Wandering Data

One of the interesting things about very popular blogs like Boing Boing is that you get all kinds of people commenting, from idiot ranters saying they could do the whole project in a weekend to the well informed and smart. So in the comments to Chunky crapgadget used to conduct the US census (kind of) [see my earlier post] you hear from people using the HTC devices and the Sybase software in the field. You can read all the comments there - some positive, some negative. Here are some of the more positive ones:

Comment #40: I’m currently a QC Enumerator (read: census worker), and to be perfectly honest, I have not experienced a single problem with my handheld. The maps (guess what) are only as accurate as they have been entered by previous census employees. It is the fault of the employee if the roads are entered incorrectly, not the fault of the GPS. In fact, part of what I get paid $17.50 to do is to correct dated or incorrect information. We don’t get to keep the handhelds at the end. They are not designed to be a workers new toy or to make you feel hip and stylin’; they are for entering information, and so far as my experience goes, they do that job.

Comment #42: …the Census Bureau is way ahead of schedule on the Address Canvassing operation. They expected to be able to canvass 1.5 million addresses a day, but have achieved 9 million.

Comment #55: …The HHC? Not bad at all — the main problem was in being able to get the You-Are-Here indicator on the GPS to display on rainy days. But GPS accuracy? Spot on. Some people had trouble learning how to use the GPS function, but that was cockpit error.

Comment #62: I would like to address some of the points made by non-Census folks.

The Harris hand-held computer: I think that the HHC does its job pretty well. Occasionally it “hiccups” during use but that is usually resolved by cycling power or doing a manual reset of the unit. Having used my HHC fulltime for the past four weeks, I think it is a pretty reliable unit. It has never lost any info. Is it overpriced? After a $700 billion bailout, a $787 billion stimulus package, and a $3 trillion increase in the national debt, how can anyone seriously ask this question?

GPS: With the built-in GPS, you can move a foot in any direction and you will see a change in the map. The GPS seems pretty accurate — I don’t think it has been off by more than 10 feet at any time.

Maps: Census mapping details and previously recorded addresses are downloaded to the HHC as needed. These maps do contain errors: Bad addresses, misnamed streets, and even street positioning errors. It is the Census lister’s job to get accurate information which includes correcting all errors in the database.

Xmit/Recv: The cell phone link (which I believe is encrypted both ways) takes maybe 30 seconds to upload a day’s worth of addresses and download any messages.

Security: The fingerprint scanner (which works reliably for me), the login password, and the two-way data encryption are necessary… In fact, we hand out a confidentiality notice to each resident who answers his/her door. As you may know, the decennial Census is mandated by the U.S. Constitution and numerous federal laws. As Census employees, we are subject to a jail term and/or fine if we reveal any information we have collected during our work. I take this subject pretty seriously, as I am sure my fellow employees do.

RIM now has an official blog, at Inside BlackBerry - The Official BlackBerry Blog.

Ethan Zukerman chats with a census worker and makes some notes about the handheld computer she is using. Cory Doctorow of Boing Boing links, and pokes fun at the chunky crapgadget used to conduct the 2010 US Census (the address canvassing part of it, to be precise). We’ll probably see more about the program on tech blogs as Census workers cover the entire US.

It’s true that HTC will win no prizes for style with this one, but there are good reasons for using a custom device if you want many cheap, rugged units and if you want privacy to be taken seriously (and you do). Plus, any project of this size takes a couple of years, so you can’t just order up tens of thousands of the latest consumer phone and hand them out. So I don’t take the blog jabs too seriously here.

The software doesn’t get much mention on today’s blog entries, but some of the software on the device comes from Sybase, as announced a couple of years ago. SQL Anywhere provides the mobile database and the data synchronization software that moves data back and forth between all those devices and the central data store, while Afaria is the device management software.

It’s been a big project, and now those folks are out all across the USA using them and it’s live, and that’s exciting.

In a comment on a post by Leena Rao, prominent tech commentator Michael Arrington says this about the new Mobile GMail.

This [the new Gmail mobile web app] is so timely. I’ve been fighting MobileMe all morning and am about to give up on desktop apps entirely. syncing is for chumps.

The irony is that the new mobile Gmail he is so keen on is, with its local data store, all about synchronization.

I collected some data from BlackBerry App World on its first day of business. There were over 900 applications, which is pretty impressive, broken down into thirteen top-level categories (Entertainment, Games, Maps & Navigation and so on).

The overall price distribution was like this (note, it’s all fixed prices - no monthly pricing):

Free:          206 applications
0 - $5:        252
$5 - $10:    268
$10 - 15:    42
$15 - $20:  80
$20 - $30:  38
$30 - $40:  12
$40 - $50:  8
$50 - $60:  14
$60 - $70:  2
$70 - $200: 2 (both near to $200)

Or, if you prefer pictures:

The prices differ a bit among the categories. Here are all the applications under $50 (which is all but two of them), grouped by category:

You can see that Reference & eBooks have some relatively high-priced titles - mainly a set of professional spell checkers in a variety of languages. On the other hand, some of the categories like News & Weather or Sport & Recreation are dominated by free applications. Some of these are little more than shortcuts to a web site, and it would be pretty cheeky to charge for that.

A few nuggets:

• Just over 20% of apps are free, which is comparable to the proportion on the iPhone AppStore, which has 30 times as many applications (24,000 according to Charles Teague, which is a little less than some other counts).
• On the AppStore, the average price of the top 100 paid apps has been dropping. It’s not clear how to get at this information for App World - I could see not web access to the data and just had to copy numbers off my BlackBerry into a SQL Anywhere database. Have I missed something? If not, I hope the data gets easier to access, because I won’t be doing this with 10,000 applications I can tell you.
• Today, I see that after a few days all the top 25 downloads are free and 9 of the 12 featured applications are free. It may be difficult for paid applications to get into these high-profile lists. If it doesn’t get easier to navigate then it will be difficult for people to make money off applications - but they have some time to expand the navigation of the store as it grows. On the iPhone AppStore, it seems that the emerging business model is of free light versions of applications with paid full versions (see Charles Teague again).
• The most expensive application is the PEPID Emergency Physician Suite, a professional point-of-care application.

I have been puzzled by the reports of Netbook manufacturers taking an interest in Android as an OS. Why would you use a phone OS for a Netbook, I wondered, when you have so many real Linux alternatives available?

Research company Ovum says that Android may have a place because Netbooks should be thought of as appliances rather than PCs. That is, the core market may be for devices which sell at $200 rather than $500, which have long battery life, use ARM rather than x86 chips, and which focus on things like fast start-up times and support for a core set of applications rather than a general purpose device.

That view does make some sense - at least in the consumer market. I can’t see it in the enterprise market, where the ability to fix up a device with just what is needed for the workforce is more important.

Still, Intel is not letting the ARM = low-power equation stand without a fight, and Ubuntu and other Linux distributors are paying more attention to low-power, netbook form factors, so while this piece makes a case for Android, it is far from a done deal. Myself, I’d bet on general purpose distributions winning over Android, but then, that view is worth what you paid to read it.

Link: Google phone OS to transform Linux netbooks, says researcher • Register Hardware
Also: at my other place, a review of Jonathan Zittrain’s “The Future of the Internet and How To Stop It”, on appliances versus general purpose end points.

Are here. The key table is in the middle of the article.

Worldwide: Smartphone Sales to End Users by Vendor, 4Q08 (Thousands of Units)

Note: Under the name HTC, Gartner counts only the company’s own-branded devices. The devices that HTC designs for mobile operators are shown separately under the operators’ names in these statistics.
Source: Gartner (March 2009)

I’ve gone through the following cycle too many times.

1. Get the data I need in a database
2. Develop the query I need to illustrate a point
3. Export the query results to a csv file
4. Import into Excel
5. Create a graph
6. Oops - new data. Go back to 1.

Well, no more. Working on a project at home the other day I came across matplotlib, a python library for graphing. Put it together with SQL Anywhere’s sqlanydb python driver and think my Excel days are behind me. Like any plotting library, it’s a bit more work to get the first plot done but then the reproducibility is great and it seems a lot more flexible than Excel’s graphs.

Here’s a Hello World, assuming that you have python, sqlanydb and matplotlib installed, drawing a scatter plot of a set of random points, normally distributed in two dimensions.

Here is a main function, showing the connection for the SQL Anywhere database:

import sqlanydb
import matplotlib.pyplot as plt

def main():
    # Create a connection object
    conn = sqlanydb.connect(userid=”DBA”,
                            password=”sql”,
                            eng=”helloworld”,
                            dbf=”helloworld.db”
                            )
    # create the plot
    scatterplot(conn)

    # close the connection, and finish
    conn.close()

And here is a routine to get a result set from the database and to plot it.

def scatterplot(conn):
    # Execute the query and get the result set into a rowset
    sql = “SELECT x, y FROM t”
    cursor = conn.cursor()
    cursor.execute(sql)
    rowset = cursor.fetchall()
    cursor.close()       
       
    # Put the x and y values into their own arrays
    cx = []
    cy = []
    for row in rowset:
        cx.append(row[0])
        cy.append(row[1])

    # Now put together a plot.
    # Most of the work is done on a subplot or axis object
    ax = plt.figure().add_subplot(111)
    ax.scatter(cx, cy)
    plt.title(’Hello World’)
    plt.xlabel(’x')
    plt.ylabel(’y')
    plt.show()

In this case the graph is a scatter plot, but many other types are available. Running this routine  displays a graph on the screen which you can save to a png vector graphic file, so it can be nicely resized. Here is the result:

Not fancy, but it does the job in only a half doze or so lines of code.  And matplotlib has many ways to do more fancy stuff, within the confines of a two-dimensional plotting library (so don’t expect 3D graphs). Here is a gallery, together with code for each, if you want to do more.

is using a mobile phone, according to the UN. The Guardian reports:

The speed and scale of the world’s love affair with mobile phones was revealed yesterday in a UN report that showed more than half the global population now pay to use one.

The survey, by the International Telecommunications Union (ITU), an agency of the UN, also found that nearly a quarter of the world’s 6.7 billion people use the internet.

But it is the breathtaking growth of cellular technology that is doing more to change society, particularly in developing countries where a lack of effective communications infrastructure has traditionally been one of the biggest obstacles to economic growth.

We are bringing our mobile database to the iPhone.

We are working on porting the UltraLite mobile database and synchronization client to the iPhone and, while we have no formal announcements yet, you can now sign up to get information about this project as we get closer to release. If we go through an early-experience program we will let you know about that too.

If you are interested in the possibilities of UltraLite on iPhone, please go to http://www.sybase.com/sqlanyiphone and sign up. We’ll keep you in touch.