Wandering Data

A rotating collection of links called Carnival of the Mobilists has been running for a couple of years now, pulling together the best in mobile blogging. I’m happy to say that this blog appears in its 206th edition, no less, hosted by mobile marketing company mobiThinking. You can find the carnival here you can find a set of posts that are, not surprisingly given the time of year, looking forward to see what’s coming next in the worlds of mobile computing.

This piece was written by Ben St. Pierre, a co-op student from the University of Waterloo who spent a very productive term with us in the fall. He wrote this in December, just before taking off for the wintry north of Ontario. Thanks for a great term Ben.

---

With the popularity of smart phones on the rise, and the ever increasing consumer hunger for bandwidth, the sustainability of the mobile data network has come into question. Recently, AT&T has come under fire for its lackluster wireless 3G coverage in the two most heavily used wireless data networks, San Francisco Bay and Manhattan [link]. Is this the beginning of the mobile data apocalypse, or will carriers and consumers adapt?

The Current Trend

Earlier this year, Cisco published a white-paper which presents their mobile traffic projections and growth trends [link]. They project that mobile data traffic will double every year, increasing 66 times over from 2008 to 2013 with Latin America having the strongest growth, followed by Asia-Pacific. This growth will mostly be driven by mobile video streaming and carrier-subsidized 3G-equipped laptops and netbooks [link].

This growth may seem wonderful at first, but the network infrastructure is not equipped to handle such a large jump in usage and the projected profits from this usage are expected to grow at a much slower rate [link].

To make matters worse, FCC chairman Julius Genachowski is warning of a “looming spectrum crisis” [link]. Available spectrum is scarce, and reassigning spectrum is a process that takes years to implement.

Mitigating Factors

Although the current projections show unsustainable growth, some factors may help keep mobile data technologies in the hands of the mass-market.

Public Education

AT&T has cited that 3% of their smartphone users are currently consuming 40% of the network capacity [link]. This seems to be due to the lack of education of their users. When contacting these customers, AT&T found that they were using their smartphones by streaming audio around the clock or watching large numbers of streaming videos, both of which consume excessive amounts of data.

The End of Unlimited Data Plans

The easiest way for carriers to educate their users is by giving them a comprehensive data plan. Unlimited plans make it easy for users to abuse the network since they don’t need to be aware of their usage. If users need to be aware of how much data they are using due to usage-based plans, excessive use would likely be less of a problem. Carriers have already hinted that unlimited plans are not sustainable and that in the long term, a switch to usage-based plans is likely [link].

Long Term Evolution (LTE) and 4G

4G wireless technology promises faster speeds and better scalability than the current 3G networks. The FCC has a four-part plan to help ease the deployment of 4G which includes unleashing spectrum and removing obstacles for ubiquitous 4G deployments [link]. However, the projected bandwidth requirements still exceed the capacity of 4G technology [link] and a widespread deployment is still years away from available.

Femtocells

Femtocells are small network appliances (similar to network routers) which use a 3G antenna and a regular broadband internet connection (such as DSL or cable) to connect 3G wireless devices to their mobile carriers. AT&T has already released a limited number of 3G Femtocells as a test program which supports up to 4 wireless phones in a residential setting [link]. Sprint-Nextel also has its own femtocell offering through its AIRAVE product [link].

The next generation femtocells are also already in development. LTE groups have indentified femtocells as a priority item for 4G deployment. This could increase early 4G adoption by consumers who wish to get 4G handhelds in areas where 4G is not yet available and thus help alleviate 3G network congestions.

Wifi

Many consumers already know that a Wifi network is preferable to using 3G. They know this due to the visible speed difference in bandwidth-hungry applications. This has lead to consumers looking for wireless devices which also includes a Wifi interface. In a recent study, it was found that 77 percent of mobile phone users want Wifi on their next handset and 75 percent of users who already have Wifi on their phone use it regularly [link] The same study also projects that 90 percent of smartphones will have Wifi by 2014, up from the current 44 percent.

Conclusion

Because of the enormous expected growth, many opportunities have arisen for businesses to solve and although the traffic projections look apocalyptic in growth, there are many mitigating factors which may lessen the impact. Whether or not these factors will be enough to keep the mobile data networks running smoothly however, has yet to been seen.

Not only are our colleagues in the Afaria team releasing their Afaria for iPhone product, but today the SQL Anywhere for iPhone Beta Program is open.

It’s been a long time since we announced that we’d be providing SQL Anywhere software for the iPhone but today, after a long and winding road, we are finally launching the SQL Anywhere for iPhone Beta program. If you register here, then you’ll be up and going quickly. Anyone who registered previously will hear from us today. The official announcement is here.

Here are some questions and answers about the release.

Q. What are the key features of SQL Anywhere for iPhone?

• Data synchronization between device and enterprise. With SQL Anywhere for iPhone, developers can build rich applications that use a local database on the device and that share data between the device and a server.
• Integration with enterprise databases. Developers can build applications that share data between the iPhone and relational databases such as Oracle, Sybase ASE and SQL Anywhere, IBM DB2, Microsoft SQL Server, and MySQL.
• Integration with enterprise systems. Developers can build applications that share data between the iPhone and enterprise systems.
• A common infrastructure for multiple devices. SQL Anywhere already supports Windows Mobile and BlackBerry. With iPhone support, SQL Anywhere now supports the three major smartphone platforms for business applications.

Q. What are the key benefits of SQL Anywhere for iPhone?

Businesses looking to provide line-of-business applications to their workforce can bring those applications to the iPhone using the same SQL Anywhere technology deployed on hundreds of thousands of other mobile devices.

Application developers looking to sell line-of-business applications can now use a single platform for all three major smartphone platforms for business applications.

Q. How is this different from other SQL Anywhere offerings? Why?

SQL Anywhere for iPhone brings most of the data management and data synchronization features available on other mobile platforms, such as Windows Mobile and BlackBerry, to the iPhone.

SQL Anywhere for iPhone includes the UltraLite database management system and synchronization client. This is a port of the UltraLite database for Windows Mobile already used in many enterprise deployments: the features and implementation are the same between the two platforms, so customers can be assured that this is already a mature and well-recognized technology.

At the server, a single SQL Anywhere MobiLink server can serve clients on all kinds of mobile devices, including the iPhone. The same set of enterprise systems are supported across all mobile devices.

To develop applications for the iPhone using SQL Anywhere, you need a Mac and a copy of the iPhone SDK. When it is reaches general availability, SQL Anywhere for iPhone will be distributed as part of SQL Anywhere for Mac OS.

Database encryption on device is supported, but encryption between the iPhone and the server is not currently supported. Push synchronization is not currently supported.

While UltraLite on Windows Mobile can be run either in-process or as a separate process, only the in-process version is available for iPhone, as iPhone does not support multiple processes running at the same time.

Q. How will developers deploy applications that include SQL Anywhere for iPhone?

Developers compile SQL Anywhere functionality into their application, and then deploy their application in the same way as any other application – through the App Store or through the same enterprise deployment mechanisms Apple recommends for other mobile applications.

Q. What iPhone versions does SQL Anywhere support?

This release supports iPhone version 3.1 and back as far as version 2.0.

Q. What version of SQL Anywhere is needed?

To synchronize data between the iPhone and a server, developers must have the SQL Anywhere 11 MobiLink synchronization server.

And here we are…

View Larger Map

Two or three years ago, the state of the art CPU for mobile handsets was the PXA270, an XScale ARM processor originally from Intel and then sold to Marvell. It chugged along at 625MHz or so.

Nowadays, we are far from that. Today’s cutting edge phones and mobile devices use… oh right, the same chip. Yes, for about three years, Moore’s law has taken a break when it comes to mobile devices. I don’t know enough about hardware to know the reasons – is it something about the chip architecture? is it having to deal within the limits of the crappy battery technology we are plagued with? Well, I don’t know, but it certainly puts a crimp in mobile software.

But maybe things are finally changing. A couple of weeks ago Motorola released their latest round of enterprise handhelds (the old Symbol line), and the MC9500 is based on the Marvell PXA320 chip, which runs at 806 MHz and yet which apparently has better power consumption than the last lot. And now it looks like faster chips may be reaching consumers as the Acer A1, a new Android-based phone, sports a Qualcomm 8250 processor running at 768MHz. So it looks like the 625MHz barrier is finally broken, and that’s good news for mobile computing.

So these new devices will scream. At least, pre-release versions of the Motorola one did. It has a built-in accelerometer and emitted a scream when dropped, as a safety feature. Unfortunately, according to the Register, that just made it too much fun to drop so it was kind of counter productive. Production units will be boringly silent when they crash, unharmed, to the ground off the top of whatever surface you want to knock them from.

SQL Anywhere is Sybase’s mobile database, and UltraLiteJ is included in that product as a mobile database for the BlackBerry.

We’ve been working for some time to get UltraLiteJ to support the internal flash (device memory) and SD Card stores on BlackBerry devices, and now I’m happy to say that it’s done and available. Right now, existing SQL Anywhere users can get this technology by downloading the latest Express Bug Fix (EBF) release. We’ll be upgrading the free SQL Anywhere Developer Edition download soon and I’ll post a link when that’s done, but for now you have to go the EBF route.

As this technology is being provided in an EBF we haven’t updated the documentation, so here is how you use it. In this post, I’m assuming you know how to use UltraLiteJ already.

First, get the software. Assuming you have SQL Anywhere installed, click here for the EBF (you need to sign up for a free Sybase account). Click SQL Anywhere, scroll down to the Windows x86 platform, and click “SQL Anywhere – Express Bug Fix, Update of 11.0.1 to build 2308 (all components)”, or any later build as they become available.

Once you’ve installed it, you will see that under the SQL Anywhere\UltraLite\UltraLiteJ folder there is now a BlackBerry 4.2 folder. That’s the folder containing the JAR and COD files you need.

There’s not a separate JAR or COD file for the flash memory or SD Card support, the same software can access databases in the persistent store and in file-based locations. It works by providing a new subclass of the ConfigPersistent object, which is what you use when connecting to or creating a database. The new class is ConfigFileME, and it inherits many of the same methods that ConfigObjectStore does, so anything you have about cache size and so on remains the same. The only difference is that when creating the configuration object the database name is a bit different.

For a database in the persistent store (all we have had until now) you would do something like this to connect to a database, using a string (no path of course) to specify the database in the persistent store:

String dbName = "my_persistent_store_database";
_config = (ConfigObjectStore)DatabaseManager.createConfigurationObjectStore(dbName);
Connection conn = DatabaseManager.connect( _config );

For a database in the internal flash (device memory) you would do this instead:

String dbName = “file:///store/user/home/my_file_database.jdb”;
_config = (ConfigFileME)DatabaseManager.createConfigurationFileME(dbName);
Connection conn = DatabaseManager.connect( _config );

And that’s it. Once you have a connection, all the other database operations are exactly the same as if you were using the persistent store. Now here are a few more details, stolen from my colleague Philippe Bertrand:

Databases names must follow the format for a fully qualified, absolute path file name as described in the file URL format as part of IETF RFCs 1738 & 2396. That RFC dictates that a file URL takes the form:

   file://<host>/<path>

In the form above, <host> is the fully qualified domain name of the system on which the <path> is accessible, and <path> is a hierarchical directory path of the form
<root>/<directory>/<directory>/…/<name>.

For BlackBerry devices, the internal flash is accessed using URIs starting with file:///store/ (case sensitive) while the SD media card is accessed using URIs starting with file:///SDCard/ (case sensitive).  File paths are case insensitive except for the host portion which is case sensitive.  Percent (%) characters have special meaning and relative paths (directories "." and "..") are not allowed.  For more information on file name restrictions, please consult the BlackBerry JDE (4.2 or later) API Reference for the javax.microedition.io.file package.

You will find that, for data access, persistent store is faster than internal flash and internal flash is faster than SD cards.  On the other hand, databases stored on internal flash or media cards can keep more rows in memory and/or use a bigger cache to improve performance.

Chris Brandsma at The ElegantCode blog reports some performance metrics on Microsoft SQL CE, SQLite, our own UltraLite and McObject Perst. The write up is here: Mobile DB Performance Testing. It’s good to see people sharing their experiments with a broader community in this way – thanks Chris. [Update: the write-up is now here and is done on devices].

Performance testing and benchmarks are tricky. Like other vendors we don’t publish our own comparative performance tests. That’s not because we want to hide things, it’s because license terms mean that we are not allowed to report performance tests on competitors’ databases. And it makes some sense because after all, we know our own software so we’ll probably get more out of it. And that assumes we’re honest – which we are of course, but how are you to know that?

So I’m not going to say much about Chris’s numbers at ElegantCode, just some things about the methodology. Chris does give a fair set of disclaimers, which you should look at too. I will say that the results of our own tests are different. For example, on most tests I’ve done UltraLite comes out quicker than SQLite for inserts, although slower for queries. And SQL CE comes out quicker than Chris’s numbers suggest. I haven’t done Perst testing, so I can’t comment on that.

The likely reason for much of the difference is that Chris’s numbers are recorded on an emulator rather than a device. The comparative CPU to I/O speeds are quite different, so you end up seeing different trends and there are different bottlenecks on an actual device.

There are some other things to look out for too if you do your own performance testing. Here are a few things to keep in mind.

• Cache: a lot of database performance is about avoiding I/O. The more data you have in cache, the better. The default cache size of SQL CE is 640KB; for SQLite it’s 2MB (2000 1KB pages); for UltraLite it’s a more conservative 250KB. This may explain the slow performance Chris sees on a join query – with a small cache you end up swapping rows in and out of the cache. You can change any and all of these cache settings around, and it will make  a difference. It’s worth setting them to the same value. SQL Anywhere, by the way, uses a dynamic cache so if there is spare RAM on the device it will use it, and if other applications need the space it gives it up. It’s the right way to do things.
• Commits: the default behaviour on SQL CE when you commit data is to do a “logical commit” which means that you can rollback to this point, but the changes are not actually written out so if your application crashes then you may lose your data. If you use UltraLite a commit is a physical commit, which takes longer but is more secure. Again, you can change the behaviour of both databases to mimic the other, but to compare apples with apples both should be set to the same.
• Change tracking: UltraLite is intended as a sync client for our MobiLink synchronization technology. That means it does change tracking for every operation you carry out on the database. If you want to synchronize a SQLite database you are going to have to add your own change tracking, and that’s going to slow it down and to add to the size of the database file on disk. And by default I believe SQL CE has change tracking turned off as well.

Put those together with the results Chris reports and I think you can see why we feel fairly good about UltraLite’s performance. But of course, what else would I say?

I keep a graph of Gartner’s quarterly smartphone sales numbers. The latest  numbers from are here, for Q2 2009. They are reported in a slightly different way. so some of the lines can’t be extrapolated easily. It’s almost as if they don’t care about my graphs. Anyway, here is what they tell us about the smartphone market by operating system.

The big news is that Microsoft (green), whch was in 2nd place two years ago and in third place one year ago, is now clearly in fourth place.
The big Symbian numbers do raise a question as to whether this is all one market or not. The New York Times recently ran a fascinating article about why Japan’s phones, which are way ahead in terms of hardware, have had so little success getting exported. Maybe the world I see from southern Ontario is a distorted one, but for 3rd party developers it seems like Symbian devices are different – used different ways, and with different user expectations – to the BlackBerry/Apple/Microsoft devices. The Android numbers are included in Linux in earlier reports: for this report Gartner mentions that Android sales are about 2million (not shoddy, and should be much better soon) but it hasn’t broken them out from the overall Linux numbers (which it does not report this time around).

Via BlackBerry Cool.Maybe this is not too surprising. RIM just announced the end of life of their MDS Runtime and MDS Studio development tools. The announcement is here. A discussion thread for developers is here and includes this comment, in response to a “What do MDS developers do now?” question:

What I would suggest is to get started mocking up your UI in HTML and get the flow of your application from screen to screen.  HTML will give you much more freedom on how you want to layout and interact with your UI than the MDS Runtime solution.  Also start looking at how you can leverage XMLHttpRequest (AJAX) to submit and request data in the background.  We also support push data with BlackBerry widgets in the exact same manner as pushing data to a Java application.

For your data storage needs you can start looking at using the Gears Database API.  This API is a full SQLite implementation for storing your data offline.

Also start thinking about which portions of your application you want to have in the Client and which make sense to stay on the server.  BlackBerry widgets will allow you to “mash-up” both local and server side content in your widget as long as you whitelist the domain.

Also, if you are still wanting to target older devices, MDS Runtime is your solution.  BlackBerry widgets will work on OS 5.0 and up

Of course, a better solution for local data storage would be to use UltraLiteJ. Just saying.

Demand for Nand – Get ready for more memory in your smartphone. As Apple’s 3GS and now Nokia’s N97 put 32GB of flash on their phones, eWeek reports that the demand for NAND is skyrocketing. Quote from iSuppli’s Michael Yang: “The more NAND in a smartphone, the more useful it becomes,” said Yang, pointing out that it enables the device to “store more songs and video clips, to hold more map data and download more programs from an application store.” Yay!

No More Goople? – David Pogue’s article in the New York Times about Apple’s refusal of the Google Voice application makes a lot of sense. Here is the end of his article:

In short, what Apple and AT&T have accomplished with their heavy-handed, Soviet information-control style is not to bury these useful apps. Instead, Apple/AT&T have elevated them to martyr status—and, in effect, thrown down a worldwide challenge to programmers everywhere.

“Get around THIS,” they’re saying.

But guess what? It won’t take long. They’ve put a rock in the river, but the water will just find a way around it.

Already, Google says it is readying a replacement for the Google Voice app that will offer exactly the same features as the rejected app—except that it will take the form of a specialized, iPhone-shaped Web page. For all intents and purposes, it will behave exactly the same as the app would have; you can even install it as an icon on your Home screen.

What’s Apple going to do now? Start blocking access to individual Web sites?

No question about it: the next chapter has yet to be written. But I think you’re going to like it.

Open is the New Closed – Andreas Constantinou at Vision Mobile has am interesting take on open source in the world of mobile computing, focusing on the different governance models for open source projects, rather than on the more commonly-discussed licensing models. The paper by Wes and O’Mahony that he draws on is full of much more detail. Constantinou does a good job of taking their general considerations and applying them to the world of mobile computing. The fact of closed phones rather than open PCs gives the dynamics of mobile a twist, as Jonathan Zittrain and others have recently pointed out.

Just in case anyone is reading this who has dropped their phone in water: Popular Mechanics has some ideas on how to recover it. Of course, it only talks about water.