SilverLight and Paging with Azure Data

If you’ve been watching by blog at all lately, you know that I’ve been playing with some larger data sets and Azure storage, specifically Azure table storage. Last week I found myself working with a SilverLight application to visualize the resulting data and display it to the user, however I did not want to use the ADO.NET Data Services client (ATOM) due to the size of data in transmission. Consequently, I set up a web role that proxied the data calls and fed them back to the caller as JSON. Due to the limitation on Azure table storage of only returning 1,000 rows at a time, I needed to access the response headers in my SilverLight client to determine after each request if there were more rows waiting… and that was the rub… every time I tried to access the response headers collection (tried both with a WebClient and HttpWebRequest), I received a System.NotImplementedException.

I pounded my head on this for a few days with no success until a helpful twitterer (@silverfighter) provided me a link that got me rolling. The root of the problem was my ignorance of how SilverLight’s networking stack functioned. As I (now) understand it, by default any networking calls (WebClient or HttpWebRequest) are actually handled by the browser and not .NET. This results in you getting access to only what the browser object hands you, which in my case, did not include the response headers.

The key here is that SilverLight 3 provides you the ability to tell the browser that you’d rather handle those requests yourself. By simply registering the http protocol (you can actually do it as granular as a site level) as handled by the Silverlight client, “magic” happens and you suddenly have access to the properties of the WebClient (ResponseHeaders) and HttpWebRequest (Response.Headers) objects that you would have expected to. The magic line you need to add prior to issuing any calls is as follows:

bool httpResult = WebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);

(yes… that’s it…)

The links to the appropriate articles are as follows:

http://blogs.msdn.com/carlosfigueira/archive/2009/08/15/fault-support-in-silverlight-3.aspx 

http://msdn.microsoft.com/en-us/library/dd470096(VS.95).aspx

http://blogs.msdn.com/silverlight_sdk/archive/2009/08/12/new-networking-stack-in-silverlight-3.aspx

Silverlight and Azure Table Data Paging

I’m playing around with a data visualization app using Silverlight and data hosted in Azure Tables and have been learning quite a bit in the process. Firstly, Azure tables only allows you to return 1000 records in a given query. If you issue a query that has a larger matching result set, Azure will return some extra headers indicating as such (x-ms-continuation-NextPartitionKey and x-ms-continuation-NextRowKey). It wasn’t hard to find an example of data paging using Azure table data, however it used the Execute() method of the DataServiceQuery object. Unfortunately, this isn’t available in Silverlight as you have to use the asynchronous methods (BeginQuery and EndQuery). I’m a bit slow, and for whatever reason translating the MSDN sample for synchronous to the asynchronous model took me longer than it should have. I’m posting this so that maybe the next person will find this, get the answer they need, and move on and not waste the same amount of time I did.

My button event handler looks quite a bit different from the MSDN sample but is pretty easy to figure out:

I instantiate the context, create a query based on that context, cast that query as a DataServiceQuery<t> and then call the BeginExecute method passing my callback method and the query as my state object. (Note: in case you are wondering about the Where clause in the query above, I know that all of the data that matches the first conditional is located in a given partition within the Azure table and have found that specifying the partition greatly increases the performance).

My callback method (ProcessDataRequest) does a bit of recursion to support the unknown number of subsequent calls needed to retrieve all of the matching records.

The contents of the ProcessDataRequest method are listed below:

Note that unlike some samples that are simply focusing on async data calls and don’t handle paging via headers, I cast the output of EndExecute as a QueryOperationResponse object which allows me to subsequently access the headers and interrogate them for the continuation keys. If I find the continuation keys, I create a new query object, set the additional query options, and execute it in the same fashion as the original call.  The AddPointsToScreen method simply processes the values and renders them as polygons to the screen. I’ve included it here not because there is anything special in it, but rather for completeness.