Return of the Windows Azure GAC Viewer

I’m pleased to announce that the excellent utility – the Azure GAC Viewer – is once again online and available for general use. You can access it at http://gacviewer.cloudapp.net. This tool shows you a dynamically generated list of all of the assemblies present in the GAC for an Azure instance. Additionally, it also allows you to upload your project file (*.csproj or *.vbproj) to have the references scanned and let you know if there are any discrepancies between what you are using and what is available (by default) in Azure. You can then adjust your project file (copy-local=true) to ensure your application can run successfully.

If you are familiar with the tool, you may be thinking “Wait! you aren’t Wayne Berry, and besides, the URL has changed!” – and you would be correct on both counts. Wayne developed the tool and posted about it back in September of last year. Since that time, however, Wayne has accepted a position on the Windows Azure team and is unable to continue to maintaining the site full time. As a gesture of kindness to the community, he has passed the source code to me and given me his blessing to re-launch the tool.

As it stands today, the tool is nearly exactly as Wayne developed, with a few tweaks to have it use Guest OS 2.1 rather than 1.6. I’ve also added a contributors page to give credit to Wayne and to the organizations that are allowing me to maintain and keep the site online.

In the future, I hope to make the source code available on CodePlex as well as to add to the list of tools that live on the site. If you have any bugs with the current site or ideas for future changes, please feel free to contact me.

The Danger with using a framework…

The danger with using a framework is that sometimes it does things that you aren’t aware of that can send you in circles for  quite some time before you figure them out.

I’ve been working on some tests of some “creative” ways to get data in and out of cloud platforms at rates above the norm and I’ve written a test harness that I’ve been using that will grab a bunch of files, one at a time, and record the file size, duration, etc. for the transfer. I’ve been doing this in a single-threaded fashion for quite some time with reasonable success. The problem began when I attempted to use to run a test that did multi-threaded downloads (multiple threads each grabbing a portion of the file).

NOTE/Crazy Quirk: I don’t yet know why this is the case, but the problem I’m preparing to explain did *not* appear while I had Fiddler running… only when it was *not* running. I’m guessing that this is due to some “magic” that Fiddler does to the HTTP/networking stack..

The behavior I was seeing, was that after two threads would execute, all subsequent threads would fail or timeout. Obviously, when one is doing a significant amount of data movement, this is sub-ideal. The culprit turned out to be the ServicePointManager’s DefaultConnectionLimit. By default, this is configured to 2 which means you can, at most, have 2 open connections to the same TLD at the same time. When I was doing this in serial, there was no problem as the connections were managed/re-used on the main (only) thread.  When doing a number of operations to the same URL (TLD) from multiple threads (especially when you are setting up/tearing them down quickly), it appears that the ServicePointManager is unable to re-use them (not surprising) but neither is it able to determine that the thread is now gone as should be the connection count. (yes, I was behaving and closing my connections).

The solution I came up with was to first shorten the time to live for idle threads, next to monitor the number of threads currently “consumed” and to increase the limit based on how many I needed for the current operations, all while ensuring an upper bound and stand-off mechanism should things get too far out of bounds.

// ensure that we don't have lingering connections that will hamper our 
// ability to continue...
// Start by getting the ServicePoint for our current Url 
ServicePoint servicePoint = 
    ServicePointManager.FindServicePoint(new Uri(url));

// see how many connections currently exist...
int existingConnections = servicePoint.CurrentConnections;

// if we are above our upper bound, wait a bit to let things settle down...
while (existingConnections >= 64)
{
    Console.WriteLine("Connection count too high. sleeping for a bit.");
    Thread.Sleep(1000);
}

// ensure that we have enough room to do what we need
if ((existingConnections + options.ConcurrentThreads + 1) > 
    servicePoint.ConnectionLimit)
{
    servicePoint.ConnectionLimit = existingConnections + 
        options.ConcurrentThreads + 1;
}

// only give them a few seconds (5) to time out...
ServicePointManager.MaxServicePointIdleTime = 5000;

Console.WriteLine("Pre-Existing Connections: {0}", existingConnections);
Console.WriteLine("Connection Limit: {0}", servicePoint.ConnectionLimit);

Hopefully, this will be helpful for someone else hitting the same issue.

Automated Chart Generation

It’s late on the Friday afternoon before Christmas week which means things are pretty quiet around the office. This quiet has the net-effect of allowing me to get quite a bit done. The last few days have been very productive with respect to our research project and Azure work (more on that coming soon) which is now in full swing. We are currently working on collecting performance data from our codes running in Azure (and soon in the Amazon cloud) and are also doing some testing of transfer speeds of data both to/from the cloud as well as between compute and storage in the cloud.

I’ve been working to automate much of this testing so we can do things in a repeatable fashion as well has have something that others could run (both other users like ourselves as well as possibly vendors should we come across something that requires a repro scenario). So far, running tests and generating data in CSV or XML format is pretty simple, but I found myself wanting to automatically generate charts/graphs of the data as part of the test process to allow a quick visualization of how the test performed. I spent a good bit of the day looking at old tools for command-line generation of charts (i.e. RDTool, etc.) and none of them were exactly what I was looking for – not to mention my proclivity to using C# and VS.NET tools and my desire to have something that looked refined/polished and not overly raw.

Thankfully, I stumbled upon something I should have remembered existed but simply hadn’t had the need to use before – the System.Windows.Forms.DataVisualization.Charting class. If you aren’t familiar with this assembly, it was released at PDC08 and has a companion Web class for performing similar operations in ASP.NET applications. In my basic testing I was able to build a console application that would ingest the CSV output from my testing harness and then generate some fairly nice looking charts based on that data. The following shows a chart (click the chart to see it full size) generated from ~1800 data points, and automatically generates a 50% band and 90% band allowing the viewer to very easily ascertain the averages and data points. This was generated using a combination of the FastPoint and BoxPlot chart types.

Web Service Studio

Those of you who interact with and develop XML web services on a regular basis are probably very familiar with this tool or have other tools that you have purchased/developed to provide similar functionality, but I came across the Web Service Studio (http://www.gotdotnet.com/Community/UserSamples/Details.aspx?SampleGuid=65a1d4ea-0f7a-41bd-8494-e916ebc4159c) last night while working on a little web service project and I think it is great… It is straight-forward in its operation, and doesn’t have alot of “jazz” to the UI, but it does what it is designed to do… which is providing you with a means of interacting with Xml web services in an ad hoc fashion.  I used it primarily as a testing tool, to validate that the web services I was creating worked as they should.  There are many other tools that one could use for performing such a validation… the built-in test page (only works on the local machine), building a custom client such as a winform app (this works, but requires a custom build for each web service which is hardly efficient), using a tool such as InfoPath to quickly build a test form (again, very effective, but requires a new form for each web service).  Web Service Studio allows me to simply point to a url after which it will read the WSDL, enumerate the methods available to me, generate the proxy, and let me test/probe each request right there… better yet, it allows me to see the raw XML request being sent back and forth for each “post” so I can see what the request/response needs to look like for non-webservice-friendly (i.e. you have to hand-craft the XML) SOAP clients.  Anyway, if you are working with Xml Web Services on a regular basis and need a simple (and free) testing tool, this tool is definitely worth looking at.

VisualStudio.Net.2005.Beta2(firstExperience) == Really.Good;

Wow… early this morning was the first time that I was really able to spend some time with the new Visual Studio 2005 beta… and let me join the majority of the blogsphere in saying that I’m quite impressed.  My task today was a simple one… that of converting a primarily content-focused website that I had build using Dreamweaver MX to ASP.NET 2.0.  The following are some general impressions…

1. Vault and the beta work great together…. We are actually running an older version of Vault (2.06) yet it still connected up and worked like a champ… in fact, I was amazed at how fast the source control operations (checkin/out) were compared to RTM version of VS 2003 and the same Vault environment… a very nice start…

2. Project-less websites are great … no more adding files to the solution/project… VS simply “knew” about all of the files in the directories I was working on… it is very nice.

3. Master pages “just work”… As someone who has used the templating feature in Dreamweaver, and has also built an ASP.NET master-pages “engine” (capitalizing heavily on the work of Rob Howard and team, supplemented by Paul Wilson) I wasn’t “amazed” by the master pages feature… I just wanted it to work better/faster than my current tools and hopefully have a better implementation.  I was not disappointed… the Master Pages support was easy, and quick to utilize…. now if only they could get nested master pages to work in the designer…

4. XHTML compliance… finally… Some people probably don’t care so much about standards compliance, but I’ve been working pretty hard lately on some of these sites to make them XHTML compliant… this site in particular had been compliant until I added some ASP.NET controls (text boxes, buttons, validators, etc.).  I moved things over this morning, fired up the page, ran the W3C validator (thanks to the Checky extension for FireFox) and it validated (XHTML Strict) on the first run… very impressive

5. System.Web.Mail is deprecated… most of you all probably knew this already… and yes, I probably have had my head in the sand, but I was surprised when the compiler griped at me this morning about my form that sends mail based on user input… I took some time and without too much work was able to get the form processing code using the System.Net.Mail namespace which seems to be the new “right” way to do it… works like a champ.

I’m very excited… don’t have time to talk about the built-in webserver, or direct FTP support… which is great….  I’ll post more later.

So... InfoPath IS useful....

One of the unfortunate realities of being a busy consultant and developer is that your exposure to technologies and tools are often limited to what you have to have to use to get your job done, or what your client needs and/or is interested in.  Obviously, you work hard to keep yourself exposed to as much and varied technology as possible, however some things invariably slip through the cracks.

Once such technology for me, as been Microsoft Office InfoPath.  I've played with it a few times, spent 30 minutes here and there, and never had much success... so I basically wrote it off as not applicable to me... a sentiment that is rapidly changing...

I'm currently on a business trip in the UK so I spent the weekend reviewing our company's client-facing Extranet (WSS based) and the various features we either offer or are planning to offer.  After doing a bit of digging, and a little experimentation, I now have our extranet site extended with a InfoPath driven status report form library... I still have a number of additional thinks I'd like to do (i.e. basic workflow, approval, etc.) but I'm pretty impressed with the ease with which it all came together. 

My next task is to get InfoPath working as a quick and dirty tool for testing XML Web Services as part of client projects... should be easy enough...

I'd be happier with the product if it didn't always require the client, but the richness of the client experience sort of explains those requirements.