30 Nov 11 Ramblings on a Private Cloud (part 1 of ?)
As many of you are aware, a few months ago I changed jobs (more like positions) and with that change has come a shift in the focus of much of my work. I’m spending less time in the abstract (i.e. “how could we solve problem X" in the cloud”) and working more in the concrete (“we have agency Y that has problem X that needs to be deployed in the cloud yesterday… what do you think?”). One unfortunate (at least in my way of thinking) is that many of these projects have legislative requirements that prohibit the use of public cloud platforms (at least currently). While I’m always looking for ways to avoid building individual, private “Cirrus1” clouds, and I remain hopeful that the day will come when we all can leave the infrastructure build-outs to someone else, the reality is that in today’s world there is some data that simply cannot live outside of an organization’s boundaries – leaving us to look at “private cloud” approaches.
Overview
We have been experimenting with a few different approaches for some of the projects I’ve been working on and are learning a number of interesting things. I want to be quick to say that I don’t think we’ve “arrived” by any stretch of the imagination and our work (and this space in general) is changing/evolving rapidly. Our initial requirements included using free/OSS solutions wherever possible and to, as much as possible, avoid vendor lock-in (always great in theory, frequently falls apart in application). The hardware used for this exercise includes 45 physical nodes spread over two racks with a total of 1160 cores, 3.8 TB of RAM, 200 TB (RAW) of local node storage and 288 TB (RAW) of iSCSI-connected storage. Our plan is to provide 8 VLANS across the environment (Management, Dev, Test, Prod + corresponding disk-traffic networks for each). The physical nodes/nics would all default to the management network and all of the actual compute resources (VMs used by researchers) would live in one of the other three zones (Dev/Test/Prod). For reasons that I won’t get to in this post, we standardized on CentOS for the primary OS and Xen as the target hypervisor. While some of the uses of the platform include data serving (i.e. web sites, data services, etc.), most of the workloads will tend to be heavy data analytics. The above scenario is complicated by the fact that the entire environment needed to be air-gapped (i.e. not connected to the Internet or other networks in any way).
Open Stack
While we have a good bit of experience using Eucalyptus on the Ubuntu stack, our initial plan was to go with the current wisdom of the crowds and deploy OpenStack as our cloud orchestration layer. The intent was to buy in fully and deploy their storage, image, and compute services. Unfortunately, while there are some very interesting things happening in this realm, we didn’t find this toolset to be at the level we needed it to be given our platform selections. After a few weeks of trying to get this working, we bailed due to simply needing to get something working2. We are hopeful that this situation will evolve in the future and we will reconsider down the road, but for the present we had to scrap it and move on.
Refactoring / Current Approach
Having burned a bit of our grace period, we were faced with the need to get something running fast and spent a weekend digging through our options. Where we ended up is using the free XenServer (via Citrix Systems) as the host OS in combination with XenCenter for managing the nodes. This – while not a cloud or cloud orchestration layer – allowed us to quickly meet some of our sponsor’s needs while buying some time to fill in the gaps. Our team is currently evaluating CloudStack as the cloud/orchestration layer to sit on top of XenServer and be the researcher-facing interface to the platform. Our hope is that as the story evolves (Citrix, XenServer, XenCenter, CloudStack, OpenStack, etc) that the deployment of future platforms will become easier and the “best” approach will become clearer.
XenCenter management console. Image credit Citrix Systems (http://community.citrix.com/display/xs/XenCenter)
Notes
1. I spent a good bit of time looking on the web for the official name of the tiniest cloud but didn’t come up with anything better than “Cirrus” which is defined as a “thin, wispy cloud.” Not exactly what I was looking for, but I’ll use it for now.
2. Our issues included (among other things) Python version conflicts between what OpenStack needed and the version of Centos (5.6) we were running, the lack of a good Web UI/self-service portal, getting the VLANS talking properly between hosts, vhosts, storage, etc., and a number of smaller miscellaneous items.
20 Oct 11 Windows Azure Development Cookbook
I should admit that I didn’t pay much attention to the front matter or explanation of the book and just dove right in. I mention this only because it was a bit jolting due to the fact that (as could easily be gleaned from the title) this is a cookbook. This means that there is not a lot of un-necessary ensemble, but rather a collection of highly focused technical nuggets. While this structure became obvious rather quickly, I decided to continue on and read it straight through just to see what I learned.
I appreciated the fact that the book was devoid of a large section of text dedicated to the now-worn-out question of “what is cloud computing”. Nor was there any prologue describing Windows Azure to be found. Instead, the assumption (I presume) is that if you’ve picked up the book, you likely know the answer to both of those questions (within reason) and simply need help getting past some of nuances of the platform. If this describes you, this book is for you.
Light on fluff, heavy on details, this is a solid book that deals with a number of real-world issues using the Azure platform. This book works great as a reference tool: have a problem, look it up in the index or table of contents, read the recipe, put it back on the shelf.
One of the things that impressed me about the book was Neil’s work to point the reader to external resources. There were a number of places where there is something along the lines of “for a more detailed explanation of topic X, visit person Y’s website at http://….” [and, in case you are wondering, this comment was not influenced by Neil’s excellent external references on blob storage interactions… at least not much] Further, I thought that the pointing of the reader to external tools and libraries that were not necessarily required to solve the stated problem but add significant value to the actual solution was great (such as the library for handling connection failures when working with SQL Azure and AppFabric). It is attention to detail such as this that gives the reader confidence that the author wasn’t just pounding out tasks to meet a deadline but rather was sharing solutions that he had used to solve real-world problems.
Taking a more critical view of the book, I’d mention just a few things. The first is that there are a number of key points that begin with “Note:” or something similar that have key tips that are very important to the success of the recipe however (at least in the eBook version I have) they are easily lost in the rest of the text. This is likely due to the format/structure of the book and the intention is for you to read one recipe end-to-end and be done rather than reading start to finish as I did, but I would encourage the reader to be sure to read the entire recipe text and not just copy/paste the code. Neil often uses the code to teach concepts and if you just copy the code you will miss this instruction.
My second criticism is that there are a number of places in the text where the author says something along the lines of “xyz is related to this. See the Using XYZ recipe for details”. While not possible in the print copy, it would have been great in the eBook version for these to be hyperlinks to the referenced section
Being that it is a first edition, there are also a few places where there are minor errors such as task numbers not lining up exactly with the numbers used in the related “how it works” section, but in such cases it was rather easy to intuit what was being referred to and didn’t detract from the book.
All told, it is a good book and I’d quickly recommend it as a reference tool for Azure developers.
In the interest of full disclosure, I was sent a copy of the book and asked to read it and post a review.
05 Sep 11 Slides from DevLink 2011
I had the privilege of speaking at DevLink 2011 a few weeks ago in Downtown Chattanooga, TN. I have been a bit OBE (overcome by events) since I left the conference and have been unable to post my slides until now. I hope to get the videos and other materials up in the coming week or so. If you came to one of these sessions – thanks – the attendance at both was great and I appreciated the questions from the audience.
09 Jun 11 Using Nevron Controls in Azure
I’ve been playing around with the Nevron Controls for an Azure application I’m building (hopefully more on that soon) and I’ve been fighting with a simple problem that I’m posting here for my own remembrance and hopefully to help a few others.
The problem has been, that the Nevron controls worked fine when I was testing the web app directly, but would cause the dev fabric to blow up if I tried to run it there. I even tried to simply deploy it to Azure assuming that possibly it was a “feature” of the dev fabric – no dice.
Well, today I had some time to dig to the bottom of it and found that it was a simple problem with the way the http handlers were registered. By default, I had the handlers registered like this:
<system.web>
<httpHandlers>
<add verb="*" path="NevronDiagram.axd" type="Nevron.Diagram.WebForm.NDiagramImageResourceHandler" validate="false"/>
<add verb="GET,HEAD" path="NevronScriptManager.axd" type="Nevron.UI.WebForm.Controls.NevronScriptManager" validate="false"/>
</httpHandlers>
</system.web>
However, as Shan points out in this post: http://social.msdn.microsoft.com/Forums/en/windowsazure/thread/0103ca2d-e952-4c28-8733-47630535c05c, you need to use the newer IIS 7 integrated pipeline. A closer look at the official Nevron samples, shows that they accounted for this and I simply missed it. The setup should be something like this:
<system.webServer>
<validation validateIntegratedModeConfiguration="false"/>
<handlers>
<add name="NevronDiagram" preCondition="integratedMode" verb="*" path="NevronDiagram.axd" type="Nevron.Diagram.WebForm.NDiagramImageResourceHandler"/>
<add name="NevronScriptManager" preCondition="integratedMode" verb="*" path="NevronScriptManager.axd" type="Nevron.UI.WebForm.Controls.NevronScriptManager"/>
</handlers>
</system.webServer>
Notice in particular that not only is the structure a little different, the declarations are under the system.webServer node rather than the system.web node.
06 Jun 11 A Comparison of AWS and Azure
This past weekend at CodeStock, I gave a double-length session that was a side-by-side comparison of Amazon Web Services and Microsoft Windows Azure. The objective was to introduce the products, walk through the similarities and differences, and have a discussion around where the different offerings fit various needs better than the other (or not).
The sessions were fairly well attended and we had some good conversations. The slides from both sessions are provided below and, if you attended, I’d appreciate it if you’d also take a minute to rate the sessions (button below) and provide feedback as to how they might be improved for the next time.
At the end of the second session, we walked through some code that demonstrated a guestbook on both the Amazon and Azure platforms. The source code bundles are available here:
06 Jun 11 Cloud Futures 2011–Scaling Document Clustering
I was honored to be able to give a talk at the Microsoft Research Cloud Futures 2011 conference this past week. I joined a number of other researchers and academics from around the world and discussed what folks were doing with the cloud, where issues remained, and where progress was being made. Having been in attendance at last year’s event, I was quite pleased to see the advancements both in logistics and content… the level of material was definitely stronger this year.
The talk I gave was on some early work we are doing in scaling a document clustering algorithm (Piranha) using cloud primitives. The slide deck is below and, if you attended the talk, I’d appreciate if you’d take a minute to rate the talk using the button below.
06 May 11 Hands On with Amazon Web Services
[updated 6/1/2011 with embedded video]
I have the opportunity to talk at StirTrek today and wanted to make the slides available from today’s session. I’ll update this post a bit more following the session.
11 Feb 11 Moving Applications To The Cloud with Windows Azure
I just finished reading a book from the Microsoft Patterns & Practices group called Moving Applications to the Cloud on the Microsoft Windows Azure Platform. I’ve had the book for a few months, and my when I first received it, I read the first chapter or two, decided it wasn’t worth the read, and set it aside.
Lately, however, I picked it up again – finished the book, and am glad I did. Don’t get be wrong, it didn’t magically morph into a superb spectacle of literary greatness, but I did find that as I read further, the authors moved further from the very basics of the Windows Azure platform and the content became increasingly interesting.
If you are new (or relatively so) to the Windows Azure platform and contemplating the moving of existing applications to the cloud, this is a worthwhile discussion of a fictitious scenario that did just that. The scenario is slightly on the cheesy side, but realistic enough to help you think through issues you may be facing in your business.
If you are well experienced with the platform, you will likely find this a bit dry – especially the first portions. You’ll also likely be distracted or bothered by the not-so-covert marketing that takes place. That said, the book covers some more complex topics such as multiple tasks/threads sharing the same physical worker role, various optimization topics, and more. In the end, I’m glad I read it and feel that I learned some things from the book.
My last thought has nothing to do specifically with the book, but rather a growing frustration of mine with the Windows Azure platform – the design of the table storage platform. Upon reading books such as this I’m reminded (they stress it *many* times) how important your partition key/row key strategy is, and how literally hosed you are if you get it wrong. This compares with my recent experiences with Amazon’s SimpleDB product, and the delta couldn’t be more striking. Both platforms solve essentially the same problem, but in the case of SDB, it is effortless (at least by comparison). I don’t have to think of partition keys, or be overly concerned with how the underlying storage platform works… I just put data in it. Additionally, *every* column is indexed and performs reasonably under queries. I can’t shake the feeling that the Azure team is missing it here – there has to be a way to get a well-designed, horizontally scaling table structure without placing such a design burden on the users.
24 Jan 11 Return of the Windows Azure GAC Viewer
I’m pleased to announce that the excellent utility – the Azure GAC Viewer – is once again online and available for general use. You can access it at http://gacviewer.cloudapp.net. This tool shows you a dynamically generated list of all of the assemblies present in the GAC for an Azure instance. Additionally, it also allows you to upload your project file (*.csproj or *.vbproj) to have the references scanned and let you know if there are any discrepancies between what you are using and what is available (by default) in Azure. You can then adjust your project file (copy-local=true) to ensure your application can run successfully.
If you are familiar with the tool, you may be thinking “Wait! you aren’t Wayne Berry, and besides, the URL has changed!” – and you would be correct on both counts. Wayne developed the tool and posted about it back in September of last year. Since that time, however, Wayne has accepted a position on the Windows Azure team and is unable to continue to maintaining the site full time. As a gesture of kindness to the community, he has passed the source code to me and given me his blessing to re-launch the tool.
As it stands today, the tool is nearly exactly as Wayne developed, with a few tweaks to have it use Guest OS 2.1 rather than 1.6. I’ve also added a contributors page to give credit to Wayne and to the organizations that are allowing me to maintain and keep the site online.
In the future, I hope to make the source code available on CodePlex as well as to add to the list of tools that live on the site. If you have any bugs with the current site or ideas for future changes, please feel free to contact me.
06 Jan 11 Book Review: Host Your Web Site In The Cloud
Over the holiday break I spent some time getting ready for the cloud computing precompiler at CodeMash and as part of that effort I read Jeff Barr’s Host Your Web Site In The Cloud, Amazon Web Services Made Easy. This book is one of the few physical paper books I’ve gotten recently, and is unique to me in that it is the only book I have that is signed by the author.
That aside, I’d like to recommend this book to anyone who is looking at Amazon Web Services, or would consider themselves a beginner with AWS. I found the writing style to be very easy to read and, while I’m not a PHP developer, the code samples and walkthroughs were clear and simple to follow.
AWS is a fast moving target, and even though Jeff is on the team, I’m certain it was difficult to get a book to market that wasn’t completely outdated by the time it hit the shelves, but I think he does a good job of addressing the basics, providing a foundation on which you can build your knowledge, and even slips in a few notes regarding late breaking updates (as of press time) such as EC2 instances being bootable from EBS.
In my mind, this book is similar to the Windows Azure Training Kit in that it gives you most everything you need to get your feed wet, get rolling with the technology, and provides you with the framework by which you can add to your skills.
I am a software engineer working in the Computational Data Analytics Group at Oak Ridge National Laboratory. I spend most of my time building prototypes of the ideas of the researchers I have the opportunity to work with. [more...]
