Entries in cloud (32)

Thursday
Apr122012

Building My Personal Cloud

78881z2p31zg232This is a bit of an odd post in that it is more a “wondering aloud” then anything specifically prescriptive or informative. I was asked yesterday by a student if I knew of a Windows box he could use to test something on. It was a simple request – he had been building an application on his Mac, and before handing it over to be used, wanted to validate that it worked on Windows. It had the one unique requirement that it had to run on our network at Work due to accessing some IP-restricted content.  The “official” answer was to have him contact our help desk and they would set him up, however the answers he got from them were (understandably) geared towards long-term use of a Windows “desktop” … and came with the associated costs (licenses, labor, etc.). What he really needed, was a VM running a trial version of Windows that he could use for two hours and then throw away… What he needed, was a place he could go, select a Windows machine from a catalog, click “go”, run for a few hours and then click “done” and have it go away… What he needed, was access to a cloud platform.

Unfortunately, I didn’t have a good answer for him, and am still a bit uncertain how he is going to go about fixing the issue, but it did get me thinking about what it would take to have/run a “personal” cloud… nothing fancy, but say I have a single “beefy” workstation/server and I’d like to have a thin hypervisor, and a web-based interface to provision/de-provision running instances… maybe setup a catalog virtual machines that I’ve used before as “starting points” for new machines. The platform should support a wide variety of OS choices… Windows and Linux without noticeable compromises for either. It would be nice if the platform could support various virtual networking (nothing fancy, but the ability, at times to create a private network that only two machines can talk on – for testing various things). The final two requirements are that it should support *modest* horizontal scaling (say, I add one or two new physical boxes) and it should be free (or very low cost). This is for personal use or a small group test platform… nothing fancy or official… just something that works. As soon as you get into anything significant cost-wise, you raise the attention of corporate IT, project budgets, etc. and your simple idea just ballooned into something with a budget line item, project managers, and no chance of coming to life.

Options/Solutions: The following is a list of some of the options I’ve been considering along with some commentary on each. I’d be very interested in feedback, but I would caveat that I don’t want the platform itself to be a research endeavor… it should basically “just work” out of the box and not require significant fiddling to get it stable/working.

  • I have a leaning towards Microsoft products (mostly due to familiarity) so I considered them first, but quickly ruled them out. VirtualPC is not anything what I’m looking for, and HyperV/SystemCenterVMM/<insertManyMoreAcronymsHere> seems ok for a lager-scale deployment but seems like *way* too much overhead for what I’m looking for.
  • Eucalyptus – This is a platform I’ve worked with in the past, and is amazingly easy to setup/run but its support for Windows platforms (at lest in my recollection) isn’t that great.
  • VMWare Workstation has some interesting things, and to large degree, this would be my front-runner short of the fact that there is no web UI for a remote user to provision/interact with a machine. This solution works fairly well, however, for a single user directly interacting with the physical host. While not free, the cost seems reasonable enough (~$200).
  • VMWare vSphere Essentials is promising, especially if used in conjunction with VMWare Workstation. It comes with a web-UI, can scale to multiple physical hosts, has broad OS support, and comes from, arguably, one of the longest-standing leaders in this space. It isn’t free, but at <$500, it falls in the reasonable category – esp. considering it supports up to three physical machines. The price does, however, tend to eliminate it from consideration for my home network.
  • CloudStack + <nameYourHypervisor> – CloudStack is another interesting option and it certainly has a nice UI, however our recent experiences have shown that, while very powerful and great for building large cloud deployments, it seems a bit over-complicated for a one or two server installation. Further, we had significant issues with networking performance for our Windows machines (note: this may be an artifact of the hypervisor we chose: Citrix’s XenServer). Further, we found that there was significant amount of “tinkering” that had to be done to get it working… i.e. download patch X from this svn repo, only use version such-and-such of python, etc. Most of these issues may have been environmental and/or solvable by using other distros/hypervisors, but it was a non-trivial deployment.
  • OpenStack – OpenStack seems to be what all of the cool kids are using these days, but our experiences once again hinted at a lack of maturity in the computing platform for the general sense (range of OS support) and the lack of a solid web-UI eliminates it from consideration.
  • XenServer – Another option I considered briefly is to use the free XenServer and then the windows desktop client to manage the servers and create new images/machines. This would work (we are actually doing this in one of our lab environments) but fails the self-provisioning portion of the exercise (no web-based portal, etc). Additionally, we’ve had issues with the networking stack for non-Linux machines and have spent a large amount of time tracking down driver issues and performance issues.

At this point, I don’t have a solution… I’m still looking, and am guessing that whatever I end up with will require an amount of compromise. Comments/suggestions are welcome…

Image courtesy of Pixomar.

Wednesday
Nov302011

Ramblings on a Private Cloud (part 1 of ?)

As many of you are aware, a few months ago I changed jobs (more like positions) and with that change has come a shift in the focus of much of my work. I’m spending less time in the abstract (i.e. “how could we solve problem X" in the cloud”) and working more in the concrete (“we have agency Y that has problem X that needs to be deployed in the cloud yesterday… what do you think?”). One unfortunate (at least in my way of thinking) is that many of these projects have legislative requirements that prohibit the use of public cloud platforms (at least currently). While I’m always looking for ways to avoid building individual, private “Cirrus1” clouds, and I remain hopeful that the day will come when we all can leave the infrastructure build-outs to someone else, the reality is that in today’s world there is some data that simply cannot live outside of an organization’s boundaries – leaving us to look at “private cloud” approaches.

Overview

We have been experimenting with a few different approaches for some of the projects I’ve been working on and are learning a number of interesting things. I want to be quick to say that I don’t think we’ve “arrived” by any stretch of the imagination and our work (and this space in general) is changing/evolving rapidly. Our initial requirements included using free/OSS solutions wherever possible and to, as much as possible, avoid vendor lock-in (always great in theory, frequently falls apart in application). The hardware used for this exercise includes 45 physical nodes spread over two racks with a total of 1160 cores, 3.8 TB of RAM, 200 TB (RAW) of local node storage and 288 TB (RAW) of iSCSI-connected storage. Our plan is to provide 8 VLANS across the environment (Management, Dev, Test, Prod + corresponding disk-traffic networks for each). The physical nodes/nics would all default to the management network and all of the actual compute resources (VMs used by researchers) would live in one of the other three zones (Dev/Test/Prod). For reasons that I won’t get to in this post, we standardized on CentOS for the primary OS and Xen as the target hypervisor. While some of the uses of the platform include data serving (i.e. web sites, data services, etc.), most of the workloads will tend to be heavy data analytics. The above scenario is complicated by the fact that the entire environment needed to be air-gapped (i.e. not connected to the Internet or other networks in any way).

Open Stack

While we have a good bit of experience using Eucalyptus on the Ubuntu stack, our initial plan was to go with the current wisdom of the crowds and deploy OpenStack as our cloud orchestration layer. The intent was to buy in fully and deploy their storage, image, and compute services. Unfortunately, while there are some very interesting things happening in this realm, we didn’t find this toolset to be at the level we needed it to be given our platform selections. After a few weeks of trying to get this working, we bailed due to simply needing to get something working2. We are hopeful that this situation will evolve in the future and we will reconsider down the road, but for the present we had to scrap it and move on.

Refactoring / Current Approach

Having burned a bit of our grace period, we were faced with the need to get something running fast and spent a weekend digging through our options. Where we ended up is using the free XenServer (via Citrix Systems) as the host OS in combination with XenCenter for managing the nodes. This – while not a cloud or cloud orchestration layer – allowed us to quickly meet some of our sponsor’s needs while buying some time to fill in the gaps. Our team is currently evaluating CloudStack as the cloud/orchestration layer to sit on top of XenServer and be the researcher-facing interface to the platform. Our hope is that as the story evolves (Citrix, XenServer, XenCenter, CloudStack, OpenStack, etc) that the deployment of future platforms will become easier and the “best” approach will become clearer.

main-search-menu

XenCenter management console. Image credit Citrix Systems (http://community.citrix.com/display/xs/XenCenter)

 

 

Notes
1. I spent a good bit of time looking on the web for the official name of the tiniest cloud but didn’t come up with anything better than “Cirrus” which is defined as a “thin, wispy cloud.” Not exactly what I was looking for, but I’ll use it for now.

2. Our issues included (among other things) Python version conflicts between what OpenStack needed and the version of Centos (5.6) we were running, the lack of a good Web UI/self-service portal, getting the VLANS talking properly between hosts, vhosts, storage, etc., and a number of smaller miscellaneous items.

Monday
Sep052011

Slides from DevLink 2011

I had the privilege of speaking at DevLink 2011 a few weeks ago in Downtown Chattanooga, TN. I have been a bit OBE (overcome by events) since I left the conference and have been unable to post my slides until now. I hope to get the videos and other materials up in the coming week or so. If you came to one of these sessions – thanks – the attendance at both was great and I appreciated the questions from the audience.

 

Source code for GPGPU Talk

 

Source code for AWS Guest Book Demo

Source code for Azure Guest Book Demo

Monday
Jun062011

A Comparison of AWS and Azure

This past weekend at CodeStock, I gave a double-length session that was a side-by-side comparison of Amazon Web Services and Microsoft Windows Azure. The objective was to introduce the products, walk through the similarities and differences, and have a discussion around where the different offerings fit various needs better than the other (or not). 

The sessions were fairly well attended and we had some good conversations. The slides from both sessions are provided below and, if you attended, I’d appreciate it if you’d also take a minute to rate the sessions (button below) and provide feedback as to how they might be improved for the next time.

 

At the end of the second session, we walked through some code that demonstrated a guestbook on both the Amazon and Azure platforms. The source code bundles are available here:

Friday
May062011

Hands On with Amazon Web Services

[updated 6/1/2011 with embedded video]

I have the opportunity to talk at StirTrek today and wanted to make the slides available from today's session. I'll update this post a bit more following the session.

Friday
Feb112011

Moving Applications To The Cloud with Windows Azure

appsinthecloud I just finished reading a book from the Microsoft Patterns & Practices group called Moving Applications to the Cloud on the Microsoft Windows Azure Platform. I’ve had the book for a few months, and my when I first received it, I read the first chapter or two, decided it wasn’t worth the read, and set it aside.

Lately, however, I picked it up again – finished the book, and am glad I did. Don’t get be wrong, it didn’t magically morph into a superb spectacle of literary greatness, but I did find that as I read further, the authors moved further from the very basics of the Windows Azure platform and the content became increasingly interesting.

If you are new (or relatively so) to the Windows Azure platform and contemplating the moving of existing applications to the cloud, this is a worthwhile discussion of a fictitious scenario that did just that. The scenario is slightly on the cheesy side, but realistic enough to help you think through issues you may be facing in your business.

If you are well experienced with the platform, you will likely find this a bit dry – especially the first portions. You’ll also likely be distracted or bothered by the not-so-covert marketing that takes place. That said, the book covers some more complex topics such as multiple tasks/threads sharing the same physical worker role, various optimization topics, and more. In the end, I’m glad I read it and feel that I learned some things from the book.

My last thought has nothing to do specifically with the book, but rather a growing frustration of mine with the Windows Azure platform – the design of the table storage platform. Upon reading books such as this I’m reminded (they stress it *many* times) how important your partition key/row key strategy is, and how literally hosed you are if you get it wrong. This compares with my recent experiences with Amazon’s SimpleDB product, and the delta couldn’t be more striking. Both platforms solve essentially the same problem, but in the case of SDB, it is effortless (at least by comparison). I don’t have to think of partition keys, or be overly concerned with how the underlying storage platform works… I just put data in it. Additionally, *every* column is indexed and performs reasonably under queries. I can’t shake the feeling that the Azure team is missing it here – there has to be a way to get a well-designed, horizontally scaling table structure without placing such a design burden on the users.

Monday
Jan242011

Return of the Windows Azure GAC Viewer

I’m pleased to announce that the excellent utility – the Azure GAC Viewer – is once again online and available for general use. You can access it at http://gacviewer.cloudapp.net. This tool shows you a dynamically generated list of all of the assemblies present in the GAC for an Azure instance. Additionally, it also allows you to upload your project file (*.csproj or *.vbproj) to have the references scanned and let you know if there are any discrepancies between what you are using and what is available (by default) in Azure. You can then adjust your project file (copy-local=true) to ensure your application can run successfully.

gacviewer

If you are familiar with the tool, you may be thinking “Wait! you aren’t Wayne Berry, and besides, the URL has changed!” – and you would be correct on both counts. Wayne developed the tool and posted about it back in September of last year. Since that time, however, Wayne has accepted a position on the Windows Azure team and is unable to continue to maintaining the site full time. As a gesture of kindness to the community, he has passed the source code to me and given me his blessing to re-launch the tool.

As it stands today, the tool is nearly exactly as Wayne developed, with a few tweaks to have it use Guest OS 2.1 rather than 1.6. I’ve also added a contributors page to give credit to Wayne and to the organizations that are allowing me to maintain and keep the site online.

In the future, I hope to make the source code available on CodePlex as well as to add to the list of tools that live on the site. If you have any bugs with the current site or ideas for future changes, please feel free to contact me.

Thursday
Jan062011

Book Review: Host Your Web Site In The Cloud

hostyourwebsiteinthecloudOver the holiday break I spent some time getting ready for the cloud computing precompiler at CodeMash and as part of that effort I read Jeff Barr’s Host Your Web Site In The Cloud, Amazon Web Services Made Easy. This book is one of the few physical paper books I’ve gotten recently, and is unique to me in that it is the only book I have that is signed by the author.

That aside, I’d like to recommend this book to anyone who is looking at Amazon Web Services, or would consider themselves a beginner with AWS. I found the writing style to be very easy to read and, while I’m not a PHP developer, the code samples and walkthroughs were clear and simple to follow.

AWS is a fast moving target, and even though Jeff is on the team, I’m certain it was difficult to get a book to market that wasn’t completely outdated by the time it hit the shelves, but I think he does a good job of addressing the basics, providing a foundation on which you can build your knowledge, and even slips in a few notes regarding late breaking updates (as of press time) such as EC2 instances being bootable from EBS.

In my mind, this book is similar to the Windows Azure Training Kit in that it gives you most everything you need to get your feed wet, get rolling with the technology, and provides you with the framework by which you can add to your skills.

Tuesday
Jan042011

Speaking at the CodeMash Precompiler

timidI’m thrilled to be speaking at the CodeMash Precompiler next week. I’m going to be joined by Mike Wood and helped by Brian Prince and Michael Collier. Together, we’ll have nearly 8 hours of instruction and hands on labs covering both the Amazon and Microsoft cloud computing platforms. Below I’ve listed the abstracts for each of the sessions as well as the prerequisites for those planning on joining us. If you are going to be in Sandusky next Wednesday, be sure to drop by.

An Introduction to Amazon Web Services (half-day, afternoon)

AWS has been in the cloud computing space longer than most anyone, and they are the de facto standard when it comes to Infrastructure as a Service. While most developers are comfortable with the notion of virtual machines, reviewing the AWS offering can sometimes look like alphabet soup (EC2, S3, SNS, SDB, SQS). Join us to learn the power behind these acronyms and the tools that they can provide your next project. We'll discuss the major components, some of the trade-offs between different implementation choices (i.e. boot from S3/boot from EBS, etc.) and provide you with the opportunity to work through some labs, deploy some code, and begin to experience the Amazon cloud for yourself.

Examples are in .NET, but fundamental concepts apply to all platforms.

 

An Introduction to Windows Azure (half-day, morning)

Steve Ballmer has made it very clear that Microsoft is "all in" when it comes to the cloud and by now most have heard about Microsoft's Windows Azure platform... but what does that mean for you? Whether you are an experienced .NET developer who is wondering what all this cloud stuff means for how you write code, or maybe you are a traditional *nix developer looking to understand how to integrate your existing code with the Microsoft version of the cloud, join us for an in-depth discussion on what Platform as a Service is, how Microsoft has implemented it, what scenarios it best addresses, and a collection of hands-on-labs to get you started.
Examples are in .NET, but fundamental concepts apply to all platforms.

 

Prerequisites

The sessions will be part presentation, part hands on labs.  While you aren't required to bring a laptop, you'll get much more out of the sessions if you have one available to work through the labs with (but, there might be some people willing to pair as well!).  Please make sure to bring your power cord! 

Here are the prerequisites to have loaded:

An Introduction to Windows Azure

· Operating Systems Supported: Windows 7 (Ultimate, Professional, and Enterprise Editions); Windows Server 2008; Windows Server 2008 R2; Windows Vista (Ultimate, Business, and Enterprise Editions) with either Service Pack 1 or Service Pack 2

· Microsoft Visual Studio 2010 (full version or the free trial).

· SQL Server 2005 Express Edition (or above) (this is usually installed with Visual Studio)

· Install the Windows Azure Tools for Microsoft Visual Studio (and some hotfixes)

· Install the AppFabric SDK

· Install the Windows Azure Platform Training Kit

An Introduction to Amazon Web Services

· Amazon AWS SDK for .NET

· Requires Microsoft .NET Framework 2.0 or later.

· Use the AWS SDK for .NET with any of the following Visual Studio editions:

o Microsoft Visual Studio 2008 Professional Edition or later

o Microsoft Visual C# 2008 Express Edition (free!)

o Microsoft Visual Web Developer 2008 Express Edition (free!)

You might be thinking, "Hey, What a second!  This is CodeMash, you just listed all Microsoft tools there!".  Just like CodeMash, both Windows Azure and Amazon AWS are happy to mix in multiple development stacks.  Our labs and demos will be shown using Visual Studio, but don't let that stop you from following along or trying out the cloud platforms from your Mac, or using Java, PHP and Ruby on Windows.  Below are links to other SDKs for each cloud platform.  Please, feel free to explore your options and load these SDKs or libraries up if you prefer them.

For Windows Azure

· Windows Azure SDK For Java

o AppFabric: http://www.jdotnetservices.com/

· Windows Azure SDK for PHP

o AppFabric: http://dotnetservicesphp.codeplex.com/

o and tools http://azurephptools.codeplex.com/

o and Companion http://www.interoperabilitybridges.com/projects/windows-azure-companion

o Oh, and some love for Eclipse via a plug in: http://www.windowsazure4e.org/

· Windows Azure AppFabric SDK For Ruby

For Amazon AWS

· AWS Java Developer Center

· AWS PHP Developer Center

· AWS Python Developer Center

· AWS Ruby Developer Center

Tuesday
Dec212010

Planet Technologies Launches GovCloud

This is just a quick post to let those interested know that the company I work for - Planet Technologies - has launched a new cloud-focused practice focused on helping public sector/government agencies utilize cloud computing. Services range from high-level assessments, hands-on migration assistance, and most everything in between.

You can read the press release here: http://www.technology-digital.com/planet-technologies-launches-govcloud-new-cloud-practice-designed-specifically-assist-government-ag-0

And visit the website here: http://govcloud.com/