Technology

Big Data 101: Handling Millions of Files

2013-02-02T20:39:52Z

I've been talking a bit recently with members of my team about some of the basic tools that need to be in any data scientist's toolbox. Things that, if you want to lay any claim to working with "big data" should be second nature. Many of these things are not terribly complicated, nor does one have to be overly clever to employ them - however the lack of knowledge as to when to properly apply them could cost you dearly (lost time, lost data, needless system maintenance, etc).

The first such topic came up a week or so ago when one of our younger team members mentioned that his machine fell over after he had written around 3,000,000 files to the same directory. This reminded me of a lesson I learned back in late 2000 when I was working with Microsoft on the "Millon Mailbox March" and the MCIS mail platform. MCIS contained a mail platform designed by Microsoft for the ISP industry (later replaced by Exchange). This mail platform used an interesting approach to store the potentially millions of mailboxes it housed on the file system. Similar approaches can be (and often have been) applied to modern day storage problems within the Big Data space.

So, I came up with the following exercise/challenge for my students and colleagues - I hope you find it interesting. If you've faced a similar problem in the past, you are likly jumping to solutions and know exactly how you would solve it. It will be interesting to see the solutions presented by our team. I'll post any particularly interesting ones here.

Challenge #1: Handling Millions of Files.
Design and implement a solution for storing 100,000,000 files on a "normal" file system (NTFS, ext4, etc.). The solution should be tested/verified and should be reasonably balanced. The system should provide a naming convention that ensures against collisions. The solution should function properly on Windows, Mac and Linux. Finally, you need to be able to explain the reasoning behind each design decision implemented in your solution.

Deliverables:

A short writeup defining your approach and any incremental steps along the way. Remember: details around interim "failed" attempts are as important as the final solution.
All code used in your solution. By "All" this means everything necessary to recreate your scenario and solution. This includes any means you used to measure and analyize your results.
Timing of the overall activity is important. For example: how long did it take for you to create the file set and analyze the results? While speed of operation is not the primary goal for this exercise (efficacy is), timing information is always informative
Extra credit is given for striking a clear balance between robustness and simplicity.

Support Files:

This exercise does not require any initial data sets.

Assumptions:

Disk space should not be an issue during this experiment.
The solution both can (and should) assume that the target file system (NTFS/ext4/etc.) is of sufficient size to house the files/data in a contiguous set and single namespace.

Would you like a Cassette for your data?

2012-11-01T12:07:00Z

A colleague of mine sent me a link to this story about using cassettes for storing data and asked for my thoughts. This was clearly a good-natured jab at me in light of a prior conversation we had debating the appropriateness of tape or disk for a research data storage platform. I had been arguing against tape as an out-moded and inappropriate storage mechanism.

The problem is… I was wrong.

And so was he.

I am using the word “wrong” not in the moral sense but to mean “less than the ideal”, “unfortunate”, “sad”, “depressing”, .

You see, I was wrong in that as he argued (and is well articulated in the article) there is simply too much data being generated to make disks a tractable solution. Even with recent and projected growth in hard drives (60TB expected by 2016) our ability to produce data – particularly in an automated means via sensors and scientific instruments – already does, and will continue to out-pace these advancements. Even if hard drives were able to keep up with the space demands, the power required to keep those drives running quickly becomes prohibitive. And let’s not even talk about disk transfer rate issues.

Whether or not he was “wrong” is probably a bit more subjective (an admission I’m certain he’ll enjoy). My frustration with his position is that it tends to be synonymous with “slow” or “laborious”. That said, an unfortunate reality in the digital sciences is that with data sets of any significant size, a researcher often has to plan well in advance of his experiment to stage the data. Data has to be loaded into online storage from some cold storage mechanism (often a tape library). This significantly limits one’s curiosity and causes some questions to go unanswered (i.e. “I wonder if… well, it’s probably not worth the time/effort to load up the data just to see…”). I suppose that if I’m honest, I’m simply manifesting the “Google effect” – the expectation that I can as a question of tons of data and get an answer instantly. I’d love for this to be possible of all scientific data – but as any data scientist will tell you, that desire is simply naïve. Providing platforms such as Google’s is hard work, and not achieved without significant planning and effort. Admitting this still doesn’t mean I can’t hope for it…

The real nugget buried in the article and underlying our friendly debate, is that storage technologies are nowhere close to where we need them to be. No matter which option we choose it will be a compromise between a.) discarding data – unfortunate no matter how you look at it, b.) spending astronomical amounts of money on both hardware and power, or c.) using slow, offline, and deterioration-prone devices such as tapes. Frankly, none of these options are attractive. There is a little part of me that dies when I think of a researcher having to choose to discard data simply because he doesn’t have space to store it and doesn’t currently think it is important to his work. What if he’s wrong? What if that data is (or was) the key to solving part of the problem, he just didn’t know it yet? Or maybe it is the key to solving a problem he doesn’t yet know he has…

So here’s hoping that the researchers working on storage technologies will be successful. That they will develop means and methods for us to store massive amounts of data, access it in increasingly shorter times, and with a power envelope that is reasonable and sustainable. That’s not asking too much, is it?

Debugging and Reversing Basics 0.01

2012-10-15T17:54:52Z

[Note: the title is what it is because I consider myself a n00b at this and these are likely things anyone else already knows]

I'm always trying to learn more and recently topics in the infosec world have garnered my attention. To further my understanding of the space, I've been reading a bit and this weekend was reading part of Gray Hat Python: Python Programming for Hackers and Reverse Engineers and found myself purpsefully doing that which you shouldn't do when learning a new subject: not following the instructions. You see, the author specifically indicates that the samples were written on/tested on a Windows x86 machine and his assumption is that you will be running on the same. In my case, I haven't run a 32-bit OS in years (since Vista was released) and I made two assumptions: 1.) It probably doesn't matter that much and 2.) even if it does, it's probably a good thing to learn what the differences b/t 32 and 64 bit debugging/reversing are. Well, after a few hours of playing around, I can tell you the first assumption was flat wrong and the second is probably accurate.

The fun beings in chapter 3 where you build a simple debugger. I got stuck on the very first step which was a simple demonstration of attaching to an existing process (calc.exe). I would run the script and simply get an error: "[*] Unable to attach to the process." I figured I must have done something wrong, so I poked around a bit and even diff'd my code against the reference and still didn't see any important differences. As a side note: If you've not yet looked at the errata for the book, you need to do so. There are a number of code/bug fixes that are required to get things working.

The key came from a blog post I came across written by A. H. where he hints that the problem may have to do with the architecture of the application I am attempting to attach to. He suggests adjusting the error line in the script as follows:

print "[*] Unable to attach to the process. %s" % FormatError(kernel32.GetLastError())

and if the error ends with "The request is not supported" you can rest sure that your problem is a 32/64 bit issue. Unfortunately, A.H.'s solution was to simply use a 32-bit box for the rest of the testing.

What Version of Python am I running?

The next issue that occured to me was to determine which version of Python (bit-ness) I was running. I simple search brought up Ned Deily's answer on Stack Overflow which indicated that one simple way to check would be to run the following:

python -c "import struct;print( 8 * struct.calcsize('P'))"

You will get either 32 or 64 as a result - in my case, 32. Great. So I know that my debugging thread is a 32-bit application, what is the image type of calc.exe?

DumpBin

Some poking around led me to an article by Frank Chism on the Windows HPC blog that pointed to being able to run a tool called dumpbin to see if an exe was 32 or 64 bit. I followed the instructions on his post, opened a VS 2010-enabled command shell and typed the following:

dumpbin /headers c:\Windows\system32\calc.exe|findstr "magic machine"

Which resulted in:

14C machine (x86)
    32 bit word machine
10B magic # (PE32)

Ok... so my debugging thread is 32-bit, and the executeable that I'm running is 32-bit, so why am I unable to attach to the thread?

Process Explorer

At this point, I pulled up the trusty Sys Internals Process Explorer to see if it would shed any light on the issue. From within Process Monitor, if you select the View menu and then click on "Select Columns" you can tick the box for "Image Type" which will allow you to see for each process/executeable running what the image type is. And, after quickly checking, I see that calc.exe is running as a 64-bit image. How in the world is this happening?

WOW64

64 Bit Windows has a feature called the File System Redirector which seems to be the root of my issues. If I understand how this works (dubious), this is a layer within the OS that "magically" redirects you to the proper version of the application based on the calling process. For example, if a 64-bit process attempts to open the 64-bit image of calc.exe (located in C:\Windows\System32), it will work just fine. However, if a 32-bit process attempts to do the same thing, it will get magically re-directed to the 32-bit version of the application which is located in C:\Windows\SysWOW64 (don't even ask why the folders are named the way they are based on the versions of the applications that they house). What this means, is that if you simply hit Windows+R and type calc, you are calling it from a 64-bit process (the shell) and therefore you get the 64-bit version of the application. If, however, you reference calc.exe from a 32-bit process (i.e. dumpbin), you get redirected to the 32-bit version.

If you specifically need the 32-bit version (as I did to complete my testing), you can open a command prompt, navigate to c:\Windows\SysWOW64 and then type calc.exe or, you can have it launched from any 32-bit process. To see this second option in action, open a command prompt, navigate to c:\windows\SysWOW64 and then type cmd.exe. Via Process Explorer you can confirm that you are running a 32-bit version of cmd.exe. Then navigate wherever you'd like (i.e. c:\) and then type calc.exe. You will now get the 32-bit version of the application (and can confirm it in Process Explorer).

From here, I can attach to the process (calc.exe as 32-bit) from my python code. This moves me forward a bit but doesn't solve the "how do I bind to the 64-bit image" question. That will be a problem for another day.

DevLink: Wireless Network Security

2012-08-31T13:12:29Z

On Wednesday I had the privelege of speaking at DevLink on the topic of wireless network security. I had a great time giving the talk and had great audience participation (including some who were unknowning victims to my man-in-the-middle attack). The slides from the talk are posted below.

DevLink - WiFu: You think your wireless is secure? from Rob Gillen

Windows 8 Release Preview on Samsung Slate

2012-07-11T00:31:39Z

I've been playing with Windows 8 on a Samsung 700T1A slate for a number of months and was quite excited with the Release Preview was announced and attempted to install it straight away. Unfortunately, I was unable to get it installed right away and set it aside for awhile, trying occasionally, failing, and setting it aside.

The problem I was having, was that the slate wouldn't boot to the Windows 8 media - DVD, USB, no matter what I burned it to, it wouldn't work. I even verified that the media was valid by using it to install on other machines.

Tonight, I finally got it working and the problem was both so odd, and simple, that I thought I'd post it here to maybe help someone else who comes along searching for the same problem.

It seems that the slate, when shipped, has a bios setting that has "Support for Legacy USB" devices enabled. However, as soon as the system is updated with a purchase date, it automatically flips this switch (presumably for faster boots). Unfortunately, this also causes it to not check for bootable USB devices during POST (cf. http://skp.samsungcsportal.com/integrated/popup/FaqDetailPopup3.jsp?cdsite=hk_en&seq=431318)

The post that tipped me off was this: http://skp.samsungcsportal.com/integrated/popup/FaqDetailPopup3.jsp?cdsite=hk_en&seq=431320

(image courtesy of Samsung)

However, the bios on my slate didn't look like this - there is no Fast BIOS Mode menu item. However, there is a menu item that says "Support Legacy USB Devices". Based on this article and the previous one, I took a guess that changing this would fix it and, magically, everything worked just as you would have expected.

CodeStock 2012: Buffer Overflow Attack

2012-06-18T01:29:03Z

We had a great time at CodeStock a few days ago discussing buffer overflow attacks, showing developers how they are discovered and exploited and a bit about how to avoid creating software that is vulnerable to these types of attacks. Below are the slides and video from the session:

CodeStock 2012: You Think Your WiFi is Safe?

2012-06-18T01:09:39Z

This past Friday I had the privilege of speaking on WiFi security at CodeStock 2012. I had a blast both preparing for the talk and delivering it and I hope it was beneficial to some of those who attended.

As promised (although a bit late), the following are the slides and video from the session:

Manually Interacting with the MSF database

2012-04-16T20:39:20Z

Building My Personal Cloud

2012-04-12T09:01:48Z

Speaking at CodeStock 2012

2012-04-09T21:09:29Z