Windows Home Server v2 'Vail' Beta: Drive Extender v2 Dissected

Yesterday Microsoft released the first public beta of the next version of Windows Home Server, currently going under the codename of Vail (or as we like to call it, WHS v2). WHS v2 has been something of a poorly kept secret, as word leaked out about its development as early as 2008. In more recent times an internal beta leaked out late last year, confirming that WHS v2 existed and giving everyone an idea of what Microsoft has in store for the next iteration of their fledgling home server OS.

As was the case with WHS v1, WHS v2 is having what we expect to be a protracted open beta period to squish bugs and to solicit feedback. This is very much a beta release by Microsoft’s definition (as opposed to Google’s definition) so it’s by no means production-worthy, but it’s to the point where it’s safe to throw it on a spare box or virtual machine and poke at it. Being based on Windows Server 2008 R2 (WHS v1 was based on Server 2003), the underlying OS is already complete, so what’s in flux are WHS-specific features.

 

We’ll cover WHS v2 more in depth once it’s closer to shipping, but there’s one interesting thing that caught our eye with WHS v2 that we’d like to talk about: Drive Extender.

Drive Extender v2: What’s New

Drive Extender was the biggest component of the secret sauce that made WHS unique from any other Microsoft OS. It was Drive Extender that abstracted the individual hard drives from the user so that the OS could present a single storage pool, and it was Drive Extender that enabled RAID-1 like file duplication on WHS v1. Drive Extender was also the most problematic component of WHS v1 however: it had to be partially rewritten for WHS Power Pack 1 after it was discovered that Drive Extender was leading to file corruption under certain situations. The new Drive Extender solved the corruption issues, but it also slightly changed the mechanism of how it worked, doing away with the “landing zone” concept in favor of writing files to their final destination in the first place.

For WHS v2, Microsoft has once again rewritten Drive Extender, this time giving it a full overhaul. Microsoft hasn’t released the full details on how Drive Extender v2 works, but we do know a few things that are worth discussing.

At the highest level, Drive Extender v1 was file based, sitting on top of the file system with the network shares going through it to access files. Hard drives in turn were NTFS formatted and contained complete files, with Drive Extender directing the network sharing service to the right drive through the use of NTFS tombstones and reparse points.

Drive Extender v2 is turning this upside down. The new Drive Extender is implemented below the file system, putting it between NTFS and the hard drives. With this change also comes a change in how data is written to disk – no longer are the drives NTFS formatted with the files spread among them, rather now they’re using a custom format that only WHS v2 can currently read. With Drive Extender v2 sitting between NTFS and the hard drives, anything that hits NTFS direct now receives full abstraction without the need to go through shared folders. In a nutshell, while Drive Extender v1 was file based, v2 is block based.

With the move to block based storage and a custom file system also comes a change in how data is stored on hard drives. In place of storing files on a plain NTFS partition, WHS v2 now uses a custom file system that stores data in 1GB chunks, with files and parts of files residing in these chunks. In the case of folder duplication, the appropriate chunks are duplicated instead of files/folders as they were on WHS v1.

Microsoft also threw in a bit more for data integrity, even on single-drive systems. Drive Extender v2 now keeps an ECC hash of each 512 byte hard drive sector (not to be confused with a chunk) in order to find and correct sector errors that on-drive ECC can’t correct – or worse, silently passes on. MS’s ECC mechanism can correct 1-bit and 2-bit errors, though we don’t know if it can detect larger errors. MS pegs the overhead at this at roughly 12% capacity, so our best guess is that they’re using a 64bit hash of each sector.

The Good & The Bad

All of these changes to Drive Extender will bring some significant changes to how WHS will operate. In terms of benefits, the biggest benefit is that from a high-level view WHS now operates almost exactly like NTFS. The NTFS volumes that Drive Extender v2 generates can be treated like any other NTFS volumes, enabling support for NTFS features such as Encrypted File System and NTFS compression that WHS v1 couldn’t handle. This also goes for application compatibility in general – Drive Extender v2 means that application compatibility is greatly improved for applications that want to interact directly with the drive pool, with this being one of Microsoft’s big goals for WHS v2.

Meanwhile at the low-level, using block based storage means that Drive Extender v2 doesn’t behave as if it was tacked on. Open files are no longer a problem for Drive Extender, allowing it to duplicate and move those files while they’re open, something Drive Extender v1 couldn’t do and which would lead to file conflicts. Speaking of duplication, it’s now instantaneous instead of requiring Drive Extender to periodically reorganize and copy files. The move to block based storage also means that WHS v2 data drives and their data can be integrated in to the storage pool without the need to move the data in to the pool before wiping and adding the drive, which means it’s going to be a great deal easier to rebuild a system after a system drive failure. Finally, thanks to chunks file size is no longer limited by the largest drive - a single file can span multiple drives by residing in chunks on multiple drives.

 

However this comes with some very notable downsides, some of which MS will have to deal with before launching WHS v2. The biggest downside here is that only WHS v2 can read MS’s new data drive file system – you can no longer pull a drive from a WHS and pull off files by reading it as a plain NTFS volume. This was one of WHS’s best recovery features, as a failing drive could be hooked up to a more liberal OS  (i.e. Linux) and have the data pulled off that way.

The other major downside to the new Drive Extender is that the chunking system means that a file isn’t necessarily stored on a single disk. We’ve already touched on the fact that excessively large files can be chopped up, but this theoretically applies to any file big enough that it can’t fit in to leftover space in an existing chunk. This in turn gives WHS v2 a very RAID-0 like weakness to drive failures, as a single drive failure could now take out many more files depending on how many files have their pieces on the failed drive. Our assumption is that Drive Extender v2 is designed to keep files whole and on the same disk if at all possible, but we won’t be able to confirm this until MS releases the full documentation on Drive Extender v2 closer to WHS v2’s launch.

First Thoughts

In discussing WHS v2 and Drive Extender v2, there’s one thing that keeps coming to mind: ZFS. ZFS is Sun/Oracle’s next-generation server file system that has been in continuing development now for a few years. ZFS is a block based file system for use in creating storage pools that features easy expandability and block hashing (among other features) making it not unlike Drive Extender v2. ZFS ultimately goes well beyond what Drive Extender v2 can do, but the fundamental feature set the two share is close enough that Drive Extender v2 can easily be classified as a subset of ZFS. Whether this is intentional or not, based on what we know thus far about Drive Extender v2, Microsoft has created what amounts to a ZFS-lite low-level file system with NTFS sitting on top of it. Make no mistake, this could prove to be a very, very cool feature, and is something we’re going to be keeping an eye on as WHS v2 approaches release.

In the meantime, Microsoft is still taking feature suggestions and bug reports on WHS v2 through Microsoft Connect. We don’t have any specific idea on how long WHS v2 will be in beta, but we’d bet on a Winter 2010 release similar to WHS v1 back in 2007, which means it would be in beta for at least a couple more months.