Kick Ass Hypervisor Nesting!

Remember how at the Black Hat Vegas 2007 I said that we still didn't support virtualization of full VMMs, like e.g. Virtual PC 2007 with hardware virtualization enabled, and that currently we could only run very simple hypervisors inside our New Blue Pill (like e.g. other NBPs inside NBP)? Remember how I said that we were working on this and should have a solution in about 2 months from then?

So, just about 2 weeks ago we did it! We can now virtualize complex hypervisors, like e.g. Virtual PC 2007 or Virtual Box with SVM turned on (BTW, we can also run VMWare Workstation, but that doesn't count, as on AMD processors it doesn't make use of SVM instructions). We also have a prototype code that allows to run nested hypervisors on VT-x but that code requires a bit of more polishing (oh, didn’t you know that our NBP also supports VT-x these days?).

I couldn't resist not to use my favorite Matrix analogy to describe what we do here: imagine Neo, who bravely followed The White Rabbit and finally decided to swallow The Red Pill, eventually awakes on The Nebuchadnezzar ship just to find out later that this whole "real world" is... just another Matrix...

I don't have a nice Matrix picture for that, so instead I will just show you a picture of a Virtual PC 2007 running inside an already bluepilled Vista and running Windows XP as its own guest. You can see that we use our "bpknock" testing program just to show we can intercept events in both the guest (i.e. the Vista that hosts the VPC hypervisor) as well as in the nested guest (the XP running inside the Virtual PC). This bpknock program simply executes CPUID instruction with some magic value in the RAX register and NBP intercepts that and answers with a magic RAX. BTW, there was no special reason to chose CPUID instruction for that, normally we don't need to intercept CPUID on AMD at all, so we could have chosen pretty much anything else, e.g. magic output to some magic I/O port.



It's worth mentioning that the only other working example of nested hardware virtualization I'm aware of is the IBM z/VM hypervisor for the IBM z series mainframe. If anybody knows any other example, please send me a link.

The research on nested virtualization has been supported by Phoenix Technologies, as the nested virtualization has also some positive applications. Phoenix is working on a cool product called HyperSpace. It consists of a hypervisor (called the "HyperCore") that allows running a few unmodified OSes inside hardware virtual machines so users can switch between them just like if they were virtual spaces on Mac or Linux. At the beginning there will be two virtual machines available: one running standard Vista and the other one based on Linux, that would contain some useful functionality like e.g. a Web browser, an email client and a multimedia suite, and also there would be something called the "ManageSpace" to manage this all.

So, how this is going to be different from e.g. XEN? The difference is that XEN is focused on server applications, while HyperSpace is intended for notebooks, which means it puts lots of efforts to offer comparable graphics (and other devices) performance as we have on normal non-virtualized laptops. This all will be possible because of the recent virtualization technology advances like e.g. VT-d/IOMMU.

At the RSA conference in San Francisco next week, I will be giving a speech that will discuss some technical problems we had to solve in order to get hardware nested hypervisoring working on AMD and also how the situation looks on Intel. I will also discuss how this changes the security battlefield and why virtualization vendors should care.

Back to Blue Pill -- the brand new source code with full virtualization support on AMD is now available on bluepillproject.org (you will need WDK6000 or newer to build it). Note that the (experimental) code for nested virtualization on Intel VT-x has been removed in this public version, leaving only the basic functionality if we run NBP on an Intel processor.

Also, please note that the code for AMD-v, even though it proved to be very stable, is still just a proof of concept. This means for example, that we don’t do any error-checks in the SVM instruction handlers, so it’s trivial for the nested hypervisor to simply crash the whole system if executing one of the SVM instructions with incorrect arguments or in an incorrect situation (e.g. CPL > 0). But that is hardly a problem for Blue Pill, as the guest isolation has never been a goal here. Of course, this could be simply addressed by adding a few more lines of code to each handler that would check for error conditions and inject #UD or #GP back to the nested hypervisor if it executed something incorrectly. Of course, we’re too lazy to code that ;)

So, what’s next? Well, we hope to show something even cooler at this year’s Black Hat Vegas, but I won’t say anything more now.