Monday, March 31, 2008

Kick Ass Hypervisor Nesting!

Remember how at the Black Hat Vegas 2007 I said that we still didn't support virtualization of full VMMs, like e.g. Virtual PC 2007 with hardware virtualization enabled, and that currently we could only run very simple hypervisors inside our New Blue Pill (like e.g. other NBPs inside NBP)? Remember how I said that we were working on this and should have a solution in about 2 months from then?

So, just about 2 weeks ago we did it! We can now virtualize complex hypervisors, like e.g. Virtual PC 2007 or Virtual Box with SVM turned on (BTW, we can also run VMWare Workstation, but that doesn't count, as on AMD processors it doesn't make use of SVM instructions). We also have a prototype code that allows to run nested hypervisors on VT-x but that code requires a bit of more polishing (oh, didn’t you know that our NBP also supports VT-x these days?).

I couldn't resist not to use my favorite Matrix analogy to describe what we do here: imagine Neo, who bravely followed The White Rabbit and finally decided to swallow The Red Pill, eventually awakes on The Nebuchadnezzar ship just to find out later that this whole "real world" is... just another Matrix...

I don't have a nice Matrix picture for that, so instead I will just show you a picture of a Virtual PC 2007 running inside an already bluepilled Vista and running Windows XP as its own guest. You can see that we use our "bpknock" testing program just to show we can intercept events in both the guest (i.e. the Vista that hosts the VPC hypervisor) as well as in the nested guest (the XP running inside the Virtual PC). This bpknock program simply executes CPUID instruction with some magic value in the RAX register and NBP intercepts that and answers with a magic RAX. BTW, there was no special reason to chose CPUID instruction for that, normally we don't need to intercept CPUID on AMD at all, so we could have chosen pretty much anything else, e.g. magic output to some magic I/O port.

It's worth mentioning that the only other working example of nested hardware virtualization I'm aware of is the IBM z/VM hypervisor for the IBM z series mainframe. If anybody knows any other example, please send me a link.

The research on nested virtualization has been supported by Phoenix Technologies, as the nested virtualization has also some positive applications. Phoenix is working on a cool product called HyperSpace. It consists of a hypervisor (called the "HyperCore") that allows running a few unmodified OSes inside hardware virtual machines so users can switch between them just like if they were virtual spaces on Mac or Linux. At the beginning there will be two virtual machines available: one running standard Vista and the other one based on Linux, that would contain some useful functionality like e.g. a Web browser, an email client and a multimedia suite, and also there would be something called the "ManageSpace" to manage this all.

So, how this is going to be different from e.g. XEN? The difference is that XEN is focused on server applications, while HyperSpace is intended for notebooks, which means it puts lots of efforts to offer comparable graphics (and other devices) performance as we have on normal non-virtualized laptops. This all will be possible because of the recent virtualization technology advances like e.g. VT-d/IOMMU.

At the RSA conference in San Francisco next week, I will be giving a speech that will discuss some technical problems we had to solve in order to get hardware nested hypervisoring working on AMD and also how the situation looks on Intel. I will also discuss how this changes the security battlefield and why virtualization vendors should care.

Back to Blue Pill -- the brand new source code with full virtualization support on AMD is now available on (you will need WDK6000 or newer to build it). Note that the (experimental) code for nested virtualization on Intel VT-x has been removed in this public version, leaving only the basic functionality if we run NBP on an Intel processor.

Also, please note that the code for AMD-v, even though it proved to be very stable, is still just a proof of concept. This means for example, that we don’t do any error-checks in the SVM instruction handlers, so it’s trivial for the nested hypervisor to simply crash the whole system if executing one of the SVM instructions with incorrect arguments or in an incorrect situation (e.g. CPL > 0). But that is hardly a problem for Blue Pill, as the guest isolation has never been a goal here. Of course, this could be simply addressed by adding a few more lines of code to each handler that would check for error conditions and inject #UD or #GP back to the nested hypervisor if it executed something incorrectly. Of course, we’re too lazy to code that ;)

So, what’s next? Well, we hope to show something even cooler at this year’s Black Hat Vegas, but I won’t say anything more now.


Saso said...

She is alive! Nice :)

Nishad Herath said...

Awesome Joanna! Keep up the good work :-) Looking forward to your BlackHat Vegas presentation!

mdpac said...

Are you planning to include the full VT-x nested hypervisor support in the public version at some point in the future?

joanna said...

@mdpac: most likely not.

Anonymous said...

Joanna... wow. I should Say that, you are veeeryyy good. Keep Doin´ It. Small Comment 4 a Big Genius.


cdp_xe said...

Nice to read something new in your blog.

Do you still work on some of your (passive) covert channel projects?

Melissa said...

I tried to launch this program in my Vista notebook, but it didn't work:(
and in case of Linux systems it has a stricted option.

joanna said...

@Melissa: There could be many reasons for you failing to run NBP on your laptop, e.g. Vista Kernel Protection or maybe having SVM disabled in BIOS or maybe just not knowing that it requires dbgclient.sys if compiled with some debug macros. Also, I'm not quite sure what Linux has to do with all of this, as current NBP is implemented as a Windows kernel driver...

Please do not ask such questions on this blog -- this is not a "Kernel Drivers Building HOWTO". If you can't load NBP and you can't figure out why, then probably you should stay away from it ;)

heraux said...

MCP looks and LCP
LCP looks at Session(VM0)
and Session (VM1)
Session (VM0) is
talking to
Session (VM1)
Session (VM0) Linux
Session (VM1) Windows

but they actually hate

MCP is obviously called VN
not VM :D :P

水天一线 said...

I am worship you

joanna said...

@heraux: I have no idea what you mean by your comment, but I guess it's just me being too stupid to understand it ;)

Mikk said...

Hi Joanna,

I was wondering...does the license on the released version of Blue Pill allow it to be used at universities for academic research projects?


joanna said...

@mikk: yes. You can't use it to run e.g. commercial trainings/workshops though.

Anonymous said...


1) Can Bluepill be slid underneath a Hyper-V enabled parent OS and the guests would function as normal?

2) Do you see any way your software could attack ESXi as it is a specialized hypervisor-OS and does no provide general API:s?

3) Do you see any way Blue Pill could be able to fool VMSafe as it will be able to inspect the VM fom "underneath"?

Best regards
Henrik / Sweden

joanna said...

@Henrik: you will find answers to some(all?) of your question at this year's Black Hat Vegas. Stay tuned! :)

Gallus said...

I'm in love with the beautiful and ridiculous notion of a nested hypervisor loop, without any physical hardware -- a ring of virtual machines powered by their own self-belief. Something to shoot for, I suppose.

Anonymous said...

kvm also has patches to support nested virtualization