1984?

I can't believe this is happening for real...

So, how can we enforce Google to never do read() on a /var/users/john_smith/heath_records.db? How about a read() implemented from within a kernel-level via RAW-disk access that would never be logged?

I wonder when we would get another cool service from Google, e.g. "Google Thoughts", where people would be able to store their most private and personal thoughts, so that they could "access and managed them from all over the world", "in a secure fashion", of course. Right, add the Thought Police to this picture and welcome to Orwell's Oceania!

There is a difference between using the Web for blog writing vs. giving away all the private aspects of your life for free to some corporation in an unencrypted form. I wonder whether all the people who understands the notion of the local hard disk will be vaporized some time...

Vegas Training 2008

Last year we debuted with our Understanding Stealth Malware training at the Black Hat Vegas. We had about 70 participants and I think it was a reasonable success, especially that the training was announced very late. Since then we have done a couple of on-site classes and also have been continually updating the training.

During our 2nd public edition, at Black Hat Europe 2008 in March this year, we significantly extended the part about virtualization, e.g. by adding discussion of nested virtualization on AMD-v and showing and analyzing the actual code for implementing this. Also we have used the New Blue Pill code with VT-x support (previously it worked only on AMD-V), making it possible to use both AMD and Intel machines for the class. This allowed us to offer this training in a "Bring Your Own Laptop" fashion, that we know is much preferred by attendees, who simply feel better when using their own, known, work environment.

At the upcoming Black Hat Vegas 2008 we are also going to offer this class. That would be our 3rd public edition. Again, we hope to improve it even more beyond what we have presented at BH Europe 2008. Similarly as last time, we will not provide the computers, but rather expect the attendees to bring their own systems. At the end of this article are the requirements that should be met by your machine, if you would like to use it during the training and be able to do all the exercises. Of course, you should back up all your important data before coming to the class, as the computer might become corrupt after doing some of the exercises (although this has never happened so far).

There will be only one class offered on August 4/5 (the weekday class). You can view the detailed training agenda that we used for the BH Europe class in March here. Please note that the exact shape of the Vegas class is subject to be a bit different, as we are planning to add new material again.

This might be the very last chance for you to attended this specific training, as it's quite possible that next year we will be offering some other class, focused on Virtualization security entirely. Don't worry, however, if you don't get a seat in the Vegas class, there is still a chance to have that class presented on-site in your town.

You can register for the Vegas training here.

See you in Vegas!

Hardware Requirements

1. 64-bit (x64) AMD or Intel processor with hardware virtualization support (AMD-v or VT-x)
2. DVD-ROM
3. 2GB RAM (for convenient work with VMWare)

Software Requirement

1. 64-bit Vista OS (primary OS, non virtualized)
2. Windows Driver Kit (WDK) 6000 or newer (available via MSDN subscription).
3. VMWare Workstation 6.x or VMWare Player 2.x (the latter is free)
4. Optionally: IDA Pro 5.x disassembler (for exercises that involve finding bugs in drivers)

AMD Processors
Most modern AMD mobile processors, like e.g. AMD Turion and Athlon, used in modern laptops support AMD-v technology. Unfortunately there is no single place on AMD website that would provide the complete description of all CPUs that support AMD-v technology or provide an answer whether a given model does support it. When in doubt use google and always verify with the CHKSVMX program described below.

Intel Processors
Most modern Intel processors used in notebooks support Intel VT-x virtualization technology, this include Core 2 Solo, Core 2 Duo (except T5500, T5550 and T5750 models) and Core 2 Extreme. You can check your own model starting at this website, then chose your processor family and chose "Specifications" tab. Make sure the processor supports "Intel® 64 architecture" and "Intel® Virtualization Technology".

Using Mac for the training
You can very easily use MacBook or MacBook Pro for this training. You can easily install Windows on a second partition using the Boot Camp program that ships with all the newer Macs. You simply start Boot Camp application when running Mac OS X and then it automatically shrinks your current Mac partition, creates a new one for Windows, and asks to insert the installation media and reboots the system and you then perform normal Windows setup (after installation is complete your Vista should find all the necessary drivers via Windows Update). You might also want to use the free AutoHotKey program for the right-click emulation on your newly installed Vista. Please don't worry that Boot Camp tells that you should install a 32-bit Vista - you can ignore this and insert a 64-bit Vista installation disk.

Testing your machine with CHKSVMX
We have prepared a special little program, CHKSVMX, to test whether a given machine indeed supports hardware virtualization technology. The CHKSVMX program can be downloaded from here

The program doesn't introduce any persistent changes to the OS and doesn't require any installation procedure. It checks for virtualization support (on both AMD and Intel processors) not only by reading the CPUID information but also by trying to actually enable virtualization mode and then disable it again. Although most of the laptops available these days support hardware virtualization, in many cases this feature is disabled or locked down in the BIOS. If the virtualization is reported as "locked", please try to enable it in the BIOS. Please note that in most cases you will have to fully power down your system for the BIOS changes to take effect (reboot is not enough)!

Additionally CHKSVMX checks whether a 64-bit edition of Windows is running, as such OS is required for the training.

DISCLAIMER: The test program is digitally signed with the Invisible Things Lab's certificate and we assure that the program does not perform any malicious actions. ITL is, however, not responsible for any accidental damage or system instability issues the test program might cause.

Research Obfuscated

This article has been brought to my attention recently. It’s an “Open Letter to Joanna Rutkowska”, by Christofer Hoff over at the “Rational Survivability” blog. I decided to spend time reading and answering this piece as 1) technorati.com reported the blog’s authority as above 100 which suggests it has a reasonable number of readers, and also 2) because I believe this is a good example of the social engineering techniques used by my opponents and I couldn’t refrain myself from not commenting about this. Besides I felt a bit flattered that some individual decided to write an “Open Letter” to me, sort of like if I was a prime minister or some other important person ;)

Let me now analyze the letter, point by point:

1. Fire rules! The first thing that Hoff accuses me of in his letter is myself being an irresponsible individual, not caring about safety of my audience (not a joke!):
   

   “As the room filled to over capacity before your talk began, you were upset and couldn't seem to understand why the conference organizers would not let people spill over from seats and sit on the floor and in the aisles to hear you speak. The fact that fire and safety codes prohibit packing a room beyond capacity was something you attributed to people being "...crazy in America." Go figure.”

   Dear Christofer, if you only read my recent blog post about this very specific incident, read thoroughly shall I say, you would notice this paragraph undoubtedly:
   

   “Interestingly it was perfectly ok for the additional people to stay in the room, provided they arranged for additional chairs for themselves. In other words it was fine for people to sit and block the main aisle, provided they sit on chairs, but they couldn’t stay and sit on the same aisle without having a chair (maybe a "certificated" chair also), as that would be against the fire regulations!”

   Conclusion: I was not so much picking upon the fire regulations that forced people to leave the room, but rather on the idiotic rule, that allowed those same people to stay in this very same room, provided they also had additional chairs with them.
   


2. Type I vs. Type II hypervisors confusion. Hoff then switches to the actual content of the presentation and writes this:
   

   “When I spoke to you at the end of your presentation and made sure that I understood correctly that you were referring specifically to type-2 hosted virtualization on specific Intel and AMD chipsets, you conceded that this was the case.”

   This simply is an incorrect statement! On the contrary, when describing the security implications of nested virtualization (which was the actual new thing I was presenting at the RSA), I explicitly gave an example of how this could be used to compromise type I hypervisors. Kindly refer to slides 85-90 of my presentation that can be downloaded here.
   
   I said that the code we posted on bluepillproject.org indeed targets type II hypervisors and the only reason for that being that it has been built on top of our New Blue Pill code that was designed as a Windows kernel driver.


3. Shit not giving. Mr. Hoff goes even further:
   

   “When I attempted to suggest that while really interesting and intriguing, your presentation was not only confusing to many people but also excluded somewhere north of 80% of how most adopters have deployed virtualization (type-1 "bare-metal" vs. type-2 hosted) as well as excluding the market-leading virtualization platform, your response (and I quote directly) was: I don't give a shit, I'm a researcher.”

   Now that was a hard blow! I understand that the usage of such a slang expression by an Eastern European female during an informal conversation with a native speaker must have made an impression on him! However, I couldn’t give such an answer to this very question, simply because of the reasons given in point #2 (see above).
   
   If I remember correctly, I indeed used this very American expression to answer somebody’s concern (undoubtedly our Christofer Hoff’s) that most of the type I hypervisors out there are based on monolithic hypervisor architecture, and not on the micro-hypervisor architecture (and that I should not try to convince people to switch to micro-hypervisor architecture). In that context it makes it more logical for me to use the “I’m a researcher” as an excuse for not caring so much that most people use monolithic based hypervisors. Obviously, the usage of micro-hypervisors would allow to better secure the whole VMM infrastructure. And I also said, that I don’t care what people are using today, because I try to help to build a product that would be secure in the future (Phoenix’s HyperCore).


4. No obfuscation postulate. Hoff then comes up with some postulates that:
   

   “[I], as a researcher who is also actively courting publicity for commercial gain and speaking at conferences like RSA which are less technical and more "executive" in nature, you have a responsibility to clarify and not obfuscate (intentionally or otherwise) the facts surrounding your research.”

   This postulate is cleverly constructed because it also contains an embedded accusation of me being a commercially motivated researcher. Well, I never tried to hide that fact, and the reason for this is very simple: I consider security research as my job, and one of the primary goals of any job is to… bring commercial gain to the individual doing the job.
   
   Second, I really don’t understand what Hoff means by asking me to not obfuscate my research?! Maybe he was just disappointed that the presentation was too technical for an average CISSP to understand it? But, well, this presentation was classified as “Advanced Technical”, which was displayed in the conference program. I still did my best so that, say 70% of the material, was understandable to an average IT people, but, come on, there always must be some deep technical meat in any non-keynote-presentation, at least this is my idea for how a conference should look like.


5. Commercially motivated. Hoff accuses me of presenting commercial product, i.e. the Phoenix’s HyperSpace, during my speech:
   

   “No less than five times during your presentation, you highlighted marketing material in the form of graphics from Phoenix, positioned their upcoming products and announced/credited both Phoenix and AMD as funding your research.”

   Well, let me tell you this – this was one of the main reasons why I decided to speak at the RSA – just to announce this very product that I try to help to secure. Why would that be wrong?
   
   BTW, I have no idea how Mr. Hoff concluded that AMD was founding my research. I never said that, nor did I have it in my slides. Needles to say, AMD has not been founding my research. NOTE: interestingly I consider this particular mistake by Hoff to be accidental – at least I don’t see how this could be connected to any PR campaign, in contrast to all the other incorrectness he made use of.


6. Independence. Hoff, for some reason, apparently known only to him, tries to argue that I’m not an “independent researcher”:
   

   “I think it's only fair to point out that given your performance, you're not only an "independent researcher" but more so an "independent contractor." Using the "I'm a researcher" excuse doesn't cut it.”

   “I know it's subtle and lots of folks are funded by third parties, but they also do a much better job of drawing the line than you do.”

   Well, I found this one to be particularly amusing, as, for at least several years now, I have not claimed I have been an independent researcher.


7. Final hit. You might have been wondering by now – why this gentleman, nah, I think “the guy” would fit better here, so why the guy decided to spent so much time to write all those points, all those quasi-arguments and why he made so many “mistakes”? Well he seems to give an answer right in this paragraph:
   

   “I care very much that your research as presented to the press and at conferences like RSA isn't only built to be understood by highly skilled technicians or researchers because the continued thrashing that they generate without recourse is doing more harm than good, quite frankly.”

   Aha, now all is clear. May I ask then, which virtualization vendor you write PR for? ;)

So, then Hoff quotes the Forbes article that was written after my presentation and accuses me that the article (written by some Forbes reporter) was too sensationalist. I definitely agree the article was very sensationalist (but correct) and when I saw the article I even got angry and even wanted to write a blog about it (but as the article was actually correct, I had no good arguments to use against it). And you know why I was so angry? Because I actually spent over 40 minutes with this very Forbes reporter in the RSA’s speaker’s lounge just after my speech, I spent that time on clarifying to that guy what my presentation was about and what it was not about and what was the main message of the presentation. Still, the reporter had his own vision of how to write about it (i.e. make it into a sensation) and I hardly, as it turned out, could do anything about it…

So, what was the main massage of my presentation? Interestingly Mr. Hoff forgot to mention that… Let me then remind it here (a curious reader might want to have a look at the the slide #96 in my presentation):

• Virtualization technology could be used to improve security on desktop systems


• However there are non-trivial challenges in making this all working well...


• ... and not to introduce security problems instead...

Additionally, the message I was trying to pass during the whole presentation was:

“Keep hypervisors simple, do not put drivers there, as otherwise we would get to the same point where we are with current OSes these days, i.e. no kernel security at all!”

Now I wonder, maybe Christofer Hoff doesn’t do PR for any VMM vendor, maybe he just didn’t listen carefully to my presentation. Maybe he’s just one of those many guys who always know in advance what they want to hear and selectively pick up only those facts that match their state of mind? Otherwise, why would he not realize that my presentation was actually a pro-virtualization one and needed no (false) counter-arguments?

The Most Stupid Security News Ever

Seems like the BBC reporters have a shortage of subjects to write about these days… Maybe the next winter we will also be able to read about how many snowflakes fell during Christmas all over the world or something like that (which BTW, would still be way more interesting that the news quoted above).

I remember that some time ago, a group of researchers used automatic generators to create a few tens of thousands of variants of some malware, just to do some testing of A/V engines. And I remember how all the A/V people were complaining how irresponsible that was bla bla bla, as now they would have to work after hours to fight all this new malware. What a BS!

For any given class of a bug (think: exploits), or a file infection method (think: viruses), or a system compromise technique (think: rootkits, stealth malware), one can come up with pretty much infinite number of examples that would be exploiting the specific bug, the specific file infection method, or the specific system compromise technique. One virus would display you a “Hello, you’re being 0wned, sir.” Message, while the other one would just flash your keyboard leds. Sure, two different beings, but if exploiting the same mechanisms, also the protection against them is the same.

But, I know, it looks so cool in the news to read: “The number of viruses, worms and trojans in circulation has topped the one million mark”. It’s most definitely a good way to scare all the housewives and make them to rush to the computer shop at the coroner to buy the brand new A/V product that already can detect 99.9% out of all those scary things out there.

The RSA Absurd

Today I was giving a speech at the RSA Conference in San Francisco. The RSA is a really big conference and also seems to me like a very well organized one – e.g. they have all those computers at the registration hall where you put your name and then it immediately says to which check-in counter you should proceed and then when you get there they already have a badge waiting for you. Pretty cool stuff.

So my speech turned out to be scheduled in a very small room, say with seats for 100-200 people only (I haven't counted exactly). But then it turned out that there are more people interested in seeing the speech, so, as it usually happens on conferences, people started seating on the floor and also standing at the back of the room. I would say there was about 30% overflow, but still they could fit ok in the room. And then came this guy from the conference and said that all people who don’t have a seat should leave the room! It turned out that this is a fire regulation.

Interestingly it was perfectly ok for the additional people to stay in the room, provided they arranged for additional chairs for themselves. In other words it was fine for people to sit and block the main aisle, provided they sit on chairs, but they couldn’t stay and sit on the same aisle without having a chair (maybe a "certificated" chair also), as that would be against the fire regulations!

Yes, I know there are more examples of stupid pseudo-security rules (think airports), but, come on, this is on of the most well known security conference...

That situation annoyed me so much (because, of course, it turned out to be impossible to arrange for the additional chairs, so all those people had to leave) that I decided to submit this story to my blog using the totally unsecured public WiFi in my hotel. It was really unwise for me to do that, as Google’s Blogger uses HTTPS only for authentication (i.e. the login screen) but then it switches back to the good old plain text HTTP, making it possible for some evil guy sitting in the lobby to hijack my session. Is it that I miss something here or Google simple forgot that it is 2008 and not the 90’s anymore? Anyway, I'm just taking this risk bravely, hoping that the potential attacker, seeing my determination here, would refrain themselves from compromising this blog.

I know, I know, instead of complaining about Google, I should just move my blog to some other place. One day that’s gonna happen for sure :)

Kick Ass Hypervisor Nesting!

Remember how at the Black Hat Vegas 2007 I said that we still didn't support virtualization of full VMMs, like e.g. Virtual PC 2007 with hardware virtualization enabled, and that currently we could only run very simple hypervisors inside our New Blue Pill (like e.g. other NBPs inside NBP)? Remember how I said that we were working on this and should have a solution in about 2 months from then?

So, just about 2 weeks ago we did it! We can now virtualize complex hypervisors, like e.g. Virtual PC 2007 or Virtual Box with SVM turned on (BTW, we can also run VMWare Workstation, but that doesn't count, as on AMD processors it doesn't make use of SVM instructions). We also have a prototype code that allows to run nested hypervisors on VT-x but that code requires a bit of more polishing (oh, didn’t you know that our NBP also supports VT-x these days?).

I couldn't resist not to use my favorite Matrix analogy to describe what we do here: imagine Neo, who bravely followed The White Rabbit and finally decided to swallow The Red Pill, eventually awakes on The Nebuchadnezzar ship just to find out later that this whole "real world" is... just another Matrix...

I don't have a nice Matrix picture for that, so instead I will just show you a picture of a Virtual PC 2007 running inside an already bluepilled Vista and running Windows XP as its own guest. You can see that we use our "bpknock" testing program just to show we can intercept events in both the guest (i.e. the Vista that hosts the VPC hypervisor) as well as in the nested guest (the XP running inside the Virtual PC). This bpknock program simply executes CPUID instruction with some magic value in the RAX register and NBP intercepts that and answers with a magic RAX. BTW, there was no special reason to chose CPUID instruction for that, normally we don't need to intercept CPUID on AMD at all, so we could have chosen pretty much anything else, e.g. magic output to some magic I/O port.



It's worth mentioning that the only other working example of nested hardware virtualization I'm aware of is the IBM z/VM hypervisor for the IBM z series mainframe. If anybody knows any other example, please send me a link.

The research on nested virtualization has been supported by Phoenix Technologies, as the nested virtualization has also some positive applications. Phoenix is working on a cool product called HyperSpace. It consists of a hypervisor (called the "HyperCore") that allows running a few unmodified OSes inside hardware virtual machines so users can switch between them just like if they were virtual spaces on Mac or Linux. At the beginning there will be two virtual machines available: one running standard Vista and the other one based on Linux, that would contain some useful functionality like e.g. a Web browser, an email client and a multimedia suite, and also there would be something called the "ManageSpace" to manage this all.

So, how this is going to be different from e.g. XEN? The difference is that XEN is focused on server applications, while HyperSpace is intended for notebooks, which means it puts lots of efforts to offer comparable graphics (and other devices) performance as we have on normal non-virtualized laptops. This all will be possible because of the recent virtualization technology advances like e.g. VT-d/IOMMU.

At the RSA conference in San Francisco next week, I will be giving a speech that will discuss some technical problems we had to solve in order to get hardware nested hypervisoring working on AMD and also how the situation looks on Intel. I will also discuss how this changes the security battlefield and why virtualization vendors should care.

Back to Blue Pill -- the brand new source code with full virtualization support on AMD is now available on bluepillproject.org (you will need WDK6000 or newer to build it). Note that the (experimental) code for nested virtualization on Intel VT-x has been removed in this public version, leaving only the basic functionality if we run NBP on an Intel processor.

Also, please note that the code for AMD-v, even though it proved to be very stable, is still just a proof of concept. This means for example, that we don’t do any error-checks in the SVM instruction handlers, so it’s trivial for the nested hypervisor to simply crash the whole system if executing one of the SVM instructions with incorrect arguments or in an incorrect situation (e.g. CPL > 0). But that is hardly a problem for Blue Pill, as the guest isolation has never been a goal here. Of course, this could be simply addressed by adding a few more lines of code to each handler that would check for error conditions and inject #UD or #GP back to the nested hypervisor if it executed something incorrectly. Of course, we’re too lazy to code that ;)

So, what’s next? Well, we hope to show something even cooler at this year’s Black Hat Vegas, but I won’t say anything more now.

Razor-Thin Hypervisors

I just came back from Stockholm where I attended the Virtualization Forum, and saw several, quite interesting vendor presentations. One that caught my attention was a talk by VMware, and especially the part that talked about the new ESX 3i hypervisor and presented it as "razor-thin". This "razor-thin" hypervisor will have, according to VMWare, the footprint "of only 32MB".

My regular readers might sense that I’m a bit ironic here. Well, 32MB of code is definitely not a "razor-thin" hypervisor in my opinion and it’s not even close to a thin hypervisor... But why am I so picky about it? Is it really that important?

Yes, I think so, because the bigger the hypervisor the more chances that there is a bug somewhere out there. And one of the reasons for using virtual machines is to provide isolation. Even if we use virtualization because of business reasons (server consolidation), still we want each VM to be properly isolated, to make sure that if an attackers "gets into" one VM, she will not be able to 0wn all the other VMs on the same hardware… In other words, isolation of VMs, is an extremely important feature.

During my presentation I also talked about thin hypervisors. I first referenced a few bugs that were found in various VMMs in the recent months by other researchers (congrats to Rafal Wojtczuk of McAfee for some interestingly looking bugs in VMWare and Microsoft products). I used them as an argument that we should move towards very thin hypervisor architecture, exploiting hardware virtualization extension as much as possible (e.g. Nested Paging/EPT, IOMMU/DEV/NoDMA, etc) and avoiding doing things "in software".

Nobody should be really surprised seeing VMMs bugs – after all we have seen so many bugs in OS kernels over years, so no surprise we will see more and more bugs in VMMs, unless we switch to very thin hypervisors, so thin that it would be possible to understand and verify their code by one person. Only then we would be able to talk about security advantage (in terms of isolation) offered by VMMs comparing to traditional OSes.

I couldn’t refrain myself from mentioning that the existence of those bugs in popular VMMs clearly shows that having a VMM already installed doesn’t currently prevent from the "Blue Pill threat" – a point often expressed by some virtualization vendors, who notoriously try to diminish the importance of this problem (i.e. the problem of virtualization based malware).

I also announced that Invisible Things Lab has just started working with Phoenix Technologies. Phoenix is the world leader in system firmware, particularly known for providing BIOSes for PCs for almost 25 years, and currently is working on a new product called HyperCore that would be a very thin and lightweight hypervisor for consumer systems. ITL will be helping Phoenix to ensure the security of this product.

HyperCore hypervisor will use all the latest hardware virtualization extensions, like e.g. Nested Paging/EPT to minimize the unnecessary complexity and to provide negligible performance impact. For the same reasons, the I/O access will go through almost natively, just like in case of our Blue Pill...

Speaking about Blue Pill – Phoenix is also interested in further research on Blue Pill, which will be used as a test bed for trying various ideas – e.g. nested virtualization, which might be adopted in the future versions of HyperCore to allow users to use other commercial VMMs inside their already-virtualized OSes. Blue Pill’s small size and minimal functionality makes it a convenient tool for experimenting. Phoenix will also support The Blue Pill Project which means that some parts of our research will be available for other researchers (including code)!

In case you still feel like having a look into my slides, you can get them here.