Ah, there is no feeling like seeing your name in the news when drinking your morning coffee... In this piece some Steve Riley, a senior security strategist at Microsoft, decided to "rebute" our recent Black Hat presentations research results.
Mr. Riley had been quoted by ZDnet as saying:
"Her [Joanna Rutkowska] insistence is that you can replace the hypervisor without anybody knowing... Our assertion is that this is incorrect," Riley told the audience. "First of all, to do these attacks you need to become administrator at the root. So that's going to be, on an appropriately configured machine, an exceedingly difficult thing to happen."
Apparently, Mr. Riley has never seen our Black Hat presentations (or slides at least) that he is referring to (oh, wait, that is the typical case with all our "refuters", how come?)...
First, we never said anything about replacing the hypervisor. I really have no idea how this idea was born in Mr. Riley's head? Replacing the hypervisor - that would indeed be insane for us to do!
Second, it is not true that the attacker needs to become an administrator "at the root" (he mean the root partition or administrative domain here I assume). The attack we presented in our second speech, that exploited a heap overflow in the Xen hypervisor FLASK module, could have been conducted from the unprivileged domain, as we demonstrated during the presentation.
Mr. Riley continues with his vision:
"Because you [the attacker] didn't subject your own replacement hypervisor through the thorough design review that ours did, I'll bet your hypervisor is probably not going to implement 100 percent of the functionality as the original one," Riley said. "There will be a gap or two and we will be able to detect that."
Well, if he only took the effort of looking into our slides, he would realize that, in case of XenBluePill, we were slipping it beneath (not replacing!) the original hypervisor, and then run the original one as nested. So, all the functionality of the original hypervisor was preserved.
Mr. Riley also shares some other ground breaking thoughts in this article, but I think we can leave them uncommented ;)
This situation is pretty funny actually - we have here the words and feelings of some Microsoft executive vs. our three technical presentations, all the code that we released for those presentations, and also a few of our demos. Yet, it's apparently still worth getting into the news and reporting what the feeling of Mr. Riley are...
Let me, however, write one more time, that I'm (still) not a Microsoft hater. There are many people at Microsoft that I respect: Brandon Baker, Neil Clift, the LSD guys, Mark Russinovich, and probably a few more that I just haven't had occasion to meet in person or maybe forgot about at the moment. It's thus even more sad that people like Mr. Riley are also associated with Microsoft, even more they are the face of Microsoft for the majority of people. Throwing a party in Vegas and Amsterdam once a year certainly is not enough to change the Microsoft's image in this case...
Interestingly, if Mr. Riley only attended our Xen 0wning Trilogy at Black Hat, then he would notice that we were actually very positive about Hyper-V. Of course, I pointed out that Xen 3.3 certainly has a more secure architecture right now, but I also said that I knew (from talking to some MS engineers from the virtualization group) that Hyper-V is going to implement similar features in the next version(s) and that this is very good. I also prized the fact it has only about 100k LOC (vs. about 300k LOC in Xen 3.3).
So, Mr. Senior Security Strategist, I suggest you do your homework more carefully next time before throwing mud at others and trying to negate the value of their work (and all the efforts of Microsoft's PR people).
On a separate note, I found it quite unprofessional that ZDNet's Liam Tung and Tom Espiner, the authors of the news, didn't ask me for a commentary before publishing this. Not to mention that they also misspelled Rafal's name and forgot to mention about Alex, the third co-author of the presentations.
Sunday, September 07, 2008
Saturday, September 06, 2008
Xen 0wning Trilogy: code, demos and q35 attack details posted
We have posted all the code that we used last month during our Black Hat presentations about Xen security, and you can get it here. This includes the full source code for:
1) The generic Xen Loadable Modules framework
2) Implementation of the two Xen Hypervisor Rootkits
3) The Q35 exploit
4) The FLASK heap overflow exploit
5) The BluePillBoot (with nested virtualization support on SVM)
6) The XenBluePill (with nested virtualization support on SVM)
Beware the code is by far not user-friendly, it requires advanced Linux/Xen, C and system-level programming skills in order to tweak some constants and run it successfully on your system. Do not send us questions how to compile/run it, as we don’t have time to answer such questions. Also do not send questions how the code works – if you can’t figure it out by reading our slides and the source code, then it means you should probably spend more time on this yourself. On the other hand, we would appreciate any constructive feedback.
The code is our gift to the research community. There is no warranty and Invisible Things Lab takes no responsibility for any potential damage that this code might cause (e.g. by rebooting your machine) or any potential malicious usage of this code, or any other code built on top of this code. We believe that by publishing this code we help to create more secure systems in the future.
Additionally, we also posted the full version of our second Black Hat talk, which now includes all the slides about the Q35 bug and how we exploited it. Those slides had to be previously removed during our Black Hat presentation, as the patch was still unavailable during that time.
1) The generic Xen Loadable Modules framework
2) Implementation of the two Xen Hypervisor Rootkits
3) The Q35 exploit
4) The FLASK heap overflow exploit
5) The BluePillBoot (with nested virtualization support on SVM)
6) The XenBluePill (with nested virtualization support on SVM)
Beware the code is by far not user-friendly, it requires advanced Linux/Xen, C and system-level programming skills in order to tweak some constants and run it successfully on your system. Do not send us questions how to compile/run it, as we don’t have time to answer such questions. Also do not send questions how the code works – if you can’t figure it out by reading our slides and the source code, then it means you should probably spend more time on this yourself. On the other hand, we would appreciate any constructive feedback.
The code is our gift to the research community. There is no warranty and Invisible Things Lab takes no responsibility for any potential damage that this code might cause (e.g. by rebooting your machine) or any potential malicious usage of this code, or any other code built on top of this code. We believe that by publishing this code we help to create more secure systems in the future.
Additionally, we also posted the full version of our second Black Hat talk, which now includes all the slides about the Q35 bug and how we exploited it. Those slides had to be previously removed during our Black Hat presentation, as the patch was still unavailable during that time.
Tuesday, September 02, 2008
The three approaches to computer security
If we looked at the computer systems and how they try to provide security, I think we could categorize those attempts into three broad categories:
1) Security by Correctness
2) Security by Isolation
3) Security by Obscurity
Let's discuss those categories in more detail below.
Security by Correctness
The assumption here is obvious: if we can produce software that doesn't have bugs (nor any maliciously behaving code), then we don't have security problems at all. The only problem is that we don't have any tools to make sure that a given code is correct (in terms of implementation, design and ethical behavior). But if we look at various efforts in computer science, we will notice a lot of effort has been made to achieve Security by Correctness: "safe" languages, code verifiers (although not sound ones, just heuristic based), developer's education, manual code audit, etc. Microsoft's famed Secure Development Life-cycle is all about Security by Correctness. The only problem is: all those approaches sometimes work and sometimes do not, sometimes they miss some bug and also there are problems that I simple don't believe can be addresses by automatic code verifiers or even safe languages, like e.g. logic/design bugs or deciding on wheatear a given code behaves maliciously or not (after all this is an ethical problem in many cases, not a computer science problem).
To sum it: I think that in some more or less distant future (some people think abuout a timeframe of 50 years or so), we would get rid of all the implementation bugs, thanks to safe languages and/or sound code verifiers. But I don't believe we could assure correctness of software on any higher level of abstraction then implementation level.
Security by Isolation
Because of the problems with effectively implementing Security by Correctness approach, people, from the very beginning, has also taken another approach, which is based on isolation. The idea is to split a computer system into smaller pieces and make sure that each piece is separated from the other ones, so that if it gets compromised/malfunctions, then it cannot affect the other entities in the system. Early UNIX's user accounts and separate process address spaces, things that are now present in every modern OS, are examples of Security by Isolation.
Simple as it sound, in practice the isolation approach turned out to be very tricky to implement. One problem is how to partition the system into meaningful pieces and how to set permissions for each piece. The other problem is implementation - e.g. if we take a contemporary consumer OS, like Vista, Linux or Mac OSX, all of them have monolithic kernels, meaning that a simple bug in any of the kernel components (think: hundreds of 3rd party drivers running there), allows to bypass of the isolation mechanisms provided by the kernel to the rest of the system (process separation, ACLs, etc).
Obviously the problem is because the kernels are monolithic. Why not implement Security by Isolation on a kernel level then? Well, I would personally love that approach, but the industry simply took another course and decided that monolithic kernels are better then micro-kernels, because it's easier to write the code for them and (arguably) they offer better performance.
Many believe, including myself, that this landscape can be changed by the virtualization technology. Thin bare-metal hypervisor, like e.g. Xen, can act like a micro kernel and enforce isolation between other components in the system - e.g. we can move drivers into a separate domain and isolate them from the rest of the system. But again there are challenges here on both the design- as well as the implementation-level. For example, we should not put all the drivers into the same domain, as this would provide little improvement in security. Also, how to make sure that the hypervisor itself is not buggy?
Security by Obscurity (or Security by Randomization)
Finally we have the Security by Obscurity approach that is based on the assumption that we cannot get rid of all the bugs (like in Security by Isolation approach), but at least we can make exploitation of those bugs very hard. So, it's all about making our system unfriendly to the attacker.
Examples of this approach include Address Space Layout Randomization (ASLR, present in all newer OSes, like Linux, Vista, OSX), StackGuard-like protections (again used by most contemporary OSes), pointer encryption (Windows and Linux) and probably some other mechanisms that I can't remember at the moment. Probably the most extreme example of Security by Obscurity would be to use a compiler that generates heavily obfuscated binaries from the source code and creates a unique (on a binary level) instances of the same system. Alex did his PhD on this topic and his an expert on compilers and obfuscators.
The obvious disadvantage of this approach is that it doesn't prevent the bugs from being exploited - it only make the meaningful exploitation very hard or even impossible. But if one is concerned also about e.g. DoS attacks, then Security by Obscurity will not prevent them in most cases. The other problem with obfuscating the code is the performance (compiler cannot optimize the code for speed) and maintenance (if we got a crash dump on an "obfuscated" Windows box, we couldn't count on help from the technical support). Finally there is a problem of proving that the whole scheme is correct and that our obfuscator (or e.g. ASLR engine) doesn't introduce bugs to the generated code and that we will not get random crashes later (that we would be most likely unable to debug, as the code will be obfuscated).
I wonder if the above categorization is complete and if I haven't forgotten about something. If you know an example of a security approach that doesn't fit here (besides blacklisiting), please let me know!
1) Security by Correctness
2) Security by Isolation
3) Security by Obscurity
Let's discuss those categories in more detail below.
Security by Correctness
The assumption here is obvious: if we can produce software that doesn't have bugs (nor any maliciously behaving code), then we don't have security problems at all. The only problem is that we don't have any tools to make sure that a given code is correct (in terms of implementation, design and ethical behavior). But if we look at various efforts in computer science, we will notice a lot of effort has been made to achieve Security by Correctness: "safe" languages, code verifiers (although not sound ones, just heuristic based), developer's education, manual code audit, etc. Microsoft's famed Secure Development Life-cycle is all about Security by Correctness. The only problem is: all those approaches sometimes work and sometimes do not, sometimes they miss some bug and also there are problems that I simple don't believe can be addresses by automatic code verifiers or even safe languages, like e.g. logic/design bugs or deciding on wheatear a given code behaves maliciously or not (after all this is an ethical problem in many cases, not a computer science problem).
To sum it: I think that in some more or less distant future (some people think abuout a timeframe of 50 years or so), we would get rid of all the implementation bugs, thanks to safe languages and/or sound code verifiers. But I don't believe we could assure correctness of software on any higher level of abstraction then implementation level.
Security by Isolation
Because of the problems with effectively implementing Security by Correctness approach, people, from the very beginning, has also taken another approach, which is based on isolation. The idea is to split a computer system into smaller pieces and make sure that each piece is separated from the other ones, so that if it gets compromised/malfunctions, then it cannot affect the other entities in the system. Early UNIX's user accounts and separate process address spaces, things that are now present in every modern OS, are examples of Security by Isolation.
Simple as it sound, in practice the isolation approach turned out to be very tricky to implement. One problem is how to partition the system into meaningful pieces and how to set permissions for each piece. The other problem is implementation - e.g. if we take a contemporary consumer OS, like Vista, Linux or Mac OSX, all of them have monolithic kernels, meaning that a simple bug in any of the kernel components (think: hundreds of 3rd party drivers running there), allows to bypass of the isolation mechanisms provided by the kernel to the rest of the system (process separation, ACLs, etc).
Obviously the problem is because the kernels are monolithic. Why not implement Security by Isolation on a kernel level then? Well, I would personally love that approach, but the industry simply took another course and decided that monolithic kernels are better then micro-kernels, because it's easier to write the code for them and (arguably) they offer better performance.
Many believe, including myself, that this landscape can be changed by the virtualization technology. Thin bare-metal hypervisor, like e.g. Xen, can act like a micro kernel and enforce isolation between other components in the system - e.g. we can move drivers into a separate domain and isolate them from the rest of the system. But again there are challenges here on both the design- as well as the implementation-level. For example, we should not put all the drivers into the same domain, as this would provide little improvement in security. Also, how to make sure that the hypervisor itself is not buggy?
Security by Obscurity (or Security by Randomization)
Finally we have the Security by Obscurity approach that is based on the assumption that we cannot get rid of all the bugs (like in Security by Isolation approach), but at least we can make exploitation of those bugs very hard. So, it's all about making our system unfriendly to the attacker.
Examples of this approach include Address Space Layout Randomization (ASLR, present in all newer OSes, like Linux, Vista, OSX), StackGuard-like protections (again used by most contemporary OSes), pointer encryption (Windows and Linux) and probably some other mechanisms that I can't remember at the moment. Probably the most extreme example of Security by Obscurity would be to use a compiler that generates heavily obfuscated binaries from the source code and creates a unique (on a binary level) instances of the same system. Alex did his PhD on this topic and his an expert on compilers and obfuscators.
The obvious disadvantage of this approach is that it doesn't prevent the bugs from being exploited - it only make the meaningful exploitation very hard or even impossible. But if one is concerned also about e.g. DoS attacks, then Security by Obscurity will not prevent them in most cases. The other problem with obfuscating the code is the performance (compiler cannot optimize the code for speed) and maintenance (if we got a crash dump on an "obfuscated" Windows box, we couldn't count on help from the technical support). Finally there is a problem of proving that the whole scheme is correct and that our obfuscator (or e.g. ASLR engine) doesn't introduce bugs to the generated code and that we will not get random crashes later (that we would be most likely unable to debug, as the code will be obfuscated).
I wonder if the above categorization is complete and if I haven't forgotten about something. If you know an example of a security approach that doesn't fit here (besides blacklisiting), please let me know!
Sunday, August 31, 2008
Teamwork & Crediting
As the technology is getting more and more complex, security research, especially offensive security research on a system level, becoming more and more difficult to be done by one person. NX/XD, ASLR, various StackGuard-like things, VT-d, TXT, etc... - all those technologies leave less and less space for the interesting system-level attacks. On the other hand, the widespread "deployment" of Web 2.0 creates a whole new area to explore, but that is a whole different world (plus there are still all those "human factor" attacks that exploit user stupidity, but again, this is a different area).
Our Xen 0wning Trilogy is a good example of how a team of researchers can still come up with interesting new system-level attacks against the very recent and securely design system. Take XenBluePill as an example.
It has first been months of research and coding done by Alex and myself to support nested hardware virtualization on AMD. Then there was months of Rafal's research about how to load code into the running Xen on the fly ("Xen Loadable Modules"). That required ability to access Xen's memory in the first place and Rafal's way for doing that was to use the DMA attack. But then it turned out that the Xen 3.3 uses VT-d protection to protect against this very kind of attacks. So then I came up with the "Q35 attack" that exploited a problem with recent Intel BIOSes on recent motherboards (details are coming this week). But I based my attack on a similar SMM attack that Rafal came up with a few months earlier on a different chipset, when he was looking into ways to compromise SMM handler, as we started thinking about HyperGuard project back then and Rafal was curious reliable the SMM protection is. In the meantime, Alex "converted" our working New Blue Pill that had full support for nested virtualization but was essentially a Windows driver, into a piece of code that was completely OS-independent (own memory management, etc.). Then I finally took Rafal's XLM framework, added a few minor things that were needed to load our "Windows-independent Windows driver" into Xen using XLM, fixed some minor stuff and... it finally worked! But that was possible only because of the joint work by all the three people together.
So, it is simply unfair to attribute all the glory and fame for our research to "Rutkowska" or "Rutkowska and team", as many news portals did. Please don't forget to credit all the co-authors! If you really would like to use a generic term, then "Invisible Things Lab team" would probably serve better.
Speaking of our team, I also have an announcement that starting this month our team has officially been extended by yet another person: Rong Fan from Beijing, China.
Rong is a software engineer, focusing on Intel's hardware virtualization technology (VT). A few months ago he wrote to me with some advanced questions regarding the implementation of our New BluePill that we published after the last year's Black Hat. Turned out that Rong, as part of his after-hour activity, is porting Bluepill to VT-x. After he succeeded, we decided to share our nested virtualization code for AMD with him so that he could investigate how to do it on VT-x. And about 2 months ago Rong succeeded with implementing full nested virtualization support for our NBP on Intel VT-x! During that time Rong has had an opportunity to find out that working with ITL is quite fun, so he decided to quit his job at Lenovo and joined ITL full time. Right now Rong is busy adding nested VT-x support to a normal Xen hypervisor.
So, Invisible Things Lab is all about the team work. The whole idea behind ITL is to gather together a bunch of smart people, so that we could all work on the most exciting problems together. Problems that might be too complex or time-consuming for just one person to solve. But it takes more then just money to get people to be creative and devote themselves to work. Getting recognition is one of the additional factors often needed. That's why ITL is not interested in "hiding" its employees, but rather in promoting their work and fairly crediting them.
Our Xen 0wning Trilogy is a good example of how a team of researchers can still come up with interesting new system-level attacks against the very recent and securely design system. Take XenBluePill as an example.
It has first been months of research and coding done by Alex and myself to support nested hardware virtualization on AMD. Then there was months of Rafal's research about how to load code into the running Xen on the fly ("Xen Loadable Modules"). That required ability to access Xen's memory in the first place and Rafal's way for doing that was to use the DMA attack. But then it turned out that the Xen 3.3 uses VT-d protection to protect against this very kind of attacks. So then I came up with the "Q35 attack" that exploited a problem with recent Intel BIOSes on recent motherboards (details are coming this week). But I based my attack on a similar SMM attack that Rafal came up with a few months earlier on a different chipset, when he was looking into ways to compromise SMM handler, as we started thinking about HyperGuard project back then and Rafal was curious reliable the SMM protection is. In the meantime, Alex "converted" our working New Blue Pill that had full support for nested virtualization but was essentially a Windows driver, into a piece of code that was completely OS-independent (own memory management, etc.). Then I finally took Rafal's XLM framework, added a few minor things that were needed to load our "Windows-independent Windows driver" into Xen using XLM, fixed some minor stuff and... it finally worked! But that was possible only because of the joint work by all the three people together.
So, it is simply unfair to attribute all the glory and fame for our research to "Rutkowska" or "Rutkowska and team", as many news portals did. Please don't forget to credit all the co-authors! If you really would like to use a generic term, then "Invisible Things Lab team" would probably serve better.
Speaking of our team, I also have an announcement that starting this month our team has officially been extended by yet another person: Rong Fan from Beijing, China.
Rong is a software engineer, focusing on Intel's hardware virtualization technology (VT). A few months ago he wrote to me with some advanced questions regarding the implementation of our New BluePill that we published after the last year's Black Hat. Turned out that Rong, as part of his after-hour activity, is porting Bluepill to VT-x. After he succeeded, we decided to share our nested virtualization code for AMD with him so that he could investigate how to do it on VT-x. And about 2 months ago Rong succeeded with implementing full nested virtualization support for our NBP on Intel VT-x! During that time Rong has had an opportunity to find out that working with ITL is quite fun, so he decided to quit his job at Lenovo and joined ITL full time. Right now Rong is busy adding nested VT-x support to a normal Xen hypervisor.
So, Invisible Things Lab is all about the team work. The whole idea behind ITL is to gather together a bunch of smart people, so that we could all work on the most exciting problems together. Problems that might be too complex or time-consuming for just one person to solve. But it takes more then just money to get people to be creative and devote themselves to work. Getting recognition is one of the additional factors often needed. That's why ITL is not interested in "hiding" its employees, but rather in promoting their work and fairly crediting them.
Tuesday, August 26, 2008
Intel patches the Q35 bug
Yesterday Intel has published an official advisory that addresses the Q35 bug and attack, that we used during Black Hat as one of the ways to subvert Xen 3.3 on a VT-d enabled system (the alternative way was to use the Xen-specific FLASK exploit, that worked even from an unprivileged domain).
One small clarification though: in the advisory they stated that: "Software running administrative (ring 0) privilege can under certain circumstances change code running in System Management Mode." But in fact an attacker might also use this bug to directly modify the hypervisor memory, without jumping into the SMM first, just as we did it with our exploit. Also, in case of e.g. Linux systems, the Ring0 access is not strictly required to perform the attack, as it's just enough for the attacker to get access to the PCI config space of the device 0:0:0, which e.g. on Linux can be granted to usermode applications via the iopl() system call.
You can download a new firmware for your motherboard from here.
Intel did a good job on handling this bug - not only they recognized the importance of the attack, but also released the patch promptly. Quite positively surprising as for such a big company.
So, now we're free to publish all the missing slides about how we exploit this vulnerability that we had to remove from our Black Hat presentation, as well as the exploit code. However, as I'm going to give 2 presentations at the upcoming ISF conference in Sweden early next week, I thought it would be logical to wait with disclosing this material and present it at this conference, during my technical speech (I will also deliver the keynote for this conference). Of course, as soon as I will get back home (Thursday next week), we will publish the full slides, exploit codes and all the demos, as promised earlier.
Speaking of speaking: also next month, Rafal will fly to Oregon, to Intel campus, for the Intel Virtualization Security Summit, where he will deliver a "compressed" version of our Xen 0wning Trilogy to the technical crowd of Intel employees. Rafal will provide some more details about the HyperGuard project that we do in cooperation with Phoenix Technologies. Also, in October, Alex will visit Kuala Lumpur and present an updated Bluepilling the Xen Hypervisor talk at the Hack In The Box conference.
One small clarification though: in the advisory they stated that: "Software running administrative (ring 0) privilege can under certain circumstances change code running in System Management Mode." But in fact an attacker might also use this bug to directly modify the hypervisor memory, without jumping into the SMM first, just as we did it with our exploit. Also, in case of e.g. Linux systems, the Ring0 access is not strictly required to perform the attack, as it's just enough for the attacker to get access to the PCI config space of the device 0:0:0, which e.g. on Linux can be granted to usermode applications via the iopl() system call.
You can download a new firmware for your motherboard from here.
Intel did a good job on handling this bug - not only they recognized the importance of the attack, but also released the patch promptly. Quite positively surprising as for such a big company.
So, now we're free to publish all the missing slides about how we exploit this vulnerability that we had to remove from our Black Hat presentation, as well as the exploit code. However, as I'm going to give 2 presentations at the upcoming ISF conference in Sweden early next week, I thought it would be logical to wait with disclosing this material and present it at this conference, during my technical speech (I will also deliver the keynote for this conference). Of course, as soon as I will get back home (Thursday next week), we will publish the full slides, exploit codes and all the demos, as promised earlier.
Speaking of speaking: also next month, Rafal will fly to Oregon, to Intel campus, for the Intel Virtualization Security Summit, where he will deliver a "compressed" version of our Xen 0wning Trilogy to the technical crowd of Intel employees. Rafal will provide some more details about the HyperGuard project that we do in cooperation with Phoenix Technologies. Also, in October, Alex will visit Kuala Lumpur and present an updated Bluepilling the Xen Hypervisor talk at the Hack In The Box conference.
Wednesday, August 20, 2008
Attacking Xen: DomU vs. Dom0 consideration
As it usually happens, there is some confusion regarding the attacks presented in our Xen 0wning Trilogy. Some people think they are possible only from Dom0 (the Xen's privileged, administrative domain) while some other people have the impression that all the attacks are possible from any unprivileged domain (DomU in Xen’s terminology). The truth is in the middle though.
Most of the attacks we presented do indeed require that the attacker first obtained access to Dom0 and only from there can launch further attacks. For example the DMA attacks that allow to overwrite hypervisor memory do indeed assume Dom0 access. The same applies to the Q35 exploit - this one is similar to the above mentioned DMA attacks in that it also requires access to certain hardware (that is possible from Dom0), but has an advantage that can bypass the hypervisor VT-d protection in the recent Xen 3.3.
There are several reasons why those attacks are still very important though:
1) First note, that the requirement for the attacker to have access to Dom0 in order to install e.g. hypervisor rootkits, is similar to the requirement that in order to install a Windows or Linux rootkit, one first needs to obtain administrator's privileges. Yet we know that Windows or Linux rootkits is a serious security problem.
2) Concerning the Xen-based systems specifically: over the last year several bugs have been discovered and published, that allowed an attacker to gain control of the Domain0 from an unprivileged domain (i.e. escape from the virtual machine). Rafal has discovered one such bug in December 2007.
3) Recent versions of Xen make deliberate effort to protect the hypervisor even from the Dom0. On systems that have IOMMU support (e.g. Intel's VT-d), the hypervisor memory is protected from tampering using both the processor's ring3/ring0 separation mechanism as well as the IOMMU protection. We showed that those protections can be bypassed.
The attempt to isolate hypervisor and protect it even from attacks originating from Dom0 is not surprising. After all if we would like to treat the hypervisor as a root of trust, then we should make sure that its code base is minimal. If we now allow Dom0 to effectively be the hypervisor (i.e. if we don't care about Dom0-to-hypervisor escalations) then we should include all the Dom0 code to the hypervisor code base, when evaluating hypervisor security. This would result in our "extended hypervisor" having not ~300k lines of code (like current Xen does), but millions of lines of code!
Having said that all about how important it is to prevent all the possible Dom0-to-hypervisor attacks, I should stress that we also presented an attack that does not require Dom0 access and that can be spawned from an unprivileged DomU domain. As far as I'm aware, Rafal's FLASK bug & exploit (presented in the 2nd presentation) was the first public example of a successful exploitation of an overflow in a bare-metal hypervisor. The bug was a heap overflow and Rafal presented some clever tricks of how to control the Xen's heap allocations in order to make this bug exploitable.
Please note that all the rootkit-like stuff that we also presented, i.e. Rafal's Xen Loadable Modules framework and his hypervisor rootkits, as well as Alex's and mine XenBluePill, can all be used with all the above mentioned attacks. So, e.g. if we are on a machine that has VT-d support and run Xen 3.3 we can still use the Q35 attack and get the XLM framework running and then use it to install e.g. XenBluePill on top of running Xen, as showed during the 3rd presentation. Similarly, we could use the FLASK exploit and get XLM running again and again used it for installation of the other stuff.
Hope this clears some confusion about our presentations. As already promised, the codes and demos and full version of the 2nd talk slides (with the Q35 attack details) will be posted after Intel release the patch for their motherboards. Stay tuned.
Most of the attacks we presented do indeed require that the attacker first obtained access to Dom0 and only from there can launch further attacks. For example the DMA attacks that allow to overwrite hypervisor memory do indeed assume Dom0 access. The same applies to the Q35 exploit - this one is similar to the above mentioned DMA attacks in that it also requires access to certain hardware (that is possible from Dom0), but has an advantage that can bypass the hypervisor VT-d protection in the recent Xen 3.3.
There are several reasons why those attacks are still very important though:
1) First note, that the requirement for the attacker to have access to Dom0 in order to install e.g. hypervisor rootkits, is similar to the requirement that in order to install a Windows or Linux rootkit, one first needs to obtain administrator's privileges. Yet we know that Windows or Linux rootkits is a serious security problem.
2) Concerning the Xen-based systems specifically: over the last year several bugs have been discovered and published, that allowed an attacker to gain control of the Domain0 from an unprivileged domain (i.e. escape from the virtual machine). Rafal has discovered one such bug in December 2007.
3) Recent versions of Xen make deliberate effort to protect the hypervisor even from the Dom0. On systems that have IOMMU support (e.g. Intel's VT-d), the hypervisor memory is protected from tampering using both the processor's ring3/ring0 separation mechanism as well as the IOMMU protection. We showed that those protections can be bypassed.
The attempt to isolate hypervisor and protect it even from attacks originating from Dom0 is not surprising. After all if we would like to treat the hypervisor as a root of trust, then we should make sure that its code base is minimal. If we now allow Dom0 to effectively be the hypervisor (i.e. if we don't care about Dom0-to-hypervisor escalations) then we should include all the Dom0 code to the hypervisor code base, when evaluating hypervisor security. This would result in our "extended hypervisor" having not ~300k lines of code (like current Xen does), but millions of lines of code!
Having said that all about how important it is to prevent all the possible Dom0-to-hypervisor attacks, I should stress that we also presented an attack that does not require Dom0 access and that can be spawned from an unprivileged DomU domain. As far as I'm aware, Rafal's FLASK bug & exploit (presented in the 2nd presentation) was the first public example of a successful exploitation of an overflow in a bare-metal hypervisor. The bug was a heap overflow and Rafal presented some clever tricks of how to control the Xen's heap allocations in order to make this bug exploitable.
Please note that all the rootkit-like stuff that we also presented, i.e. Rafal's Xen Loadable Modules framework and his hypervisor rootkits, as well as Alex's and mine XenBluePill, can all be used with all the above mentioned attacks. So, e.g. if we are on a machine that has VT-d support and run Xen 3.3 we can still use the Q35 attack and get the XLM framework running and then use it to install e.g. XenBluePill on top of running Xen, as showed during the 3rd presentation. Similarly, we could use the FLASK exploit and get XLM running again and again used it for installation of the other stuff.
Hope this clears some confusion about our presentations. As already promised, the codes and demos and full version of the 2nd talk slides (with the Q35 attack details) will be posted after Intel release the patch for their motherboards. Stay tuned.
Friday, August 08, 2008
Our Xen 0wning Trilogy Highlights
Below you can find highlights of the three presentations, collectively referred to as "Xen 0wning Trilogy", that Alex, Rafal and I gave today at the Black Hat conference in Las Vegas.
Talk #1
1) Practical implementation of reliable and portable DMA attacks from Domain 0 to the Xen hypervisor memory.
2) Xen Loadable Modules :) A framework that allows to load arbitrary C code modules into the running Xen hypervisor. It uses DMA attack from the previous point to get access to Xen memory.
3) Two implementations of Xen Hypervisor Rootkits. This was the first time that working hypervisor rootkits have been presented (note the distinction between hypervisor rootkit vs. virtualization based rootkits).
Talk #2
1) Discussed how Xen 3.3 makes use of the Intel VT-d technology to protect the hypervisor.
2) Then we discussed how to bypass such VT-d protection on certain motherboards, like e.g. Intel DQ35 board.
3) An extra bonus: our attack from the previous point allows also to subvert the SMM handler and e.g. install an SMM rootkit in the system.
4) Discussed other Xen security mechanisms like driver domains, stub domains, PV GRUB and also attempted to quickly compare the state of Xen security design with the Hyper-V and ESX hypervisor.
5) Showed an exploitable heap overflow bug in the Xen hypervisor. The bug was in the FLASK module -- the NSA implementation of Xen Security Modules. FLASK, however, is not turned on by default, so even though we showed how to successfully exploit this heap overflow (which results in an escape from an unprivileged domain directly to the hypervisor), this is not a bug that can be used to 0wn The Planet. It shows, however, what happens when people start adding more and more code into the hypervisor.
6) Introduced HyperGuard -- a project done in cooperation with Phoenix Technologies. HyperGuard is going to be a SMM-based integrity scanner for Xen-like hypervisors. With HyperGuard we take a different approach then other integrity scanners do -- rather than ensuring the correctness of the code and data of the hypervisor, which might be very tricky, we instead ensure there is no untrusted code in the hypervisor, which is a much simpler task.
Talk #3
1) Provided detailed description of how to implement nested hardware based virtualization on AMD-V and VT-x (a copy of the slides from my RSA speech in April).
2) Showed how to use this nested virtualization to implement Blue Pill Boot, that can be used to virtualize the system right from the boot stage. We mentioned the best defend against this kind of system compromises is a trusted boot mechanism, either SRTM or DRTM, as implemented e.g. by Xen's tboot.
3) Consequently we showed Xen Blue Pill that is able to move a running Xen system into a virtual machine on the fly. This, on the other hand, cannot be prevented by neither the SRTM nor DRTM technology. XBP is a good example that running a legitimate hypervisor doesn't always prevent bluepill-like malware from being installed in the system.
4) Finally, discussed the XBP detection. First, we noted that all the "VMM detectors", proposed over the last years, that try to detect if there is a hypervisor running above, are useless in the case of a bluepilled Xen system. The only one approach that could be used is the direct timing analysis of the #VMEXIT times in order to distinguish between the native Xen case vs. bluepilled Xen case. We noted however, that direct timing analysis will not observe any differences when run from PV domains on AMD processors, and that it will observe little difference when run from HVM domains (7k vs. 5k cycles). The detection is easier on Intel processors, because of the unconditional #VMEXIT that we cannot get rid of.
All the three talks can be found here.
Talk #1
1) Practical implementation of reliable and portable DMA attacks from Domain 0 to the Xen hypervisor memory.
2) Xen Loadable Modules :) A framework that allows to load arbitrary C code modules into the running Xen hypervisor. It uses DMA attack from the previous point to get access to Xen memory.
3) Two implementations of Xen Hypervisor Rootkits. This was the first time that working hypervisor rootkits have been presented (note the distinction between hypervisor rootkit vs. virtualization based rootkits).
Talk #2
1) Discussed how Xen 3.3 makes use of the Intel VT-d technology to protect the hypervisor.
2) Then we discussed how to bypass such VT-d protection on certain motherboards, like e.g. Intel DQ35 board.
3) An extra bonus: our attack from the previous point allows also to subvert the SMM handler and e.g. install an SMM rootkit in the system.
4) Discussed other Xen security mechanisms like driver domains, stub domains, PV GRUB and also attempted to quickly compare the state of Xen security design with the Hyper-V and ESX hypervisor.
5) Showed an exploitable heap overflow bug in the Xen hypervisor. The bug was in the FLASK module -- the NSA implementation of Xen Security Modules. FLASK, however, is not turned on by default, so even though we showed how to successfully exploit this heap overflow (which results in an escape from an unprivileged domain directly to the hypervisor), this is not a bug that can be used to 0wn The Planet. It shows, however, what happens when people start adding more and more code into the hypervisor.
6) Introduced HyperGuard -- a project done in cooperation with Phoenix Technologies. HyperGuard is going to be a SMM-based integrity scanner for Xen-like hypervisors. With HyperGuard we take a different approach then other integrity scanners do -- rather than ensuring the correctness of the code and data of the hypervisor, which might be very tricky, we instead ensure there is no untrusted code in the hypervisor, which is a much simpler task.
Talk #3
1) Provided detailed description of how to implement nested hardware based virtualization on AMD-V and VT-x (a copy of the slides from my RSA speech in April).
2) Showed how to use this nested virtualization to implement Blue Pill Boot, that can be used to virtualize the system right from the boot stage. We mentioned the best defend against this kind of system compromises is a trusted boot mechanism, either SRTM or DRTM, as implemented e.g. by Xen's tboot.
3) Consequently we showed Xen Blue Pill that is able to move a running Xen system into a virtual machine on the fly. This, on the other hand, cannot be prevented by neither the SRTM nor DRTM technology. XBP is a good example that running a legitimate hypervisor doesn't always prevent bluepill-like malware from being installed in the system.
4) Finally, discussed the XBP detection. First, we noted that all the "VMM detectors", proposed over the last years, that try to detect if there is a hypervisor running above, are useless in the case of a bluepilled Xen system. The only one approach that could be used is the direct timing analysis of the #VMEXIT times in order to distinguish between the native Xen case vs. bluepilled Xen case. We noted however, that direct timing analysis will not observe any differences when run from PV domains on AMD processors, and that it will observe little difference when run from HVM domains (7k vs. 5k cycles). The detection is easier on Intel processors, because of the unconditional #VMEXIT that we cannot get rid of.
All the three talks can be found here.
Monday, July 07, 2008
0wning Xen in Vegas!
At this year’s Black Hat conference in Las Vegas in August we will be presenting three talks about the Xen hypervisor (in)security. The three presentations have been designed in such a way that they complement each other and create one bigger entirety, thus they can be referred as “Xen 0wning Trilogy” for brevity.In the first presentation, Subverting the Xen hypervisor, Rafal will discuss how to modify the Xen’s hypervisor memory and consequently how to use this ability to plant hypervisor rootkits inside Xen (everything on the fly, without rebooting Xen). Hypervisor rootkits are very different creatures from virtualization based rootkits (e.g. Bluepill). This will be the first public demonstration of practical VMM 0wning (proof of concept code will be released, of course).
In the second talk, Detecting and Preventing the Xen hypervisor subversions, Rafal and I will discuss various anti-subverting techniques (IOMMU, Xen’s driver- and stub- domains) and whether they really can protect the Xen (or similar) hypervisor from compromises. After demonstrating that those mechanisms can be bypassed, we will switch to discussing hypervisor integrity scanning and will present some prototype solutions to this problem.
Our trilogy wouldn’t be complete without discussing virtualization based malware in the context of bare-metal hypervisor compromises. Thus, in the third speech, Bluepilling the Xen hypervisor, Alex and I will discuss how to insert Bluepill on top of the running Xen hypervisor. We will show how to do that both with and without restart (i.e. on the fly). To make this possible, our Bluepill needs to support full nested virtualization, so that Xen can still function properly. We will also discuss how the “Bluepill detection” methods proposed over the last 2 years, as well as the integrity scanning methods discussed in the previous speech, fit into this new scenario and how far we are from the stealth malware’s Holy Grail ;)
Special thanks to Black Hat organizers for scheduling all the three presentations one after another in a dedicated Virtualization track on the 2nd day of the conference (August 7th).
It’s worth noting that we chose Xen as the target not because we think it’s insecure and worthless. On the contrary, we believe Xen is the most secure bare-metal hypervisor out there (especially with all the goodies in the upcoming Xen 3.3). Still we believe that it needs some improvements when it comes to security. We hope that our presentations will help making Xen (and similar hypervisors) more secure.
Tuesday, July 01, 2008
Rafal Wojtczuk joins Invisible Things Lab
I’m very happy to announce that a well known researcher, Rafal Wojtczuk, will join our team this month.
For anybody who is serious about OS security research it is hard not to know Rafal’s work. I remember reading his Defeating Solar Designer non-executable stack patch article somewhere around 1998, when I was still a Linux newbie, learning shell programming back at that time ;)
Since then Rafal published many other articles, advisories and exploits, mostly Linux-related. To mention just a few – the *BSD procfs vulnerability (2000), the Linux Ptrace vulnerability (2001), the famous Advanced return-into-lib(c) paper (2002), the vulnerability in the SELinux (2003), a tool for automatic integer overflow discovery in Win32 binaries (2005) and many others. He’s also known for his libnids project.
Recently Rafal has been doing a lot of research in the area of virtualization and VMM security. In the recent months he found vulnerabilities that potentially allowed to escape a VM jail in all the major virtualization software from Microsoft, VMWare and, of course, Xen.
I wrote “and, of course, Xen”, as Rafal will be presenting a talk at the upcoming Black Hat about Subverting the Xen hypervisor. His talk will be the first one in the series of 3 presentations about Xen (in)security that Invisible Things Lab prepared for this year’s Black Hat. Stay tuned for more details in the coming days.
Rafal has been with McAfee Avert Labs until this month.
For anybody who is serious about OS security research it is hard not to know Rafal’s work. I remember reading his Defeating Solar Designer non-executable stack patch article somewhere around 1998, when I was still a Linux newbie, learning shell programming back at that time ;)
Since then Rafal published many other articles, advisories and exploits, mostly Linux-related. To mention just a few – the *BSD procfs vulnerability (2000), the Linux Ptrace vulnerability (2001), the famous Advanced return-into-lib(c) paper (2002), the vulnerability in the SELinux (2003), a tool for automatic integer overflow discovery in Win32 binaries (2005) and many others. He’s also known for his libnids project.
Recently Rafal has been doing a lot of research in the area of virtualization and VMM security. In the recent months he found vulnerabilities that potentially allowed to escape a VM jail in all the major virtualization software from Microsoft, VMWare and, of course, Xen.
I wrote “and, of course, Xen”, as Rafal will be presenting a talk at the upcoming Black Hat about Subverting the Xen hypervisor. His talk will be the first one in the series of 3 presentations about Xen (in)security that Invisible Things Lab prepared for this year’s Black Hat. Stay tuned for more details in the coming days.
Rafal has been with McAfee Avert Labs until this month.
Tuesday, May 20, 2008
1984?
I can't believe this is happening for real...
So, how can we enforce Google to never do read() on a
I wonder when we would get another cool service from Google, e.g. "Google Thoughts", where people would be able to store their most private and personal thoughts, so that they could "access and managed them from all over the world", "in a secure fashion", of course. Right, add the Thought Police to this picture and welcome to Orwell's Oceania!
There is a difference between using the Web for blog writing vs. giving away all the private aspects of your life for free to some corporation in an unencrypted form. I wonder whether all the people who understands the notion of the local hard disk will be vaporized some time...
So, how can we enforce Google to never do read() on a
/var/users/john_smith/heath_records.db? How about a read() implemented from within a kernel-level via RAW-disk access that would never be logged?I wonder when we would get another cool service from Google, e.g. "Google Thoughts", where people would be able to store their most private and personal thoughts, so that they could "access and managed them from all over the world", "in a secure fashion", of course. Right, add the Thought Police to this picture and welcome to Orwell's Oceania!
There is a difference between using the Web for blog writing vs. giving away all the private aspects of your life for free to some corporation in an unencrypted form. I wonder whether all the people who understands the notion of the local hard disk will be vaporized some time...
Wednesday, April 30, 2008
Vegas Training 2008
Last year we debuted with our Understanding Stealth Malware training at the Black Hat Vegas. We had about 70 participants and I think it was a reasonable success, especially that the training was announced very late. Since then we have done a couple of on-site classes and also have been continually updating the training.
During our 2nd public edition, at Black Hat Europe 2008 in March this year, we significantly extended the part about virtualization, e.g. by adding discussion of nested virtualization on AMD-v and showing and analyzing the actual code for implementing this. Also we have used the New Blue Pill code with VT-x support (previously it worked only on AMD-V), making it possible to use both AMD and Intel machines for the class. This allowed us to offer this training in a "Bring Your Own Laptop" fashion, that we know is much preferred by attendees, who simply feel better when using their own, known, work environment.
At the upcoming Black Hat Vegas 2008 we are also going to offer this class. That would be our 3rd public edition. Again, we hope to improve it even more beyond what we have presented at BH Europe 2008. Similarly as last time, we will not provide the computers, but rather expect the attendees to bring their own systems. At the end of this article are the requirements that should be met by your machine, if you would like to use it during the training and be able to do all the exercises. Of course, you should back up all your important data before coming to the class, as the computer might become corrupt after doing some of the exercises (although this has never happened so far).
There will be only one class offered on August 4/5 (the weekday class). You can view the detailed training agenda that we used for the BH Europe class in March here. Please note that the exact shape of the Vegas class is subject to be a bit different, as we are planning to add new material again.
This might be the very last chance for you to attended this specific training, as it's quite possible that next year we will be offering some other class, focused on Virtualization security entirely. Don't worry, however, if you don't get a seat in the Vegas class, there is still a chance to have that class presented on-site in your town.
You can register for the Vegas training here.
See you in Vegas!
Hardware Requirements
Most modern AMD mobile processors, like e.g. AMD Turion and Athlon, used in modern laptops support AMD-v technology. Unfortunately there is no single place on AMD website that would provide the complete description of all CPUs that support AMD-v technology or provide an answer whether a given model does support it. When in doubt use google and always verify with the CHKSVMX program described below.
Intel Processors
Most modern Intel processors used in notebooks support Intel VT-x virtualization technology, this include Core 2 Solo, Core 2 Duo (except T5500, T5550 and T5750 models) and Core 2 Extreme. You can check your own model starting at this website, then chose your processor family and chose "Specifications" tab. Make sure the processor supports "Intel® 64 architecture" and "Intel® Virtualization Technology".
Using Mac for the training
You can very easily use MacBook or MacBook Pro for this training. You can easily install Windows on a second partition using the Boot Camp program that ships with all the newer Macs. You simply start Boot Camp application when running Mac OS X and then it automatically shrinks your current Mac partition, creates a new one for Windows, and asks to insert the installation media and reboots the system and you then perform normal Windows setup (after installation is complete your Vista should find all the necessary drivers via Windows Update). You might also want to use the free AutoHotKey program for the right-click emulation on your newly installed Vista. Please don't worry that Boot Camp tells that you should install a 32-bit Vista - you can ignore this and insert a 64-bit Vista installation disk.
Testing your machine with CHKSVMX
We have prepared a special little program, CHKSVMX, to test whether a given machine indeed supports hardware virtualization technology. The CHKSVMX program can be downloaded from here
The program doesn't introduce any persistent changes to the OS and doesn't require any installation procedure. It checks for virtualization support (on both AMD and Intel processors) not only by reading the CPUID information but also by trying to actually enable virtualization mode and then disable it again. Although most of the laptops available these days support hardware virtualization, in many cases this feature is disabled or locked down in the BIOS. If the virtualization is reported as "locked", please try to enable it in the BIOS. Please note that in most cases you will have to fully power down your system for the BIOS changes to take effect (reboot is not enough)!
Additionally CHKSVMX checks whether a 64-bit edition of Windows is running, as such OS is required for the training.
DISCLAIMER: The test program is digitally signed with the Invisible Things Lab's certificate and we assure that the program does not perform any malicious actions. ITL is, however, not responsible for any accidental damage or system instability issues the test program might cause.
During our 2nd public edition, at Black Hat Europe 2008 in March this year, we significantly extended the part about virtualization, e.g. by adding discussion of nested virtualization on AMD-v and showing and analyzing the actual code for implementing this. Also we have used the New Blue Pill code with VT-x support (previously it worked only on AMD-V), making it possible to use both AMD and Intel machines for the class. This allowed us to offer this training in a "Bring Your Own Laptop" fashion, that we know is much preferred by attendees, who simply feel better when using their own, known, work environment.
At the upcoming Black Hat Vegas 2008 we are also going to offer this class. That would be our 3rd public edition. Again, we hope to improve it even more beyond what we have presented at BH Europe 2008. Similarly as last time, we will not provide the computers, but rather expect the attendees to bring their own systems. At the end of this article are the requirements that should be met by your machine, if you would like to use it during the training and be able to do all the exercises. Of course, you should back up all your important data before coming to the class, as the computer might become corrupt after doing some of the exercises (although this has never happened so far).
There will be only one class offered on August 4/5 (the weekday class). You can view the detailed training agenda that we used for the BH Europe class in March here. Please note that the exact shape of the Vegas class is subject to be a bit different, as we are planning to add new material again.
This might be the very last chance for you to attended this specific training, as it's quite possible that next year we will be offering some other class, focused on Virtualization security entirely. Don't worry, however, if you don't get a seat in the Vegas class, there is still a chance to have that class presented on-site in your town.
You can register for the Vegas training here.
See you in Vegas!
Hardware Requirements
- 64-bit (x64) AMD or Intel processor with hardware virtualization support (AMD-v or VT-x)
- DVD-ROM
- 2GB RAM (for convenient work with VMWare)
- 64-bit Vista OS (primary OS, non virtualized)
- Windows Driver Kit (WDK) 6000 or newer (available via MSDN subscription).
- VMWare Workstation 6.x or VMWare Player 2.x (the latter is free)
- Optionally: IDA Pro 5.x disassembler (for exercises that involve finding bugs in drivers)
Most modern AMD mobile processors, like e.g. AMD Turion and Athlon, used in modern laptops support AMD-v technology. Unfortunately there is no single place on AMD website that would provide the complete description of all CPUs that support AMD-v technology or provide an answer whether a given model does support it. When in doubt use google and always verify with the CHKSVMX program described below.
Intel Processors
Most modern Intel processors used in notebooks support Intel VT-x virtualization technology, this include Core 2 Solo, Core 2 Duo (except T5500, T5550 and T5750 models) and Core 2 Extreme. You can check your own model starting at this website, then chose your processor family and chose "Specifications" tab. Make sure the processor supports "Intel® 64 architecture" and "Intel® Virtualization Technology".
Using Mac for the training
You can very easily use MacBook or MacBook Pro for this training. You can easily install Windows on a second partition using the Boot Camp program that ships with all the newer Macs. You simply start Boot Camp application when running Mac OS X and then it automatically shrinks your current Mac partition, creates a new one for Windows, and asks to insert the installation media and reboots the system and you then perform normal Windows setup (after installation is complete your Vista should find all the necessary drivers via Windows Update). You might also want to use the free AutoHotKey program for the right-click emulation on your newly installed Vista. Please don't worry that Boot Camp tells that you should install a 32-bit Vista - you can ignore this and insert a 64-bit Vista installation disk.
Testing your machine with CHKSVMX
We have prepared a special little program, CHKSVMX, to test whether a given machine indeed supports hardware virtualization technology. The CHKSVMX program can be downloaded from here
The program doesn't introduce any persistent changes to the OS and doesn't require any installation procedure. It checks for virtualization support (on both AMD and Intel processors) not only by reading the CPUID information but also by trying to actually enable virtualization mode and then disable it again. Although most of the laptops available these days support hardware virtualization, in many cases this feature is disabled or locked down in the BIOS. If the virtualization is reported as "locked", please try to enable it in the BIOS. Please note that in most cases you will have to fully power down your system for the BIOS changes to take effect (reboot is not enough)!
Additionally CHKSVMX checks whether a 64-bit edition of Windows is running, as such OS is required for the training.
DISCLAIMER: The test program is digitally signed with the Invisible Things Lab's certificate and we assure that the program does not perform any malicious actions. ITL is, however, not responsible for any accidental damage or system instability issues the test program might cause.
Monday, April 14, 2008
Research Obfuscated
Update 07-Sept-2008: Four months later after writing his open letter to me (see below), Christofer Hoff experienced on his own difficult it is for one to control the press, making sure it correctly reports what you say. In this blog entry he describes how he was terribly misquoted by a report after his Black Hat presentation and he also explicitly admits that "[I] was essentially correct in [my] assertion during our last debate that you cannot control the press, despite best efforts." and that "[he] humbly submit[s] to [me] on that point." :)
This article has been brought to my attention recently. It’s an “Open Letter to Joanna Rutkowska”, by Christofer Hoff over at the “Rational Survivability” blog. I decided to spend time reading and answering this piece as 1) technorati.com reported the blog’s authority as above 100 which suggests it has a reasonable number of readers, and also 2) because I believe this is a good example of the social engineering techniques used by my opponents and I couldn’t refrain myself from not commenting about this. Besides I felt a bit flattered that some individual decided to write an “Open Letter” to me, sort of like if I was a prime minister or some other important person ;)
Let me now analyze the letter, point by point:
So, what was the main massage of my presentation? Interestingly Mr. Hoff forgot to mention that… Let me then remind it here (a curious reader might want to have a look at the the slide #96 in my presentation):
This article has been brought to my attention recently. It’s an “Open Letter to Joanna Rutkowska”, by Christofer Hoff over at the “Rational Survivability” blog. I decided to spend time reading and answering this piece as 1) technorati.com reported the blog’s authority as above 100 which suggests it has a reasonable number of readers, and also 2) because I believe this is a good example of the social engineering techniques used by my opponents and I couldn’t refrain myself from not commenting about this. Besides I felt a bit flattered that some individual decided to write an “Open Letter” to me, sort of like if I was a prime minister or some other important person ;)
Let me now analyze the letter, point by point:
- Fire rules! The first thing that Hoff accuses me of in his letter is myself being an irresponsible individual, not caring about safety of my audience (not a joke!):
“As the room filled to over capacity before your talk began, you were upset and couldn't seem to understand why the conference organizers would not let people spill over from seats and sit on the floor and in the aisles to hear you speak. The fact that fire and safety codes prohibit packing a room beyond capacity was something you attributed to people being "...crazy in America." Go figure.”
Dear Christofer, if you only read my recent blog post about this very specific incident, read thoroughly shall I say, you would notice this paragraph undoubtedly:“Interestingly it was perfectly ok for the additional people to stay in the room, provided they arranged for additional chairs for themselves. In other words it was fine for people to sit and block the main aisle, provided they sit on chairs, but they couldn’t stay and sit on the same aisle without having a chair (maybe a "certificated" chair also), as that would be against the fire regulations!”
Conclusion: I was not so much picking upon the fire regulations that forced people to leave the room, but rather on the idiotic rule, that allowed those same people to stay in this very same room, provided they also had additional chairs with them. - Type I vs. Type II hypervisors confusion. Hoff then switches to the actual content of the presentation and writes this:
“When I spoke to you at the end of your presentation and made sure that I understood correctly that you were referring specifically to type-2 hosted virtualization on specific Intel and AMD chipsets, you conceded that this was the case.”
This simply is an incorrect statement! On the contrary, when describing the security implications of nested virtualization (which was the actual new thing I was presenting at the RSA), I explicitly gave an example of how this could be used to compromise type I hypervisors. Kindly refer to slides 85-90 of my presentation that can be downloaded here.
I said that the code we posted on bluepillproject.org indeed targets type II hypervisors and the only reason for that being that it has been built on top of our New Blue Pill code that was designed as a Windows kernel driver. - Shit not giving. Mr. Hoff goes even further:
“When I attempted to suggest that while really interesting and intriguing, your presentation was not only confusing to many people but also excluded somewhere north of 80% of how most adopters have deployed virtualization (type-1 "bare-metal" vs. type-2 hosted) as well as excluding the market-leading virtualization platform, your response (and I quote directly) was: I don't give a shit, I'm a researcher.”
Now that was a hard blow! I understand that the usage of such a slang expression by an Eastern European female during an informal conversation with a native speaker must have made an impression on him! However, I couldn’t give such an answer to this very question, simply because of the reasons given in point #2 (see above).
If I remember correctly, I indeed used this very American expression to answer somebody’s concern (undoubtedly our Christofer Hoff’s) that most of the type I hypervisors out there are based on monolithic hypervisor architecture, and not on the micro-hypervisor architecture (and that I should not try to convince people to switch to micro-hypervisor architecture). In that context it makes it more logical for me to use the “I’m a researcher” as an excuse for not caring so much that most people use monolithic based hypervisors. Obviously, the usage of micro-hypervisors would allow to better secure the whole VMM infrastructure. And I also said, that I don’t care what people are using today, because I try to help to build a product that would be secure in the future (Phoenix’s HyperCore). - No obfuscation postulate. Hoff then comes up with some postulates that:
“[I], as a researcher who is also actively courting publicity for commercial gain and speaking at conferences like RSA which are less technical and more "executive" in nature, you have a responsibility to clarify and not obfuscate (intentionally or otherwise) the facts surrounding your research.”
This postulate is cleverly constructed because it also contains an embedded accusation of me being a commercially motivated researcher. Well, I never tried to hide that fact, and the reason for this is very simple: I consider security research as my job, and one of the primary goals of any job is to… bring commercial gain to the individual doing the job.
Second, I really don’t understand what Hoff means by asking me to not obfuscate my research?! Maybe he was just disappointed that the presentation was too technical for an average CISSP to understand it? But, well, this presentation was classified as “Advanced Technical”, which was displayed in the conference program. I still did my best so that, say 70% of the material, was understandable to an average IT people, but, come on, there always must be some deep technical meat in any non-keynote-presentation, at least this is my idea for how a conference should look like. - Commercially motivated. Hoff accuses me of presenting commercial product, i.e. the Phoenix’s HyperSpace, during my speech:
“No less than five times during your presentation, you highlighted marketing material in the form of graphics from Phoenix, positioned their upcoming products and announced/credited both Phoenix and AMD as funding your research.”
Well, let me tell you this – this was one of the main reasons why I decided to speak at the RSA – just to announce this very product that I try to help to secure. Why would that be wrong?
BTW, I have no idea how Mr. Hoff concluded that AMD was founding my research. I never said that, nor did I have it in my slides. Needles to say, AMD has not been founding my research. NOTE: interestingly I consider this particular mistake by Hoff to be accidental – at least I don’t see how this could be connected to any PR campaign, in contrast to all the other incorrectness he made use of. - Independence. Hoff, for some reason, apparently known only to him, tries to argue that I’m not an “independent researcher”:
“I think it's only fair to point out that given your performance, you're not only an "independent researcher" but more so an "independent contractor." Using the "I'm a researcher" excuse doesn't cut it.”
“I know it's subtle and lots of folks are funded by third parties, but they also do a much better job of drawing the line than you do.”
Well, I found this one to be particularly amusing, as, for at least several years now, I have not claimed I have been an independent researcher. - Final hit. You might have been wondering by now – why this gentleman, nah, I think “the guy” would fit better here, so why the guy decided to spent so much time to write all those points, all those quasi-arguments and why he made so many “mistakes”? Well he seems to give an answer right in this paragraph:
“I care very much that your research as presented to the press and at conferences like RSA isn't only built to be understood by highly skilled technicians or researchers because the continued thrashing that they generate without recourse is doing more harm than good, quite frankly.”
Aha, now all is clear. May I ask then, which virtualization vendor you write PR for? ;)
So, what was the main massage of my presentation? Interestingly Mr. Hoff forgot to mention that… Let me then remind it here (a curious reader might want to have a look at the the slide #96 in my presentation):
- Virtualization technology could be used to improve security on desktop systems
- However there are non-trivial challenges in making this all working well...
- ... and not to introduce security problems instead...
“Keep hypervisors simple, do not put drivers there, as otherwise we would get to the same point where we are with current OSes these days, i.e. no kernel security at all!”Now I wonder, maybe Christofer Hoff doesn’t do PR for any VMM vendor, maybe he just didn’t listen carefully to my presentation. Maybe he’s just one of those many guys who always know in advance what they want to hear and selectively pick up only those facts that match their state of mind? Otherwise, why would he not realize that my presentation was actually a pro-virtualization one and needed no (false) counter-arguments?
Saturday, April 12, 2008
The Most Stupid Security News Ever
Seems like the BBC reporters have a shortage of subjects to write about these days… Maybe the next winter we will also be able to read about how many snowflakes fell during Christmas all over the world or something like that (which BTW, would still be way more interesting that the news quoted above).
I remember that some time ago, a group of researchers used automatic generators to create a few tens of thousands of variants of some malware, just to do some testing of A/V engines. And I remember how all the A/V people were complaining how irresponsible that was bla bla bla, as now they would have to work after hours to fight all this new malware. What a BS!
For any given class of a bug (think: exploits), or a file infection method (think: viruses), or a system compromise technique (think: rootkits, stealth malware), one can come up with pretty much infinite number of examples that would be exploiting the specific bug, the specific file infection method, or the specific system compromise technique. One virus would display you a “Hello, you’re being 0wned, sir.” Message, while the other one would just flash your keyboard leds. Sure, two different beings, but if exploiting the same mechanisms, also the protection against them is the same.
But, I know, it looks so cool in the news to read: “The number of viruses, worms and trojans in circulation has topped the one million mark”. It’s most definitely a good way to scare all the housewives and make them to rush to the computer shop at the coroner to buy the brand new A/V product that already can detect 99.9% out of all those scary things out there.
I remember that some time ago, a group of researchers used automatic generators to create a few tens of thousands of variants of some malware, just to do some testing of A/V engines. And I remember how all the A/V people were complaining how irresponsible that was bla bla bla, as now they would have to work after hours to fight all this new malware. What a BS!
For any given class of a bug (think: exploits), or a file infection method (think: viruses), or a system compromise technique (think: rootkits, stealth malware), one can come up with pretty much infinite number of examples that would be exploiting the specific bug, the specific file infection method, or the specific system compromise technique. One virus would display you a “Hello, you’re being 0wned, sir.” Message, while the other one would just flash your keyboard leds. Sure, two different beings, but if exploiting the same mechanisms, also the protection against them is the same.
But, I know, it looks so cool in the news to read: “The number of viruses, worms and trojans in circulation has topped the one million mark”. It’s most definitely a good way to scare all the housewives and make them to rush to the computer shop at the coroner to buy the brand new A/V product that already can detect 99.9% out of all those scary things out there.
Wednesday, April 09, 2008
The RSA Absurd
Today I was giving a speech at the RSA Conference in San Francisco. The RSA is a really big conference and also seems to me like a very well organized one – e.g. they have all those computers at the registration hall where you put your name and then it immediately says to which check-in counter you should proceed and then when you get there they already have a badge waiting for you. Pretty cool stuff.
So my speech turned out to be scheduled in a very small room, say with seats for 100-200 people only (I haven't counted exactly). But then it turned out that there are more people interested in seeing the speech, so, as it usually happens on conferences, people started seating on the floor and also standing at the back of the room. I would say there was about 30% overflow, but still they could fit ok in the room. And then came this guy from the conference and said that all people who don’t have a seat should leave the room! It turned out that this is a fire regulation.
Interestingly it was perfectly ok for the additional people to stay in the room, provided they arranged for additional chairs for themselves. In other words it was fine for people to sit and block the main aisle, provided they sit on chairs, but they couldn’t stay and sit on the same aisle without having a chair (maybe a "certificated" chair also), as that would be against the fire regulations!
Yes, I know there are more examples of stupid pseudo-security rules (think airports), but, come on, this is on of the most well known security conference...
That situation annoyed me so much (because, of course, it turned out to be impossible to arrange for the additional chairs, so all those people had to leave) that I decided to submit this story to my blog using the totally unsecured public WiFi in my hotel. It was really unwise for me to do that, as Google’s Blogger uses HTTPS only for authentication (i.e. the login screen) but then it switches back to the good old plain text HTTP, making it possible for some evil guy sitting in the lobby to hijack my session. Is it that I miss something here or Google simple forgot that it is 2008 and not the 90’s anymore? Anyway, I'm just taking this risk bravely, hoping that the potential attacker, seeing my determination here, would refrain themselves from compromising this blog.
I know, I know, instead of complaining about Google, I should just move my blog to some other place. One day that’s gonna happen for sure :)
So my speech turned out to be scheduled in a very small room, say with seats for 100-200 people only (I haven't counted exactly). But then it turned out that there are more people interested in seeing the speech, so, as it usually happens on conferences, people started seating on the floor and also standing at the back of the room. I would say there was about 30% overflow, but still they could fit ok in the room. And then came this guy from the conference and said that all people who don’t have a seat should leave the room! It turned out that this is a fire regulation.
Interestingly it was perfectly ok for the additional people to stay in the room, provided they arranged for additional chairs for themselves. In other words it was fine for people to sit and block the main aisle, provided they sit on chairs, but they couldn’t stay and sit on the same aisle without having a chair (maybe a "certificated" chair also), as that would be against the fire regulations!
Yes, I know there are more examples of stupid pseudo-security rules (think airports), but, come on, this is on of the most well known security conference...
That situation annoyed me so much (because, of course, it turned out to be impossible to arrange for the additional chairs, so all those people had to leave) that I decided to submit this story to my blog using the totally unsecured public WiFi in my hotel. It was really unwise for me to do that, as Google’s Blogger uses HTTPS only for authentication (i.e. the login screen) but then it switches back to the good old plain text HTTP, making it possible for some evil guy sitting in the lobby to hijack my session. Is it that I miss something here or Google simple forgot that it is 2008 and not the 90’s anymore? Anyway, I'm just taking this risk bravely, hoping that the potential attacker, seeing my determination here, would refrain themselves from compromising this blog.
I know, I know, instead of complaining about Google, I should just move my blog to some other place. One day that’s gonna happen for sure :)
Monday, March 31, 2008
Kick Ass Hypervisor Nesting!
Remember how at the Black Hat Vegas 2007 I said that we still didn't support virtualization of full VMMs, like e.g. Virtual PC 2007 with hardware virtualization enabled, and that currently we could only run very simple hypervisors inside our New Blue Pill (like e.g. other NBPs inside NBP)? Remember how I said that we were working on this and should have a solution in about 2 months from then?
So, just about 2 weeks ago we did it! We can now virtualize complex hypervisors, like e.g. Virtual PC 2007 or Virtual Box with SVM turned on (BTW, we can also run VMWare Workstation, but that doesn't count, as on AMD processors it doesn't make use of SVM instructions). We also have a prototype code that allows to run nested hypervisors on VT-x but that code requires a bit of more polishing (oh, didn’t you know that our NBP also supports VT-x these days?).
I couldn't resist not to use my favorite Matrix analogy to describe what we do here: imagine Neo, who bravely followed The White Rabbit and finally decided to swallow The Red Pill, eventually awakes on The Nebuchadnezzar ship just to find out later that this whole "real world" is... just another Matrix...
I don't have a nice Matrix picture for that, so instead I will just show you a picture of a Virtual PC 2007 running inside an already bluepilled Vista and running Windows XP as its own guest. You can see that we use our "bpknock" testing program just to show we can intercept events in both the guest (i.e. the Vista that hosts the VPC hypervisor) as well as in the nested guest (the XP running inside the Virtual PC). This bpknock program simply executes CPUID instruction with some magic value in the RAX register and NBP intercepts that and answers with a magic RAX. BTW, there was no special reason to chose CPUID instruction for that, normally we don't need to intercept CPUID on AMD at all, so we could have chosen pretty much anything else, e.g. magic output to some magic I/O port.
It's worth mentioning that the only other working example of nested hardware virtualization I'm aware of is the IBM z/VM hypervisor for the IBM z series mainframe. If anybody knows any other example, please send me a link.
The research on nested virtualization has been supported by Phoenix Technologies, as the nested virtualization has also some positive applications. Phoenix is working on a cool product called HyperSpace. It consists of a hypervisor (called the "HyperCore") that allows running a few unmodified OSes inside hardware virtual machines so users can switch between them just like if they were virtual spaces on Mac or Linux. At the beginning there will be two virtual machines available: one running standard Vista and the other one based on Linux, that would contain some useful functionality like e.g. a Web browser, an email client and a multimedia suite, and also there would be something called the "ManageSpace" to manage this all.
So, how this is going to be different from e.g. XEN? The difference is that XEN is focused on server applications, while HyperSpace is intended for notebooks, which means it puts lots of efforts to offer comparable graphics (and other devices) performance as we have on normal non-virtualized laptops. This all will be possible because of the recent virtualization technology advances like e.g. VT-d/IOMMU.
At the RSA conference in San Francisco next week, I will be giving a speech that will discuss some technical problems we had to solve in order to get hardware nested hypervisoring working on AMD and also how the situation looks on Intel. I will also discuss how this changes the security battlefield and why virtualization vendors should care.
Back to Blue Pill -- the brand new source code with full virtualization support on AMD is now available on bluepillproject.org (you will need WDK6000 or newer to build it). Note that the (experimental) code for nested virtualization on Intel VT-x has been removed in this public version, leaving only the basic functionality if we run NBP on an Intel processor.
Also, please note that the code for AMD-v, even though it proved to be very stable, is still just a proof of concept. This means for example, that we don’t do any error-checks in the SVM instruction handlers, so it’s trivial for the nested hypervisor to simply crash the whole system if executing one of the SVM instructions with incorrect arguments or in an incorrect situation (e.g. CPL > 0). But that is hardly a problem for Blue Pill, as the guest isolation has never been a goal here. Of course, this could be simply addressed by adding a few more lines of code to each handler that would check for error conditions and inject #UD or #GP back to the nested hypervisor if it executed something incorrectly. Of course, we’re too lazy to code that ;)
So, what’s next? Well, we hope to show something even cooler at this year’s Black Hat Vegas, but I won’t say anything more now.
So, just about 2 weeks ago we did it! We can now virtualize complex hypervisors, like e.g. Virtual PC 2007 or Virtual Box with SVM turned on (BTW, we can also run VMWare Workstation, but that doesn't count, as on AMD processors it doesn't make use of SVM instructions). We also have a prototype code that allows to run nested hypervisors on VT-x but that code requires a bit of more polishing (oh, didn’t you know that our NBP also supports VT-x these days?).
I couldn't resist not to use my favorite Matrix analogy to describe what we do here: imagine Neo, who bravely followed The White Rabbit and finally decided to swallow The Red Pill, eventually awakes on The Nebuchadnezzar ship just to find out later that this whole "real world" is... just another Matrix...
I don't have a nice Matrix picture for that, so instead I will just show you a picture of a Virtual PC 2007 running inside an already bluepilled Vista and running Windows XP as its own guest. You can see that we use our "bpknock" testing program just to show we can intercept events in both the guest (i.e. the Vista that hosts the VPC hypervisor) as well as in the nested guest (the XP running inside the Virtual PC). This bpknock program simply executes CPUID instruction with some magic value in the RAX register and NBP intercepts that and answers with a magic RAX. BTW, there was no special reason to chose CPUID instruction for that, normally we don't need to intercept CPUID on AMD at all, so we could have chosen pretty much anything else, e.g. magic output to some magic I/O port.
It's worth mentioning that the only other working example of nested hardware virtualization I'm aware of is the IBM z/VM hypervisor for the IBM z series mainframe. If anybody knows any other example, please send me a link.
The research on nested virtualization has been supported by Phoenix Technologies, as the nested virtualization has also some positive applications. Phoenix is working on a cool product called HyperSpace. It consists of a hypervisor (called the "HyperCore") that allows running a few unmodified OSes inside hardware virtual machines so users can switch between them just like if they were virtual spaces on Mac or Linux. At the beginning there will be two virtual machines available: one running standard Vista and the other one based on Linux, that would contain some useful functionality like e.g. a Web browser, an email client and a multimedia suite, and also there would be something called the "ManageSpace" to manage this all.
So, how this is going to be different from e.g. XEN? The difference is that XEN is focused on server applications, while HyperSpace is intended for notebooks, which means it puts lots of efforts to offer comparable graphics (and other devices) performance as we have on normal non-virtualized laptops. This all will be possible because of the recent virtualization technology advances like e.g. VT-d/IOMMU.
At the RSA conference in San Francisco next week, I will be giving a speech that will discuss some technical problems we had to solve in order to get hardware nested hypervisoring working on AMD and also how the situation looks on Intel. I will also discuss how this changes the security battlefield and why virtualization vendors should care.
Back to Blue Pill -- the brand new source code with full virtualization support on AMD is now available on bluepillproject.org (you will need WDK6000 or newer to build it). Note that the (experimental) code for nested virtualization on Intel VT-x has been removed in this public version, leaving only the basic functionality if we run NBP on an Intel processor.
Also, please note that the code for AMD-v, even though it proved to be very stable, is still just a proof of concept. This means for example, that we don’t do any error-checks in the SVM instruction handlers, so it’s trivial for the nested hypervisor to simply crash the whole system if executing one of the SVM instructions with incorrect arguments or in an incorrect situation (e.g. CPL > 0). But that is hardly a problem for Blue Pill, as the guest isolation has never been a goal here. Of course, this could be simply addressed by adding a few more lines of code to each handler that would check for error conditions and inject #UD or #GP back to the nested hypervisor if it executed something incorrectly. Of course, we’re too lazy to code that ;)
So, what’s next? Well, we hope to show something even cooler at this year’s Black Hat Vegas, but I won’t say anything more now.
Subscribe to:
Posts (Atom)
