As the technology is getting more and more complex, security research, especially offensive security research on a system level, becoming more and more difficult to be done by one person. NX/XD, ASLR, various StackGuard-like things, VT-d, TXT, etc... - all those technologies leave less and less space for the interesting system-level attacks. On the other hand, the widespread "deployment" of Web 2.0 creates a whole new area to explore, but that is a whole different world (plus there are still all those "human factor" attacks that exploit user stupidity, but again, this is a different area).
Our Xen 0wning Trilogy is a good example of how a team of researchers can still come up with interesting new system-level attacks against the very recent and securely design system. Take XenBluePill as an example.
It has first been months of research and coding done by Alex and myself to support nested hardware virtualization on AMD. Then there was months of Rafal's research about how to load code into the running Xen on the fly ("Xen Loadable Modules"). That required ability to access Xen's memory in the first place and Rafal's way for doing that was to use the DMA attack. But then it turned out that the Xen 3.3 uses VT-d protection to protect against this very kind of attacks. So then I came up with the "Q35 attack" that exploited a problem with recent Intel BIOSes on recent motherboards (details are coming this week). But I based my attack on a similar SMM attack that Rafal came up with a few months earlier on a different chipset, when he was looking into ways to compromise SMM handler, as we started thinking about HyperGuard project back then and Rafal was curious reliable the SMM protection is. In the meantime, Alex "converted" our working New Blue Pill that had full support for nested virtualization but was essentially a Windows driver, into a piece of code that was completely OS-independent (own memory management, etc.). Then I finally took Rafal's XLM framework, added a few minor things that were needed to load our "Windows-independent Windows driver" into Xen using XLM, fixed some minor stuff and... it finally worked! But that was possible only because of the joint work by all the three people together.
So, it is simply unfair to attribute all the glory and fame for our research to "Rutkowska" or "Rutkowska and team", as many news portals did. Please don't forget to credit all the co-authors! If you really would like to use a generic term, then "Invisible Things Lab team" would probably serve better.
Speaking of our team, I also have an announcement that starting this month our team has officially been extended by yet another person: Rong Fan from Beijing, China.
Rong is a software engineer, focusing on Intel's hardware virtualization technology (VT). A few months ago he wrote to me with some advanced questions regarding the implementation of our New BluePill that we published after the last year's Black Hat. Turned out that Rong, as part of his after-hour activity, is porting Bluepill to VT-x. After he succeeded, we decided to share our nested virtualization code for AMD with him so that he could investigate how to do it on VT-x. And about 2 months ago Rong succeeded with implementing full nested virtualization support for our NBP on Intel VT-x! During that time Rong has had an opportunity to find out that working with ITL is quite fun, so he decided to quit his job at Lenovo and joined ITL full time. Right now Rong is busy adding nested VT-x support to a normal Xen hypervisor.
So, Invisible Things Lab is all about the team work. The whole idea behind ITL is to gather together a bunch of smart people, so that we could all work on the most exciting problems together. Problems that might be too complex or time-consuming for just one person to solve. But it takes more then just money to get people to be creative and devote themselves to work. Getting recognition is one of the additional factors often needed. That's why ITL is not interested in "hiding" its employees, but rather in promoting their work and fairly crediting them.
Sunday, August 31, 2008
Tuesday, August 26, 2008
Intel patches the Q35 bug
Yesterday Intel has published an official advisory that addresses the Q35 bug and attack, that we used during Black Hat as one of the ways to subvert Xen 3.3 on a VT-d enabled system (the alternative way was to use the Xen-specific FLASK exploit, that worked even from an unprivileged domain).
One small clarification though: in the advisory they stated that: "Software running administrative (ring 0) privilege can under certain circumstances change code running in System Management Mode." But in fact an attacker might also use this bug to directly modify the hypervisor memory, without jumping into the SMM first, just as we did it with our exploit. Also, in case of e.g. Linux systems, the Ring0 access is not strictly required to perform the attack, as it's just enough for the attacker to get access to the PCI config space of the device 0:0:0, which e.g. on Linux can be granted to usermode applications via the iopl() system call.
You can download a new firmware for your motherboard from here.
Intel did a good job on handling this bug - not only they recognized the importance of the attack, but also released the patch promptly. Quite positively surprising as for such a big company.
So, now we're free to publish all the missing slides about how we exploit this vulnerability that we had to remove from our Black Hat presentation, as well as the exploit code. However, as I'm going to give 2 presentations at the upcoming ISF conference in Sweden early next week, I thought it would be logical to wait with disclosing this material and present it at this conference, during my technical speech (I will also deliver the keynote for this conference). Of course, as soon as I will get back home (Thursday next week), we will publish the full slides, exploit codes and all the demos, as promised earlier.
Speaking of speaking: also next month, Rafal will fly to Oregon, to Intel campus, for the Intel Virtualization Security Summit, where he will deliver a "compressed" version of our Xen 0wning Trilogy to the technical crowd of Intel employees. Rafal will provide some more details about the HyperGuard project that we do in cooperation with Phoenix Technologies. Also, in October, Alex will visit Kuala Lumpur and present an updated Bluepilling the Xen Hypervisor talk at the Hack In The Box conference.
One small clarification though: in the advisory they stated that: "Software running administrative (ring 0) privilege can under certain circumstances change code running in System Management Mode." But in fact an attacker might also use this bug to directly modify the hypervisor memory, without jumping into the SMM first, just as we did it with our exploit. Also, in case of e.g. Linux systems, the Ring0 access is not strictly required to perform the attack, as it's just enough for the attacker to get access to the PCI config space of the device 0:0:0, which e.g. on Linux can be granted to usermode applications via the iopl() system call.
You can download a new firmware for your motherboard from here.
Intel did a good job on handling this bug - not only they recognized the importance of the attack, but also released the patch promptly. Quite positively surprising as for such a big company.
So, now we're free to publish all the missing slides about how we exploit this vulnerability that we had to remove from our Black Hat presentation, as well as the exploit code. However, as I'm going to give 2 presentations at the upcoming ISF conference in Sweden early next week, I thought it would be logical to wait with disclosing this material and present it at this conference, during my technical speech (I will also deliver the keynote for this conference). Of course, as soon as I will get back home (Thursday next week), we will publish the full slides, exploit codes and all the demos, as promised earlier.
Speaking of speaking: also next month, Rafal will fly to Oregon, to Intel campus, for the Intel Virtualization Security Summit, where he will deliver a "compressed" version of our Xen 0wning Trilogy to the technical crowd of Intel employees. Rafal will provide some more details about the HyperGuard project that we do in cooperation with Phoenix Technologies. Also, in October, Alex will visit Kuala Lumpur and present an updated Bluepilling the Xen Hypervisor talk at the Hack In The Box conference.
Wednesday, August 20, 2008
Attacking Xen: DomU vs. Dom0 consideration
As it usually happens, there is some confusion regarding the attacks presented in our Xen 0wning Trilogy. Some people think they are possible only from Dom0 (the Xen's privileged, administrative domain) while some other people have the impression that all the attacks are possible from any unprivileged domain (DomU in Xen’s terminology). The truth is in the middle though.
Most of the attacks we presented do indeed require that the attacker first obtained access to Dom0 and only from there can launch further attacks. For example the DMA attacks that allow to overwrite hypervisor memory do indeed assume Dom0 access. The same applies to the Q35 exploit - this one is similar to the above mentioned DMA attacks in that it also requires access to certain hardware (that is possible from Dom0), but has an advantage that can bypass the hypervisor VT-d protection in the recent Xen 3.3.
There are several reasons why those attacks are still very important though:
1) First note, that the requirement for the attacker to have access to Dom0 in order to install e.g. hypervisor rootkits, is similar to the requirement that in order to install a Windows or Linux rootkit, one first needs to obtain administrator's privileges. Yet we know that Windows or Linux rootkits is a serious security problem.
2) Concerning the Xen-based systems specifically: over the last year several bugs have been discovered and published, that allowed an attacker to gain control of the Domain0 from an unprivileged domain (i.e. escape from the virtual machine). Rafal has discovered one such bug in December 2007.
3) Recent versions of Xen make deliberate effort to protect the hypervisor even from the Dom0. On systems that have IOMMU support (e.g. Intel's VT-d), the hypervisor memory is protected from tampering using both the processor's ring3/ring0 separation mechanism as well as the IOMMU protection. We showed that those protections can be bypassed.
The attempt to isolate hypervisor and protect it even from attacks originating from Dom0 is not surprising. After all if we would like to treat the hypervisor as a root of trust, then we should make sure that its code base is minimal. If we now allow Dom0 to effectively be the hypervisor (i.e. if we don't care about Dom0-to-hypervisor escalations) then we should include all the Dom0 code to the hypervisor code base, when evaluating hypervisor security. This would result in our "extended hypervisor" having not ~300k lines of code (like current Xen does), but millions of lines of code!
Having said that all about how important it is to prevent all the possible Dom0-to-hypervisor attacks, I should stress that we also presented an attack that does not require Dom0 access and that can be spawned from an unprivileged DomU domain. As far as I'm aware, Rafal's FLASK bug & exploit (presented in the 2nd presentation) was the first public example of a successful exploitation of an overflow in a bare-metal hypervisor. The bug was a heap overflow and Rafal presented some clever tricks of how to control the Xen's heap allocations in order to make this bug exploitable.
Please note that all the rootkit-like stuff that we also presented, i.e. Rafal's Xen Loadable Modules framework and his hypervisor rootkits, as well as Alex's and mine XenBluePill, can all be used with all the above mentioned attacks. So, e.g. if we are on a machine that has VT-d support and run Xen 3.3 we can still use the Q35 attack and get the XLM framework running and then use it to install e.g. XenBluePill on top of running Xen, as showed during the 3rd presentation. Similarly, we could use the FLASK exploit and get XLM running again and again used it for installation of the other stuff.
Hope this clears some confusion about our presentations. As already promised, the codes and demos and full version of the 2nd talk slides (with the Q35 attack details) will be posted after Intel release the patch for their motherboards. Stay tuned.
Most of the attacks we presented do indeed require that the attacker first obtained access to Dom0 and only from there can launch further attacks. For example the DMA attacks that allow to overwrite hypervisor memory do indeed assume Dom0 access. The same applies to the Q35 exploit - this one is similar to the above mentioned DMA attacks in that it also requires access to certain hardware (that is possible from Dom0), but has an advantage that can bypass the hypervisor VT-d protection in the recent Xen 3.3.
There are several reasons why those attacks are still very important though:
1) First note, that the requirement for the attacker to have access to Dom0 in order to install e.g. hypervisor rootkits, is similar to the requirement that in order to install a Windows or Linux rootkit, one first needs to obtain administrator's privileges. Yet we know that Windows or Linux rootkits is a serious security problem.
2) Concerning the Xen-based systems specifically: over the last year several bugs have been discovered and published, that allowed an attacker to gain control of the Domain0 from an unprivileged domain (i.e. escape from the virtual machine). Rafal has discovered one such bug in December 2007.
3) Recent versions of Xen make deliberate effort to protect the hypervisor even from the Dom0. On systems that have IOMMU support (e.g. Intel's VT-d), the hypervisor memory is protected from tampering using both the processor's ring3/ring0 separation mechanism as well as the IOMMU protection. We showed that those protections can be bypassed.
The attempt to isolate hypervisor and protect it even from attacks originating from Dom0 is not surprising. After all if we would like to treat the hypervisor as a root of trust, then we should make sure that its code base is minimal. If we now allow Dom0 to effectively be the hypervisor (i.e. if we don't care about Dom0-to-hypervisor escalations) then we should include all the Dom0 code to the hypervisor code base, when evaluating hypervisor security. This would result in our "extended hypervisor" having not ~300k lines of code (like current Xen does), but millions of lines of code!
Having said that all about how important it is to prevent all the possible Dom0-to-hypervisor attacks, I should stress that we also presented an attack that does not require Dom0 access and that can be spawned from an unprivileged DomU domain. As far as I'm aware, Rafal's FLASK bug & exploit (presented in the 2nd presentation) was the first public example of a successful exploitation of an overflow in a bare-metal hypervisor. The bug was a heap overflow and Rafal presented some clever tricks of how to control the Xen's heap allocations in order to make this bug exploitable.
Please note that all the rootkit-like stuff that we also presented, i.e. Rafal's Xen Loadable Modules framework and his hypervisor rootkits, as well as Alex's and mine XenBluePill, can all be used with all the above mentioned attacks. So, e.g. if we are on a machine that has VT-d support and run Xen 3.3 we can still use the Q35 attack and get the XLM framework running and then use it to install e.g. XenBluePill on top of running Xen, as showed during the 3rd presentation. Similarly, we could use the FLASK exploit and get XLM running again and again used it for installation of the other stuff.
Hope this clears some confusion about our presentations. As already promised, the codes and demos and full version of the 2nd talk slides (with the Q35 attack details) will be posted after Intel release the patch for their motherboards. Stay tuned.
Friday, August 08, 2008
Our Xen 0wning Trilogy Highlights
Below you can find highlights of the three presentations, collectively referred to as "Xen 0wning Trilogy", that Alex, Rafal and I gave today at the Black Hat conference in Las Vegas.
Talk #1
1) Practical implementation of reliable and portable DMA attacks from Domain 0 to the Xen hypervisor memory.
2) Xen Loadable Modules :) A framework that allows to load arbitrary C code modules into the running Xen hypervisor. It uses DMA attack from the previous point to get access to Xen memory.
3) Two implementations of Xen Hypervisor Rootkits. This was the first time that working hypervisor rootkits have been presented (note the distinction between hypervisor rootkit vs. virtualization based rootkits).
Talk #2
1) Discussed how Xen 3.3 makes use of the Intel VT-d technology to protect the hypervisor.
2) Then we discussed how to bypass such VT-d protection on certain motherboards, like e.g. Intel DQ35 board.
3) An extra bonus: our attack from the previous point allows also to subvert the SMM handler and e.g. install an SMM rootkit in the system.
4) Discussed other Xen security mechanisms like driver domains, stub domains, PV GRUB and also attempted to quickly compare the state of Xen security design with the Hyper-V and ESX hypervisor.
5) Showed an exploitable heap overflow bug in the Xen hypervisor. The bug was in the FLASK module -- the NSA implementation of Xen Security Modules. FLASK, however, is not turned on by default, so even though we showed how to successfully exploit this heap overflow (which results in an escape from an unprivileged domain directly to the hypervisor), this is not a bug that can be used to 0wn The Planet. It shows, however, what happens when people start adding more and more code into the hypervisor.
6) Introduced HyperGuard -- a project done in cooperation with Phoenix Technologies. HyperGuard is going to be a SMM-based integrity scanner for Xen-like hypervisors. With HyperGuard we take a different approach then other integrity scanners do -- rather than ensuring the correctness of the code and data of the hypervisor, which might be very tricky, we instead ensure there is no untrusted code in the hypervisor, which is a much simpler task.
Talk #3
1) Provided detailed description of how to implement nested hardware based virtualization on AMD-V and VT-x (a copy of the slides from my RSA speech in April).
2) Showed how to use this nested virtualization to implement Blue Pill Boot, that can be used to virtualize the system right from the boot stage. We mentioned the best defend against this kind of system compromises is a trusted boot mechanism, either SRTM or DRTM, as implemented e.g. by Xen's tboot.
3) Consequently we showed Xen Blue Pill that is able to move a running Xen system into a virtual machine on the fly. This, on the other hand, cannot be prevented by neither the SRTM nor DRTM technology. XBP is a good example that running a legitimate hypervisor doesn't always prevent bluepill-like malware from being installed in the system.
4) Finally, discussed the XBP detection. First, we noted that all the "VMM detectors", proposed over the last years, that try to detect if there is a hypervisor running above, are useless in the case of a bluepilled Xen system. The only one approach that could be used is the direct timing analysis of the #VMEXIT times in order to distinguish between the native Xen case vs. bluepilled Xen case. We noted however, that direct timing analysis will not observe any differences when run from PV domains on AMD processors, and that it will observe little difference when run from HVM domains (7k vs. 5k cycles). The detection is easier on Intel processors, because of the unconditional #VMEXIT that we cannot get rid of.
All the three talks can be found here.
Talk #1
1) Practical implementation of reliable and portable DMA attacks from Domain 0 to the Xen hypervisor memory.
2) Xen Loadable Modules :) A framework that allows to load arbitrary C code modules into the running Xen hypervisor. It uses DMA attack from the previous point to get access to Xen memory.
3) Two implementations of Xen Hypervisor Rootkits. This was the first time that working hypervisor rootkits have been presented (note the distinction between hypervisor rootkit vs. virtualization based rootkits).
Talk #2
1) Discussed how Xen 3.3 makes use of the Intel VT-d technology to protect the hypervisor.
2) Then we discussed how to bypass such VT-d protection on certain motherboards, like e.g. Intel DQ35 board.
3) An extra bonus: our attack from the previous point allows also to subvert the SMM handler and e.g. install an SMM rootkit in the system.
4) Discussed other Xen security mechanisms like driver domains, stub domains, PV GRUB and also attempted to quickly compare the state of Xen security design with the Hyper-V and ESX hypervisor.
5) Showed an exploitable heap overflow bug in the Xen hypervisor. The bug was in the FLASK module -- the NSA implementation of Xen Security Modules. FLASK, however, is not turned on by default, so even though we showed how to successfully exploit this heap overflow (which results in an escape from an unprivileged domain directly to the hypervisor), this is not a bug that can be used to 0wn The Planet. It shows, however, what happens when people start adding more and more code into the hypervisor.
6) Introduced HyperGuard -- a project done in cooperation with Phoenix Technologies. HyperGuard is going to be a SMM-based integrity scanner for Xen-like hypervisors. With HyperGuard we take a different approach then other integrity scanners do -- rather than ensuring the correctness of the code and data of the hypervisor, which might be very tricky, we instead ensure there is no untrusted code in the hypervisor, which is a much simpler task.
Talk #3
1) Provided detailed description of how to implement nested hardware based virtualization on AMD-V and VT-x (a copy of the slides from my RSA speech in April).
2) Showed how to use this nested virtualization to implement Blue Pill Boot, that can be used to virtualize the system right from the boot stage. We mentioned the best defend against this kind of system compromises is a trusted boot mechanism, either SRTM or DRTM, as implemented e.g. by Xen's tboot.
3) Consequently we showed Xen Blue Pill that is able to move a running Xen system into a virtual machine on the fly. This, on the other hand, cannot be prevented by neither the SRTM nor DRTM technology. XBP is a good example that running a legitimate hypervisor doesn't always prevent bluepill-like malware from being installed in the system.
4) Finally, discussed the XBP detection. First, we noted that all the "VMM detectors", proposed over the last years, that try to detect if there is a hypervisor running above, are useless in the case of a bluepilled Xen system. The only one approach that could be used is the direct timing analysis of the #VMEXIT times in order to distinguish between the native Xen case vs. bluepilled Xen case. We noted however, that direct timing analysis will not observe any differences when run from PV domains on AMD processors, and that it will observe little difference when run from HVM domains (7k vs. 5k cycles). The detection is easier on Intel processors, because of the unconditional #VMEXIT that we cannot get rid of.
All the three talks can be found here.
Monday, July 07, 2008
0wning Xen in Vegas!
At this year’s Black Hat conference in Las Vegas in August we will be presenting three talks about the Xen hypervisor (in)security. The three presentations have been designed in such a way that they complement each other and create one bigger entirety, thus they can be referred as “Xen 0wning Trilogy” for brevity.In the first presentation, Subverting the Xen hypervisor, Rafal will discuss how to modify the Xen’s hypervisor memory and consequently how to use this ability to plant hypervisor rootkits inside Xen (everything on the fly, without rebooting Xen). Hypervisor rootkits are very different creatures from virtualization based rootkits (e.g. Bluepill). This will be the first public demonstration of practical VMM 0wning (proof of concept code will be released, of course).
In the second talk, Detecting and Preventing the Xen hypervisor subversions, Rafal and I will discuss various anti-subverting techniques (IOMMU, Xen’s driver- and stub- domains) and whether they really can protect the Xen (or similar) hypervisor from compromises. After demonstrating that those mechanisms can be bypassed, we will switch to discussing hypervisor integrity scanning and will present some prototype solutions to this problem.
Our trilogy wouldn’t be complete without discussing virtualization based malware in the context of bare-metal hypervisor compromises. Thus, in the third speech, Bluepilling the Xen hypervisor, Alex and I will discuss how to insert Bluepill on top of the running Xen hypervisor. We will show how to do that both with and without restart (i.e. on the fly). To make this possible, our Bluepill needs to support full nested virtualization, so that Xen can still function properly. We will also discuss how the “Bluepill detection” methods proposed over the last 2 years, as well as the integrity scanning methods discussed in the previous speech, fit into this new scenario and how far we are from the stealth malware’s Holy Grail ;)
Special thanks to Black Hat organizers for scheduling all the three presentations one after another in a dedicated Virtualization track on the 2nd day of the conference (August 7th).
It’s worth noting that we chose Xen as the target not because we think it’s insecure and worthless. On the contrary, we believe Xen is the most secure bare-metal hypervisor out there (especially with all the goodies in the upcoming Xen 3.3). Still we believe that it needs some improvements when it comes to security. We hope that our presentations will help making Xen (and similar hypervisors) more secure.
Tuesday, July 01, 2008
Rafal Wojtczuk joins Invisible Things Lab
I’m very happy to announce that a well known researcher, Rafal Wojtczuk, will join our team this month.
For anybody who is serious about OS security research it is hard not to know Rafal’s work. I remember reading his Defeating Solar Designer non-executable stack patch article somewhere around 1998, when I was still a Linux newbie, learning shell programming back at that time ;)
Since then Rafal published many other articles, advisories and exploits, mostly Linux-related. To mention just a few – the *BSD procfs vulnerability (2000), the Linux Ptrace vulnerability (2001), the famous Advanced return-into-lib(c) paper (2002), the vulnerability in the SELinux (2003), a tool for automatic integer overflow discovery in Win32 binaries (2005) and many others. He’s also known for his libnids project.
Recently Rafal has been doing a lot of research in the area of virtualization and VMM security. In the recent months he found vulnerabilities that potentially allowed to escape a VM jail in all the major virtualization software from Microsoft, VMWare and, of course, Xen.
I wrote “and, of course, Xen”, as Rafal will be presenting a talk at the upcoming Black Hat about Subverting the Xen hypervisor. His talk will be the first one in the series of 3 presentations about Xen (in)security that Invisible Things Lab prepared for this year’s Black Hat. Stay tuned for more details in the coming days.
Rafal has been with McAfee Avert Labs until this month.
For anybody who is serious about OS security research it is hard not to know Rafal’s work. I remember reading his Defeating Solar Designer non-executable stack patch article somewhere around 1998, when I was still a Linux newbie, learning shell programming back at that time ;)
Since then Rafal published many other articles, advisories and exploits, mostly Linux-related. To mention just a few – the *BSD procfs vulnerability (2000), the Linux Ptrace vulnerability (2001), the famous Advanced return-into-lib(c) paper (2002), the vulnerability in the SELinux (2003), a tool for automatic integer overflow discovery in Win32 binaries (2005) and many others. He’s also known for his libnids project.
Recently Rafal has been doing a lot of research in the area of virtualization and VMM security. In the recent months he found vulnerabilities that potentially allowed to escape a VM jail in all the major virtualization software from Microsoft, VMWare and, of course, Xen.
I wrote “and, of course, Xen”, as Rafal will be presenting a talk at the upcoming Black Hat about Subverting the Xen hypervisor. His talk will be the first one in the series of 3 presentations about Xen (in)security that Invisible Things Lab prepared for this year’s Black Hat. Stay tuned for more details in the coming days.
Rafal has been with McAfee Avert Labs until this month.
Tuesday, May 20, 2008
1984?
I can't believe this is happening for real...
So, how can we enforce Google to never do read() on a
I wonder when we would get another cool service from Google, e.g. "Google Thoughts", where people would be able to store their most private and personal thoughts, so that they could "access and managed them from all over the world", "in a secure fashion", of course. Right, add the Thought Police to this picture and welcome to Orwell's Oceania!
There is a difference between using the Web for blog writing vs. giving away all the private aspects of your life for free to some corporation in an unencrypted form. I wonder whether all the people who understands the notion of the local hard disk will be vaporized some time...
So, how can we enforce Google to never do read() on a
/var/users/john_smith/heath_records.db? How about a read() implemented from within a kernel-level via RAW-disk access that would never be logged?I wonder when we would get another cool service from Google, e.g. "Google Thoughts", where people would be able to store their most private and personal thoughts, so that they could "access and managed them from all over the world", "in a secure fashion", of course. Right, add the Thought Police to this picture and welcome to Orwell's Oceania!
There is a difference between using the Web for blog writing vs. giving away all the private aspects of your life for free to some corporation in an unencrypted form. I wonder whether all the people who understands the notion of the local hard disk will be vaporized some time...
Subscribe to:
Posts (Atom)
