Saturday, January 21, 2012

Thoughts on DeepSafe

Several people asked me recently what I though about DeepSafe. So, below I present my opinion...

First, for any AV system (or Host IPS, or Personal Firewall, etc) to work effectively, there are three problems that must be addressed:
  1. How to protect the AV agent (code and data) from tampering (from the rest of the OS)?
  2. How can the AV agent get reliable access to (sensitive pieces of) the system memory and registers, and/or provide reliable memory protection for the (sensitive pieces of) the OS.
  3. What are those "sensitive pieces of” memory that should be monitored or protected?
From reading various PR materials, it seems like the #1 above is the primary differentiation factor for DeepSafe (DS). So, let's consider this problem in the context of e.g. a Windows OS. In order to protect its code and data, DS uses, as it is heavily advertised, Intel VT-x virtualization technology. Now, that sounds really secure -- after all what can be more secure than a hardware virtualization, right? ;)

But VT-x (including EPT) is only about CPU virtualization, which in our case translates to protecting the DS memory and registers from CPU-originating accesses. But, as every regular to this blog knows, there is also another method of accessing memory on any PC system, and this is through DMA transactions from devices. The OS (so also the kernel malware) is free to program one of the many devices in the system to issue DMA reads or writes to any physical memory it wants...

Now, in order to protect some portion of the system memory (DRAM, cache) against DMA accesses, we have the Intel VT-d technology... So, one would think that DS must be also using VT-d in order to protect itself.

Very well, let's assume then that the DeepSafe is not a total ripoff, and that it implements also VT-d protection for its agent, although I haven't found this mentioned in any of the public papers or press materials I found on the web...

This, however, would be a bit complex to do correctly, because the OS (so, also the kernel malware) still has a full control over the chipset (MCH), which is the entity... that controls the VT-d.

Now, in order to prevent the OS (or the kernel malware) from playing with the chipset for fun and profit, and e.g. disabling VT-d protection, DS would have to virtualize the chipset.

If you look at some consumer VMMs, such as VMware or Xen/Qemu, you would see that they all virtualize the chipset for their guests (of course), but that the chipset they provide this way is some kind of an ancient Pentium MCH. I don't think any of the consumers would be especially happy if they found out that after installing DS on their brand new 2012 laptop, Windows suddenly see a Pentium-era chipset... And this is not without a reason – chipsets, specifically MCHs, are one of the most complex devices, perhaps only beaten by GPUs in this category. There are virtually hundreds of configuration registers exposed by the chipset, some of them control the VT-d, some other control system memory maps and permissions, PCIe configuration, and many other things that I even have no idea about, and this all makes virtualizing the chipset a very challenging task.

So, it's either that McAfee and Intel found some interesting way of how to securely virtualize the chipset while preserving all of its (very rich) functionality, or that they... don't bother with VT-d protection and chipset virtualization at all, assuming that even without VT-d, DeepSafe is good enough and “rises the bar” anyway (sarcasm intended).

(Can somebody from McAfee or Intel confirm in the comments below what does DP really do?)

Anyway, let's assume they do have VT-d protection and they do virtualize the chipset somehow...

Now, we're moving on to the #2 point from the list of tasks above -- about the reliable
memory access or reliable protection.

So, let say that the DS agent decided that some part of the system memory, e.g. the IDT table, is sensitive and should be monitored/protected. So it sets up EPT traps to trigger an VT-x/EPT intercept on any access to that memory (or IDT base register), in order to find kernel malware that tried to modify IDT. That sounds really nice, but what if the malware uses DMA to modify IDT? DS would not be able to catch such access! (So far we considered the, hypothetical, use of VT-d only to protect the DS agent code).

One might think that DS is programming VT-d to sandbox each and every device in the system (so including GPU, USB controllers, NICs, SATA, etc) so they never be allowed to touch any of those sensitive parts of the system, such as IDT. Let's assume they do it this way...

And here we've arrived to the last point from the list at the beginning: which of the system memory constitutes those "sensitive pieces" that should be protected/monitored? IDT? Sure. What about all the code sections of the all the kernel modules? Yes. Are we fine now? Well, no, because the malware can hook some pointers other than the well known IDT. Some public NDIS data structure? Ok, we can add those to the protected areas. But, what about some undocumented NDIS structures? And this is just NDIS subsystem, one of the many subsystems in the Windows kernel... When we think about it, it should be intuitively obvious that in a general purpose Operating System like Windows, it is not possible (at least for 3rd party) to make a satisfactory list of all the sensitive pieces of memory that should be monitored/protected, in order to detect all the system compromises.

Greg Hoglund, Jamie Butler, Alex Tereshkin, and myself, have been researching this area actively in the early years of this millennium. In addition to the Alex's paper linked above, you might also check out one of my last slides from this period.

I don't think anything has changed since that time. It was also the reason why I gave up on writing Windows compromise detectors, or forensic tools, and moved on to researching other ways to secure OSes, which finally gave birth to Qubes OS, many years later.

So, back to DS -- in order to provide a somehow satisfactory protection level for your general purpose OS, such as Windows, it would need to:
  1. Use VT-d to protect its own memory,
  1. Virtualize the chipset, at least some (sensitive) parts of it,
  1. Program VT-d permissions for each device to exclude all the sensitive areas in the system that should be protected, and also protect one device from DMAing into/from another device memory (e.g. NIC stealing GPU framebuffer, or inserting instructions to the GPU instruction buffer, or keystrokes to USB controller buffer). Ideally, this could be done by programming VT-d to grant each device only access to its own DMA buffer, but as far as I know, this would be very hard to implement, if not impossible for a 3rd party, on a Windows OS (in contrast to Linux, which mostly support this model). Please correct me, if the recent Windows version allows for such use of VT-d.
  1. Finally, and the most hard thing to solve, how to define all the "sensitive pieces of memory" in the system that should be protected and/or monitored? Although this is a somehow more generic problem, not specific to DS, but applying to any A/V, HIPS, or forensic tool.
So, is DeepSafe another piece of crap not worth any special attention, or has McAfee and Intel came up with some novel methods, e.g. for chipset virtualization, and other problems? Unless I see some technical info to backup the latter, I would have to assume, unfortunately, the former. But I would like to be mistaken – after all DeepSafe seems to be just a new incarnation of my Bluepill ;)


Anonymous said...

I'm a assemnbly programmer and operating systems developer ,
I like to have full control of the hardware and i am bored and tired of all this security.
If governments, the military and any other institution wants
such assurance, Intel should produce models of processors and chipsets targeted to these needs
without affecting the,desktop models, where nobody want to steal the latest mp3s that you downloaded :-)
Much of the security of a PC is operating system dependent ... chipset and processors are becoming
more complicated with unnecessary things like virtualization ..
Before I could easily handle all the hardware
including SMI (System Management Interrupt),
now has become more difficult to take control
over SMI,Intel has added new MSR registers in the new chipset.
I spend the money for the computer to use it,
not to run a shit code like the bios, or stupid high level windows and linux.

Joanna Rutkowska said...

BIOS is not a useless (or obsolete) piece of code as most people think -- it handles essential tasks, such as DRAM initialization, which is a very complex task, requiring probably thousands lines of code on modern chipsets. This is probably the price we pay for ultra fast DRAMs we all got used so easily.

But another thing is e.g. the SMI handler, where OEMs often put lots of USELESS CRAP, and this should definitely be somehow banned. Anybody willing to sue an OEM for putting Tetris into your SMI?

Paul Harper said...

Vapourware comes to mind when I look at Deepsafe.

Ross Anderson author of the classic 'Security Engineering hit the nail on the head when it come to Windows in particular in the article, 'Security and your mother's Linux box'.

"LXF: OK then, what steps should an ordinary citizen take to improve their data security?

RA: Buy a Linux box or a Mac. I bought my wife a Mac, last time the Windows box got filled up with loads of spyware."

RA "While to be fair to them they have improved they have too much legacy to ever really fix themselves."


Joanna Rutkowska said...

There is really nothing that makes Linux or Mac (except iOS) any more secure than Windows. All of those OSes are inherently insecure, architecturally. They are only safer than Windows, because of the much smaller user base. iOS is a different story, though -- I consider it to be significantly safer, architecturally (but not necessarily implementation-wise).

SMI said...

2) Another question that comes to mind is how the Deepsafe software would know it is indeed running at the bare metal. A rookit could, through nested virtualization, fool Deepsafe into thinking it was in control even though it wasn't. Bluepill is a proof of this concept.

One defence against this might be Trusted Computing technologies... such a bluepill
attack would leave a trace in the PCR registers. However, many/most PC's don't have TPM's so for those it wouldn't be an option. Further, even for PC's with TPM's, the rootkit could still fool Deepsafe into thinking it was running on bare metal (there are many ways, one could be to virtualize the TPM, and issuing it a certificate from a "fake vendor", and - at runtime - patching Deepsafe into trusting this certificate... or it could make Deepsafe bypass the checks altogether). The fact that the system was compromised can only be seen by an external, trusted computer who can carry out a challenge-response protocol against the host computer's TPM (using the TPM_QUOTE facility) which would ultimately be able to reveal the forgery. I guess the Deepsafe software could be made to contact a remote system (maybe as part of the regular AV signture update) which could use the TPM_QUOTE mechanism - but the rootkit could patch Deepsafe to bypass this facility also, while still ensuring the user (in the "Windows security center") that everything was fine and dandy! One way to provide some protection against this would be to make the servers sent out an email to the owner of the computer in case of missing/unsuccessful authentication from the machine in question since this would provide for an out-of-band notification of the exposure (if the user could read this from a phone etc.). However, I can see some practical problems in this mechanism but they might not be impossible to overcome.

SMI said...

So does Deepsafe provide any benefit at all? Maybe, maybe not. If we assume a PC with a TPM, and that Deepsafe does remote attestation to the AV servers who can send an out-of-band notification to the owner in case of compromise, they could theoretically realize
the property of "being able to guarentee that the security software is running as a hypervisor". The server would be able to setup a secure tunnel end-to-end with the AV software to get info on how much has been scanned, how much has been found etc. All this info could be relayed to the user out-of-band. This would be a new property that was not possible before. The question is then, how much is it worth? In light of Johannas points and my own points on the difficulty of detecting viruses even if you have "full memory and I/O access", I don't think this is the panacea of antivirus software. But I still think it's nice to have this extra property since AV's are in general pretty good at detecting viruses even if they aren't perfect. So I still think this is a step in the right direction if Intel doesn't oversell the benefits.

On a side note, as far as I've been able to read, Deepsafe doesn't rely on any new CPU features. It seems to only rely on the existing VMX etc. features. This rasises the question why McAfee (or another vendor) couldn't have done this without being bought by Intel? After all, VMX is fully documented. Intel could just have entered into a collaboration with McAfee and help them built this tech into the CPU. What is preventing
other AV vendors from doing this (patents? by Intel? If so, Intel could have taken out those patents and licensed them to all vendors, potentially earning more than through buying ;cAfee). This makes me think there might be more to this Deepsafe tech than we think (I had executed the collaboration to result in some sort of 'AV-features-in-the-CPU' type of thing).
It could also have been realized by other means than VMX, for instance through Intel AMT. If the point is to remotely inventory the antivirus, AMT seems like an excellent choice since a lot of groundwork in those respects have already been done by Intel. Another possibility could be pure software virtualization which can also be pretty low-overhead.

SMI said...

Hi Joanna,

thanks for your thoughts! I too have wondered about how Deepsafe could possibly work and add security over what is possible now. The following is just speculation and is
based on the assumption that Deepsafe is simply a "AV-in-a-hypervisor" solution.

Regarding your own points, I agree that Deepsafe couldn't know how to protect all operating system vectors (see below). But it would probably at least be able to protect itself from being overwritten (even by DMA etc.) so it could stay in control (assuming it was running on the bare metal, see pt 2 below). Also,
other VM vendors are able to make hypervisors that can protect themselves from the guest. The real question is -- how much can Deepsafe protect the guest, even if it can guarentee that it itself can not be compromied?

I've been thinking of the problem more in the context of malware detection, not memory protection which you primarily discuss. However, malware detection has its own set of problems that it would seem
challenging for Deepsafe to address. While running at the bare metal provides ultimate power
to inspect I/O and memory and in principle control all software that is executed,
it also presents the following problem: How can Deepsafe "know" what the guest software
(i.e. the normal operating system/software stack) is up to? It could scan e.g. network
and disk I/O for viruses but what if this is encrypted? What if the user opens an
encrypted Truecrypt volume and executes the file from there? Deepsafe I/O monitoring
wouldn't be able to detect a virus flowing over the I/O in this case. Or how about if a virus is downloaded over a https:// connection? Another strategy could be for Deepsafe to do memory scanning. After all,
it would have unrestricted access to the physical address space. Still, this is hardly
fool-proof. Some viruses are known to rewrite their own x86 machine code and can produce
an infinity of variants. Other viruses encrypt themself and dynamically generates a
decoder which is only used when the virus activates - making its time in cleartext in memory very brief and unlikely to be caught by periodic memory scanning. Maybe Deepsafe could be very smart and through clever use of the paging virtualization facilities scan all pages containing code on the first access - and then every time they are rewritten (or if the page tables are updated to contain new executable pages). This
would still not help it catch the metamorphic viruses.
If Deepsafe were to implement a more behavior-based type of detection it becomes even more difficult. How can it hook into the proper vectors in Windows? It would have to, as an "outsider" look at the running Windows kernel, drivers, page tables, try to make sense of it all and do some brain surgery on it. Doesn't seem like a recipe for stability in the face of the many versions, fixes, drivers etc. It could be made more stable if it could have a piece of helper software running on the "inside" (like a kind of VMWare tools) which could help it interact with windows etc. put this immediately raises the question whether you can trust any information gathered by such
a helper, since he's after all running in the untrusted space you are trying to safeguard!

Anonymous said...

But, as every regular to this blog knows, there is also another method of accessing memory on any PC system, and this is through DMA transactions from devices.

Good point.

To put this in context how many rootkits exploit this method to compromise a system?


Joanna Rutkowska said...


How many? How do you count/distinguish them? By different hash value of their installer code? Or in-memory code fingerprint? In any case I can generate easily as many variants as you want...