My SSTIC 2011 slides

A few days ago I had a privilege to give an opening keynote at the SSTIC conference in Rennes, France, which is believed by many to be the most important security conference in France. You can find my slides here.

SSTIC seems to be a very interesting conference indeed, with a strong emphasis on system-level security, which is quite unusual these days where most conferences focus on networking, apps, and web-apps. What a pity all those interestingly-looking talks have been encoded in an obscure language used only by some 3% of the population of the planet...

Anyway, it was a pleasure to talk to some ANSSI people I met before the conference (one of the organizers of the event) who really seemed to understand well the challenges we face with building secure operating systems, and generally seemed well versed in the topic. Perhaps some other nations should learn from France, instead of proposing ridiculous and superficial means that can't really solve any real problem.

From Slides to Silicon in 3 years!

Remember our Xen 0wning Trilogy at Black Hat in summer 2008, specifically the presentation on Detecting & Preventing the Xen Hypervisor Subversions?

One of the things we were discussing there was a proposal to include an additional restriction to Intel processors that would disallow execution of usermode pages from within supervisor mode (ring0). Such a feature, we argued, apart from obviously making many ring3-to-ring0 exploits much harder, including the very Xen heap overflow exploit we presented in the slides, would also bring us closer to efficient runtime code integrity checkers for kernels and hypervisors, as discussed in the slides.

Slide #97, Detecting and Preventing Xen Hypervisor Subversions, Black Hat USA, July, 2008

Fast forward 3 years. On June 1^st, 2011, an Intel engineer is submitting a patch for Xen to support a mysterious new processor feature called SMEP (Supervisor Mode Execution Protection). He writes the feature is not yet documented in SDM, but soon will be. In fact, the May 2011 update of Intel SDM already contains the details:

Intel SDM, vol. 3a, May 2011, source: intel.com

Some other people spotted this feature earlier, because of another patch submitted by another Intel engineer to Linux kernel a few weeks ago. Here's a good write up by Dan Rosenberg discussing how this patch makes writing Linux kernel exploits harder, and how it's still possible to write them.

The SMEP feature still doesn't seem to be present in the processors available on the market, including the latest Sand Bridge processors, but there's no question it's coming, now that the feature made it into SDM.

It is quite rewarding to see your idea implemented in a processor... I guess this is how physicists feel when they introduce a new particle as part of a new quantum model, and later discover evidences to support the existence of this very particle in the wild...

USB Security Challenges

When we think about “USB Security” there are lots of things that come to mind...

First there are all the physical attacks that could be conducted with the help of USB devices. These are generally not so interesting, because if one includes physical attacks in the threat model, then it really opens up lots of possibilities of various attacks, and generally a physical attacker always wins. Still, there are a few very cheap and easy physical attacks that one would like to avoid, or make harder, such as the Evil Maid Attacks or the Cold Boot Attacks. Strictly speaking these are not problems inherent to USB itself, but rather with lack of Trusted Boot, or OS not cleaning properly secrets from memory upon shutdown. They are just made simple thanks to bootable USB sticks.

Much more interesting USB-related physical attacks are those that take advantage of the specifics of the USB standard. One example here would be a malicious USB device that exposes intentionally malformed info about itself in order to exploit a potential flaw in a USB Host Controller driver that processes this info upon each new USB device connect. Or a malicious USB device that would trick the OS (Windows at least) into downloading a known buggy USB driver (or even an intentionally malicious driver, legally submitted to WHQL by the attacker) and then exploit the driver.

Another class of physical attacks made possible by the USB specification are malicious USB devices that pretend to be a keyboard or mouse. The input devices, such as keyboard, are actually the most security sensitive devices, because an attacker who controls the keyboard can do everything the user can do, which basically means: can do everything, at least with regards to the user's data.

Finally, the USB, as the names stands, is a bus interconnect, which means all the USB devices sharing the same USB controller are capable of sniffing and spoofing signals on the bus. This is one of the key differences between USB and PCI Express standards, where the latter uses a peer-to-peer interconnect architecture.

Ok, so these all above were physical attacks. Let's now look at, much more fatal, software attacks.

The infamous class of attacks exploiting various autorun or auto-preview behaviors is the most known example, but also the easiest, at least in theory, to protect against.

Much more interesting are software attacks that attempt to exploit potential flaws in the USB stacks – similarly like the physical attacks mentioned above, just that this time not requiring any hardware-level modifications to the USB device. Exposing a malformed partition table is a great example of such an attack. Even if we have all the autorun mechanisms disabled, still, when we're inserting a storage medium the OS always attempts to parse the partition table in order to e.g. create devices symbolizing each partition/volume (e.g. /dev/sdbX devices).

Now, this is really a problematic attack, because the malformed partition table can be written onto a fully legitimate USB stick by malware. Imagine e.g. you have two physically separated machines (air-gapped), belonging to two different security domains, and you want to transfer files from one to another. You insert the USB stick into the first machine, copy files, and then insert the stick to the second machine. If the first machine was compromised, it could have altered the partition table on the USB stick, and now when this stick is inserted into the other machine its malformed partition table might exploit a buffer overflow in the code used by the second OS to parse the stick's partition information. Air-gapped systems, huh? We avoid this attack vector in Qubes by using a special inter-domain file copy mechanism that doesn't require any metadata parsing.

A variation of the above attack would be to expose a malicious file system metadata, but this time the target OS would have to actually mount the partition for the attack to work (and, of course, there would have to be bugs in the OS file system parsing code, although these  seem to be quite common on most OSes).

Having quickly summarized the USB security-related threats, let's now think about how we could design an OS to mitigate most of those attacks, and at the very least the software-based attacks. This is, in fact, precisely the challenge we've been facing in Qubes, so the divagations below necessarily focus mostly on the Qubes architecture.

First we should realize that USB devices, unlike PCI Express devices, cannot be independently delegated to different domains (VMs). This is because IOMMU technologies, such as Intel VT-d, operate only on PCIe device granularity. This means we can only delegate a whole USB controller to a domain, including all of the USB devices connected to this controller/hub.

Imagine now two internal devices, both connected via internal USB bus: a keyboard, and a 3G wireless modem. Chances are high that you will have those two devices connected to the same USB controller – usually one controller is used for all the internal devices, like those I just mentioned, plus camera, fingerprint reader, etc, and the other controller is used for all the externally visible USB connectors (at least this is true for modern systems: Intel Series 5 chipsets and newer).

We would like to be able to delegate the 3G modem to the NetVM (an untrusted domain on Qubes where all the networking drivers and stacks are kept; it's considered untrusted because its compromise is equivalent to a compromise of a WiFi network or home router, or some other router, and any reasonable person always assumes that the network is compromised, and deals with that using crypto, such as SSL or SSH). But assigning the USB controller, to which the 3G modem is connected to, to the NetVM, would also assign the USB keyboard to the NetVM! And this is precisely what we don't want to do, because control over the keyboard is equivalent to the control over the whole system!

Currently, in Qubes Beta 1, we keep all the USB controllers assigned to Dom0. This, however, causes two annoyances:

First, the user cannot use any of the USB-connected networking devices, such as 3G modems (because there is no networking in Dom0).

Second, if somebody connects a USB disk and later delegates it to some domain (this could easily be done via block-attach mechanism, supported by the same backend that handles storage virtualization for domains), and this domain turns out to be compromised, it might alter e.g. the stick's partition table and later attack Dom0 as explained above.

We can eliminate the second problem by modifying the Dom0's kernel to not parse the partition table of any removable devices automatically, and instead expect some kind of explicit consent from user to actually do that (we still must allow to mount USB disks in Dom0 to allow easy backups of all domains at once).

To allow the use of USB-connected networking devices in NetVM, we could use a PVUSB backend that can virtualize single USB devices without moving the whole USB controller to the domain. But that would require introducing a whole lot of new code to Dom0 – code that would be directly reachable from VMs (in other words that would be processing lots of untrusted input coming from untrusted domains).

So another option is to delegate all the non-security-critical USB controllers, i.e. those controllers that don't have any security-sensitive USB devices connected, such as keyboard, to a dedicated “USB” domain, and later share the USB devices via PVUSB backend from this USB domain. This time, the extra PVUSB backend runs in the USB domain, not in Dom0, so we don't worry that much about potential bugs in this backend. Of course, this way you cannot delegate the USB controller to which the keyboard, and potentially also other security-critical devices, such as camera, are connected to, which in practice rules out the integrated3G modem. Fortunately many modern laptops do not use USB-connected keyboard and touchpad (they use PS/2-connected keyboards instead), and the face camera can be easily disabled with a piece of sticker (although that sucks, because it means we cannot really use the camera).

With this approach (a dedicated USB domain) you can now delegate your 3G modem to the NetVM, and other USB devices, such as removable disks to other domains, e.g. for file exchange. This seems the most reasonable setup, although it requires that either 1) your laptop doesn't have USB-connected keyboard, or 2) you don't use internal USB devices connected to the same controller that your USB keyboard/touchpad from other domains than Dom0 (in practice: no 3G modem in NetVM).

As we can see proper handling of USB devices is quite a challenge for OS architects. It might have been much less of a challenge if the engineers designing the USB, chipsets, and motherboards were a bit more security-conscious. Even such simple practice as never mixing security critical devices (keyboard, touchpad, camera, fingerprint reader), with non-security ones (3G modem), onto the same USB controller, would help tremendously. Or ability to somehow dynamically configure their connectivity, e.g. in BIOS?

(Un)Trusting the Cloud

Everybody loves The Cloud these days, and it is not hard to understand why. When every person owns computers (devices), the cloud is really hard to beat when it comes to syncing all your digital life back and forth between all those devices, and also sharing with your family members, friends, and colleagues at work. From task lists, through calendars, through health & fitness data, to work-related documents. And I'm not even mentioning all the unencrypted email that is out there.

One doesn't need to be especially smart or security conscious to realize how much this might be a threat to security and privacy. How much easier would it be to attack somebody's laptop if I knew precisely in which hotel and when he or she is planning to stay? How much more expensive would my health and life insurance be, if they could get a look at my health and fitness progress? Etc.

But we're willing to sacrifice our privacy and security in exchange for easy of syncing and sharing of our data. We decide to trust The Cloud. What specifically does that mean?

First, it means we trust the particular cloud-based service vendor, such as the provides of our training monitoring app and service. We trust that this vendor is: 1) non-malicious and ethical, and so is not going to sell our private data to some other entity, e.g. insurance company, and 2) that the software written by this vendor is somehow secure, so it would not be easy for an attacker to break into their cloud service and download all the user's data (and then sell to health insurance companies).

Next, we trust the cloud infrastructure provider, such as Amazon EC2. We trust that the cloud provider is 1) non-malicious and ethical, and that they won't really read the memory of the virtual machine on which the previously mentioned cloud-service is running (and won't make it available to a local government officials, e.g. in China), and 2) that they secured their infrastructure properly (e.g. it wouldn't be easy for one customer to “escape” from a VM and read all the memory of the VMs belonging to other customers).

Finally we trust all the infrastructure that is in the middle between us and the service provider, such as e.g. the networking protocols, are safe to use (e.g. we trust all the engineers working in any of the ISP we use won't sniff/spoof our communication, e.g. by using some fake or quasi-fake SSL certs).

So, that's a hell of a lot of trusting! And the stake is high. Do we really need to make such a sacrifice? Do we really need to hand in all our private data to all those organizations? Of course we don't!

First, notice that in majority of cases, the cloud is only used basically as a on-line storage. No processing, just dump storage. Indeed, what kind of server-side processing does your task list or calender require? Or your freestyle swimming results? Or your conference slides? None.

And we know for very long how to safely keep secrets on untrusted storage, don't we? This is achieved via encryption (and digital signatures for integrity/authenticity). So, the idea is very simple: let's encrypt all the data before we send them to the cloud. The point here is, the encryption must be done by the app that is running on our client device. Not in the cloud, of course.

Ok, so let's say I have my calendar records encrypted in the cloud, how do I share it with my other devices and other people, such as my partner and colleagues at work? Very simple – you encrypt each record with a random symmetric key and then, for every other device or person who you want to grant access to your calendar you make the symmetric key available to this person, by encrypting it with their public key (if you're paranoid, you can even verify fingerprints using some out-band communication channel, such as phone, to ensure the cloud/service provider didn't do MITM attack on you). What if you want to share only some events (or some details) with some group of people (e.g. only your availability info)? Very simple – just encrypt those records you want to share in non-full access with some other symmetric key and publish only this key to those people/devices you want to grant such non-full access.

Implementing the above would require writing new end-user apps, or plugins for existing apps (such as Outlook), so that they do encryption/decryption/signing/verification before sending the data out to the cloud. But what stops the malicious vendor from offering apps that would be leaking out our secrets, e.g. the keys? Well, nothing actually. But this time, the vendor would need to explicitly build in some kind of backdoor into the app. The same could be done with any other vendor, and any other, non-cloud-based app. After all, how do we know that MS Word, which is not cloud-based yet, is not sending out fragments of our texts to Agent Smith? Note how different this is from a situation when the vendor already owns all our data, unencrypted, brought legitimately to their servers, and all they need to do is to read them from their own disks. No need to plant and distribute any backdoors!

In practice few vendors would be risking their reputation and would be willing to build in a backdoor into an app that is then made available to customers. Because every backdoor in such client-exposed code will sooner or later be found (You would really not believe what great lengths all those young people aimed with disassembler and debugger would go to, to win an economy class ticket to the middle of desert in the hottest summer season, just to be able to deliver a presentation on how evil/stupid a company X is ;).

One problem is, however, with accessing our encrypted cloud over a Web Browser. In contrast to apps, the web browser content is much less identifiable. An app can have a digital signature – everybody know its an App v 1.1, published by X. As explained above it would be rather stupid for X to plant a backdoor into such an app. But a Web-delivered Javascript is much more tentative, and it's very possible for X to e.g. deliver various versions of scripts to different customers. Digital signature on client-side scripts, paired with ability to whitelist allowed client-side-scripts, would likely solve this problem.

So, why we still haven't got client-side-encrypted cloud-services? The question is rhetorical, of course. Most vendors actually loves the idea of having unlimited access to their customers data. Do you think Google would be happy to give up an opportunity to data mine all your data? This might affect their ad business, health research, or just Secret Plan To 0wn The World. After our dead body, I can almost hear them yelling! After all they have just came up with Chrome OS to bring even more data into their data mining machine...

To sum it up, there is no technical reason we must entrust all those people with our most private data. Sooner or later somebody will start selling client-side-encrypted cloud services, and I would be the first person to sign up for it. Hopefully it will happen sooner than later (to late?).

This post also hopefully shows, again, one more aspect – that we can, relatively easy, move most of the IT infrastructure out of the “TCB” (Trusted Computing Base, used as metaphor here). In other words, we can design our systems and services so that we don't need to trust a whole lot of things, including servers and the networking infrastructure (except for its reliability, but not for its security). But, there always remains one element that we must trust – these are our client devices. If they are compromised, the attacker can steal everything.

Strangely most people still don't get it, or get it backwards. Just the fact that “information is not stored on the iPad but kept safe on the corporate network”, doesn't change anything! Really. If the attacker owns your iPad, then she also can do anything that the legitimate user could do from this iPad. So if you could get to the company's secret trade data from your iPad's Receiver, so would be able to do the malware/attacker.

The App-oriented UI Model and its Security Implications

Most of the desktop OSes today, such as Windows or Mac, expose and encourage a File-oriented UI model. You pick a file in the file manager, click it, and then the file manager automagically determines the best app to handle the file, starts the app, and passes the file to it.

Back in the MS-DOS days we used a different model: an app-oriented model – you started an app first, e.g. Word Perfect, or Lotus 1-2-3, and then you opened a file from within the app (Norton Commander and similar programs somehow changed that later).

Interestingly this very same app-oriented model is now becoming popular again thanks to systems such as iOS and Android. There is no such thing as a global File Explorer or Finder on an iPad. Only the apps. One must first pick an app, and then it's the application's responsibility to expose an option for opening one of your “files”, if the app supports it (e.g. the calendar or task list apps would always open your default calendar or task list without asking for anything).

I actually like this app-oriented model a lot! It's much less confusing to the user. Just think about all those attacks in the past where an attacker could prepare a file with some innocently-looking extension but which in fact was an MZ executable. Or how many times people are not even aware which app they use! One might argue that user should not be distracted by such “unimportant” things as what app he or she uses for her work, but I disagree. Apparently Apple, and millions of iPhone and iPad users, disagree too.

But the main reason why I like this app-oriented model is because it just fits greatly into the Security by Isolation philosophy.

Just think about it: if it's possible to get users to consciously select an app, and we now know it is possible thanks to the millions of app-oriented devices sold, then it should be not much more difficult to get them to also consciously select the domain or area, such as “work”, or “personal”, which they wish to use. Just imagine that instead of one “Mail” app, you would have two apps (and two icons): “Mail Work”, and “Mail Personal”.

There are some technicalities here – such as e.g. how to isolate apps between each other? Do we need to build another layer of isolation in a form of VMs to isolate “Mail Work” from “Mail Personal”, or should the (new) OSes and the (new) APIs be designed in such a way, that they were thin and secure, and allow for very good isolation between processes without using virtualization?

In Qubes we must use this additional layer of abstraction (virtualization), because we want to use Linux apps (and in the future also Windows apps), and they require huge POSIX/X API (and Win32 API) to work correctly. And those APIs are not easily isolate-able. So we use VMs as “API providers”. Same with isolating networking drivers and stacks – we need Linux kernel API to get those drivers and stacks running, so that's why we use a Linux-based “NetVM” for isolating networking. For this reason we expect users to explicitly define domains, such as “work”, “personal”, etc. This is because we cannot afford to run every single app in a separate AppVM (more precisely we cannot afford to create a working copy of this huge POSIX/X API for each app).

But we could very well imagine a well constructed API for apps that would just be easily isolate-able (I'm not saying iOS or Android has such an API), and so there would be no need to define domains explicitly. Still, we would need a possibility to define more than one instance of each app – such as the previously mentioned “Mail Work” and “Mail Personal”.

The app-oriented model seems to be the future. And so seems the Security by Isolation philosophy!

Following the White Rabbit: Software Attacks Against Intel VT-d

Today we publish a new paper which is a result of our several month long in-depth evaluation of Intel VT-d technology. To quote the abstract:

We discuss three software attacks that might allow for escaping from a VT-d-protected driver domain in a virtualization system. We then focus on one of those attacks, and demonstrate practical and reliable code execution exploit against a Xen system. Finally, we discuss how new hardware from Intel offers a potential for protection against our attacks in the form of Interrupt Remapping (for client systems available only on the very latest Sandy Bridge processors). But we also discuss how this protection could be circumvented on a Xen system under certain circumstances...

I think the attack is likely the most complex and surprising out of all the things we have presented so far. Parts of it are even funny (if you share our weird sense of humor), such as the use of ICMP ping to generate MSIs. The paper also covers the vendors' response. You can download the paper here.

The Linux Security Circus: On GUI isolation

There certainly is one thing that most Linux users don't realize about their Linux systems... this is the lack of GUI-level isolation, and how it essentially nullifies all the desktop security. I wrote about it a few times, I spoke about it a few times, yet I still come across people who don't realize it all the time.

So, let me stress this one more time: if you have two GUI applications, e.g. an OpenOffice Word Processor, and a stupid Tetris game, both of which granted access to your screen (your X server), then there is no isolation between those two apps. Even if they run as different user accounts! Even if they are somehow sandboxed by SELinux or whatever! None, zero, null, nil!

The X server architecture, designed long time ago by some happy hippies who just thought all the people apps are good and non-malicious, simply allows any GUI application to control any other one. No bugs, no exploits, no tricks, are required. This is all by design. One application can sniff or inject keystrokes to another one, can take snapshots of the screen occupied by windows belonging to another one, etc.

If you don't believe me, I suggest you do a simple experiment. Open a terminal window, as normal user, and run xinput list, which is a standard diagnostic program for Xorg (on Fedora you will likely need to install it first: yum install xorg-x11-apps):

$ xinput list

It will show you all the pointer and keyboard devices that your Xorg knows about. Note the ID of the device listed as “AT keyboard” and then run (as normal user!):

$ xinput test id

It should now start displaying the scancodes for all the keys you press on the keyboard. If it doesn't, it means you used a wrong device ID.

Now, for the best, start another terminal window, and switch to root (e.g. using su, or sudo). Notice how the xinput running as user is able to sniff all your keystrokes, including root password (for su), and then all the keystrokes you enter in your root session. Start some GUI app as root, or as different user, again notice how your xinput can sniff all the keystrokes you enter to this other app!

Yes, I can understand what is happening in your mind and heart right now... Don't worry, others have also passed through it. Feel free to hate me, throw out insults at me, etc. I don't mind, really (I just won't moderate them). When you calm down, continue reading.

In Qubes the above problem doesn't exist, because each domain (each AppVM) has it own local, isolated, dummy X server. The main X server, that runs in Dom0 and that handles the real display is never exposed to any of the AppVMs directly (AppVMs cannot connect to it via the X protocol). For details see this technical overview.

You can repeat the same experiment in Qubes. You just need to use the ID of the “qubesdev” device, as shown by xinput list (should be 7). Run the xinput in one of your domains, e.g. in the “red” one. Because we actually use the same device for both mouse and keystrokes, you should now see both the key scancodes, as well as all the mouse events. Notice how your xinput is able to sniff all the events that are destined for other apps belonging to the same domain where you run xinput, and how it is unable to sniff anything targeted to other domains, or Dom0.

BTW, Windows is the only one mainstream OS I'm aware of, that actually attempts to implement some form of GUI-level isolation, starting from Windows Vista. See e.g. this ancient article I wrote in the days when I used Vista at my primary laptop. Of course, it's still easy to bypass this isolation, because of the huge interface that is exposed to each GUI client (that also includes GPU API). Nevertheless, they at least attempt to prevent this at the architecture level.