Qubes Alpha 3!

We have just uploaded the new packages for the Qubes Alpha 3 milestone. A lot of under the hood work went into this release, including:

• Support for fast-booting Disposable VM (see also implementation description here)

• Dynamic memory balancing between AppVMs

• Redesigned networking and NetVM support (for VT-d system)

• Reasonably stable S3 sleep support (suspend-to-RAM), that works even with a NetVM!

• Improved GUI virtualization (all known bugs fixed finally!)

Disposable VMs are really a killer feature IMO. The screenshot below shows the user's experience:

The user righ-clicks on a PDF file, chooses "Open in Disposable VM", and then waits 1... 2... 3... 4 seconds (assuming a reasonably modern laptop) and the document automagically opens in a fresh new Disposable VM. If you make some changes to the document (e.g. if it was a PDF form, and you edited it), those changes will propagate back to the original file in the original AppVM.

So, within 4-5 seconds, Qubes creates a new VM, boots it up (actually refreshes from a savefile), copies the file in question to the VM, and finally opens the application that is a registered MIME handler for this type of documents, e.g. a PDF viewer. We're pretty confident this time could be further decreased down to some 2 seconds, or maybe even less. This is planned for some later Beta release.

Dynamic memory balancing allows to better utilize system physical memory by moving it between running AppVMs in realtime, according to the VM's real needs. This allows to run more VMs, compared to a scheme with static memory allocation, and also dramatically eliminates system hiccups, that otherwise occur often in a static scheme when one of the VMs is short of memory and initiates swapping.

The screenshot above shows the memory usage on my 6GB laptop when writing this blog post. As you can see I can easily run a dozen of AppVMs (most users will not need that many, but I'm a bit more paranoid I guess ;) and could probably even start a few more if there was such a need (e.g. open some Disposable VMs). Of course, this all depends on the actual type of workload the user runs in each VM - most of my AppVMs run just one or two applications, usually a Web browser (Firefox), but some, e.g. the work, and personal AppVMs run much more memory-hungry applications such as Open Office, or Picasa Photo Browser. I very rarely see more than 1 GB of memory allocated to a single VM, though. Generally speaking, the new memory management in Qubes works pretty nice.

Currently, the biggest slow-down factor for Qubes is somehow poor disk performance, most likely caused by the joint impact of the Xen backend, Linux dm, and kcryptd (we use the simplest possible Xen block backend for security reasons, will move to more sophisticated backends when we introduce untrusted storage domain in Qubes 2.0).

Now, most of the under-the-hood work for Qubes 1.0 seems to be complete, and now it time for all the polishing of the user experience, which will be the main focus of the upcoming Beta development. Just reminding that we're currently looking to hire developers for this effort.

The Installation instructions can be found here. Enjoy!

ITL is hiring!

We're looking to hire one or two full time developers, who will be working on the open source version of Qubes OS, with the primary task of advancing it from Alpha to Beta stage, and then finally to a production quality version.

We're looking to hire developers, not necessarily security researchers! Specifically we expect the following from candidates:

• Many years of experience with Linux/GNU development, including system-level and kernel-level Linux development, documented by the actual projects,
  
• Familiarity with virtualization technologies, and specifically with Xen hypervisor,
• Basic understanding of the Qubes architecture and excitement about the project :)
• Product-oriented approach (polishing, testing, packaging, understanding of user needs),
• Good communication skills in written English
  

In return we offer the following benefits:

• Decent, full-time salary,
• Opportunity to be part of a renown security team,
• Opportunity to work on an exciting product,
  
• Work on a GPLed project with all the benefits it gives to the developer (visibility, rights to the code)

If you're interested in joining our team, please send a message to joanna at invisiblethingslab.com.

Please do not send typical resumes: don't write about schools you finished, certificates you obtained, driving license, scuba trainings, etc. We are only interested in a short bio (keep it below 100 words please), and links to your past or current projects. Include your geographic location.

While it would be great if you were based in Warsaw (or somewhere in Poland), as it would allow for regular face-to-face meetings, this is not a critical factor. ITL doesn't have a physical office, and everybody work from their apartments, so there is no need to relocate to Warsaw, in case you happened to be based somewhere else.

On Thin Clients Security

I'm constantly being asked about it, and so I thought I would write a handy blog post, so I could just referrer to it in the future, when yet anther person asks me if I think the use of Thin Clients is a game-changer to desktop security...

It is not! Thin Clients do not improve your desktop security in any way, and that's because:

1. You still run a regular full-blown OS, such as Widows and all the regular applications, such as those buggy PDF readers, Web browsers, etc - it's just you run them all on some corporate server, rather on your laptop. The fact that you run the OS on the corporate server, doesn't make it any less prone to compromises, compared to if you run it locally on your laptop.



2. A compromise of your laptop, even if it's just a dump terminal, is still fatal! This is because if your laptop's kernel (or MBR, or BIOS, or some PCI device's firmware, or GPU) is compromised, the attacker can intercept/steal/spoof all the data that you work on remotely, because it is still your laptop that processes the input (keystrokes, mouse events) and output (pixels). So, an Evil Maid attack on your laptop when you use it as a Thin Client, would be just as devastating, as it is otherwise (and don't fool yourselves that crypto tokens can help)

We really need secure end-user systems, even if we just want to use them as dump terminals only! There is really no way we could skip this step (and e.g. focus only on infrastructure, or services security).

(Un)Trusting your GUI Subsystem

Why do we need secure desktop systems? Why support from hardware is necessary to build secure desktop OSes? Does virtualization make things more, or less complex? Why Dynamic RTM (Intel TXT) is better than Static RTM? Can we have untrusted GUI domain/subsystem?

I tried to cover those questions in my recent keynote at ETISS, and you can grab the slides here.

Particularly, the slide #18 presents the idealistic view of an OS that could be achieved through the use of hardware virtualization and trusted boot technologies. It might look very similar to many other pictures of virtualized systems one can see these days, but what makes it special is that all the dark gray boxes represent untrusted domains (so, their compromise is not security-critical, except for the potential of a denial-of-service).

No OS currently implements this architecture, even Qubes. We still have Storage and GUI subsystem in Dom0 (so they are both trusted), although we already know (we think) how to implement the untrusted storage domain (this is described in detail in the arch spec), and the main reason we don't have it now is that TXT market adoption is so poor, that very few people could make use of it.

The GUI subsystem is, however, a much bigger challenge. When we think about, it should really feel impossible to have an untrusted GUI subsystem, because the GUI subsystem really "sees" all the pixmaps that are to be displayed to the user, so also all the confidential emails, documents, etc. The GUI is different in nature than the networking subsystem, where we can use encrypted protocols to prevent the netvm from sniffing or meaningfully intercepting the application-generated traffic, or the storage subsystem, where we can use fs-encryption and trusted boot technologies to keep the storage domain off from reading or modifying the files used by apps in a meaningful ways. We cannot really encrypt the pixmaps (in the apps, or AppVMs), because for this to work we would need to have graphics cards that would be able to do the decryption and key exchange (note how this is different from the case of an untrusted storage domain, where there is no need for internal hardware encryption!), and the idea of putting, essentially an HTTPS webserver on your GPU is doubtful at best, because it would essentially move the target from the GUI domain to the GPU, and there is really no reason why lots-of-code in the GPU were any harder to attack than lots-of-code in the GUI domain...

So we came out recently with an idea of a Split I/O model that is also presented in my slides, where we separate the user input (keyboard, mouse), and keep it still in dom0 (trusted domain), from the output (GUI, audio), which is moved into an untrusted GUI domain. We obviously need to make sure that the GUI domain cannot "talk" to other domains, to make sure it cannot "leak out" the secrets that it "sees" while processing the various pixmaps. For this we need to have the hypervisor ensure that all the inter-domain shared pages mapped into the GUI domain are read-only for the GUI domain, and this would imply that we need the GUI protocol, exposed by the GUI domain to other AppVMs, to be unidirectional.

There are more challenges though, e.g. how to keep the bandwith of timing covert channels, such as those through the CPU caches, between the GUI domain and other AppVMs on a reasonably low level (please note the distinction between a covert channel, which require cooperation of two domains, and a side-channel, which requires just one domain to be malicious - the latter are much more of a theoretical problem, and are of a concern only in some very high security military systems, while the former are easy to implement in practice usually, and present a practical problem in this very scenario).

Another problem, that was immediately pointed out by the ETISS audience, is that an attacker, who compromised the GUI domain, can manipulate the pixmaps that are being processed in the GUI subsystem to present false picture to the user (remember, the attacker should have no way to send them out anywhere). This includes attacks such as button relabeling ("OK" becomes "Cancel" and the other way around), content manipulation ("$1,000,000" instead of "$100", and vice-versa), security labels spoofing ("red"-labeled windows becoming "green"-labeled), and so on. It's an open question how practical these attacks are, at least when we consider automated attacks, as they require ability to extract some semantics from the pixmaps (where is the button, where is the decoration), as well as understanding the user's actions, intentions, and behavior (just automatically relabeling my Friefox label to "green" would be a poor attack, as I would immediately realize something is going wrong). Nevertheless this is a problem, and I'm not sure how this could be solved with the current hardware architecture.

But do we really need untrusted GUI domain? That depends. Currently in Qubes the GUI subsystem is located in dom0, and thus it is fully trusted, and this also means that a potential compromise of the GUI subsystem is considered fatal. We try to make an attack on GUI as hard as possible, and this is the reason we have designed and implemented special, very simple GUI protocol that is exposed to other AppVMs (instead of e.g. using the X protocol or VNC). But if we wanted to add some more "features", such as 3D hardware acceleration for the apps (3D acceleration is already available to the Window Manager in Qubes, but not for the apps), then we would not be able to keep the GUI protocol so simple anymore, and this might result in introducing exploitable fatal bugs. So, in that case it would be great to have untrusted GUI domain, because we would be able to provide feature-rich GUI protocols, with all the OpenGL-ish like things, without worrying that somebody might exploit the GUI backend. We would also not need to worry about putting all the various 3rd party software in the GUI domain, such as KDE, Xorg, and various 3rd party GPU drivers, like e.g. NVIDIA's closed source ones, and that some of it might be malicious.

So, generally, yes, we would like to have untrusted GUI domain - we can live without it, but then we will not have all the fancy 3D acceleration for games, and also need to carefully choose and verify the GUI-related software (which is lots of software).

But perhaps in the next 5 years everybody will have a computer with a few dozens of cores, and also the CPU-to-DRAM bandwidth will be orders of magnitude faster than today, and so there will be no longer a need to offload graphic intensive work to a specialized GPU, because one of our 64 cores will happily do the work? Wouldn't that be a nicer architecture, also for many other reasons (e.g. better utilization of power/circuit real estate)? In that case nobody will need OpenGL, and so there will be no need for a richer GUI protocol than what is already implemented in Qubes...

It's quite exciting to see what will happen (and what we will come up for Qubes) :)

BTW, some people might confuse X server de-privileging efforts, i.e. making the X server run without root privileges, which is being done in some Linux distros and BSDs, with what had been described in this article, namely making the GUI subsystem untrusted. Please note that a de-priviliged X server doesn't really solve any major security problems related to GUI subsystem, as whoever controls ("0wns") the X server (depriviliged or not) can steal or manipulate all the data that this X server is processing/displaying. Apparently there are some reasons why people want to run Xorg as non-root, but in case of typical desktop OSes this provides little security benefit (unless you want to run a few X servers with different user accounts, and on different vt's, which most people would never do anyway).

Qubes, Qubes Pro, and the Future...

The work on Qubes OS has been extremely exciting and also very challenging for us. While most of the work we have been doing so far relates to solving various technical, under-the-hood challenges, the more important goals in the long-term are related more to mitigating the so called "human factor", i.e. making the system not only easy to use, but tolerant to user absentmindedness. This includes e.g. ensuring the user uses a correct AppVM (e.g. do the banking in the "banking" AppVM, and not in the "random web browsing" AppVM, and also not the other way around: don't do random surfing in the "banking" AppVM), and generally making the whole isolation between AppVMs as seamless as possible, but without sacrificing the security at the same time.

This is becoming very important, as the technical level of security in Qubes is already very high, and so the "human factor" might easily become a low hanging fruit for the attacker. (In contrast to other OSes)

But for Qubes to become something more than just an interesting OS for Linux geeks and security enthusiasts, it is also critical to have better application support. Right now Qubes lets users run Linux apps, because each AppVM is Linux-based. But, and let's not be afraid to admit this: Linux sucks when it comes to application support! (Take Open Office as an example - it not only looks like MS Office 97, but is also terribly user-unfriendly, especially their presentation program, the Impress. Why is it so difficult to make it look and behave more like Apple Keynote?)

There is only one way to provide better application support to Qubes: make it support Windows-based, or Mac-based, AppVMs. Just imagine that: being able to run most of your Windows (or Mac) applications, but at the same time benefit from the Qubes strong isolation and seamless integration on one common desktop...

In order to implement support for Windows-based AppVMs (or alternatively Mac-based AppVM) we would need to engage significant resources (5+ very skilled developers, working full time for 1+ year), and so we're currently looking for an investor that would be able to provide funding for such an endeavor. The idea is to create a dedicated spin-off company that would focus entirely on Qubes and Qubes Pro, and in the future will make a profit from selling Qubes Pro licenses. Qubes Pro will become a commercial product, still based on the open source Qubes, but adding support for Windows-based or Mac-based AppVMs. I would be happy to discuss the details and business plan via email with interested potential investors.

Speaking about the future of Qubes: next week I will speak at the European Trusted Infrastructure Summer School, where I will talk about some general stuff like why we need secure desktop systems and why trusted computing might be a way to go, but will also dive a little bit into some new things we plan for Qubes 2.0, such as storage domain and split I/O graphics model. The conference features some very reputable speakers in system-level security field, such as David Grawrock (the father of Intel TXT and TPM), and Loic Duflot (our venerable competitor in the filed of offensive system-level research), so I consider a honour to deliver an opening keynote there (Check the agenda here).

I will have my Qubes laptop with me, of course, so if anybody is interested to see Qubes OS live (including Disposable VMs!), I would be happy to do a quick demo on the spot.

The MS-DOS Security Model

Back in the '80s, there was an operating system called MS-DOS. This ancient OS, some readers might not even remember it today, had a very simple security model: every application had access to all the user files and other applications.

Today, over two decades later, overwhelming majority of people still use the very same security model... Why? Because on any modern, mainstream OS, be that Linux, Mac, or Windows, all the user applications still have full access to all the user's files, and can manipulate all the other user's applications.

Does it mean we haven't progressed anywhere from the MS-DOS age? Not quite. Modern OSes do have various anti-exploitation mechanisms, such as ASLR, NX, guard pages (well, Linux has it since last week at least), and even some more.

But in my opinion there has been too much focus on anti-exploitation, and on bug finding, (and on patching, of course), while almost nothing has been done on the OS architecture level.

Does anybody know why Linux Desktops offer ability to create different user accounts? What a stupid question, I hear you saying - different accounts allow to run some applications isolated from user's other applications! Really? No! The X server, by design, allows any GUI application to mess with all the other GUI applications being displayed by the same X server (on the same desktop). So, what good it is to have a "random_web_browsing" user, if the Firefox run under this user account would still be able to sniff or inject keystrokes to all my other GUI applications, take screenshots of them, etc...?

[Yes, I know, the user accounts allows also to theoretically share a single desktop computer among more than one physical users (also known as: people), but, come on, these days it's that a single person has many computers, and not the other way around.]

One might argue that the progress in the anti-exploitation, and also safe languages, would make it nearly impossible to e.g. exploit a Web browser in the next few years, so there would be no need to have a "random_web_browsing" user in the first place. But, we need isolation not only to protect ourselves when somebody exploits one of our application (e.g. a Web Browser, or a PDF viewer), but also, and perhaps most importantly, to protect from maliciously written applications.

Take summer holiday example: imagine you're a scuba diver - now, being also a decently geeky person, no doubt you will want to have some dive log manager application to store the history of your dives on a computer. There are a dozen of such applications on the web, so all you need to do is to pick one (you know, the one with the nicest screenshots), and... well you need to install it on your laptop now. But, hey, why this little, made by nobody-knows-who, dive application should be given unlimited access to all your personal files, work email, bank account, and god-know-what-else-you-keep-on-your-laptop? Anti-exploitation technology would do exactly nothing to prevent your files in this case.

Aha, it would be so nice if we could just create a user "diving", and run the app under this account. In the future, you could throw in some advanced deco planning application into the same account, still separated from all the other applications.

But, sorry, that would not work, because the X server doesn't provide isolation on the GUI-level. So, again, why should anybody bother creating any additional user accounts on a Linux Desktop?

Windows Vista made a little step forward in this area by introducing integrity levels, that, at least theoretically, were supposed to prevent GUI applications from messing with each other. But they didn't scale well (IIRC there were just 3 or 4 integrity levels available), and it still isn't really clear if Microsoft treats them seriously.

So, why do we have user accounts on Linux Desktops and Macs is beyond me (I guess Mac's X server doesn't implement any GUI-level isolation either - if I'm wrong, please point me out to the appropriate reference)?

And we haven't even touched the problems that might arise from the attacker exploiting a bug in the (over-complex) GUI server/API, or in the (big fat) kernel (with hundreds of drivers). In order for those attacks to become really interesting (like the Rafal's attack we presented yesterday), the user would have to already be using e.g. different X servers (and switch between them using Ctrl-Shift-Fn), or some sandboxing mechanisms, such as SELinux sandbox, or, in case of Vista, a scheme similar to this one.

Skeletons Hidden in the Linux Closet: r00ting your Linux Desktop for Fun and Profit

A couple of months ago, while working on Qubes GUI virtualization, Rafal has come up with an interesting privilege escalation attack on Linux (a user-to-root escalation), that exploits a bug in... well, actually it doesn't exploit any concrete bug, which makes it so much more interesting.

The attack allows a (unpriviliged) user process that has access to the X server (so, any GUI application) to unconditionally escalate to root (but again, it doesn't take advantage of any bug in the X server!). In other words: any GUI application (think e.g. sandboxed PDF viewer), if compromised (e.g. via malicious PDF document) can bypass all the Linux fancy security mechanisms, and escalate to root, and compromise the whole system. The attack allows even to escape from the SELinux's "sandbox -X" jail. To make it worse, the attack has been possible for at least several years, most likely since the introduction of kernel 2.6.

You can find the details of the attack, as well as the discussion of possible solutions, including the one that has eventually been implemented, in the Rafal's paper.

One important aspect the attack demonstrates, is how difficult it is to bring security to a desktop platform, where one of the biggest challenges is to let applications talk to the GUI layer (e.g. X server in case of Linux), which usually involves a very fat GUI protocol (think X protocol, or Win32 GUI API) and a very complex GUI server, but at the same time keep things secure. This was one of the key priories for us when designing Qubes OS architecture. (So, we believe Qubes is much more secure than other sandboxing mechanisms, such as BSD jails, or SELinux-based sandboxes, because it not only eliminates kernel-level exploits, but also dramatically slims down GUI-level attacks).

The kernel-level "patch" has been implemented last week by Linus Torvalds, and pushed upstream into recent stable kernels. RedHat has also released an advisory for this attack, where they rated its severity as "high".

ps. Congrats to Brad Spengler for some good guessing :)