Thursday, March 21, 2013

Introducing Qubes Odyssey Framework

Qubes OS is becoming more and more advanced, polished, and user friendly OS.
But Qubes OS, even as advanced as it is now, surely have its limitations. Limitations, that for some users might be difficult to accept, and might discourage them from even trying out the OS. One such limitation is lack of 3D graphics support for applications running in AppVMs. Another one is still-far-from-ideal hardware compatibility – a somehow inherent problem for most (all?) Linux-based systems.
There is also one more “limitation” of Qubes OS, particularly difficult to overcome... Namely that it is a standalone Operating System, not an application that could be installed inside the user's existing OS. While installing a new application that increases system's security is a no-brianer for most people, switching to a new, exotic OS, is quite a different story...
Before I discuss how we plan to address those limitations, let's first make a quick digression about what Qubes really is, as many people often get that wrong...
What Qubes IS, and what Qubes IS NOT?
Qubes surely is not Xen! Qubes only uses Xen to create isolated containers – security domains (or zones). Qubes also is not a Linux distribution! Sure, we currently use Fedora 18 as the default template for AppVMs, but at the same time we also support Windows VMs. And while we also use Linux as GUI and admin domain, we could really use something different – e.g. Windows as GUI domain.
So, what is Qubes then? Qubes (note how I've suddenly dropped the OS suffix) is several things:
  • The way how to configure, harden, and use the VMM (e.g. Xen) to create isolated security domains, and to minimize overall system TCB.
  • Secure GUI virtualization that provides strong gui isolation, while at the same time, provides also seamless integration of all applications running in different VMs onto one common desktop. Plus a customized GUI environment, including trusted Window Manager that provides unspoofable decorations for the applications' windows.
  • Secure inter-domain communication and services infrastructure with centrally enforced policy engine. Plus some “core” services built on top of this, such as secure file exchange between domains.
  • Various additional services, or “addons”, built on top of Qubes infrastructure, such as Disposable VMs, Split GPG, TorVM, Trusted PDF converter, etc. These are just few examples, as basically the sky is the limit here.
  • Various additional customizations to all the guest OSes that run in various domains: GUI, Admin, ServiceVMs, and AppVMs.
Introducing Qubes HAL: Hypervisor Abstraction Layer
Because Qubes is a bunch of technologies and approaches that are mostly independent from the underlying hypervisor, as discussed above, it's quite natural to consider if we could easily build an abstraction layer to allow the use of different VMMs with Qubes, instead of just Xen? Turns out this is not as difficult as we originally thought, and this is exactly the direction we're taking right now with Qubes Odyssey! To make this possible we're going to use the libvirt project.
So, we might imagine Qubes that is based on Hyper-V or even Virtual Box or VMWare Workstation. In the case of the last two Qubes would no longer be a standalone OS, but rather an “application” that one installs on top of an existing OS, such as Windows. The obvious advantage we're gaining here is improved hardware compatibility, and ease of deployment.
And we can go even further and ask: why not use Windows Native Isolation, i.e. mechanisms such as user account separation, process isolation, and ACLs, to implement domain isolation? In other words why not use Windows OS as a kind of “VMM”? This would further dramatically improve then lightness of the system...
Of course the price we pay for all this is progressively degraded security, as e.g. Virtual Box cannot be a match to Xen in terms of security, both architecturally and implementation-wise, and not to mention the quality of isolation provided by the Windows kernel, which is even less.

But on the other hand, it's still better than using “just Windows” which offers essentially only one “zone”, so no domain isolation at all! And if we can get, with minimal effort, most of our Qubes code to work with all those various isolation providers then... why not?
Being able to seamlessly switch between various hypervisors is only part of the story, of course. The remaining part is the support for different OSes used for various Qubes domains. Currently we use Linux, specifically Fedora 18, in our GUI & Admin domain, but there is no fundamental reason why we couldn't use Windows there instead. We discuss this more in-depth in one of the paragraphs below.
The diagram below tries to illustrate the trade-offs between hardware compatibility and ease of deployment vs. security when using different isolation backends with Qubes. Some variants might also offer additional benefits, such as “super-lightness” in terms of CPU and memory resources required, as is the case with Windows Native Isolation.

Some example configurations
Let's now discuss two extreme variants of Qubes – one based on the baremetal Xen hypervisor and the other one based on Windows Native Isolation, so a variant from the opposite end of the spectrum (as shown on the illustration above).
The diagram below shows a configuration that uses a decent baremetal hypervisor, such as Xen, with abilities to securely assign devices to untrusted service domains (NetVM, UsbVM). So, this is very similar to the current Qubes OS.

Additionally we see separate GUI and Admin domains: the GUI domain might perhaps be based on Windows, to provide users with a familiar UI, while the Admin domain, tasked with domain management and policy enforcement, might be based on some minimal Linux distribution.
In the current Qubes OS there is no distinction between a GUI and an Admin domain -- both are hosted within one domain called “dom0”. But in some cases there are benefits of separating the GUI domain from the Admin domain. In a corporate scenario, for example, the Admin domain might be accessible only to the IT department and not to the end user. This way the user would have no way of modifying system-wide policies, and e.g. allowing their “work” domain to suddenly talk to the wild open Internet, or to copy work project files from “work” to “personal” domains (save for the exotic, low-bandwidth covert channels, such as through CPU cache).
The ability to deprivilege networking and USB stacks by assigning corresponding devices (NICs, and USB controllers) to untrusted, or semi-trused, domains provides great security benefits. This automatically prevents various attacks against the bugs in WiFi stacks or USB stacks.
What is not seen on the diagram, but what is typical for baremetal hypervisors is that they are usually much smaller than hosted hypervisors, implementing less services, and delegating most tasks, such as the infamous I/O emulation to (often) unprivileged VMs.
Let's now look at the other extreme example of using Qubes – the diagram below shows an architecture of a “Qubized” Windows system that uses either a hosted VMM, such as Virtual Box or VMWare Workstation, or even the previously mentioned Windows Native Isolation mechanisms, as an isolation provider for domains.

Of course this architecture lacks many benefits discussed above, such as untrusted domains for networking and USB stacks, small hypervisors, etc. But it still can be used to implement multiple security domains, at a much lower “price”: better hardware compatibility, easier deployment, and in case of Windows Native Isolation – excellent performance.
And it really can be made reasonable, although it might require more effort  than it might seem at first sight. Take Windows Native Isolation – of course just creating different user accounts to represent different domains is not enough, because Windows still doesn't implement true GUI-level isolation. Nor network isolation. So, there is a challenge to do it right, and “right” in this case means to make the isolation as good as the Windows kernel can isolate processes from different users from each other.
Sure, a single kernel exploit destroys this all, but it's still better than “one application can (legally) read all my files” policy that 99% of all desktop OSes out there essentially implement today.
Now, probably the best thing with all this is that once we implement a product based on, say, Qubes for Windows, together with various cool “addons” that will take advantage of the Qubes services infrastructure, and which shall be product-specific, it should then be really easy to upgrade to another VMM, say Hyper-V to boost security. And the users shall not even notice a change in the UI, save for the performance degradation perhaps (well, clearly automatic creation of VMs to handle various users tasks would be more costly on Hyper-V than with Windows Native Isolation, where “VMs” are just... processes).
Qubes building blocks – implementation details
Let's have a look now at the repository layout for the latest Qubes OS sources – every name listed below represents a separate code repository that corresponds to a logical module, or a building block of a Qubes system:

Because current Qubes R2 still doesn't use HAL layer to support different hypervisors, it can currently be used with only one hypervisor, namely Xen, whose code is provided by the vmm-xen repository (in an ideal world we would be just using vanilla Xen instead of building our own from sources, but in reality we like the ability to build it ourselves, slightly modifying some things).
Once we move towards the Qubes Odyssey architecture (essentially by replacing the hardcoded calls to Xen's management stack, in the core-admin module, with libvirt calls), we could then easily switch Xen for other hypervisors, such as Hyper-V or Virtual Box. In case of Hyper-V we would not have access to the sources of the VMM, of course, so would just be using the stock binaries, although we still might want to maintain the vmm-hyperv repository that could contain various hardening scripts and configuration files for this VMM. Or might not. Also, chances are high that we would be just able to use the stock libvirt drivers for Hyper-V or Virtual Box, so no need for creating core-libvirt-hyperv or core-libvirt-virtualbox backends.
What we will need to provide, is our custom inter-domain communication library for each hypervisor supported. This means we will need to write core-vchan-hyperv or core-vchan-virtualbox. Most (all?) VMMs do provide some kind of API for inter-VM communication (or at least VM-host communication), so the main task of such component is to wrap the VMM-custom mechanism with Qubes-standarized API for vchan (and this standardization is one thing we're currently working on). All in all, in most cases this will be a simple task.
If we, on the other hand, wanted to support an “exotic” VMM, such as the previously mentioned Windows Native Isolation, which is not really a true VMM, then we will need to write our own libvirt backend to support is:
... as well as the corresponding vchan module (which should be especially trivial to write in this case):
Additionally, if we're building a system where the Admin domain is not based on Linux, which would likely be the case if we used Hyper-V, or Virtual Box for Windows, or, especially, Windows Native Isolation, then we should also provide core-admin-windows module, that, among other things, should provide Qubes qrexec implementation, something that is highly OS-dependent.
As can be seen above, we currently only have core-admin-linux, which is understandable as we currently use Linux in Dom0. But the good news is that we only need to write core-admin-XXX once for each OS that is to be supported as an Admin domain, as this code should not be depend on the actual VMM used (thanks to our smart HAL).
Similarly, we also need to assure that our gui-daemon can run on the OS that is to be used as a GUI domain (again, in most cases GUI domain would be the same as Admin domain, but not always). Here the situation is generally much easier because “with just a few #ifdefs” our current GUI daemon should compile and run on most OSes, from Linux/Xorg to Windows and Macs (which is the reason we only have one gui-daemon repository, instead of several gui-daemon-XXX).
Finally we should provide some code that will gather all the components needed for our specific product and package this all into either an installable ISO, if Qubes is to be a standalone OS, like current Qubes, or into an executable, in case Qubes is to be an “application”. The installer, depending on the product, might do some cool things, such as e.g. take current user system and automatically move it into one of Qubes domains.
To summary, these would be the components needed to build “Qubes for Windows” product:

Additionally we will likely need a few qubes-app-* modules that would implement some "addons", such as perhaps automatic links and documents opening in specific VMs, e.g.:
Here, again, the sky's the limit and this is specifically the area where each vendor can go to great lenghts and build killer apps using our Qubes framework.
Now, if we wanted to create "Qubes for Hyper-V" we would need the following components:

Here, as an example, I also left optional core-agent-linux and gui-agent-linux components (the same that are to be used with Xen-based Qubes OS) to allow support for also Linux-based VMs – if we can get those “for free”, then why not!
It should be striking how many of those components are the same in both of those two cases – essentially the only differences are made by the use of different vmm-* components and, of course, the different installer
It should be also clear now how this framework now enables seamless upgrades from one product (say Qubes for Windows) to another (say Qubes for Hyper-V).
Our business model assumes working with vendors, as opposed to end users, and licensing to them various code modules needed to create products based on Qubes.
All the code that comprises the base foundation needed for creation of any Qubes variant (so core-admin, gui-common, gui-daemon, qubes-builder and qubes-manager) will be kept open source, GPL specifically. Additionally all the code needed for building of Xen-based Qubes OS with Linux-based AppVMs and Linux-based GUI and Admin domains, will continue to be available as open source. This is to ensure Qubes OS R3 that will be based on this framework, can remain fully open source (GPL).
Additionally we plan to double-license this core open source code to vendors who would like to use it in proprietary products and who would not like to be forced, by the GPL license, to share the (modified) sources.
All the other modules, especially those developed to support other VMMs (Hyper-V, Virtual Box, Windows Native Isolation), as well as those to support Windows OS (gui-agent-windows, core-agent-windows, core-admin-windows, etc) will most likely be proprietary and will be available only to vendors who decide to work with us and buy a license.
So, if you want to develop an open source product that uses Qubes framework, then you can freely do that as all the required core components for this will be open sourced. But if you would like to make a proprietary product, then you should buy a license from us. I think this is a pretty fair deal.
Current status and roadmap
We're currently working on two fronts: one is rewriting current Qubes code to support Qubes HAL, while the other one is adding a backend for Windows Native Isolation (which also involves doing things such as GUI isolation right on Windows).
We believe that by implementing two such extreme backends: Xen and Windows Native Isolation we can best show the flexibility of the framework (plus our customer is especially interested in this backend;)
We should be able to publish some code, i.e. the framework together with early Qubes OS R3 that will be based on it, sometime in fall or maybe earlier.
We obviously are determined to further develop Xen-based Qubes OS, because we believe this is the most practically secure OS available today, and we believe such OS should be open source.
Qubes R2 will still be based on the Xen-hardcoded code, because it's close to the final release and we don't want to introduce such drastic changes at this stage. The only thing that Qubes R2 will get in common with Qubes Odyssey is this new source code layout as presented above (but still with hardcoded xl calls and xen-vchan).
So, this is all really exciting and a big thing, let's see if we can change the industry with this :)
Oh, and BTW, some readers might be wondering why the framework was codenamed “Odyssey” -- this is obviously because of the “HAL” which plays a central role here, and which, of course, also brings to mind the famous Kubrick's movie.


Anonymous said...

Sounds awesome!

Anonymous said...

Joanna, what you and your team are working on is really revolutionary.

Most cyber attacks could be stopped in the US government would use this technology in all of its departments.

Andrew said...

I am amazed at what you have created. We are looking at how we can integrate your work in our commercial application. I have been watching since R1 and I am really excited by your progress. Have you thought about a BSD release With your duel licensing approach? Rather then Linux? This will allow more flexibility for commercial firms. (Qubes OS BSD)

Thomas Hoberg said...

I can only welcome your new approach to generalize the Qubes concept and thus have it appeal to a broader user and OS base. And I certainly wish you the very best in your attempt to attract commercial vendors: I guess doing all this good work on consultant fees only is too much living on the edge.

I see strong parallels to my two other favorite open source projects, OpenVZ and openQRM as far as the commercialization is concerned.

But just like with the OpenVZ version for Windows I'm a bit afraid there is only two options in that camp:
a) your work will be largely ignored when it should be mainstream

b) Microsoft will make you an offer you can't refuse

Meanwhile I just hope some patent troll can't steal or block your wonderful work.

Unfortunately a technically well designed and from an end user perspective attractive product rarely is a good sell or downright shot down by vendors who fear losing a bit of grip on their customers.

Thomas Hoberg said...

Here are my thoughts from a corporate perspective:

I design technical infrastructure in a payment industry company so security is very dear to me also professionally.

My first impression was that Qubes was first and formost a proof of concept and largely driven by the use case of being your personal productivity workstation driver.

Now that you are reaching out into the commercial space, there are currently three primary use cases in our company (and most likely many others):

1. Server
2. Worktation/Fat Client
3. VDI/Server based fat client + display-only thin client

And I don't see Qubes yet playing very well in any of these use cases which seem best in terms of revenue stream and security needs.

I could certainly see us paying for something ready to use in all of the above, because our company is a very likely target for the criminal industry.

Our servers are of course GUI-less and we'd need a mix of OpenVZ we currently use to consolidate virtualized server loads at 1:50 or better as well as Qubes to isolate the I/O and management stacks. I/O overhead and latencies here would need to be optimized as much as possible and automated deployment via kickstart/PXE boot like mechanisms is a must.

The run-time aspect of workstation is probably best covered today, but Qubes would need some unique deployment advantages to appeal to workstation admins. Ironically our vPro desktop was almost ideal about two years ago, when we depoyed almost exclusively Q based chipsets, but these days cost pressures is losing the potential window of opportunity to H based chipsets which have VT-d fused off.
AMD APU parts would have been a great alternative because they always come full featured in terms of virtualization support, but with AMD fighting for survival and dropping IOMMU2 software support they seem a very unsafe bet: There is also no APU based corporate client hardware out there.

For me VDI servers are a next logical place to deploy the Qubes concept, but what you have currently doesn't seem to integrate easily with terminal servers or server based VMs. Of course the major players in that market like Citrix, VMware and I guess Microsoft take no prisoners and won't easily tolerate a cross vendor Qubes for VDI and TS.

Like so often before I can see all the differnt parts out there fitting together in an ideal fashion for potential clients, but very little chance for this actually happening without major gotchas.

Hope you prove me wrong and are successful in all those use cases!!

test said...

Do you have plan for porting QuebesOS on mobile platform? I planning to study and develop the mobile platform base on Xen-ARM (just for personal open source project, nonprofit). What do you think about QuebesOS mobile? Can I ask about your opinion as a developer? For example, in a point of view performance, utility, or any kind of things.

Joanna Rutkowska said...

@Yeongdeok: at this moment we don't such plans, but with Qubes Odyssey Framework this is certainly possible. Questions about performance should really be asked to the specific VMM developers (e.g. Xen Arm Project).

Anonymous said...

First, let me say - great job with everything so far!

Another interesting thing to look into might be using hardware machines instead of VMs - maybe even with different CPU architectures.

Something like a physical network of Raspberry PI machines, controlled by an Admin and/or GUI which runs on a high-end laptop or PC. The setup can have firewall-machine and net-machine roles, just like in the regular Qubes OS.

Or a future single-board system that has multiple chips as separate systems.

Or a mix of Xen VT-d-enabled VMs and hardware machines.

The end result can look the same as running Qubes OS on a single system with VT-d.

Joanna Rutkowska said...


Surely doable, but the networking would have to be handled smartly, so that the TCP/IP and WiFi/BT stacks not be part of the TCB. Doable, I think.

Timo Ollech said...

What about KVM? It is supported by libvirt, and if one wants to stick with linux entirely it would be first choice - will Qubes Oddyssey support it as well?

Anonymous said...

speaking of Qubes Odyssey: Wouldn´t you converge to products like eg Bromium (which you compared to cubes here And your mentioned attack vectors like usb attacks and hooking into the GUI subsystem would also apply?
regards, joerg

Joanna Rutkowska said...


As has been mentioned multiple times in the article, the purpose of Odyssey is to allow a whole spectrum of products, balancing security vs. hardware compatibility/ease of deployment. Even though the variants from the "low-security" end of the spectrum share some of the weakness of other products, the big advantage is that one can almost seamlessly upgrade to the more secure variants from the other end of the spectrum.

Qubes Odyssey is all about giving the customer freedom in making the decision how they want to balance security vs. hardware compatibility and ease of deployment. Qubes Odyssey provides infrastructure to build very secure client systems. Whether people would like to use its full potential is up to the customer...

As for us, ITL, we will, of course, continue the development of the open source Qubes OS, also based on Odyssey Framework as well as on Xen and Linux, which we believe is the most practically secure solution for desktop computing.

Anonymous said...

I have just installed beta 3. I had beta 2 installed but didn't have much time to fool with it. I am clueless when it comes to the level of knowledge you all are at. I don't understand much of the discussion but I do understand hardware fairly well. But what I do know is that I was able to install Qubes with no problem and can use the default appVMs as designed. I have, in beta 2, also created some appVMs. While I may know a little more than the average user I don't see installation and use as being any more difficult than Windows 8 (8 disks (OEM disks), 4 hours and 3 tries before it installed properly, 1 disk, 40 minutes, 1 try to install Qubes). It may seem more difficult than it is when a rook like me looks at the blog and comments but you are talking about things most users will never concern themselves with. With some education and good plain language guidance most of the commercial market could easily be using Qubes. I like Qubes a lot, I would like it to be my every day, go-to OS. Right now that isn't possible (need some Win only hardware) but with the many improvements in each update, Qubes will soon be standing alone on my system. Great OS! Great job! Keep up the good work!

Anonymous said...

Just curious, where does kvm-qemu sit in all this, I see no mention of it, do you see it a liability security-wise, worse than windows???

Will be watching.

Anonymous said...

Hello Joanna,

Incredible work, thank you. Surely and sorely needed in today's computing environment.

Regarding the Windows process level implementation option, there is a commercial product I use that does something similar to this called WinPatrol, maybe worth having a look at for ideas.

Anonymous said...

Joanna and the team,
Everyone says "thank you" so I will do the opposite ;-)
Go to hell, all (cyber/IT)criminals will loose their jobs and then what? Many security officers will loose their jobs. And then what? ;-)
PS. Good job. I am observing this project quite for some time and I am more and more impressed.