Qubes Beta 3!

-->

A new ISO with the just released Qubes Beta 3 is now available for download here.

Beta 3 fixes lots of annoying problems discovered in Beta 2 and earlier releases, and also implements a bunch of useful feature:

This includes the qvm-block tool and infrastructure for easy attachment of block devices to any AppVM, no matter which system VM is handling the actual USB controller. So, this allows to have untrusted USB domain(s), almost seamlessly integrated in the desktop system. One can consider to use it in order to prevent various USB attacks. The next release (the 1.0) will bring this feature to the Qubes GUI manager as well, making it easy to use for non-command-line users too.

Also, we have now introduced fully automatic Qubes build system, that allows to build all the Qubes packages, and also create the installation ISO, with just one command. More information on this system and on how to use it can be found in the wiki.

We have also updated to Fedora 15-based template as a default. Unfortunately F16-based template would require too much work to get all the Gnome 3 stuff working correctly. (The challenge here, is that we don't run a normal Windows and Desktop manager in every domain, in order to make the VMs light weight, and so we need to sometimes work around various problems this causes...).

Finally, we have added two new Qubes-specific applications:

• A plugin for Thunderbird (it is automatically installed in the template), that allows for one click opening of attachments in Disposable VMs, as well as one-click saving of the attachment to select VM.

• And something we call “split GPG”, that I will describe in a separate article later.

Those Qubes-specific applications are based on our Qubes RPC, introduced in Beta 2.

This is likely the last release before the “final 1.0”, that is scheduled to follow soon (TM). The only major work for 1.0 is GUI manager improvements to expose most of the Qubes functionality via clickable GUI, and command line tools cleanup and documentation. Plus testing and bugfixing :)

And then, the next thing we will be working on will be support for HVM domains, e.g. Windows. This work is starting actually just about now, but code will be released only after Qubes 1.0.

Thoughts on DeepSafe

Several people asked me recently what I though about DeepSafe. So, below I present my opinion...

First, for any AV system (or Host IPS, or Personal Firewall, etc) to work effectively, there are three problems that must be addressed:

1. How to protect the AV agent (code and data) from tampering (from the rest of the OS)?
2. How can the AV agent get reliable access to (sensitive pieces of) the system memory and registers, and/or provide reliable memory protection for the (sensitive pieces of) the OS.
3. What are those "sensitive pieces of” memory that should be monitored or protected?

From reading various PR materials, it seems like the #1 above is the primary differentiation factor for DeepSafe (DS). So, let's consider this problem in the context of e.g. a Windows OS. In order to protect its code and data, DS uses, as it is heavily advertised, Intel VT-x virtualization technology. Now, that sounds really secure -- after all what can be more secure than a hardware virtualization, right? ;)

But VT-x (including EPT) is only about CPU virtualization, which in our case translates to protecting the DS memory and registers from CPU-originating accesses. But, as every regular to this blog knows, there is also another method of accessing memory on any PC system, and this is through DMA transactions from devices. The OS (so also the kernel malware) is free to program one of the many devices in the system to issue DMA reads or writes to any physical memory it wants...

Now, in order to protect some portion of the system memory (DRAM, cache) against DMA accesses, we have the Intel VT-d technology... So, one would think that DS must be also using VT-d in order to protect itself.

Very well, let's assume then that the DeepSafe is not a total ripoff, and that it implements also VT-d protection for its agent, although I haven't found this mentioned in any of the public papers or press materials I found on the web...

This, however, would be a bit complex to do correctly, because the OS (so, also the kernel malware) still has a full control over the chipset (MCH), which is the entity... that controls the VT-d.

Now, in order to prevent the OS (or the kernel malware) from playing with the chipset for fun and profit, and e.g. disabling VT-d protection, DS would have to virtualize the chipset.

If you look at some consumer VMMs, such as VMware or Xen/Qemu, you would see that they all virtualize the chipset for their guests (of course), but that the chipset they provide this way is some kind of an ancient Pentium MCH. I don't think any of the consumers would be especially happy if they found out that after installing DS on their brand new 2012 laptop, Windows suddenly see a Pentium-era chipset... And this is not without a reason – chipsets, specifically MCHs, are one of the most complex devices, perhaps only beaten by GPUs in this category. There are virtually hundreds of configuration registers exposed by the chipset, some of them control the VT-d, some other control system memory maps and permissions, PCIe configuration, and many other things that I even have no idea about, and this all makes virtualizing the chipset a very challenging task.

So, it's either that McAfee and Intel found some interesting way of how to securely virtualize the chipset while preserving all of its (very rich) functionality, or that they... don't bother with VT-d protection and chipset virtualization at all, assuming that even without VT-d, DeepSafe is good enough and “rises the bar” anyway (sarcasm intended).

(Can somebody from McAfee or Intel confirm in the comments below what does DP really do?)

Anyway, let's assume they do have VT-d protection and they do virtualize the chipset somehow...

Now, we're moving on to the #2 point from the list of tasks above -- about the reliable

memory access or reliable protection.

So, let say that the DS agent decided that some part of the system memory, e.g. the IDT table, is sensitive and should be monitored/protected. So it sets up EPT traps to trigger an VT-x/EPT intercept on any access to that memory (or IDT base register), in order to find kernel malware that tried to modify IDT. That sounds really nice, but what if the malware uses DMA to modify IDT? DS would not be able to catch such access! (So far we considered the, hypothetical, use of VT-d only to protect the DS agent code).

One might think that DS is programming VT-d to sandbox each and every device in the system (so including GPU, USB controllers, NICs, SATA, etc) so they never be allowed to touch any of those sensitive parts of the system, such as IDT. Let's assume they do it this way...

And here we've arrived to the last point from the list at the beginning: which of the system memory constitutes those "sensitive pieces" that should be protected/monitored? IDT? Sure. What about all the code sections of the all the kernel modules? Yes. Are we fine now? Well, no, because the malware can hook some pointers other than the well known IDT. Some public NDIS data structure? Ok, we can add those to the protected areas. But, what about some undocumented NDIS structures? And this is just NDIS subsystem, one of the many subsystems in the Windows kernel... When we think about it, it should be intuitively obvious that in a general purpose Operating System like Windows, it is not possible (at least for 3^rd party) to make a satisfactory list of all the sensitive pieces of memory that should be monitored/protected, in order to detect all the system compromises.

Greg Hoglund, Jamie Butler, Alex Tereshkin, and myself, have been researching this area actively in the early years of this millennium. In addition to the Alex's paper linked above, you might also check out one of my last slides from this period.

I don't think anything has changed since that time. It was also the reason why I gave up on writing Windows compromise detectors, or forensic tools, and moved on to researching other ways to secure OSes, which finally gave birth to Qubes OS, many years later.

So, back to DS -- in order to provide a somehow satisfactory protection level for your general purpose OS, such as Windows, it would need to:

1. Use VT-d to protect its own memory,

2. Virtualize the chipset, at least some (sensitive) parts of it,

3. Program VT-d permissions for each device to exclude all the sensitive areas in the system that should be protected, and also protect one device from DMAing into/from another device memory (e.g. NIC stealing GPU framebuffer, or inserting instructions to the GPU instruction buffer, or keystrokes to USB controller buffer). Ideally, this could be done by programming VT-d to grant each device only access to its own DMA buffer, but as far as I know, this would be very hard to implement, if not impossible for a 3rd party, on a Windows OS (in contrast to Linux, which mostly support this model). Please correct me, if the recent Windows version allows for such use of VT-d.

4. Finally, and the most hard thing to solve, how to define all the "sensitive pieces of memory" in the system that should be protected and/or monitored? Although this is a somehow more generic problem, not specific to DS, but applying to any A/V, HIPS, or forensic tool.

So, is DeepSafe another piece of crap not worth any special attention, or has McAfee and Intel came up with some novel methods, e.g. for chipset virtualization, and other problems? Unless I see some technical info to backup the latter, I would have to assume, unfortunately, the former. But I would like to be mistaken – after all DeepSafe seems to be just a new incarnation of my Bluepill ;)

Trusted Execution In Untrusted Cloud

Wouldn't it be nice if we could actually own our data and programs in the cloud? By “owning” here I mean to have control over their confidentiality and integrity. When it comes to confidentiality and integrity for the data, it's not much of a rocket since, as the classic crypto (and secure client systems) is all that we need. I have already wrote about it in an earlier post.

But it would also be nice, if we could somehow get the same confidentiality and integrity assurance for our programs that we upload for the execution in the cloud...

For example, a company might want take their database application, that deal with all sorts of corporate critical sensitive data, and then upload and safely run this application on e.g. Amazon's EC2, or maybe even to some China-based EC2-clone. Currently there is really nothing that could stop the provider, who has a full control over the kernel or the hypervisor under which our application (or our VM) executes, from reading the contents of our process' memory and stealing the secrets from there. This is all easy stuff to do from the technical point of view, and this is also not just my own paranoia...

Plus, there are the usual concerns, such as: is the infrastructure of the cloud provider really that safe and secure, as it is advertised? How do we know nobody found an exploitable bug in the hypervisor and was not able to compromise other customer's VMs from within the attacker-hired VM? Perhaps the same question applies if we didn't decided to outsource the apps to a 3^rd party cloud, but in case of a 3^rd party clouds we really don't know about what measures have been applied. E.g. does the physical server on which my VMs are hosted also used to host some foreign customers? From China maybe? You get the point.

Sometimes all we really need is just integrity, e.g. if we wanted to host an open source code revision system, e.g. a git repository or a file server. Remember the kernel.org incident? On a side note, I find the Jonathan Corbet's self-comforting remarks on how there was really nothing to worry about, to be strikingly naive... I could easily think of a few examples of how the attacker(s) could have exploited this incident, so that Linus & co. would never (not soon) find out. But that's another story...

But, how can one protect a running process, or a VM, from a potentially compromised OS, or a hypervisor/VMM?

To some extent, at least theoretically, Intel Trusted Execution Technology (TXT), could be used to implement such protection. Intel TXT can attest to a remote entity, in that case this would be the cloud customer, about the hash of the hypervisor (or kernel) that has been loaded on the platform. This means it should be possible for the user to know that the cloud provider uses the unmodified Xen 4.1.1 binary as the hypervisor and not some modified version, with a built-in FBI backdoor for memory inspection. Ok, it's a poor example, because the Xen architecture (and any other commercially used VMM) allow the administrator who controls Dom0 (or equivalent) to essentially inspect and modify all the memory in the system, also that belonging to other VMs, and no special backdoors in the hypervisor are needed for this.

But let's assume hypothetically that Xen 5.0 would change that architecture, and so the Dom0 would not be able to access any other VM's memory anymore. Additionally, if we also assumed that the Xen hypervisor was secure, so that it was not possible to exploit any flaw in the hypervior, then we should be fine. Of course, assuming also there were also no flaws in the TXT implementation, and that the SMM was properly sandboxed, or that we trusted (some parts of) the BIOS (these are really complex problems to solve in practice, but I know there is some work going on in this area, so there is some hope).

Such a TXT-bases solution, although a step forward, still requires us to trust the cloud provider a bit... First, TXT doesn't protect against bus-level physical attacks – think of an attacker who replaces the DRAM dies with some kind of DRAM emulator – a device that looks like DRAM to the host, but on the other end allows full inspection/modification of its contents (well, ok, this is still a bit tricky, because of the lack of synchronization, but doable).

Additionally for Remote Attestation to make any sense, we must somehow know that we “talk to” a real TPM, and not to some software-emulated TPM. The idea here is that only a “real” TPM would have access to a private key, called Endorsement Key, used for signing during Remote Attestation procedure (or used during the generation of the AIK key, that can be used alternatively for Remote Attestation). But then again who generates (and so: owns) the private endorsement keys? Well, the TPM manufacturer, that can be... some Asian company that we not necessarily want to trust that much...

Now we see it would really be advantageous for customers, if Intel decided to return to the practice of implementing TPM internally inside the chipset, as they did in the past for their Series 4 chipsets (e.g. Q45). This would also protect against the LCP bus-level attacks against TPM (although somebody told me recently that TPM in current systems cannot be so easily attacked from LCP bus, because of some authentication protocol being used there – I really don't know, as physical attacks have not been the area we ever looked at extensively; any comments on that?).

But then again, the problem of DRAM content sniffing always remains, although I would consider this to be a complex and expensive attack. So, it seems to me that most governments would be able to bypass such TXT-ensured guarantees in order to “tap” the user's programs executing in the cloud provides that operate within their jurisdictions. But at least this could stop malicious companies from staring up fake cloud services with an intent to easily harvest some sensitive data from unsuspecting users.

It seems that the only way to solve the above problem of DRAM sniffing attacks is to add some protection at the processor level. We can imagine two solutions that processor vendors could implement:

First, they could opt for adding an in-processor hardware mechanism for encrypting all the data that leave the processor, to ensure that everything the is kept in the DRAM is encrypted (and, of course, also integrity-protected), with some private key that never leave the processor. This could be seen as an  extension to the Intel TXT.

This would mean, however, we still needed to relay on: 1) the hypervisor to not contain bugs, 2) the whole VMM architecture to properly protect VM's memory, specifically against the Dom0, 3) Intel TXT to not be buggy either, 4) SMM being properly sandboxed, or alternatively to trust (some parts of) the BIOS and SMI handler, 5) TPM's EK key to be non-compromised and verifiable as genuine, and 6) TPM bus attacks made impossible (those two could be achieved by moving the TPM back onto the chipset, as mentioned above), and finally, 7) on the encryption key used by the processor for data encryption to be safely kept in the processor.

That's still quite a lot of things to trust, and it requires quite a lot of work to make it practically really secure...

The other option is a bit more crazy, but also more powerful. The idea is that the processor might allow to create untrusted supervisors (or hypervisors). Bringing this down to x86 nomenclature, it would mean that kernel mode (or VT-x root) code cannot sniff or inject code into (crypto-protected) memory of the usermode processes (or VT-x guests). This idea is not as crazy as you might think, and there has even been some academic work done in this area. Of course, there are many catches here, as this would require specifically written and designed applications. And if we ever considered to use this technology also for client systems (how nice it would be if we could just get rid of some 200-300 kLOC of the Xen hypervisor from the TCB in Qubes OS!), the challenges are even bigger, mostly relating to safe and secure trusted output (screen) and, especially, input (keyboard, mouse).

If this worked out, then we would need to trust just one element: the processor. But we need to trust it anyway. Of course, we also need to trust some software stack, e.g. the compilers we use at home to build our application, and the libraries it uses, but that's somehow an unrelated issue. What is important is that we now would be able to choose that (important) software stack ourselves, and don't care about all the other software used by the cloud provider.

As I wrote above, the processor is this final element we always need to rust. In practice this comes down to also trusting the US government :) But we might imagine users consciously choosing e.g. China-based, or Russia-based cloud providers and require (cryptographically) to run their hosted programs on US-made processors. I guess this could provide reasonable politically-based safety. And there is also ARM, with its licensable processor cores, where, I can imagine, the licensee (e.g. an EU state) would be able to put their own private key, not known to any other government (here I assume the licensee also audits the processor RTL for any signs of backdoors). I'm not sure if it would be possible to hide such a private key from a foundry in Hong Kong, or somewhere, but luckily there are also some foundries within the EU.

In any case, it seems like we could make our cloud computing orders of magnitude safer and more secure than what is now. Let's see whether the industry will follow this path...

Exploring new lands on Intel CPUs (SINIT code execution hijacking)

Today we're releasing a new paper where we describe exploiting a bug in Intel SINIT authenticated code module that allows for arbitrary code execution in what we call an “SINIT mode”. So, to the already pretty-well explored “lands” on Intel processors, that include ring 3 (usermode), ring 0 (kernelmode), ring “-1” (VT-x root), and ring “-2” (SMM), we're now adding a new “island”, the SINIT mode, a previously unexplored territory inhabited so far only by the Intel-blessed opcodes.

What is really interesting about the attack are the consequences of SINIT mode hijacking, which include ability to bypass Intel TXT, LCP, and also compromise system SMRAM.

It's also interesting how difficult was this vulnerability for Intel to patch, as they had to release not only updated SINIT modules, but also updated microcode for all the affected processors, and also work with the BIOS vendors so they release updated BIOSes that would be unconditionally loading this updated microcode (plus provide anti-rollback mechanisms for both the BIOS and microcode). Quite an undertaking...

You can get the paper here.

Intel also published an advisory yesterday, which can be downloaded from their website here. The advisory is peculiar in a few ways, however...

First, the advisory (I'm referring to the revision 1.0) never explicitly mentions that the attack allows to bypass TXT launch itself, only that the attack “may compromise certain SINIT ACM functionality, including launch control policy and additionally lead to compromise of System Management Mode (SMM). Intel also recommend to disable TXT altogether in the BIOS, as a preventive measure, in case the user doesn't “actively running Intel® TXT”... This reminds me how various vendors started actively disabling Intel VT-x after certain virtualization rootkits have been demonstrated some 5 years ago, and how many laptops still ship with this technology disabled today (or VT-d at least) to the questionable delight of many users.

Second, the advisory assigns only an “Important” rating to this vulnerability, even though another Intel advisory, published some two years ago for a problem also reported by us, and which which was strictly a subset of the current vulnerability in terms of powers that it gave to the attacker (in other words the current vulnerability provides the attacker with everything that the previous one did, plus much more), was given a “Critical” rating... This is called evolution, I guess, and I wonder what would be considered critical by Intel these days?

UPDATE (Dec 7th, 2011): Intel has just released an updated advisory (release 1.1) that now explicitly states that the vulnerability also bypasses Intel TXT.

This is the last paper co-authored with Rafal Wojtczuk, who recently decided to try some new things and to leave ITL. Rafal has been the most talented exploit writer I have worked with, and I will surely miss his ingenious insights, such as e.g. how to practically win an absolutely hopeless race condition with ICMP-delivered MSI! But then again, how many times can one break Intel technologies, before getting bored? At the same time ITL is really transforming now into a development company, with all our efforts around Qubes and architecting, rather than on breaking. I wish Rafal all the best with his new endeavors, and thank him for all the excellent contributions he made while working for ITL over the past 3+ years.

Playing with Qubes Networking for Fun and Profit

Today, I would like to showcase some of the cool things that one can do with the Qubes networking infrastructure, specifically with all the new features that have been brought by the just released Qubes Beta 2. This will cover the use of multiple Net VMs for creating isolated networks, the use of a Proxy VM for creating a transparent Tor Proxy VM, as well as demonstration of how to use a Standalone VM with manually assigned devices, to create a “WiFi pen-testing” VM, which surely represents the “for fun” aspect of this post.

Qubes Networking Intro

From the networking point of view there are three types of VMs in Qubes:

• Net VMs, that have networking devices assigned to them, such as e.g. a WiFi or Ethernet card. Each Net VM contains a Xen network backend that is used to provide networking to all VMs that are connected to this Net VM.
• Regular VMs (AppVMs) that use the networking provided by Net VMs (so they have Xen network frontends that provide virtual interfaces that are backed by the backend in the corresponding Net VM.
• Proxy VMs that combine both of the above: to Net VMs they look like regular AppVMs, because they are consumers of the networking they provide, but to other AppVMs they act as if they were Net VMs themselves, allowing other VMs to connect to them. Of course the Proxy VMs do not have directly assigned networking devices – they use the networking provided by the Net VM that they connect to. One can chain many Proxy VMs, as we will see below.

The virtual interfaces in client VMs are called ethX, and are provided by the xen_netfront kernel module, and the corresponding interfaces in the Net/Proxy VM are called vifX.Y and are created by the xen_netback module.

Each Net and Proxy VM implements NAT, specifically masquerading, for all the connected VMs. Additionally to this SNAT, each Net or Proxy VM provides also DNAT redirection for DNS resolutions, so that each VM behind a Proxy or Net VM thinks that it uses a DNS in the Net/Proxy VM, but in fact all the DNS request are DNAT-ed by all the Proxy and Net VMs down the original DNS that is provided to the final Net VM. This smart trick allows us to avoid running a DNS caching server in Proxy/Net VMs.

Also, any VM-to-VM traffic, among the VMs connected to the same Net/Proxy VM is blocked by default.

Additionally, each Proxy VM enforces system-wide firewaling rules, specifically the rules for all the directly connected VMs. Those firewalling rules are centrally managed in Dom0 and exposed to each Proxy VM through Xen store. One useful application of this firewalling mechanism is to limit certain VMs to only specific type of white-listed traffic to minimize likelihood of user mistakes. A good example could be a work VM that might be limited to network connectivity only with the select corporate servers and denied all other traffic. This way, when the user receives an email message with an embedded http link (possibly leading to a malicious website) and accidentally clicks on it, nothing wrong happens.

The current infrastructure doesn't support IPv6 routing, but we will likely add this support in the upcoming Beta 3.

The default networking topology in Qubes OS

When you proceed with the default installation of Qubes Beta 2, then your initial networking topology looks like on the diagram below:

The default network configuration in Qubes.

So, by default there is one Net VM, called 'netvm', that is automatically assigned all the networking devices in the system. There is also one Proxy VM, called 'firewallvm' that is directly connected to the default Net VM, and which provides networking to all other VMs in the system. This Proxy VM is used for firewall rules enforcement. Each such service VM consumes 200MB of RAM by default.

Network-isolated VMs

For some VMs it might be desirable to completely disconnect them from any kind of networking access. This can be easy done using the following command (issued from Dom0's konsole):

[dom0]$ qvm-prefs -s netvm none

For example I have a 'vault' VM that I use for keeping my master PGP keys, and other secrets, and this machine is not connected to any network.

Using multiple Net VMs for physically isolated networks 

In some scenarios the machine might be connected to two or more physically separate networks (e.g. safe corporate intranet, reachable via ethernet cable on the user's desk, and the unsafe and evil Internet, reachable via WiFi card).

It is easy to use more than one Net VMs in Qubes, and assign different networking devices to different Net VMs, and also decide which VMs are connected to which Net VMs. The diagram below presents an exemplary such setup:

A simple setup with two isolated networks, and one fully isolated domain ('vault').

 

We could created such a setup using the following commands (issued in Dom0):

[dom0]$ qvm-create netvm1 --net --label red

[dom0]$ qvm-create netvm2 --net --label yellow

Currently qvm-create when used with the --net option automatically assigns all networking devices to the just created VM, so in the example above you would want to remove extra devices from each Net VM using qvm-pci -d, leaving only those you really want, e.g.: 

[dom0]$ qvm-pci -l netvm1 # to get a list of currently assigned devices

[dom0]$ qvm-pci -d netvm1 02:00.0

Now we should create the Firewall VMs:

[dom0]$ qvm-create firewallvm1 --proxy --label green

[dom0]$ qvm-create firewallvm2 --proxy --label green

... and connect them to proper Net VMs:

[dom0]$ qvm-prefs -s firewallvm1 netvm netvm1

[dom0]$ qvm-prefs -s firewallvm2 netvm netvm2

And now, for any other VM, just set the appropriate Net VM (either firewallvm1 or firewallvm2, or 'none), to get it assigned to either of the isolated networks, e.g.:

[dom0]$ qvm-prefs -s banking netvm firewallvm1

[dom0]$ qvm-prefs -s xfiles netvm firewallvm2

[dom0]$ qvm-prefs -s vault netvm none

...

This configuration provides very strong isolation between the VMs belonging to network #1, and the VMs belonging to network #2. Specifically, this becomes significant if we fear about potential remotely exploitable bugs in the client code of the core TCP/IP stack (in this case the Net VM could potentially compromise all the connected VMs -- but the same problem applies to even physically separated machines that use the same network).

Setting up Tor Proxy using a Proxy VM

Let's now play a bit with Proxy VMs and see how we can use it to create a simple Tor proxy VM. Such a VM would provide anonymized networking to all its clients, so would allow to easily create VMs for anonymous Internet access. The simple setup we would like to prepare is depicted on the figure below:

The 'torvm' Proxy VM provides anonymized networking to 'anon-web' and 'anon-bitcoin' VMs. All the traffic generated by the VMs behind 'torvm' is either fed into the Tor network, or discarded. Furthermore, any app running in those VMs is not able to read any global system identifiers, such as the external IP, external MAC address, etc.

Our Tor proxy would forward only the Tor traffic, so we don't have to fear about some Tor-not-aware applications, or even intentionally malicious ones to compromise the privacy of our connection. This is because such applications have no way to generate traffic to the outside world without going through our Tor proxy (unless they could exploit a hypothetical vulnerability in the Tor process running in the Tor VM). Also, the applications running in any VM behind the Tor proxy are not able to determine any globally identifiable IDs, such as the user's external IP address, the real MAC address used by real NICs, etc.

Interestingly just after writing the above paragraph, I discovered that one of our xenstore keys had wrong permissions and, as a result, any VM could read it and get to know the actual external IP (the key is used by a Net VM to communicate the external IP configuration to the connected Proxy VMs, so they could know when to update the firewall configuration). The fix for this problem is here, and the update (qubes-core-dom0-1.6.32) is now available for Dom0 (just do qvm-dom0-update to get it installed).

 

So, this represents a rather strong setup for use with Tor. Let's now have a look at how to practically create such a configuration, step by step.

First, let's create the VM that will become our Tor proxy:

[dom0]$ qvm-create torvm --proxy --label green

This will create a Proxy VM named 'torvm', based on the default template. We will need to now start the template VM and install the Tor client there:

[dom0]$ qvm-run -a fedora-14-x64 gnome-terminal

Alternatively, if we didn't trust the Tor client rpm package to be non-malicious, specifically for its installation scripts to be non malicious, we could have based this on a different template, e.g. one used for less trusted VMs, or we could installed the Tor client in /usr/local, that is backed by the VM's private storage, but this would require compiling Tor from sources.

Now, in the just started template VM, lets install the Tor client and (optionally) the Vidalia graphical frontend:

[fedora-14-x64]$ sudo yum install tor vidalia

And then power off the template VM. Now, every VM based on this template, started after the template shutdown, will also see the Tor binary in its filesystem.

Let's now configure our torvm to properly start Tor proxying at boot:

[dom0]$ qvm-run -a torvm gnome-terminal

Now, we will create the following script for starting up the Tor transparent proxy and setting up traffic redirection using iptables:

[torvm]$ vim /rw/config/start_tor_proxy.sh

...and now paste the following into this file:

#!/bin/sh

killall tor

QUBES_IP=$(xenstore-read qubes_ip)

TOR_TRANS_PORT=9040

if [ X$QUBES_IP == X ]; then

echo "Error getting QUBES IP!"

echo "Not starting Tor, but setting the traffic redirection anyway to prevent leaks."

QUBES_IP="127.0.0.1"

else

/usr/bin/tor \

--SocksPort 0 \

--TransListenAddress $QUBES_IP --TransPort $TOR_TRANS_PORT \

--DNSListenAddress $QUBES_IP --DNSPort 53 \

--RunAsDaemon 1 --ControlPort 9051 \

|| echo "Error starting Tor!"

fi

echo “0” > /proc/sys/net/ipv4/ip_forward

/sbin/iptables -t nat -F

/sbin/iptables -t nat -A PREROUTING -i vif+ -p udp --dport 53 -j DNAT --to-destination $QUBES_IP:53

/sbin/iptables -t nat -A PREROUTING -i vif+ -p tcp -j DNAT --to-destination $QUBES_IP:$TOR_TRANS_PORT

/sbin/iptables -I INPUT 1 -i vif+ -p udp --dport 53 -j ACCEPT

/sbin/iptables -I INPUT 2 -i vif+ -p tcp --dport 9040 -j ACCEPT

/sbin/iptables -F FORWARD

echo “1” > /proc/sys/net/ipv4/ip_forward

Except for the “QUBES_IP=$(xenstore-read qubes_ip)” line that reads the torvm's IP address, there is nothing Qubes-specific in the above listing. It's just a standard way of setting up transparent Tor proxy.

It is important that this file be located in the /rw directory, as this directory is backed by the VM's private storage and will survive VM reboots. The VM's root file-system is read-only and all the changes to it are lost on VM shutdown (VM gets an illusion of the root fs being writeable thanks to Copy-On-Write mechanism, but the actual COW backing device is cleared upon each VM shutdown).

We should also modify the /rw/config/rc.local script, to ensure that our Tor proxy is automatically started -- just paste the following into this script:

#!/bin/sh

# Uncomment this if you would like to use a custom torrc file:

#rm -f /rw/config/log

#ln -sf /rw/config/torrc /etc/tor/torrc

chkconfig qubes_netwatcher off

chkconfig qubes_firewall off

/rw/config/start_tor_proxy.sh

Finally we should also provide a script that would restart our proxy in case the user dynamically switched the NetVM, which would result in the completely different routing. This could be done by creating a script with predefined name qubes_ip_change_hook within /rw/config/ directory:

#!/bin/sh

/rw/config/start_tor_proxy.sh

Make sure that all the scripts are executable (chmod +x). And that's all. Now, shutdown the torvm:

[dom0]$ qvm-run --shutdown --wait torvm

From now on, every time you start the torvm (or when Qubes starts it in response to start of some other VM that uses torvm as its Net VM), the Tor transparent proxy should be automatically started.

Let's test this by creating a VM that would be using the just created Tor proxy:

[dom0]$ qvm-create anon-web --label black

[dom0]$ qvm-prefs -s anon-web netvm torvm

Now, every time you start the anon-web VM (e.g. by clicking on the Web browser icon in the anon-web's start menu), Qubes will also ensure that torvm is up and running, and this in turn would configure all the Tor proxying for this VM.

Fo additional control one might want to use Vidalia, the graphical front end for Tor (this should be installed within the template VM that has been used for torvm). We could easily start Vidalia by just typing:

[dom0]$ qvm-run -a torvm vidalia

We should however make sure to disable "Start the Tor software when vidalia starts" option in Settings/General in Vidalia. Otherwise, Vidalia might kill your original Tor (that has transparent proxy open) and start own without transparent proxy enabled.

The web browser runs in the 'anon-web' VM that uses 'torvm' for networking access, and thus all the traffic generated by 'anon-web' is routed through the Tor network, or discarded if it's a different traffic than TCP or DNS.

Of course one case easily create more VMs that would be using torvm as their Net VM, as so would have anonymized network access. The beauty of this solution is that in case one of my anonymized VM gets compromised, others do not. Plus, the already mentioned benefit, that no matter whether apps in those VMs are buggy, or even intentionally malicious, they would not be able to leak out the user's external IP address.

Creating a WiFi pen-testing VM

Finally let's have some fun and create a WiFi pen-testing VM. The desired config is depicted below:

Because we would like to use all sorts of l33t h4x0r t00lz pen-testing security software in this VM, it would make sense to create it as a Standalone VM, which means that it would get its own copy of the whole file-system (as opposed to just the home directory, /rw and /usr/local, as it is the case with regular Qubes VMs). This would ease the installation of all the extra software we would need there, and also ensure that even if the install/build scripts were malicious, the damages would be contained only to this very VM and nothing else. Also, for some reason the standard Linux WiFi stack and drivers still don't support injection on (all?) most of the WiFi cards out of the box, so we would need to patch the actual kernel drivers -- yet another reason to use a Standalone VM in this case.

So, let's create the VM first, and assign a WiFi card to it:

[dom0]$ qvm-create wififun --standalone --label yellow

[dom0]$ qvm-prefs -s wififun memory 800 # ensure at least this mem at startup

[dom0]$ qvm-prefs -s wififun kernel none # use own copy of kernel and modules

[dom0]$ qvm-pci -a wififun

You can easily find the BDF address of any device using the lspci command in Dom0 -- this would be something like e.g. “02:00.0”. You should make sure that this WiFi card is not used by any other VM, specifically by your default Net VM (called 'netvm' in a standard Qubes installation). Ideally you could just use a dedicated Express Card-based WiFi card, leaving the built in WiFi assigned to your default Net VM.

Because it's a Standalone VM, Qubes will make a copy of the whole root filesystem, and thus it would eat about 5GB of your disk (normal VMs would take only as much space as their private fs takes up).

Let's now start the VM...

[dom0]$ qvm-run -a wififun gnome-terminal

... and then install the prerequisite software there, starting with downloading the reasonably new compat-wireless sources, together with the required injection patches, and then building and installing the new kernel modules. All actions below are now executed within the VM. This stuff here is really nothing Qubes- or Xen-specific -- one would do more or less the same on any Linux in order to get injection working (so, treat this as a free bonus WiFi hacking tutorial on Linux).

[wififun]$ wget http://linuxwireless.org/download/compat-wireless-2.6/compat-wireless-2011-07-14.tar.bz2

[wififun]$ wget http://patches.aircrack-ng.org/channel-negative-one-maxim.patch

[wififun]$ wget http://patches.aircrack-ng.org/mac80211-2.6.29-fix-tx-ctl-no-ack-retry-count.patch

[wififun]$ wget http://patches.aircrack-ng.org/mac80211.compat08082009.wl_frag+ack_v1.patch

[wififun]$ sudo yum install kernel-devel patch gcc

[wififun]$ tar xjf compat-wireless-2011-07-14.tar.bz2

[wififun]$ cd compat-wireless-2011-07-14

[wififun]$ patch -p1 < ../channel-negative-one-maxim.patch

[wififun]$ patch -p1 < ../mac80211-2.6.29-fix-tx-ctl-no-ack-retry-count.patch

[wififun]$ patch -p1 < ../mac80211.compat08082009.wl_frag+ack_v1.patch

[wififun]$ make

[wififun]$ sudo make unload

[wififun]$ sudo make install

Now, lets reboot the VM to ensure that all the patched drivers will get properly loaded on each VM boot:

[dom0]$ qvm-run --shutdown --wait wififun

[dom0]$ qvm-run -a wififun gnome-terminal

Let's first see if the WiFi driver got properly loaded and if the interface has been created (look for wlanX interface):

[wififun]$ ifconfig -a

If yes, then proceed with the steps below (if not, then have a look into dmesg and see what was the problem):

[wififun]$ sudo bash

[wififun]# yum install aircrack-ng dnsmasq

[wififun]# airmon-ng start wlan0

[wififun]# iptables -F INPUT

[wififun]# iptables -F FORWARD

[wififun]# echo “1” > /proc/sys/net/ipv4/ip_forward

Note that you don't need to add any explicit masquerading rules, as they are applied by default on Qubes VMs (you can take a look at the nat table in the VM if you want to see by yourself).

Edit the /etc/dnsmasq.conf, so that it contains at least the following:

interface=at0

dhcp-range=192.168.0.50,192.168.0.150,12h

and then start the dnsmasq daemon -- we will use it for providing DHCP to our fake AP (the at0 interface will be created by airbase-ng and emulates the “uplink” of a traditional AP):

[wififun]# /etc/init.d/dnsmasq start

And finally the fake AP:

[wififun]# airbase-ng -e free_wifi mon0

and on another console (before any client connects, but after airbase-ng got started), configure the at0 interface (make sure it matches what you wrote into dnsmasq.conf):

[wififun]# ifconfig at0 192.168.0.1 up

(you can also add an udev rule to that automatically).

and just to verify it really is working:

[wififun]# tcpdump -i at0

... and now, just wait for a client to connect to your AP. What you do next is only limited by your imagination... But hey, this article is about Qubes networking and not about 0wning client systems ;)

Here's an innocent example using Moxie's sslstrip (amazing this attack still works so well at the end of 2011...):

My 'wififun' VM in action using a simple sslstrip attack, that surprisingly still works pretty nice...

Please note that as your wififun VM is a regular Qubes VM, it is automatically connected to the default Net VM, which in turn provides networking to it. That's why it is so easy to create a fully functioning fake AP.

When using custom driver domains, there are currently some catches you should be aware:

Catch #1: When you start a driver domain late after system boot, so after some days of uptime and extensive use of VMs, Xen might not be able to allocate enough continues (in terms of MFNs) memory for a driver domain. And PV driver domains, unlike normal domains or HVM driver domains, do require MFN-continuous memory for their DMA buffers (HVM domains do not need that, because IOMMU can create an illusion of this; even though IOMMU is also used for PV driver domains, for protection, it doesn't actively translate bus addresses into GMFNs).

This is usually not a big problem in practice, because in most cases all the driver domains are started early at system boot, when there is still plenty of non-fragmented memory available. However it might become a problem when one wishes to start e.g. the WiFi pen-testing at some later time. The work around is to close as many VMs as possible before starting such driver domain, and then also reducing, for a moment, the amount of memory assigned to Dom0:

[dom0]$ xm mem-set 0 1600m

and then starting the driver domain should be fine. Now we can start all other domains, and that should no longer be problematic for the already running driver domain.

Catch #2: Some network cards, notably Express Cards, might not work well with the 3.0.4 pvops kernel that we use in all VMs by default. In that case you might want to try to use the 2.6.38.3 xenlinux kernel in your WiFi fun VM -- to do that, follow these steps:

[dom0]$ sudo qvm-dom0-update kernel-qubes-vm-2.6.38.3-10.xenlinux.qubes

[dom0]$ cp /var/lib/qubes/vm-kernels/2.6.38.3/* /var/lib/qubes/appvms/wififun/kernels/

[dom0]$ qvm-prefs wififun -s kernelopts "swiotlb=force"

And then, in the VM:

[wififun]$ sudo yum install kernel-devel-2.6.38.3-10.xenlinux.qubes

And rebuild the compat-wireless, unload, install modules, and then load drivers again.

Summary

As you can see, Qubes Beta 2 now offers a very advanced networking infrastructure that allows more advanced users to create very sophisticated configurations, allowing for pretty good isolation between various domains and networks. Qubes leaves it up to the user (or admin) to figure out what would be the best configuration -- most users would be happy with the default simple setup with just one Net VM and one Firewall VM, while others would go for much more advanced setups.

A bit more advanced networking setup. The usbvm has a 3G modem assigned, and it is possible to dynamically switch between the Net VMs without restarting any other VMs.

Qubes Beta 2 Released!

I'm proud to announce that we have just released Qubes Beta 2! You can view installation instructions and download the ISO here.

We faced quite a few serious problems with this release that were caused by an upgrade to Xen 4.1 (from Xen 3.4) that we used in Beta 1. But finally we managed to solve all those problems and all in all I'm very happy with this release. It includes many performance optimizations compared to Beta 1 (CPU- and memory-wise) and also many bugfixes.

We also introduced a couple of new features:

• Generic mechanism for inter-domain services with a centralized policy enforcement (more)

• Network-less update mechanism for Dom0 (more)

• VM management improvements: easy device assignment for driver domains, dynamic netvm switching, flexible VM kernel configuration, etc (see the new qvm-prefs utility)

• Easy management of appmenus (shortcuts in the Start Menu)

• Update to Xen 4.1 that offers, among other things, better VT-d support and more lightweight management stack (we have ported Qubes to use the new xl now, instead of the slow and heavy xend), and also to 2.6.38-xenlinux kernel for Dom0, and to 3.0.4 pvops kernel for VMs (better hardware compatibility, better power management)

I will write some more posts shortly that would present in detail some of the new features and what cool things one could do with them.

We have also created a dedicated wiki page that enumerates all the security-critical code for Qubes OS. We hope this page would be useful for security researchers that might attempt to find weaknesses in Qubes OS either in our code or in the 3rd party code that we rely on (Xen hypervisor, select Xen backends). Whether your motives are noble (gaining immortal fame, helping create a secure client OS), or not (proving ITL wrong), we would appreciate your efforts! And you might even get a job at ITL.

Speaking of which, I'm happy to announce that Marek Marczykowski, who has effectively become the key Qubes developer over the past few months, has now officially joined ITL :)