When we think about “USB Security” there are lots of things that come to mind...
First there are all the physical attacks that could be conducted with the help of USB devices. These are generally not so interesting, because if one includes physical attacks in the threat model, then it really opens up lots of possibilities of various attacks, and generally a physical attacker always wins. Still, there are a few very cheap and easy physical attacks that one would like to avoid, or make harder, such as the Evil Maid Attacks or the Cold Boot Attacks. Strictly speaking these are not problems inherent to USB itself, but rather with lack of Trusted Boot, or OS not cleaning properly secrets from memory upon shutdown. They are just made simple thanks to bootable USB sticks.
Much more interesting USB-related physical attacks are those that take advantage of the specifics of the USB standard. One example here would be a malicious USB device that exposes intentionally malformed info about itself in order to exploit a potential flaw in a USB Host Controller driver that processes this info upon each new USB device connect. Or a malicious USB device that would trick the OS (Windows at least) into downloading a known buggy USB driver (or even an intentionally malicious driver, legally submitted to WHQL by the attacker) and then exploit the driver.
Another class of physical attacks made possible by the USB specification are malicious USB devices that pretend to be a keyboard or mouse. The input devices, such as keyboard, are actually the most security sensitive devices, because an attacker who controls the keyboard can do everything the user can do, which basically means: can do everything, at least with regards to the user's data.
Finally, the USB, as the names stands, is a bus interconnect, which means all the USB devices sharing the same USB controller are capable of sniffing and spoofing signals on the bus. This is one of the key differences between USB and PCI Express standards, where the latter uses a peer-to-peer interconnect architecture.
Ok, so these all above were physical attacks. Let's now look at, much more fatal, software attacks.
The infamous class of attacks exploiting various autorun or auto-preview behaviors is the most known example, but also the easiest, at least in theory, to protect against.
Much more interesting are software attacks that attempt to exploit potential flaws in the USB stacks – similarly like the physical attacks mentioned above, just that this time not requiring any hardware-level modifications to the USB device. Exposing a malformed partition table is a great example of such an attack. Even if we have all the autorun mechanisms disabled, still, when we're inserting a storage medium the OS always attempts to parse the partition table in order to e.g. create devices symbolizing each partition/volume (e.g. /dev/sdbX devices).
Now, this is really a problematic attack, because the malformed partition table can be written onto a fully legitimate USB stick by malware. Imagine e.g. you have two physically separated machines (air-gapped), belonging to two different security domains, and you want to transfer files from one to another. You insert the USB stick into the first machine, copy files, and then insert the stick to the second machine. If the first machine was compromised, it could have altered the partition table on the USB stick, and now when this stick is inserted into the other machine its malformed partition table might exploit a buffer overflow in the code used by the second OS to parse the stick's partition information. Air-gapped systems, huh? We avoid this attack vector in Qubes by using a special inter-domain file copy mechanism that doesn't require any metadata parsing.
A variation of the above attack would be to expose a malicious file system metadata, but this time the target OS would have to actually mount the partition for the attack to work (and, of course, there would have to be bugs in the OS file system parsing code, although these seem to be quite common on most OSes).
Having quickly summarized the USB security-related threats, let's now think about how we could design an OS to mitigate most of those attacks, and at the very least the software-based attacks. This is, in fact, precisely the challenge we've been facing in Qubes, so the divagations below necessarily focus mostly on the Qubes architecture.
First we should realize that USB devices, unlike PCI Express devices, cannot be independently delegated to different domains (VMs). This is because IOMMU technologies, such as Intel VT-d, operate only on PCIe device granularity. This means we can only delegate a whole USB controller to a domain, including all of the USB devices connected to this controller/hub.
Imagine now two internal devices, both connected via internal USB bus: a keyboard, and a 3G wireless modem. Chances are high that you will have those two devices connected to the same USB controller – usually one controller is used for all the internal devices, like those I just mentioned, plus camera, fingerprint reader, etc, and the other controller is used for all the externally visible USB connectors (at least this is true for modern systems: Intel Series 5 chipsets and newer).
We would like to be able to delegate the 3G modem to the NetVM (an untrusted domain on Qubes where all the networking drivers and stacks are kept; it's considered untrusted because its compromise is equivalent to a compromise of a WiFi network or home router, or some other router, and any reasonable person always assumes that the network is compromised, and deals with that using crypto, such as SSL or SSH). But assigning the USB controller, to which the 3G modem is connected to, to the NetVM, would also assign the USB keyboard to the NetVM! And this is precisely what we don't want to do, because control over the keyboard is equivalent to the control over the whole system!
Currently, in Qubes Beta 1, we keep all the USB controllers assigned to Dom0. This, however, causes two annoyances:
First, the user cannot use any of the USB-connected networking devices, such as 3G modems (because there is no networking in Dom0).
Second, if somebody connects a USB disk and later delegates it to some domain (this could easily be done via block-attach mechanism, supported by the same backend that handles storage virtualization for domains), and this domain turns out to be compromised, it might alter e.g. the stick's partition table and later attack Dom0 as explained above.
We can eliminate the second problem by modifying the Dom0's kernel to not parse the partition table of any removable devices automatically, and instead expect some kind of explicit consent from user to actually do that (we still must allow to mount USB disks in Dom0 to allow easy backups of all domains at once).
To allow the use of USB-connected networking devices in NetVM, we could use a PVUSB backend that can virtualize single USB devices without moving the whole USB controller to the domain. But that would require introducing a whole lot of new code to Dom0 – code that would be directly reachable from VMs (in other words that would be processing lots of untrusted input coming from untrusted domains).
So another option is to delegate all the non-security-critical USB controllers, i.e. those controllers that don't have any security-sensitive USB devices connected, such as keyboard, to a dedicated “USB” domain, and later share the USB devices via PVUSB backend from this USB domain. This time, the extra PVUSB backend runs in the USB domain, not in Dom0, so we don't worry that much about potential bugs in this backend. Of course, this way you cannot delegate the USB controller to which the keyboard, and potentially also other security-critical devices, such as camera, are connected to, which in practice rules out the integrated3G modem. Fortunately many modern laptops do not use USB-connected keyboard and touchpad (they use PS/2-connected keyboards instead), and the face camera can be easily disabled with a piece of sticker (although that sucks, because it means we cannot really use the camera).
With this approach (a dedicated USB domain) you can now delegate your 3G modem to the NetVM, and other USB devices, such as removable disks to other domains, e.g. for file exchange. This seems the most reasonable setup, although it requires that either 1) your laptop doesn't have USB-connected keyboard, or 2) you don't use internal USB devices connected to the same controller that your USB keyboard/touchpad from other domains than Dom0 (in practice: no 3G modem in NetVM).
As we can see proper handling of USB devices is quite a challenge for OS architects. It might have been much less of a challenge if the engineers designing the USB, chipsets, and motherboards were a bit more security-conscious. Even such simple practice as never mixing security critical devices (keyboard, touchpad, camera, fingerprint reader), with non-security ones (3G modem), onto the same USB controller, would help tremendously. Or ability to somehow dynamically configure their connectivity, e.g. in BIOS?