Security “in the hypervisor”

One of the dilemmas facing virtualization vendors (in particular vmware) is whether to open up the hypervisor and allow third-party plug-ins for things like security. It will be interesting to see how this plays out.

If we follow the path that operating systems took, we may see solutions that add "hooks" (which are essentially micro-man-in-the-middle attacks) into the hypervisor software to bolster some security capability like antivirus scanning. Nowadays, there may be much more screaming about rootkit-like behavior, even though pretty much every endpoint-based security product pre-Vista uses the technique in the Windows world.

I think hooks like these are inevitable, but they won’t necessarily be unauthorized. With vmware’s Vsafe program, the company is already opening up some APIs to security companies. This, unfortunately, is both necessary and a slippery slope – it almost automatically increases the attack surface of the hypervisor, loosens the control over the code, and generally creates a problem of deciding how to "trust" the third-party vendors.

Why doesn’t Xen have this problem? Well, it does, but since it is open source, changes are made all the time. This is a different kind of problem. And Microsoft’s Hyper-V? I don’t know yet… but the Hyper-V architecture still relies heavily on the operating system, so I don’t see it with as big of a problem.

Btw, keep in mind that one can make the (marketing) claim of being "in the hypervisor" even if a solution is simply a virtual machine — all VMs are "in the hypervisor" at some basic level. It is worth asking exactly what components are customized to specifically work in a virtual environment. (Remember, virtual environments are intended to be transparent to traditional applications).