There will always be a need for one more privilege level

When I was involved in Intel’s virtualization, I found something like seven different privilege layers.

Not just user/OS. Not just user/OS/hypervisor. Not just user/OS/M-mode. There's always the need for one more layer or more.

note that sometimes I call these layers, sometimes I call them virtual machines, sometimes ...

Here's how I see it:

0. The actual hardware, what we see if you guys used to emulate instructions and so on

1. a software layer that might be used to patch bugs in the hardware, or possibly to implement features like state machines that you don't want to spend hardware to do, or to trap and emulate instructions

   1. This might or might not have been Intel x86 microcode
   2. But even given microcode, it would have been nice to have a software layer in addition to microcode
   • this might be DEC Alpha PAL code or RISC-V M-mode
     • except that RISC-V M-mode is also being used for other stuff like TEE

2. The next layer was used by Intel chipset guys, again to emulate features in their chipset. I remember them telling me that they wouldn’t be able to ship if they couldn’t patch. At one point they flipped the polarity of a bit by this sort of trapping emulate, although at that time they trapped to Intel‘s SMM mode

3. the same sort of thing for people who bought Intel‘s chips and put them in motherboards, like Compaq. SMM mode was originally used by Compaq to do power management when the OS was not providing that feature.

4. The true hypervisor

5. the operating system

6. user level processes running on the operating system

7. Additional virtual machine layers.

   • yes: I do mean that it could be possible to run an entire guest virtual machine, with all of its layers, from a user level process, without requiring any kernel level support
     • usually, however, nested virtual machine support, although it might look like it is within the operating system, actually requires some hypervisor level support.
   • as far as I know no existing virtual machine implementation supports this
   • although I think that an IBM SIE style architecture could support it

PLUS: do you nearly always want to be able to dynamically insert at least one layer of virtual machine so that you can do debugging. Sometimes there’s the system level debugger and the user level deugger running at the same time. moreover, such a debug layer is not necessarily the fixed level of the stack. Sometimes the CPU guys would need to be able to debug the software layer that they have provided.

The key to making this work is delegation. Not necessarily complete virtualization, not necessarily transparent. protection of more privilege layers from less privileged layers, but not necessarily hiding. The ability to delegate certain features, like manipulating interrupts and so on, to less privileged layers, but always under the control of the more privileged layers.

---

IMHO one of the fundamental sins of many virtual machines/hypervisor/privilege architectures is that they assume that there are only N layers. often, they assume there is only one.

E.g. there's user and kernel. How can use anything more?

• in honesty, early machines might've a few more layers.
  • E.g. x86's 4 rings. although few operating systems used anything other than rings 0 and 3.
  • e.g. MIPS's K, S, and U modes. ditto, usually only two were used

Well, eventually we learned that we needed a hypervisor layer. So there's hypervisor and non-hypervisor. and the non-hypervisor is itself divided into the two existing layers kernel and user. (H,(K,U)).

Well, except before we got around hypervisors, people like Compaq wanted to implement features like power management before the Microsoft OSes were ready to do so. So, Intel added SMM (System Management Mode). the guy who designed this new he really wanted to implement a virtual machine, but couldn't afford to do so, so he made compromises. Regrettable perhaps, but successful in the marketplace, no matter how many problems SMM eventually caused.

SMM was such a good idea that ... unfortunately, instead of adding a new mode in a clean manner, people started putting things that should've belonged in different privilege domains in the same SMM mode. First power management. Then an OS independent debugger. Then patching of the chipset bugs. Then ...

The problem with this is that independent things have independent bugs. They collide.

the point of this rant is that if you have code that comes from different origins and really should be protected from each other, there really should be a protection system. If the code runs independently, different parallel privilege domains, like processes or virtual machines. If however one such facility uses features from another, e.g. if one such facility is used to trap and emulate instructions executing in another facility, then they are layered in some way. You don't necessarily need to transition each layer - indeed, you want to be able to skip as many layers as possible whenever possible - but there may be such a total order of layers in terms of trapping emulate.

given any stack of privilege layers there is always the need for one more layer. If nothing else, to debug bugs in that existing layer.