NVMM
Created:08-09-2018
Updated:27-02-2019

NetBSD Virtual Machine Monitor


NVMM provides hardware-accelerated virtualization support for NetBSD. It is made of an ~MI frontend, to which MD backends can be plugged. A virtualization API is shipped via libnvmm, that allows to easily create and manage virtual machines via NVMM. Two additional components are shipped as demonstrators, toyvirt and smallkern: the former is a toy virtualizer, that executes in a VM the 64bit ELF binary given as argument, the latter is an example of such binary.

Resources


The kernel code: browse.
The libnvmm API: browse.
The demonstrators: download.
The man page: open.
The regression tests: browse.

Qemu+NVMM


Make sure you have the latest NVMM headers installed. If you don't:     # cp /usr/src/sys/dev/nvmm/nvmm.h         /usr/include/dev/nvmm/
    # cp /usr/src/sys/dev/nvmm/nvmm_ioctl.h   /usr/include/dev/nvmm/
    # cp /usr/src/sys/dev/nvmm/x86/nvmm_x86.h /usr/include/dev/nvmm/x86/
Make sure you have the latest version of libnvmm installed. If you don't:     # cd /usr/src/lib/libnvmm/
    # make
    # make install
Load the NVMM kernel driver:     # cd /usr/src/sys/modules/nvmm/
    # make
    # modload ./nvmm.kmod
You may want to run the NVMM tests to confirm that your setup is correct:     # cd /usr/src/tests/lib/libnvmm/
    # make
    # ./h_io_assist
    # ./h_mem_assist
Install the qemu-nvmm package from pkgsrc-wip. Note that this will conflict with any previous Qemu package that you installed.

Once this is done, you can finally launch Qemu with NVMM acceleration, by adding "-accel nvmm" to your usual Qemu command line. For example:     # qemu-system-x86_64 -cdrom NetBSD-8.0-amd64.iso -m 256M -accel nvmm
NetBSD on Qemu+NVMM Fedora on Qemu+NVMM
WindowsXP on Qemu+NVMM Windows10 on Qemu+NVMM
NetBSD 8 (top left), Fedora 29 (top right), Windows XP (bottom left), Windows 10 (bottom right),
all running via Qemu+NVMM on NetBSD.


Notes:

Windows 64bit requires MTRR and MCE, and both are currently masked in NVMM; support for that hasn't yet been committed.


Linux performs timer checks at boot time which fail under hypervisors; to boot Linux, pass "no_timer_check" as kernel argument.

Technical details


NVMM can support up to 128 virtual machines, each having a maximum of 256 VCPUs and 128GB of RAM.

Each virtual machine is granted access to most of the CPU registers: the GPRs (obviously), the Segment Registers, the Control Registers, the Debug Registers, the FPU (x87 and SSE), and several MSRs.

Events can be injected in the virtual machines, to emulate device interrupts. A delay mechanism is used, and allows VMM software to schedule the interrupt right when the VCPU can receive it. NMIs can be injected as well, and use a similar mechanism.

The host must always be x86_64, but the guest has no constraint on the mode, so it can be x86_32, PAE, real mode, and so on.

The TSC of each VCPU is always re-based on the host CPU it is executing on, and is therefore guaranteed to increase regardless of the host CPU. However, it may not increase monotonically, because it is not possible to fully hide the host effects on the guest during #VMEXITs.

The different intercepts are configured in such a way that they cover everything that needs to be emulated. In particular, the LAPIC can be emulated by VMM software, by intercepting reads/writes to the LAPIC page in memory, and monitoring changes to CR8 in the exit state.

Two assists (IO and Mem) are provided in libnvmm, they allow VMM software to easily handle #VMEXITs that require disassembling guest instructions.

Support


Two backends are supported currently, x86-SVM for x86 AMD CPUs, and x86-VMX for x86 Intel CPUs.