Welcome back to the second part of my blog post series on how to port an unikernel to Xen. In the first part i gave an introduction to Xen, Unikernels and HermitCore and explained the conceptual changes that have to be made to an operating system to work as a paravirtualized guest in Xen. This part will show you how i got HermitCore running as a fully virtualized guest in Xen. All the sourcecode i show in this post can be found in the xen_hvm branch of my HermitCore fork on GitLab. If you want to try it out for yourself, the docker folder contains instructions on how to build HermitCore and the vagrant folder includes a fully configured Xen hypervisor inside a Vagrant box.

Table of contents

Xen includes two main modes of operation. One is running paravirtualized guests and the other is hardware assisted virtualization. CPUs that support hardware virtualization make it possible to run unmodified guests, including operation systems such as Microsoft Windows. In Xen this is called a hardware virtual machine (HVM). Xen uses device emulation based on QEMU to provide I/O virtualization to the virtual machines. This means the virtual machines see an emulated version of a fairly basic PC.

Because the virtualization of HVM guests is based on QEMU, it is possible to run HermitCore almost unmodified as a guest in Xen. When booting HermitCore via QEMU directly, the kernel is booted via the GRUB Multiboot protocol. Unfortunately it was not possible to make the Multiboot Binary bootloader, which is included in Xen, boot HermitCore directly.

A simple other way to boot HermitCore, is to first start the GRUB bootloader and then boot into HermitCore.

Create a bootable GRUB ISO image

The easiest way to create a bootable media is to use the grub-mkrescue tool. It is included in every GRUB installation on major Linux distributions and can be used to create a bootable ISO file containing the GRUB bootloader. GRUB modules are also needed. On Ubuntu for example, they are included in the package grub-pc-bin. To create a GRUB rescue disk, the folder structure of the ISO file has to be created first. It includes the HermitCore loader, the application that is supposed to run and a GRUB config file and it should look like this:

iso/
 *-- boot
    |-- grub
    |   *-- grub.cfg
    |-- hermit_application
    *-- ldhermit.elf

The grub.cfg file has the following content:

default=0
timeout=0

menuentry "HermitCore" {
    multiboot /boot/ldhermit.elf
    module /boot/hermit_application
    boot
}

It instructs GRUB to create a menu entry called “HermitCore”, which boots the compiled binary file of the HermitCore loader via the Multiboot protocol. The actual application that is intended to run gets passed to the loader as a Multiboot-module. Additional command line options can be passed to the kernel after the multiboot statement. For example

multiboot /boot/ldhermit.elf -uart=io:0x3f8

would enable output on the serial port COM1. To create the ISO file, grub-mkrescue needs to be invoked in the following way:

grub-mkrescue -o /tmp/hermit.iso iso/

The folder structure shown above is included in the “hermit.iso” file. This ISO could now be booted via QEMU and it would work just fine. A domain configuration file is still needed to boot it via Xen.

HVM domain configuration

# This configures an HVM rather than PV guest
type = "hvm"

# Guest name
name = "hermit-single.hvm"

# Initial memory allocation (MB)
memory = 1024

# Number of VCPUS
vcpus = 1

# Network devices
vif = [ 'model=rtl8139,bridge=br0' ]

# Disk Devices
disk = [ 'file:/tmp/hermit.iso,hdc:cdrom,r' ]

#Disable VGA output
nographic = 1

#Serial Console Output
serial = [ 'file:/tmp/hermit.log' ]

tsc_mode="native"

Xen will create a HVM guest by setting type=hvm, assign it the given name, memory and one virtual CPU. HermitCore supports the rtl8139 chipset for network devices, so a device is added using this chipset and connected to the bridge br0 (The name of the device might be different on other Linux distributions). The just created ISO image is added as a CD-ROM and the graphic output is disabled. To see kernel boot messages, a serial port is attached and the output gets written into a temporary log file. While the guest is running, “tail -f” can be used to watch it boot. The last line is important, since it tells Xen how to emulate the Time-Stamp Counter (TSC) of the guest. Native mode has to be used, otherwise the time measures are wrong. A complete list of domain configuration options can be found in the xl.cfg manual file. To start the domain, xl has to be invoked with the following arguments, where domain_config is the configuration file created above.

xl create /path/to/domain_config

Create a wrapper script

The xen_hvm branch on Gitlab includes a wrapper script, which does all the steps shown above automatically. After building HermitCore it can be found in the following directory

HermitCore/build/local_prefix/opt/hermit/tools/xen-single-kernel.sh

and can be used as follows:

Usage: xen-single-kernel.sh -hvlknmc args

This script starts a HermitCore application as a single kernel HVM Domain in Xen. Make sure to destroy your domain when you are finished!

Args:
  -l  Path to hermit loader
  -k  Path to hermit application
  -n  Only create files, do not start
  -m  Domain memory in MB (Default 1024)
  -c  Number of CPU cores (Default 1)
  -v  print VERSION
  -h  this help screen

Example:
  xen-single-kernel.sh -l bin/ldhermit.elf -k x86_64-hermit/extra/tests/hello -m 512

Conclusion

HermitCore is running very well as a HVM guest in Xen. The recorded output of booting the “hello world” application can be seen below.

Hello world boot process

The only thing not working, is the ability to use multiple CPU cores. When trying to start additional CPU cores, HermitCore fails to bring them online via an Inter-processor Interrupt (IPI) which then leads to an infinite loop while waiting for these cores to come online. A possible way to make this work would be to implement hypercalls, which will be discussed in the following posts. With the help of a special hypercall, it is possible to make Xen start the additional CPU cores. Also, boot times are much slower compared to all other variants of running HermitCore. I will provide a detailed performance comparison in a future post.

This is it for part two. I hope you found it interesting again and will be back for the next one where i will show you my process of trying to make HermitCore work as a completely paravirtualized guest in Xen.

So long, Jan :-)