JanMa's Blog

Deploy systemd units in your Nomad cluster

Sun, 10 Jan 2021 00:00:00 +0000

Back in 2014 before Kubernetes even existed, CoreOs included a simple cluster-scheduler called Fleet. With Fleet you could aggregate your individual machines into a pool of resources and deploy systemd unit files to them. You could choose to either run your units globally on all machines at the same time or limit them to a set of hosts. The idea behind it was to treat your machines as if they would share an init system.

In 2018 Fleet was removed from CoreOs in favor of Kubernetes and is since then no longer maintained. Nevertheless the idea of being able to define a systemd unit and deploy it on a set of machines still seems like it could be useful. So after some tinkering I came up with a way to do exactly that using Nomad and my systemd-nspawn driver.

I am going to show you how to deploy a simple systemd unit for Consul running inside a vanilla Debian image into your Nomad cluster.

The unit file

Using the template stanza inside a Nomad job file, we will render a systemd unit for Consul in the local task directory.

template {
  data = <<EOH
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=[[ env "NOMAD_TASK_DIR" ]]/consul/consul agent -dev -bind '{{ GetInterfaceIP "host0" }}' -client '{{ GetInterfaceIP "host0" }}'
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOH
  destination = "local/systemd/consul.service"
  left_delimiter = "[["
  right_delimiter = "]]"
}

The above generates a simple unit file which runs Consul in dev mode into the local/systemd directory inside the started task. Consul is instructed to bind it’s addresses to the host0 interface which is available inside a systemd-nspawn container running with private networking enabled. To not interfere with the command line arguments of Consul, we tell Nomad to use [[ and ]] to delimit templating commands instead of the usual {{ and }}.

To download the Consul binary, we make use of the artifact stanza.

artifact {
  source = "https://releases.hashicorp.com/consul/1.9.0/consul_1.9.0_linux_amd64.zip"
  destination = "local/consul"
}

With this unit file rendered, and the necessary binary downloaded, we need a way to enable it on startup. Also we need to figure out how to make systemd load a custom unit file from the local task directory instead of /etc/systemd/system

Enabling the unit

The usual way to enable a systemd unit is to run systemctl enable <unit name>. It will create a symbolic link inside the /etc/systemd/system/multi-user.target.wants pointing to your unit file. Another way to enable a unit file without running a command, is to create a drop-in file for the multi-user target. In this file you need to define a Wants section which contains your unit name. This ensures that your unit is started before the multi-user.target, the same way as using systemctl enable does.

Using another template stanza in our job file, we can use the second method to enable the unit file we rendered to the local task directory.

template {
  data = <<EOH
[Unit]
Wants=consul.service
EOH
  destination = "local/systemd/multi-user.target.d/wants.conf"
}

Loading systemd units from a different path

Systemd includes a nice little feature with allows you to specify additional paths from where unit files are loaded on startup. All you need to do, is to set the environment variable SYSTEMD_UNIT_PATH to the directory containing your files. If it ends with a :, the usual load paths will be appended to the content of the variable. This is similar to how you set the PATH variable inside your shell.

To set this, we simply need to make sure our systemd-container is started in boot mode. Then all environment variables we pass to it, will be available to systemd on startup. Using boot mode is the default behavior of the systemd-nspawn task driver so we only need to define the image we want to use and the mentioned environment variable.

config {
  image = "consul"
  image_download {
    url = "https://cloud.debian.org/images/cloud/buster/20201214-484/debian-10-generic-amd64-20201214-484.qcow2"
    force = true
    type = "raw"
  }
  environment = {
    SYSTEMD_UNIT_PATH = "${NOMAD_TASK_DIR}/systemd:"
  }
}

The complete job

With all of the above in place, the complete job file now looks like this

job "consul" {
  datacenters = ["dc1"]
  type = "service"
  group "linux" {
    count = 1

    task "consul" {
      driver = "nspawn"
      config {
        image = "consul"
        image_download {
          url = "https://cloud.debian.org/images/cloud/buster/20201214-484/debian-10-generic-amd64-20201214-484.qcow2"
          force = true
          type = "raw"
        }
        environment = {
          SYSTEMD_UNIT_PATH = "${NOMAD_TASK_DIR}/systemd:"
        }
      }

      artifact {
        source = "https://releases.hashicorp.com/consul/1.9.0/consul_1.9.0_linux_amd64.zip"
        destination = "local/consul"
      }

      template {
        data = <<EOH
[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=[[ env "NOMAD_TASK_DIR" ]]/consul/consul agent -dev -bind '' -client ''
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOH
        destination = "local/systemd/consul.service"
        left_delimiter = "[["
        right_delimiter = "]]"
      }

      template {
        data = <<EOH
[Unit]
Wants=systemd-networkd.service systemd-resolved.service consul.service
EOH
        destination = "local/systemd/multi-user.target.d/wants.conf"
      }
    }
  }
}

If you deploy the job inside your Nomad cluster and spawn a shell inside the task, you can see that the unit file defined in the job file is properly loaded on startup.

➜  ~ nomad exec -job consul /bin/bash
root@buster:/# systemctl status consul
● consul.service - "HashiCorp Consul - A service mesh solution"
   Loaded: loaded (/local/systemd/consul.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2021-01-17 19:20:59 CET; 2min 15s ago
     Docs: https://www.consul.io/
 Main PID: 36 (consul)
   CGroup: /system.slice/consul.service
           └─36 /local/consul/consul agent -dev -bind  -client 

Jan 17 19:20:59 buster consul[36]:     2021-01-17T19:20:59.750+0100 [INFO]  agent.server: member joined, marking health alive: member=buster
Jan 17 19:20:59 buster consul[36]:     2021-01-17T19:20:59.791+0100 [INFO]  agent.server: federation state anti-entropy synced
Jan 17 19:20:59 buster consul[36]:     2021-01-17T19:20:59.826+0100 [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Jan 17 19:20:59 buster consul[36]:     2021-01-17T19:20:59.826+0100 [INFO]  agent: Synced node info
Jan 17 19:21:01 buster consul[36]:     2021-01-17T19:21:01.771+0100 [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Jan 17 19:21:01 buster consul[36]:     2021-01-17T19:21:01.771+0100 [DEBUG] agent: Node info in sync
Jan 17 19:22:53 buster consul[36]:     2021-01-17T19:22:53.841+0100 [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
Jan 17 19:22:53 buster consul[36]:     2021-01-17T19:22:53.841+0100 [DEBUG] agent: Node info in sync
Jan 17 19:22:59 buster consul[36]:     2021-01-17T19:22:59.699+0100 [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="buster.dc1 (Addr: tcp/192.168.74.222:8300) (DC: dc1)"
Jan 17 19:22:59 buster consul[36]:     2021-01-17T19:22:59.700+0100 [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="buster (Addr: tcp/192.168.74.222:8300) (DC: dc1)"
root@buster:/# 

And that’s all there is to it :-). I hope you find this as useful as I do

Jan

Orchestrating containers with Nomad and systemd-nspawn

Sun, 08 Mar 2020 00:00:00 +0000

In my last post I took a deeper look into systemd-nspawn and how to use it to run containers. Afterwards I decided to figure out the logical next step of how to orchestrate those containers. This is what this post is all about :-). I will show you how to use HashiCorp Nomad together with a custom plugin I wrote for it to orchestrate systemd-nspawn containers across multiple hosts.

About Nomad

If you have never heard of Nomad or have never used it before, I can recommend you to read the Introduction to Nomad guide. To quote the docs:

Nomad is a flexible workload orchestrator that enables an organization to easily deploy and manage any containerized or legacy application using a single, unified workflow. Nomad can run a diverse workload of Docker, non-containerized, microservice, and batch applications.

In version 0.9 Nomad introduced a plugin framework which allows you to extend it’s functionality to add new Task Drivers and Device plugins. Adding a new Task Driver allows you to execute workloads which are not manageable by the included ones like Docker or Java. You can find a list of already community supported Task Drivers in the Nomad Docs. Those include for example:

LXC
Podman
Firecracker

In the last few months I got acquainted with the new plugin framework and wrote a custom Task Driver for systemd-nspawn. I recently released version 0.1.0 and it is now in a state where I feel it is ready to be shared with the world :-). You can find the code at GitHub.

For the rest of this post, I am going to assume that you have read the guide I linked above or are otherwise already acquainted with Nomad and it’s terminology. Also you should have Nomad installed somewhere in your PATH.

Using the `nspawn` plugin

To build the Task Driver for systemd-nspawn, you need a recent version of Go installed. Checkout the repository from GitHub then then simply run go build.

git clone https://github.com/JanMa/nomad-driver-nspawn.git
cd nomad-driver-nspawn
go build -mod=vendor

This will produce a binary called nomad-driver-nspawn which can be used as a plugin by Nomad. The easiest way to test it, is to use Nomad in development mode. This starts a single node Nomad cluster on your machine, with the local agent acting as client and server. From the git root run

sudo nomad agent -dev -plugin-dir=$(pwd) -config=example/config.hcl

and visit http://127.0.0.1:4646 to see the Nomad web UI. If you click on the Clients tab, you should see your local machine and the nspawn driver showing up as healthy in the client details.

This means you are ready to start your first task.

Two simple example jobs

The driver repo contains two examples which show very basic configurations. The first one is located in the example/Debian folder. There you will find a very simple debian.hcl file which will start a plain Debian/Buster image that does exactly nothing (except running systemd).

job "debian" {
  datacenters = ["dc1"]
  type = "service"
  group "linux" {
    count = 1
    task "debian" {
      driver = "nspawn"
      config {
        image = "example/Debian/image"
        resolv_conf = "copy-host"
      }
    }
  }
}

It uses the image located in the example/Debian/image folder inside the git repo and copies the /etc/resolv.conf file of your host into the container to enable DNS resolution.

Before you are able to run the job file, you need to build the container image first. If you haven’t already, install mkosi on your machine, open another shell and run the following command inside the example/Debian folder.

sudo mkosi

Mkosi will parse the mkosi.* files in the folder and produce an image sub-folder containing a Debian/Buster file tree. Now you can start the Task by executing:

nomad run debian.hcl

If you take a look at the output of nomad status, you should see a debian job with the status running. The container should also show up if you call machinectl list.

$ nomad status
ID      Type     Priority  Status   Submit Date
debian  service  50        running  2020-03-08T09:27:20+01:00

$ machinectl list
MACHINE                                     CLASS     SERVICE        OS     VERSION ADDRESSES
debian-8eeb9876-1195-413c-4433-55dcd779f586 container systemd-nspawn debian 10      192.168.60.38…

1 machines listed.

After the job has started you can attach a shell to the container or run any command you want in it. Have a look at the output of nomad status debian and copy the allocation ID from the last line of output. Then run

nomad alloc exec <alloc ID> /bin/bash

to start a shell inside the container. You could also do the same thing by running machinectl shell <machine-name>. Note that this only works if the container was started with the boot option set to true (which it is by default).

Since this example is somewhat useless, let’s move on to the next one. In the example/Nginx folder you will find a nginx.hcl file which is a little more useful than the previous one.

job "nginx" {
  datacenters = ["dc1"]
  type = "service"
  group "linux" {
    count = 1
    task "nginx" {
      driver = "nspawn"
      config {
        image = "example/Nginx/image"
        resolv_conf = "copy-host"
        command = [
          "/bin/bash", 
          "-c", 
          "dhclient && nginx && tail -f /var/log/nginx/access.log"
          ]
        boot = false
        process_two = true
        port_map {
          http = 80
        }
      }
      resources {
        network {
          port "http" {
            static = "8080"
          }
        }
      }
    }
  }
}

The job file uses the image located in example/Nginx/image and copies the /etc/resolv.conf as before. It also sets boot to false and in turn process_two to true, which causes the Bash script configured in the command stanza to be run as process two inside the container. The container’s port 80 will be forwarded to port 8080 on your host.

To start the job, build the image the same way as before by executing sudo mkosi and then run:

nomad run nginx.hcl

When calling nomad status, you will now see an additional nginx job running. You will also see another machine in the output of machinectl list. If you’d try to start a shell inside the container now, Nomad will exit with an error since this container was not started with the boot parameter set to true.

Because the job exposes port 80 of the container to your hosts’ port 8080 you can curl it to see if Nginx is actually running. One thing to note here is that systemd-nspawn does not forward exposed ports to your loopback interface. So accessing 127.0.0.1:8080 will not work.

curl http://<your-ip>:8080 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   612  100   612    0     0   149k      0 --:--:-- --:--:-- --:--:--  199k

Since the last command in the containers Bash script is following the Nginx log file inside the container, you are able to see it when accessing the containers logs via Nomad.

nomad alloc logs <alloc ID>
192.168.1.226 - - [04/Mar/2020:22:13:48 +0100] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0"
192.168.1.226 - - [06/Mar/2020:21:41:56 +0100] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0"

Those examples are of course very simple and not suited for any real world workloads. Make sure to have a look at the drivers README page for all options it currently supports. The naming is kept close to the names of the underlying systemd-nspawn arguments.

Conclusion

In it’s current state the driver should allow you to run reasonably complex workloads with it, but it is by no means finished. For example I am trying to figure out a way to allow executing commands via nomad alloc exec in containers which are not started with the boot parameter. I also want to add support for network modes which allow you to start multiple containers in the same isolated network namespace.

Also there’s the problem of image distribution which needs to be solved somehow. Manually building container images on your hosts before being able to start tasks on them isn’t really convenient. You can currently circumvent it a bit by using the artifact stanza in your job file to download an image from some server before starting a task though. Make sure to have a look at nspawn.org for some nice pre-built container images you could use.

If you find issues or a bug in the driver or have general questions about it, please open an Issue at GitHub and I will try to help you as good as I can :-)

Jan

Running containers with systemd-nspawn

Sun, 13 Oct 2019 00:00:00 +0000

I recently discovered that apart from running services, scheduling timers, configuring your network interfaces, resolving names and a lot more you can also run containers with systemd using systemd-nspawn. This was completely new to me so I decided to take a deeper look into the necessary steps to get this up and running. First I looked into how to build images suitable for systemd-nspawn and then at the different ways to run and manage containers with the help of builtin tools. After reading this post you will hopefully also have a rough understanding about how this works and are able to run simple workloads in containers yourself using only systemd.

Prerequisites

To use systemd-nspawn you can install it on Debian based distributions via:

apt install systemd-container

On Arch based distributions it comes already pre-packaged with systemd. You also need to enable and start systemd-networkd.service and systemd-resolved.service so networking and name resolution work inside the containers.

Building an image

There are several ways in which you can build an image for use with systemd-nspawn depending on the distribution you want to use. In this post I am going to use Debian but you can of course use any distribution you like. You just have to make sure it contains a valid /etc/os-release file. It is also helpful if the distribution it uses systemd as the init system as well, but not necessary. systemd-nspwan will also run any other init system it finds inside the filesystem tree.

The default way to build images for Debian is to use debootstrap. Creating a minimal image based on the latest stable version Buster can be done by executing:

debootstrap --include=systemd-container stable /var/lib/machines/Buster

This creates a filesystem tree inside the /var/lib/machines/Buster directory which you can use with systemd-nspawn. To make everything work completely, you have to perform some post installation steps.

# systemd-nspawn -D /var/lib/machines/Buster
Spawning container Buster on /var/lib/machines/Buster.
Press ^] three times within 1s to kill container.
root@Buster:~# systemctl enable systemd-networkd
root@Buster:~# systemctl enable systemd-resolved
root@Buster:~# ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
root@Buster:~# ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
root@Buster:~# mkdir /etc/systemd/resolved.conf.d
root@Buster:~# echo "[Resolve]" > /etc/systemd/resolved.conf.d/dns.conf
root@Buster:~# echo "DNS=1.1.1.1 8.8.8.8" >> /etc/systemd/resolved.conf.d/dns.conf
root@Buster:~# echo "pts/0" >> /etc/securetty
root@Buster:~# exit
logout
Container Buster exited successfully.

Let’s go through this step by step. I first spawned a shell inside the newly created image. Then I needed to enable systemd-networkd and systemd-resolved in order to get networking and name resolution inside the container working properly. For this I also linked the stub-resolv.conf generated by systemd-resolved to /etc/resolv.conf and configured the DNS servers which systmed-resolved will use. Otherwise the running container cannot resolve anything. The DNS servers are configured inside /etc/systemd/resolved.conf.d/dns.conf. As a last step, I added pts/0 to /etc/securetty to enable root logins.

To make this process more automated, you can also use mkosi. It is a wrapper around debootstrap, pacstrap and zypper to create minimal, legacy free OS images. To install mkosi, clone the git repo somewhere, open a shell in it and simply run the install script.

git clone https://github.com/systemd/mkosi.git
cd mkosi
sudo setup.py install

For detailed information about how to use it, see the man page. Now create an empty directory somewhere and create the following files in it:

# tree
.
├── mkosi.default
└── mkosi.postinst

# cat mkosi.default

[Distribution]
Distribution=debian
Release=buster

[Output]
Format=directory
Bootable=no
Hostname=buster
Output=/var/lib/machines/buster

[Validation]
Password=root

[Packages]
Packages=
	iputils-ping
	systemd-container
	iproute2

# cat mkosi.postinst

#!/bin/sh
# make sure systemd-networkd and systemd-resolved are running
systemctl enable systemd-networkd
systemctl enable systemd-resolved
# make sure we symlink /run/systemd/resolve/stub-resolv.conf to /etc/resolv.conf
# otherwise curl will fail
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
# Configure global DNS servers
mkdir /etc/systemd/resolved.conf.d
echo "[Resolve]" > /etc/systemd/resolved.conf.d/dns.conf
echo "DNS=1.1.1.1 8.8.8.8" >> /etc/systemd/resolved.conf.d/dns.conf
# set pts/0 in /etc/securetty to enable root login
echo "pts/0" >> /etc/securetty

mkosi will read the mkosi.default file for the settings of the image. According to the file, it will create a directory at /var/lib/machines/buster containing a Debian/Buster filesystem tree, make it not bootable inside a virtual machine and set the host name and root password . It will also install some additional packages. There are actually a lot more options you could use but I will leave it rather simple for this post. After the image was created, mkosi will run the mkosi.postinst script inside the image which performs all of the steps just done by hand when using debootstrap. Make sure to set the executable flag on the file after creating it.

The nice thing about mkosi is that you can easily and in an automated fashion create OS images for a number of different distributions. Now that we have created an image, it is time to run it.

Running the image

systemd-nspawn can either be invoked via the command line or run as a system service in the background. In the service mode, each container runs as it’s own service instance using a provided systemd-nspawn@ unit template. I will first look at how to invoke it via the command line to get a better understanding about how it works and then I will use the provided unit template for a more automated approach.

There are actually three different ways you can run an image with systemd-nspawn which all work slightly different. The default way is to boot the image using it’s init system just like you would boot a VM. It is important to note here that systemd-nspawn does not boot a kernel and doesn’t start a VM. Using the boot mode will provide you with an OS container that is running multiple processes as well as it’s own init system. You can compare this mode of operation to LXC containers or BSD jails. To use it, the --boot or -b flag need to be passed when invoking it. This is the default mode of operation when using the systemd-nspawn@ unit template.

systemd-nspawn --boot -D /var/lib/machines/buster

The command above will boot the image and present you with a login shell. If you followed the steps above to build the image, you can now login with the root user and password root to look around a bit. You will notice that the container shows the same interface names and IP addresses as your host because network separation was not enabled. Any network service started in this container or port that will be exposed, will directly be available on the IPs of the host.

Instead of full-fleged OS containers, you can also start something more similar to an application container which you might now from Docker or RKT. You can either start an application directly as PID 1 by passing no extra flag at all or run a stub init process which will then start the application by passing --as-pid2. Note that not all applications are suited to run as PID 1 since they have to meet a few special requirements that the PID 1 process has. For example they need to reap all processes spawned by it and also implement sysvinit compatible signal handling. Shells are generally able to satisfy these requirements but for all other applications is recommended to use the --as-pid2 switch.

To start a shell inside the created image running as PID 2, run the following command:

systemd-nspawn -a -D /var/lib/machines/buster /bin/bash

A big caveat in this mode of operation is, that name resolution does not seem to work properly (at least I could not get it working). If this is an issue for the application you want to run I would recommended you to use the boot mode. The man page has a nice comparison of the three modes

Switch	Explanation
Neither –as-pid2 nor –boot specified	The passed parameters are interpreted as the command line, which is executed as PID 1 in the container.
–as-pid2 specified	The passed parameters are interpreted as the command line, which is executed as PID 2 in the container. A stub init process is run as PID 1.
–boot specified	An init program is automatically searched for and run as PID 1 in the container. The passed parameters are used as invocation parameters for this process.

UPDATE:

The issues with name resolution in some containers can be explained by the way systemd-nspawn handles the /etc/resolv.conf file. It is configured by the --resolv-conf command line flag:

If set to “auto” the file is left as it is if private networking is turned on (see –private-network). Otherwise, if systemd-resolved.service is connectible its static resolv.conf file is used, and if not the host’s /etc/resolv.conf file is used. In the latter cases the file is copied if the image is writable, and bind mounted otherwise. […] Defaults to “auto”.

To use the same DNS servers in the container as on the host, set it to either copy-host or bind-host.

Networking

In general it can be a good idea to contain the container in a private network so you don’t have to worry about which ports it exposes unless you explicitly forward them. To do this, systemd-nspawn offers a variety of options which differ in complexity. To simply put a container inside it’s own private /28 subnet you have to pass the --network-veth or -n option. This will create a virtual ethernet link between the container and the host. Inside the container, it will be available as host0 and on the host side it will be named after the container, prefixed with ve-. systemd-networkd comes with a default configuration to set up the virtual interface on the host and inside the container as well if it is enabled and running on both. It also takes care of setting up DHCP on the link as well as the necessary routing options. A container with private networking can be started like this:

systemd-nspawn -bD /var/lib/machines/Buster -n

Note: If you are also using docker on your system, you have to do some tweaking of iptables rules so the container can communicate with the outside world. docker changes the default behavior of iptables so you have to allow in- and outgoing traffic on the created virtual interface. Example iptables rules can be found in the paragraph below.

Managing containers

If you want to run containers via systemd-nspawn in a more automated and management friendly fashion which is similar to how you would run docker containers, you can make use of machinectl which also ships with systemd. It uses the systemd-nspawn@ unit template mentioned above to start containers with sensible default settings. Those are:

ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot \
--link-journal=try-guest --network-veth -U \
--settings=override --machine=%i

I did not cover all of them so make sure to look them up in the man page :-).

To start a container, you can first have a look at all images that are available. machinectl searches images stored in /var/lib/machines/, /usr/local/lib/machines/, /usr/lib/machines/ and /var/lib/container/.

# machinectl list-images 
NAME                                TYPE      RO USAGE CREATED                      MODIFIED                    
buster                              directory no   n/a n/a                          n/a                         

1 images listed.

Then you can simply run machinectl start buster and it will invoke the unit template with the used image.

# machinectl 
MACHINE CLASS     SERVICE        OS     VERSION ADDRESSES     
buster  container systemd-nspawn debian 10      192.168.68.28…

1 machines listed.

You can then use machinectl login or machinectl shell to login to the running container and do things or machinectl status to check the processes running inside your container. What is also pretty neat is that you can use journalctl -u systemd-nspawn@buster on your host to see all log output of the container.

As mentioned above, if you are also running docker on your system, you have to create a few iptables rules so your container can talk to the outside when you run it with private networking enabled. The easiest way to do this, is to create an override file for the systemd unit template via systemctl edit systemd-nspawn@ and adding the following content:

[Service]
ExecStartPre=-/usr/bin/iptables -A FORWARD -o ve-%i -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT ; \
-/usr/bin/iptables -A FORWARD -i ve-%i ! -o ve-%i -j ACCEPT ; \
-/usr/bin/iptables -A FORWARD -i ve-%i -o ve-%i -j ACCEPT
ExecStopPost=-/usr/bin/iptables -D FORWARD -o ve-%i -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT ; \
-/usr/bin/iptables -D FORWARD -i ve-%i ! -o ve-%i -j ACCEPT ; \
-/usr/bin/iptables -D FORWARD -i ve-%i -o ve-%i -j ACCEPT

It will invoke iptables before starting and after stopping the container to add and delete the necessary rules for the container.

Configuration per container

If you want to customize the options a container is started with using machinectl you can create a .nspawn file next to your image with the same name. On startup it will be parsed by systemd-nspawn and possibly override the default settings of the unit template. Have a look at the systemd.nspawn man page for the options. To forward port 80 of the buster container to port 8080 on the host, you could create the following buster.nspawn file in /etc/systemd/nspawn. It cannot be put next to the image since some options are privileged and therefore need to be set inside /etc/systemd/nspawn to be applied. Information about which options are privileged can also be found inside the man page.

[Network]
Port=8080:80
VirtualEthernet=yes

After creating the config file and starting the container again, port 80 on the container will be forwarded to port 8080 on your host. It is important to note that systemd-nspwan will not forward the port to your loopback interface. So it won’t be available via 127.0.0.1:8080 or localhost:8080. This caused quite some confusion for me :-)

That’s it for now. I hope I could give you a small and understandable introduction on how to run containers with the help of systemd-nspawn. I am currently trying to figure out how you could use systemd-nspawn with existing workload orchestrators like HashiCorp Nomad so stay tuned :-)

Jan

Talk: Evolution of a Microservice Infrastructure

Mon, 10 Jun 2019 00:00:00 +0000

A feew weeks back, I had the pleasure to give my first talk on a conference. I attended the Open Source Datacenter Conference (OSDC) in Berlin where I talked about the evolution of our microservice infrastructure at REWE digital.

Since the end of last year, we performed two very big changes to our microservice environment. First we changed our reverse proxy from Nginx to Traefik and afterwards, our container orchestrator from Docker Swarm (standalone) to Nomad.

In my talk I gave an overview of how this environment we build looks like and what problems we faced that led us to change these core parts of our infrastructure.

If you are interesed in the slides, you can find them at Slideshare

Jan

A hypervisor in a box

Sat, 02 Feb 2019 00:00:00 +0000

Soon after I started to work on my bachelors thesis, I had to figure out a practical and portable way to run a Xen hypervisor on which I could test my development builds. Initially I just created a simple Ubuntu virtual machine on my Desktop PC and installed the latest available version of Xen by hand. Unfortunately at the time Ubuntu only shipped Xen until version, 4.9 but I needed access to features that were only available in versions 4.10 and later.

So I switched to Alpine Linux which has a very nice version with the latest Xen already installed and set up to work as dom0. This served me well for some time but things quickly became messy once I started to also develop on my Notebook. Keeping manually set up VMs in sync across multiple machines turned out to be very difficult and annoying. I had some prior experience with Vagrant, that’s why I decided to create a Vagrant Box which includes an already configured installation of Xen.

Vagrant is an open source tool for building and managing virtual machine environments based on VirtualBox, VMWare, KVM and many more. It is developed by HashiCorp and was first released in 2010.

With the help of Vagrant, it is possible to create the same exact virtual machine containing the Xen hypervisor on different computers. Every branch of my HermitCore fork contains a vagrant directory, in which the configuration for the virtual machine is stored. Using the Vagrantfile inside this directory, it is possible to create a virtual machine based on the latest Ubuntu version 18.04. Xen and all it’s dependencies get automatically installed on the creation of the machine.

The Vagrantfile for my setup looks like this:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu1804"
  config.vm.hostname = "xen"
  config.vm.define "xen"
  config.vm.network :private_network, :ip => "192.168.50.11"
  config.vm.synced_folder '../', '/HermitCore', type: "rsync", rsync__exclude: ".git/"
  config.vm.provider :libvirt do |libvirt|
    libvirt.cpus = 2
    libvirt.cpu_mode = "host-model"
    libvirt.nested = true
    libvirt.memory = 2048
    libvirt.keymap = "de"
  end
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
    vb.cpus = "2"
    vb.customize ["modifyvm", :id, "--paravirtprovider", "kvm"]
    vb.customize ["modifyvm", :id, "--hwvirtex", "on"]
    vb.customize ["modifyvm", :id, "--nestedpaging", "on"]
    vb.customize ["modifyvm", :id, "--cpu-profile", "host"]
    vb.customize ["modifyvm", :id, "--uart1", "0x3F8", "4"]
    vb.customize ["modifyvm", :id, "--uartmode1", "file",  "./xen.log"]
  end
  config.vm.provision "shell", inline: <<-SHELL
     export DEBIAN_FRONTEND=noninteractive
     apt-get update
     apt-get upgrade -qy
     apt-get install libyajl2 qemu-system-x86 xorriso bridge-utils -qy
     apt-get install /HermitCore/vagrant/files/xen-upstream-4.11.0.deb -qy
     ldconfig
     mv /etc/grub.d/10_linux /etc/grub.d/50_linux
     mv /HermitCore/vagrant/files/grub /etc/default/grub
     mv /HermitCore/vagrant/files/50-vagrant.yaml /etc/netplan/50-vagrant.yaml
     update-grub2
     systemctl enable xen-qemu-dom0-disk-backend.service
     systemctl enable xen-init-dom0.service
     systemctl enable xenconsoled.service
     systemctl enable xendomains.service
     systemctl enable xen-watchdog.service
     echo "Reboot VM!"
     reboot
  SHELL
end

You can choose to either start a VM via VirtualBox or libvirt. Sadly VirtualBox has no support for nested virtualization, that’s why I’d recommend to use libvirt as the default virtualization provider. To use Vagrant with libvirt, you first have to install the necessary plugin:

vagrant plugin install vagrant-libvirt

Afterwards you can start the VM with:

vagrant up --provider=libvirt

This will spin up a Vagrant Box with 2 GiB of memory and two virtual CPUs. After Vagrant created the VM, the whole parent folder containing the Git repository including the built files of HermitCore will be copied into the virtual machine and are available inside the /HermitCore directory. It will also run the inline shell script that is included in the Vagrantfile. It installs a custom built version of Xen with debug output enabled and reboots the VM. Afterwards everything is set up and ready to go.

One very nice thing Vagrant brings with it, is to automatically rsync folders into the running Vagrant Box. So when I made some changes to HermitCore and built them, I only had to run

vagrant rsync

from within the vagrant folder and I could start the latest build inside the VM without much effort.

This setup proved to be very convenient to use, even across multiple systems. I just had to clone the repo, start the Vagrant Box and could dive right into developing. I hope you found this post interesting and will also consider to use Vagrant to provide you with a portable development environment for your projects

So long, Jan

Porting an unikernel to Xen: Wrapping up

Mon, 07 Jan 2019 00:00:00 +0000

Hello again and a happy new year :-)

This is the last post of my blog series on how to port an unikernel to Xen. Last time we had a look at the performance of my implementations and how it compared to the original one. The results looked very promising which led me to the conclusion that Xen is a valid choice when looking for a platform to run HermitCore on. In this post I will wrap thing up and provide a small summary about what I have done.

Over the course of the last posts, I showed how I extended the unikernel HermitCore in such a way that it can be executed in different ways as a guest in Xen. On the one hand as a fully virtualized guest whose hardware is completely emulated by Xen, which makes it possible to run practically unmodified operating systems. On the other hand as a paravirtualized guest whose OS has to be modified, but which promised better performance. First, I gave an insight into Xen to explain the conceptual changes that need to be made to a paravirtualized guest. I also provided a short introduction to unikernels and HermitCore.

The focus of my implementation was initially aimed at the support for the operation as a so-called HVM guest. I wrote a wrapper script to create a bootable ISO image from the unmodified HermitCore files and start it as a fully virtualized guest in Xen. Apart from supporting multiple CPU cores, this can be considered working completely.

The next thing I tried, was to modify HermitCore to work as a fully paravirtualized guest in Xen. I explained the necessary substantial changes at many crucial places in the source code in detail. Unfortunately it would have also been necessary to change the memory management in large parts, in order to fulfill the restrictions demanded by Xen. Since the implementation of the memory management represented a bachelor thesis on its own, I chose not to do this.

Instead, I implemented the necessary changes to run HermitCore as a hybrid PVH guest under Xen. This had the advantage that no changes to the memory management were necessary. I could adopt the already made changes to the source code with slight modifications and HermitCore was extended so that it is completely functional except for a few small restrictions.

Finally I compared the new HermitCore operating modes with the original modes using the benchmarks already included in HermitCore. It turned out that the new modes are able to keep up with the original implementation in terms of performance except for a few minor differences.

It would be interesting to see in future work, what would be necessary to implement the missing features in HermitCore to make it work completely under Xen. This includes support for multiple CPU cores, a working network driver and a working console as a PVH guest. It would also be interesting to see how extensive the changes to the memory management must be in order to run HermitCore as a completely paravirtualized guest.

Finally I can say that Xen is a very interesting additional platform for HermitCore. The effort required for porting was reasonable and the performance looks promising. Since Xen is one of the most used hypervisors in the cloud and HermitCore is also specially developed for cloud computing, it is certainly a useful addition.

I hope you enjoyed my blog post series and will be back for my future posts :-)

Jan

Porting an unikernel to Xen: Perfomance comparison

Sun, 16 Dec 2018 00:00:00 +0000

Hello,

after finishing the implementation part last time, this week it is time for some performance comparison. Although both the HVM and PVH port are not 100% feature complete, they are in a state where basic bench marking is very much possible. The results, as you will see, look very promising when comparing them with the ones of the original implementation.

Basic Benchmark
Stream Benchmark
Boot time
Conclusion

HermitCore includes a number of different benchmarks to determine the performance. This post compares the results of some of these benchmarks for HermitCore running in different environments. The following environments were tested:

KVM
HermitCore is started in a virtual machine on Linux using QEMU with KVM acceleration
QEMU
A virtual machine on Linux using only QEMU without acceleration
HVM
Running as a fully virtualized guest on Xen
PVH
A hybrid PVH guest running on Xen

For each test, the virtual machines were given the same amount of resources. They were started with one virtual CPU core and 512 megabytes of RAM. All tests were performed on the same machine, a Lenovo Thinkpad T470 with the following specifications:

Intel Core i5-7300U CPU with 2.6 GHz and 4 Cores
16 gigabytes of RAM
256 gigabytes SSD storage
Running Arch Linux

Basic Benchmark

First the overhead of a system call and a reschedule was measured. The basic benchmark invokes the system calls getpid and sched_yield up to 10.000 times after the cache has been warmed up. It measures how many CPU cycles the respective calls need on average. Getpid is the system call with the shortest runtime, it can be used to determine the general overhead of a system call. Sched_yield checks if another task is ready to be executed and switches to this task. The benchmark also checks how long it takes to allocate a megabyte of memory and how long the first write access to a page table takes.

System activity	KVM	QEMU	HVM	PVH
getpid	9	122	12	12
sched_yield	79	360	90	83
malloc	5858	51812	51311	86658
write access	3368	34626	42607	83368

Results of the Basic Benchmark

It is not surprising that HermitCore running on KVM shows the best performance overall. It is however interesting to see that a PVH and HVM guest can almost keep up with it in regards to system call performance. What is also surprising is that the memory access of a PVH guest is much slower than that of a HVM guest considering that both their mechanism for page table management is virtualized in hardware.

Stream Benchmark

The Sustainable Memory Bandwidth in Current High Performance Computers STREAM benchmark is a synthetic test written in Fortran to measure the performance of four distinct long vector operations. They represent the elementary operations on which vector codes are based and are specifically intended to eliminate data re-use. The results display the sustainable memory bandwidth in megabytes per second and the corresponding computation time for the four vector operations.

Name	Function	bytes per iteration
Copy	`a(i) = b(i)`	16
Scale	`a(i) = q * b(i)`	16
Sum	`a(i) = b(i) + c(i)`	24
Triad	`a(i) = b(i) + q * c(i)`	24

Functions used in the STREAM benchmark

Environment	Bandwidth MB/s	Avg time	Min time	Max time
		Copy
KVM	23342.8	0.009865	0.009596	0.014814
QEMU	5153.7	0.045812	0.043464	0.047941
HVM	24294.7	0.009369	0.009220	0.010860
PVH	24141.9	0.009469	0.009278	0.012305

		Scale
KVM	16556.5	0.013793	0.013529	0.017478
QEMU	1094.8	0.218594	0.204610	0.229119
HVM	17263.1	0.013157	0.012976	0.015088
PVH	17189.3	0.013252	0.013031	0.019612

		Add
KVM	19264.9	0.017679	0.017441	0.020491
QEMU	1562.1	0.225724	0.215092	0.237715
HVM	20038.9	0.016974	0.016767	0.018722
PVH	19955.0	0.017068	0.016838	0.022021

		Triad
KVM	19088.8	0.017928	0.017602	0.021669
QEMU	897.4	0.394932	0.374413	0.415066
HVM	19856.3	0.017111	0.016922	0.018756
PVH	19772.2	0.017232	0.016994	0.021154

Results of the STREAM benchmark

It is very surprising to see, that a HVM guest running in Xen outperforms all others in terms of possible memory bandwidth and corresponding computing time. Especially if considered that a HVM guest in the testing environment is actually a virtual machine running inside a virtual machine (the Xen hypervisor) running on top of Linux. Although a KVM and PVH VM achieve almost the same results with only one to four percent deviation. You can also see that the virtualization purely based on QEMU seems to be rather inefficient and slow.

Boot time

At last the time needed by the VMs to boot was compared. The included hello world test was run in all environments and the reported boot time was noted. This time is how long it takes HermitCore until it is able to start the hello world application.

Environment	Time in ms
KVM	80
QEMU	60
HVM	6140
PVH	80

Boot time for the different Environments

HermitCore takes about the same time for it in all environments. The only exception is a HVM guest. It takes this guest very long in comparison to detect and start the emulated devices it is provided by Xen, which results in a about 80 times slower boot time.

Conclusion

The results of the previous benchmarks show that when running as either a HVM or PVH guest in Xen, HermitCore is definitely able to perform as well as a the original implementation running as a KVM accelerated virtual machine in QEMU. There are some small exceptions, notably the very slow boot time of a HVM guest in comparison to all others and the long memory access times of a PVH and HVM guest in terms of CPU cycles, but the overall results are very similar.

I hope you will join me again next time for the last post of this series. There i will provide a short summary and some possible outlooks into the future.

So long Jan

Porting an unikernel to Xen: PVH guest

Sun, 09 Dec 2018 00:00:00 +0000

Welcome back to the fourth part of my blog post series. Last time i wrote about my process of trying to get HermitCore working as a fully paravirtualized PV guest in Xen. Unfortunately this didn’t work out for a number of reasons. The main one being hat i would have had to rewrite the memory management code of HermitCore. So i decided to try to get it working as a PVH guest. This worked out quite nice as you will read in this post. Once again, all source code discussed here can be found in a separate branch on GitLab.

Booting
Initializing Xen features
Creating multiboot information
Console output
Completing startup
Conclusion

The third guest type i discussed in my thesis is called a PVH guest. It can be viewed as a kind of hybrid between a HVM and a PV guest.

They work almost exactly as PV guests with one major exception: a PVH guest runs in ring 0 and has direct control over it’s page tables. This has several advantages. One of the biggest efforts when porting an operating system to Xen is the page table management code. As mentioned in the previous post, a lot of effort would have been needed to modify the existing page table and memory management code to work with the restrictions placed on a PV guest by Xen. In contrast, there is no special code needed for a PVH guest. It does need some modifications to work correctly with the Xen hypervisor but by far not as many as a PV guest.

Booting

When starting a PVH guest, Xen defines the following register state:

ebx
contains the physical memory address where the loader has placed the boot start info structure.
cr0
bit 0 (PE) must be set. All the other writable bits are cleared.
cr4
all bits are cleared.
cs
must be a 32 bit read/execute code segment with a base of ‘0’ and a limit of ‘0xFFFFFFFF’. The selector value is unspecified.
ds, es
must be a 32 bit read/write data segment with a base of ‘0’ and a limit of ‘0xFFFFFFFF’. The selector values are all unspecified.
tr
must be a 32 bit TSS (active) with a base of ’0’ and a limit of ’0x67’.
eflags
bit 17 (VM) must be cleared. Bit 9 (IF) must be cleared. Bit 8 (TF) must be cleared. Other bits are all unspecified.

All other processor registers and flag bits are unspecified. The OS is in charge of setting up it’s own stack, GDT and IDT. Xen starts the PVH guest in 32 bit mode with paging enabled, so the guest also has to provide a 32 bit entry point to Xen with the help of an ELF note. The domain builder will jump directly to the specified address in the boot code.

ELFNOTE "Xen",XEN_ELFNOTE_PHYS32_ENTRY,start

When HermitCore is not booting directly into 64 bit mode, it first has to run the included loader. Therefore all changes in the boot code have to be implemented in the entry.asm file of the loader.

The address of the start info struct gets passed to the guest in the ebx register instead of the multiboot information. This has to be saved. In addition only the mentioned ELF note has to be added. The resulting changes in entry.asm are therefore very minimal.

SECTION .mboot
global start
start:
    cli ; avoid any interrupt
    jmp stublet
...
SECTION .text
ALIGN 4
stublet:
    ; Initialize stack pointer
    mov esp, boot_stack
    add esp, KERNEL_STACK_SIZE - 16

    ; Safe Xen start info
    mov DWORD [xen_start_info], ebx
...
; jump to the boot processor's C code
extern main
jmp main
jmp $

align 4096
global shared_info, hypercall_page
shared_info:
    times 512 DQ 0
hypercall_page:
    times 512 DQ 0

SECTION .data

global mb_info, xen_start_info
ALIGN 8
mb_info:
    DQ 0
xen_start_info:
    DQ 0
    

A special hypercall_page and shared_info page have been added again as described in the previous post. Similar to a PV guest, the first thing a PVH guest has to set up are hypercalls and the shared_info page.

Initializing Xen specific features

Setting these up works a little different than in a purely paravirtualized guest. Hypercalls have to be specifically enabled and mapping the shared_info page requires a different hypercall. In addition, a hvm_start_info struct is passed to the guest instead of the start_info_t struct described previously.

The values in this struct are almost completely different to the values of the start_info_t struct but are nevertheless very important for the following steps in the boot process.

Hypercalls

As opposed to a PV guest, a PVH guest has to enable hypercalls before it can use them. The following code has been taken from Mini-OS.

#define XEN_CPUID_FIRST_LEAF 0x40000000
/*
 * Leaf 1 (0x40000x00)
 * EAX: Largest Xen-information leaf. All leaves up to an including @EAX
 *      are supported by the Xen host.
 * EBX-EDX: "XenVMMXenVMM" signature, allowing positive identification
 *      of a Xen host.
 */
#define XEN_CPUID_SIGNATURE_EBX 0x566e6558 /* "XenV" */
#define XEN_CPUID_SIGNATURE_ECX 0x65584d4d /* "MMXe" */
#define XEN_CPUID_SIGNATURE_EDX 0x4d4d566e /* "nVMM" */

static inline void wrmsrl(unsigned msr, uint64_t val)
{
    wrmsr(msr, (uint32_t)(val & 0xffffffffULL), (uint32_t)(val >> 32));
}

static void hpc_init(void)
{
    uint32_t eax, ebx, ecx, edx, base;

    for ( base = XEN_CPUID_FIRST_LEAF;
          base < XEN_CPUID_FIRST_LEAF + 0x10000; base += 0x100 )
    {
        cpuid(base, &eax, &ebx, &ecx, &edx);

        if ( (ebx == XEN_CPUID_SIGNATURE_EBX) &&
             (ecx == XEN_CPUID_SIGNATURE_ECX) &&
             (edx == XEN_CPUID_SIGNATURE_EDX) &&
             ((eax - base) >= 2) )
            break;
    }

    cpuid(base + 2, &eax, &ebx, &ecx, &edx);
    wrmsrl(ebx, (unsigned long)&hypercall_page);
    barrier();
}

To enable hypercalls, the guest first has to issue cpuid commands, until the register ebx, ecx and edx contain the XEN_CPUID_SIGNATURE values defined in the above listing. Then it can issue a wrmsr command to tell the hypervisor the address of it’s desired hypercall_page.

Shared info page

A PVH guest doesn’t have access to the same hypercalls a PV guest has. To map the shared_info page the HYPERVISOR_memory_op hypercall has to be used. It sets the page frame number (PFN) at which a specific page should appear in the guest’s address space.

shared_info_t *map_shared_info(void *p)
{
    struct xen_add_to_physmap xatp;

    xatp.domid = DOMID_SELF;
    xatp.idx = 0;
    xatp.space = XENMAPSPACE_shared_info;
    xatp.gpfn = PFN_DOWN((size_t)&shared_info);
    if ( HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp) != 0 )
        asm volatile ("hlt");

    return &shared_info;
}

The hypercall takes a xen_add_to_physmap struct and an operation as arguments which are defined in the memory.h header file.

Creating a multiboot information struct

As mentioned previously, HermitCore always expects a multiboot_info_t struct in the ebx register when booting. It contains a lot of important information about the environment HermitCore is running in and is needed in the whole startup process. Considering that Xen does not pass such a struct to the guest on startup, there are two possible ways to obtain the needed information.

One would be to rewrite the parts of the code that rely on the multiboot_info_t struct, to use information provided by Xen instead. This would have to be done for the loader and the kernel and would take a lot of changes. The second way is to simply gather all the information needed once at startup and create a multiboot_info_t struct, which then can be passed to the code that needs it. In view of it’s simpler implementation, i chose the second method and implemented a build_multiboot() function in the loader’s main.c file.

The multiboot_info_t struct is a rather large and complex structure which consists out of many variables and pointers to other structures. It’s structure can be seen above. Fortunately, HermitCore does not need all of the information which could be included in this struct. The build_multiboot function only has to create a multiboot_memory_map_t struct, a multiboot_module_t struct and set the correct flags. In addition to that, the function also calculates the CPU frequency with the help of the shared_info page.

Flags

The following flags need to be set for HermitCore to work properly:

MULTIBOOT_INFO_MEMORY
Information about the available memory is provided
MULTIBOOT_INFO_CMDLINE
Extra command line options are defined.
MULTIBOOT_INFO_MODS
There are modules passed to the operating system
MULTIBOOT_INFO_MEM_MAP
A full memory map is provided

To set these flags, the bits 1, 3, 4 and 7 have to be set to one in the flags variable of the multiboot_info_t struct

mb_tmp.flags = 0x00000001 | 0x00000004 | 0x00000008 | 0x00000040;

It is not needed to pass additional command line options to the kernel so the cmdline variable can be set to zero.

Memory map

To create a multiboot_memory_map_t struct the guest first has to issue the HYPERVISOR_memory_op hypercall to get the memory mapping from Xen. The hypercall returns a e820entry struct which has to be translated into a multiboot_memory_map_t struct.

/* PC BIOS standard E820 types and structure. */
#define E820_RAM          1
#define E820_RESERVED     2
#define E820_ACPI         3
#define E820_NVS          4
#define E820_UNUSABLE     5
#define E820_PMEM         7
#define E820_TYPES        8

struct __attribute__((__packed__)) e820entry {
    uint64_t addr;
    uint64_t size;
    uint32_t type;
};

Translating it is done in the following way:

//get memory map from Xen
struct xen_memory_map mmap; 
mmap.nr_entries = E820_MAX;
mmap.buffer = e820_map;
int rc = HYPERVISOR_memory_op(XENMEM_memory_map, &mmap);
if (rc){
	kprintf("Getting mmap failed!\n");
	HALT;	
}
kprintf("Memmap nr_entries: %d\n", mmap.nr_entries);
for ( int i = 0; i < mmap.nr_entries ; i++){
	kprintf("size: %ld addr: %lx type: %d\n", e820_map[i].size, e820_map[i].addr, e820_map[i].type);
	mboot_mmap[i].len = e820_map[i].size;
	mboot_mmap[i].addr = e820_map[i].addr;
	mboot_mmap[i].type = e820_map[i].type;
	mboot_mmap[i].size = sizeof(multiboot_memory_map_t)-sizeof(uint32_t);
}
mb_tmp.mmap_addr = (multiboot_uint32_t)&mboot_mmap;
mb_tmp.mmap_length = mmap.nr_entries * sizeof(multiboot_memory_map_t);
kprintf("mmap: 0x%lx mmap_length: 0x%lx\n", mb_tmp.mmap_addr, mb_tmp.mmap_length);    

Multiboot module information

When starting HermitCore as a PVH guest, Xen will first load the binary file of the HermitCore loader and pass the actual application that is intended to run as a ram disk. This is very similar to a HVM guest, where the application gets passed as a multiboot module directly. The virtual memory layout of a PVH guest is almost the same as for a PV guest, which is described here. The initial ram disk is located just after the relocated kernel image, which means that the ram disk starts at the beginning of the first page after the HermitCore loader. The correct start address can be determined by using the provided kernel_end variable, which contains the address where the loader ends, and calculating the start address of the next page from there.

The size of the ram disk can be calculated in a similar matter. The magic variable inside the hvm_start_info_t struct contains the address where the struct is located. Subtracting the start address of the ram disk from this address provides the size of the ram disk. This works because a PVH guest is not passed a list of allocated page frames so the hvm_start_info_t struct is located just behind the initial ram disk.

These two values are all that is needed by the build_multiboot function to create a multiboot_module_t struct.

// calculate the next page frame
uint64_t mod_addr = pfn_to_virt(PFN_UP((uint64_t)&kernel_end));
// calculate the size of the ram disk
uint64_t mod_len = (uint64_t)start_info_ptr->magic - mod_addr;
kprintf("module0 addr: 0x%lx\n", mod_addr);
kprintf("module0 len: 0x%lx\n", mod_len);
mmod[0].mod_start = mod_addr;
mmod[0].mod_end = mod_addr + mod_len;
mmod[0].cmdline = 0;
mmod[0].pad = 0;
mb_tmp.mods_count = 1;
mb_tmp.mods_addr = (multiboot_uint32_t)&mmod;    

CPU frequency

The last thing the build_multiboot function has to do, is to calculate the CPU frequency. Strictly speaking, this is not part of the multiboot_info_t struct but is nevertheless needed by HermitCore. Determining it at this point in the startup process is very convenient so that is why it is included into the build_multiboot function.

The equation to calculate the CPU frequency is very similar to the one used to calculate the current system time in a PV guest.

frequency=((10^9 << 32) ÷ tsc_to_system_mul) >> |tsc_shift|

The values needed can be gathered from the shared_info_t struct.

Console output

In theory it should have been possible to make a PVH HermitCore guest use the same virtual console a PV guest uses. Mapping the required shared pages and initializing events to communicate with the control domain works almost the same way. The necessary functions to initialize the console device and write data to it are also included in the xen_pvh branch but are not used. The reason for this is because when trying to write output to the virtual console, the data gets written correctly into the mapped shared memory page but the control domain is not notified and therefore displays no output.

Fortunately Xen provides a second way to display the output of a guest. There exists a HYPERVISOR_console_io hypercall which can be used to print data to an emergency debug console. To have access to this debug console, Xen has to be compiled from source with debug support enabled. I will provide a short description on how to do this in a following post.

Writing data to the emergency console is very easy. The HYPERVISOR_console_io hypercall takes three arguments to work.

HYPERVISOR_console_io(CONSOLEIO_write, strlen(buf), buf);

The first instructs the hypervisor to write data to the console. The second and third argument contain the string to be written and it’s length. For a more convenient usage i implemented a printf-like function xprintk in the same way as for a PV guest.

To read the data from the emergency console, the xl dmesg command has to be used inside the control domain, it will print all data written to the emergency console.

Completing startup

With the above features implemented, the HermitCore loader is able to complete it’s startup process and load the application binary. The kernel code doesn’t need much additional changes. After starting the application binary, hypercalls have to be enabled again to provide the possibility of console output and the shared_info page has to be mapped again. This works exactly the same way as in the loader. In addition to that, the detection of PCI and UART devices and the initialization of the network were disabled.

A PVH guest does not have access to any emulated hardware devices, including PCI and UART devices. So trying to initialize them only takes unnecessary time in the boot process (since none cane be found). This is the same reason why the network initialization is disabled. At the time of writing it is not possible to use networking when running HermitCore as a PVH guest on Xen. The drivers necessary to make it work, need to be ported from Mini-OS or Linux to HermitCore.

Console output from an application

To enable console output from applications to the emergency console, a small trick has been used. Normally, C-programs use printf or similar functions to display text output inside a terminal window. When they are compiled into HermitCore, they get linked against HermitCore’s included C library newlib. Inside this library, the function putchar, which is used by printf and similar functions to write a character to the standard output, is implemented in a way that each character that would be written to the terminal, gets written to the UART device instead. This is done by calling the write_to_uart function which is included in HermitCore. By modifying this function to also write the passed characters to the Xen emergency console, C-programs can still use the standard functions to print text.

static inline void write_to_uart(uint32_t off, unsigned char c)
{
    while (is_transmit_empty() == 0) { PAUSE; }

    if (uartport)
        outportb(uartport + off, c);

    // also write the output to Xen's emergency console
    xprintk("%c", c);
}   

Conclusion

A HermitCore PVH guest is working almost completely. There are only a few things missing that have to be implemented in the future, to consider it working 100 %. These include:

a working virtual console
working networking
support for multiple CPUs

The performance of a PVH guest is astonishing. It boots as fast as a HermitCore guest running on the KVM hypervisor even though it is a virtual machine, running inside a virtual machine running on Linux. The decision to move the focus of the implementation away from a purely paravirtualized PV guest to a hybrid PVH guest has clearly paid off. I will provide a more detailed performance comparison in the next post. I hope i kept it interesting and that you will be back next time.

Until then, Jan

Porting an unikernel to Xen: PV guest

Sun, 25 Nov 2018 00:00:00 +0000

Welcome back once again to my blog post series on how to port an unikernel to Xen. Last week i showed you how i got HermitCore running as a fully virtualized guest in Xen. This week it is finally time to get our hands dirty and to start modifying the operating system :-) . The source code for this post can be found in the xen_parav branch of my HermitCore fork.

ELF notes
Modifying the assembly boot code
Initializing XEN specific features
Time keeping
Events
Console output
Interrupts
Memory Management
Conclusion

To start HermitCore as a paravirtualized guest, changes have to be made to the boot code so it can be loaded by the generic ELF loader included in Xen. Since Xen boots paravirtualized guests directly into 64 bit mode, no changes have to be implemented in the ldhermit.elf loader binary and the compiled application binary can be started directly.

ELF notes

First, the standard Xen ELF notes have to be included, allowing the binary to be loaded by the Xen toolstack domain builder. When the guest is started, the application binary is read and the ELF PT_NOTE program header is parsed. The hypervisor looks in the .note sections of the ELF file for the “Xen” notes. The description fields are Xen specific and contain the required information to find out where the kernel expects its virtual base address, what type of hypervisor it can work with, certain features the kernel image can support and the location of the hypercall page, etc. All ELF note elements have the same basic structure:

The Name Size and Desc Size fields are integers, which specify the size of the Name and Desc fields (excluding padding). The Name field specifies the vendor who defined the format of the Note. Typically, vendors use names which are related to their project and/or company names. For instance, the GNU Project uses GNU as its name. The Type field is vendor specific, but it is usually treated as an integer which identifies the type of the note. The Desc field is also vendor specific, and usually contains data which depends on the note type.

Xen defines the following types for ELF notes:

XEN_ELFNOTE_INFO
XEN_ELFNOTE_ENTRY
XEN_ELFNOTE_HYPERCALL_PAGE
XEN_ELFNOTE_VIRT_BASE
XEN_ELFNOTE_PADDR_OFFSET
XEN_ELFNOTE_XEN_VERSION
XEN_ELFNOTE_GUEST_OS
XEN_ELFNOTE_GUEST_VERSION
XEN_ELFNOTE_LOADER
XEN_ELFNOTE_PAE_MODE
XEN_ELFNOTE_FEATURES
XEN_ELFNOTE_BSD_SYMTAB
XEN_ELFNOTE_HV_START_LOW
XEN_ELFNOTE_L1_MFN_VALID
XEN_ELFNOTE_SUSPEND_CANCEL
XEN_ELFNOTE_INIT_P2M
XEN_ELFNOTE_MOD_START_PFN
XEN_ELFNOTE_SUPPORTED_FEATURES
XEN_ELFNOTE_PHYS32_ENTRY

The definitions of the different types can be found in the elfnotes.h header-file. Mini-OS implements a macro to add ELF notes in the following way.

    #define ELFNOTE(name, type, desc)           \
    .pushsection .note.name               ; \
    .align 4                              ; \
    .long 2f - 1f       /* namesz */      ; \
    .long 4f - 3f       /* descsz */      ; \
    .long type          /* type   */      ; \
1:.asciz #name          /* name   */      ; \
2:.align 4                                ; \
3:desc                  /* desc   */      ; \
4:.align 4                                ; \
    .popsection

It can be called like this:

ELFNOTE(Xen, XEN_ELFNOTE_GUEST_OS, .asciz "Mini-OS-x86_64")
ELFNOTE(Xen, XEN_ELFNOTE_LOADER, .asciz "generic")

Unfortunately, Mini-OS compiles it’s assembly files with gcc, which uses the AT&T assembly syntax. HermitCore on the other hand, uses nasm which uses the Intel assembly syntax. Since they are not compatible the ELFNOTE macro needs to be rewritten. A very helpful comparison of the two styles can be found here. Without going into detail, this is an equivalent implementation in Intel assembly syntax for HermitCore:

%macro ELFNOTE 3 ; name, type, descr
    align 4
    dd %%2 - %%1
    dd %%4 - %%3
    dd %2
  %%1:
    dd %1
  %%2:
    align 4
  %%3:
    dd %3
  %%4:
    align 4
%endmacro

Which can be called like this:

SECTION .note
elf_notes:
  ELFNOTE "Xen",XEN_ELFNOTE_GUEST_OS,"HermitCore"
  ELFNOTE "Xen",XEN_ELFNOTE_GUEST_VERSION,"0.2.5"
  ELFNOTE "Xen",XEN_ELFNOTE_LOADER,"generic"
  ELFNOTE "Xen",XEN_ELFNOTE_XEN_VERSION,"xen-3.0"
  ELFNOTE "Xen",XEN_ELFNOTE_HYPERCALL_PAGE,hypercall_page
  ELFNOTE "Xen",XEN_ELFNOTE_ENTRY,_start
  ELFNOTE "Xen",XEN_ELFNOTE_FEATURES,0x3

With these notes added to the kernels entry.asm file, the Xen domain builder is able to detect the binary and tries to boot it. At this point the domain still dies almost instantly, so additional changes to the assembly boot code have to be made.

Modifying the assembly boot code

The initial boot time environment of a Xen PV guest is different from the normal initial mode of an x86 processor. Instead of starting with paging disabled in 16-bit mode, a PV guest is started in either 32 or 64 bit mode with paging enabled and runs on a first set of page tables provided by the hypervisor. These pages are set up to correspond to the required invariants and are loaded into the base register of the page table, but are not explicitly pinned.

The initial virtual and pseudo physical memory layout is described in the xen.h header file.

1. The domain is started within contiguous virtual-memory region
2. The contiguous region ends on an aligned 4MB boundary.
3. This the order of bootstrap elements in the initial 
   virtual region:
    a. relocated kernel image
    b. initial ram disk              [mod_start, mod_len]
       (may be omitted)
    c. list of allocated page frames [mfn_list, nr_pages]
       (unless relocated due to XEN_ELFNOTE_INIT_P2M)
    d. start_info_t structure        [register rSI (x86)]
       in case of dom0 this page contains the console info, too
    e. unless dom0: xenstore ring page
    f. unless dom0: console ring page
    g. bootstrap page tables         [pt_base and CR3 (x86)]
    h. bootstrap stack               [register ESP (x86)]
4. Bootstrap elements are packed together, but each is 
   4kB-aligned.
5. The list of page frames forms a contiguous 'pseudo-physical' 
   memory layout for the domain. In particular, the bootstrap 
   virtual-memory region is a 1:1 mapping to the first section 
   of the pseudo-physical map.
6. All bootstrap elements are mapped read-writable for the guest 
   OS. The only exception is the bootstrap page table, 
   which is mapped read-only.
7. There is guaranteed to be at least 512kB padding after the 
   final bootstrap element. If necessary, the bootstrap 
   virtual region is extended by an extra 4MB to ensure this.

To jump from the assembly boot code to the actual kernel C code, there are only a few things needed. When the guest is launched as explained above, the ESI or RSI register (depending on wether it is a 32 or 64 bit guest) contains a start_info_t structure which is needed later on and needs to be saved. Other than that, only a stack has to be set up. This simplifies the boot code a lot to the just few following lines:

SECTION .mboot
global _start
_start:
jmp start64

...

SECTION .ktext
align 4
start64:
cld                                 ; clear registers
add rsp, KERNEL_STACK_SIZE-16       ; set up stack
mov rdi, rsi                        ; pass start_info_t as 
extern hermit_main                  ; argument to hermit_main
call hermit_main                    ; jump into C-code
jmp $

After the above code ran, the instruction pointer is pointing at the hermit_main function defined in the kernels main.c file.

Initializing XEN specific features

The next things the guest has to set up are some XEN specific features. Upon starting, a PV guest gets passed a start_info_t structure which contains many important information for the operating system.

Particularly important are the nr_pages, shared_info and domU entries. They are needed for determining the assigned RAM, communication with the Xen hypervisor and console output. The assembly boot code passes the virtual address of the structure to the hermit_main function as the first argument.

Hypercalls

The environment presented to a Xen PV guest is not quite the same as that of a real x86 system. From the perspective of the operating system, the biggest difference is that it is running in ring 1 or ring 3 instead of ring 0. This means that it cannot perform any privileged instructions. In order to provide similar functionality, the hypervisor exposes a set of hypercalls that correspond to the instructions. A hypercall is conceptually similar to a system call. To request a service from the hypervisor, the guest calls a function in a shared memory page which gets mapped by the hypervisor.

First, a special page must be created in the assembly boot code.

global shared_info, hypercall_page
ALIGN 4096
shared_info:
    times 512 DQ 0

hypercall_page:
    times 512 DQ 0

Then the virtual address of the hypercall_page gets passed to the hypervisor in an ELF note

ELFNOTE "Xen",XEN_ELFNOTE_HYPERCALL_PAGE,hypercall_page

Hypercalls are issued by calling an address within this page. The following listing shows a macro that is used to call a hypercall without additional arguments.

extern char hypercall_page[4096];
#define _hypercall0(type, name)			                 \
({						                                 \
long __res;				                                 \
asm volatile (				                             \
"call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)"\
: "=a" (__res)			                                 \
:				                                         \
: "memory" );			                                 \
(type)__res;				                             \
})

A list of all hypercalls the guest can use, can also be found in the xen.h header file. Upon usage, the individual hypercalls will be explained in more detail.

Shared Info page

One of the first things a guest has to do is to set up the shared_info page. It contains a shared_info_t struct which holds valuable information about the virtual CPUs assigned to the guest, the system time and event channels which can be used to communicate with other domains. It is again defined in the xen.h header file. The machine address of the shared_info page is defined in the start_info_t struct. In order to map it into the virtual address space, the guest has to issue the HYPERVISOR_update_va_mapping hypercall.

HYPERVISOR_update_va_mapping(
    (unsigned long) &shared_info,            // defined in entry.asm
    (unsigned long) start_info->shared_info, // passed by Xen
    UVMF_INVLPG)                             // invalidate TLB entry 
);

After this is completed, the guest is able to use a pointer to the shared_info page as it would any other data structure.

shared_info_t *HYPERVISOR_shared_info = (shared_info_t*) &shared_info;

Time keeping

The default way of HermitCore to determine system time, is to simply get the elapsed clock ticks since boot time. This is done by receiving interrupts from the PIT or APIC timer and counting them.

 /** @brief Get milliseconds since system boot */
static inline uint64_t get_uptime() 
{ 
    return (get_clock_tick() * 1000) / TIMER_FREQ; 
}

/** @brief Returns the current number of ticks. */
static inline uint64_t get_clock_tick(void)
{
    return per_core(timer_ticks);
}

 /* Handles the timer. In this case, it's very simple: We
  * increment the 'timer_ticks' variable every time the
  * timer fires. */
static void timer_handler(struct state *s)
{
	/* Increment our 'tick counter' */
	set_per_core(timer_ticks, per_core(timer_ticks)+1);
}

The number of tics is divided by the timer frequency and multiplied by 1000 to get the elapsed milliseconds.

When running as a PV guest in Xen, the operating system does not have access to a PIT or APIC timer and thus the default time keeping method will not work. Xen provides the guest with all the information necessary to keep track of time through the shared_info_t struct.

In general, there are two types of time that a Xen guest must keep in mind. The first is the wall clock time - the elapsed real time. It is used for userspace applications that perform scheduled tasks, display clocks, and so on. The second is virtual time - the time the guest has spent executing. Virtual time is essential for scheduling tasks that are performed within a domain.

While a guest is scheduled, he receives a periodic tic event every 10 ms. This allows him to easily keep an eye on the virtual time. Real-time values are somewhat more complicated. Three different time values are needed to track real time:

Initial system time
is the time of day when system time is zero.
Current system time
is the time that has elapsed since the guest was resumed and is updated whenever the guest is scheduled.
TSC time
is the number of cycles that have elapsed since an arbitrary point in the past.

To calculate the uptime, i implemented a gettimeofday() function using the provided information.

gettimeofday()

Implementing the gettimeofday() function requires access to the shared_info page, the TSC and some simple calculations. The shared_info_t struct contains time values which are regularly updated by Xen.

To calculate the time, the guest has to wait until the last bits of the wc_version and version variables equal zero. This indicates the time values are not being updated and are save to read. Then, the current system time can be calculated with the following formula:

current system time = system_time + ((((tsc - tsc_timestamp) << tsc_shift) * tsc_to_system_mul) >> 32)

To get the wall clock time, the calculated system time has to be added to the wc_sec and wc_nsec values.

nanoseconds = wc_nsec + current system time
seconds = wc_sec + (nanoseconds ÷ 1.000.000.000)
nanoseconds = nanoseconds mod 1.000.000.000

seconds now contains the elapsed seconds since the Epoch. A complete implementation of the gettimeofday() function can be found in the file time.c.

Events

Event channels are the basic element that Xen provides for event notifications. An event is the Xen equivalent of a hardware interrupt. They essentially store one bit of information, the event of interest is signalled by switching that bit from 0 to 1. Notifications are received from a guest via an upcall from Xen that indicates when an event occurs (by setting the bit). Further notifications are masked until the bit is deleted again. Therefore guests must check the value of the bit after re-enabling the delivery of events to ensure that no missed notifications are received. Event notifications can be masked by setting a flag. This is equivalent to disabling interrupts and can be used to ensure the atomicity of certain operations in the guest kernel.

All event notifications are received by the same handler. The guest has to set up a way to dispatch events to their correct handlers when they are received. Since events are delivered completely asynchronously (much like normal hardware interrupts) they can occur at any point in execution. Upon entering the event handler it is therefore necessary for the guest to save the current state. When exiting an interrupt handler on x86, it is common to use the IRET instruction. This restores control to the interrupted process an re-enables interrupts atomically. Since events are an entirely software construct, the IRET instruction has no way of knowing how to enable them . There are two solutions for this. Xen provides an IRET hypercall that re-enables event delivery via the hypervisor. The other way it to not re-enable them atomically and handle errors when something goes wrong.

The actual implementation of the event handling has been taken in large parts from the examples provied in David Chisnall’s book “The Definitive Guide to the Xen Hypervisor”. It is rather complex and involves a lot of jumping from assembly code to C code and back. Explaining it in detail would go beyond the scope of this chapter. The code can be found in the entry.asm and event.c files. For further information an on how this works exactly, i would advise you to read chapter 7 in the mentioned book.

Console output

Xen provides the user with the possibility to connect a virtual console to a running guest. This enables the user to read the boot output of the guest and interact with it in the terminal in a way that is very similar to connecting to another computer via ssh. The console can either be attached when starting the guest by adding the “-c” flag

xl create -c domain_config

or later on by issuing the following command:

xl console <Domain>

Inside the guest, the console is also implemented using a shared memory page. The start_info_t struct shown in Figure 3.2 contains the machine page number of the console page and the event channel that is used for communication. To initialize the console, the guest first has to translate the machine page number of the console page into a physical page number. To avoid confusion, it is helpful to clarify the different types of memory addresses:

machine address
address in the (real) machine’s address space running Xen
physical address
address in the (virtual) guest machine’s address space
virtual address
virtual address inside the guest

Translating a machine page number into a physical page number is done with the help of the machine_to_phys_mapping macro defined in the xen-x86_64.h header file.

#define machine_to_phys_mapping ((unsigned long *)HYPERVISOR_VIRT_START)    
#define HYPERVISOR_VIRT_START xen_mk_ulong(__HYPERVISOR_VIRT_START)
#define __HYPERVISOR_VIRT_START 0xFFFF800000000000

The shared console page contains a xencons_interface struct which is defined in the console.h header file.

struct xencons_interface {
    char in[1024];
    char out[2048];
    XENCONS_RING_IDX in_cons, in_prod;
    XENCONS_RING_IDX out_cons, out_prod;
}; 

It consists of two ring buffers, one for input and one for output. To write output to the console, the guest essentially only has to write data into the output ring buffer. The control domain (dom0) then reads the content of the buffer and outputs it into the terminal. The complete code for Initializing the console is show in the listing below:

int console_init(start_info_t * start)
{
    console = (struct xencons_interface*)
        ((machine_to_phys_mapping[start->console.domU.mfn] << 12));
    console_evt = start->console.domU.evtchn;
    return 0;
}

Writing a string to the console works in the following way:

int console_write(char * message)
{
    struct evtchn_send event;
    event.port = console_evt;
    int length = 0;
    while(*message != '\0')
    {
        /* Wait for the back end to clear enough space in the buffer */
        XENCONS_RING_IDX data;
        do
        {
            data = console->out_prod - console->out_cons;
            HYPERVISOR_event_channel_op(EVTCHNOP_send, &event);
            mb();
        } while (data >= sizeof(console->out));
        /* Copy the byte */
        int ring_index = MASK_XENCONS_IDX(console->out_prod, console->out);
        console->out[ring_index] = *message;
        /* Ensure that the data really is in the ring before continuing */
        wmb();
        /* Increment input and output pointers */
        console->out_prod++;
        length++;
        message++;
    }
    HYPERVISOR_event_channel_op(EVTCHNOP_send, &event);
    return length;
}

Once the guest has written data into the output buffer, it sends an event channel notification to the dom0 with the help of the HYPERVISOR_event_channel_op hypercall. For a more convenient usage, i also implemented a printf-like function. The complete implementation can be found in the console.c file.

Interrupts

In addition to events, Xen provides a lower-level form of asynchronous notification in the form of traps. Unlike events that can be dynamically generated and bound, traps have a static meaning that corresponds directly to hardware interrupts. When the guest is started on a physical CPU, Xen installs an Interrupt Descriptor Table (IDT) in the guests name. Since traps correspond directly to hardware interrupts, the same code can be used to handle them.

To install an IDT, the guest has to use the HYPERVISOR_set_trap_table hypercall. It accepts an array of trap_info_t structs which contains one entry for every interrupt.

struct trap_info {
    uint8_t       vector;  /* exception vector      */
    uint8_t       flags;   /* 0-3: privilege level  */
    uint16_t      cs;      /* code selector         */
    unsigned long address; /* code offset           */
};
typedef struct trap_info trap_info_t;

Each entry contains the number of the interrupt, the highest privilege ring that can raise the interrupt and the address of the handler. All of HermitCores functions relating to creating the IDT have been rewritten to fill out such an array.

/* We have defined exactly 66 interrupts */
static trap_info_t idt[67] = {[0 ... 66] = {0, 0, 0, 0}};
uint8_t trap_counter = 0;

static void configure_idt_entry(trap_info_t *dest_entry,uint8_t num, size_t base,t16_t sel, uint8_t flags)
{
    /* The interrupt routine's base address */
    dest_entry->address = base;
    dest_entry->vector = num;
    /* The segment or 'selector' that this IDT entry will use
     *  is set here, along with any access flags */
    dest_entry->cs = sel;
    dest_entry->flags = flags;
}

void idt_set_gate(uint8_t num, size_t base, uint16_t sel, uint8_t flags)
{
    configure_idt_entry(&idt[trap_counter++], num, base, sel, flags);
}    

To configure an IDT entry, the idt_set_gate function can be called like this:

idt_set_gate(0, (size_t)isr0, FLAT_KERNEL_CS, 0);

FLAT_KERNEL_CS is a flag defined by Xen. It represents the code segment created by Xen mirroring a flat address space, where the entire space is mapped into a single segment.

When HermitCore has finished filling out the IDT entries, it calls the idt_install function.

void idt_install(void)
{
    /* Issue a hypercall to install the new idt */
    int ret = HYPERVISOR_set_trap_table(idt);
    if (ret) {
        LOG_INFO("Failed to set Trap Table!\n");
        LOG_INFO("Error: %d\n", ret);
        asm volatile ("hlt");
    }
    LOG_INFO("Installed new IDT with %d entries.\n", trap_counter);
}    

This issues the HYPERVISOR_set_trap_table hypercall, which instructs Xen to install the new IDT on behalf of the guest after it has been validated.

Memory Management

One of the original innovations of the Xen hypervisor was the paravirtualized Memory Management Unit (MMU), which enabled fast and efficient virtualization of guests using paging. To virtualize the memory subsystem, all hypervisors require an additional level of abstraction between what a guest sees as physical memory and the memory of the machine running the hypervisor. This is usually done via Physical to Machine (P2M) mapping and is managed by the hypervisor, i.e. it is hidden from the guest’s operating system. Instead, the Xen MMU model requires the guest to know of the P2M mapping. The guest’s operating system must be modified so that instead of writing page table updates to the physical address, they must be written to the machine address. To ensure that the guest cannot access memory areas he should not have access to, Xen requires that all page table updates be performed by the hypervisor. This means that the guest has read access to all page tables and must issue hypercalls when updating.

Modifying HermitCore’s memory management to work with the invariants of Xen would require to rewrite large parts of it. Considering that developing the memory management code has been a bachelor thesis on it’s own, implementing these changes would definitively go beyond the scope of my thesis. Instead, i modified HermitCore to run as a PVH guest on Xen.

Conclusion

Modifying HermitCore to work as a purely paravirtualized guest on Xen took a lot of changes. The lack of access to emulated hardware components such an an PIT or APIC and the specific constraints made by Xen make it necessary to rewrite many basic operating system functionalities. Considering that this would include modifying large parts of the memory management code, i changed the focus of the implementation. Instead of running HermitCore as a purely paravirtualized guest, i implemented the necessary changes to make it work as a PVH guest.

You will see in my next post, that the ability of a PVH guest to manage and modify it’s own page tables makes things a lot easier. Instead of rewriting large parts of the memory management code, i could just use the existing one without modifications. I hope you will be back for the next part,

Jan

Porting an unikernel to Xen: HVM guest

Sat, 17 Nov 2018 00:00:00 +0000

Welcome back to the second part of my blog post series on how to port an unikernel to Xen. In the first part i gave an introduction to Xen, Unikernels and HermitCore and explained the conceptual changes that have to be made to an operating system to work as a paravirtualized guest in Xen. This part will show you how i got HermitCore running as a fully virtualized guest in Xen. All the sourcecode i show in this post can be found in the xen_hvm branch of my HermitCore fork on GitLab. If you want to try it out for yourself, the docker folder contains instructions on how to build HermitCore and the vagrant folder includes a fully configured Xen hypervisor inside a Vagrant box.

Create a bootable image
HVM domain configuration
Startup wrapper script
Conclusion

Xen includes two main modes of operation. One is running paravirtualized guests and the other is hardware assisted virtualization. CPUs that support hardware virtualization make it possible to run unmodified guests, including operation systems such as Microsoft Windows. In Xen this is called a hardware virtual machine (HVM). Xen uses device emulation based on QEMU to provide I/O virtualization to the virtual machines. This means the virtual machines see an emulated version of a fairly basic PC.

Because the virtualization of HVM guests is based on QEMU, it is possible to run HermitCore almost unmodified as a guest in Xen. When booting HermitCore via QEMU directly, the kernel is booted via the GRUB Multiboot protocol. Unfortunately it was not possible to make the Multiboot Binary bootloader, which is included in Xen, boot HermitCore directly.

A simple other way to boot HermitCore, is to first start the GRUB bootloader and then boot into HermitCore.

Create a bootable GRUB ISO image

The easiest way to create a bootable media is to use the grub-mkrescue tool. It is included in every GRUB installation on major Linux distributions and can be used to create a bootable ISO file containing the GRUB bootloader. GRUB modules are also needed. On Ubuntu for example, they are included in the package grub-pc-bin. To create a GRUB rescue disk, the folder structure of the ISO file has to be created first. It includes the HermitCore loader, the application that is supposed to run and a GRUB config file and it should look like this:

iso/
 *-- boot
    |-- grub
    |   *-- grub.cfg
    |-- hermit_application
    *-- ldhermit.elf

The grub.cfg file has the following content:

default=0
timeout=0

menuentry "HermitCore" {
    multiboot /boot/ldhermit.elf
    module /boot/hermit_application
    boot
}    

It instructs GRUB to create a menu entry called “HermitCore”, which boots the compiled binary file of the HermitCore loader via the Multiboot protocol. The actual application that is intended to run gets passed to the loader as a Multiboot-module. Additional command line options can be passed to the kernel after the multiboot statement. For example

multiboot /boot/ldhermit.elf -uart=io:0x3f8

would enable output on the serial port COM1. To create the ISO file, grub-mkrescue needs to be invoked in the following way:

grub-mkrescue -o /tmp/hermit.iso iso/

The folder structure shown above is included in the “hermit.iso” file. This ISO could now be booted via QEMU and it would work just fine. A domain configuration file is still needed to boot it via Xen.

HVM domain configuration

# This configures an HVM rather than PV guest
type = "hvm"

# Guest name
name = "hermit-single.hvm"

# Initial memory allocation (MB)
memory = 1024

# Number of VCPUS
vcpus = 1

# Network devices
vif = [ 'model=rtl8139,bridge=br0' ]

# Disk Devices
disk = [ 'file:/tmp/hermit.iso,hdc:cdrom,r' ]

#Disable VGA output
nographic = 1

#Serial Console Output
serial = [ 'file:/tmp/hermit.log' ]

tsc_mode="native" 

Xen will create a HVM guest by setting type=hvm, assign it the given name, memory and one virtual CPU. HermitCore supports the rtl8139 chipset for network devices, so a device is added using this chipset and connected to the bridge br0 (The name of the device might be different on other Linux distributions). The just created ISO image is added as a CD-ROM and the graphic output is disabled. To see kernel boot messages, a serial port is attached and the output gets written into a temporary log file. While the guest is running, “tail -f” can be used to watch it boot. The last line is important, since it tells Xen how to emulate the Time-Stamp Counter (TSC) of the guest. Native mode has to be used, otherwise the time measures are wrong. A complete list of domain configuration options can be found in the xl.cfg manual file. To start the domain, xl has to be invoked with the following arguments, where domain_config is the configuration file created above.

xl create /path/to/domain_config

Create a wrapper script

The xen_hvm branch on Gitlab includes a wrapper script, which does all the steps shown above automatically. After building HermitCore it can be found in the following directory

HermitCore/build/local_prefix/opt/hermit/tools/xen-single-kernel.sh

and can be used as follows:

Usage: xen-single-kernel.sh -hvlknmc args

This script starts a HermitCore application as a single kernel HVM Domain in Xen. Make sure to destroy your domain when you are finished!

Args:
  -l  Path to hermit loader
  -k  Path to hermit application
  -n  Only create files, do not start
  -m  Domain memory in MB (Default 1024)
  -c  Number of CPU cores (Default 1)
  -v  print VERSION
  -h  this help screen

Example:
  xen-single-kernel.sh -l bin/ldhermit.elf -k x86_64-hermit/extra/tests/hello -m 512

Conclusion

HermitCore is running very well as a HVM guest in Xen. The recorded output of booting the “hello world” application can be seen below.

The only thing not working, is the ability to use multiple CPU cores. When trying to start additional CPU cores, HermitCore fails to bring them online via an Inter-processor Interrupt (IPI) which then leads to an infinite loop while waiting for these cores to come online. A possible way to make this work would be to implement hypercalls, which will be discussed in the following posts. With the help of a special hypercall, it is possible to make Xen start the additional CPU cores. Also, boot times are much slower compared to all other variants of running HermitCore. I will provide a detailed performance comparison in a future post.

This is it for part two. I hope you found it interesting again and will be back for the next one where i will show you my process of trying to make HermitCore work as a completely paravirtualized guest in Xen.

So long, Jan :-)

JanMa's Blog

Deploy systemd units in your Nomad cluster

The unit file

Enabling the unit

Loading systemd units from a different path

The complete job

Orchestrating containers with Nomad and systemd-nspawn

About Nomad

Using the nspawn plugin

Two simple example jobs

Conclusion

Running containers with systemd-nspawn

Prerequisites

Building an image

Running the image

Networking

Managing containers

Configuration per container

Talk: Evolution of a Microservice Infrastructure

A hypervisor in a box

Porting an unikernel to Xen: Wrapping up

Porting an unikernel to Xen: Perfomance comparison

Table of contents

Basic Benchmark

Stream Benchmark

Boot time

Conclusion

Porting an unikernel to Xen: PVH guest

Table of contents

Booting

Initializing Xen specific features

Hypercalls

Shared info page

Creating a multiboot information struct

Flags

Memory map

Multiboot module information

CPU frequency

Console output

Completing startup

Console output from an application

Conclusion

Further reading

Porting an unikernel to Xen: PV guest

Table of contents

ELF notes

Modifying the assembly boot code

Initializing XEN specific features

Hypercalls

Shared Info page

Time keeping

gettimeofday()

Events

Console output

Interrupts

Memory Management

Conclusion

Further reading

Porting an unikernel to Xen: HVM guest

Table of contents

Create a bootable GRUB ISO image

HVM domain configuration

Create a wrapper script

Conclusion

Using the `nspawn` plugin