Lightweight Virtualisation

The slides for this chapter are available here.

Motivation

Imagine a company wishes to run a website and does not want to leave a local machine up and running 24/7, so they decide to rent a virtual machine in the cloud. They choose a cloud provider, say AWS, and select a Linux distribution to install on their VM, for example Ubuntu. So inside their VM the web server, like Apache, is installed, along with its library dependencies, things like perl, libssl, etc. When Apache runs, all this software makes use of a subset of the services offered by this massive guest kernel that is Linux. The setup we just described can be illustrated as follows:

On this illustration, what really needs to run is in the blue boxes: the web server, its dependencies, and the subset of kernel features it requires, that’s it. All the grey areas are installed and maybe even running but are not needed. We call this software bloat, and it’s a bit of an issue.

Indeed, software bloat leads first to an increased attack surface: all the software installed in the Linux distribution, all the background programs running, and that you don’t really need represent potential targets for an attacker to take over and as a first step to attack your environment. Probabilistically, the more software you run, the higher the chance of a vulnerability being present somewhere. Second, software bloat represents additional costs. The VM’s tenant is paying the cloud provider for the disk, memory and CPU cycles used to store and run software they don’t even need. Third, for a fixed budget, software bloat also causes performance loss, because all this memory and CPU cycles are not used to run what really needs to run, which is your web server.

Definition

Lightweight virtualisation tackles this issue by providing the following properties, compared to traditional virtual machines:

1. Lower memory footprint, in the order of kilobytes to a few megabytes of system software overhead for each virtualised instance, compared to hundreds of megabytes or gigabytes of memory consumption for traditional VMs.
2. Fast boot times in microseconds or milliseconds, compared to seconds or minutes for traditional VMs.
3. Lower disk footprint, once again we are talking about a few kilobytes or megabytes.

Obviously, these metrics regard the per-VM systems software, in particular the operating system. The part of the initialisation time and memory/disk footprint that relates to an application will stay the same whether it runs in a lightweight or in a traditional VM.

Today there are three modern technologies that allow to achieve the aforementioned lightweightness objectives. First, stripped-down Linux VMs, called micro-VMs. These can be quite minimalist, but there are two technologies that take things one step further in terms of lightweightness: containers and unikernels. We’ll focus on containers and unikernels here; if you want to explore micro-VMs by yourself, you can check out, for example, Firecracker.

Containers

Presentation

Containers are a process-based sandboxing technology, enforced by the operating system. A container management stack differs significantly from a system-level VM-based one:

Contrary to a traditional virtual machine, a container is a process or a group of processes for which the OS restricts the visibility of system resources. This way the software running in the container is sandboxed, and it also feels it is running alone in the system, like in a virtual machine.

The resources whose visibility can be reduced and changed for the container are the filesystem, the system’s users, visible PIDs, IPCs, system clocks, among others. The OS can also control the allocation to the container of certain resources, including CPU scheduling cycles, memory available, disk and network bandwidth usable, among others.

Conceptually, by reducing or changing the visibility on resources, and limiting their allocation to a process or a group of processes, containers achieve the same isolation goals as a virtual machine, without the need for an hypervisor and a system-level VM. This is much lighter than using a traditional virtual machine. The boot time is that of spawning a process, a few microseconds, and the memory footprint is close to 0. Still containers are not perfect, and as we will see they suffer from significant security concerns.

Use Cases

Containers are useful in most scenarios where virtualisation is beneficial. They are extensively used in software development, where they allow bringing up a homogeneous environment to develop, build, and test an application, for the entire development and testing team. Containers can also be used for deployment, as they represent a lightweight way to package an application with all of its dependencies. Docker is a prime example of a container engine used extensively in software development:

Because they are so lightweight, containers can replace traditional VMs for many aspects of the cloud requiring very quick initialisation and execution of a particular task. Services such as Gmail or Facebook make extensive use of containers for such tasks. You may also have heard about AWS Lambda, which provides serverless computing services. With the serverless paradigm, the developer programs cloud machines with small stateless functions executed on demand when certain events happen, for example a user visits a web page. These functions generally run within containers.

Namespaces and Control Groups

Containers are enabled by two key technologies in Linux, restricting the view and usage of resources for processes: namespaces and control groups.

Namespaces restrict the view of the container on the following system resources:

• Filesystems and mount points: a container is generally given its own root filesystem from a base image, and it cannot access the host’s filesystem.
• Network stack: a container also has its own state of the network stack, including its own IP, with a virtual bridged and routed network.
• Processes: PIDs and IPCs: a container has also its own isolated set of PIDs, one for each process it runs. It cannot see or communicate with external processes.
• Host and domain name: a container can set the host name, which is the machine’s name, to something different than what the host sees. Same thing for the domain name.
• User IDs: usernames and IDs can also be different within the container, compared to what is on the host. In most scenarios a user will simply take the identity of root within the container.

Control groups restrict a container’s usage/access/allocation of the following system resources:

• Memory: one can set the maximum amount of memory and swap a container can use.
• CPU: the CPU usage of a container can be rate-limited, for example the container can be allocated 1.5 CPUs. Which CPU (core) a container can run on can also be restricted, and so can the scheduler’s quotas for the container.
• Devices: a container can also be restricted to seeing only certain devices.
• Block and network I/Os: a container’s disk and network throughput can be rate-limited.

Containers vs. VMs

If we list the respective benefits of containers versus traditional virtual machines, we get the following:

Containers are very lightweight, meaning they have a low memory and disk usage and very fast boot times. Their lightweightness allows creating a very high number of containers on a single machine; it’s not uncommon to run hundreds or even thousands of containers on a host. Nested virtualisation is also easy with containers, in other words it’s simple to create a container within a container. Regarding virtual machines, they are still useful when one wants to run an operating system different from Linux – something that is difficult to do efficiently with containers because they rely on the namespaces and control groups technologies available only with Linux. Several studies have also shown that performance isolation is stronger with VMs than with containers, meaning it’s more difficult for a malicious VM to steal resources by abusing them. Finally, the degree of isolation for the sandboxing enforced by virtual machines is considered as much stronger compared to containers.

Containers and Security

To understand why the isolation in VM environments is considered as stronger compared to containers, let’s consider both setups:

We have a container environment on the left, with several containers running on top of the OS kernel. And on the right a VM environment, with several VMs running on top of a hypervisor. If we place ourselves in the shoes of the cloud provider, and reason about what we trust and what we do not trust in such a setup, we shall conclude that the virtualisation layer is trusted, that is the OS for the container environment, and the hypervisor for the VM one. The instances of either containers or virtual machines are obviously untrusted; who knows who the tenants are and what they run in their VMs/containers.

As the provider of virtualised environments, the kind of attack we are most concerned about is often the following:

One of the containers or VMs is malicious, and tries to perform an escape attack, that is, to get access to the virtualisation layer’s memory, or to the memory allocated for other containers or VMs. As we have seen in the past, hardware-enforced isolation mechanisms such as the page tables or extended page tables will prevent direct access from the malicious entity to other VMs or containers. The real threat lies in the virtualisation layer, which can be invoked by the malicious VM or container. If this invocation can manage to trigger a bug in the virtualisation layer, the isolation may be broken and the attacker could access the virtualisation layer’s or other containers’/VMs’ memory.

It is quite important to determine how complex this interface between what we trust and what we don’t trust is in both cases. The reason for that is that how complex translates into how hard it is to secure this interface and make sure there are no bugs.

In the case of containers, that interface is unfortunately very complex: it is the system call interface, which is made of hundreds of system calls, some of them like ioctl presenting thousands of sub-functions. There is no way we can guarantee that the implementation of all these system calls is bug-free. In fact, automated vulnerability finding systems will regularly find bugs on that interface. Conversely, the interface between a VM and the hypervisor managing it is much simpler: it is just a few traps.

In that context, the isolation between containers is not considered to be as strong as that between VMs because of the complexity of the interface between containers and the privileged layer, the OS kernel. To illustrate this point, note that many actors running containers in production will actually run containers within virtual machines, to benefit from their strong isolation. These approaches try to reduce as much as possible the memory footprint and boot times of Linux VMs, creating what they call micro-VMs; however this still negates most of the lightweightness benefits of containers. An example of such technology is Firecracker.

Unikernels

Presentation

We have seen that traditional VMs are heavyweight but secure, and that containers are lightweight but insecure. Can we get both the lightweightness benefits of containers, combined with the security benefits of virtual machines, into a single virtualised solution? Unikernels are a relatively new operating system model that aims to answer that question.

Recall our motivational example from earlier, presenting the software bloat that occurs in many situations when using traditional VMs. Using unikernels we would address the problem as follows:

With a unikernel we compile an application’s code as well as all of its dependencies with a very small operating system layer into a static binary which merges the application and the operating system. This binary can be run as a kernel, in a virtual machine, on top of a hypervisor.

A unikernel instance is single-purpose and it runs a single application. To run multiple applications, one needs to run multiple unikernel instances. A unikernel instance is also a single process virtual machine, and once again to run a multiprocess application it is generally needed to execute several unikernel instances. Still, several unikernel models can run on multicore CPUs and leverage parallelism/concurrency with threads. Finally, as already mentioned in a unikernel instance the VM runs a single binary, containing the application, its dependencies, and the kernel. All of this code runs within a single address space, and there is no user/kernel protection. This is because there is only one application running in a unikernel instance, and the isolation between applications is already enforced by running them as separate unikernel instances.

The unikernel model was originally proposed in this seminal paper in 2013:

Madhavapeddy et al., Unikernels: Library Operating Systems for the Cloud, ASPLOS’13

Benefits & Application Domains

With that model, unikernels present a series of benefits. First, it’s a form of lightweight virtualisation. Because a unikernel instance only runs the code absolutely necessary for the application in question, and that the OS layer is so small, we get similar benefits as with containers in terms of low memory/disk footprint, and fast boot times. Second, because they run as virtual machines, unikernels are well isolated and considered a secure alternative to containers in many scenarios. Third, the OS layer within a unikernel instance can be specialised towards the application it runs: specialised kernel subsystems can bring higher performance, or lower memory footprint/power consumption for a particular application scenario. Finally, because a unikernel operating system is so small and simple, it does not get in the way of the application’s performance as much as larger OSes such as Linux. This translates into increased performance for certain applications. Another noteworthy point regarding performance is the system call latency: with unikernels, because there is no user/kernel isolation, system calls are simple function calls, which makes them much faster.

Just to illustrate the unikernel performance benefits that come from their low-latency system calls, consider these results:

Redis throughput under various setups (higher is better)

This graph is taken from the Unikraft paper:

S. Kuenzer et al., Unikraft: Fast, Specialized Unikernels the Easy Way, EuroSys’21

The graph shows the throughput for Redis, which is a very popular key-value store, in millions of requests per second. There are various setups on the X axis, including different unikernels, as well as vanilla Linux. As you can see, Unikraft is the fastest solution: even if it runs virtualised on top of Qemu/KVM, it is still a bit faster than non-virtualised Linux, and also much faster than Linux in a VM. Feel free to check out the paper for more performance evaluation.

Given these benefits, unikernels have plenty of application domains. We motivated them with cloud environments such as server or microservice software. But they have also been explored in the domains of embedded virtualisation, edge computing and IoT, network function virtualisation, high performance computing, and various security-critical domains such as VM introspection, malware analysis, and secure desktop environments. Still, an important point to note is that most unikernels are still at the stage of research prototypes. This is different from containers, which as you may know are a production-ready technology.

Unikernel Projects

A few examples of unikernel projects are listed below. Some of these are relatively unstable and poorly maintained academic research artefacts. The most mature project, the one that is the closest to production ready status, is Unikraft.

Unikernels can be classified based on the targeted language for supported applications:

• Pure memory safe languages (OCamL, Erlang, Haskell): MirageOS, LING, HalVM
• C/C++, semi-POSIX API: Unikraft, HermiTux, HermitCore, OSv, Rumprun, Lupine
• Rust/Go: Hermit, Clive
• More: https://unikernelalliance.org/projects/

Compatibility Aspects

Although they present desirable security and lightweightness benefits, unikernels also suffer from an important drawback which stems from their custom OS nature: a lack of compatibility with existing software and, to a lesser extent, hardware. On the hardware side, it is not possible for unikernel projects to integrate the large amount of drivers supported by popular OSes such as Windows or Linux. Still, using the split driver model (frontend/backend) we saw that is common in virtualised environments, a unikernel supporting popular paravirtualised (e.g. Virtio) I/O frontend drivers and running alongside e.g., Linux, will be compatible with numerous I/O devices. The real compatibility problem of unikernels rather lies on the software side.

One of the reasons unikernels are not particularly popular today, despite being around for more than a decade now, is that it is hard to run existing applications on top of them. Most unikernels require access to an application’s source to compile it with the unikernel OS, so in scenarios where the sources are not available for a given application (e.g., proprietary software), a unikernel cannot be created. Even when sources are available, as we’ve seen previously, most unikernel models support only one or a few programming languages, which limits compatibility.

Many applications will also require specific OS features available under e.g., Linux or Windows, but unsupported by most unikernel models. Porting is thus required: one can try to adapt an application to work on top of a unikernel model, or to enhance the unikernel model for it to provide the features required by the application. Often, porting involves doing a bit of both: it is a time-consuming task which requires expertise in both the application to be ported and the unikernel model to use, which discourages many prospective users.

Since that problem was identified, several research efforts attempted to address the compatibility issues of unikernels. The main idea is to require as little modifications/effort as possible to execute, as a unikernel, an application that already builds for and runs on top of a popular OS such as Linux. The compatibility can be achieved at various levels, from the weakest to the strongest:

• Source-level compatible unikernels (e.g., HermitCore, Rumprun): these require recompiling an application’s code with a custom C standard library and the unikernel kernel. It is a relatively weak form of compatibility, as the C standard library is not the only interface with the kernel in many applications. This approach also still requires recompilation and access to the sources.
• C standard library-level binary-compatible unikernels (e.g., OSv, Lupine): these interface an application at runtime with a custom C standard library, similarly to how shared libraries are loaded at runtime. These approaches may sometimes avoid recompilation, but are still limited to programs requesting OS services only through the C standard library.
• System call-level binary compatible unikernels (e.g., HermiTux, Unikraft): this is the strongest form of compatibility, targeting the standard app/OS interface: the unikernel OS hooks into the system calls made by an application compiled for a popular OS (e.g., Linux) and emulates that OS behaviour. Such compatibility at the system call level allows running unmodified Linux applications as unikernels without access to the sources/recompiling.

For more information on the topic of unikernels and application compatibility, check out this paper:

Olivier et al., A Binary-Compatible Unikernel, VEE’19