OS Models and Security
You can access the slides 🖼️ for this lecture.
OS Design Models
Since the first OSes were proposed in the 1950s, we have derived several classes of OS design models, coming from both industry and academia. Here, by OS design model we mean a collection of core design choices that define how the OS architecture is organised. Each of the main OS design models relevant today has pros and cons in terms of speed, memory footprint, reliability, real-time capabilities, and, of course, security. In this lecture we will briefly cover the OS design models that are relevant to security, and discuss their performance and security implications.
Monolithic Kernel
The monolithic kernel is one of the most widespread classes of OS design today. This is the model we focused on in this unit, as Linux is indeed a monolithic kernel. The key idea of a monolithic kernel is that all kernel components are packed together within a single trust domain: there is no isolation between kernel subsystems, and they all live within the same address space. Further, the kernel is closely coupled with applications, as both are located within the same address space: The OS is isolated from applications using CPU privilege levels: the kernel runs in supervisor mode, and the kernel’s memory is protected with the supervisor bit in page table entries. Applications are isolated from each other by running in different address spaces. We can illustrate the monolithic kernel model as follows:
The close coupling of kernel and user space software components in a monolithic kernel is done for performance reasons, to lower the overheads due to the interactions between these components. Overall, this OS model gives us a middle-ground trade-off between security and performance. From the security point of view, the isolation between the kernel and applications, and between applications themselves, is good but not ideal, as we will see next. There is also no isolation inside the kernel. This medium level of isolation means that there is a limited number of security domain crossings, which is good for performance. Many derivatives of UNIX are monolithic, the most popular examples being Linux, FreeBSD, NetBSD, or OpenBSD. MS-DOS is another example of a monolithic kernel.
The OS Design Space
As we just mentioned, the monolithic kernel design is a middle-ground solution between security and performance. Let’s have a look at other, more extreme points in that design space. We can decide to increase the degree of isolation, to improve security at the cost of performance. We can also decide to sacrifice security for the sake of performance, for example by dropping part or all of the isolation between kernel components and/or applications, with the goal of faster computations.
Microkernels are a prime example of the former solution, favouring security at the expense of performance.
Microkernels
With a microkernel, most of the OS functionalities are moved into user space processes (sometimes named servers), each running within its own address space. The microkernel itself is very small and contains only the core functionalities that cannot be offloaded to user space: memory management, scheduling, as well as inter-process communication. These IPC mechanisms are a key aspect of microkernels, because this is the way used by kernel subsystems offloaded to user space to communicate with the rest of the OS.
The microkernel is considered more secure compared to the monolithic kernel, because of the increased level of isolation between kernel components. If one server gets compromised, the attacker will be confined to the relevant address space, and it will be difficult for them to gain access to the rest of the system. Of course, this additional security comes with a performance cost: communicating via IPC is slower compared to a monolithic kernel where everything lives within a single address space. With a microkernel, although the communication can be heavily optimised, such communication does involve security domain crossings and possibly data copies, all of which is expensive. Examples of microkernels include MINIX v3 and L4.
Exokernel
We now move to the other side of the spectrum presented earlier, looking at solutions that drop some or all of the isolation in the system to be able to run faster. With the exokernel model, each application or group of applications is compiled with a library providing most of the operating system services. This is called a library operating system (already briefly discussed in the previous lecture). LibOS instances run on top of a microkernel-like kernel named the exokernel:
The exokernel acts as a microkernel, handling mostly core OS functionalities such as memory management, and exposing a low-level interface that securely multiplexes the hardware between the (untrusted) LibOSes. In its original version, the exokernel idea was to specialise each LibOS instance for the applications it runs, to maximise performance. Some applications could benefit from a particular implementation of the memory allocator, while other applications would prefer different implementations of the allocator. This is possible because we have one LibOS instance per application.
Unikernel
The exokernel idea was proposed in the 1990s but did not really take off. Fast forward to the 2010s, researchers realised that a small hypervisor could very well play the role of the exokernel, and that applications could be compiled with small library operating systems and run within virtual machines on top of that hypervisor. This is the unikernel design model:
In terms of security, unikernels are well isolated from each other because they run in separate VMs. The hypervisor is also isolated from the unikernels. Still, there is no isolation between a unikernel instance and the applications it runs. This has some performance advantages, in particular system calls become function calls which are much faster. However, it is obviously concerning from the security point of view: if an attacker can exploit a vulnerability in an application, they may easily access the entire unikernel instance.