Software Compartmentalisation: Isolation Mechanisms

You can access the slides 🖼️ for this lecture.

This last lecture on compartmentalisation covers the various mechanisms that can enforce the isolation between different compartments.

Compartmentalisation Mechanisms

A compartmentalisation mechanism enforces at runtime the isolation between compartments, as defined by policies and implemented through abstractions. All the mechanisms one wishes to use for compartmentalisation must allow the following two high-level properties: First, compartments represent different protection domains and the mechanism should provide proper isolation between them, i.e., prevent a compartment from reading, writing, or executing data and code in the memory of another compartment:

Second, the mechanism should allow safe and controlled communications between compartments that need to communicate. In other words, the CALL and RETURN abstractions we previously mentioned should be implementable with the considered mechanism. The mechanism should also ensure a form of cross-compartment control flow integrity: communication should allow compartment A to invoke compartment B only through well-defined interfaces, and not at arbitrary points in B’s code. Ideally, the data exchanged between compartments should also be limited to the data required for communication: anything more than that would be oversharing, which decreases the level of security.

Examples of Mechanisms

Several mechanisms can be used to enforce the isolation required in a compartmentalised scenario. They can be classified into two categories: hardware and software mechanisms.

Hardware Isolation Mechanisms. We have already mentioned several times process-based compartmentalisation, where each compartment runs within its own process. In that case, the mechanism used to isolate compartments is the page tables. Another mechanism that isolates the kernel from user space uses the privilege levels on the CPU. Memory protection keys is another mechanism that allows creating multiple compartments within a single address space. We also mentioned previously trusted execution environments and confidential VMs, which provide strong isolation even in the presence of malicious privileged layers such as the OS. Other examples of hardware isolation mechanisms include bound-checking hardware or hardware memory capabilities.

Software Isolation Mechanism. Software Fault Isolation (SFI) is a compiler-level technique, in which the code of compartments is generated in such a way that it cannot escape a sandbox. Control transitions such as jumps or calls are also generated in such a way that they can only target legitimate code locations, enforcing control flow integrity. Memory safe languages can be seen as a form of isolation mechanism that prevents the memory safety issues we covered previously, which could allow a compartment to access memory it is not supposed to. Finally, bounds-checking software such as FAT pointers augments the pointers with information about the bounds of the objects they point to. When these pointers are dereferenced, these bounds are checked before the access occurs, which gives more guarantees that a compartment will not be able to access memory it is not supposed to read or write.

Hardware vs. Software Isolation. Most modern compartmentalisation approaches rely on hardware mechanisms. That is because such mechanisms can provide isolation that is both strong and relatively fast. Several hardware mechanisms are also compatible with many types of software, regardless of the language in which it is written. Conversely, software mechanisms are available everywhere, regardless of the hardware. However, they are generally slower and suffer from compatibility issues.

Cross-Compartment Communications

In a compartmentalised application, compartments are not entirely isolated and need to communicate. With that in mind, isolation mechanisms must support some form of CALL and RETURN primitives to transition between compartments. In addition to that, the mechanisms should also support a way to exchange data between compartments upon such transitions. For example, to pass parameters or return values when there is a cross-compartment call, and also to handle the data pointed to by reference parameters exchanged across compartments (if any).

To achieve data exchange there are 2 main methods: message passing and shared memory.

Message Passing. The first class of cross-compartment data exchange methods is message passing: the data is sent and received between compartments over some form of communication channel, generally involving one or more data copies. This is the case with IPCs such as pipes or sockets, when using process-based compartmentalisation. Message passing is relatively slow because of the data copies it involves, and also sometimes because of the need to marshal/unmarshal transmitted/received data into/from a format suitable for communication. Message passing is also very secure, because a particular piece of data is never accessible from more than one compartment at a time (no possibility of Time of Check to Time of Use – TOCTOU – attacks).

Shared Memory. The second method for data exchange is shared memory. Compartments can establish a portion of shared memory, so they can share part or even their entire address space. Doing so, they don’t need to send and receive data; they simply need to send and receive references to that data, which are much smaller. Hence, shared memory has the potential to be much faster than message passing. However, it is also less secure. In scenarios where compartments run concurrently, because two communicating compartments can access the memory they share at the same time, there is a risk of race conditions and TOCTOU attacks.

Most mechanisms will allow one or both approaches. For example, as seen with our example program in the introduction to the topic of compartmentalisation, processes can either share memory or establish message-passing-based IPCs (e.g., pipes).

Trust Models

Each mechanism is designed with a specific trust model in mind. For example, CPU privilege levels target single-direction distrust: the kernel distrusts the application, which itself trusts the kernel. Other mechanisms target mutual distrust, such as the page table establishing different address spaces for different processes, or trusted execution environments. Mechanisms will also influence the content of the trusted computing base (TCB). In general, from the point of view of a compartment, the TCB will contain at least the compartment’s code, the system’s loader, the motherboard firmware, BIOS, the OS and hypervisor and their boot processes, as well as the machine’s physical environment. Some mechanisms, like trusted execution environments, allow reducing that TCB by removing the OS and hypervisor.

Mechanisms: Misc. Aspects

A few more aspects regarding mechanisms are worth discussing. Mechanisms will enforce different permissions, generally a combination of read/write/execute, but also address – in the sense of the ability for a compartment to create a reference to a resource or an area of memory. Not all mechanisms support all permissions: for example, Intel’s implementation of memory protection keys supports only enforcing read/write access, read-only access, or no read/write access at all, and there is no support for preventing execution.

The enforcement granularity varies depending on the mechanisms. Some, like the page tables, will allow or deny memory accesses at the granularity of 4KB memory pages. While others, such as bounds-checking software, will provide byte-level memory protection. These considerations have implications in terms of memory consumption and oversharing for compartmentalised applications using these mechanisms.

Certain mechanisms also support a limited number of domains, for example 16 only with Intel Memory Protection Keys. With MPK this number can be increased with some forms of virtualisation, at the cost of a non-negligible performance slowdown. Other mechanisms, such as processes, support an unbounded number of domains. More generally, the more compartments you have, the more scalability issues arise.

Performance Considerations

Choosing a particular mechanism for a compartmentalisation project has an important impact on the compartmentalised application’s performance. The mechanism impacts performance overheads as it defines the time taken for security domain switches, the cost to communicate data across compartment boundaries, and possibly the cost to sanitise such communication data. Creating and destroying compartments may have non-negligible performance costs, as does setting up and updating compartments’ permissions. The way memory needs to be organised to ensure the shared or private nature of data will impact performance due to memory fragmentation, access locality, and its impact on the caches. Finally, as mentioned previously, scalability issues may arise due to too many compartments, or compartments that are too large.

Today, the main source of performance concerns remains the domain switching latency. For that reason, the most up-to-date compartmentalisation approaches rely on mechanisms that work within a single address space to avoid context switches, and that can also perform security domain switches without relying on an exception, contrary to something like system calls.