Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

OS Security Concepts

You can access the slides 🖼️ for this lecture.

Introduction

We have previously covered the many responsibilities of the OS, which is a foundational component of modern computer systems. As such, the OS plays a key role in the security of the computer system it controls. The primary security goals of an OS include enforcing which operations (e.g. read, write) subjects (e.g. applications, users) can perform on objects (e.g. files, system resources), applying the principle of least privilege to maintain confidentiality, integrity, and/or availability. These security goals are often at odds with other OS objectives such as performance, resource usage, convenience of use, or compatibility.

Fundamental OS Security Invariants

The two main entities to which the permissions/restrictions considered in OS-enforced security policies are applied are processes and users. We have already seen the following process-level fundamental security invariants an OS aims to maintain:

  1. Inter-Process Isolation. By default, processes should not be able to access (in read, write or execution mode) each other’s state (mostly memory) directly. We have seen that this is enforced with virtual memory, by giving each process its own virtual address space. In most modern CPUs this is realised with the page tables defining non-overlapping virtual address spaces for processes.

  2. User/kernel isolation. Processes should not be able to access the memory reserved for the kernel directly. To request services, processes can only invoke the kernel at a safe and well-defined entry point. These two security objectives are achieved with the user/supervisor execution mode and associated bits in page table entries, as well as the system call mechanism.

In addition to the process-level invariants, the OS enforces several user-level security invariants:

  1. User authentication. Only authorised users should be able to access the system. This is enforced with authentication mechanisms such as passwords, biometric identifiers (fingerprint, face ID), or other credential-based approaches.

  2. User-controlled resource sharing. Users must be able to configure how to share or not share the resources they own with other users. In UNIX/Linux systems, this is primarily implemented with file permissions, where files abstract many types of system resources.

  3. Privileged operations. Only privileged users (administrators) should be able to accomplish security-critical tasks such as loading kernel code, shutting down the computer, mounting filesystems, or modifying system-wide configurations.

A Basic OS Trust Model

When considering the security of a computer system from the OS point of view, the trust model we have assumed so far is as follows: the entirety of the OS kernel is trusted, and local user space applications are not trusted. Remote applications (e.g., clients connecting to a server hosted by the system) are not trusted either. Similarly, local or remote non-administrator users are not trusted. This trust model can be illustrated as follows:

In reality, this model is overly simplistic and does not consider components of the system that are key to its security:

  • The hardware, which is assumed by the OS to behave correctly for any kind of security guarantee to hold in the system.
  • The system administrator has ambient authority over the system: should they be malicious, the OS security guarantees simply cannot be maintained. Further, certain applications (e.g., programs logging users in, or changing users’ passwords) are privileged. If one of these is compromised, the aforementioned OS security invariants are likely to break.
  • The system’s components involved in the boot process are also assumed to behave properly. That includes the motherboard’s firmware, the BIOS, bootloader, and OS boot process. Once again, if a component of the boot process is compromised, the system cannot be secure.

All these components are fully trusted in the OS trust model we have considered so far. A more complete illustration of that model, taking into account the components we just covered, is as follows:

The large quantity of green boxes in that diagram should give you an idea of the immense amount of software and hardware that is blindly trusted to work correctly when considering the security of a computer system. This raises the following question: does this trust model reflect the reality of today’s computer systems?

Realistic OS Trust Model

The answer to the previous question is no. It is easy to identify situations in which the components that are assumed trusted in our basic trust model may actually be faulty or malicious. The BIOS, bootloader, or other boot process software may have bugs or be corrupted. A local attacker could swap the on-disk kernel image, loaded at boot time, with a malicious one, or install a bootkit. The OS kernel itself is not exempt from bugs and vulnerabilities. This is particularly problematic in subsystems developed by third parties (e.g., drivers), which may not have the same degree of code writing quality, secure coding practices, or testing/validation discipline as core kernel subsystems. The hardware itself cannot be fully trusted, as it presents faults and vulnerabilities, see for example the Spectre and Meltdown micro-architectural side channels identified in 2017. Finally, the system administrator or the owner of the computer an application executes upon may not be fully trusted: for example, one of the reasons certain businesses refuse to offload some of their internal workloads to the cloud is because they do not trust the cloud providers with access to their data.

In that context, a more realistic OS trust model can be represented as follows:

Security concerns regarding the hardware, boot process, system administrator, or internal untrusted components of the OS have been the target of several recent technologies and protection techniques, which will be covered in the remainder of this course unit:

  • The lecture materials on trusted execution environments will consider scenarios where the system administrator/host computer is not trusted.
  • The part on software compartmentalisation will cover how to isolate untrusted components of applications and operating systems.
  • Finally, the hardware part of the unit will discuss hardware security and secure boot.

Application - Kernel Isolation

Let us zoom in on the isolation of the kernel from applications. For that we can simply consider the following basic trust model: applications distrust each other and trust the kernel, and the kernel distrusts applications:

We have seen earlier that one of the main security invariants an OS aims to maintain is inter-process isolation: applications should be isolated from each other. This is relatively intuitive: some applications in the system perform security-sensitive tasks (e.g., authenticating a user, installing new software, etc.) and handle critical data (check the validity of a password, generate and process crypto keys, etc.) while other applications are not fully trusted (they may run on behalf of an untrusted user, be faulty, or under the control of an attacker and plainly malicious) and it makes sense to isolate both classes of programs. Because it is often difficult to determine if an application is security-sensitive or is not to be fully trusted, OSes implement a policy that is safe by design: by default, each process running in the system should be isolated from the other processes.

We have also seen that the kernel is in charge of enforcing the isolation across processes, by setting up their page tables and defining their virtual address spaces to be non-overlapping. For that reason we need the kernel to be isolated from applications: this is our second security invariant. If an application could modify or access the kernel’s memory, for example, to modify page tables or run code that allows updating memory mappings, our cross-process isolation security invariant would break.

User -> Kernel Attacks

Although the kernel is well isolated from applications, these still need to be able to invoke the kernel to perform system calls: although this is made through a controlled interface, this interface still raises security concerns. Production-ready kernels such as Linux are written in memory unsafe languages, so they are subject to all the memory safety and undefined behaviour vulnerabilities we saw previously. Due to their sheer size, millions of lines of code, we cannot rule out the possibility of bugs and vulnerabilities in these OSes, for example on the processing path of system calls. Hence, although processes cannot access each other’s memory directly, a process triggering a bug in the kernel through the invocation of one or several system calls may be able to mislead the kernel into interfering with or accessing the memory of another process: from the kernel’s point of view, the system call interface is a primary attack vector:

This is particularly concerning as the system call interface is extremely large and complex. Linux has more than 350 system calls, some of them implementing up to thousands of sub-functionalities, e.g., \texttt{ioctl}. Because of that size and complexity, that interface is very difficult to fully secure, and we cannot rule out the possibility of kernel bugs and vulnerabilities triggered by applications using that interface.

Hardening The System Call Interface

Because the system call interface is the main user -> kernel attack vector, the OS considers every piece of data flowing from user space to the kernel through system calls as untrusted. That includes system call parameters and, for pointer parameters, what they point to. In that context the kernel must protect itself against the user space injecting:

  • Corrupted data structures
  • Bad indexing information
  • NULL pointers
  • References to resources (e.g., files) that the process does not have permission to access
  • Sequences of system call invocations in the wrong order
  • Etc.

The kernel must sanity check the validity of the system call data and control flow coming from user space. As mentioned earlier, getting these checks 100% correct is difficult due to the size and complexity of the system call interface.

Sanity Checking System Call Pointer Parameters

A process often passes a pointer to some of its memory to the kernel. Consider for example the readv system call:

ssize_t readv(int fildes, const struct iovec *iov, int iovcnt);

This system call performs several (iovcnt) read operations from a file identified by a file descriptor filedes. The operations are described in an array of iovec vector data structures which is located within the application’s address space, and passed to the kernel as a reference iov. That pointer could be invalid, and the content it points to could be corrupted: for safety reasons, before accessing that array the kernel needs to check the validity of the pointer (e.g., is it NULL?) and of the content it points to. These validity checks depend on the data structure in question, for iovec elements Linux will check that each vector’s length is not negative and will not overflow the target buffer, that the pointers to target buffers are valid and point to user-accessible memory, etc.

These sanity checks cannot be made directly to the area of user space memory referenced by system call pointer parameters. This would open the kernel for a type of vulnerabilities called Time of Check to Time of Use (TOCTTOU), also known as double fetch. Assume that sanity checks are made by the kernel directly in user space memory and consider the following scenario:

We have a view in time of what runs on the CPU on top, and a memory view at the bottom. A thread of an application runs on a CPU core and invokes a system call with a pointer parameter referencing a data structure in user space. The kernel takes over, sanity checks the data structure, and if the check succeeds, the kernel can access the data structure: all is well.

With this scenario, a TOCTTOU attack works as follows: the user space application can use another thread to corrupt the data structure after the check, but before the kernel accesses it:

This can be done if the application’s code can run concurrently with the kernel: corrupting the data can be done from another thread, a signal handler, or from another application sharing the area of memory in question. In effect, this attack can bypass the sanity checks made by the kernel. The solution implemented by modern kernels to protect against TOCTTOU issues is to copy into kernel space all user space data passed by reference. Validity checks can then be performed on these copies, which cannot be accessed by user space at all:

To perform these copies, the kernel uses two functions:

unsigned long __copy_from_user(void * to, const void __user * from, unsigned long n);
unsigned long __copy_to_user(void __user * to, const void * from, unsigned long n);

__copy_from_user copies n bytes of memory from user space at address from into kernel space at address to. For content that needs to flow from the kernel to user space, __copy_to_user copies n bytes of memory from kernel space at address from into user space at address to. These functions perform additional security checks, such as verifying that user space references are not NULL, that they point to memory that is mapped, etc.

Kernel Vulnerabilities and their Consequences

By exploiting a kernel vulnerability, an attacker can aim to accomplish the following:

  • Leak or tamper with kernel memory, for example to read kernel pointers and break kernel ASLR, or to escalate privilege by gaining administrator rights.
  • Access other processes’ memory, to break the cross-process isolation security invariant. Indeed, the kernel has access to the entirety of the computer’s memory.
  • Execute code, possibly arbitrarily, in the context of the kernel – that means with full privileges, i.e., in supervisor mode. This can be done for example to install and hide malicious programs (rootkits).
  • Crashes, freezes, or disturb the performance of the system or specific applications (Denial of Service attacks)
  • Etc.

In a paper by Chen et al., entitled Linux kernel vulnerabilities: State-of-the-art defenses and open problems, the authors studied 141 kernel vulnerabilities over the year 2010. They classified the vulnerabilities within the following categories:

  • Missing pointer checks
  • Missing permission checks
  • Buffer overflow
  • Integer overflow
  • Uninitialised data
  • Memory mismanagement (leaks, use-after-free, double free)
  • Miscellaneous: NULL dereference, divide by zero, infinite loop, race condition/deadlock

The following two tables are taken from that paper. The first gives the number of vulnerabilities in each category, based on the consequence of exploiting the vulnerability:

Denial of service (e.g., kernel crash) is the most common consequence, and is largely due to NULL pointer dereferences, and to a lesser extent to memory mismanagement (e.g., leaks). Next come vulnerabilities allowing an attacker to read (information disclosure) or write (memory corruption) kernel memory. Unsurprisingly, memory corruption issues are largely due to buffer and integer overflows. Information disclosure is itself mostly due to the kernel sending uninitialised or partially initialised data through I/O channels or to user space: that uninitialised data may contain old kernel data, for example in the case where the kernel’s equivalent of malloc reuses freed memory without zeroing it.

A second table studies the location of these vulnerabilities in the kernel code base:

About a third of the vulnerabilities are located in the core kernel code, that is the part of the code included with every compilation of Linux. The other two thirds of the vulnerabilities are located in kernel modules: driver, networking, filesystem, and sound management code.

Linux: Runtime Defences

Linux offers a set of runtime defences, some specific to its OS nature, others similar to the protections we already covered for applications.

Attack Surface Reduction

Kernel Memory Permissions. Linux implements a strict memory access permission model for the kernel. The goal is to restrict memory permissions for the kernel as much as possible, while still letting the OS do its job properly. It is a direct application of the least-privilege principle. A key part of the relevant policies is that kernel executable code and read-only data must not be writable. The kernel code is particularly sensitive because it runs with supervisor privileges, so preventing it from being written as far as possible is important: this is the write xor execute protection we already covered for applications. Kernel function pointers and sensitive variables are similarly sensitive, and should not be easily writable.

Two memory protection technologies help further isolate the kernel from user space attacks:

  • Supervisor Mode Execution Prevention (SMEP) prevents the kernel from executing code located in user-space memory.
  • Supervisor Mode Access Prevention (SMAP) prevents the kernel from reading/writing user-space memory.

Both technologies are enabled on most modern CPUs and protect against injection and dereference of user space pointers in the kernel (ret2usr attack). Note that user space can still indirectly control a subset of the physmap, a particular area of the kernel memory that is a direct mapping of all physical memory (that by desing include the memory allocated to user space processes). Attacks leveraging that area are called ret2dir. Note that SMAP must be temporarily disabled by the kernel during copy_to/from_user.

Reducing Applications’ Access to System Calls. Most applications make use of only a small subset of the system call interface exposed by Linux. Preventing an application from issuing the system calls it does not need to perform its job is called system call filtering:

System call filtering is a common kernel protection measure against untrusted applications, as a subverted program attempting to attack the kernel may issue system calls it does not normally invoke under legitimate execution. This is widely used in production to harden multi-tenant and sensitive environments, such as Docker containers, Android or Flatpak/Appimage software, etc. This is achieved under Linux with a software technology named seccomp.

A particular problem with system call filtering is how to determine precise, per-application system call blacklists and whitelists? Determining a good black/whitelist manually for every application in a distribution requires a huge amount of effort, so automation is needed. However, automated techniques rely on either static or dynamic analysis, with each method coming with its own downside. Static analysis overestimates the legitimate system calls that can be issued by an application, which translates into filtering rules with low strictness. The coverage limitations of dynamic analysis raise the concern of missing system calls that may be legitimately invoked by an application but are not exercised during analysis, which may lead to legitimate system calls being flagged as an attack in production.

Probabilistic Defences

Similar to user space programs, stack canaries and ASLR are supported by the kernel. These probabilistic protection techniques come with the same pros and cons as for user applications. Canaries allow detecting certain attempts at overwriting return addresses on kernel stacks, however there is only a single canary value for all stack frames on each CPU core. Should this value leak to the attacker, e.g., through a stack buffer overread, the protection may be bypassed. The kernel also applies Address Space Layout Randomisation (KASLR for Kernel ASLR) to the kernel memory areas. Same as for user space it is coarse-grain in nature: upon creation, a random offset is added to large areas including kernel and modules executable code, kernel stacks, vmalloc area, physmap, etc. Because of that coarse granularity of randomisation, a single pointer leak may allow an attacker to break ASLR for the entire area.

Memory Integrity

Regarding control flow integrity, the Linux kernel supports Indirect Branch Tracking, a CPU technology that restricts the target of function pointer calls to a set of legitimate functions. Shadow stacks are not currently supported by the kernel.

To further protect against overflows, the kernel places guard pages at the beginning and end of kernel and user space stacks. Guard pages are unmapped pages that trigger a fault when accessed. Should a linear stack over/underflow escape the stack and hit one of the guard pages, the resulting fault would allow the issue to be detected:

Other protection techniques include macros for arithmetic operations that embed overflow checks, and sanity checks applied to heap metadata to detect corruption during the dynamic allocation and deallocation of memory in the kernel.

Preventing Kernel InfoLeaks

Kernel information leaks happen when data supposed to be private to the kernel becomes inadvertently readable by untrusted security domains such as user space or remote applications. This may allow an attacker to steal sensitive data (e.g., passwords) or infer information about the kernel that may be useful to mount further attacks. A notorious example here are kernel pointers: as we have seen previously, kernel address space layout randomisation is applied on a coarse-grain basis, so having a single kernel pointer leak to user space may allow an attacker to break ASLR and infer part of the kernel memory layout, which is required for many attacks.

To avoid such issues, the kernel must take care not to send to user space buffer or data structures that are only partially initialised: because dynamic memory allocators generally reuse memory without zeroing it out (for performance reasons), such uninitialised memory may contain old kernel data. Although that old data has been freed, it can still be useful to an attacker, e.g., old kernel pointers can still allow to break ASLR. The kernel also takes care of not using addresses as resource identifiers, but rather numbered identifiers such as file descriptors.

A tempting solution to the issue of kernel leaks stemming from reusing dynamically-allocated memory would be to zero out any piece of memory freed with the kernel’s equivalent of free. That way when the memory in question is reused and possibly leaked through a partially-initialised data structure or buffer, it does not contain any information of value to an attacker. Unfortunately, this solution is generally too costly from a performance point of view: calls to free are often located on the critical path of performance-sensitive operations, and the amount of time required to zero out freed memory is often unacceptable.

Linux: Bug Detection

Similarly to applications, dynamic and static analysis techniques have been applied to the Linux kernel with the goal of detecting bugs and security vulnerabilities.

Dynamic Analysis Techniques

Several dynamic analysis techniques are applied to the Linux kernel for bug detection. Kernel sanitisers such as Kernel Address Sanitiser (KASan), Kernel Undefined Behaviour Sanitiser (KUBSan), as well as memory leaks and concurrency (race conditions) sanitisers are widely used to detect memory errors and undefined behaviour at runtime. Lockdep tracks the state of locks to detect deadlocks, double locking, and lock order inversion. Dynamic tracing and instrumentation tools such as ftrace, perf, and eBPF provide runtime observability into kernel behaviour. Finally, fuzzing tools can also be used to protect the interfaces exposed by the kernel to untrusted security domains. We have seen earlier that the system call interface represented a primary attack vector for the kernel, and understandably significant effort has been devoted to fuzzing it.

Syzkaller: Fuzzing the Linux System Call Interface. Syzkaller is a widely used kernel fuzzer targeting the system call interface. It works by injecting malformed system calls into kernel space with the hope of triggering bugs. The kernel under test is generally compiled with sanitisers enabled to maximise the chances of detecting bugs, and executed in a virtual machine. Fuzzing is controlled from the host: for each round of fuzzing a program containing a series of syscalls to execute is generated and executed in the VM. Syzkaller is a coverage-guided fuzzer, meaning that it measures the kernel code coverage executed by each injection of system calls, and uses that information to generate the next series of system calls to execute in a way that maximises the chances of hitting new coverage. Syzkaller fuzzing process is illustrated in the following diagram:

Syzkaller is a grammar-based fuzzer, meaning that the fuzzer models the interface it aims to fuzz, the Linux system call API, as a language. That language is called the Syzlang, which is a grammar used to precisely describe the different system calls Linux supports, how they interact with each other, and the data flowing in and out of that interface. The Syzlang models system calls arguments and their types, the data structures that can be passed between system calls (e.g., a file descriptor can be created by open and used later by read) for file I/O, the length parameter specifying the size of other parameters, etc. This precise knowledge of the system call interface lets Syzkaller optimise the generation and mutation of fuzzing inputs, to maximise the chances of hitting new coverage and discovering bugs.

Below is an excerpt from the Syzlang:

resource fd[int32]: 0xffffffff, AT_FDCWD
resource sock[fd]
resource sock_unix[sock]

socket(...) sock
accept(fd sock, ...) sock
listen(fd sock, backlog int32)

On top of this snippet we can see examples of definitions of data structures flowing through the system call interface: a file descriptor fd is a signed integer on 32 bits, and can take the notable values 0xffffffff or AT_FDCWD. A socket sock is a special type of file descriptor, and a Unix socket is a special type of socket. Next come three examples of system calls. The socket system call returns a socket. The accept system call takes as first parameter a socket file descriptor, and returns another socket file descriptor. Finally, the listen system call also takes a socket file descriptor as first parameter. The Syzlang lets Syzkaller generate for fuzzing series of system calls that make sense together, for example, generate a socket file descriptor with socket and then pass it on to listen.

Syzkaller is widely used by Google to continuously fuzz Linux, Android, FreeBSD, NetBSD, OpenBSD, and gVisor, through a system called Syzbot. It consists in hundreds of VMs fuzzing these OSes 24/7. Thousands of bugs have been uncovered this way, and reported automatically to the OSes’ developers. The results are accessible online on Syzbot’s website. The graph below is taken from that website, and represents the number of bugs discovered (red line) and fixed (green line):

As one can observe, there is an important amount (two thirds) of invalid bugs reported. There are multiple reasons for a bug to be classified as invalid: secondary symptom of an earlier memory corruption, duplicate of another bug, etc.

The Linux Test Project. The Linux Test Project is a repository of test cases maintained by a series of companies and institutions outside of the kernel main tree. It contains many tests targeting various subsystems of Linux: system calls, conformance to the POSIX standard, filesystems, networking, memory management, etc. It also contains test cases reproducing existing kernel CVEs.

Static Analysis Techniques

Several approaches also apply static analysis to the problem of bug detection in the Linux kernel. Pattern-based analysis tools such as Coccinelle and Smatch allow developers to describe common programming mistake patterns in source code. For example, a pattern can specify that for every call to kmalloc there should be a corresponding kfree, enabling automated detection of memory leaks. Control and data flow analysis tools like Sparse let the programmer use compiler attributes to annotate certain properties on objects and memory. For instance, the __user attribute can mark pointers that reference user-space memory (used for example in copy_to/from_user, see above), while __acquires can indicate a lock that is held on function exit but not on entry, helping to detect incorrect locking patterns. Other approaches include formal verification techniques, symbolic execution, and various compiler-based analysis methods.

Similarly to applications, a key challenge for kernel static analysis approaches is their inherent scalability issues, particularly around state explosion. These downsides are particularly problematic when applied to OS kernels, because of the sheer size of their code bases, that are often in the order of millions of lines of code.