Three weeks ago, I wrote systemd service sandboxing and security hardening 101: an introduction to Linux security features for service processes managed by systemd. It proved enormously popular, so I thought I’d follow it up with a more complete example.
This week, I’ll explore how you can use some of the more advanced security features offered by systemd. You’ll want to read the 101-introduction before proceeding with this article.
Last week, researchers at Qualys disclosed a remote code execution (RCE) vulnerability in OpenSMTPD: an open-source email server. This seems like an opportune time to make sure you’ve locked down this service. It will serve as our example service for this tutorial.
Most parts of OpenSMTPD is designed to run in unprivileged processes. However, this was a “worst-case scenario”, as Gilles Chehade put it. The vulnerability lets attackers execute remote commands with full administrative privileges. Remotely executed arbitrary code running rampant is the last thing you want on your email server.
I’m an OpenSMTPD user and my server was actively targeted by this vulnerability. Hopeful attackers were actively trying to exploit it hours after the vulnerability was disclosed publicly. I’ll be honest here: I’d not done my job properly and hadn’t secured the smtpd service running on my system. Sheer dumb luck protected my system against exploitation.
Sandboxing is all about restricting a service’s permissions, access, and capabilities to a point where it can’t perform any options except what’s expected of it. To block unexpected behavior, you’ll need to gather detailed knowledge of how it’s expected to work.
I’ll start with some of the most complex security directives first. They require a bit of effort, but can make the most positive impact to your system security.
I suggest you apply directives one at a time and thoroughly test that your service still performs as expected after applying each one. Alternatively, you can spend days and days reviewing how the software is built. However, trial and error is the quickest approach to get you up and running here.
Restricting capabilities
Linux kernel capabilities are per-process thread security policies that control access to specific kernel features. You can increase security and reduce the impact of an intrusion by limiting services to expected capabilities only.
The available capability sets are listed in the capabilities(7) manual page. The manual’s explenations for each capability assume a developer’s mindset, but you can search online for discussion of each one in more detail.
If you don’t know what capabilities your service requires then here’s how to work it out. You’ll notice that the manual page lists system calls, e.g. chroot(2). If you’re securing an open-source program service, then you can locate and search through its source code for these specific system calls.
Alternatively, you can try removing all capabilities and see if and how the service fails. Explicitly set the CapabilityBoundingSet= directive to nothing to remove all capabilities.
I was familiar with OpenSMTPD’s architecture so I knew which capabilities it would need. I’ve set the following allow-listed capabilities for my instance of the OpenSMTPD’s systemd.service:
CapabilityBoundingSet=CAP_SYS_CHROOT
CapabilityBoundingSet=CAP_SETUID
CapabilityBoundingSet=CAP_SETGID
CapabilityBoundingSet=CAP_CHOWN
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=OpenSMTPD runs as the root user and spawns unprivileged child processes in a minimal root directory. It needs the capability to change the root directory (chroot) and assign these processes user and group identities (UID/GUID) Lastly, the process needs to listen to several privileged network ports (ports below 1024).
We don’t want any of these child processes (or any compromised process spawned from the process) to inherit these capabilities. A compromised process could for example potentially take over the role as the web server (ports 80 and 443).
Set the AmbientCapabilities option to nothing to prevent any child processes from inheriting their parent’s capabilities.
Allow-listing system calls
We can also block or allow-list specific system calls (syscall) or sets of syscalls. This gives an even more granular control than kernel capability sets. The systemd.exec(5) manual page contains a few predefined syscall sets.
@system-service is one such set. It contains some common syscalls used by services. It notably removes the ability to reboot the system, interfere with swap memory, and change the system clock. Few services need these capabilities.
I’ve set the following allow-list of syscalls for OpenSMTPD:
SystemCallFilter=@system-service
SystemCallFilter=@mount
SystemCallErrorNumber=EPERMOpenSMTPD needs @mount in order to use chroot.
Limiting access to the file system
The Linux kernel can change how a process sees the file system. This can be a powerful way to limit what a compromised process has access to and how much damage it can cause to your system.
You can prevent it from reading into the /tmp (temporary files) of other processes or reading or writing to /dev (hardware devices including disks). You can also mount the entire file system read-only.
This last capability can help you prevent malicious code running on your system from getting a persistent foothold on your system. This is so important it should probably be the default, and require you to turn it off to make your system less secure.
The following configuration does all of the above:
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strictHowever, the OpenSMTPD email server wouldn’t be much use to anyone if it couldn’t write anywhere on the disk. It needs access to some specific directories to store runtime information, state, and your emails OpenSMTPD store all of these things under different points in /var.
You can override the read-only file-system restriction for specific paths using the ReadWritePaths directive. However, you can make things even more secure by creating a fake temporary file system and only mapping the required directories to it. This will act as an allow-list of read-write paths.
The following example configuration for OpenSMTPD demonstrates this approach:
TemporaryFileSystem=/var
TemporaryFileSystem=/var/empty/smtpd
TemporaryFileSystem=/var/run
BindPaths=/var/spool/clientmqueue
BindPaths=/var/spool/lpd
BindPaths=/var/spool/mail
BindPaths=/var/spool/mqueue
BindPaths=/var/spool/smtpd
BindPaths=/var/mailBy now, you’re probably wondering how to identify the above list of directories. It may be different in your configuration! I’ll walk you through the process for the /etc (configuration) directory.
You can work out the required directories by reading the available documentation and source code. However, this can take all day and you may simply not have the required programming skills.
The easiest way is to run the software through strace and use it as normal. strace can track and print the file system operations a program performs. More importantly, it can print the file paths a program interacts with.
The following command launches OpenSMTPD and traces which files it interacts with under the /etc directory.
/usr/bin/strace -f \
/usr/sbin/smtpd 2>&1 | \
egrep -o '"/etc[^"]+"'This should give you the information you need to create an allow-list. Note that you must use the program normally for a while as its behavior, and thus file operations, may change over time and when using different functionality.
Here’s the configuration I came up with for my instance of OpenSMTPD.
TemporaryFileSystem=/etc
BindReadOnlyPaths=/etc/aliases
BindReadOnlyPaths=/etc/crypto-policies/
BindReadOnlyPaths=/etc/group
BindReadOnlyPaths=/etc/hosts
BindReadOnlyPaths=/etc/opensmtpd/
BindReadOnlyPaths=/etc/letsencrypt/
BindReadOnlyPaths=/etc/localtime
BindReadOnlyPaths=/etc/nsswitch.conf
BindReadOnlyPaths=/etc/passwd
BindReadOnlyPaths=/etc/pki/
BindReadOnlyPaths=/etc/resolv.conf
BindReadOnlyPaths=/etc/services
BindReadOnlyPaths=/etc/userThese paths all fit with what we’d expect from an email service. They’re notably all read-only access. It can’t suddenly change the DNS resolvers, interfere with cyrpto-library policies, or change the configuration of a completely unrelated program.
The system’s persistent configuration should remain fairly safe in the event of a service compromise.
You can build on the above to cover other root-level directories too. However, you may want to consider a blocklist instead. It can be far quicker to come up with and you can share it among multiple different services.
The following blocklist restricts access to programs commonly invoked in shell-based remote execution vulnerabilities.
InaccessiblePaths=/usr/bin/at
InaccessiblePaths=/usr/bin/cron
InaccessiblePaths=/usr/bin/bash
InaccessiblePaths=/usr/bin/sh
InaccessiblePaths=/usr/bin/zsh
InaccessiblePaths=/usr/bin/wget
InaccessiblePaths=/usr/bin/curl
InaccessiblePaths=/usr/bin/ssh
InaccessiblePaths=/usr/bin/scp
InaccessiblePaths=/usr/bin/python
InaccessiblePaths=/usr/bin/perl
InaccessiblePaths=/usr/local/Your email server is unlikely to have a legitimate reason to execute any of these programs. Neither do many of your other services.
Prohibit risky and obscure behavior
There are a couple of more security directives in systemd.exec(5) manual page. You can likely enable all of the following directives for most of the services on your systems with no ill effects. (Except for the first directive.)
I recommend you enable them one by one and test your service after enabling each one.
MemoryDenyWriteExecute=true
ProtectHostname=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
LockPersonality=true
RestrictSUIDSGID=true
SystemCallArchitectures=nativeI’ll not go into more detail on any of these directives. Some of them are self-explanatory but they all prohibit the use of rarely used and obscure features. You can find more information about each of them in the manual page.
I hope this article has raised your awareness of some of the security features you can enable through systemd. Over time, I believe many Linux distributions will enable at least some of these directives by default for most of their systemd.service. Ultimately, it will be up to systems administrators to configure more restrictive and secure policies based on their usage.
These directives combined would have stopped the specific remote code execution vulnerability that afflicted OpenSMTPD. However, the key takeaway is that you should strive to sandbox long-running and internet-exposed services. There’s no need for your web server to be able to load a kernel module, your email server to change the hostname, or your DNS server to launch wget and schedule reoccurring tasks with cron.
You shouldn’t rely on systemd alone to save the day. However, it can be an additional layer of security to help protect your systems in the event of an acively exploited remote code execution vulnerability.