Security-Enhanced Linux (SELinux) is a type of Mandatory Access Control (MAC) in the Linux kernel. It can prevent software from performing unexpected — such as abusive or malicious actions — on your Linux systems. However, … it’s also an unmanageable mess, and I have a much greater understanding of why people recommend that people disable it.
SELinux is one of many layers of security that helps protect your Linux servers (and desktops) from the lions, and tigers, and bears — oh, my! SELinux policies specify which programs, sockets, and files are allowed to interact with each other. It requires everything on the system to be properly labeled with a security context that gets enforced through a policy that maps which labels are allowed to interact.
That might all sound like gobbledygook. Okay, here’s a concrete example: The Lighttpd web server executable is stored at /usr/sbin/lighttpd and is labeled with labeled httpd_exec_t. It reads files from /var/www/ which are labeled as httpd_sys_content_t. Every time Lighttpd wants to read a file, the Kernel checks the current policy set and verifies that httpd_exec_t is allowed to read files labeled httpd_sys_content_t. The Kernel denies the read if it isn’t and logs a policy denial.
Everything happens, mostly, invisible to the system administrator (sysadmin). Everything works seamlessly unless the sysadmin tries to do something not allowed by the current set of policies. Now, most sysadmins are probably not aware of SELinux nor its policy set until they can’t achieve something because it violates the policies.
People then seek advice on the web on how to fix the problem. I’ve always cringed at what I’ve perceived to be inexperienced users suggesting to others that they disable SELinux — which is a terrible idea, right? The right solution is to learn about SELinux, and then make a tiny tweak to the policies to specifically allow what they want to achieve.
I can’t provide easy-to-follow steps on how to solve the problem, though. You need to learn a full set of tools to have any hope of adjusting things. There are very few good documentation sources available that describe SELinux and how to manage it.
It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study.
Over time, I’ve realized that the advice to turn it off isn’t necessarily bad advice. Either SELinux won’t give you any problems as you stay within the parameters of the system policy, or you need to decide between learning a complicated subject or turning off a security feature.
At the heart of the problem is that the SELinux policies themselves are sort of magical. The policies have probably been provided by the maintainers of your Linux distribution, e.g., Fedora Linux. There’s nowhere on the system where you can view the policies and look up why something might or might not work. The policies also change over time, without any warning.
Whatever the reason, you’ll likely encounter some software that stops working after a system update or after you’ve changed its configuration. The change resulted in an SELinux enforcement action causing something, like a file read operation, to be denied on the system.
The SELinux denial audit log messages are too vague. You’re told that a label was denied reading from another label. Okay, what do those labels mean? Which programs? Which files, sockets, or whatever? You’re given the right information about why something was denied, but it’s unactionable without context.
In the best-case scenario, you’ll get (bad) advice from the denial message. The denial message might tell you to run some magical commands to allow stuff to happen. Great, that’s what you want! So, you run the audit2allow command as instructed, and end up with some policy blob files.
God only knows what changes the blobs do; you can’t be expected to, nor are you given enough information to evaluate them. Then you’re told to install it with the semodule command. You say, “okay”, and your problems have gone away for now.
Then six months later, an update to the system made some changes to the default system policies. The module you installed earlier references some labels or something that’s no longer defined, or some other conflict occurs. Then what do you do? You don’t know what the module did, where it was installed, or how to remove it.
This isn’t just something that happens when you blindly follow suggestions to run audit2allow. Package updates and policy changes often happen out-of-sync. A package’s SELinux policy is — most often — not part of the package but is installed as part of a larger meta package containing tens of thousands of policies (e.g., the Fedora Linux project’s selinux-policy mega-package). So, you’ll end up with breakage from the delay between upstream project changes making their way into an updated package, and the policy package receiving its next update.
The solution? Keep updating your system and hope that, within a week or so, the problem will have been resolved. That’s not always an option with business (or fun) critical software.
You can downgrade the package to the earlier version. Downgrading a security patch isn’t advisable and not all software even supports running older versions on top of configuration and databases that have been tainted by a newer version of itself. You risk introducing more problems.
You can try to run the magical commands to create a local policy patch, but then you might run into problems again further down the line. The best option in this situation is to, at least temporarily, put SELinux into permissive mode. Permissive mode logs denials but doesn’t enforce them the way the regular enforcing mode does.
Red Hat Enterprise Linux (RHEL) has some of the most accessible documentation on creating custom policies. (Red Hat is the same organization that manages Fedora Linux.) The documentation essentially says to run some commands that auto-generate policies for you. The RHEL documentation is written in a way that makes it clear you’re expected to pay Red Hat for consultants to manage custom SELinux policy needs for you.
The thing I’ve started to realize over time is that it’s probably best to just leave SELinux in permissive mode. The enforcing mode might prevent malicious actors from taking advantage of a weak point in my system’s security. However, it’s definitely preventing me from getting my work done.
In the last two weeks, completely unrelated issues have cost me hours and days to troubleshoot and deploy fixes to issues introduced by SELinux. The issues seem to have developed on their own and out of my control. Some were caused by policy changes, some by software package changes, and some by my configuration changes. I keep up to date on changes in critical packages, but I didn’t anticipate faults introduced by the SELinux policy.
That’s because SELinux doesn’t enforce my policies. I rely on the Fedora Linux project to develop and maintain policies for the software I rely on. However, I’m not familiar with the policies that are in place on my system. There’s no configuration file or reference tool I can check to see or modify the policies. They’re just there.
The issues I’ve ran into lately have been severe, too. One system (with no custom policies) was left unbootable after. The server that handles my blog’s newsletter sent an empty newsletter as the email signing agent could no longer talk to the mail transport agent (MTA).
From my perspective, SELinux broke a perfectly functional system. My system was built on top of someone else’s expectations and policies, and I wasn’t even aware of them, nor was I given a heads up before they changed.
I also decided to try the Fedora Linux 36 Beta on my laptop the same week. It’s just a few days before the final release, so I figured it would be fine. After the update, I was locked out as the login screen no longer could read some files needed to authenticate my login. Disabling SELinux was the only quick solution. (Relabeling the system didn’t fix the problem.)
Magical incantations — whether repeated from an online tutorial, documentation, or generated by a management tool — doesn’t really address the fundamental problem: SELinux is just too damned difficult to configure! It’s time consuming and complex even to inspect and parse the default policies just to understand what’s going on in the system. Audit log messages that suggest you run some random commands is just a band-aid on a more serious usability problem.
I want more control of SELinux policies on my system, and I want much easier tools to manage them. Auto-generated magical modules that vanish deep into the system only to pop up causing issues months later simply won’t do. I’d much prefer some plain text configuration files kept in /etc/selinux with a simple rule syntax and the ability to comment on what the policy does.
Frankly, I much prefer the security settings provided by systemd for the services it manages. They don’t provide the same controls as SELinux, but they get much of the same job done and are far easier to observe, modify, and understand.
SELinux is essentially a bag of security magic tricks orchestrated by unseen puppeteers. All I can do to influence the show is to pull more and more blobs out of the red magician’s hats/utilities.
The only reliable alternative where you have any hope of staying on top of this is to scrap everything provided by your Linux distribution and start with a clean slate. That’ll cost you weeks of your life to set up even for a basic web server. The only rational option is to leave it on but turn it off if it ever gets in your way.