Most Linux distributions with enforced Security-Enhanced Linux (SELinux) policies won’t allow the Apache Web Server to connect to an OCSP responder server by default. Here is how you adjust your SELinux policies to allow Apache to perform OCSP stapling, and what it means for your server security.
This tutorial assumes that you’re running a Linux distribution with a default SELinux policy set resembling those from SELinux Reference Policy by Tresys Technology. (Including recent versions of CentOS, Fedora Linux, Red Hat Enterprise Linux (RHEL), and optionally most others.)
OCSP stapling is a process where your server will attach up-to-date and cryptographically signed response from an OCSP server about your certificate’s current validity to your certificate. Visitors will receive both the certificate for your domain and a signed OCSP response from your web server in one-go rather than having to send a separate OCSP request themselves. This can speed up website load times especially for visitors on mobile or high-latency connections.
Permission denied
The policy that causes trouble for OCSP stapling is meant to protect against code, possibly from injected scripts through a vulnerability in a web application, being allowed to open sockets and connect to network locations. Something like the CGI or PHP interpreter running in a web server could be used to download remote code or transmit data out of the system.
Apache’s error log messages will show that Apache’s process (httpd, in this example) failed to obtain permissions to connect to the OCSP responder (Let’s Encrypt, in this example):
[ssl:error] Permission denied: AH01974: could not connect to OCSP responder 'ocsp.int-x3.letsencrypt.org' [ssl:error] AH01941: stapling_renew_response: responder error
This would manifest as error “SEC_ERROR_OCSP_TRY_SERVER_LATER” in Firefox who’ll see a stapled OCSP error response from Apache and block visitors from accessing the website. You’ll find a matching message in the SELinux audit log showing that Apache was denied permissions to connect to a remote port 80 over a TCP socket:
type=AVC msg=audit(1514069285.431:489): avc: denied { name_connect } for pid=1856 comm="httpd" dest=80 scontext=system_u:system_r:httpd_t:s 0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=0
Granting access
At this point, you could use audit2allow to generate a new policy from the above log entry to allow Apache access to open a tcp_socket. However, there’s a ready-made boolean switch which in the SELinux Reference Policy that covers this specific scenario. You can enable the httpd_can_network_connect boolean to allow Apache to connect to network locations with the following command:
setsebool -P httpd_can_network_connect 1You must restart the web server after issuing this command for it to take effect on a running server instance. This is the same boolean you’d use to allow connections to a reverse or intermediary proxy, or to allow network connections from a script executed as the Apache user.
Browsers who trust stapled OCSP response, such as Firefox, won’t retry a OCSP request on their own in error conditions. Personally, I believe it’s better to not staple error responses (in case of server configuration issues like this, or a network issues) and let clients retry for themselves. This potentially increases the load on the OCSP responder, but at least clients won’t give up on connecting to your website. You can change this behavior by modifying the following option in your Apache configuration:
SSLStaplingReturnResponderErrors offLocking down network access again
After you’ve changed the above option, you’ve also opened up your web server to some types of attacks. However, please do note that this doesn’t make your system any less secure than an instance of Apache running on a system that doesn’t enforce any SELinux policies.
You can re-impose more specific limitations by configuring the system firewall to only allow outgoing connections initiated on the system to go to a allow-listed set of addresses, including the OCSP responder for your certificates (the specific address is embedded in the individual certificate files.) Maintaining an allow-list of allowed remote hosts in the firewall will also help secure the system from other attack vectors, but it can be complex to manage.
Practically speaking, you must either accept the risk of allowing your web server to initiate remote connections or disable OCSP stapling to remove the need for making remote connections. Please note that removing OCSP stapling can negatively affect your website performance for your visitors as each client will have to do their own OCSP requests instead.
Didn’t this “just work” before?
Yes, it did work out-of-the-box sometimes on CentOS, Fedora, and RHEL. These distributions have an extra SELinux policy boolean called httpd_graceful_shutdown that allows TCP connections to port 80 (this isn’t clear from the name.) Many certificate comes with OCSP responder servers listening on port 80, so they would happen to work by in the default configuration.
The default value of httpd_graceful_shutdown was changed to deny in , causing all OCSP stapling requests and checks to fail by default starting with Fedora Linux 27 and later releases. This doesn’t affect systems that are updated from older policy sets, but any system installed after the change will have the new default policies.
Enabling the httpd_can_network_connect policy boolean instead of httpd_graceful_shutdown will ensure that all OCSP responders will work regardless of which TCP port they listen on. The policy also has much clearer consequences than the more ambiguously named httpd_graceful_shutdown policy.