Last week was a busy week for me email server-wise. Here’s what happened and why you were sent an empty email newsletter this week.
Roskomnadzor, the Russian telecommunications regulator, is on the warpath against privacy focused European email service providers. Last week, it ordered Russian internet service providers to block ProtonMail and StartMail.
My primary email provider is Mailbox.org (review) and it has a similar focus on privacy. Roskomnadzor has already come after them seeking to block access to their services in Russia.
Update (): “[…] Roskomnadzor [has] declared their intention to withdraw the petition to ban access to mailbox.org in Russia”, according to the Mailbox.org blog.
This set me off scrambling to set up a backup mail server (or “backup MX.”) Email is a distributed system and domain administrators can configure multiple email servers to handle incoming emails to a domain.
The email system wasn’t designed to bypass state-issued blocking of an email service provider. Never the less, you can use an unblocked backup MX to get your emails delivered despite the primary mail servers being blocked.
With a backup MX hosted on an unblocked domain, email servers located in Russia can still deliver emails. Delivery may take longer than normal as the sender’s email server will need to try the blocked servers first and then fall back to the backup MX. This is also good for redundancy in case of service interruptions at Mailbox.
I already have a self-hosted email server used for the blog’s newsletter. I didn’t want to manage two email servers. Instead, I reconfigured the newsletter email service to also act as the backup mail server for my domains.
While I was busy reconfiguring my domains and email server (it only took about ten minutes), news hit about a remote code execution vulnerability in OpenSMTPD. OpenSMTPD is the open-source email server software I’m using. I needed to double-check on something with my configuration and Bing helpfully put the news of the vulnerability at the top of the results.
I quickly checked and confirmed attempts at exploiting the vulnerability in my email server logs. My server wasn’t vulnerable because I was running an outdated version of OpenSMTPD. The OpenSMTPD package in Fedora Linux’s software repositories hadn't been updated in two years. For once, I’m grateful that they didn’t keep up with updates.
I didn’t realize that my server wasn’t vulnerable right away, however. Instead, I spent the next half an hour updating the package from upstream sources, testing, and deploying the newer version.
There had been a configuration format change between the versions that required I spend even more time on it. One of OpenSMTPD lead developer’s had provided a useful migration guide that made the migration fairly smooth.
I later confirmed that my version wasn’t vulnerable to this or any other vulnerability relevant to my configuration.
I’ve also been busy tightening the Linux kernel-provided security protections managed through systemd for the OpenSMTPD service. I had some in place but there was plenty of room to tighten the service’s security even further. I’ve written about this as a more advanced guide/follow-up to service sandboxing and security hardening 101.
I’m sorry for sending everyone an empty email newsletter this week. It miraculously had nothing to do with me messing about with the email delivery server.
For the last two weeks, I’ve suffered from acute bronchitis with coughing fits that have kept me awake all night. Long story short, there were no new articles to send out in the newsletter this week.
In this situation, my newsletter system is supposed to defer sending it until there are at least two new articles. Unfortunately, the number zero exposed a bug in the sending routine that hadn’t come up when I was able to keep up with my regular writing.
In my defense, this could only have happened in a weakly-typed programming language where an empty array can evaluate to a boolean false. Oh, wait. I chose to write this in PHP so that’s on me as well.
It was an unfortunate mistake and I’m sorry about cluttering your inboxes with a pointless email. I’ve added even more checks to prevent this from happening again.
So, should everyone run their own email servers and deliver their own email newsletters? My recent experiences haven’t dissuaded me from believing that it’s still the right thing to do. Russia’s decision to block my own email provider reinforced my belief that SMTP must remain decentralized for the greater good.