I’ve been using XFS as a very conservative file system choice for the last four years or so after being burned by BtrFS file system corruptions a couple of times in a row.
However, I’ve been working more with many small files lately to build this blog using a static website generator. Performance hasn’t been great and I’ve also found a need for recording file creation times, something which isn’t natively supported by XFS.
My hardware setup has consisted of two old fashioned spinning software mirrored 4 TB hard drives for my home directory. I’ve used these drives as a backing drive with a Samsung 960 EVO NVMe solid state drive as a LVM cache drive for performance.
I stopped using LVM cache disk a few months ago as I wanted to enable full-drive encryption and it also prevented my system from hibernating; a problem LVM caches share with BCache. I considered trying BCache anyway, but luckily decided against it just days before people started noticing drive corruptions with BCache on GCC 9 and Linux 5.
After a careful review, I decided to slim down my home directory and move over to the NVMe with an LUKS encrypted BtrFS file system. I’ve had bad experiences with this file system in the past, but surely something has happened in the last four years, right?
I decided to give it a fourth chance in the interest of getting a more modern file system. I’d to move a lot of files over to a separate drive, but cleaning up your home directory never hurt anyone (assuming you’ve proper backups.)
I saw bad transfer speeds when copying files from my old home directory residing on a hard disk drive over to BtrFS. The NVMe drive is theoretically capable of 1,28 GiB/s random writes under ideal conditions, but write-speeds were bottlenecked at around 30–45 MiB/s. These were quite underwhelming results.
After everything had been transferred over and setup I also noted lower than expected performance gains with my static website generator. Both of these workloads involve a lot of random file access and writes.
The Ext4 file system was my second choice of file system. I rebooted and repeated my tests on BtrFS a dozen times to get some performance metrics, and decided to reformat and repeat my testing on Ext4. This time around the read-speeds from the old hard disk drive bottlenecked me at around 90–105 MiB/s, which was what I’d expected to see all along. This time around the performance gains with my static website generator were also more significant.
| Hardware | SATA HDD | NVMe SSD | |
|---|---|---|---|
| File system | XFS | BtrFS | Ext4 |
nanoc compile |
89 s | 78 s | 37 s |
nanoc check |
305 s | 260 s | 196 s |
The XFS on a SATA drive isn’t a fair direct comparison, but it serves as a reference for what performance was like before I decided to do something about it.
The static website generator needs to read file system metadata for 8200 files during compilation. Given the nature of this data, we can assume that these are mostly random reads. These numbers were gathered from six dry-runs with no changed files and therefore no writes. The 12,3 % reduction in compilation time with BtrFS is a nice improvement but pales compared to the 58,4 % reduction seen with Ext4 on the same hardware doing the same task.
The check task is involves reading roughly 2000 small files in their entirety and performing some processor intensive tasks on them. The checking task improved 14,6 % with BtrFS and 35,7 % with Ext4.
At this point both the compilation time and checking time is bottlenecked by my processor. Nanoc’s lead developer, Denis Defreyne, is currently working on compiling in parallel to take on the processor bottleneck. I expect that these tasks should complete significantly faster once that’s done, but I suspect that I’d not seen all that much benefit if I’d stayed on my old hardware.
Okay, so what do the numbers in this article tell us? Obviously, it confirms that hardware with faster I/O is indeed faster. However, it also confirms the findings in Phoronix’s Linux 5.0 file system performance benchmarks for randoms and writes with a real-world workload instead of using a synthetic benchmark.
As a nice and unexpected bonus, the time it takes before Linux prompts me for my drive decryption password during boot-up have dropped from about 45-ish seconds to about 6 seconds.