I first tried Syncthing a couple of years ago, but abandoned it because it kept running into file synchronization conflicts. A couple of months ago, I once again tried keeping my files synchronized across devices using Syncthing. I’ve had much more success with it this time.
Syncthing is an open-source peer-to-peer inspired file synchronization utility. It keeps your files encrypted while transferring between your computers and can keep folders in sync over the internet without involving a commercial third-party service provider.
I’ve experienced intermittent critical process failures after setting up Syncthing with directories I use for ongoing projects. The core problem in all cases came down to files that had disappeared from the file system. I didn’t know the cause nor did I even think to blame Syncthing for the issues. I thought the problems must have been caused by a bug in my own software that processed the files in these directories.
The problem would only happen once every ten days or so and I didn’t think much of it. However, the number of files involved has grown and the problem eventually happened several times a day. I first wasted time trying to figure out where in my processes the files were being deleted. Then I turned to the file system itself.
I used a program to monitor all file operations (inotifywatch) to monitor the directories with the disappearing files. I’ll let a summarized version of the program’s output continue the narrative:
./ CREATE example.file ./ DELETE example.file ./ CREATE example.file ./ MOVED_FROM example.file ./.stversions/ MOVED_TO example~20190916-162900.file
The output shows how the example.file is created and deleted, and that a new file with the same name is created. So, far everything works as expected. However, then the newly created file is renamed and moved by Syncthing into a hidden directory called .stversions/.
This directory is used by Syncthing to preserve copies of files that have been changed or deleted on a remote client. It doesn’t preserve copies of file changes or deletions that originated on the local system.
I tried disabling the version control feature on my local client, but the problem didn’t go away. Syncthing would instead outright delete the files instead of moving them aside. I had to disconnect my remote Syncthing instances to stop the problem from reoccurring. This completely defeated Syncthing's purpose, however.
I’ve only seen this problem happen when syncing a directory with thousands of files to two or more remote instances.
I haven’t dug into Syncthing to work out the exact details on what’s happening. However, the following is my best guess at what goes wrong based on the observed behaviors.
I believe what happens is that Syncthing starts syncing the file creation and deletion to the remote instances. While that’s going on and the remote instances talk among each other — this can take some time — I’ve created a new file in the same location. At some point, Syncthing gets a message from one of the remote instances telling it that they’ve deleted that specific path and that it should do the same thing to be in sync.
Syncthing proceeds to diligently delete the file at the specified path without having any checking to make sure it’s the same file and not a different file that just happens to be stored at the same file path.
Overall, I’m impressed with how well Syncthing works and its flexible and secure peer-to-peer design. It’s a good tool but I keep bumping into corner cases and problems with it. For now, I’ll work around the issue mentioned in this blog post and stick with using Syncthing. However, I’d not recommend anyone trust it with their critical files unless they have a separate snapshotting and backup systems in place.
Syncthing is great! But it’s also terrible.
I used Syncthing version 1.2.2.