Ctrl.blog

Introduction to distribution via BitTorrent

BitTorrent is a protocol where files that people want to download are cut up into small pieces. Anytime someone wants to download a file, they can download pieces from multiple other people and put it back together using a BitTorrent client. For larger file transfers, this is often much faster than downloading directly from one server that almost always imposes bandwidth limits on each user to avoid one user throttling the server. The distributed nature of BitTorrent means no one user, called a ‘peer’, will be carrying all the distribution load and can thus be used to lower bandwidth and server costs while users often get much greater total download speeds from multiple peers.

Distributing files through BitTorrent is very easy. Any client can create torrents, and all you need to do to publish them is to either share a torrent file or a Magnet link with the world. The client you created the torrent on will seed the file to anyone who is interested in downloading it. But how do you share files over BitTorrent and ensure that they will remain available for a long time period? You’ll either need a dedicated commitment to never shutting down your laptop where the client resides, or set up a server to seed the torrent from. I’ll tackle the latter approach.

The peer swarm, the information about who has which part of which file, has been tracked by a centralized server called a tracker. When you create your torrent file, you can embed tracker information inside the file to tell everyone where to go to discover who has the file (or which pieces) and is also online.

The more modern approach of relying on the Distributed Hash Table (DHT), is entirely decentralized and requires no central tracking server. Each user of BitTorrent manages a tiny fraction of the overall DHT network and the users themselves manage the role among themselves that the tracker previously handled.

Distributing through BitTorrent means you give up fine-control over the distribution of your files, but you receive free bandwidth from everyone who wants to download the file in return. For large files, the cost savings in bandwidth and infrastructure can be enormous for a popular file. By not relying on a tracker, your server doesn’t even need to coordinate the download between users and everything is self-managing. All you need to do is to share your files with the BitTorrent community and continue to share it over time to ensure long-time availability of your files. That is where this tutorial comes in.

Note that as the requirements stated above may suggest, this setup will not provide fast transfer speeds but only act as a low-cost and slow seeder that can remain available over time. You can experiment with increasing memory, peer, and bandwidth limits if you want to ensure fast download as well as availability.

Installing and configuring Transmission

No special software is needed to share a file on BitTorrent and any client can be used. I’ve found the Transmission client very to be the best alternative for use on a server as it has good compatibility with other clients, support all the recent protocol standards, is easy to use, and has a low memory footprint.

To get started, install the Transmission daemon (non-graphical variant for server use) from your package manager and start it once to create the default files and directories that you’ll need to work with later.

dnf install transmission-daemon
systemctl start transmission-daemon

Then create the /var/lib/transmission/seeding directory and assign it to the new “transmission” user account that was created when you installed Transmission:

mkdir -p /var/lib/transmission/seeding
chown transmission:transmission /var/lib/transmission/seeding

You should now have a set of directories and default configuration files for Transmission. Make sure you stop the Transmission service before adjusting Transmission’s main configuration file. Otherwise, your changes will be overwritten when Transmission exits.

systemctl stop transmission-daemon

Then proceed to open up and modify /var/lib/transmission/.config/transmission-daemon/settings.json in your favorite text editor. It should already contain all the default configuration options in their default states. Here is a synopsis of the changes that you should make to the defaults:

"download-dir": "/var/lib/transmission/seeding",
"incomplete-dir": "/var/lib/transmission/seeding",
"dht-enabled": true,
"pex-enabled": true,
"peer-limit-per-torrent": 10,
"port-forwarding-enabled": false,
"queue-stalled-enabled": false,
"speed-limit-down": 50,
"speed-limit-down-enabled": true,
"speed-limit-up": 50,
"speed-limit-up-enabled": true,
"upload-slots-per-torrent": 5,
"user-has-given-informed-consent": true,
"watch-dir": "/var/lib/transmission/seeding",
"watch-dir-enabled": true,

The rest of this section will cover each of these options in more depth. You may wish to skip ahead to the next section if you feel comfortable that you understood the above options and values.

watch-dir and watch-dir-enabled: Transmission will look for new torrent files in the destination directory, and will add these to the downloading and seed list. This directory is where you’ll add new torrents later that you wish to distribute.

The peer-limit-for-torrent and upload-slots-per-torrent: The first value should be approximately double the value of the latter. The peer limit controls how many people your server is willing to communicate to about the torrent, and the latter how many it’s willing to upload it to at any given time. These values should remain low even when interest for the files is high as the distributed nature of BitTorrent means that other people will also be distributing your files on your behalf. Keeping the values low is one of the most important metrics for keeping Transmission’s memory usage low.

pex-enable: With Peer-Exchange (PEX) enabled, both people with and without an upload slot will be informed about other people who have the entire or partial file available. By sharing this information, people can more easily discover other users who do have available upload slots. Asking users who already know about other users who has a file is usually more efficient than the slower to act DHT network. This system is why your peer queue limit should be higher than your upload slot limit.

speed-limit-down and speed-limit-up: Controls the maximum upload and download bandwidth in KiB. Setting this low many reduce your bandwidth costs as clients will be more likely to choose other peers when available over your slower peer. As you’ll not wish to bear the brunt of the distribution bandwidth yourself, these should only be increased if you’ve more bandwidth and memory to spear.

queue-stalled-enabled: Set to disabled to prevent Transmission from ever stopping to seed your files in periods when there’s low interest in your files. As Transmission is serving as a long-term availability distribution node for your files, it shouldn’t ever pause any of your torrents.

dht-enabled: Register your client and torrents in the global distributed hash table (DHT) network so that peers can locate you as a seeder. On by default, and you shouldn’t turn this off even if you’ve chosen to use a tracker.

More information about Transmission’s many configuration options is available from the ConfigFiles wiki page.

Configuring the system for Transmission

Next, we’ll tweak some of the system configurations for Transmission including the system service file (‘initscript’) and the system firewall. We’ll start by adjusting the system service file to keep Transmission’s footprint as minimal as possible.

If you’re not intending to run Transmission on a low-end system or have unlimited bandwidth, you’ll want to carefully review the man pages of systemd.exec and systemd.resource-control. For everyone using anything between a reasonably priced hosted server and a Raspberry Pi, the below suggestions should work great.

Start by copying the system-provided systemd service file for Transmission to the location for customized service files to avoid having your modifications overwritten with future system updates:

cp /usr/lib/systemd/system/transmission-daemon.service /etc/systemd/system/

The open up and modify /etc/systemd/system/transmission-daemon.service in your favorite text editors. Add these statements at the bottom of the existing [Service] section of the file (omitting the ellipsis):

[Service]
Nice=16
IOSchedulingClass=idle
CPUSchedulingPolicy=idle
MemoryMax=25M
MemoryHigh=35M
ProtectSystem=true

These options specify that the process is to run with low system priority and with low impact on the CPU and disk, as well as keep the memory usage within low parameters. This will prevent Transmission from eating all your available system resources.

After installing the new copy of the service file, you’ll need to reload the systemd service cache in order for the new file to be visible to the system:

systemctl daemon-reload

The last step before we start the Transmission service and start looking into adding torrents, is to allow BitTorrent to talk through the system firewall (FirewallD):

firewall-cmd --permanent --service="transmission-client" --add-port "51413/tcp"
firewall-cmd --permanent --zone="public" --add-service="transmission-client"
firewall-cmd --reload

Note that the default transmission-client service only opens up TCP port 51413. The DHT and µTP protocol extensions require UDP to be available on the same port well. Pending a fix for this in firewalld, you must manually include UDP port 51413 as shown below when adding the transmission service to your firewall rules.

If there are any intermediary proxies, firewalls, or NAT devices you may need to configure port-forwarding or otherwise allow TCP/UDP traffic with your server on port 51413.

You’re now ready to start up the transmission-daemon, and if that runs with no errors continue to enable the new service as a default service that will start on every system boot:

systemctl enable --now transmission-daemon

By now, you should have gotten yourself a new BitTorrent seedbox server!

Adding torrents to seed

A BitTorrent seedbox is only as good as the torrent it distributes. You can create a torrent form any file using any BitTorrent client such as Transmission (available to all operating systems with graphical user-interface), or using a specialist tool to generate the torrent such as my mitorrent program for generating torrents. You shouldn’t set a tracker address as your seedbox should be discoverable over DHT. Operating a tracker is a very bandwidth and server resource-intensive task, and with DHT there’s no longer any need for it either.

As we configured Transmission to watch for new torrents and to store its downloaded data in the /var/lib/transmission/seeding directory, you can simply place your torrent files and data files in that directory.

Alternatively, you can start seeding from a BitTorrent client on your PC transfer the data files and the torrent file itself over BitTorrent. To tell your seedbox about the torrents you want, execute the transmission-remote command and add the torrent’s Magnet link. You can find this by right-clicking on transfer item in most BitTorrent clients and choosing “Copy Magnet link”. The next example command will add and download a reference to this article on your seedbox; just to demonstrate how you can add a Magnet link:

transmission-remote --add "magnet:?xt=urn:btih:8fd5f93e4159ae2341812c5c3645fafc4c5492a7"

If you’ve set up multiple seed boxes, say at different data centers around the world, then you only need to send either the torrent file or Magnet link to each server. They’ll download and distribute the data files among themselves as soon as one of them — or even your local BitTorrent client — starts sharing the file. It’s quite quick and easy to scale. Now you’re thinking with torrents.

Hosted options

If you’re not willing or able to configure and host your files, you may want to consider some hosted BitTorrent distribution options.

Amazon Simple Storage Server (S3) offers zero-configuration BitTorrent that allows you to distribute your files via Amazon’s BitTorrent trackers. This is only an option if you want to use stateful tracker-based peer discovery as Amazon doesn’t publish your torrents to the DHT network.

Amazon charge per gigabyte/month of storage, plus per gigabit transfer from their servers. Costs will be reduced by the distributed nature of BitTorrent, but as Amazon control the tracker and are financially motivated to serve your files themselves … let us just say I wouldn’t be surprised if the tracker favors Amazon’s hosted peers that they can charge for over other peers.

Mininova offers the cost-free Mininova Content Distribution for “high-quality content producers”. It’s unclear who this is aimed at, but they only seem to accept ebooks, music, and video produces. If you’re interested in distributing other materials through them, I’d suggest you contact them and ask anyway.

Update (2017-09): Mininova was discontinued in April 2017, and no longer offer BitTorrent distribution services.

Like Amazon S3, Mininova will host and seed your files through their own BitTorrent tracker but not over DHT. This is again a weird decision as I would believe that they’d be interested in keeping costs down as the service is provided free of charge. I contacted Mininova to ask how long they host the content, and was told they’ll host it indefinitely but noted that “[Mininova] might start dropping old content that nobody is downloading”. Amazon or your seedbox seems to be a more reliable long-term option, though Mininova definitely has the lowest barrier to entry.