Feed readers can be quite wasteful when it comes to bandwidth. A good implementation of HTTP caching, compression, and feed deltas can save both power and bandwidth for users and servers alike. Here is my detailed list of best practices for Atom/RSS feed publishers and feed readers.
These best practices are aimed at feed readers that store feed entries in a database and is only interested in new or changed feed entries. This is a guide to advanced caching techniques put in context, and only a few are specific to syndication feeds. Some familiarity with HTTP headers is assumed.
I’ll now go through each of the bandwidth-saving techniques listed above in order.
Compressed responses
Let us start out big by making everything smaller. Compression is a method where the server and feed reader agrees on a common format for exchanging data in a minimized format. This significantly reduces the bandwidth needed on both ends to update feeds.
A feed reader should include the Accept-Encoding: gzip, deflate request header to announce support for gzip and deflate compression methods. (Bonus points for reading up on and implementing support for newer and more efficient compression methods too!) Likewise, a server announces support by sending the Vary: Accept-Encoding response header with every response.
When a server sees a response with an encoding it supports (like gzip), it should apply that compression format to the response body (not the headers), set the Content-Encoding: gzip header, and process the request as normal. Most modern web servers will perform this automatically by default (or with a little configuration). There’s likely very little need to change anything in an application.
Cache revalidation
When pulling a feed for updates, there will quite often not be any new entries available. Rather than transferring the full contents of the feed again, the feed reader and server can agree that the version of the feed that the feed reader have already got stored is still good enough and abort the transfer. This is achieved using a method called cache revalidation.
When a server wants to indicate that it supports cache revalidation, it will set either or both of the Last-Modified and ETag response headers. A feed reader should store these headers’ values and return them in subsequent requests to the same server as the values of the If-Modified-Since and If-None-Match request headers.
Servers shouldn’t bother with sending an ETag response header. They’re inelegant, cumbersome, and more feed readers support Last-Modified than ETags, so I’ll focus on the more common for reasons that will become apparent in the next section. The Last-Modified header is quite intuitive, and represents the datetime that any of the entries in the feed was last published or modified.
A feed reader should calculate the datetime to set in the If-Modified-Since request header in one of three ways: Either by storing the Last-Modified value from the last successful request. HTTP status codes 200–299 plus 304 indicate successful requests. Don’t just check for 200 OK! Or by storing the time that it last successfully pulled the feed. Or by sorting all the datetimes included in the feed itself, and returning the newest datetime. Which of these methods are used is up to the implementer, but the first is preferred and the two others are good options when the first method can’t be implemented for whatever reason.
When a server sees a request with the If-Modified-Since request header set, it should evaluate whether the feed has indeed been modified since the datetime sent by the feed reader. If the feed haven’t been altered, the server should respond with a 304 Not Modified response status, and abort the request without sending the feed’s contents. This can save a significant amount of bandwidth, especially with feed readers set to pull feeds often. Of the feed have been modified, the full feed is returned as it would be with a normal request. In both scenarios, updated Last-Modified and Cache-Control response headers (covered later) should be returned with the response so that the feed reader will know what to do next.
Feed delta updates
Expanding even further on cache revalidation, a server can use the datetime in the If-Modified-Since request header combined with the A-IM: feed header to filter out undeeded entries from a feed: just needing to transfer the changes since the feed was last modified. This method is called feed delta updates.
A feed reader should announce support for delta feeds by including the A-IM: feed request header when at least one of If-Modified-Since or If-None-Match are also set. If the server supports delta feeds, just the modified entries and the newly published entries will be returned. Feed readers should treat a 226 IM Used response status as it was a regular 200 OK response status.
A server should announce support for feed deltas by including the Vary: A-IM, If-Modified-Since and Cache-Control: im response headers. Usually, you wouldn’t include If-Modified-Since in the Vary header, but when working with delta update; we need to preserve the exact value of this request header as a variant qualifier to ensure everyone will get the correct variant from any intermediary proxies or web caches.
When a server receives a request with both the A-IM: feed and the If-Modified-Since header set, it can use what is known as instance manipulation to modify the contents of the feed itself. The datetime from the If-Modified-Since should be used to determine if any entries have been modified or published since said datetime. If no entries are found, the server should respond with a 304 Not Modified response status and abort without sending back the feed contents.
However, if the sever does have new or modified entries things start to get interesting. Older/stale entries from before the client provided datetime should be filtered out so that only the new and modified entries will be included. This list of only new and modified entries makes up a delta update to the feed. A delta feed should return the IM: feed response header along with either the 226 IM Used or regular 200 OK response status. This reduces the size of the transfer by not retransferring entries that the feed reader have already cached. This can be a significant bandwidth reduction for feeds that publish “full content” or include a lot of embedded media assets.
A server should take care not to resurface too outdated entries when considering which modified entries to include. Consider excluding entries that were published more than a month ago (or maybe ten days) from reappearing in the feed even when it has been modified: it probably isn’t newsworthy anymore, right?
If you’re publishing with WordPress, check out my Feed Delta Updates plugin for WordPress.
Server-hinted next-pull interval
Feed readers traditionally update every feed about at the same time on a fixed schedule. This is a great fallback strategy, but many feeds tell the feed reader when they should next pull for updates. This is communicated through the Cache-Control: max-age=seconds response header, or optionally through Expires: datetime.
A server should send either or both the Cache-Control or the Expires response headers (the “freshness interval headers”). The value of Cache-Control should be set to max-age=seconds, e.g. max-age=14400 for four hours in seconds. This communicates to the feed reader that it doesn’t need to schedule another pull of the same feed until after either now + max-age or the expiration datetime.
Feed readers should apply their default freshness policies (“update every nth hour”) for all feeds, but adjust it on a per-feed basis to accommodate the freshness interval headers that was last returned by the server. Be aware that these change, and that they should be updated for responses that either return a valid feed with status codes in the 2xx range or 304 Not Modified response status.
The freshness interval headers are merely suggestions for when the feed should be pulled next, so feed readers are free to concatenate multiple expired feeds and pull them all in batches. This reduces the feed readers network chattiness, allows wireless communication antennas to enter sleep mode, and saves on battery by limiting CPU and network activities to batches rather than an inefficient low background tricky.
When a server has new entries scheduled to be published in the future, it can lower its max-age responses beforehand the publication time to make sure feed readers will get the scheduled entry sooner. However, be sure to introduce an element of random time drift to avoid having every feed reader pull the feed at the exact same second! This is know as the “thundering horde”. If you’re publishing with WordPress (or want to see a reference implementation), check out my Cache-Control plugin for WordPress that implements the described behavior.
See RFC 7234§4.2 for more details about freshness and caching.
Permanent updates to subscription endpoint
Feed readers fetch the address their told to, and that’s usually what users want them to be doing. However, sometimes websites move domains, change protocol, or settle in to a new publishing system. Most of the time, this also involves permanently redirecting the feed address to a new location.
Update (): Read the followup article Permanent redirects are supposed to be permanent for more details.
The server will signal that the feed reader should update the subscription endpoint with either a 301 Moved Permanently or a 308 Permanent Redirect response status. The same response will include the new address in the Location: URI response header.
Feed readers should follow the redirected location and verify that the new URI endpoint is a working and valid feed. After confirming the new destination is a working feed, the existing subscription endpoint should be update with the new address. This improves pull performance and increases the robustness of the subscription by eliminating the need for repeating meaningless redirects. A feed reader should perform this maintenance task on its own without involving the user. However, if the origin (protocol+domain) changes, the user should be prompted. For example, a dialog saying “A feed subscription from ‘old-domain.com’ has requested to be changed to ‘new-domain.blog’. [Confirm] [Unsubscribe]”.
A feed reader might want to keep track of a redirect, and only update the subscription endpoint after the redirect has been in place for a week or two of days. This is to mitigate the risks of any configuration mistakes or potential “subscription-hijacking” scenarios on the server.
Many feed readers will follow the same redirect on every update several times per day for years on end, even when the server has provided a newer endpoint. This is incredibly pointless as well as a violation of the HTTP specification. Chances are you’ve never heard of permanent redirects being used like this. That is a clear indication that you’ve been exposed to nothing but poor HTTP implementations in the past. To put your mind at ease, here’s the relevant section from RFC 7538 (also applicable to 301 statuses!):
The 308 (Permanent Redirect) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs.
Clients with link editing capabilities ought to automatically re-link references to the effective request URI to one or more of the new references sent by the server, where possible.
Support WebSub
Feed readers and publishers should support WebSub. WebSub turns feed distribution on its head; making it a push instead of pull media. Feed readers can register their subscriptions with a WebSub hub, and the hub will be responsible for pushing updates to the reader.
Feed publishers either need to maintain a hub server or use one of the many freely available hub services. There are many free and open-source hub implementations available. The publisher also needs to notify the hub about udates to their feeds. The update pings can be automated by the publisher.
A WebSub subscription is more efficient for every party, but requires that the feed reader can maintain a public web server. This makes it ideal for web clients, but client apps may have trouble receiving WebSub.
As a closing note, I’d like to point out that there are more headers sent by the server than are being picked up and used on the client side by feed readers. These extra headers, like Vary, are there to support “web accelerators”, reverse-proxies, corporate/network web proxies, and other middleware that needs to be aware of and accommodate for caching.
Please note that every header mentioned in this article that’s being set multiple times (like Vary, Cache-Control, A-IM, and Accept-Encoding) can either be sent individually as duplicates or be combined in to a comma-separated list in a single header. The latter is the strongly preferred for compatibility.