I keep finding myself explaining how the Vary HTTP response header works and what effects it has on reverse-proxies and caching. Let’s find out if I’ve gotten any good at explaining it!
The Vary response header lists out request headers that affect the returned response. It sounds simple, but what does that mean? Before explaining the header itself, I’ll drop in a short explanation of how web caches work.
A “web accelerator”, reverse-proxy, corporate/network web cache all have one thing in common: They act as intermediaries between the user and the server. Caches use HTTP response headers like Cache-Control and Last-Modified to ensure that it can consider a file “fresh”/up-to-date. While files are still fresh, caches may return the cached file variant to the user without involving the server. If the cached file is considered outdated, the cache will try to re-validate and retrieve an updated version of the cached file. These types of caches are used for many reasons, including to reduce network costs, load balancing and geographical distribution of servers, and content filtering.
However, things are never that simple. Servers can be running complex applications that return different responses that are optimized for the user based on the characteristics of the user’s request. For example, the Accept-Language: fr, en-CA request header means the user has a preference for either a French or Canadian English language variant of the page. An Accept-Language: en-CA, en-US;q=0.8, en;q=0.6 request header means the user prefers a Canadian, American, or other English variants.
Caches can’t know which request headers the upstream server will consider important without explicit instructions. This is where the Vary response header comes in. For the above example, the server should respond with the header Vary: Accept-Language. This would tell any intermediary caches that the returned file should only be returned to users with the exact same Accept-Language header as the user who triggered the page generation. If another user’s request headers don’t match a cached variant of the file, the cache must ask the upstream server for a response matching the new request parameters.
With some more HTTP-literate caches, you can improve variant-matching and cache efficiency by setting more response headers. By extending the above example with the two Accept-Language request headers used above, the upstream server could include a Content-Language: en-CA response header to indicate that the page is in Canadian English. Now, the cache could return the same cached variant to both users, who both had en-CA as preferred language, even though their request headers weren’t exact matches.
More complex situations
User agents often send many headers that affect the returned response. Every header the server considered important must be set either as separate Vary headers or into a comma-separated list of headers. The first variant with multiple headers is allowed, but discouraged as not all implementations are all that clever in how they parse headers. The more headers a response varies on, the more cache variants must be maintained, and the less effective caching will be.
You can expect to find the following Vary response header on top websites with multi-lingual content: Vary: Accept-Encoding, Accept-Language, Cookie, User-Agent. The last two headers, the Cookie and User-Agent, is often omitted by mistake as developers often forget to set them. Forgetting to declare variations on the first request header, Cookie, can lead to information leakage; whereas the last can lead to usability and compatibility problems.
An incorrectly configured Vary: Accept-Encoding response header can lead to problems such as double-gzipping (or even gzip-deflate-gzipping) where compression is applied and then reapplied through multiple caching layers. This will, of course, lead to the returned page appearing scrambled and broken to the user, as the User Agent wouldn’t know what to do with the returned page.
A very common implementation error that I’ve come across in many places is to only include a request header in the Vary response header for requests that have that header set. Even if a request doesn’t contain the header you’re varying on, its omission is in itself a unique variant.
Headers that are used for cache revalidation (If-Modified-Since, If-None-Match) don’t need to be included in the Vary response header. However, these headers should still be included if their exact values are used; as is the case with syndication feed delta updates.
In conclusion, the Vary header is very useful when used correctly and can lead to all sorts of information leakage and compatibility/usability issues when omitted or used incorrectly.