Evan Martin (evan) wrote in evan_tech,
Evan Martin

deflate compression in googlebot

Random bit of Google trivia: Googlebot recently added support for "deflate" compression, despite no real demand for it. Why? Go ahead and guess; the answer follows.

This allowed changing the Accept-Encoding header to gzip,deflate, which then makes it better match the Accept-Encoding sent by web browsers. This then allows proxies like Squid to share cache entries between requests from browsers and requests from Googlebot. Here's an interesting thread on the way the Accept-Encoding and Vary headers interplay to ruin things for proxies. (Here's a page I skimmed that goes into a bunch more detail.) You could argue the blame lies with Squid, which apparently(?) treats the Accept-Encoding header value as an opaque string rather than a list of encodings. On the other hand, doing something smarter depends on Squid magically knowing what circumstances caused a server to choose which Accept-Encoding combination.

[N.B.: I'm not an expert on any of this stuff, so feel free to correct me. Thankfully, I was also not involved in the implementation details, so I can excuse myself from blame!]

This was all prompted by an observation by one of the wikipedia folks, who emailed me about it. However, I asked him if this Google-side change helped and he never replied. :~(
Tags: google

  • blog moved

    As described elsewhere, I've quit LiveJournal. If you're interested in my continuing posts, you should look at one of these (each contains feed…

  • dremel

    They published a paper on Dremel, my favorite previously-unpublished tool from the Google toolchest. Greg Linden discusses it: "[...] it is capable…

  • treemaps

    I finally wrote up my recent adventures in treemapping, complete with nifty clickable visualizations.

  • Post a new comment


    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.