I don’t think better planning would help alleviate this much, either. While we do our best, it’s impossible to anticipate every way things will need to work in the future.
And now to the meat of this post: LJ’s userpics were slow, and, as usual, Brad had another scheme to make them fast.
LJ’s userpic serving is complicated. The pictures are all stored in LJ’s databases (so they’re available when uploading new pics, for example), but they’re mostly served from machines somewhere on the East coast because the bandwidth is cheaper. Additionally, the URLs are of the form /picid/userid, with no extension, and we continue to support them (see above about how this is difficult). These together mean there are three major complications:
- The userpic servers don’t necessarily always have every picture, as new ones are being uploaded to the main site all the time.
- Those picture URLs can’t be mapped straight through to the file system because there are millions of user pictures and they can’t all be stored in a single directory.
- To properly send the HTTP Content-type header, we have to read the first few bytes of the file to determine the file’s type.
Before, we were using Apache with mod_perl to do the hard work, but that was too heavyweight. I rewrote that about a week ago to use mod_rewrite with an external rewriting engine that did the tricky work: when a file wasn’t found, a temporary HTTP redirect was returned that forwarded the user’s browser to the relevant picture on the main LJ site, and a subprocess was forked off to retrieve the picture for the userpic servers. Then Apache’s mod_mime_magic was used to determine the file’s type.
It was still too slow. Today, I downloaded thttpd and made some custom modifications:
- URLs in the proper userpic form are rewritten internally to the proper filesystem paths, which is hashed into subdirectories.
- Because userpics never change, any request that includes an If-Modified-Since: header immediately gets at 304 Not Modified response.
- When a file is mapped, we switch() on the first byte to tell the file’s type; LJ only allows three image formats and they each begin with a different byte.
The result? Before, the Apaches were putting out 3mbit/sec. Now, the thttpds (yes, there is more than one machine pushing this data) are each putting out 20mbit/sec. That’s a pretty significant increase! And now userpics work again, and the machines aren’t too loaded. Hooray!
(And isn’t that an incredible amount of bandwidth? Each machine is pushing literally megabytes of data out each second.)
I’ll soon be able to post about my other project, which has been much more complicated and interesting, too. Work was interesting today. :)