evan_tech

Previous Entry Share Next Entry
06:26 pm, 15 Sep 03

aggregation sucks

One of the many things I’m interested in is weblog aggregation. LJ is, of course, my favorite aggregator, even though when compared to a real RSS aggregator it’s terrible, because it perfectly aggregates pretty much everyone I want to read. But scalability (and relatedly, reliability) problems keep me from thinking it’s the end-all solution.

The Atom effort is well-intended, but to me they don’t really address any of the interesting questions. Sure, RSS is a mess and a well-defined spec ought to replace it, but taking a step back it’s clear to me that polling static feeds will simply not scale. (The Atom API question is independent, but from what I could tell it seemed to be mostly SOAP versus REST versus your favorite TLA wankery, again missing the forest for the trees.)

On top of that, trying to follow Atom (and in general, that whole contingent of the web) makes me unbelievably angry in a way I can’t quite put my finger on. It’s something to do with the self-congratulatory smugness/name-dropping/egos, and a little to do with the blindness to the more difficult technical issues that are so familiar to me from looking at something the size of LJ, that prevent any real progress from being made. (Or maybe it’s bitterness because I’m not involved in it anymore. :P)

With that said, I’m pretty sure I don’t have any good solutions handy, either. I think if even if I did magically come up with a good solution and tried to present it, it wouldn’t be adopted anyway (see also: NIH syndrome) and that would be frustrating. So maybe I should stop worrying about the problem at all and hope that other people solve it. :)

Syndication is a big mess, anyway, and I fear that ultimately even a well-designed format would still have people providing partial content, forcing us to fire up a web browser anyway. The other path to go is the jwz route: everything sucks, so I might as well screen-scrape. (For context, note that jwz’s LJ client just posts to the web form, rather than use the LJ protocol.) It would be interesting to make tools to automate screen-scraping that interacted well with an aggregator...

A step back from all of this, though: getting away from worrying about any of this and from reading LJ would probably do me some good as well.