03:41 pm, 20 May 06
screen scraping with ruby
For various reasons I seem to often need to screen-scrape websites. I've tried Rubyful Soup but it seems to take tens of seconds to parse relatively small files. I just now tried htmltools as well and it's also incredibly slow -- 9.7 seconds to parse a 400kb file on a P4 3.8ghz machine!
The file is really straightforward, too: it's just one huge table. How hard is this to parse quickly? Firefox seems to have no trouble with it...
So my question for you: what should I be using instead? (I'm happy to switch languages --
gaal/brad, which Perl HTML parsing library is the most pleasant? Preferably something that doesn't use SAX or DOM APIs...)
The file is really straightforward, too: it's just one huge table. How hard is this to parse quickly? Firefox seems to have no trouble with it...
So my question for you: what should I be using instead? (I'm happy to switch languages --
gaal/brad, which Perl HTML parsing library is the most pleasant? Preferably something that doesn't use SAX or DOM APIs...)
But Text::CSV apparently just eats my UTF-8 without complaining. Thanks. I guess I can do it manually.
Every time I try use Perl I get bitten by one of its misfeatures. This time it was using "==" on a string. Heh.
I ended up using the above Perl modules, where it's nearly instantaneous.
donno about how much scraping you can get from it
I have several utilities that “screen-scrape” websites based on the CPAN HTML::TokeParser module. Based on tokens (start tag, text, end tag) rather than DOM. I've written table-type parsers before just by making it tab after every closing TD, newline after every closing TR and strip any other tags or text that isn't inside a table at all, giving a nice tab-delimited output file that can be processed by a separate tool.
You may want to take a look at HTML::TokeParser::Simple – makes for much nicer coder than HTML::TokeParser by itself.
(But do give XML::LibXML in HTML parsing mode a spin.)
Try XML::LibXML. It’s got an HTML parsing mode, it’s got XPath, it’s a speed demon, it’s parsimonious with memory. libxml2 rocks.
$parser->parse_html_*, though I now realize I'd been using this module for machine-generated XML, not possibly sucky random HTML. Well, at least if it fails, it'll fail fast :-/It doesn’t handle everything, but you can throw some pretty badly formed markup at it and it will do just fine. It fails with very screwy input that other parsers still manage to make sense of (at least, HTML Tidy does), but that hasn’t been an issue for me yet.
I hate to say it but almost always resort to regexps for screen scraping in ruby. I know it represents a maintenance problem but I rarely come across valid xml/html (I scrape a lot of government regulation websites) and it's just not worth the hassle of running it through Tidy just to manipulate it with rexml or libxml.
Regardless, I feel your pain here.
TagSoup
TagSoup is Java is fast (but SAX based)Nick
I haven't managed to really figure out XQuery, but if you can learn Haskell I'm sure it's easy by comparison.