Evan Martin (evan) wrote in evan_tech,
Evan Martin

Wikipedia: I broke down and wrote it in C++. Night and day performance difference. Sigh. (See also: graydon's comment on my last post.)

It turns out at the C level there are two interfaces for fetching MySQL results: one pulls the entire response into the client, and then satisfies repeated requests for rows pull from there, while the other pulls a row at a time. The first requires more memory, but the second locks up the database if the application is slow. The O'Caml (and I imagine other) bindings use the first. No wonder my SELECT cur_title FROM cur was slow.

I also realized pushing the links back into the same database was slow, so now I have a packed binary on-disk representation: 45megabytes, of the form { src: int, count: short, [ { dst: int, weight: short } ] }.

I think my first project will be shortest paths, but that will probably have to wait for another weekend. (I notice that the A* algorithm page links to a thorough and generally great page by Amit, a coworker. (And, now that I poke around, I see his PhD work was on an OO extension for ML!))

  • blog moved

    As described elsewhere, I've quit LiveJournal. If you're interested in my continuing posts, you should look at one of these (each contains feed…

  • dremel

    They published a paper on Dremel, my favorite previously-unpublished tool from the Google toolchest. Greg Linden discusses it: "[...] it is capable…

  • treemaps

    I finally wrote up my recent adventures in treemapping, complete with nifty clickable visualizations.

  • Post a new comment


    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.