evan_tech -- ruby sqlite3 = :(

12:18 pm, 20 Jan 06

ruby sqlite3 = :(

To better support this offline project, I rewrote LogJam's journal export in Ruby. (It's gotta be like 10% of the length of the original code, mostly due to blocks and the GC.)

Unfortunately, libsqlite3-ruby 1.1.0, the latest available, is buggy. A deterministic program dies with a variety of errors when ran multiple times in a row:

/usr/lib/ruby/1.8/sqlite3/driver/native/driver.rb:169:in `sqlite3_step': undefined method `call' for 21:Fixnum (NoMethodError)

/usr/lib/ruby/1.8/sqlite3/driver/native/driver.rb:169:in `sqlite3_step': undefined method `call' for true:TrueClass (NoMethodError)

/usr/lib/ruby/1.8/sqlite3/driver/native/driver.rb:169: [BUG] Segmentation fault
ruby 1.8.4 (2005-12-16) [i486-linux]

Looks like a memory bug. The probably either forgot to register something with the GC or just used something uninitialized. In any case, I need to figure out what approach will be the least painful.

(Update: using DBI's sqlite3 wrapper fixed it, for some reason or another...)

dan_erat

I haven't seen that (also using 1.1.0), but there's obviously something that I don't understand about sqlite's locking model. Processes that were just occasionally doing SELECT queries were leaving the database locked, preventing other processes from opening the DB. I finally just gave up and worked around it by closing the connection each time a query finished and then reopening it the next time I needed to do one. Laaame. :(

reply to this

snej

Locking is fairly simple -- BEGIN starts a transaction, END ends it. The lock is on the entire database file, not per-table as in MySQL. (It's using OS file locking.) You can find more details at sqlite.org.

The misbehavior you're describing doesn't make sense to me, and I've got a fair bit of sqlite experience. In particular, SELECT queries never cause the database to lock. I would suspect a bug in the Ruby wrapper code.

reply to this

dan_erat	I must've been smoking crack, or otherwise doing something wrong. I just wrote a little test program doing the same thing that I remember doing before, and it works just fine.
reply to this

snej	I've seen many a warning in the docs for installing Rails, about needing to make sure the native driver code for the sqlite3 module is installed, else you get a pure-Ruby emulation of sqlite that's buggy. But from the console logs it looks like the native driver is being invoked. Weird! I've been using sqlite3 in my Rails app, with no problems. But Rails probably uses DBI?
reply to this

evan	This'll be hard to track down, 'cause when I tried to reproduce it in a more simple setting I couldn't. It's likely that somehow my more complicated code tickles something the wrong way (maybe uses the API the wrong way), though I still blame the wrapper for an unuseful error message ("segfault!").
reply to this

xaosenkosmos

Random Debian gotcha

For whatever reason, you have to have libsqlite0's ruby bindings installed to get this to work under Debian etch. It only took a few hours of trying to debug things sanely before i gave up and decided to try the really obvious, stupid thing that belongs in the package management system's dependency tracking.

A symptom of this is:

0] jbm@brevity:/tmp$ ruby test.rb
/usr/lib/ruby/1.8/dbi/dbi.rb:499:in `load_driver':  is not a class/module (TypeError)
        from /usr/lib/ruby/1.8/dbi/dbi.rb:401:in `_get_full_driver'
        from /usr/lib/ruby/1.8/dbi/dbi.rb:381:in `connect'
        from test.rb:6

(This kind of thing is probably why i'm so negative all the time. Not only did the package management system fail on deps, but DBI fails to check that it found a, um, thingy for specified DBD module. It's actually "(nil) is not a class/module" there, if you poke and prod at things...)

reply to this

evan	Re: Random Debian gotcha Hmm... I don't see a libsqlite0 ruby binding... (Ubuntu breezy, though.)
reply to this

xaosenkosmos

Re: Random Debian gotcha

Err, i was partially wrong. It's not explicitly SQLite0, but there is a SQLite3 one. So i assumed it was 0, by elimination:

0] jbm@brevity:~$ dpkg -l '*sqlite*ruby*' | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                   Version        Description
+++-======================-==============-============================================
ii  libdbd-sqlite-ruby     0.0.23-3       Ruby/DBI driver for SQLite  <-- what you need
ii  libdbd-sqlite-ruby1.8  0.0.23-3       Ruby/DBI SQLite driver for Ruby 1.8 <--+
pn  libdbd-sqlite3-ruby             (no description available)
ii  libdbd-sqlite3-ruby1.8 0.0.23-3       Ruby/DBI SQLite driver for Ruby 1.8
pn  libsqlite3-ruby                 (no description available)
ii  libsqlite3-ruby1.8     1.1.0-1        SQLite3 interface for Ruby 1.8

In any case, the dependencies appear to be implicit but not defined in the dpkg. Either that, or the SQLite3 DBD binding is broken =\

reply to this