I wonder if using "begin exclusive" in db.newid would solve this.

Doing this should start a write transaction on the first select. It
should block if some other process has an exclusive lock. This will
block any read or write attempt until the commit() call after the
update. It should prevent the same value from being read twice.

     def newid(self, classname):
         """ Generate a new id for the given class
         """
+        print "|%r, %r, %r|"%(self.conn, self.conn.isolation_level,
self.cursor.connection.isolation_level)
+
+        # prevent other processes from reading while we increment
+        # otherwise multiple processes can end up with the same
+        # new id and hilarity results.
+        self.conn.isolation_level = 'EXCLUSIVE';
+
+        # force the creation of a new cursor and close out
+        # any existing transaction
+        self.sql_commit();
+
         # get the next ID
         sql = 'select num from ids where name=%s'%self.arg
         self.sql(sql, (classname, ))
         newid = int(self.cursor.fetchone()[0])

         # update the counter
         sql = 'update ids set num=%s where name=%s'%(self.arg, self.arg)
         vals = (int(newid)+1, classname)
         self.sql(sql, vals)
 
+        self.conn.isolation_level = ''; # go to standard begin transaction
+        # sql_commit creates a new cursor.
+        self.sql_commit();
+
         # return as string
         return str(newid)

The first self.sql_commit() may not be needed. The documentation
doesn't state that a change of isolation_level takes effect on the
next sql execute call.  So by committing I am deleting the old cursor
and starting a new cursor that has the requested isolation_level.

One test in the python code seems to indicate that changing the
isolation_level should be able to be done with existing cursors, so
the first self.sql_commit() should not be needed, but I can't be sure.

Matt are you able to try testing this patch? It sounds like you have
the environment to produce the error. Also if you can test without the
first self.sql_commit() and see if it works.

This does slow down some tests,
e.g. test/test_sqlite::testIDGeneration goes from 16 seconds to 22
seconds on my laptop. So it's doing something 8-).

-- rouilj