Tidbits @ Kassemi

A collection of opinions, thoughts, tricks and misc. information.

Saturday, February 04, 2006


More sqlobject fun!

I got a hold of a load of locational information from a few government sources, and, being government sources, the information is scattered everywhere. I currently have a database with location information for the US, including zip codes and, most importantly, latitude and longitude. There are tons of areas in the world that the programs I was writing wouldn't reach, which is unfortunate for business. So I decided I'd find international information I could use. With the amount of trouble I had obtaining a US zip code database, I knew I wasn't going to be able to find information quite as specific, so I set my goal on latitude and longitude for cities around the world. I came across a ton of geographical data at that link, and downloaded their file.

I updated my database schema to accept new values, this time based more on the city, province, country values than the zips themselves. My python script was supposed to parse all this data (big file), adding it to the new schema, and then pull the data from my old schema and append it, as well. I ran the program, went to watch some TV, and found a hefty little memory leak. My computer was losing ram, and losing it quickly. After about four hours staring at my code I figured it wasn't my fault, and took a look more deeply at what sqlobject was doing. It turns out that it holds a cache of the connection used to create each instance (I'm insantiating my sqlobject model to enter a record). This is a HUGE problem when you're entering as much data as I am, so I disabled it. Simply override the default _SO_finishCreate() method of the SQLObject class in your own model:

class YourModel(SQLObject):
blah = StringCol()
etc = FloatCol()


def _SO_finishCreate(self, id=None):
setters = self._SO_createValues.items()
names = [self.sqlmeta.columns[v[0]].dbName for v in setters]
values = [v[1] for v in setters]
if not self.sqlmeta.lazyUpdate:
del self._SO_createValues
self._SO_createValues = {}
del self.sqlmeta._creating
id = self._connection.queryInsertID(self,
id, names, values)

Now you'll notice your ram not exceeding what you start off with during those inserts. Hope I've described this well enough for another person to find when they need it. Wish I could have found it earlier :)


Comments: Post a Comment

<< Home


August 2005   September 2005   October 2005   November 2005   December 2005   January 2006   February 2006   March 2006   April 2006   June 2006   July 2006   August 2006   September 2006   October 2006   November 2006  

This page is powered by Blogger. Isn't yours?