Tidbits @ Kassemi

Hey everyone! I'm finally finished with the creation of my first real project, matchstd.com, a matching service designed for those with any type of sexually transmitted disease, which is completely free, and as anonymous as those things can get...

There were a lot of challenges, but I think everything is pretty well worked out. It's done entirely in python, and hosted by the wonderful people over at http://www.python-hosting.com... I had a few problems getting a few features to work in their environment, but nothing too difficult. It's up and running now, and, I should add, in BETA, as I will be making bug fixes as they come up...

Take it easy everyone, and take a look!

-James

I got a hold of a load of locational information from a few government sources, and, being government sources, the information is scattered everywhere. I currently have a database with location information for the US, including zip codes and, most importantly, latitude and longitude. There are tons of areas in the world that the programs I was writing wouldn't reach, which is unfortunate for business. So I decided I'd find international information I could use. With the amount of trouble I had obtaining a US zip code database, I knew I wasn't going to be able to find information quite as specific, so I set my goal on latitude and longitude for cities around the world. I came across a ton of geographical data at that link, and downloaded their file.

I updated my database schema to accept new values, this time based more on the city, province, country values than the zips themselves. My python script was supposed to parse all this data (big file), adding it to the new schema, and then pull the data from my old schema and append it, as well. I ran the program, went to watch some TV, and found a hefty little memory leak. My computer was losing ram, and losing it quickly. After about four hours staring at my code I figured it wasn't my fault, and took a look more deeply at what sqlobject was doing. It turns out that it holds a cache of the connection used to create each instance (I'm insantiating my sqlobject model to enter a record). This is a HUGE problem when you're entering as much data as I am, so I disabled it. Simply override the default _SO_finishCreate() method of the SQLObject class in your own model:



Now you'll notice your ram not exceeding what you start off with during those inserts. Hope I've described this well enough for another person to find when they need it. Wish I could have found it earlier :)

-James

Hello everybody. Like I was saying the other day, I got my first big site on the internet: matchstd.com, a location where individuals who have tested positive for any number of different stds (or just one), can find other individuals with the same stds... It's a good site, and it looks great (I'm no designer, but I'll give myself a pat on the back).

I learned quite a few things putting this system up. My development computer has apache 2, cherrypy 2.2beta and a plethora of other installed packages that make my life easier. The transition, however, to the main server (python-hosting.com), made my life a little more difficult. Their service is great, but I had never moved a site that utilized so many features (I've stuck up PHP sites on various services, as well as some simple CGI and static pages), and was so complex. I'm just happy a solution out there existed for python hosting that already included most of what I needed (python 2.4, sqlobject, kid, etc.) Anyway, a few pointers:

• Never program for a specific OS (I knew this before, but it's a good tip anyway). Always make sure that your application uses all the cross-platform tools available, instead of depending on hard-coded paths and such


• Organize your pages in a neat fashion.
  By placing pages in a huge mess throughout a large folder, you're going to eventually lose track of exactly where they are (or you'll depend on instinct for the path names)... Doing that will force you to re-organize on the server... Not good. I'll lay out a good organization model below...


• Keep track of your changes. Use a repository such as subversion or CVS. If you haven't used a repository before, go with subversion off the bat.


• Make sure you've either programmed a nice way to keep log information organized, or use a program from the internet. I'd recommend webalizer, a completely free system that you cron, which produces lovely graphs and statistical printouts. It's extremely useful.

I'll put more up later... Or maybe not, but I hope some of those work out for you. Now, to the organization I mentioned. I've found the following setup makes things very easy to create both a web site, and programs that utilize the site's hidden engine:

• AppRoot




• html
  
  Use a templating system, and put all templates here. Your pages directory will include the code to display these.


• libraries
  
  For every function that your web application needs to perform, it's a good idea to implement it here, in such a way that doesn't require the server. This lets you write simple non-web applications that can interact with your site. Although you won't need this at first, it's always good for later.


• models
  
  Database definitions. Place all code to create your tables here.


• pages
  
  Place all actual page logic display here... Basically, the code
  that constructs each page.


• server.py
  
  When I'm using CP, I like to put my server startup code here...
  
• server.conf
  
  Also with CP, it's nice to have the configuration here.

Okay. If anybody wants some input, feel free to comment!

-James