programming

It’s a law of programming that early optimization is a very bad idea. Why fill your kitchen with gadgets when you don’t know how to cook?

On the other hand, what if I’m using an oxy-acetylene torch to cut onions? Shouldn’t I go to the market to see if knives are for sale?

So I’ve looked into replacing sqlite with MySQL. Maybe it’s faster, maybe it’s better; all the web programming books teach PHP and MSQL forchrissakes, and I’m using neither. So today I tried downloading MySQL and learned a couple things:

1. Wow, was that installation hard work. This means ANOTHER interplanetary programming detour and I’m sick of detours. I want to get to work.

2. MySQL’s python for windows implementation REALLY sucks.

Discouraged, I thought I’d read around a bit. At stackoverflow I found a great post that lead to an even better podcast with the inventor of Sqlite.

The upshot appears to be this: if you don’t need lots of simultaneous WRITING to your database, sqlite is probably better. Another heuristic: if your website fits on one computer (!?), sqlite is probably better. Yet another: if you get fewer than 100,000 hits a day (!!!) sqlite is the winner.

The big downside to sqlite is that it only allows one user to write data into it at one time: the database “locks”. The big upside? Just about everything else, I think.

Anyway, that little bit of research makes this whole question absolutely ridiculous for my purposes, so I’m sticking with sqlite.

The weekend project has me downloading a bunch of weather forecast data and working on some processes for analyzing it. I’ve been trying to not EVER delete any raw data because I never know I’ll need to redo something I screwed up.

This habit has saved my can a few times.

Somewhat recently, I learned SQLITE so I could stop storing everything on the hard drive in flat files, which is probably ridiculously inneficient for the computer and is definitely a pain in the ass to monitor.

But here’s the thing with SQL, it’s a tricky and brutally unforgiving language. You can ACTUALLY delete things by mistake and never see them again. And last week that’s what happened to ALL of my backup data. I still don’t know what on earth I did.

Luckily it was only the backup data (still had the main database) and even more luckily I backed up the backup a week earlier when I left town on a trip.

In the end, I lost a few days worth of backup info forever. The upshot is that I quickly moved backup management to the top of the to-do list. The thought of losing all this work gives me the willies. I’d probably have a nervous breakdown.

So, here’s my ‘backup plan’:

I recently bought this nifty router that can host an external hard drive on an ftp server. I just plug it in and presto, it’s accessible online from anywhere. Perfect.

But netowrk programming is finicky. Connections interrupt for reasons I cannot fathom, which means that the old drag and drop method fails when the interweb gods frown upon me.

So I spent all day today figuring out a way to cycle through my directory and compare all the files to the existing backups on the ftp site and keep the new ones. If the process gets interrupted it just waits until I get it back. No restarts.

Cycling through the installation-related junk slows me down, but brtue-force backupping is good for peace of mind. I know it’s all going to be there.

This is the latest in a long series of programming detours I’ve had to make. It’s amazing how little time I spend doing the real analysis that is the ENTIRE POINT of this project. If I knew how hard this whole thing was going to be, I’d have been a bit less cavalier in committing myself to it.

Two months and I’m not even remotely close to a working prototype.

My target is the last weekend in July, now. We shall see.

I’m a gigantic fan of several blogs.

And these blogs have big archives that, realistically, I’ll never go through even if I know (and I do know) they have lots of cool posts that I would find interesting.

So here’s my idea: why not build a little tool that lets you specify a few blogs and, at the push of a button, summons an archived blog post from one at random? I figure there are three possible outcomes:

1. You find a post from the archives you never read.

2. You find a post form the archives you read and remember.

3. You find a post form the archives you read but don’t remember.

Now, I know I like these blogs so I’m likely to enjoy the posts right away. The cool part, though, is that I’m also likely to feel little remorse at refreshing away a post I’m not too keen on and getting another.

I like this idea.

From an implementation standpoint, I don’t much feel building and maintaining an archive of all the blogs I like. So my solution is to do this by date. Here’s the method:

Pick a date to start: let’s say jan 1, 2000.

Pick a random day between today and that date.

Test whether there is a post from that blog in that day. If not, pick another random day. If there is more than one, index the posts from that day and pick one of THEM at random.

This will bias towards older dates, sure, but that’s the idea.

I bet that I would enjoy flicking through a bunch of old posts even more than I would enjoy reading the ‘cutting edge’ posts from my favorite bloggers. Current events don’t tend to interest me much, it’s the thought processes and insights that I love.

So I’m going to build this.