Author: David Wright

I’m building a monte carlo simulator right now. This means that what I’m really doing is:

1. Pick my statistical distributions
2. Figure out a way of generating random numbers from those distributions.

For my (reinsurance) purposes, I am trying to answer the following questions: how many claims are there and what do they cost?

So I’ve learned to just model the frequency of claims with a poisson distribution and the severity of claims with something else: Lognormal, Beta, InverseGauss (the last is our favorite because it’s so damn obscure).

It’s one thing to use something like @risk, which has all the distributions packaged up, but what if you don’t HAVE @risk and you’re too cheap to buy it?

Well, you roll up your sleeves and find something on the Internet is what.

A problem here is that math is mostly written in another language. Here’s the formula for evaluating the probability of k things happening if you’re expecting lambda things happening:

Oh, says Donald Knuth, well you can generate random numbers for that with this:

algorithm poisson random number (Knuth):
    init:
         Let L ← e

^−λ

, k ← 0 and p ← 1.
    do:
         k ← k + 1.
         Generate uniform random number u in [0,1] and let p ← p × u.
    while p > L.
    return k − 1.

Right. Well, neither of these languages is English.

That Knuth program is super interesting, though, because I feel like its constituent ideas are within my grasp. What it’s saying is that you multiply two random numbers between 0 and 1 together. If that number is higher than e^(-lambda), you multiply another. Then check again and continue until you have a number that is smaller than the e^(-lambda) threshold.

We all know the probability of two things happening together is their probability multiplied together. So each of these 0-1 random numbers has an expected value of 0.5. If you multiplied 30 of these together, because you’re expecting 30 as your lambda, you get 0.5^30.

But the distribution isn’t uniform. The higher the lambda, the bigger the left side of the distribution and so mean value sits at a progressively higher percentile. Put another way, the ratio of 0.5^lambda / e^-lambda increases.

I’m still stumped, though. I don’t understand e well enough to figure out what it’s doing to this whole thing and I want to go to bed.

Perhaps more tomorrow!

It’s a law of programming that early optimization is a very bad idea. Why fill your kitchen with gadgets when you don’t know how to cook?

On the other hand, what if I’m using an oxy-acetylene torch to cut onions? Shouldn’t I go to the market to see if knives are for sale?

So I’ve looked into replacing sqlite with MySQL. Maybe it’s faster, maybe it’s better; all the web programming books teach PHP and MSQL forchrissakes, and I’m using neither. So today I tried downloading MySQL and learned a couple things:

1. Wow, was that installation hard work. This means ANOTHER interplanetary programming detour and I’m sick of detours. I want to get to work.

2. MySQL’s python for windows implementation REALLY sucks.

Discouraged, I thought I’d read around a bit. At stackoverflow I found a great post that lead to an even better podcast with the inventor of Sqlite.

The upshot appears to be this: if you don’t need lots of simultaneous WRITING to your database, sqlite is probably better. Another heuristic: if your website fits on one computer (!?), sqlite is probably better. Yet another: if you get fewer than 100,000 hits a day (!!!) sqlite is the winner.

The big downside to sqlite is that it only allows one user to write data into it at one time: the database “locks”. The big upside? Just about everything else, I think.

Anyway, that little bit of research makes this whole question absolutely ridiculous for my purposes, so I’m sticking with sqlite.

The weekend project has me downloading a bunch of weather forecast data and working on some processes for analyzing it. I’ve been trying to not EVER delete any raw data because I never know I’ll need to redo something I screwed up.

This habit has saved my can a few times.

Somewhat recently, I learned SQLITE so I could stop storing everything on the hard drive in flat files, which is probably ridiculously inneficient for the computer and is definitely a pain in the ass to monitor.

But here’s the thing with SQL, it’s a tricky and brutally unforgiving language. You can ACTUALLY delete things by mistake and never see them again. And last week that’s what happened to ALL of my backup data. I still don’t know what on earth I did.

Luckily it was only the backup data (still had the main database) and even more luckily I backed up the backup a week earlier when I left town on a trip.

In the end, I lost a few days worth of backup info forever. The upshot is that I quickly moved backup management to the top of the to-do list. The thought of losing all this work gives me the willies. I’d probably have a nervous breakdown.

So, here’s my ‘backup plan’:

I recently bought this nifty router that can host an external hard drive on an ftp server. I just plug it in and presto, it’s accessible online from anywhere. Perfect.

But netowrk programming is finicky. Connections interrupt for reasons I cannot fathom, which means that the old drag and drop method fails when the interweb gods frown upon me.

So I spent all day today figuring out a way to cycle through my directory and compare all the files to the existing backups on the ftp site and keep the new ones. If the process gets interrupted it just waits until I get it back. No restarts.

Cycling through the installation-related junk slows me down, but brtue-force backupping is good for peace of mind. I know it’s all going to be there.

This is the latest in a long series of programming detours I’ve had to make. It’s amazing how little time I spend doing the real analysis that is the ENTIRE POINT of this project. If I knew how hard this whole thing was going to be, I’d have been a bit less cavalier in committing myself to it.

Two months and I’m not even remotely close to a working prototype.

My target is the last weekend in July, now. We shall see.

I’m a gigantic fan of several blogs.

And these blogs have big archives that, realistically, I’ll never go through even if I know (and I do know) they have lots of cool posts that I would find interesting.

So here’s my idea: why not build a little tool that lets you specify a few blogs and, at the push of a button, summons an archived blog post from one at random? I figure there are three possible outcomes:

1. You find a post from the archives you never read.

2. You find a post form the archives you read and remember.

3. You find a post form the archives you read but don’t remember.

Now, I know I like these blogs so I’m likely to enjoy the posts right away. The cool part, though, is that I’m also likely to feel little remorse at refreshing away a post I’m not too keen on and getting another.

I like this idea.

From an implementation standpoint, I don’t much feel building and maintaining an archive of all the blogs I like. So my solution is to do this by date. Here’s the method:

Pick a date to start: let’s say jan 1, 2000.

Pick a random day between today and that date.

Test whether there is a post from that blog in that day. If not, pick another random day. If there is more than one, index the posts from that day and pick one of THEM at random.

This will bias towards older dates, sure, but that’s the idea.

I bet that I would enjoy flicking through a bunch of old posts even more than I would enjoy reading the ‘cutting edge’ posts from my favorite bloggers. Current events don’t tend to interest me much, it’s the thought processes and insights that I love.

So I’m going to build this.