Stanford Course Notes

Still at it. I am loving this course.

Today we go to logistic regression, which is a fancy term and means that it is used to predict binary outcomes.

Binary outcomes are super-risky evaluations because while math doesn’t like discrete data, humans love it. Think about medical evaluations: you’re either ‘sick’ or ‘not sick’ in your own mind, but according to mathematized science, you have a particular combination of abnormal scores on a blood test, etc. These combine to produce a binary evaluation, “sick”, but that’s only because we need to cross a decision boundary to take action (begin treatment).

Logistic regression tackles this in a few ways. First, it lets you set where you think your decision boundary is going to be, when evaluated against a series of inputs (blood cell count, let’s say) and set an overall threshold for the evaluation. Let’s say that you assign a certain number of points to each input: 50 points per 100 red blood cells, -20 points if you work out every week, + 10 points for every cigarette you smoke. Then we say, if this person has more than 750 points, we declare them sick.

Now this point system isn’t perfect, there will be people we should have labeled sick with 300 points and people who are actually fine at 1000 points. Logistic regression gets around this by imposing a non-linear cost for being wrong. When fitting the curve (and figuring out that 750 level), the algorithm is penalized more heavily for misses at 1000 points than at 500.

Error in logistic regression is ALWAYS non-zero.

Not much to report or record at the moment, as all of the lectures and exercises this week have the goal only of teaching us the Octave programming language, which is an open source version of Matlab.

I’m constantly impressed by the power of expressing spreadsheets as matrices and vectors and thinking of analytical operations as expressed by linear algebra. Lots of new things to think about here, but gotta work through the programming exercises first.

Another machine learning class.

We wrapped up linear regression by discussing work with error functions. I hadn’t really focused on this before, even though I’ve built some crude ‘by-hand’ error-minimizing functions at work.

The error functions are the sum of the squared values of the error terms in a regression. The squaring part of this is important because it really penalizes big individual deviations.

For simple linear regression, this is pretty easy because there are only going to be two function terms (x and y). When you start packing in the variables (or exponents of variables), minimizing this error function gets tough.

We then take this error function and multiply it by 1/(2 * the sample size) for some reason. This some reason is to be discussed later.

That’s where the gradient descent algorithm comes in,  whose purpose is tofind a local minimum in the error function. It does this in a clever way: by just adding the value of the slope to each of the parameter terms. Eventually, you get smaller and smaller slopes and wind up at zero slope. Cool!

This is dampened (or amplified, I suppose) by another term to take bigger or smaller steps ‘down the slope’. This dampening term is called the ‘learning rate’. I find that terminology a bit confusing.

One key is to make ure you apply the slope to each term simultaneously, remembering that the slope in each ‘direction’ is going to be a bit different. If you visualize the whole process as a three-dimensional park with hills and valleys (I can’t find the picture he used in the notes) it’s easier to see that the slope in respect of each variable will be a bit different. So the equations for calculating that slope will be a bit different.

And there’s finally a discussion about resource use. We’re told that gradient descent is faster than the closed-form linear algebra solution at large database sizes.

That’s interesting and I wonder how sensitive such a conclusion is to language type. One thing I’ve learned is that speed of calculation is actually a multi-dimensional problem. Dataset size is one issue, of course, but so is language compilation/interpretation time. If you implemented the closed-form solution in C but did the gradient descent in some slower language like VBA or something, is the answer so clear? Maybe we could derive a cost function for compile times and figure it out!

Dicussion on XML:

There are two ways of organizing an XML document: using something called a DTD and something called a Schema. One is somewhat more strict than the other (Schema) and one is easier to deal with (DTD).

The point to me is that there is SOME way of automatically validating a document. The most common ‘dirty data’ problems I’ve come across are with databases that are not validated properly against some kind of template. In the world of validaion, I separate problems into human-type errors and computer-type errors.

Human-type errors are things like miskeyed addresses, duplicating entries and writing in a premium value for the policy limit, for example. A computer-type error would be something like repeating the column heading as a value in every single field.

The point is that Computer-type errors are really easy for humans to spot and Human-type errors are easy for computers to spot. That’s why automatic validation is so powerful: if you can combine a computer validation process with a human dataset, you’re probably going to get something that works. And vice versa.

From a user-experience standpoint, a good validation process can be maddening (I have to CAPITALIZE the first letter every single time?!) because most of it is perceived as fiddly formatting busywork. Luckily, increases in computing power have ushered in the dawn of the autofill era and this bug has become a feature (we’ll help you type!).

Anyway, I’m supposed to learn that a DTD is a bit messier because the ID aren’t typed and there is no control for sets of keys, blah blah blah, but these problems aren’t there when using a Schema/xsd. The XSD document can be daunting so my ‘homework’ is to  download schema for XML and play around with it.

How boring. I feel like I’m in “school” now, a feeling I loathe.

Speaking of which, relational algebra is a ridiculous topic. I don’t care if it’s the “underpinnings” of query languages. I hardly needed to learn C to build Python scripts, even though C is its “underpinnings”. If I had some deep problem with SQL I’d be happy to mess around with relational algebra to figure it out. But until then, keep me in the dark, please.