My Sandy Timeline

Mid-April: I move to Hoboken, NJ with my 6-month pregnant wife and Bree and Max, my two 10-pound dogs.

Some time in May: a relatively minor storm floods our parking garage and the nearby street. WTF? Lesson: Hoboken is really bad for flooding and we live in one of the worst parts.

End of July: I sign up for an actuarial exam for the end of October AND my son is born.

October 20: “A strong ridge of high pressure parked itself over Greenland beginning on October 20, creating a “blocking ridge” that prevented the normal west-to-east flow of winds over Eastern North America. Think of the blocking ridge like a big truck parked over Greenland. Storms approaching from the west or from the south were blocked from heading to the northeast.”

Some additional background from the same link:

We expect hurricanes to move from east to west in the tropics, where the prevailing trade winds blow that direction. But the prevailing wind direction reverses at mid-latitudes, flowing predominately west-to-east, due to the spin of the Earth. Hurricanes that penetrate to about Florida’s latitude usually get caught up in these westerly winds, and are whisked northeastwards, out to sea.

Bottom line: normal no longer applies.

October 22th: Tropical Depression 18 forms.

October 24th: Now Hurricane Sandy, the storm hits Cuba hard. The possibility of a US landfall dawns on the NHC for the first time.

October 27th (Saturday): I get a mass email from my building manager saying that the area flooded even during the over-hyped Irene last year and big floods trigger the fire alarm. I reply: as in the building-wide fire alarm? Yep, he fires back, and we can’t turn it off and it’s LOUD.

Well that sews this one up, but where do we go? Here’s our criteria:

  1. Town that has a hotel that wasn’t full
  2. Oh, yeah, and isn’t on a river
  3. That hotel needs to take dogs
  4. Is near a place where I can take my exam (still studying through all this!).

I pull up the intertubes and hit the phones. The answer? Three-hour away Albany.

October 28th (Sunday): no point studying, got to pack up an infant and dogs and supplies and whatnot and hit the road. That takes most of a day. The hotel is great and guest cancellations are rolling in. Papa John’s pizza and a practice exam for me.

October 29th (Monday): Holy Cow this is for real. Glued to CNN. Albany? A brisk wind is the worst we saw. Incredible luck.

October 30th (Tuesday): Hoboken is underwater. Everything is underwater. Uh, oh, when are we going to be able to get back?

October 31st (Wednesday): I write my exam in the morning. I’m the only one sitting it since the CAS let affected folks put it off. Not for me, let’s do this. That’s four hours.

Back at the hotel it’s becoming clear, as I scarf down yet more takeout, that we’re not going home. Looks like it’s back to Canada to my in-laws’. But first someone’s got to go back to Hoboken to get our travel documents. Saddle up!

Driving down the roads I see that about one in ten gas stations in Northern New Jersey is open and each has a gigantic queue of cars at it.

You know what that means: rationing by time instead of price. Far more importantly, however, it means that overall supply is lower. Here’s Yglesias who has been covering this very well:

Chris Christie, also put out a weekend press release warning that “price gouging during a state of emergency is illegal” and that complaints would be investigated by the attorney general. Specifically, Garden State merchants are barred from raising prices more than 10 percent over their normal level during emergency conditions (New York’s anti-gouging law sets a less precise definition, barring “unconscionably extreme” increases).

The bipartisan indignation is heartening, but there’s one problem. These laws are hideously misguided. Stopping price hikes during disasters may sound like a way to help people, but all it does is exacerbate shortages and complicate preparedness

And more:

But when it comes to things like gasoline and bottled water, neither the short-term nor the long-term supplies are genuinely fixed. Transportation routes into the area have been severely disrupted and many gas stations’ supplies are hard to access due to power outages, but it’s not impossible to transport this fuel from where it is into people’s cars and generators. It’s just much more annoying and difficult than usual. But the possibility of windfall profits is exactly the lure we need to get people to start making extraordinary efforts to get more fuel to the people who need it. There are things people will do to sell gasoline for $10 a gallon that they won’t do to sell gasoline for $3.40 a gallon (note that in Norway this is what gas costs all the time) and that’s what we need.

Power lines were down everywhere and electrical crews were working away. Roads were closed, though, and it took forever to get back to Hoboken. Eventually I had to park about a mile away and, now under the cover of darkness, run into town with my rubber boots, flashlight and backpack.

Very post-apocalyptic.

The phone networks were overloaded so there were definitely people around. You could see refugees sitting in cars charging their devices before going back up to, I dunno, play angry birds by candlelight, I suppose. The gold standard of disaster certifications is of course an on-location broadcast by Anderson Cooper, which happened while I was there! I didn’t see AC360 himself, though.

Anyway, got my stuff and booted it back to the car. Back to Albany by 11pm. Phew. what a day.

November 1st: quick check of the newswires. Still flooded. Ok, back in the car for 8 hours to Canada!

Post Scripts:

The insurance loss is getting picked at 10-20bn, which should put this after Katrina and Andrew as the third most costly hurricane in US history. That’s probably about right. There’s also a debate about whether hurricane deductibles (higher than normal storm deductibles) are going to apply to this “Post-Tropical Cyclone”. See here for example.

There is also a debate about the role of FEMA in these kinds of disasters. Here is an interesting point (via MR):

We’ve nationalized so many of the events over the last few decades that the federal government is involved in virtually every disaster that happens. And that’s not the way it’s supposed to be. It stresses FEMA unnecessarily. And it allows states to shift costs from themselves to other states, while defunding their own emergency management because Uncle Sam is going to pay. That’s not good for anyone.

When FEMA’s operational tempo is 100-plus disasters a year, it’s always having to do stuff. There’s not enough time to truly prepare for a catastrophic event. Time is a finite quantity. And when you’re spending time and money on 100-plus declarations, or over 200 last year, that taxes the system. It takes away time you could be spending getting ready for the big stuff.

…I think another issue is some people see the failed response to Hurricane Andrew as the reason George H.W. Bush lost Florida to Clinton. So now, you have presidents who are very concerned about the potential impact, from an election standpoint, of disasters. That created an incentive to nationalize things.

Finally, here’s a statistical wrap-up (great image at the link):

Death toll: 160 (88 in the U.S., 54 in Haiti, 11 in Cuba)

Damage estimates: $10 – $55 billion

Power outages: 8.5 million U.S. customers, 2nd most for a natural disaster behind the 1993 blizzard (10 million)

Maximum U.S. sustained winds: 69 mph at Westerly, RI

Peak U.S. wind gusts: 90 mph at Islip, NY and Tompkinsville, NJ

Maximum U.S. storm surge: 9.45′, Bergen Point, NJ 9:24 pm EDT October 29, 2012

Maximum U.S. Storm Tide: 14.60′, Bergen Point, NJ, 9:24 pm EDT October 29, 2012

Maximum wave height: 33.1′ at the buoy east of Cape Hatteras, NC (2nd highest: 32.5′ at the Entrance to New York Harbor)

Maximum U.S. rainfall: 12.55″, Easton, MD

Maximum snowfall: 36″, Richwood, WV

Minimum pressure: 945.5 mb, Atlantic City, NJ at 7:24 pm EST, October 29, 2012. This is the lowest pressure measured in the U.S., at any location north of Cape Hatteras, NC (previous record: 946 mb in the 1938 hurricane on Long Island, NY)

Destructive potential of storm surge: 5.8 on a scale of 0 to 6, highest of any hurricane observed since 1969. Previous record: 5.6 on a scale of 0 to 6, set during Hurricane Isabel of 2003.

Diameter of tropical storm-force winds at landfall: 945 miles

Diameter of ocean with 12′ seas at landfall: 1500 miles

New Year’s Resolution

It’s a bit late for this, I know, but I’ve been flicking through the back catalogue of blog posts and I’ve realized something: I’m way too negative on this blog.

I don’t like to think of myself as a negative person. Negative people are boring and annoying and bring you down. It’s easy to be negative, it’s easy to say no. I’m happiest when I’m enthusiastic about stuff and drinking all the kool-aid I can get my hands on. The problem with blogging is that you get caught up in this silly people are wrong on the Internet thing:

I don’t like to be like this and too much of my stuff is negative. I’ve noticed as well that my positive posts get more hits and attention from people. So here’s my New Year’s resolution: be positive in print!

You Are What You Do All Day, Until You Are

Malcom Gladwell has a knack for catching the public consciousness, doesn’t he?

Ever since that Outliers book (I’ve never read a Gladwell book), much of the really interesting writing I’ve come across probably owes its inspiration to him. Every personal experiment runs into its Gladwell moment:

Let me get to the big point here: You are what you do all day.

What does all this research about your body adapting to circumstances tell us?You are what you do all day.

What does all this research about brain plasticity and rewiring tell us? You are what you do all day.

That probably scares the shit out of a lot of people. And it should.

What did you do today? Is that what you want to be a genius at in 13 years? Is that what you want to become?

Nietzsche spoke of the eternal recurrence. Basically, you should live in such a way so that if I said to you, “You’re going to have to repeat this life over and over again for infinity” that your response would be “Sounds great.”

Do you feel that way? If not, are you going to change what you do tomorrow?

Your brain will change to adapt. Your body will change to adapt.

But will you?

And I’m sitting at one right now, actually. These Stanford courses have whetted my appetite for serious learning and I’m completely uninterested in actually shelling out real dough fo the privilege of packing my brain.

So I’m going to do the reverse of making my hobby my day job. I’m going to, once again, make my day job my hobby. I’m going to become an actuary.

By a Gladwellian definition, I’m already an actuary, or getting close, anyway. It’s what I do all day and, though I don’t quite have 10,000 hours under my belt, I’ve probably got about 3-4,000 hours of pure actuarial experience in the bag. My job has been migrating this way over the last five years or so and when I moved down here to New York it’s pretty much become full time.

And I love it. I really do.

My TIPPING POINT came when I downloaded an actuarial text for pure extra reading and reference for work. I thought to myself: how many people are stupid enough to read something like this and choose to spend zero extra effort to join one of the highest-status clubs on earth with infinite job security and economic prospects?

I had a flick through the syllabi of the 5 exams I’m going to need (I did one years ago between CFA exams) and I was surprised at how much I already know. See paragraph above. Facepalm.

So the first one is in January and I’m going to pack as many as I can in next year, which is probably going to be three. If it’s going super-duper well, I might chance a fourth. We shall see.

Machine Learning Course Notes – Bittersweet

Finished this week’s exercises in a 5-hour marathon starting at 4:30am this morning. Today’s meta-lesson: implementation is way harder than reading slides and ‘kinda getting it’. My god is it hard to actually write a program that uses even what appear to be simple concepts.

So there are three tracks for this course: first is the spectator track (my term), where you just do the basic assignments (enough to be dangerous and spew plausible-sounding BS).

There’s the ‘advanced’ track, which I’ve chosen, which asks you to do some actual programming assignments (this morning’s marathon). Within the advanced track there are ‘extra credit’ assignments, which ask you to implement even more of the course material in Octave (a programming language). I haven’t gotten to the extra credit stuff. More on this later.

The final track is the ‘real’ track, where you pay real money, show up to class and all the rest. I read a discussion thread on the course website that speculates that my ‘advanced’ track covers about 40%-50% of the real course material. The real course is about 1.5x as long (3 months instead of 2), so let’s say we’re about 60%-75% of the pace of a real university course.

I’m starting to think it was a mistake to take two of these courses. I just don’t have enough time to learn everything I want to learn. I want to do the extra credit stuff, because what’s the point of reading the slides on stuff if you don’t REALLY get it? And my first crack at the extra credit stuff shows that I don’t REALLY get it.

And there are all these dudes (yes, all dudes) carpet-bombing the discussion boards who obviously REALLY get this stuff, while I only kinda get it. How many times in University did I wish I were smarter?  That I wish I had really learned the background material in high school like I should have and I could have picked this up quicker?

Anyway, I’m done complaining and it’s just too time-costly for me to learn more of this right now, so I won’t. I wish it were different but that’s just too bad for me, isn’t it.

Stanford Machine Learning Notes

Wow, I’m really loving this class. Lecture4 slides.

[I should probably point out to regular readers that these notes aren’t really fit for mass consumption. I’m not going to bother even trying to build a complete understanding of each of the concepts so they’re really just for personal use.

That being said, they’re on here and if you’re interested in seeing what I spend just about all my spare time doing, read on!]

Ok, today we covered multi-variate regression and we’re venturing into some virgin territory for me, now. It’s pretty awesome stuff.

So we started out with the insight that we could express a multivariate regression as a transposed matrix multiplication. What a mouthful. Believe me, it’s simpler than it sounds.

The idea is that you have a sec of values (slope values for the dependent variable) and a set of inputs (independent variables) and matrix multiplication just gives us a clean way of grouping them and then mashing them all together at once. This is clearly a programming optimization. If you did it by hand, it wouldn’t really be any easier.

The second idea is to express the error function of the gradient descent algorithm as a matrix. I’m barely holding on at this point, actually, and am looking forward to my first actual exercise.

Feature and mean scaling are next. These are neat little tricks to optimize the program. The idea is that if you have two features, sq footage and age of a house for example, which take on values of massively different magnitudes, your application of a uniform transformation of the slopes (the alpha term) will really frig up the algorithm’s progress.

So let’s say the slope of the sq. footage term is 350 and age is 5. If you apply a 0.01 modification to adjust the algorithm, you’ll barely move the age term. If you apply 0.5, you’ll be blazing away on the sq footage.

There’s some talk about graphing the error term of the error function so you can see your progress. I like visuals, so I’m on board.

There’s also a neat discussion about how to use the variables supplied to build your own variables. Using length and depth to compute sq footage, for example. Also arbitrarily raising some variables to some power: price of a house being related to the sq footage and negatively related to the sq of the sq footage. This is a nice way of introducing non-linearities.

We closed with a discussion of a closed-form solution for some of these problems. I finally lost my grip and will need to spend more time learning about the ‘normalized’ equation, which involves transposing the coefficient matrix and multiplying it by the training vector.

I totally get that there are tradeoffs to this approach versus the iterative gradient descent solution, though. Specifically, the trick is transposing that matrix of coefficients. Once you start transposing 10,000 x 10,000 matrices, it takes quite some time. I wonder if the transpose function in Octave is just an iterative function itself?

Back to the drawing board to deepen my understanding…

Down Memory Lane: Stanford Machine Learning Course Notes

These courses are serving up distinct reminders of why I’ve always done poorly at school: I’m lazy and sloppy. Very lazy and very sloppy. And my god do schools punish you for these personality traits.

The DB course is teaching me about my laziness. I’ve learned to call my brand of laziness “programmer’s laziness“. I would rather spend a bajillion hours building something that prevents me from doing 5 hours of work, as long as I can satisfy two conditions:

  1. I find a way of engineering the task in a way that interests me (this is easier than it sounds: lots of things interest me)
  2. Nobody tells me to do it this way

Usually the ratio isn’t a bajillion : 5. Usually I save a bit of time doing it because it would probably take me longer to use the conventional method. I suppose it’s not really laziness, as in an aversion to work; rather, it’s an extreme aversion to doing things in a manner I don’t enjoy/choose.

My second problem is that I’m sloppy. This one KILLS me in math-related courses. Now, my brand of sloppiness doesn’t really manifest itself in the workplace because the one-shot-and-done testing environment doesn’t really exist in real life.

Real math and real problem solving happen in an iterative, collaborative and failure-laden environment. I normally get so excited about solving a problem that I stop concentrating on stuff. I can go back later, realize I’ve been screwing it up and crunch away harder than I possibly could on the first pass. Computers take care of the arithmetic and, presto, the product improves. This makes me a TERRIBLE test-taker.

And I’m turning out some TERRIBLE test results right now. Ick.

Stanford DB-class notes

Well, I’m still in the course despite my grumblings. I’m determined to not screw this up.

I have a history with classes like this. In my second year of University, I took a finance course and COMPLETELY effed it up. Like, completely.

It was a tactical error, actually. I focused on the concepts and didn’t drill the equations. I’m still pissed off about that, 10 years later (holy *#$@, TEN years?!).

Anyway, this is clearly a course that looks to teach wrote-learning and I want to redeem myself. So I’m drilling* Relational Algebra this morning and XML data structures this weekend. Making the 7-hour drive back up to the in-laws, so maybe I can find a way for my wife to quiz me.

Ha!

*As an aside, I forgot how much I prefer to write on the right-hand page of a spiral-bound notebook, rather than the left. It just always feels cleaner. I am right-handed, of course.

Don’t Read With Sharp Objects Nearby

So I came home tonight in a bit of a mood and elected to shelve today’s work on the weekend project, crack a beer, order some food and curl up with the kindle backlog until my wife came home.

And now I’m depressed.

The first two articles tonight (one by Peter Thiel and the second by Neal Stephenson) were of the TGS variety: big long 3000-ish word whinge-fests on how we’ve stopped advancing technologically. Ugh…

The third (on Google’s dominance) brought me back from the brink but was still so shot through with ominy* that I remain perturbed.

Well, at least my beer hasn’t let me down. Whole Foods. Fantastic selection.

*does this word exist? I want it to be a collective noun for ominous things. Meh, it’s my blog, I can do what I want. There. Done.

Ok, I Did It

Fair warning to blog readers. I’m going to use this thing as a crutch for the Machine Learning and Database courses from Stanford, which I just finalized my enrollment in. Look out for lecture summaries and coursework as I think through the problems I encounter.

I’ve looked into each course and I am a bit skeptical about the Machine Learning stuff. Machine Learning is when you set a few algorithms loose on a gigantic dataset of uncertain value. Apparently humans are much better when guessing with even the slightest of an idea for what conclusions should pertain.

As a big believer in intuition, this suggests that Machine Learning is of limited use. Am I wasting my time? Should I be spending these hours on my weekend project (which I shall not forget)?

We shall see. God I hope I have the dedication to stick to this…