Unlike Kim Kardashian, Reddit Got Backend Implants

That’s the nice thing about the scalable servers: they can handle 48MB per second.

In preparation for the IAMA, we initially added 30 dedicated servers (20%~ increase) just for the comment thread. This turned out not to be enough, so we added another 30 dedicated servers to the mix. At peak, we were transferring 48 MB per second of reddit to the internet. This much traffic overwhelmed our load balancers which caused a lot of the slowness you probably experienced on reddit. This IAMA gave us a lot of information on what we need to do next not only to handle the next megathread on reddit, but also to make reddit faster in general. We’re going to redesign our load balancer system to be more flexible under this kind of load, and we’ve got a bunch of things we need to do to make our code more efficient for giant comment threads. Both of these improvements should make day-to-day reddit use faster as well.

Read the rest of the post, which opens with this.

Without any fanfare ahead of time, The President of the United States spent 30 minutes answering questions from anonymous users with no mediation on a website run by 20 people, and in a forum that was organically created by volunteers. Tens of thousands of people actively participated by voting and/or commenting and millions watched, and The President even broke some news in the Q & A (by suggesting a new amendment for the first time). Did it go perfectly smooth? Nope. Is this the absolute perfect format for politicians to answer questions? Nope. But it worked, and it was a big step, not just for reddit, but for online communities everywhere. No matter how you feel about the IAMA, reddit, the President, or politics in general, we hope that what took place this past Wednesday will inspire other online communities and start-ups, and encourage other politicians to seek out more ways to talk directly with their constituents. We thank the President and his team for making this happen.

It’s true that the AMA is one of the coolest emergent phenomena on the web.

Obama Clobbers Reddit’s Backend

Pun intended.

The AMA is the usual Obama stuff. The interesting meta discussion, though, is on HN:

It looks like the Obama AMA has really caused some serious fires in reddit’s backend infrastructure. The site’s been down for the last 10 minutes.

I would’ve thought that they’d have brought in some additional computing power for such an event, should’ve been easy for them considering they have a cloud deployment. Maybe this gives them greater reason to hire more engineers. I found it impressive that they served billions of impressions with just 2 engineers a short while ago..

Also, it says a lot about the “Come Cloud with us, we’ll help you scale” marketing bandwagon. We’ve seen time and again issues with EC2s infrastructure and if EC2 doesn’t have issues right now (http://status.aws.amazon.com/) then it’s just sad that they can’t order a gazillion instances for this event and have it scale easily.

Definitely makes me think that we still have a long way to go to compute in a truly ‘elastic’ way.

reddit definitely does have some crazy infrastructure in place but this would’ve been one of the most important moments in reddit history (so far..) and I’m sad to see that their engineers are probably going to get blamed for this..

EDIT: Okay, they’re back in read only mode.. I wonder how they’ll hack in some write access for the AMA while keeping everything else read only. Time for some app server redeployments! Funsies!

EDIT2: And they’re gone again sigh

And this:

No, it’s probably due to the massive number of reads/writes pumping through their system. I’m pretty sure that their database isn’t able to handle what’s going on at the moment.
Also, if you were to use Varnish as a cache, that wouldn’t help you much with write access because every time there is a write (a new post..) you’d have to invalidate the cache. In a high volume scenario such as this (tons of posts coming in..), the page is as good as dynamic, even if you DO cache it for a bit.

And:

reddit’s backend could be written in assembly hand-tuned by God Himself, and having half the world hit it at once would still cause bottlenecks at the database.

More On The Future Of Education

Great new article on the possible pending disruption of higher education. Here are some quotes.

First what some ideas are:

That preppy-looking guy near the barbecue? He’s launching a company called Degreed, which aims to upend the traditional monopoly that colleges and universities hold over the minting of professional credentials; he wants to use publicly available data like academic rank and grade inflation to standardize the comparative value of different college degrees, then allow people to add information about what they’ve learned outside of college to their baseline degree “score.” It’s the kind of idea that could end up fizzling out before anyone’s really heard of it, or could, just maybe, have huge consequences for the market in credentials. And that woman standing by the tree? She’s the recent graduate of Columbia University who works for a company called Kno, which is aiming to upset the $8 billion textbook industry with cheaper, better, electronic textbooks delivered through tablet computers. And then there’s the guy standing to her right wearing a black fleece zip-up jacket: five days ago, he announced the creation of the Minerva Project, the “first new elite American university in over a century.”

And on what scale is possible:

To drive home the point of just how cheap it is to be Quizlet, one of its executives asks me how much money the United States spends per year to educate a single student in K-12 education. About $15,000, I say. That’s more than what it costs us per month to host the entire site, serving millions, the executive responds. Quizlet has no sales force, a very small marketing department, and more than seven million monthly unique visitors. (There are about fifty million public school students in the United States.) Quizlet, in its busiest months, during the school year, is among the top 500 most visited sites on the entire Internet. Now they’ve expanded beyond flash cards. You can create study groups, convert your content into multiplayer games, and search for cards and games that other people have created. We think we can get to 40 million users, then 100 million, says the executive. The question that drives the company, he says, is this: How can we create amazing learning tools for one billion people? This is the way most of the people in the valley talk.

And finally:

I can go online right now and get everything I need to learn—courses, textbooks, videos, other students to study with—for free. And if I need to know what someone else has learned, I can look at their Linked-In profile or their blog to find out.

At a certain point, probably before this decade is out, that parallel universe will reach a point of sophistication and credibility where the degrees—or whatever new word is invented to mean “evidence of your skills and knowledge”—it grants are taken seriously by employers. The online learning environments will be good enough, and access to broadband Internet wide enough, that you won’t need to be a math prodigy like Eren Bali to learn, get a credential, and attract the attention of global employers.

Here’s Yglesias:

The way it works is that you charge the same price for all the courses. When I tookPatrice Higonnet‘s five-person seminar on Vichy France, I didn’t need to pay a premium tuition over what I paid to take his 150-person lecture survey course on the French Revolution. Part of the way the college works is that the large courses generate profits that subsidize other activities, including the small seminars. The seminars themselves happen in part because some of the faculty wants to do them, and in part as an investment in the value of the brand. But while it would be very difficult to replicate the value of the small Vichy seminar, it’s pretty easy to imagine a French Revolution MOOC that’s both higher quality than your average French Revolution lecture-format survey course and radically cheaper

Profile Of A Gentrivolution

I moved to the city of Hoboken, NJ (a Manhattan bedroom community) from the big city a few months ago. It’s tiny at just a bit more than a square mile, bordered by waterways on the North, South and East and a big honking cliff to the West.

It’s much more distinct from the local social and urban geography than the map below suggests and it’s one of the most densely populated municipalities in the US.

It’s also a town whose socioeconomic makeup has changed radically since the 90s. Consider some stats:

2000 2010*
Population 38,577 50,005
Median Age 30.4 31.2
Median Household Income 62,550 101,782
Median Income Males Only 54,870 90,878
Elementary School Enrollment 1,781 2,156
Bachelor Degree or Higher 59.4% 71.6%
Currently Married 31.1% 32.5%

*The population is a 2010 figure and the rest are 2006-2010 averages. Here is a link to the 2000 data. Here is the 2006-2010 American Survey data. Here is Wikipedia.

In the early 90s, Hoboken boasted two communities: students (of the local Stevens Institute of Technology) and recent graduates fueled a big drinking scene while a ‘townie’ population of mostly blue collar workers and residents of a ‘projects’ in the bottom corner of the town were holdovers from the town’s industrial roots. It’s got Northeast college town written all over it to this day.

But the table above shows where things are heading. High-income, educated families with good jobs in the city, young children and stable marriages are filling the new ‘luxury condominium’ developments going up everywhere.

What’s interesting about this is that the boom has been projected many, many times in the past. During the West Village and Brooklyn renaissance of the 80s and 90s everyone was ready for the gentry to cross the Hudson. It’s been such a ‘sure thing’ that when it came the boom has been a horribly disappointing overdevelopment. Building projects abandoned (expecting a far larger boom), developers going bust, retail locations empty.

I’m sure the coincidence with the nation-wide housing bubble didn’t help. A lot of people in the area have lost a lot of money on housing developments. This is one town, though, where the units will eventually be filled.

The Angry Birds Era Is Over

I mean that in a commercial sense. The game will obviously live on.

Buying a game is a thing of the past because developers have figured out the revenue model that lets games be free.

Right now, 18 of the top 25 grossing of all apps are Free To Play Games (72%).  Also, it should be noted that 22 of the 25 top grossing apps are in the games category (88%), confirming the fact you need to be into games if you want to have the biggest potential payout.  The reason for this is people have a stronger emotional attachment to games than any other type of app, therefore they are more likely to spend money.

How are these Free to Play games crushing it?

After digging deeper in these top grossing apps, you can see they consist of nearly every free to play genre there is… Social games, click games, gambling games, turn based games, card games, etc but all of these have TWO things in common:  They each have lots of in app purchases and they encourage the user to buy stuff (a call to action).

This is the basics, but it’s SUPER IMPORTANT, here’s how:

A very small percentage of people buy stuff in games.  Of this small percentage you have people who will spend a LOT.

More here.

As many note on the HN discussion, this is really really annoying. But so are television commercials.

7-Year-Old Wins Ram Groping Contest (In Iceland)

Seven-year-old Jón Haukur Vignisson unexpectedly won the highest score among non-professionals in the annual national ram groping tournament organized by the Sheep Farming Museum in Hólmavík, the Strandir region in the West Fjords, last weekend.

More here. Amusing stuff. And the winner gets…

Ram gropers are of all ages and walks of life. Among the prizes is sperm from the Insemination Center of West Iceland, which is much appreciated among sheep farmers.

Hardware Fantasy Links (hey hey hey, keep it clean)

I am a big fan of computer programming but deep in my heart I’m a frustrated electrical engineer. I am captivated by the fundamentals of computing and hardware interaction.

Here are some things I’ve enjoyed.

Code (by Charles Petzold, published in 1999). This book winds up in all kinds of “best programming books” lists and what a revelation it is. Mostly it’s concerned with answering the question: “how would you build a computer with 19th-century technology?”. The rest is an extremely detailed look at what computer hardware actually DOES and how software interacts with it. Far more readable than it sounds.

Once you get through the basics of how computers work, Petzold machine-guns you with a quick explanation for just about every acronym, file format, compression technique and common technology in 1999. What is a bitmap? How do scanners work? What is an analog signal and how is it converted to digital? Where did MS-DOS come from? How do modems work? Just about every paragraph of the last 20 pages gave me an “ah-HA!” jolt.  I haven’t put a book down and wished for more in a while but an update on Internet technologies and perhaps a chapter on mobile hardware/software are surely in the works!

Programming Throwdown podcast – Specifically the ones on Assembly and C. I’m throwing C into the low-level programming boat because I can. Not that I really understand this link (yet?), but you can write linux device drivers with it!

Technometria (podcast). These guys are doing a series on the “Internet of Things”, discussing trends in hardware programming, specifically as it connects to the web.

Personal data ecosystem (podcast). This is an interesting series on ways that personal data is being collected and used. Some of it has to do with business models, much of it has to do with privacy, which is boring.

One interesting aspect of the Internet of Things phenomenon is that it is most closely associated with home automation but home automation as a business idea died a long time ago. Nobody will pay for it. People in this field are constantly trying to distance themselves from those applications.

The real advances in the Internet of Things are typically concerned with automating processes that already are fairly well automated, squeezing the last few drops of human input (cost) from things like building cars or monitoring traffic. The revolutions in these fields happened a long time ago

So the Internet of Things is a very 20th century pursuit. A 21st century engineering challenge would be about cracking the human to human economy (ie beating Turing tests). If you can make computerized social workers you’ll change the world.

Coca-Cola’s Value: Still Marking to Model

Three people have been arrested for allegedly attempting to sell Coca-Cola(Charts) company secrets to PepsiCo (Charts), according to the Department of Justice.

Two residents of Georgia and one resident of New York City are purported to have participated in the scheme to sell the Coke secrets to rival Pepsi for $1.5 million.

In May, PepsiCo told Coca-Cola that it had received a letter from a person calling himself “Dirk” offering “very detailed and confidential information” about Coke’s products for a fee, according to the DOJ press release.

More here.

Time Flies When You’ve Practiced Your Butt Off

One of these ideas that I like being hammered with over and over is that performance (skill) is about repetition. It’s about muscle memory for athletes and autopilot recall for exam-takers.

And this is what it’s about for Navy Seals:

The best preparation for any scenario is having done it before — or the closest thing to that.

Via Daniel Coyle’s excellent book The Little Book of Talent: 52 Tips for Improving Your Skills:

When U.S. Navy SEAL Team 6 mounted its May 2011 raid on Osama bin Laden’s compound in Pakistan, it prepared by constructing full-scale replicas of the compound in North Carolina and Nevada, and rehearsing for three weeks. Dozens of times the SEALs simulated the operation. Dozens of times, they created various conditions they might encounter. They used the power of repetition to build the circuitry needed for the job.

This is one of the key principles of deliberate practice, the best system for building expertise.

Another way of putting this is that these SEALs have planned and practiced reactions to every conceivable scenario. That way they can maximize how many situations they react to with ‘instinct’, which is just a way of saying they’re on mental autopilot. When trained properly, autopilot is simply a faster and more reliable mode of human thought than focused attention.

The cornerstone of my exam strategy is mental autopilot. I maximize what I can achieve without really thinking, saving my mental resources for the very end. Here is the progression I go through at exam time:

  1. Comb through the test and read every problem. If I know the answer INSTANTLY, I work the problem (category #1 problems). If I have any hesitation at all, I skip it. If I don’t complete more than half of the test on this pass, I am in big trouble. Time: 25-33% of total allotment (45 mins-1hr in a 3-hour test).
  2. Go to the problems (category #2 problems) that I think I can figure out quickly and skip the ones that are really tough. Most of these problems are easy problems disguised as really hard problems. Once I crack it, I switch on autopilot again and blow through the sucker. This should get me to about 75% complete. This is probably about another 25% of the exam time.
  3. Go back over every question and (re-)work them all. I’ve probably screwed up a few category #1 problems (which are actually category #2) and I correct these here. It takes discipline to keep skipping the really tough problems but I need to save those. Get the easy ones right!
  4. The real toughies should be <10% of the total problem set. Now you tackle them.  You’ve probably got about 20-30 mins left on the exam and your brain is nearly fried. These last few minutes will take FOREVER because once you switch off autopilot the world slows down.

This is why return journeys feel quicker, why childhood feels like it takes FOREVER and why time flies when you’re having fun. Thinking is HARD. Thinking is PAINFUL. Your brain only has a finite amount of focus and attention and lighting up all those neurons is costly.

When you are studying, your goal is to have an autopilot that is as complete as possible. Understand that to sign up for an exam is to sign up for mental torture: you need to think a LOT to learn and, as I said, thinking is PAINFUL. That’s ok: no pain, no gain and you’re here to make something of yourself, dammit.

If you don’t train properly, you need to do more thinking on the test, which is costly. You need to work dozens/hundreds/thousands of problems to have well worn grooves in your mind for the problems you encounter.

Generally speaking, kids have no grooves. Older folks have nothing but grooves. Neither group likes putting down those grooves; kids just have no choice in the matter.

What Is A Good Company? A Bad One?

In good organizations, people can focus on their work and have confidence that if they get their work done, good things will happen for both the company and them personally. It is a true pleasure to work in an organization such as this. Every person can wake up knowing that the work they do will be efficient, effective and make a difference both for the organization and themselves. These things make their jobs both motivating and fulfilling.

In a poor organization, on the other hand, people spend much of their time fighting organizational boundaries, infighting and broken processes. They are not even clear on what their jobs are, so there is no way to know if they are getting the job done or not. In the miracle case that they work ridiculous hours and get the job done, they have no idea what it means for the company or their careers…

That’s Ben Horowitz. There is also a discussion about a company called Go, which you’ve never heard of but are about to learn something from:

When I first met my friend Bill Campbell, he was chairman of Intuit, on the board of Apple and a mentor to many of the top CEOs in the industry, including Steve Jobs and Jeff Bezos. However, those things did not impress me nearly as much as his time running a company called GO Corporation. GO essentially attempted to build an iPhone in 1992. The company raised more money than almost any other venture capital back startup in history and lost nearly all of it before selling itself for nearly nothing to AT&T in 1994.

Now that probably doesn’t sound impressive. In fact, it probably sounds like a horrible failure. But I’d met tens of GO employees in my career, including great people like Mike Homer, Danny Shader, Frank Chen and Stratton Sclavous, and the amazing thing was that every GO employee that I’d ever met counted GO as one of the greatest work experiences of their lives. The best work experience ever despite the fact that their careers stood still, they made no money and they were front-page failures. GO was a good place to work.

My favorite test for whether a company is excellent company or not is whether the people who were a part of it go on to do extraordinary things. Think of the Paypal mafia.  It’s not clear to me that GO passes this test.

In the (re)insurance business, there are two Paypal mafias that come to mind: AIG’s actuarial department in the 80s and F&G Re. The top ranks of my business have over the last 20 years been massively over-represented by people with one of these two lines on their resume.

There is also an interesting discussion on HN about a rather controversial passage in the piece. Here’s the setup:

At Opsware I used to teach a management expectations course because I deeply believed in training. In it, I made it clear that I expected every manager to meet with her people on a regular basis. I even gave instructions on how to conduct a 1:1 meeting so there could be no excuses.

Then one day while I happily went about my job, it came to my attention that one of my managers hadn’t had a 1:1 with any of his employees in over six months.

Ben threatened to fire this manager and the manager’s boss if they didn’t fix this in 24 hours. A commenter thought that was harsh and perhaps a bit arbitrary. Another one responded with this:

It’s even worse than that, the people that work for you will make their number one priority not getting fired.

I’ve worked at a company like that before. Management worked hard on whatever problem the CEO noticed last, while doing their best to hide any other problems from him.
As a manager you do much better at aligning everyone’s interests so that your staff does what they want to do, which just happens to work towards the outcome you want. It’s more about gentle course corrections ahead of time than grabbing the wheel from them.

This is evocative to me of the hardest management problem in insurance. In insurance, like in all businesses, there is pressure for companies to grow. In insurance, also like in other businesses, growing by cutting your prices to the point where you lose money is one self-defeating option.

Unlike in other businesses, though, the break-even price for insurance is basically unknowable when you charge it. So you charge something that is probably close and work out differences across time.

This means insurance managers can be a schizophrenic bunch. One day they’re focused on growth, the next day they’re worried that they’re cutting their prices too much to grow profitably. The worst managers expose their employees to the full horror of this uncertainty. The best find a balance and help their subordinates find their own balance.

I’m not sure this criticism is justified in the story, but the lesson still stands: understand your management priorities and be consistent in applying them.

And one of those priorities is that your turf should be a good place to work.