The Complaints about Facebook’s Study are Ludicrous and Bad for Science

Screen Shot 2014-07-02 at 3.17.16 PMI wasn’t surprised to see Slate write the click-baity article “Facebook’s Unethical Experiment” about Facebook’s controversial study where researchers showed that increasing positive messages in a user’s news feed lead that user to post more positive messages.  But I’ve been shocked by how strongly the entire internet has come down against this research and I’m really concerned about the stifling effects of this misinformed outrage. Every website you visit is manipulating you and most are doing  controlled studies right now to figure out how to manipulate you better.  And overall, this is good for you.  Google constantly uses experimental groups to test subtle changes in the ranking of results to make them more relevant to you and to make you happier.  Amazon endlessly experiments on you to see if they can get you to buy more, but this isn’t usually a bad thing – their recommendation engine is amazing and often truly useful..  Remember when sign-up flows used to have a million steps and credit card forms were clunky?  Through controlled experiments, millions of companies have learned that bad UI makes users unhappy and is bad for business. So why are people in such a frenzy about this Facebook study?  If it’s because it measured the effect of manipulating emotion directly, that seems misguided/crazy  It would be a great thing if websites tried to measure their effect on emotion more directly – maybe they could try to optimize for long-term user happiness rather than click-through rates. Some people are angry because they think academics should be held to a higher standard than companies.  But in this case holding academics to a “higher standard” just forces corporations to never collaborate with them and makes these interesting and potentially important results necessarily unpublishable.  This study was a minor variation on something every consumer internet company does all the time and is nothing like the Stanford prison experiment. If journalists want to worry about technology companies manipulating people, I think they should worry about game companies that use Skinner Box style reinforcement and endless experimentation to hook people into playing games over and over.   Facebook investigating whether positive messages in a news feed lead to positive feelings is a good thing. I hope they keep looking into it, although after the media’s reaction, I’m sure they will never share their results publicly again.  Important research will remain in the hands of large companies rather than shared across research institutions.

Hiring by Consensus

Every company I’ve worked for has followed a similar official hiring process.   We put out a job description, resumes come in, we sift through them, screen the promising candidates and then bring the finalists in for a day of interviews.   Then all the interviewers convene and everyone gives a clear “HIRE” or “NO HIRE”.

I’ve come to the conclusion that this is a bad process.

When I look at the truly exceptional hires that I’ve made, they have rarely followed this organized process.  I met Zack Kass, now my VP of Revenue and one of the most important hires I’ve ever made at a poker game, after which he harassed me on twitter until I unilaterally gave him a contract position.  I hired John Le, one of the best engineers I’ve ever worked with, after seeing his Putnam score and giving him an impossibly difficult programming test.  I recruited Tatiana Josephy, now my VP of product, without a clear role for her at the time, just knowing that she would crush whatever she ended up doing.

Hiring is the only crucial decision I can think of that companies make by consensus.  We don’t set goals by consensus, we don’t set product strategy by consensus, we generally know that consensus leads to mediocrity.

People often point out that a bad hire costs 1.5 times a year’s salary.  That sounds pretty expensive.  But a great hire is worth at least 10 times a year’s salary.  Focusing on the down side of a bad hire ignores the opportunity cost of a mediocre hire vs. a great one.

We viscerally feel the negative impact of a bad hire, but it’s much harder to see the opportunity cost of missing out on a great one.

I once had a summer intern, Howie Liu, who was obviously an exceptional guy.  He wanted (what seemed like) a ludicrously generous offer to come on board full-time.  I turned him down and he went off and started two successful companies.  Clearly I made a huge mistake not fighting harder to bring him on, but I somehow don’t remember that failure as vividly as the day-to-day frustration of dealing with the bad hires I’ve made.  When I ask my team to review my performance they always bring up bad hires I’ve made, but they’ve never once brought up a great person I failed to bring on board.

All of the great people I’ve worked with have had flaws that were tied to their greatness.   Many of them would rub certain people the wrong way in a standard interview process that seems designed to reward candidates with a lack of weaknesses over candidates with exceptional strengths.  I want to hire for upside and greatness and I want a process that supports that.

Being ENTJ doesn’t make you a CEO, being a CEO makes you ENTJ.

Steve-Jobs-HomesteadIn Silicon Valley we have stereotypes.  We have the aggressive, intuitive Steve Jobs style leader.  We have the competent, disciplined Sheryl Sandberg style leader.   We have the spastic, unfocused startup founder.  We imagine that Steve Jobs was born being Steve Jobs, because we’ve only seen him as the CEO of Apple.

We massively underestimate how much the environment we put someone in shapes their personality.

I’ve seen this first-hand.  When you are running a company rapidly growing from 5 to 25 people, and you deeply care about its success you will be spastic.  There’s so much to build, so few processes in place.  I’ve never seen a founder not become the stereotype of a startup founder, harassing engineers, rapidly changing priorities.

When your company is suddenly valued at over a billion dollars, and you have all the opportunities and problems that come with that, it’s hard not to become confident and forceful.  I’ve watched several introspective, nerdy founders get nine and ten figure valuations and over the next year or two gain confidence and become aggressive, decisive CEOs.  I’ve never seen it not change someone.

Looking back over the last ten years, watching some people achieve massive success and others fail, I’m more and more convinced that success in the long run it has a lot to do with the person and less to do with luck. But I’m also convinced that environments dramatically shape and change personalities. When we look at early stage founders and try to predict success, its easy to assume we want to find replicas of the CEOs we already know have achieved success.  I think that’s a mistake. Instead, I would look for people that are adaptable and pragmatic.  By the time the IPO comes around, they will look like they’ve been doing it their whole lives.

Recording Sensations

How much hard drive space would be required to record all the sensations that we experience in a day?

The most natural way for humans to store an image is an array of intensities red, green and blue values.  This makes sense because our eyes have sensors (cones) that have a peak response to red, green and blue light.

The most natural way for humans to store sound is two arrays of frequencies.  This makes sense because we have sensors in each of our ears that respond to frequencies of pressure.

I think the most natural way from humans to store taste is a five dimensional array of flavors for the intensities of sweet, sour, bitter, salty and umami.  But maybe there would also need to be spiciness and even the temperature of the food?  Has anyone ever tried to record a taste and replay it?

I’m less sure about smells.  I think every smell is a different chemical reaction in our nose, so it seems like the best way to describe a smell would be a long list of the intensities of various chemicals.  But maybe there is a hidden deeper structure.

I wonder even more about touch.  Surely to fully describe all the touch sensations in a moment in time you would need to record temperature and pressure everywhere in the body.  I’ve read that the feeling of wetness is just the feeling of cold plus light temperature and that it’s surprisingly easy to trick the body into feeling wetness with cold light pressure.  What about itchiness and numbness?  Is the heat from a rash the same feeling as heat from a stove?

I am the most unsure about internal sensations.  Is there one kind of nausea or many types?  I think I can imagine at least two.  One kind of headache?  What about tiredness?  Intoxication?

And does it even make sense to include feelings in this discussion?  Certainly we recognize a common set of feelings such as happiness and anger and we know that we can combine them in different ways.  We can even combine opposites – I think that it’s possible to be happy and sad at the same time, which means they would have to be recorded separately. 

Even if you add up all the different smells and internal feelings, I think recording all of the light that hits our eyes might be more data than all of the other sensations combined.

Go and the Beauty of Efficient Resource Allocation

One thing that Go players learn quickly is a visceral, aesthetic appreciation for an efficient allocation of resources.

Even a beginner would recognize the black stones in the first diagram as beautifully placed.  It is the minimum number of stones to securely enclose the White group in the upper left.

go1The next diagram adds just a single stone and is immediately recognizable as ugly.  Even though it is a significantly more secure shape, it is not an efficient allocation of resources.


The third diagram adds yet another stone and is an even stronger shape in practice, but it is such an inefficient allocation of stones that it would never occur in a real game.


If only it was this easy to tell if a company was allocating resources efficiently.

Metrics and Hiring

Virtually every CEO claims to be extremely metrics driven.  As far as I can tell all of these CEOs are completely full of shit, at least when it comes to their own hiring.

We all hire based on intuition but we rarely go back and rigorously check our mistakes.  In my career, I’ve hired or been part of hiring 129 people that I know or remember well enough to confidently grade their performance.  Before last weekend, not once had I gone back and looked at the data.  Of course, that hasn’t stopped me from having strong opinions about interviewing and hiring.


So I went through and labeled every hire on a scale from disaster to superstar.  I was involved to different degrees in every hire, but unfortunately I don’t have notes on the process that happened for each candidate.  The distribution of hires was about what I expected.

Next, I labeled the referral quality on a three point scale from unknown (0) to close friend (3) and the school quality from unknown (0) to ivy league (2).  I looked at the average school quality and referral quality for each type of outcome.  The results of my unscientific survey were striking and statistically significant.


Referral quality was incredibly important – the eight worst hires that I’ve been involved in were all unknown to me and everyone at the company at the time of hiring.  Digging into the data showed that employee referrals were basically as good as my own personal referrals.  The average hire was a “might hire again” but the average employee or personal referral was a “definitely hire again”.

School quality was also positively correlated with performance, which was a surprise to me – I’ve always felt that people tend to overweight school quality in hiring decisions, but the data says that I probably underweight its effect.

Ask for Stories, Not Advice

People have a deep, intrinsic need to feel heard.  I’ve found the best way to charm anyone is to ask for advice.  This is especially effective with old men, who generally control the most resources.  Hence the old adage “Ask for money, get advice.  Ask for advice, get money.”

Paul Buchheit describes the fundamental problem with advice well, “Limited Life Experience + Overgeneralization = ADVICE”. 

As a first time founder, I was inundated with unsolicited advice.  Now I’m inundated with people pretending to ask for advice when they really want my money or introductions to people with more money.

My angel investor/mentor Travis gave me the most memorable advice I ever got: ask for stories, not advice.  Stories are much more fun to listen to and more fun to tell.  The listener can draw their own conclusions.