Thursday, May 19, 2011

Chatterbox, and the Failure of the Hawthorne Effect

Hawthorne Effect: "a form of reactivity whereby subjects improve or modify an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied..."

I found out about the Hawthorne Effect by reading about Chris Anderson's TEDxSV talk about Living by Numbers.  In it he discusses HE's beneficial effect when you share measurements with a social circle (e.g. all your Facebook friends knowing how much your weight changes).  He's got a point.

The interesting thing about both the original experiment and Anderson's versions of HE in action is that they're on polar opposites of the freedom scale -- the original experiment measured subjects which had absolutely no ability to opt out of being measured; Anderson's subjects explicitly opt in. 

And in the middle of the freedom spectrum ... that's where Chatterbox comes in.

I mentioned Caffeinator in a previous post, where I also mentioned Nominum's very chat-intensive work environment.  At the time, I managed the QA organization, and so was one of the 3-5 managers in the engineering organization. 

At some point, we noticed that some people seemed to be chatting a lot on our chat system, and in some cases having much of that chat be not work-related.  This was doubly an issue because starting lots of idle chat means other people will join it.  At the same time, and probably at least partially because the work/life boundary was extremely porous, engineers were occasionally grumbling they worked many hours (despite basically setting their hours).

This seems suboptimal, and someone suggested we start talking to people we think are doing a lot of idle chatting.  To me, this seemed unnecessarily heavy-handed; not to mention, it would have required us to make some off-the-cuff decisions about who was chatting a lot without information.  It seemed .. lame.  I had a better idea (now's the time to look atop this post and note the name of this blog). 

I had already built a bot which interacted with our chat system.  I figured we'd build a new bot that just kept stats about who was chatting where and would provide aggregates of this information.  I figured we'd post this information publicly and people -- upon seeing the numbers -- would self-manage.  "Oh hey, I sent 450 messages to a non-work-related discussion last week, and only 38 messages to the discussion about my own product."  Makes sense, right?

Chatterbox was pretty elegant.  It did a bunch of aggregates -- per-hour, work-hours, non-work hours, per discussion, for work-related discussions, for non-work-related discussions, etc.  I even got a license of ChartDirector so we could make pretty graphs.  It was great.  I was proud of it, and the other management thought it was pretty cool.  We had high hopes.

And then everyone rebelled. 

We (management) knew we intended to not do anything with this information.  Frankly, the whole thing was created with the express intention that it would allow us to not manage employees around their chat participation.  Employees? Despite the fact Nominum was (and still is) the most laid back company I've ever worked in, from a management/engineer interaction basis, people got incredibly paranoid. 

Overnight, many engineers quit all discussions in protest.  In other cases, private discussions were created as analogs of the public discussions which excluded Chatterbox.  At least one engineer credibly threatened to resign if Chatterbox wasn't turned off.

We tried to address some of those concerns by turning down retention to one week so old information could never be used against anyone.  It did no good.  If I recall correctly, Chatterbox was decommissioned about 2-3 weeks after it became operational. 

I've made worse mistakes in my professional career, but in hindsight, this was probably one of the most interesting ones. 

Silver Linings

This post is relevant to the name of this blog due to the fact that it started with me, today, managing to forget my laptop at home.  this is probably the second time in my career at Netflix (689 days), and feels like a pretty debilitating problem -- I live on my laptop.  Everything's on it.  I back it up, so I'm not too concerned about losing it (there's nothing that's both confidential and unencrypted on it), but being without it for a day, last time, was a pain. 

And yet I'm sitting here and looking at the cloud's impact on how I work and ... I'm loving it.  I got a loaner laptop from our desktop people.  It's a delightfully sexy Thinkpad T410s which is great and ... a completely different architecture from my actual laptop (a 15" MBP). 

In the year or so since I last left my laptop at home, I've made some changes in how I work and where I keep my information.  This included:
  1. I use 1password to keep track of all of my passwords; 
  2. I use Dropbox to sync a small set of data to the cloud.  Dropbox hasn't been hugely integrated into how I work, but 1password integrates with it natively to back up its password cache to/from the cloud; 
  3. When I left the PC platform behind to go to the Mac, one of the big pain points I had was in abandoning Microsoft OneNote for note-taking -- but I replaced it with Evernote, which is 
    1. Free; and
    2. Auto-syncs to the cloud; and 
    3. Allows me to sync notes with various people (e.g. my previous boss)
  4. And of course, my mail/contacts/calendar have never been primarily stored on my laptop, since I work in a corporate environment (whether you're using Exchange or Google Apps, that point remains)
  5. My IM client (Trillian) stores most of its information (and all of the contact info) in the cloud
That means that, today, I spent about ten minutes installing 1password, dropbox,  Evernote, and Trillian.  And once that was done ... I'm basically functional.  Life is good.

I'm digging the cloud right now, for reasons entirely unrelated to Netflix's use of it.

Monday, May 16, 2011

Coffee and its Effects on Feature Creep

Starting in mid-2004 and until the end of 2006, I worked at a small DNS and DHCP software company called Nominum.  These days Nominum offers both software and services, but at its core what has allowed it to be as successful as it has been is the fact that its development group is, singularly, the most brilliant and personally committed to the company group of developers I've ever seen.  Most of the developers with whom I worked had been there from just about the beginning of the company.  It was the sort of environment where my VP was still doing the Costco shopping (though admittedly she enjoyed it), everyone knew every other employee's spouse's name, and we got together to have our dogs play. 

That's less relevant, however.  The two introductory bits of information necessary for this posting are:
  1. Nominum at the time had tons of RPI alumni, and because of this had implemented an RPI-sourced chat system named Lily.  Lily is more IRC-like than Yahoo Messenger-like in the sense that it's largely oriented toward offering multi-person chats (in IRC these are channels; in Lily these are called discussions); you can also send private messages, but it's typically used less often; Lily was very heavily used (one of the earliest lessons I had to learn was that if it was 1AM and my boss just sent me a private message asking me a question or asking me to take care of something, it was OK to say "actually, I was about to go to sleep; I'll deal with it in the morning."  It took me some time to learn this lesson);
  2. We worked approximately a mile away from Starbucks, and we went for coffee every day.
The problem was that the coffee ordering process was just terribly inefficient.  At some point in the day, someone would pop up on the "coffee" discussion and send out a broadcast "coffee?" to all interested parties.  At that point, people would have to say on the discussion what they want (e.g. "I'll have my venti non-fat 180 degrees vanilla  latte with two equals, extra squirt of vanilla, and whipped cream."  No, seriously, that was my drink at the time), the person who was volunteering to go would then write the order down (with a pen and a piece of paper), they'd go to Starbucks where the barista would have to decipher their handwriting, and make the drinks.  Typically, the person who went to pick up would pay for the whole order and then would try to keep track of how much each person owed them (people did not often have the exact cost of their drink available).

It doesn't take too many iterations of this to get a little sick of the inefficiency.

Lily was really easy to interact with -- it was a telnet-based interface, and we weren't doing any encryption.  Writing a bot to interact with Lily proved to be a trivial task.  I called it Caffeinator.

By the end of the first weekend of development (this was, obviously, a rather extra-curricular sort of effort), Caffeinator could see that ordering was opened, then listen to people saying "I'd like X," from which it would create a shopping list which it could post as a web page.  When ordering was closed, the person doing the shopping could visit the web page and print it out.  The order takers were happy.  Starbucks was happy.  Success.

It was still painful to have to say "I'll have my venti non-fat 180 degrees vanilla latte with two equals, extra squirt of vanilla, and whipped cream" every day.  So it was easy to have Caffeinator start keeping track of your favorite.  Saying "my favorite gnfl is a grande non-fat latte," for example, meant you could next time simply say "I'll have my gnfl" and Caffeinator would note you wanted a grande non-fat latte."

In rapid succession, we added the ability for it to optimize ordering for the Starbucks drink carriers (Starbucks drink carriers accommodate four drinks; ideally, all four drinks are similarly sized for balance, but at minimum it's preferred to have two pairs of identically-sized drinks).  We also added the ability to order for other people (so if I'm in John's cube when coffee ordering is open, John could say "Roy will have his gnfl") and the ability to define other people's drinks, if they let you (yes, I implemented coffee drink definition ACLs).  And that was still not that big of a deal.  It all sort of made sense.

In hindsight, it was probably right around the time that I implemented debt tracking into Caffeinator that I should have taken a break from enhancing it and reconsidered whether feature creep had gone way too far. 

Remember the part where someone comes back with your drink and you find out you don't have the $4.15 you owe them? It's annoying.  So after a particularly fun weekend, suddenly debts could be declared to Caffeinator.  Obviously, the potential for mischief when you can say "John owes me $4.15" is high, so the first implementation required people to declare their own debt ("I owe Roy $4.15.").  Of course, I also had to implement debt repayment ("John paid me $4.15").  The next implementation allowed for a proposed debt ("John owes me $4.15," which would result in Caffeinator telling you "John says you owe them $4.15.  If you agree, send me a message including this key: 'xxxasdfasdf325'").   Then, of course, I got curious about the total balance of debt/credit and so had Caffeinator report your TCNW (Total Caffeinator Net Worth) -- your total credits minus your total debts. 

It didn't take long to see some people's TCNW fluctuate way more highly than simply coffee ordering could account for, and realize Caffeinator became the default way in which social monetary debts were being tracked.  Your coworker with Amazon Prime order something for you? "I owe Matt $88.95." 

Once the network of debt and credit became saturated enough, it turned out that most people had dozens of people with whom they carried either a Caffeinator debt or a Caffeinator credit, and I came up with a way to allow people to simplify their credit/debt situation by reassigning debt.  Imagine Jim owes you $10, and you owe Bob $10.  Well, that's easy.  Tell Caffeinator "reassign $10 from Jim to Bob" and suddenly Jim owed Bob $10, and you were out of the picture.  Heck, two people owe you $5 and you owe four people $2.50 each? You can eliminate six credit/debt relationships at once. 

Of course, this being an environment full of mischief, the next thing we had to deal with was people gaming the system just for the sake of annoying others.  Bob owes you $1? Well, declare that you owe twenty people $.05 each, then reassign $.05 of Bob's debt to each of these people.  Suddenly, these twenty people find Bob owes them $.05 for no particular reason and Bob's really unhappy with you.

My last enhancement to Caffeinator was to allow people to opt out of debt reassignment.  That seemed to stop most of the gaming.

And that's how I set out to simplify ordering Starbucks and created an internal banking system. 

Why Now?

When I interviewed at Netflix, one of the more interesting questions I got was a question Allison Hopkins asked me.  She asked me to tell her about a time I had made a decision and, later, with the benefit of hindsight, thought to myself "oh boy, I did the wrong thing there."

It was a struggle to come up with a good answer.  This was largely due to the fact that I've made a bunch of these sorts of decisions in my life.  I pride myself on not engaging in analysis paralysis and on taking quick, reasonable risks.  It's not always worked out the way I thought it would -- sometimes for the better (see my later post about Caffeinator), sometimes ... the other way. 

And today (5/16/2011) someone pointed out to me an article about a Chris Anderson talk, where I encountered the term Hawthorne Effect.  Suddenly, I was reminded of one of my most hilarious failures, and thought I'd take a moment to document it so it transcends its current status as one company's piece of oral history.

Hence this blog, and hence its name.