Chatterbox, and the Failure of the Hawthorne Effect

Hawthorne Effect: "a form of reactivity whereby subjects improve or modify an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied..."

I found out about the Hawthorne Effect by reading about Chris Anderson's TEDxSV talk about Living by Numbers.  In it he discusses HE's beneficial effect when you share measurements with a social circle (e.g. all your Facebook friends knowing how much your weight changes).  He's got a point.

The interesting thing about both the original experiment and Anderson's versions of HE in action is that they're on polar opposites of the freedom scale -- the original experiment measured subjects which had absolutely no ability to opt out of being measured; Anderson's subjects explicitly opt in. 

And in the middle of the freedom spectrum ... that's where Chatterbox comes in.

I mentioned Caffeinator in a previous post, where I also mentioned Nominum's very chat-intensive work environment.  At the time, I managed the QA organization, and so was one of the 3-5 managers in the engineering organization. 

At some point, we noticed that some people seemed to be chatting a lot on our chat system, and in some cases having much of that chat be not work-related.  This was doubly an issue because starting lots of idle chat means other people will join it.  At the same time, and probably at least partially because the work/life boundary was extremely porous, engineers were occasionally grumbling they worked many hours (despite basically setting their hours).

This seems suboptimal, and someone suggested we start talking to people we think are doing a lot of idle chatting.  To me, this seemed unnecessarily heavy-handed; not to mention, it would have required us to make some off-the-cuff decisions about who was chatting a lot without information.  It seemed .. lame.  I had a better idea (now's the time to look atop this post and note the name of this blog). 

I had already built a bot which interacted with our chat system.  I figured we'd build a new bot that just kept stats about who was chatting where and would provide aggregates of this information.  I figured we'd post this information publicly and people -- upon seeing the numbers -- would self-manage.  "Oh hey, I sent 450 messages to a non-work-related discussion last week, and only 38 messages to the discussion about my own product."  Makes sense, right?

Chatterbox was pretty elegant.  It did a bunch of aggregates -- per-hour, work-hours, non-work hours, per discussion, for work-related discussions, for non-work-related discussions, etc.  I even got a license of ChartDirector so we could make pretty graphs.  It was great.  I was proud of it, and the other management thought it was pretty cool.  We had high hopes.

And then everyone rebelled. 

We (management) knew we intended to not do anything with this information.  Frankly, the whole thing was created with the express intention that it would allow us to not manage employees around their chat participation.  Employees? Despite the fact Nominum was (and still is) the most laid back company I've ever worked in, from a management/engineer interaction basis, people got incredibly paranoid. 

Overnight, many engineers quit all discussions in protest.  In other cases, private discussions were created as analogs of the public discussions which excluded Chatterbox.  At least one engineer credibly threatened to resign if Chatterbox wasn't turned off.

We tried to address some of those concerns by turning down retention to one week so old information could never be used against anyone.  It did no good.  If I recall correctly, Chatterbox was decommissioned about 2-3 weeks after it became operational. 

I've made worse mistakes in my professional career, but in hindsight, this was probably one of the most interesting ones.