What A/B testing can find, and the hill it cannot see past
Testing climbs the nearest hill well. Choosing which hill to climb is a different kind of decision.
You can achieve a shallow local maximum with A/B testing, but you’ll never win hearts and minds.
Picture a walker in fog who can only feel the slope under their feet. Taking every step that goes uphill will bring them to a summit, but only the nearest one, and they will have no idea whether a far higher peak stands a mile away. Mathematicians call that summit a local maximum, and A/B testing is a very efficient way of walking uphill in fog.
Jeff Atwood made the argument in July 2010 in a post on his blog, Groundhog Day, or, the Problem with A/B Testing. He used the film as his parable: Bill Murray’s character relives the same day over and over, refining each move in his pursuit of the woman he wants, and the perfectly tuned performance still fails because it has nothing genuine behind it. A product built only from winning variants risks the same result, polished in every detail and without a reason for anyone to care.
The best-known complaint of this kind came from Doug Bowman, who left his job as a visual designer at Google in March 2009. He wrote that design questions there were routinely settled by data, including a test of forty-one shades of blue to decide the color of a link. The test surely found the best of the forty-one. It could not say whether the page needed a different idea altogether.
The case for testing is strong where the question is narrow. During the 2008 presidential campaign, Barack Obama’s team tested combinations of images and button labels on its sign-up page, and the winner raised the sign-up rate by roughly forty percent over the original. Across millions of visitors that meant a very large number of additional supporters and donations. Nobody would call that shallow in money terms.
Both things are true at once. Testing is excellent at choosing among options someone has already imagined, and it cannot supply an option nobody has imagined. It also measures what can be measured in a week: clicks, sign-ups, purchases. The reasons people stay loyal to a product for years show up late and faintly in that data.
A sensible division of labor follows. Decide the direction by judgment, conversation with users and taste, then use tests to tune the details once the direction is set. If every test has started producing gains of a fraction of a percent, that is often a sign the team has reached the top of its hill and needs a new one rather than another variant.