Hunches first, data later, and how to tell when later has come
An early product has too few users for data to say much. A grown product has too much at stake to rely on one person's instinct.
Early in a startup, product decisions should be hunch driven. Later on, product decisions should be data driven.
Fred Wilson said this in a talk at Harvard Business School, and in a February 2011 post on his blog he noted that the line had been repeated widely since. He also said where he thinks good hunches come from: from being a heavy user of the products in a category and from a long obsession with the problem being solved. A hunch in his sense is not a guess. It is judgment built from experience that has not yet been written down as numbers.
The reason for the rule is statistical. A product with a few hundred users cannot run a meaningful test on most questions, because the differences are lost in noise. By the time enough data arrives, the moment has passed. Early on, the founder who has used every competing product and talked to dozens of customers is the best instrument available.
Later the balance flips. A product with millions of users can measure the effect of a change precisely, and the cost of a wrong hunch grows with the size of the audience. Large consumer companies test almost everything, from button colors to the order of search results. At that scale, a change that improves sign-ups by half a percent is worth real money, and only a test can detect it.
That second half has critics. In 2009 Douglas Bowman, a visual designer at Google, left the company and wrote that it had tested dozens of shades of blue for a link rather than trust a designer’s choice. Measurement can optimize details while missing the bigger decisions no test can answer, such as what product to build next. A company that runs entirely on data tends to climb to the top of the hill it is already on.
The rule also has exceptions early. Some questions can be tested cheaply before launch, such as whether anyone clicks on a price or signs up for a waiting list. A founder who could have checked and relied on instinct instead has not followed Wilson’s advice; he has ignored free information.
The practical question is when to switch. A reasonable signal is when tests start producing clear answers within a few days. Before that point, hunches carry most decisions and data checks them. After it, data carries most decisions and hunches choose what to test. Neither ever disappears.