Managers regularly implement new ideas without evidence to back them up. They act on hunches and often learn very little along the way. That doesn’t have to be the case. With the help of broadly available software and some basic investments in building capabilities, managers don’t need a PhD in statistics to base consequential decisions on scientifically sound experiments.
Some companies with rich consumer-transaction data—Toronto-Dominion, CKE Restaurants, eBay, and others—are routinely testing innovations well outside the realm of product R&D. As randomized testing becomes standard procedure in certain settings (website analysis, for instance), firms learn to apply it in other areas as well. Entire organizations that adopt a “test and learn” culture stand to realize the greatest benefits.
That said, firms need to determine when formal testing makes sense. Generally, it’s much more applicable to tactical decisions (such as choosing a new store format) than to strategic ones (such as figuring out whether to acquire a business). Tests are useful only if managers define and measure desired outcomes and formulate logical hypotheses about how proposed interventions will play out.
To begin incorporating more scientific management into your business, acquaint managers at all levels with your organization’s testing process. A shared understanding of what constitutes a valid test—and how it jibes with other processes—helps executives to set expectations and innovators to deliver on them. The process always begins with creating a testable hypothesis. Then the details of the test are designed, which means identifying sites or units to be tested, selecting control groups, and defining test and control situations. After the test is carried out for a specified period, managers analyze the data to determine results and appropriate actions. Results ideally go into a “learning library,” so others can benefit from them.
Thomas H. Davenport. "How to Design Smart Business Experiments," Harvard Business Review, February 2009.
Some companies with rich consumer-transaction data—Toronto-Dominion, CKE Restaurants, eBay, and others—are routinely testing innovations well outside the realm of product R&D. As randomized testing becomes standard procedure in certain settings (website analysis, for instance), firms learn to apply it in other areas as well. Entire organizations that adopt a “test and learn” culture stand to realize the greatest benefits.
That said, firms need to determine when formal testing makes sense. Generally, it’s much more applicable to tactical decisions (such as choosing a new store format) than to strategic ones (such as figuring out whether to acquire a business). Tests are useful only if managers define and measure desired outcomes and formulate logical hypotheses about how proposed interventions will play out.
To begin incorporating more scientific management into your business, acquaint managers at all levels with your organization’s testing process. A shared understanding of what constitutes a valid test—and how it jibes with other processes—helps executives to set expectations and innovators to deliver on them. The process always begins with creating a testable hypothesis. Then the details of the test are designed, which means identifying sites or units to be tested, selecting control groups, and defining test and control situations. After the test is carried out for a specified period, managers analyze the data to determine results and appropriate actions. Results ideally go into a “learning library,” so others can benefit from them.
Thomas H. Davenport. "How to Design Smart Business Experiments," Harvard Business Review, February 2009.