Test and Learn: Making Confident Decisions on the Commercial Side

Because rigorous application of the scientific method is at the foundation of the pharmaceutical development cycle, readers of Pharm Exec may be surprised to learn that the industry has much to learn from companies in other sectors where scientific testing on the commercial side of the business is rapidly becoming the norm.

The power of insights drawn from in-market testing has increased dramatically because of advances in information technology, large-scale data management, and computation speed. Catching this wave, a growing number of leading retailers, banks, insurance companies, restaurants, and hotel chains have adopted the scientific method to accelerate innovation while mitigating risk. These organizations have termed the process "Test & Learn," and they are applying it to understand the impact of digital marketing programs, new advertising content and targeting, promotion effectiveness, sales force management tactics, and customer communications. Test & Learn enables organizations to learn which ideas work and which don't before taking the risk and expense of rolling them out. Though most companies have tried to measure commercial tests ad-hoc, the best practitioners of Test & Learn are institutionalizing this capability and prioritize, design, and measure tests using dedicated software, ultimately enabling a consistent decision-making process.

A few leading pharmaceutical companies have recently begun applying Test & Learn. Given the industry's depth of experience with lab-based testing, it may seem ironic that market-based experimentation is just now catching on in Big Pharma. But interest in the Test & Learn approach has been sparked by ever tightening commercial budgets and the industry's long-term mission to reach the right customers with the right products more successfully, quickly, and efficiently.

In one recent example, a leading pharmaceutical company operating across multiple therapeutic units took a sophisticated Test & Learn approach to building and evaluating a DTC advertising campaign supporting a blockbuster drug. The brand team launched a print advertising campaign in select markets, targeting the most attractive demographic and customers. The team was having a difficult time evaluating the results to determine the cause-and-effect relationship between the ad campaign and new scripts—if any.

The company used advanced Test & Learn software to get an accurate and useful understanding of ROI. Analysts used the Test & Learn software to create optimal control groups for the test markets, in order to remove the inherent volatility in prescription data that occur by-location, by-physician, and by-patient. Changes in prescriptions in the test markets (receiving the campaign) were then compared to changes in scrips of the custom control group (not receiving the campaign).

Test & Learn analysis showed that the advertising was driving a significant increase in new scripts overall. This sales lift was generating profits well in excess of the cost of the advertising—a sign that the ads were working well.

Moreover, by de-averaging the results, the brand team learned that the program was most successful in driving scripts from primary care physicians with a history of prescribing the product, and in markets with higher median income. Based on these findings, the team doubled its advertising investment, targeting the rollout to specific markets predicted to cause a significant increase in overall scripts and profitability.

Test & Learn's advantages are increasingly acclaimed by academic and business leaders. Jim Collins wrote in "Great by Choice" that, "In the face of instability uncertainty and rapid change...analytic skills still matter, but empirical validation matters much more." And a Harvard Business Review article entitled "A Step-by-Step Guide to Smart Business Experiments," argues that, "It's easier to draw the right conclusions using data generated through experiments than by studying historical transactions."

As it applies to commercial decisions, Test & Learn means trying an idea (such as a new marketing, sales force, or pricing program) with a small group of markets, patients, physicians, or reps and comparing the change in behavior of that group with the behavior of a carefully-matched control group that has not been exposed to the idea. When effectively applied, Test & Learn answers three key questions about any proposed business idea, before taking on the risk and expense of rollout:

» Will the program work?

» With which patients, reps, physicians, or markets will the program work better?

» How can we fine-tune the idea so that we are only moving forward with the most effective components?

These questions are addressed by a variety of large and small companies across the business frontier: from large companies such as Walmart and Target, Walgreens and CVS, Hilton and Holiday Inn, Kraft and Anheuser-Busch InBev, McDonald's and Starbucks, to smaller companies such as Guitar Center and Banfield the Pet Hospital. Some of these companies' internal estimates of the value of Test & Learn eclipse $100 million per year in added profits.

As pharmaceutical companies move forward with Test & Learn they are learning two things. First, analytically it is very different from patient-level testing. A test of a new sales force targeting program, which can be applied at the market or prescribing physician level, and the analytical challenges of driving reliable results from such a test are quite different than the challenges involved with the traditional double-blind patient-level test. Different analytical approaches are needed for test design, test and control group selection, and test evaluation. Ongoing developments in big data, coupled with extraordinary advances in algorithm development, have allowed analysts to overcome issues of business volatility, competitive initiatives, the confounding influence of payers and patients, as well as continuous macro-economic changes, to allow organizations to isolate the incremental impact of new ideas with high confidence.

Second, and more importantly, companies in our industry are learning that Test & Learn is much more precise and actionable than the more conventional conjoint, correlation, or regression analyses that evaluate historical data in an effort to inform future alternatives. Test & Learn provides direct insight into the specific impact of the new idea, as well as opportunities to modify it for maximum effectiveness and target its rollout for maximum returns.

Leading pharmaceutical companies can drive millions of dollars in shareholder value by applying Test & Learn tools to measure, refine, and target national and local media campaigns, reimbursement strategies (including couponing), speaker program investments, electronic marketing programs, sales strategies and incentives, and patient communication. Early results of Test & Learn being applied at pharmaceutical companies are compelling and it is believed those companies that adopt this methodology soon will have a long-standing competitive advantage over those companies conducting business as usual.

As Tom Davenport argues in an HBR article titled, "How to Design Smart Business Experiments," "testing, and learning from testing, should become central to any organization's decision making. The principles of the scientific method work as well in business as in any sector of life. It's time to replace 'I'll bet' with 'I know.'"

Ben Bonifant is President of Bonifant Insights Group, LLC. He can be reached at bbonifant@bonifantinsights.com

Scott Setrakianis Managing Director at Applied Predictive Technologies.He can be reached atscott.setrak@aptmail.com