Trial by Design

I recently worked with a client on their clinical trial protocol for a huge study involving thousands of patients and hundreds of independent clinical investigators across the country. The study's protocol—the detailed plans and regimens that investigators must follow to test a drug in clinical development—was precise. Not only did it have lots of tests to determine the efficacy of the drug, but the protocol included a raft of criteria to ensure that only patients with a precisely defined set of health characteristics would participate in the study.

However, the company felt that such a rigorous design would create some problems. By creating such careful screening criteria, it would be difficult to find enough patients to fill the trial.

Because the tests to be administered were burdensome (such as multiple blood draws, 24-hour ambulatory blood-pressure monitoring), they might deter qualified patients from participating.

The length and complexity of the protocol were bound to cause difficulties with patient drop-outs, driving up the time and cost to run the trial.

The company's worries were well-founded—and they're not alone. Pharma manufacturers of every size are running into problems operationalizing the protocols they set forth for their clinical trials.

Every R&D professional has a host of horror stories about protocols that went wildly awry: multiple amendments to the protocol, enrollment delays, and poor data quality, to name a few. But when asked, most people believe that the protocols are just fine—they see the science as generally strong, and the methodologies have a high likelihood of successfully determining the efficacy of the drug.

This disconnect between people's perception of protocol quality (high) and actual protocol quality (low) has led us to believe that there are actually two different kinds of protocol quality. Through our conversations with heads of R&D, medical management, and clinical operations at a dozen large and medium-size pharmas, we've seen that R&D organizations are set up to focus on only one of them. This organizational stumbling block thwarts effective execution of the protocol, and costs the typical R&D organization millions of dollars a year in lost productivity and months or years in lost cycle time.

This article explores these two types of protocol qualities. It offers some solutions to companies that want to improve protocol quality—based around execution and clinical trial operations—and in the process, bring their clinical trials under control, completing them on budget and on schedule.

Protocol Quality, Defined

During the development of a clinical trial protocol, most pharma R&D organizations focus on "scientific" protocol quality—that is, making sure that the scientific goals of the protocol are sound and that there is a reasonable chance the protocol will achieve study goals. Questions such as, "What tests do we need to determine whether our drug is effective?" or "How many patients are necessary to assure statistical accuracy of our results?" receive well-deserved and careful attention.

Protocol development teams often bring in medical consultants to help hone the scientific aspects of the protocol. But in their efforts, they often overdesign it, such as when they create too-stringent criteria for patient enrollment, making the protocol nearly impossible to execute. Protocol Review Committees tend to focus on the scientific aspects as well, poking and prodding the protocol to ensure it has a high probability of meeting its scientific goals. As a result, the team often amends the protocol, which then must be approved by FDA. Given that the average Phase III protocol has five amendments, costing almost $200,000 a piece in out-of-pocket costs, companies are spending nearly a million dollars on amendments per protocol. But when all is said and done, the scientific quality of a protocol tends to be reasonably good.

There is a second type of protocol quality that is focused around the quality of the execution of the clinical trials. "Execution" quality addresses issues such as whether effective practices and processes are being used in the protocol; whether the protocol can be executed within the time and cost specified; whether the inclusion/exclusion criteria, procedures, visit schedule, etc. are necessary for the achievement of the protocol's goals and endpoints; and whether any of those events will hinder execution tasks such as subject enrollment/retention or data collection. While execution quality doesn't deal with the protocol from a scientific standpoint, it is nonetheless critical to a successful outcome. No matter how good the science is, a protocol that can't be executed effectively has very little real value.

When considered together, these two types of quality complement each other: Scientific quality tends to expand a protocol's complexity in the name of good science, while execution quality tends to limit it in the name of efficiency and effectiveness. When they are in balance, the two work together to create an optimum protocol. Out of balance, the protocol becomes either too bloated to be executed effectively or too limited to achieve its scientific goals.

Left Out of the Equation

Most organizations favor scientific over execution-protocol quality. At the base of that disconnect is the lack of any true ownership for ensuring optimal execution of trials. As a result, the clinical trial process rarely runs smoothly.

R&D's scientific experts clearly own a protocol's scientific quality. They have the responsibility, the expertise, the authority, and the accountability for creating a scientifically high-quality protocol. However, the scientific experts usually hand off the protocol to operations experts for execution.

It would be natural to assume that the operations experts own execution quality, but that's not the case. While operations experts have responsibility and expertise for creating execution quality, they generally only gain authority after the protocol is finalized. By then, the crucial execution-quality decisions have been made already.

Rarely do scientific experts bring operations experts in as partners during the development of the protocol. That happens for several reasons. Often, the scientific and operational experts are siloed into medical and operations departments, and the "throw it over the transom" mentality presides in which one group independently develops the protocol and then simply hands it off to the second group to execute. Team-based organizations attempt to address this gap by creating a single, unified team. But execution experts remain left out of the protocol-development process because their position often is seen as subordinate to the scientific experts when it comes to protocol decisions. Even in a team environment, scientific experts' opinions receive undue weight, thereby rendering team-based decision-making ineffective.

Furthermore, neither the scientific nor the execution experts have significant accountability for execution-quality failures. If a protocol runs late or over budget, management typically assumes external or other unknowable factors were to blame and no one is held responsible. And if the drug is successfully launched, virtually all execution problems are forgiven. The result is that poor protocol quality never gets resolved. Enrollment and site problems crop up again and again but the root cause – poor execution quality – is never addressed and a huge amount of waste is allowed to occur during the protocol-execution phase.

Steps to Success

There are four steps companies can take to bring scientific- and execution-protocol quality into balance.

1. Make the two types of protocol quality a management priority. R&D management tends to focus on big picture issues and schedules, while the actual protocol quality, execution times, and budgets are delegated to lower level groups in the organization. These lower-level groups, such as the clinical teams or medical management and clinical operations groups, often have conflicting priorities and goals that prevent them from creating the balance between scientific and execution quality. For example, one group may develop milestone dates, but a different group is responsible for the quality standards associated with that date.

Because of their exclusively big-picture perspective, management tends to assume that schedule and budget problems are an inevitable consequence of the development process. But trial budgets, schedules, and quality can be successfully managed, if measured. Senior executives should measure scientific quality, execution quality, and the success of each protocol. They also should set goals for execution quality and then insist on trial-over-trial improvement in meeting those goals. Management may insist, for example, that the organization's clinical trials move from 90% late to 50% late in the first year, followed by 25% late in the second.

2. Put one person in charge. Across the industry, no single R&D group has ownership for the end-to-end success of the clinical trial protocol. The only sure-fire way to solve this problem is to invest authority, responsibility, and accountability for protocol success in a single individual or group.

Many pharmas have attempted to do this, most notably by creating a protocol manager or director position. However, history has shown that while this person ends up with responsibility for protocol success, he or she never has true authority to compel the compromises needed to solve conflicting scientific- and execution-quality requirements. The result is that these managers are perceived as ineffective, when in reality, they never had enough authority to be successful in the first place.

To solve the problem, companies need to appoint an individual that sits above both medical and operations experts, and who has clear responsibility, accountability, and authority, as well as unwavering senior-management backing. This person ensures that execution quality concerns receive the same up-front attention as is afforded to scientific quality currently.

3. Create accountability, reward success. Typically, R&D management takes an authoritarian approach to the execution of clinical trials by dictating a schedule and budget. They set out schedules by saying things such as, "This trial needs to be completed in 18 months for $10 million." These budgets and timelines may fit nicely into management's overall product-launch scenario, but they often appear arbitrary and unrealistic to the folks who have to write and execute the protocol. In this way, they force the team to make unrealistic promises, skip crucial planning and review steps, and ultimately compromise execution quality.

A better solution is to get senior management to (1) negotiate a viable clinical trial schedule and budget with the team that must write and execute the protocol, (2) hold that team accountable for achieving the schedule and budget, (3) provide executive support when hard decisions and compromises must be made, and (4) reward their success.

Finally, it is critical to reward those teams that bring their trials to conclusion within their agreed-upon schedules and budgets, rather than rewarding those teams that miss their targets but then exert Herculean efforts to resolve the problems.

4. Mandate the use of new tools and processes. The typical protocol has a scientific review and a cursory enrollment feasibility analysis, but rarely anything more. Instead, the protocol teams need top-notch tools to obtain the most cutting-edge scientific and execution quality. In particular—if execution quality is really to be emphasized— management must supply more sophisticated tools and mandate that they be used via a systematic development and review process.

Some of the tools that can be particularly effective at increasing execution quality include:

» Bayesian enrollment analysis conducted at the protocol synopsis stage to ensure an accurate enrollment forecast and optimized inclusion/exclusion criteria (this helps avoid amending the inclusion/exclusion criteria later and eliminates non-enrolling sites)

» Walk-through of the protocol at the penultimate draft stage to ensure that visit schedules, procedures, and inclusion/exclusion criteria are in line with execution-quality requirements (this helps to ferret out execution problems that will cause data problems and amendments later)

» Detailed, systematic, objective reading of the final protocol to remove inconsistencies, conflicts, and errors (this keeps investigators from getting confused and making protocol errors)

» Protocol-quality checklist that provides the team with criteria for judging the quality of the protocol as they develop it (this helps the team incorporate lessons learned from previous protocols).

In the face of time, budget, and staffing constraints, it is often convenient to skip these steps. However, there is ample evidence that skipping this up-front work results is scads of preventable errors that later end up costing much more in schedule, budget, and data quality.

The Way Forward

Shifting R&D to a more balanced focus on both scientific and execution quality will not be easy. Despite the rising costs and increasing timeframes of clinical trials, most R&D organizations seem unable to implement any fundamental, systematic improvements to protocol development and execution. Rather, they continue to make changes at the margins, where resistance to change—and positive impact on the bottom line—are minimal.

Management should expect serious resistance to these efforts. Every argument from "the tools aren't useful in my situation" to "my trial is particularly difficult and therefore shouldn't be measured" or "the schedule and budget were mandated by others" will be put up in defense of poor performance. But by remaining steadfast, management will convey their resolve, and, eventually, the performance measures will start to show improvement.

Taking aggressive steps to create ownership within the R&D organization of both scientific- and execution- protocol quality will yield significant increases in R&D value, from decreased costs per trial and increased staff efficiency to increased speed to market and increased portfolio integrity and flexibility.

David S. Zuckerman is president of Customized Improvement Strategies. He can be reached at dave@rx-business.com.