Too Many Drug Trials Stopped Early

'Benefits' from these highly publicized treatments are often fleeting, study finds

Please note: This article was published more than one year ago. The facts and conclusions presented may have since changed and may no longer be accurate. And "More information" links may no longer work. Questions about personal health should always be referred to a physician or other health care professional.

En Español

By E.J. Mundell
HealthDay Reporter

TUESDAY, Nov. 1, 2005 (HealthDay News) -- It's a common scenario in the world of medical research: Investigators announce in a major journal that the drug they've been testing has proven so effective they've stopped the trial early, to provide the drug's benefits to all patients involved.

This type of headline-making "breakthrough" means publicity for everyone involved: the research team, the drug, its manufacturer and even the journal in which the study appears.

However, new research published in the Nov. 2 issue of the Journal of the American Medical Association suggests that, in most cases, the purported "benefit" to patients is exaggerated or evaporates over the long term.

"Even when things are done right, stopped-early trials run a risk of overestimating treatment effects substantially, and that risk gets even greater when things aren't done right," said study author Dr. Gordon Guyatt, a professor of medicine at McMaster University in Hamilton, Ontario, in Canada.

Guyatt led an international team of independent experts who reviewed 143 clinical trials published over the past 15 years in "high-impact" medical journals. All of these randomized, controlled trials were stopped early because of an announced treatment benefit to patients.

And without more background information, the average physician -- and patient -- takes such dramatic trial results at face value, often changing their treatment patterns accordingly, the team pointed out.

But in their own close look, the reviewers concluded that, all too often, the trials were terminated far too soon, before they gathered the statistical power to determine if the "benefit" was real, or just a short-lived statistical fluke.

And Guyatt said the problem is growing: His team found the percentage of stopped-early trials published in major journals has more than doubled over the past 15 years -- from 0.5 percent in 1990 to 1.2 percent by 2004.

"This paper is a real wake-up call to all the groups involved to say 'Look, the way things are going, we've made a mistake,'" Guyatt said.

His team of Canadian, U.S. and European experts took a hard look at data on the 143 stopped-early studies -- typically industry-funded drug trials focused on common, deadly conditions such as heart disease, cancer and HIV/AIDS. Most were published in respected, high-profile journals, including the New England Journal of Medicine (55 studies), The Lancet (27 studies), and Journal of the American Medical Association (6 studies).

Terminating a trial before its planned end date usually meant ending the participant recruiting process early, since most trials continue to recruit new participants even after they are up and running.

In fact, the trials had recruited an average 63 percent of their planned cohort by the time they were stopped, the researchers report. And while many trials are designed to collect data for years, those stopped early because of good results did so at a median of just 13 months.

In almost all cases, researchers publishing their results failed to provide (and journal editors failed to insist on) a detailed, statistical justification for prematurely terminating the trial.

"That's part of the purpose of this report, to encourage people to get stricter standards for reporting on why they are stopping early," Guyatt said.

In too many cases, he added, the number of "events" -- endpoints such as death, heart attack, stroke or other important markers -- that went into the stop-early decision were too few to draw any firm conclusion that the medication would continue to provide benefit over the longer term.

"When sample sizes are small, and the number of events are low, we run a high risk of substantially overestimating the magnitude of the effect, and sometimes, in the extreme, there's actually no effect at all," Guyatt said.

His team cited a number of examples, including one trial comparing the drug bisoprolol for heart patients undergoing vascular surgery.

That trial was stopped early, at a time when only 112 of the planned 266 patients had been recruited. Researchers announced that those patients taking bisoprolol had experienced a 91 percent reduction in death or heart attack. However, Guyatt and his fellow reviewers noted that a benefit this extreme, so soon, "is likely too good to be true."

That turned out to be the case: After the fanfare surrounding the trial died down, subsequent, longer-term studies found the overwhelming "benefit" to be no more than a short-term statistical fluke, something researchers call a "random high."

Exercising caution before stopping trials early is important, "because we need to get the right answers," stressed Stuart Pocock, a professor of medical statistics at the London School of Hygiene and Tropical Medicine, and the author of a related commentary on the study.

Longer-term studies also give time for drug side effects to emerge, "and not just side effects, but what the long-term merit to the treatment might be, even its efficacy. Because [in stopping trials early] you're only looking at short-term follow up," he added.

Why the steady increase in trials stopped early over the past decade? According to Guyatt, under the current system, everyone -- except, perhaps, the public -- stands to benefit.

While pharmaceutical companies may be sincere in their belief that patients will benefit from stopping a trial early, Guyatt said, "if you stop early and get big treatment results, it [also] improves the likelihood of marketing your drug."

Clinical trials are also expensive, so stopping them early can save drug makers millions.

Journals publishing "breakthrough" results stand to gain, too. "One of the ways these journals compete is through getting into the press -- JAMA or New England Journal of Medicine get into the New York Times, and that's a big success for them," Guyatt said.

But Pocock believes more benign motives may be at play. "The quality of trial design has improved over the years," he said. "One aspect is that we often make them bigger than we used to. The bigger the trial, the more opportunity there is to see results halfway through, and to stop. But of course [those results] are open to misinterpretation, too."

Whatever the reasons, both experts agreed that researchers, pharmaceutical companies, journal editors and the independent review boards that typically oversee major drug trials need to put the brakes on early stoppage.

The dilemma for everyone involved is real: If a drug is of substantial, sometimes lifesaving benefit, is it ethical to withhold it from those patients in the trial who are getting an older drug or a placebo?

Lasting benefit can be tough to prove in the short term, however. Based on the findings, Guyatt and Pocock contend that, in most cases, only time -- and responsible oversight by everyone involved -- will distinguish a "random high" from something that's truly valuable to patients over the long term.

"On the one hand, we have to balance the best interests of patients in the trial, here and now, and the longer-term judgment on what's going to be the right thing to do in the long run, to get the right answer for future patients," Pocock said. "That's what we're juggling with."

More information

To learn more about how clinical trials are conducted, head to the U.S. National Institutes of Health.

SOURCES: Gordon Guyatt, M.D., professor, medicine, McMaster University, Hamilton, Ontario; Stuart Pocock, Ph.D., professor, medical statistics, London School of Hygiene and Tropical Medicine, U.K.; Nov. 2, 2005, Journal of the American Medical Association

Last Updated: