Drug researchers hide thousands of negative studies. A new campaign is trying to change that.
Updated by Julia Belluz on January 14, 2015
@juliaoftoronto julia.belluz@voxmedia.com
http://www.vox.com/2015/1/14/7546569/clinical-trials-transparency
Imagine making a decision about whether or not to buy a new home based on seeing only half the property. The real estate agent invites you in to look around — the bedrooms, the kitchen. But you’re not allowed to visit the dining area, and the basement is off limits.
Do you buy the house? The choice may seem more like an uninformed guess. And that’s essentially how doctors are made to operate today when it comes to decisions about which medical interventions (drugs, tests, procedures) to use on us.
In medicine, clinical trials inform the choices health professionals and regulators make about the safety and effectiveness of drugs and devices. An estimated half of these studies are never published, and those with negative or unpromising results are more likely to remain hidden from view (a phenomenon called 'publication bias.')
This unpublished data may contain information on patient harms or side effects, clues to failed studies or unpromising drugs that shouldn’t be re-tested, and information that independent researchers could use to vet the quality of research findings.
WE WERE READY FOR A TIPPING POINT EVENT Without this data, doctors are left to act like realtors selling houses they themselves have hardly seen — in decisions that, quite literally, are life or death.
In recent years, there has been a movement to fix this broken system: to make sure the results of studies don’t go missing and that all clinical trials are registered in public databases (such as America's ClinicalTrials.gov) for the world to see. Today, the transparency agenda might be nudged a little further with a new report out of the Institute of Medicine.
The report — "Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risks" — basically says the status quo is now unacceptable. In it, a group of stakeholders including doctors, journal editors, and patient groups write that data sharing needs to be "the expected norm" in medical research, and they outline what clinical trials information should be made transparent and when.
"We were ready for a tipping point event," said Deborah Zarin, director of ClinicalTrials.gov. "This is a statement that the expectations have changed: if you're a researcher or a research sponsor, there's going to be an expectation that you're going to make data available at the end of your study."
New recommendations about "patient-level" data
In the report, the IOM said that instead of just designing and planning a study, scientists need to plan and document how they're going to share the data from that study so that its usable to others who may want to re-analyze it.
One of the most provocative recommendations included a push for sharing "participant-level" data. Right now, only the summaries of results from studies are typically communicated and registered to clinical trials databases.
But, as one New England Journal of Medicine editorial notes, a lot of information can get lost when researchers apply statistical methods to bring together and analyze all the quirks and individual findings from patients. Different methods for summarizing data can also lead to very different conclusions.
For example, an MRI scan on a patient needs to be read and interpreted by a doctor; notes from the emergency department inform whether a trial participant actually had a heart attack. When independent researchers look at these scans and notes, they may come to very different conclusions.
Right now, researchers usually lock this kind of data away at the end of a study. But the IOM says this needs to change. They recommend that six months after the publication of trial results (or 18 months after trial completion), researchers share the raw data about study participants through open databases.
The IOM also recommends that:
Data sharing becomes an essential step in running a clinical trial: As soon as researchers register a trial and before they enroll the first patient, the people running the trial have a plan in place for which data will be shared and when.
When a study is completed, within a year, scientists should share a summary of their results, including any adverse events.
Within 18 months of a trial's end, scientists should share the full data sets from the clinical trial (ie. the participant-level data).
Data sharing initiatives should be transparent and overseen by independent panels of experts and the public.
The slow movement toward clinical trials data sharing
The IOM's suggestions are in line with other initiatives to push the clinical trials transparency agenda further.
Health Human Services released a new plan late last year to make ClinicalTrials.gov — the world's largest database of clinical trials, run by the National Institutes of Health — even more comprehensive.
The AllTrials campaign, led by a group of crusading scientists and doctors in Europe, has been pushing for every clinical trial (past, present, and future) around the world to be registered, with their full methods and results reported.
More recently, drug companies including GlaxoSmithKline, Novartis, and Sanofi, created a web portal where independent researchers can request anonymized patient-level data from trials. In this case, the drug companies act as the gatekeepers of the data.
Yale University has been collaborating with industry partners, including Medtronic and Johnson & Johnson, to open up raw data from clinical studies for independent researchers to see.
Because the transparency landscape in medicine is advancing, some think that the IOM group could have been even more provocative.
"I would have liked the report to go further," Harlan Krumholz, a professor of medicine and advocate for data transparency at Yale University, "with respect to shortening the time to sharing, being more inclusive of the types of studies in which data should be shared.
"That being said, several years ago it would have been hard to imagine an IOM report that makes clear the issues we face in the current culture and the need to change."
Peter Doshi, assistant professor at the University of Maryland School of Pharmacy and associate editor at the BMJ, said the IOM should have put more pressure on the Food and Drug Administration to release the data they have from the drug approvals process.
THE NEW RECOMMENDATIONS ARE NOT ON THE LEADING EDGE OF OPEN SCIENCE
"We know they hold more data than anybody else, for medicines people are using today," he said. Their European counterpart — the European Medicines Agency — has already agreed to publish its data for independent analyses, he added. "The FDA has vastly more data than EMA, and needs to follow suit."
He also noted that there was some vagueness around the definition of "commercial confidential information," and that may leave space for drug companies or others who run trials to hide data.
"We have seen that what constitutes 'commercial confidential' is very subjective and debatable. I wish the IOM had laid down more details here, also clearly saying what potential commercial confidential information also has public health value, and when the public health value must trump concerns of commercial confidentiality."
The road to open data in medicine is going to be a long one. And the IOM is a scholarly non-governmental organization, so it doesn't have any direct legal or policy levers. Still, as Krumholz said, "The IOM's recommendations are not on the leading edge of open science, but it does add momentum to the efforts."
Updated by Julia Belluz on January 14, 2015
@juliaoftoronto julia.belluz@voxmedia.com
http://www.vox.com/2015/1/14/7546569/clinical-trials-transparency
Imagine making a decision about whether or not to buy a new home based on seeing only half the property. The real estate agent invites you in to look around — the bedrooms, the kitchen. But you’re not allowed to visit the dining area, and the basement is off limits.
Do you buy the house? The choice may seem more like an uninformed guess. And that’s essentially how doctors are made to operate today when it comes to decisions about which medical interventions (drugs, tests, procedures) to use on us.
In medicine, clinical trials inform the choices health professionals and regulators make about the safety and effectiveness of drugs and devices. An estimated half of these studies are never published, and those with negative or unpromising results are more likely to remain hidden from view (a phenomenon called 'publication bias.')
This unpublished data may contain information on patient harms or side effects, clues to failed studies or unpromising drugs that shouldn’t be re-tested, and information that independent researchers could use to vet the quality of research findings.
WE WERE READY FOR A TIPPING POINT EVENT Without this data, doctors are left to act like realtors selling houses they themselves have hardly seen — in decisions that, quite literally, are life or death.
In recent years, there has been a movement to fix this broken system: to make sure the results of studies don’t go missing and that all clinical trials are registered in public databases (such as America's ClinicalTrials.gov) for the world to see. Today, the transparency agenda might be nudged a little further with a new report out of the Institute of Medicine.
The report — "Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risks" — basically says the status quo is now unacceptable. In it, a group of stakeholders including doctors, journal editors, and patient groups write that data sharing needs to be "the expected norm" in medical research, and they outline what clinical trials information should be made transparent and when.
"We were ready for a tipping point event," said Deborah Zarin, director of ClinicalTrials.gov. "This is a statement that the expectations have changed: if you're a researcher or a research sponsor, there's going to be an expectation that you're going to make data available at the end of your study."
New recommendations about "patient-level" data
In the report, the IOM said that instead of just designing and planning a study, scientists need to plan and document how they're going to share the data from that study so that its usable to others who may want to re-analyze it.
One of the most provocative recommendations included a push for sharing "participant-level" data. Right now, only the summaries of results from studies are typically communicated and registered to clinical trials databases.
But, as one New England Journal of Medicine editorial notes, a lot of information can get lost when researchers apply statistical methods to bring together and analyze all the quirks and individual findings from patients. Different methods for summarizing data can also lead to very different conclusions.
For example, an MRI scan on a patient needs to be read and interpreted by a doctor; notes from the emergency department inform whether a trial participant actually had a heart attack. When independent researchers look at these scans and notes, they may come to very different conclusions.
Right now, researchers usually lock this kind of data away at the end of a study. But the IOM says this needs to change. They recommend that six months after the publication of trial results (or 18 months after trial completion), researchers share the raw data about study participants through open databases.
The IOM also recommends that:
Data sharing becomes an essential step in running a clinical trial: As soon as researchers register a trial and before they enroll the first patient, the people running the trial have a plan in place for which data will be shared and when.
When a study is completed, within a year, scientists should share a summary of their results, including any adverse events.
Within 18 months of a trial's end, scientists should share the full data sets from the clinical trial (ie. the participant-level data).
Data sharing initiatives should be transparent and overseen by independent panels of experts and the public.
The slow movement toward clinical trials data sharing
The IOM's suggestions are in line with other initiatives to push the clinical trials transparency agenda further.
Health Human Services released a new plan late last year to make ClinicalTrials.gov — the world's largest database of clinical trials, run by the National Institutes of Health — even more comprehensive.
The AllTrials campaign, led by a group of crusading scientists and doctors in Europe, has been pushing for every clinical trial (past, present, and future) around the world to be registered, with their full methods and results reported.
More recently, drug companies including GlaxoSmithKline, Novartis, and Sanofi, created a web portal where independent researchers can request anonymized patient-level data from trials. In this case, the drug companies act as the gatekeepers of the data.
Yale University has been collaborating with industry partners, including Medtronic and Johnson & Johnson, to open up raw data from clinical studies for independent researchers to see.
Because the transparency landscape in medicine is advancing, some think that the IOM group could have been even more provocative.
"I would have liked the report to go further," Harlan Krumholz, a professor of medicine and advocate for data transparency at Yale University, "with respect to shortening the time to sharing, being more inclusive of the types of studies in which data should be shared.
"That being said, several years ago it would have been hard to imagine an IOM report that makes clear the issues we face in the current culture and the need to change."
Peter Doshi, assistant professor at the University of Maryland School of Pharmacy and associate editor at the BMJ, said the IOM should have put more pressure on the Food and Drug Administration to release the data they have from the drug approvals process.
THE NEW RECOMMENDATIONS ARE NOT ON THE LEADING EDGE OF OPEN SCIENCE
"We know they hold more data than anybody else, for medicines people are using today," he said. Their European counterpart — the European Medicines Agency — has already agreed to publish its data for independent analyses, he added. "The FDA has vastly more data than EMA, and needs to follow suit."
He also noted that there was some vagueness around the definition of "commercial confidential information," and that may leave space for drug companies or others who run trials to hide data.
"We have seen that what constitutes 'commercial confidential' is very subjective and debatable. I wish the IOM had laid down more details here, also clearly saying what potential commercial confidential information also has public health value, and when the public health value must trump concerns of commercial confidentiality."
The road to open data in medicine is going to be a long one. And the IOM is a scholarly non-governmental organization, so it doesn't have any direct legal or policy levers. Still, as Krumholz said, "The IOM's recommendations are not on the leading edge of open science, but it does add momentum to the efforts."