Exposing bad science in medical research publishing

Ben Goldacre is fighting for greater openness and review of research results, including those published at the highest level.

In his latest research project, Goldacre and his colleagues have investigated how five renowned medical journals fail to publish critical reviews of research articles.

This is the first empirical study that addresses the willingness of large academic journals to publish comparative and objective corrections of errors in high-impact articles.

Science is about accuracy

You can hear Ben Goldacre live on stage during the The Big Challenge science festival in Trondheim in June.

“Science is supposed to be about accuracy. Academic journals are supposed to be the gatekeepers,” Goldacre writes on his Facebook profile, where he shares his latest research article.

“We decided to test whether they’re doing this job properly. We checked every clinical trial published in the world’s five biggest medical journals. Almost every trial had engaged in “outcome switching,” despite the journal claiming to follow guidelines to police this,” Goldacre says.

The researchers sent a correction letter to the respective journal on every misreported trial. What happened next?

“Most of our letters were rejected. In return, we received a fascinating and unexpected stream of false claims and ad hominem attacks from editors. […T]he journals’ responses showed that – in many cases – editors don’t understand the importance of correct outcome reporting, or how to implement the very rules they’ve endorsed” in the Consolidated Standards of Reporting Trials (CONSORT).

• You may also like to read a summary of the scientific article in BMC Medicine: COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time.

Tom Nilsen, professor of epidemiology at NTNU’s Department of Public Health and Nursing, believes the correction letters are an important initiative.

“Openness about the research and publishing process is essential in order to be able to trust the results and for them to be verified by others,” he says.

Rock star in critical research journalism

Goldacre is referred to as a rock star in evidence-based research journalism. As a doctor, researcher, book author and active social commentator, he clearly states that all medical research must be made publicly available so that “undesirable” research results cannot be hidden.

He has described himself as a crass critic of the unscientific dissemination of health and medicine in the media. During the eight years he wrote the Bad Science column in The Guardian newspaper, he reached a lot of readers. In 2008, he published the book Bad Science, and secured a large international audience with it as well through its translation into over 30 languages.

You can hear and see Goldacre in his 2011 TED Talk:

Bought by the pharmaceutical industry

In 2012, Goldacre put the powerful pharmaceutical industry under the microscope in his book Bad Pharma.

He reported how some of the major pharmaceutical companies, instead of basing medicine on real science, produce their own “research results” and mislead patients by exaggerating the benefits of the products they sell.

Goldacre also suggests that health workers and patient groups have been bought by the pharmaceutical industry. The result is that patients do not get the treatment they need, which can have fatal consequences.

Publishing practices under scrutiny

In his latest research article, Goldacre and his colleagues scrutinize renowned scientific medical journals such as The Lancet to examine their publishing practices.

These journals are regarded as upholding a high standard of research integrity and thoroughly evaluating the trial results to be published. Goldacre and his colleagues show this to be far from the case.

Their study has revealed many fundamental shortcomings in the actual research – and in the publishing practices that have failed to reveal the deficiencies. The study also found major differences between various journals’ publishing practices.

The background for the study, writes Goldacre, is that “[d]iscrepancies between pre-specified and reported outcomes are an important source of bias in trials.” Goldacre is the first author of the research article.

He notes that outcome misreporting continues to occur despite legislation, guidelines and journals’ public commitments to correct reporting.

“We aimed to document the extent of misreporting, establish whether it was possible to publish correction letters on all misreported trials as they were published, and monitor responses from editors and trialists to understand why outcome misreporting persists despite public commitments to address it,” says Goldacre.

Not following the gold standard

The five scientific journals evaluated by the researchers have committed to abide by the Consolidated Standards of Reporting Trials (CONSORT), considered the “gold standard” for research publication.

These guidelines are designed to improve the reporting of various types of health research, and to improve the quality of research used in healthcare decision-making processes.

The medical journals investigated by the researchers were The Lancet, New England Journal of Medicine, Journal of the American Medical Association, British Medical Journal and Annals of Internal Medicine.

All revealed extensive breaches of the CONSORT guidelines, and most declined to publish correction letters that documented deficiencies.

The researchers compared published research results in the five journals against different protocols and registry entries. Where they found misreported research trials in the published articles, they sent correction letters to the medical journal that had published the article.

Of the 67 scientific articles assessed, 53 showed discrepancies that required correction letters, according to the CONSORT guidelines. Only 23 of the correction letters submitted by the researchers were published.

The journals that did publish correction letters took an average of just over three months to do it. The slowest journal took more than eight months before publishing the correction letter.

Qualitative analysis showed extensive misunderstandings among the editors about correct result reporting and about the CONSORT standard.

Findings disturbing, but not unexpected

“The CONSORT guidelines for reporting experimental trials, and comparable criteria for other study designs, were developed precisely to make research more transparent and verifiable,” says Professor Tom Nilsen of NTNU’s Department of Public Health and Nursing.

All quantitative research risks generating some results that are due to statistical coincidences. In order to avoid putting too much emphasis on such random findings, the CONSORT guidelines state that researchers have to report in advance what they want to study. The final research report then needs to be in accordance with the stated objectives.

Nilsen says it is disturbing that Goldacre and his colleagues found that a majority of the studies violate this guideline, but not unexpected. The news value of the research results is often driven by how strong the effects are and whether these can be described as statistically significant.

“Researchers and scientific journals both tend to focus excessively on new significant relationships, even when chances are good that they don’t reflect real effects that can be replicated in other studies,” says Nilsen.

“The prestige of both the researchers and the journals is at stake, and so they end up not observing the reporting guidelines. Hopefully, the Goldacre study will encourage researchers and journal editors to do better at following best practice guidelines and to present research results that can be trusted,” Nilsen adds.

These issues have considerable relevance beyond the medical and health research field as well.

Other recent initiatives have also addressed challenges related to reporting research results, including a letter in the journal Nature signed by more than 800 researchers.

Want all data to be shared openly

Goldacre and his colleagues are recommending changes and improvements in the way scientists and journals correspond. They would like to see:

• greater insight into how peer reviews are conducted
• changes in how CONSORT guidelines are enforced
• new strategies for research methods and reporting

“We discuss the benefits of a potential research methodology that shares all data openly and proactively in real time, such as feedback on critically reviewed studies,” the researchers write in the BMC Medicine article.

The AllTrials.net initiative, led by Goldacre, was created to register all clinical trials being conducted and their findings. The goal is for all research to become accessible and transparent.

The AllTrials petition has been signed by nearly 100 000 people and by 748 organizations.

Goldacre heads the EBM DataLab – Evidence-Based Medicine DataLab – at the University of Oxford. Researchers at EBM DataLab are developing innovative tools to make research data more accessible.

Geir Wenberg Jacobsen, professor emeritus at NTNU’s Department of Public Health and Nursing, says the study addresses a very relevant topic that should be probed more closely.

“These issues have considerable relevance beyond the medical and health research field as well,” he says.