US effect results in scientific exaggeration

Scientists who study human behaviour are more likely than average to report exaggerated or eye-catching results if they are based in the United States, according to an analysis of more than 1,000 research papers in psychiatry and genetics. This bias could be due to the research culture in the US, authors of the analysis said, which tends to preferentially reward scientists for the novelty and immediate impact of a piece of work over the quality or its long-term contribution to the field. Alok Jha reports.

Scientists who study human behaviour are more likely than average to report exaggerated or eye-catching results if they are based in the United States, according to an analysis of more than 1,000 research papers in psychiatry and genetics. This bias could be due to the research culture in the US, authors of the analysis said, which tends to preferentially reward scientists for the novelty and immediate impact of a piece of work over the quality or its long-term contribution to the field.

HT Image

Daniele Fanelli, University of Edinburgh, one of the authors of the latest analysis, said that there was intense competition in the US for research funds and, subsequently, pressure to report novel findings in prestigious, high-impact scientific journals.

“We don’t know what causes the US effect but we think the most likely explanation is that it’s about the research environment in the US,” he says. “Somehow the researchers there are subtly more pressured than elsewhere in the world to make strong discoveries. This very idea that you do science to make strong discoveries is natural but it’s a problem to science itself. Science should be about doing good, precise studies. Not necessarily about getting exciting new results every time.”

Working in a research environment where careers depend on publishing the most exciting and strongest results might unconsciously draw researchers to exaggerate their findings, he added. “The problem, if we’re right about the US effect, is that the US itself should re-think the way they are rewarding researchers. They shouldn’t reward researchers only because they get a lot of papers in a lot of high-ranking journals. They should reward research that is methodologically highly accurate.”

Fanelli worked with John Ioannidis of Stanford University on the study. They looked at 1,172 results reported on in 82 meta-analyses on a range of biological and behavioural questions in the fields of genetics and psychiatry. Each meta-analysis took the results of 10 to 20 individual research studies that examined the same question, such as whether a particular drug had an effect on a patient in the expected way. For each study, Fanelli and Ioannidis recorded how far the strength of the reported effect was, from the average across many studies, for that question.

They found that behavoural studies were more likely to report more extreme results than studies that involved biological or chemical measurements in a person. Behavioural studies with lead scientists in the US were also more likely than average to report extreme effects that confirmed the initial hypotheses of the studies. The results are published today in the Proceedings of the National Academy of Sciences.

The “US effect”, as the authors called it, is not huge. When studying the strength of a particular effect – the efficacy of a drug, say, or a type of psychiatric intervention – studies with US-based first authors tended to have a 5% stronger effect than the average across all countries. GNS