Replication is a foundational concept in science. Researchers take an older study that they want to test and then try to reproduce it to see if the findings hold up.
Testing, validating, retesting — it's all part of a slow and grinding process to arrive at some semblance of scientific truth. But this doesn't happen as often as it should, our respondents said. Scientists face few incentives to engage in the slog of replication. And even when they attempt to replicate a study, they often find they can’t do so. Increasingly it’s being called a "crisis of irreproducibility."
The stats bear this out: A 2015 study looked at 83 highly cited studies that claimed to feature effective psychiatric treatments. Only 16 had ever been successfully replicated. Another 16 were contradicted by follow-up attempts, and 11 were found to have substantially smaller effects the second time around. Meanwhile, nearly half of the studies (40) had never been subject to replication at all.
More recently, a landmark study published in the journal Science demonstrated that only a fraction of recent findings in top psychology journals could be replicated. This is happening in other fields too, says Ivan Oransky, one of the founders of the blog Retraction Watch, which tracks scientific retractions.
As for the underlying causes, our survey respondents pointed to a couple of problems. First, scientists have very few incentives to even try replication. Jon-Patrick Allem, a social scientist at the Keck School of Medicine of USC, noted that funding agencies prefer to support projects that find new information instead of confirming old results.
Journals are also reluctant to publish replication studies unless "they contradict earlier findings or conclusions," Allem writes. The result is to discourage scientists from checking each other's work. "Novel information trumps stronger evidence, which sets the parameters for working scientists."
The second problem is that many studies can be difficult to replicate. Sometimes their methods are too opaque. Sometimes the original studies had too few participants to produce a replicable answer. And sometimes, as we saw in the previous section, the study is simply poorly designed or outright wrong.
Again, this goes back to incentives: When researchers have to publish frequently and chase positive results, there’s less time to conduct high-quality studies with well-articulated methods.
Belluz, J., Plumer B., & Resnick (2016). The 7 biggest problems facing science, according to 270 scientists. Retrieved from https://www.vox.com/2016/7/14/12016710/science-challeges-research-funding-peer-review-process
