There is a credible case to be made that most published research is false. This notion was popularized in a 2005 essay by Stanford epidemiologist John Ioannidis titled “Why Most Published Research Findings Are False.” [1] As of this writing, the article has been viewed 2,833,883 times and cited 3,271 times.
Several reasons have been identified for why this might be true such as data dredging [2], P-hacking [3], or basic researcher biases. [4] There is an abundant array of articles summarizing the problem in detail [5] [6] [7] [8], but suffice it to say the problem is real and has consequences in all areas of science and medicine.
The seriousness and scope of the problem can be illustrated by one example. The Biotechnology firm, Amgen, tried to reproduce the results of 53 landmark studies to validate pursuing a particular line of research. In the end, they could reproduce the results of the originally published work in only 6 cases. That meant 47 out of 53 were irreproducible. [9] This finding is not an outlier – other researchers have struggled to reproduce the results of previous studies.
The purpose of this article is not to relitigate the case for saying most research is false. It’s really to consider how that might impact the medical care you receive and what you choose to believe as true.
What does that mean for you?
I recently wrote about the high rate of medical reversals (medical practices that were adopted based on poor evidence and replaced by a practice with better evidence) and the observation that many physicians continue to recommend treatments that have been shown to be ineffective. [10] If most published research is false and your own doctor may be recommending treatments based not based on verified evidence, how does one decide what treatment is right for them? It’s not reasonable to think that every person has the knowledge, the inclination, or the time to evaluate every breakthrough treatment that appears in the press – or is recommended by your physician.
The emotion quotient
If we assume most people will not make evidence-based decisions and that most research is false anyway, then what’s left in the decision-making process? There are two things left – one leading to the other: emotion and faith. Anyone who has read this blog knows by now that it’s my position that humans tend to make decisions based primarily on emotions – not facts. [11] It’s the process Jonathon Haidt calls motivated reasoning. [12]
If people are unencumbered by the need for considering facts, they are free to pursue any treatment, concept, or world view that is consistent with what their “gut” is telling them. In the area of improving one’s health, that can lead down the rabbit hole of pursuing complementary or alternative medicine (CAM) treatments.
If you’re incapable of evaluating the evidence, science shows that most research is wrong, and your doc is still using flawed treatments, then taking a CAM approach means you can be no worse off. Plus, you don’t have to think -it’s a matter of faith. That’s why CAM treatments and “medications” remain a flourishing industry.
It’s not logical (at least to me) that a ground-up duck heart diluted to nothingness could possibly be an effective treatment for flu symptoms. And yet, Oscillococcinum is still on the shelves at CVS and Walgreens. [13] People buy it not because it’s been shown to be effective. They buy it because they believe it works, or maybe a friend told them “it works for me.” How do you compete with the “it works for me” mentality? I’m not sure you really can. People are fundamentally emotion-driven beings.
The least-worse approach
The best option is to adopt the least-worse decision-making process. First, that means tossing out anything that doesn’t have some credible evidence. A good place to start is FDA approval. To my knowledge, the FDA hasn’t approved any ground up fowl innards as an effective therapy for anything. The next step is to view any new discovery in the news as preliminary. As an exercise, one can look at the news-making discovery and simply check the source of the research, check the sample size, and verify the subjects were human. A study of ten rats responding to a compound may very well make the news, but it’s a long, long way from being shown to be a meaningful therapy for humans.
Finally, be an informed, critically-thinking patient. Ask your physician questions. Asking why or how a treatment works is always a fair and reasonable question. If your doc declines to answer, then you can add finding a new doc to your list of actions.
[1] Why Most Published Research Findings Are False, https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
[2] Data dredging, bias, or confounding- They can all get you into the BMJ and the Friday papers https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1124898/
[3] An investigation of the false discovery rate and the misinterpretation of p-values https://royalsocietypublishing.org/doi/full/10.1098/rsos.140216
[4] Identifying and Avoiding Bias in Research https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2917255/
[5] How science goes wrong https://www.economist.com/leaders/2013/10/21/how-science-goes-wrong (PDF)
[6] The Replication Myth: Shedding Light on One of Science’s Dirty Little Secrets https://blogs.scientificamerican.com/guest-blog/the-replication-myth-shedding-light-on-one-of-sciencee28099s-dirty-little-secrets/
[7] A summary of the evidence that most published research is false https://simplystatistics.org/2013/12/16/a-summary-of-the-evidence-that-most-published-research-is-false/
[8] Believe It Or Not, Most Published Research Findings Are Probably False https://bigthink.com/neurobonkers/believe-it-or-not-most-published-research-findings-are-probably-false
[9] Raise standards for preclinical cancer research https://www.nature.com/articles/483531a
[10] How medical reversals impact your health care. https://confrontingmediocrity.net/2019/09/14/how-medical-reversals-impact-your-health-care/
[11] Belief-based thinking: Irrational fear of GMOs and Donald Trump as President. https://confrontingmediocrity.net/2017/02/20/belief-based-thinking-irrational-fear-of-gmos-and-donald-trump-as-president/
[12] The Righteous Mind http://righteousmind.com/
[13] A trip down the rabbit hole of homeopathy https://confrontingmediocrity.net/2012/01/29/a-trip-down-the-rabbit-hole-of-homeopathy/
Confronting Mediocrity 