Your grandma bakes the best chocolate chip cookies, so when your friend asks for a good recipe to impress her new neighbors, you happily pass hers along. Later, your friend thanks you with a few of the cookies they baked, but they just don't taste right. You bake them yourself for a movie night, but they're still not as good as you remember. Determined, you bake them again, and again, but they still aren't as good as your grandma's batch. Are Grandma's cookies just a fluke? Are you forgetting a detail? What's different—is it the flour, the oven temperature, or the bake time?

This is what the replication crisis looks like—only in baking science. A study makes a bold claim, people believe it's true, and then when others try to replicate the findings, they can't. Whether it's a perfect cookie or a celebrated psychological theory, the replication crisis has made us question which “recipes” in science can truly be trusted. Back in the 2010s, it felt like a true crisis, but since then, it has also sparked positive change. Researchers are now more transparent about their practices, sharing detailed documentation of their data and analytical methods. Practices like preregistration and using larger sample sizes have become common requirements for experiments to be published and taken seriously.¹ 

Still, the replication crisis has led to an erosion of public trust. Many people began to doubt the reliability of scientific findings, undermining confidence in experts—an issue that continues to grow today. To ensure that research advances knowledge rather than fueling skepticism and causing polarization, the scientific community is working to prevent future replication challenges through greater transparency and accountability.¹ 

Peer Review: A process in which experts in the same field evaluate a research study before it is published to ensure quality, accuracy, and validity. This systematic review helps with quality control for scientific works. 

P-Hacking: Manipulating data analysis until statistically significant results appear, often by trying multiple tests or selective reporting of findings, which increases false positives. 

Publication Bias: The tendency for studies with significant or positive results to be published more often than studies with null or negative results, disturbing the balance of findings. 

Preregistration: The practice of publicly documenting a study's hypotheses, methods, and analysis plan before data collection to increase transparency and prevent p-hacking and HARKing. 

HARKing: “Hypothesizing After Results are Known” involves presenting a post hoc explanation of the results, which can misrepresent the true reasoning behind a study and make the research seem more conclusive or impactful than it truly is.

Black Box: When inputs and outputs are visible, but the internal workings (in this case, methodology) are not transparent or understandable, making outputs hard to assess or replicate. 

Concerns about the reproducibility of studies have existed for decades, but it wasn't until the 2000s that systematic attempts to replicate research began revealing alarming inconsistencies. Many psychological concepts were developed in the mid-to-late 1800s and throughout the 1900s, a time when there were fewer rules and ethical guidelines governing publication.³ Questionable research practices (QRPs) such as p-hacking and selective reporting were rarely discussed, and academic culture placed heavy emphasis on producing as many publications as possible—hence the mantra “publish or perish." New theories were often built on the perceived validity of previous findings, compounding the problem when earlier works proved to be less reliable than originally thought. 

The issue was first officially raised by Greek-American physician John Ioannidis in 2005 with his paper "Why Most Published Research Findings Are False,”⁴ but the crisis reached its breaking point years later in the early 2010s, when suspicions of misconduct led to the retraction of several high-profile studies. Multiple scandals shook the psychology community, like the case of Dutch psychologist Diederik Stapel. Stapel was accused of committing academic fraud after he was found to have fabricated data in dozens of studies, including some of his most famous works, which were even published in one of the world's top academic journals, Science.⁵ Investigators at Tilburg University confirmed that at least 30 of his papers were fraudulent. 

Around the same time, Daryl Bem of Cornell University published a paper in the renowned Journal of Personality and Social Psychology that seemed to demonstrate that humans possess precognition, a type of ESP, after participants appeared to predict the outcomes of random future events. The peer-reviewed publication of Bem’s conclusions on our supposed aptness for “feeling the future” sparked widespread disbelief and intense scrutiny.⁶ Cases like these indicated that reform was necessary in the research community, whether through a culture shift or increased regulation; if sketchy methodology could sneak past review processes undetected, the current system was not working. 

From 2015 onward, the crisis ushered in a reform era that marked the rise of open science practices. The Open Science collaborative, founded by psychologist Brian Nosek, is a movement promoting transparency and accountability in scientific research, which conducted an attempt to replicate 100 psychology studies from top journals. Although 97% of the studies claimed significant results, only 39% of the findings were replicated, confirming widespread concerns.⁷ Organizations like the Center for Open Science and the Psychological Science Accelerator led the push for transparency, supporting new tools like AsPredicted (where researchers preregister their studies) and centralized data repositories that enable open data sharing.^8,9,10

Regardless of reforms, ongoing tension remains. Some argue that the replication crisis has been exaggerated by the media, while others insist that deep structural issues persist.¹ Metascience—the study of how science itself is conducted—has grown into its own subfield, reflecting the profound and lasting impact of the replication crisis on how research is conducted and evaluated.¹¹

John Ioannidis

A Greek-American professor at Stanford University who first theorized the replication crisis in his 2005 paper, “Why Most Published Research Findings Are False.” Focusing primarily on medicine, he argued that clinical research was long overdue for reform and provided the first framework for the replication crisis discussion.¹²

Brian Nosek

An American social psychologist and professor at the University of Virginia who founded the Open Science Center and led the Open Science collaboration project, which provided the first large-scale empirical evidence for the replication crisis. He also co-founded the Society for the Improvement of Psychological Science and Project Implicit.¹³

The replication crisis has shaken confidence in many scientific findings, driving important reforms within the research community. Scientific credibility has taken a hit, but thanks to the combination of more rigorous research standards and a culture shift, we are pushing toward science that is more trustworthy than ever.

Improved research transparency

So, what are some of the practices that have been implemented to boost research credibility? One is open data, where researchers share their raw data to ensure that their findings aren't “black boxes,” thereby making results more transparent and reducing the risk of selective reporting or data fabrication.¹⁴ Another common practice is pre-registering a hypothesis, design, and planned analysis before data collection to prevent p-hacking (when data analysis is manipulated to favor a particular result), or HARKing (hypothesizing after the results are known).¹⁴ Finally, researchers increasingly follow reporting standards that ensure they write up their findings clearly and accurately.¹⁴ For example, CONSORT for randomized controlled trials includes a 25-item checklist requiring detailed descriptions of participants, interventions, and analysis. Another is PRISMA for systematic reviews and meta-analyses, which outlines how different studies were selected and evaluated.¹⁵

From “publish or perish” to trust and verify

The replication crisis has sparked a cultural shift in academia. In the past, researchers often felt pressure to publish flashy, attention-grabbing studies to stay relevant; however, now, careful and replicable science is being increasingly rewarded. Journals are becoming more appreciative of null results, recognizing that studies that didn't find an effect still represent an advancement of knowledge, though, according to survey results, we still struggle to publish them.¹⁶ Some journals, like Nature Human Behavior and eLife, now offer registered reports, which is a format that emphasizes the quality of the research question and methodology by conducting a peer review before data collection.^17,18

The science community is also less likely to readily accept bold claims, especially after high-profile studies such as “power posing” (adopting confident body language to influence feelings or behavior), or certain types of priming (subconsciously activating certain thoughts to influence a person's subsequent response) failed to replicate.^19,20 There is now greater caution in interpreting “groundbreaking” research, and a stronger culture of calling out questionable practices. Funding has also expanded for large-scale replication initiatives that test high-impact findings, such as ManyBabies in developmental psychology, which were rarely supported before 2015.²¹

The replication crisis has sparked heated debates, and with conspiracy theories gaining alarming ground, the scientific community cannot afford to continue to undermine public trust. Whether it's due to the theory or the methodology, the replication crisis has discredited a plethora of studies that were once seen as scientific progress, causing scientific research to remain highly contentious. At the same time, some researchers disagree on the meaning and severity of the crisis, arguing that good science is about more than just replicability.

Erosion of public trust

When the 2015 reproducibility project found that fewer than half of landmark psychology findings could be replicated, media coverage simplified the news to “scientists can't be trusted.” The backlash was amplified because famous findings that had long been legitimized were debunked. For non-experts, changing conclusions can be confusing and disorienting.²² For example, headlines are constantly claiming that coffee is good, then bad, and then good again, fostering frustration among readers who don't follow scientific literature and are mostly exposed to clickbait narratives. This dynamic has fueled scientific denial, with growing skepticism about issues like climate change and vaccines, which are backed by an abundance of evidence.²³ Distrust has also spread within academia itself, especially after cases like that of Diederik Stapel, which imparted that some scientists are "gaming the system.” 

The theory and the method

A factor behind the replication crisis is the process by which a psychological theory is developed. Many theories are vague or underspecified, lacking precise predictions, like “priming affects behavior.” When studying subconscious mechanisms, it becomes difficult to pinpoint what is the cause of an effect. This can lead researchers to build theories around findings rather than pre-existing predictions, otherwise known as HARKing.²⁴

Methods themselves pose another problem: poor study designs, small sample sizes, and practices like p-hacking undermine reliability. This is why there has been an emphasis on reporting standards, like CONSORT and PRISMA. When strong theories are paired with weak methods, false negatives can lead to inconsistent results, while strong methods applied to weak theories can result in premature abandonment of ideas or overapplication of a theory.¹

Is the replication crisis actually a crisis?

Reproducibility alone doesn't guarantee good research; even transparent and reproducible studies can lack rigor if they're inherently noisy or poorly designed. Many leading researchers disagree about the significance of the replication crisis, or have different points of view on how it could be improved. For example, psychology researcher Lisa Feldman Barrett says, “Science is not a body of facts that emerge, like an orderly string of light bulbs, to illuminate a linear path to universal truth,” maintaining that a failure to replicate is not a cause for alarm, rather a reflection of how science actually works; it is simply not possible to replicate the findings of every study.³⁰ 

In response, American statistician Andrew Gelman argues that “If, like Barrett, you want to dismiss replications and say there’s no crisis in science: Fine. But then publish everything and accept that all data are telling you something. Don’t privilege something that happens to have been published once and declare it true.” We have a habit of treating one-off published findings as “true,” while ignoring contradictory publication attempts, illustrating a double standard. This highlights the need for a change in perspective surrounding replication; people dismiss replication attempts as flawed or irrelevant, without questioning the initial findings.³¹

Some scientists argue that the replication crisis is a front for a “publication bias crisis.” Ottoline Leyser, a British plant biologist, has the viewpoint that “there are many good reasons why two ‘identical’ experiments might not give the same result, such as unknown differences that have not been considered.” Leyser contends that the issue lies in the fact that there is an excessive focus on publishing groundbreaking scientific findings, and that “the reproducibility crisis is actually a publication bias crisis which is driven by the reward structures in the research system.”³²

There are multiple ways to interpret the replication crisis, and scientists continue to debate its causes and its possible implications. What remains crucial, however, is the persistent commitment to open data and transparency by engaging in practices that not only help clarify the roots of the issue but also ensure that high-quality research continues, even if replication isn’t always feasible.

Power posing

A study conducted by American social psychologist Amy Cuddy suggested that adopting expansive, confident body postures for as little as two minutes can influence levels of hormones like testosterone and cortisol. She called this practice “power posing,” and the findings went viral for their potential to influence business culture, generating widespread interest. Cuddy's research reported that cortisol dropped by 25% and testosterone rose by 19% in both men and women when adopting a “power pose.” Participants in the “high-power pose” group were also more willing to take risks, with 86% choosing to gamble $2 compared to 60% in the “low-power pose” group.²⁵ As a professor at Harvard Business School, Cuddy's affiliation lent her work additional credibility, making it easy for the public to assume the study was definitive. 

However, many follow-up studies failed to replicate the hormonal effects that Cuddy put forward. While the influence of power posing on hormones remains debated, evidence does suggest that it can affect behavior.²⁶ Exciting findings from seemingly credible sources are often accepted by the public without hesitation and critical evaluation, as seen with the “power posing” study. Many longstanding studies that were widely accepted for years have also faced similar challenges, like ego depletion theory, which claimed that self-control functions are like a muscle that tires with use, but later failed to replicate consistently.²⁷ 

Other fields

As mentioned, the replication crisis is not exclusive to psychology; sectors like medicine and economics have also been impacted. In 2011, Bayer Healthcare attempted to replicate 67 influential drug target studies but succeeded with only 14 of them, showing that clinical applications occasionally fail despite billions of dollars being spent on drug development.²⁸ This led to growing skepticism about what researchers were working toward, as questionable practices like selective reporting and publication bias came to light. 

In economics, behavioral economist Colin Camerer led a large-scale reproducibility project on 18 influential studies. While 61% of findings replicated, their effect sizes were reduced by about 25-50%, raising serious questions about past research practices in economics. Camerer found that key theories like anchoring and default effects held up, but more nuanced theories did not.²⁹ Strengthening reproducibility and reliability measures is essential to maintaining public trust in experts, but also ensures that valuable resources are invested in research that actually can drive real-world impacts. 

Look-elsewhere effect

Sometimes scientists keep looking for a statistically significant result after failing to find one initially—what is known as the “look-elsewhere effect.” As a major contributing factor to the replication crisis, it has impeded scientific progress and led scientists to incorrect conclusions. This article explains where this all started and how we can avoid it. 

Pitfalls of Behavioral Science: How to Avoid Untrustworthy Data

From questionable research practices (QRPs) to publication bias, how can we avoid the hazards that plague behavioral science research? TDL provides a short guide to improving research trustworthiness, including how to support replication, design strength, and ethical vigilance. 

1. Piper, K. (2020, October 14). Science has been in a “replication crisis” for a decade. Have we learned anything? Vox. https://www.vox.com/future-perfect/21504366/science-replication-crisis-peer-review-statistics
2. Rawat, S., & Meena, S. (2014). Publish or perish: Where are we heading? Journal of Research in Medical Sciences : The Official Journal of Isfahan University of Medical Sciences, 19(2), 87. https://pmc.ncbi.nlm.nih.gov/articles/PMC3999612/
3. McLeod, S. (2023, December 7). Psychology Research Ethics. Simply Psychology. https://www.simplypsychology.org/Ethics.html
4. Berkeley Initiative for Transparency in the Social Sciences. (2018, April 25). Why most published research findings are false. Berkeley Initiative for Transparency in the Social Sciences. https://www.bitss.org/education/mooc-parent-page/week-2-publication-bias/is-there-a-credibility-crisis/why-most-published-research-findings-are-false/
5. Enserink, M. (2021). Dutch University Sacks Social Psychologist Over Faked Data. Science.org. https://doi.org/10.1126/article.28354
6. Ratner, P. (2020, December 6). Are psi phenomena real? A study on precognition once exploded science. Big Think. https://bigthink.com/neuropsych/psi-phenomena-precognition-study/
7. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716
8. American Psychological Association. (2022). Home News & advocacy Podcasts Speaking of Psychology Speaking of Psychology: How “open science” is changing psychological research, with Brian Nosek, PhD. Apa.org. https://www.apa.org/news/podcasts/speaking-of-psychology/open-science
9. Psychological Science Accelerator. (2025). Psychological Science Accelerator. Psychological Science Accelerator. https://psysciacc.org/
10. AsPredicted. (2025). AsPredicted: Home. Aspredicted.org. https://aspredicted.org/
11. Metascience. (2020, February 6). Metascience - The Emerging Field of Research on the Scientific Process. Metascience.com; Metascience. https://metascience.com/mission/
12. Stanford Profiles. (2010). John P.A. Ioannidis’ Profile | Stanford Profiles. Stanford.edu. https://profiles.stanford.edu/john-ioannidis
13. Science, C. for O. (2025). Brian Nosek | Center for Open Science. Www.cos.io. https://www.cos.io/team/brian-nosek
14. The American Psychology Association. (2021). Open Science Badges. Apa.org. https://www.apa.org/pubs/journals/resources/open-science-badges
15. Grech, V., & Eldawlatly, A. A. (2024). STROBE, CONSORT, PRISMA, MOOSE, STARD, SPIRIT, and other guidelines – Overview and application. Saudi Journal of Anaesthesia, 18(1), 137–141. https://doi.org/10.4103/sja.sja_545_23
16. Udesky, L. (2025). Researchers value null results, but struggle to publish them. Nature. https://doi.org/10.1038/d41586-025-02312-4
17. Nature Human Behavior. (2025). Registered Reports | Nature Human Behaviour. Nature.com; Nature. https://www.nature.com/nathumbehav/submission-guidelines/registeredreports
18. eLife. (2016, June 23). Registered Reports. ELife. https://elifesciences.org/articles/registered-report
19. ScienceDaily. (2017). “Power poses” don’t work, eleven new studies suggest. ScienceDaily. https://www.sciencedaily.com/releases/2017/09/170911095932.htm
20. Ulrich Schimmack. (2025, May 8). Priming Research is Past its Prime - Replicability-Index. Replicability-Index - Improving the Replicability of Empirical Research. https://replicationindex.com/2025/05/08/priming-research-past-its-prime/
21. ManyBabies. (2025). ManyBabies. ManyBabies. https://manybabies.org/
22. Warren, M. (2019, October 31). The replication crisis lowers the public’s trust in psychology. BPS; The British Psychological Society. https://www.bps.org.uk/research-digest/replication-crisis-lowers-publics-trust-psychology
23. Krause, N. M., Freiling, I., & Scheufele, D. A. (2025). Our changing information ecosystem for science and why it matters for effective science communication. Proceedings of the National Academy of Sciences, 122(27). https://doi.org/10.1073/pnas.2400928121
24. Wilson, M. (2025). HARKing: What is it and why is it bad? https://health.ucdavis.edu/media-resources/ctsc/documents/pdfs/harking-2021.pdf
25. Hanna, J. (2010, September 20). Power Posing: Fake It Until You Make It. Harvard Business School. https://www.library.hbs.edu/working-knowledge/power-posing-fake-it-until-you-make-it
26. Elsesser, K. (2020, October 2). The debate on power posing continues: Here’s where we stand. Forbes. https://www.forbes.com/sites/kimelsesser/2020/10/02/the-debate-on-power-posing-continues-heres-where-we-stand/
27. Englert, C., & Bertrams, A. (2021). Again, No Evidence for or Against the Existence of Ego Depletion: Opinion on “A Multi-Site Preregistered Paradigmatic Test of the Ego Depletion Effect.” Frontiers in Human Neuroscience, 15. https://doi.org/10.3389/fnhum.2021.658890
28. Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: how much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10(9), 712–712. https://doi.org/10.1038/nrd3439-c1
29. Camerer, C. F. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918
30. Barrett, L. F. (2015, September 1). Opinion | Psychology Is Not in Crisis (Published 2015). The New York Times. https://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html
31. LSE Blog. (2015). To understand the replication crisis, imagine a world in which everything was published. Google.com. https://www.google.com/url?q=https://blogs.lse.ac.uk/impactofsocialsciences/2015/09/09/replication-crisis-imagine-a-world-in-which-everything-was-published/&sa=D&source=docs&ust=1754585309972369&usg=AOvVaw2I0MGJc8zCzyTqFpIQt7rE
32. The Conversation. (2017, March 21). Three Views on Addressing the “Reproducibility Crisis” - Social Science Space. Social Science Space. https://www.socialsciencespace.com/2017/03/three-views-addressing-reproducibility-crisis/