Imagine you are conducting a test to see whether using a fertilizer will improve plant growth in your garden. In an experiment, you always begin with a null hypothesis, which suggests that there is no statistical relationship between the variable (using the new fertilizer) and the outcome (plant growth), and an alternative hypothesis, which states that the variable does have a significant impact on the outcome.¹

To conduct your experiment, you use fertilizer for one of your lilac shrubs and leave the other shrub without. After a month, you measure the growth of each lilac shrub and find that the fertilized lilac has grown bigger than the unfertilized one. You assume that the fertilizer significantly improves plant growth and reject the null hypothesis. 

However, later on, when you start using the fertilizer for all your plants, you realize that it does not seem to be causing improved growth. You realize that the reason the first lilac shrub grew bigger was actually because it was closer to the window and received more sunlight. 

In this instance, you have made a type I error: a false positive conclusion. Type I errors occur because of random chance (which is more likely to happen with a small sample size, such as comparing only two lilac bushes) or improper testing techniques (not controlling other variables like sunlight or concluding the experiment too early).²

Type I errors cause people to conclude results are statistically significant when, in reality, they are not.³ Imagine that you started telling your friends about the fertilizer and its impact on growth, only for them to waste their money on it and see no results—you may have a lot of annoyed friends! While in this scenario, the consequences of the type I error are not too serious, they can have far-reaching consequences in medical research. If a researcher concludes that a drug improves patient outcomes, it would pass to market and be recommended for treatment. In this case, the false conclusion about an ineffective medication could be a matter of life or death.

Null Hypothesis (H0): A hypothesis that suggests there is no statistically significant relationship between a variable and outcome. It is a default assumption that is part of good research design, as it establishes a baseline from which to test variables. As a researcher, you should strive to reject the null hypothesis rather than prove the alternative hypothesis (what you believe) to minimize researcher bias.⁵ 

Alternative Hypothesis (H1): The hypothesis that contradicts the null hypothesis and states that there is a significant relationship between a variable and outcome. Usually, the alternative hypothesis is aligned with what researchers believe to be true and are trying to determine through the experiment.⁵

P-value: A number that describes how likely it is that you would get the same results if the null hypothesis is true. Essentially, the p-value indicates how probable it is that your results happened by chance. To avoid a type I error, you only assign statistical significance to a low p-value, below at least 0.05, meaning there’s only a 5% possibility the results occurred by chance. The p-value is calculated using a test statistic that measures how much your findings differ from what would be expected under the null hypothesis.⁶

Alpha (α) Level (Significance Level): A chosen threshold set at the beginning of the experiment to assess the statistical probability of making a type I error. If your p-value is less than or equal to your alpha level, you can usually be confident in rejecting the null hypothesis.³

Power: The likelihood that an experiment will correctly reject the null hypothesis. The greater the power, the better the design of the test and the less likely you are to commit a type I error. The power of a test is impacted by sample size, effect size, and significance level.⁷

Neyman-Pearson Lemma: A mathematical rule in hypothesis testing that provides a tool for designing the most powerful test for a given significance level. A more powerful test would diminish the likelihood of a type I error occurring.⁸ 

Type II error: A false negative conclusion in an experiment. This occurs when you fail to reject the null hypothesis that there is no relationship between a variable and the outcome and incorrectly reject the alternative hypothesis when, in fact, there is a statistically significant relationship.⁹

In the 1920s, British statistician Ronald Fisher formalized experimental methodology by introducing guidelines for experimental design. Fisher realized that the quality of experimental analysis was undermined by poor design, which made it difficult to validate results. He believed that to be able to conclude meaningful insights, experiments had to be well-designed.¹⁰ 

In addition to outlining rigorous experimental design features, Fisher introduced statistical probability into the mix and developed a way to calculate the p-value to determine the probability of results being significant.¹¹ He advocated for experiments to be designed with rigor, ensuring that significant results were detected when they truly existed.¹² 

In 1933, building on previous integration of statistical probability with experiment design, Polish mathematician Jerzy Neyman and British statistician Egon Pearson introduced the concept of type I error in their paper “On the Problem of the Most Efficient Tests of Statistical Hypotheses.” Their paper stated that there were four potential outcomes of an experiment: the null hypothesis could be true but rejected (type I error), false and rejected (correct decision), true and not rejected (correct decision), or false but not rejected (type II error).¹³

In this paper, Neyman and Pearson also introduced a framework for hypothesis testing known as the Neyman-Pearson lemma.¹⁴ The lemma served as a guide for making optimal decisions and increasing the power of a test when faced with conflicting hypotheses. Neyman and Pearson formalized the concept of binary hypothesis testing, which involves a null hypothesis and an alternative hypothesis. The lemma provided insights into tests that would maximize statistical power for a given significance level (alpha), making it less likely for a type I error to occur. Previously, methods of significance testing were commonly based on Ronald Fisher’s framework, which focused on rejecting the null hypothesis based on p-values, rather than minimizing type I and type II error.  Neyman and Pearson’s framework provided a way for researchers to balance both errors in the design of an experiment.⁸

Ronald A. Fisher

Considered the father of experimental design, Fisher pioneered the use of statistical measures in the design of scientific experiments, which was prompted by his realization that it was difficult to validate the conclusions of experimental analysis because experiments were not well designed.10 Some of Fisher’s biggest contributions to scientific research include variation, sample randomization, and a method for calculating the p-value.¹⁵ 

Jerzy Neyman

A Polish mathematician who developed the concept of type I error alongside Egon Pearson. Neyman had previously pivoted away from statistics before receiving a letter from Egon Pearson, who was looking for a way to test whether there was a significant difference between the mean results of two groups. This led to Neyman and Pearson collaborating to develop and introduce the Neyman-Pearson lemma in their 1993 paper “On the Problem of the Most Efficient Tests of Statistical Hypotheses.”¹⁶

Egon Pearson

A British statistician and the son of Karl Pearson (considered the founder of modern statistics) who co-introduced type I errors and the Neyman-Pearson lemma with Neyman. Pearson was greatly interested in robustness studies, exploring ways to make experiments and their results more reliable and conclusive. Other than his ten-year-long collaboration with Neyman, Pearson is also well-known for working with H. O. Hartley, a German-American statistician, to update his father’s Tables for Statisticians and Biometricians, to provide statistical resources in an era before computers and calculators.¹⁷

Type I errors can lead to significant consequences, particularly in fields like medicine and law enforcement, where false positives can result in unnecessary treatments or wrongful arrests. These errors often arise due to pressures such as publication bias, where studies showing significant findings are more likely to be published, contributing to the replication crisis and undermining the reliability of research findings.

Consequence of False Positives

Type I errors are also known as false positives because they show a relationship that isn’t really there (e.g., believing the plant fertilizer leads to improved growth when, in fact, it has no significant effect). 

Unfortunately, some of the most common instances of type I errors are with medical tests, which can have serious consequences. For example, blood tests are used to screen for diabetes. Patients are asked to fast before the test, as high blood sugar after fasting would indicate the patient has diabetes. However, a patient may lie about whether they fasted, or may be taking medication that contributes to unusually high blood sugar readings. The doctor might conclude that they have Type II diabetes when, in fact, the high blood sugar can be explained by another factor. The patient may start taking medication unnecessarily, which can have negative side effects.¹⁸

Another field where false positives have ethical implications is the use of facial recognition technology for law enforcement. Facial recognition systems will analyze an image or video, and then compare the faceprint with a database of faces to try and find a match. However, even complex technologies aren’t immune to type I errors. The technology may indicate a match when there isn’t one, which could lead to wrongful arrests.¹⁹

There has been a lot of discourse around facial recognition systems, demonstrating that they are more likely to make a type I error for racialized faces. In fact, a study conducted by the federal U.S. government showed that African American and Asian faces were up to 100 times more likely to be misidentified than white faces, and Native Americans were most likely to be victims of type I errors.²⁰ 

Pressure to Publish

One of the reasons type I errors occur is because of publication bias. Studies that show significant findings are more likely to be published than studies with non-significant findings. While this intuitively makes sense—it is a lot more interesting to read how a drug led to improved patient outcomes than a study that only proves a lack of a relationship—it leads to increased pressure on researchers to find relationships even when there may not be one.¹⁹

Research by the journal Nature found that 2% of researchers admitted they had falsified studies before, and 14% said they knew someone who did.²¹ This pressure can come from the companies paying for the research. For example, if a pharmaceutical company develops a new drug to diminish headache symptoms, they hope that the trials will show that it is an effective painkiller. If a researcher finds no significant relationship, they may be asked to adjust the sample or the alpha level to show the drug’s effectiveness or asked not to publish their findings. Research demonstrating significance is more likely to be published even if other studies fail to reject the null hypothesis, so people would be led to believe it is an effective painkiller to treat headaches when that might not be the case. 

Replication Crisis

When type I errors occur, the research suggests a relationship where there is none. That means that future studies will struggle to replicate the findings. Many scientific fields have found themselves in a “replication crisis” where many findings from psychology, sociology, and medicine are not validated through further research, highlighting that early findings may be false positives.²² 

This replication crisis has led to a distrust in published findings, as we don’t always know which ones are reliable. This can lead to valuable scientific knowledge being dismissed and uncertainty about which results can be relied upon to inform policy, practice, or scientific progression. The replication crisis has led to advocacy for more robust research methodologies and less bias toward publishing significant findings.

Type I errors are controversial due to their prevalence in high-stakes fields like psychology and medicine, where false positives can lead to misguided policies or harmful medical practices. However, efforts to avoid these errors sometimes come with unintended consequences, such as stifling innovation or increasing the likelihood of type II errors.

Notable Cases of Type I Errors

Two of the fields that are commonly criticized for the prevalence of type I errors are psychology and medicine. 

In the past, many psychological studies did not use rigorous methodologies compared to other scientific fields, making it difficult for future researchers to replicate experiments. A lot of psychology studies use qualitative research methods, which makes it harder to copy the study exactly. A 2015 study on the reproducibility of psychological studies found that findings could only be replicated in 39 out of 100 studies.²² This can be dangerous, as psychological research is often used to implement policies, and if the findings are false, then those policies will be misguided.²³

In medicine, due to pressure from pharmaceutical companies and the demand for quick publication, type I errors are common. Type I errors aren’t always conscious decisions to falsify findings, but if we’re looking for an effect, we’re more likely to accept results that are consistent with our assumptions due to confirmation bias. Other times, however, the data really is fake. 

One researcher who falsified data in order to prove an alternative hypothesis was cardiologist Don Poldermans. From 1999 to the early 2010s, Poldermans published multiple articles that showed that beta blockers should be administered before surgery, as they lowered blood pressure. Based on Poldermans’ research, both European and US guidelines recommended the administration of beta blockers before cardiac surgeries. It wasn’t until 2014 that a meta-analysis found that beta blockers increased the chance of patients dying within 30 days of their surgery by 27%. The type I error promoted by Poldermans was estimated to have led to the death of 800,000 people. While some people argue that the figure may be inflated, it still stands that this type I error led to the death of thousands of people.24 

Stifling Innovation and Progress

In order to avoid type I errors and diminish the risk of publishing results that are later found to be false, researchers might become very conservative in their methodologies. Not wanting to rock the boat, they might be more likely to proceed with status-quo methods and only conduct experiments that have a high likelihood of yielding positive results, stifling innovation. Fear of failure prevents people from trying at all.25

Many of the most groundbreaking studies have been controversial and may not have happened at all if researchers didn’t face their fear of failure. Focusing on avoiding type I errors can slow down the rate of progress and prevent new, innovative treatments from being approved or recommended.

For example, mindfulness-based cognitive therapy has been shown to positively impact people experiencing depression, anxiety, or chronic stress. There are many anecdotal stories demonstrating a positive effect and even some clinical trials supporting the treatment. However, these trials often have small sample sizes or are non-randomized, therefore failing to produce a large effect size. Because of the challenge of meeting rigorous statistical standards, mindfulness-based cognitive therapy is therefore not very widely adopted by therapists, and patients are missing out on its benefits.²⁶

Increases Likelihood of Type II Errors

Unfortunately, when researchers put measures in place to reduce the likelihood of a type I error, such as lowering the significance level, they increase the chance of a type II error. Any results that fall under the significance level, even if they show that a variable has some impact on the outcome, are dismissed as insignificant. This may mean that a study will fail to reject the null hypothesis when the alternative hypothesis may be true. 

If we consider a medical test, while a type I error can lead to unnecessary treatment, a type II error would mean a diagnosis is missed.²⁷ For example, it has been found that type II errors in breast cancer screening can affect women who have dense breasts. Mammograms search for abnormalities, like lumps or calcifications in the breast. Women with dense breasts have more fibroglandular tissue compared to fatty tissue. Because fibroglandular tissue and tumors both show up white on X-rays, doctors will sometimes miss an abnormality. This type II error causes some women’s breast cancer to go undiagnosed in the early stages, where interventions are most successful.²⁸ 

This demonstrates the tradeoff between type I and type II errors: reducing the likelihood of one will increase the likelihood of the other. Researchers must play a balancing act based on the severity of each consequence.

False Positives in COVID Testing

During the COVID-19 pandemic, many of us took at-home rapid antigen tests to test whether we had contracted the virus. Whether we saw one line or two lines would drastically change our behavior: if the test was positive, we had to isolate and inform anyone we had been in contact with recently, which could lead to widespread fear and anxiety. If the test was negative, we could go on as normal. 

A report by The New England Journal of Medicine showed that although rare, some tests exhibited a type I error, and this was more common for women and people with autoimmune disorders. They found that some people would report a positive rapid antigen test, but when they completed an RT-PCR test within 48 hours, the results came back negative. This only occurred in 1.7% of tests on over 11,000 participants—however, if we look specifically at the 604 people who received a positive antigen result and later took a PCR test, 191 of them then tested negative. That means that for almost a third of the individuals who initially tested positive,  it was a false positive result. For 13 individuals, most of whom were women and several with autoimmune disorders, the false positives were persistent across multiple rapid antigen tests.²⁹

In this case, a type I error is less detrimental than a type II error, as isolating when unnecessary is better than the risk of infecting others due to a false negative. However, the study provided valuable insight that if you are female or have an autoimmune disease, you should consider getting a PCR test to confirm your diagnosis.³⁰

Debunking The Ego Depletion Theory

In 1998, social psychologist Roy F. Baumeister published an article introducing the theory of ego depletion.³¹ The article suggested that when people use up all of their willpower or self-control for one task, it drains their energy and they are unable to exert the same level of self-control in subsequent tasks. For example, if you are trying to quit smoking, you may resist the first two times someone asks if you’d like a cigarette, but by the third time, your willpower is too low to refuse.³²

Baumeister and his colleagues came up with the theory of ego depletion after conducting four experiments. In the first experiment, participants waited in a room where there was a plate of chocolate chip cookies and a plate of radishes. Participants were assigned to either eat chocolate chip cookies, or radishes. The researcher left the room while the participants ate, and afterward, they were asked to complete a puzzle. Participants were unaware the puzzles were actually unsolvable. The researchers found that individuals who had the radishes—and therefore had to exhibit self-control to not eat the cookies—quit the puzzle faster.³³

In the second experiment, participants had to record a persuasive speech either advocating for or against a tuition raise. One group was not given a choice of which to record and was assigned the pro-tuition speech, while the other group was given a choice but strongly encouraged to choose the pro-tuition option. After recording their speeches, participants were asked to complete an unsolvable puzzle. Participants who were given an option and chose to make an anti-tuition speech—exhibiting self-control to stick to their beliefs instead of giving in to the researchers—were less persistent when trying to solve the puzzle.³³

The other two experiments showed similar effects, which led Baumeister to develop the ego depletion theory. Other researchers were interested in this theory, leading to subsequent studies involving over 30,000 participants that exhibited similar findings. Later, Baumeister conducted studies to show that self-control relies on glucose, so when it is used up, willpower diminishes. However, in 2011, physiologists challenged Baumeister’s glucose theory and it came to light that Baumeister had hidden studies that showed no statistical significance between glucose and willpower. He defended himself, stating: “We initially submitted [the paper] with more studies, some of which had weaker results. The editor said to delete those.”³¹

In 2016, a large-scale study called Many Labs 2 aimed to find whether previous studies and accepted findings could be replicated. Ego depletion was one of the theories that Many Labs 2 tried to replicate. Twenty-three different studies were conducted with over 2,000 participants, and the researchers found little to no evidence of ego depletion, concluding that the original findings were likely a type I error.³⁴

From Gossiping to High-Quality Research for Decision-Making

Everybody gossips—it’s a natural part of being human. However, sometimes we use something we learned through gossip to make a decision, without first validating if the gossip is true. Have you ever wondered how you can transform gossip into high-quality research? If you just rely on gossip, without validating whether the information is true, you may make a type I error: assuming something is true when actually it is false. In this article, our writer, Itzel Cabrero Iriberri, provides guidance on conducting qualitative research to show you how to gather and analyze gossip to guide your decisions. 

Why Do We Think Some Things Are Related When They Aren’t?

A type I error occurs when we falsely conclude that there is a relationship between a variable and outcome and reject the null hypothesis. Outside of scientific experiments, people make type I errors on a regular basis because we tend to think things are related even when they aren’t. This article explores our tendency to find illusory correlations, associating events between variables when they aren’t associated, and outlines the pitfalls that this bias can lead to.

1. Hayes, A. (2024, September 21). Null hypothesis. Investopedia. https://www.investopedia.com/terms/n/null_hypothesis.asp
2. Hotjar. (n.d.). Type I error. Hotjar. https://www.hotjar.com/conversion-rate-optimization/glossary/type-1-error/
3. Bhandari, P. (2021, January 18). Type I and type II errors. Scribbr. https://www.scribbr.com/statistics/type-i-and-type-ii-errors/
4. Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part I. Biometrika, 20A(1/2), 175-240. https://doi.org/10.1093/biomet/20A.1.175
5. Hayes, A. (2024, September 21). Null hypothesis. Investopedia. https://www.investopedia.com/terms/n/null_hypothesis.asp
6. Bevans, R. (2020, July 16). P-value. Scribbr. https://www.scribbr.com/statistics/p-value/
7. Bhandari, P. (2021, February 16). Statistical power. Scribbr. https://www.scribbr.com/statistics/statistical-power/
8. The Learning Channel of Quantitative Sciences. (2020, October 5). The Neyman Pearson lemma in statistics [Video]. YouTube. https://www.youtube.com/watch?v=fiPoS3qiiB4
9. Shreffler, J., & Huecker, M. R. (2023, March 13). Type I and type II errors and statistical power. StatPearls. https://www.ncbi.nlm.nih.gov/books/NBK557530/
10. Braithwaite, L. (n.d.). Experimental design. The Decision Lab. https://thedecisionlab.com/reference-guide/statistics/experimental-design
11. Biau, D. J., Jolles, B. M., & Porcher, R. (2009, November 17). P value and the theory of hypothesis testing: An explanation for new researchers. Clinical Orthopaedics and Related Research, 468(3), 885–892. https://doi.org/10.1007/s11999-009-1164-4
12. Schneiter, R. (n.d.). Historical overview of hypothesis testing. Utah State University. https://www.usu.edu/math/schneit/StatsStuff/Inference/HistoricalHT/
13. Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231, 289-337. https://www.jstor.org/stable/91247
14. Oxford University Press. (n.d.). Type I error. In Oxford reference. https://www.oxfordreference.com/display/10.1093/oi/authority.20110803100232947
15. Bodmer, W., Bailey, R. A., Charlesworth, B., Eyre-Walker, A., Farewell, V., Mead, A., & Senn, S. (2021, January 15). The outstanding scientist, R.A. Fisher: His views on eugenics and race. Nature Reviews Genetics. https://doi.org/10.1038/s41576-020-00331-x
16. St. Andrews University. (n.d.). William Sealy Gosset. Retrieved from https://mathshistory.st-andrews.ac.uk/Biographies/Gosset/
17. Pearson, E. S. (2017). On the history of the statistical approach to hypothesis testing. Amstat News. Retrieved from https://magazine.amstat.org/wp-content/uploads/2017/08/pearsonegon.pdf
18. University of Hawaii at Manoa. (n.d.). False positives and false negatives. Exploring Our Fluid Earth. Retrieved from https://manoa.hawaii.edu/exploringourfluidearth/chemical/matter/properties-matter/practices-science-false-positives-and-false-negatives
19. FasterCapital. (n.d.). Examples of type I error in research studies. Retrieved from https://fastercapital.com/topics/examples-of-type-i-error-in-research-studies.html
20. Amnesty International. (n.d.). Racial bias in facial recognition algorithms. Retrieved from https://www.amnesty.ca/features/racial-bias-in-facial-recognition-algorithms/
21. Davey, R. (n.d.). What is the replication crisis? News-Medical. Retrieved from https://www.news-medical.net/life-sciences/What-is-the-Replication-Crisis.aspx
22. Piper, K. (2020, October 14). Science has been in a “replication crisis” for a decade. Have we learned anything? Vox. https://www.vox.com/future-perfect/21504366/science-replication-crisis-peer-review-statistics
23. Rodgers, J. L., & S, P. E. (2018). Psychology’s replication crisis as scientific opportunity: A précis for policymakers. SAGE Open, 8(4), 1-11. https://doi.org/10.1177/2372732217749254
24. Piper, K. (2024, August 26). Should scientific research fraud be a crime with jail time? Vox. https://www.vox.com/future-perfect/368350/scientific-research-fraud-crime-jail-time
25. Chatterjee, A. (2023, June 18). The value of named innovation: Going beyond the fear of type I error. LinkedIn. https://www.linkedin.com/pulse/value-named-innovation-going-beyond-fear-type-i-error-chatterjee/
26. Chiesa, A., & Serretti, A. (2011, May 30). Mindfulness based cognitive therapy for psychiatric disorders: A systematic review and meta-analysis. Psychiatry Research, 187(3), 441-453. https://doi.org/10.1016/j.psychres.2010.08.011
27. Amplitude. (n.d.). Type 1 and type 2 errors explained. Amplitude. Retrieved January 17, 2025, from https://amplitude.com/explore/experiment/type-1-and-type-2-errors-explained
28. European Society of Breast Imaging. (2022). Breast cancer screening in women with extremely dense breasts: Recommendations of the European Society of Breast Imaging (EUSOBI). European Radiology, 32(6), 4036–4045. https://doi.org/10.1007/s00330-022-08617-6
29. Persistent False Positive Covid-19 Rapid Antigen Tests. (2024, February 21). New England Journal of Medicine, 390(8), 764-765. https://doi.org/10.1056/NEJMc2313517
30. Hui-Anderson, A. (2024, March 1). How common is a false positive COVID test? Verywell Health. https://www.verywellhealth.com/how-common-is-a-false-positive-covid-test-8602420
31. Baumeister, R. F., Ego Depletion, Meta-Analysis, Power, R-Index, & R-Index4Science. (2016, April 18). Replicability Report No. 1: Is Ego-Depletion a Replicable Effect? R-Index4Science.
32. Cherry, K. (2021, May 27). Ego depletion: What it is and how it works. Verywell Mind. https://www.verywellmind.com/ego-depletion-4175496
33. Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (2000). Ego depletion: Is the active self a limited resource? Self and Identity, 9(1), 1–10. https://doi.org/10.1207/S1532706XID0901_1
34. Hagger, M. S., Chatzisarantis, N. L. D., & Zwienenberg, M. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 468–478.https://doi.org/10.1177/1745691616652873