We’ve all heard that drinking a coffee late will make it harder to fall asleep, but does caffeine really impact our sleep? You’re not sure, so you decide to conduct an at-home experiment.

Like any good experiment, you start with two potential hypotheses:

Null hypothesis (H0): Drinking coffee in the evening has no effect on your sleep

Alternative hypothesis (H1): Drinking coffee in the evening negatively impacts your sleep

The first day, you don’t drink any coffee and go to bed at 10:00 pm. You fall asleep quickly and, according to your sleep-tracking app, achieve an 87% sleep score. For two days after, you drink coffee at 7:00 pm and go to bed at 10:00 pm. You’re still able to fall asleep quickly, and your sleep score doesn’t vary much from the first day. Given the evidence, you retain the null hypothesis that coffee doesn’t affect your sleep and reject the alternative hypothesis.

However, in reality, the caffeine does impact your sleep—the experiment you designed just didn’t detect the effect. It was only a small sample size of one participant, you only compared one day without coffee to two days with coffee, and other factors—like how active you were those days, fatigue, stress, or screen time—all may have impacted your sleep quality. Even though you tried to control variables by maintaining a consistent bedtime and attempted to measure your sleep quality in an unbiased way with an app, you have failed to reject the null hypothesis. 

This type of error is known as a type II error. Type II errors occur when a null hypothesis is incorrectly retained and a conclusion is made that there is insufficient evidence to suggest that a variable (in this instance, coffee) has a particular effect on an outcome (in this instance, sleep quality). A true effect is missed, resulting in a “false negative.”¹ Your conclusion from your at-home experiment might result in caffeine intake late in the evening, and you may eventually feel the repercussions when you continuously wake up groggy. But in more serious contexts, type II errors can have higher stakes. Imagine if a medical trial testing the effect of a new drug treatment for depression falsely concluded that the drug had no effect on patient outcomes? The drug would be dismissed, never making it out of the clinical trial stage, delaying critical treatments. 

Null Hypothesis (H0): A default hypothesis stating that there is no statistically significant relationship between a variable and outcome (no effect is observed). The null hypothesis increases the strength of research design, as it minimizes bias related to research solely focused on proving a relationship (the alternative hypothesis).³ A type II error occurs when the null hypothesis is falsely retained.

Alternative Hypothesis (H1): A hypothesis that opposes the null hypothesis, stating that a statistically significant effect can be found between the independent variable and outcome. Most often, researchers are trying to prove the alternative hypothesis through their experiment. A type II error occurs when the alternative hypothesis is falsely rejected.³

Statistical Power: The likelihood of an experiment correctly detecting an effect and rejecting the null hypothesis. Statistical power is most often associated with sample size; the higher the sample size, the stronger the statistical power of the experiment. If power is low, you are more likely to make a type II error and fail to reject the null hypothesis when it is incorrect.⁴ 

Beta (β): The probability of committing a type II error, which is inversely related to power. One of the most effective ways to decrease the beta value is to increase the sample size, to ensure that sufficient data is collected to judge whether an effect truly exists. A 20% beta (80% power) is seen as acceptable for drawing conclusions.⁵ 

Type I Error: Contrasting a type II error, this is a “false positive” error. The null hypothesis is incorrectly rejected, and a relationship is believed to exist between a variable and an outcome that isn’t really there, causing the experimenter to accept the alternative hypothesis. Type I errors are thought to be more dangerous than Type II errors, as ineffective policies or medical interventions can be recommended when they do not lead to positive outcomes, potentially resulting in harmful side effects or impeding the search for better solutions. 

Neyman-Pearson Lemma: A mathematical rule to help researchers decide between a null hypothesis and an alternative hypothesis. It’s a likelihood ratio test researchers can use to determine what the likelihood of the data under each hypothesis is. If the likelihood of the alternative hypothesis is greater than the likelihood of the null hypothesis, giving a high ratio, there is stronger evidence for the alternative, which helps researchers more confidently reject the null hypothesis.⁶

The foundation of modern experimental design was first developed in the 1920s by British statistician Ronald Fisher. Fisher observed that the quality of results was often impacted by poorly designed experiments, as there were no standardized guidelines for collecting or analyzing data. 

Fisher developed meticulous guidelines in two influential books, Statistical Methods for Research Workers and The Design of Experiments, where he introduced important concepts like randomization and replication. He also introduced statistical probability, providing researchers with a way to calculate the probability of results being statistically significant and not just occurring due to chance.⁷ Initially, Fisher was primarily concerned with reducing the likelihood of false positives by requiring that results demonstrate strong evidence before rejecting the null hypothesis. He urged researchers to ignore small effects between variables and outcomes, attributing them to chance. However, this means it becomes harder to detect real effects, increasing the chance of type II errors.⁸ 

In 1933, as classical hypothesis testing frameworks were beginning to be established, Polish mathematician Jerzy Neyman and British statistician Egon Pearson published the seminal paper “On the Problem of the Most Efficient Tests of Statistical Hypotheses.” In this paper, they suggested that the null hypothesis—that there is no relationship between variable and outcome—should be the default assumption, with the alternative hypothesis—that there is a significant relationship between variable and outcome—being a competing claim. They also introduced the terms type I and type II error.⁹ A type I error occurs when the null hypothesis is true, but is incorrectly rejected, whereas a type II error occurs when the null hypothesis is false, but fails to be rejected. 

In their paper, Neyman and Pearson emphasized the need for various tests that could reduce the likelihood of both errors and allow researchers to be more confident in their conclusions. They introduced the Neyman-Pearson lemma, a mathematical rule to help researchers pick the best way to test hypotheses, given a fixed significance level. The goal of the lemma is to maximize the likelihood ratio of the alternative hypothesis to the null hypothesis, which makes it more likely that the null hypothesis is accurately rejected.⁶ For example, it would help you decide the appropriate statistical test to determine which sleep-tracking app produces the most reliable data. 

To avoid type II errors, Neyman and Pearson also outlined the statistical power test to determine the beta (β) value. Statistical power calculates the probability that your test will detect an effect, given that the effect exists, and is impacted by variables such as the size of the effect (a bigger impact is easier to detect), sample size, significance level, and how many other variables may influence outcomes. It should be conducted prior to an experiment so that these variables can be adjusted if the statistical power is too low. Neyman and Pearson suggested researchers aim for a statistical power of 80%.⁹ 

Ronald Fisher

A British statistician who is often referred to as the father of experimental design for his contribution of statistical measures to design reliable scientific experiments. When he was working in the agricultural field, he realized that poor experimental design led to low-quality data. He developed guidelines to improve experimental design so that he (and others) would be able to more accurately study the effects of nutrition and soil types on plant fertility. In his books Statistical Method for Research Workers and The Design of Experiments, Fisher introduced foundational concepts such as randomization and the importance of replication to validate data generated in experiments, which were applied beyond the agricultural field.¹⁰ 

Jerzy Neyman

A Polish mathematician who further developed experimental design with his colleague, Egon Pearson, through the development of the statistical theory of hypothesis. He helped establish the field of theoretical statistics, which explores the principles and mathematical frameworks that underlie experimental statistical methods. Neyman and Pearson’s co-authored paper, “On the Problem of the Most Efficient Tests of Statistical Hypotheses,” introduced important concepts like null and alternative hypotheses, the Neyman-Pearson lemma, type I and type II errors, and statistical power.¹¹ 

Eagon Pearson

A British statistician who was lucky to learn from the best—his father, Karl Pearson, the father of modern statistics. Together with Jerzy Neyman, his work further developed experimental design, introducing robust concepts like alternative hypotheses, the Neyman-Pearson lemma, and type I and II errors. Throughout his career, Pearson was dedicated to robustness studies, finding methods to ensure that results were reliable and could be validated.¹²

Type II errors can have serious real-world consequences by allowing true effects or risks to go undetected. These errors can delay life-saving treatments, allow guilty parties to go unpunished, and lead to harmful public policies.

Type II errors in medical trials

Medical trials are designed to test the effectiveness of a new drug or intervention in improving patient outcomes. Before the drug is brought to market, it undergoes rigorous clinical experiments so that agencies, like the Food and Drug Administration (FDA), are confident in the safety and effectiveness of the treatment.

However, type II errors can sometimes occur when the null hypothesis—that there is no relationship between the drug and patient outcomes—is falsely retained instead of being rejected in support of the alternative hypothesis—that the drug has a positive impact on patient outcomes. This may occur because the evidence does not show a statistically significant relationship between the drug and patient outcomes, which results in delayed treatment availability and a misallocation of resources.

One example of a type II error was when a medical trial on a drug developed for hypertension (high blood pressure) initially reported nonsignificant findings. Elevated blood pressure is the leading cause of mortality globally, which makes a drug designed to reduce hypertension highly sought-after.¹³ The researchers used a small sample size and were unable to demonstrate a significant relationship between the drug and improving hypertension, and the lack of evidence forced them to retain the null hypothesis. Thankfully, they didn’t give up. They conducted further clinical trials, with larger sample sizes and higher statistical power. This time, they were able to show that the drug reduced blood pressure, revealing that they had made a type II error in the initial study. Had the team believed the initial findings, they would have halted research and given up on the drug, which could have prevented thousands of patients from accessing a potentially life-saving treatment.¹⁴ 

Acquitting a guilty defendant

While type II errors are most commonly discussed in relation to scientific experiments, they can also occur in the justice system. A type II error in court could result in a criminal going free.

Let’s apply hypothesis testing to a court case. Mark has been arrested for breaking and entering and theft. The hypotheses would be:

H0    - Mark did not commit the crime (Mark, the variable, did not cause the crime, the outcome)

H1    - Mark did commit the crime (Mark caused the outcome)

The plaintiff is trying to prove the alternative hypothesis. They think that Mark is guilty, which is the charges were brought against him. However, in the justice system, people are presumed innocent until proven guilty, which provides the null hypothesis.

During a court case, the plaintiff would provide data as evidence towards the alternative hypothesis. Perhaps they have an eyewitness who saw Mark leaving the home he robbed, or a piece of jewelry missing from the house was in his pocket when police arrested him. However, the defendant would be trying to show that the evidence just occurred by chance. They may suggest that the eyewitness is unreliable or that Mark bought the jewelry from a pawn shop. For Mark to be convicted, the judge or jurors must believe beyond a reasonable doubt that he is guilty. Even if it seems like Mark is guilty, if the evidence isn’t strong enough, then the null hypothesis has to be retained, which may mean a type II error has occurred.¹⁵

Unfortunately, if a type II error occurs in court, it can mean that a guilty defendant is acquitted and allowed to go free. While some people argue this is better than a type I error in law, where an innocent person is convicted unfairly, a type II error results in criminals not facing the consequences of their actions and potentially repeating their crimes.

Implementing poor public policies

Type II errors in public policy occur either when decision-makers fail to detect negative side-effects of interventions, believing there is no relationship between the variable and negative outcomes, and therefore implementing the policy, or by failing to detect the positive side-effects of interventions and rejecting the policy.

One famous example of a type II error in public policy occurred in the 1970s. Historically, right turns at red lights were illegal in the U.S. Policy-makers believed that allowing right turns at red lights would improve traffic flow and would reduce fuel consumption, an important mandate in the Energy Policy and Savings Act of 1973, but engineers were concerned that it would lead to more accidents. Right turns at red lights became legal in Virginia. They conducted multiple studies to see if more accidents occurred after the right turns had been legalized. In a report, they stated that there was no significant relationship between right turns at red lights and accidents, retaining the null hypothesis that it had no effect.

However, it was later realized that a type II error had occurred. After right turns at red lights were almost universally legalized across North America, much larger sample sizes were available to test whether it impacted the number of accidents. It was found that right turns significantly increased accidents involving cyclists and pedestrians. As the initial studies failed to detect this effect, public policy was widely implemented that put people’s lives in danger, showing the very real consequences of a type II error.¹⁶ 

While much of statistical practice emphasizes minimizing Type I errors, there is growing concern that Type II errors—failing to detect real effects—are often overlooked. This has sparked debate about the balance between avoiding false positives and the risk of missing meaningful discoveries, which can have serious consequences in research and innovation.

Which is worse: type I or type II errors?

Most statistical advice suggests that making a type I error is more detrimental than a type II error. The thought experiment most commonly cited is to consider, in a legal situation, whether it is better for an innocent person to wrongly be accused and go to jail (a type I error), or for a guilty person to wrongly be acquitted (a type II error). Most people agree that a type I error is worse.¹⁷

However, the answer may not be so black and white. Researchers need to consider the potential consequences of each error to determine the best way to balance the risk, depending on their unique experiment. For example, let’s say a psychologist wants to test the effect of a new therapy intervention. If the potential outcome of the intervention helps patients better manage their mental health and there are few negative side effects, even if the psychologist only has limited evidence that it can improve patient outcomes from clinical trials, the potential benefits may outweigh the potential consequences. A type II error in this case may just maintain the patients’ current mental health struggles rather than worsen them. Alternatively, if there is a very expensive drug aimed at helping patients manage a life-threatening disease, and the potential negative side effects could worsen their condition or even lead to death, it would not be prudent for the FDA to approve the drug without strong statistical evidence.

Abundance of caution

As type I errors are seen as having worse consequences, there are rigid measures in place to avoid them, with the general rule of thumb being that the results must have a p-value below a 0.05 significance level, meaning that there is only a 5% probability that the outcome of the experiment is due to chance, with some experiments setting the threshold even lower. 

However, this results in researchers being more likely to commit a type II error. Researchers can only accept the alternative hypothesis if the evidence is very strong—meaning that many true effects may go undetected or unreported. In other words, by setting a strict threshold to avoid false positives, researchers increase the risk of overlooking real relationships, especially in studies with small sample sizes or subtle effects.

There are real-life implications to being too strict with the significance level and therefore committing a type II error. In a 2003 clinical trial, researchers compared the potential side effects of cardiovascular disease for two painkillers: Vioxx and naproxen. Heart attacks were reported for ten patients taking Vioxx compared to seven for naproxen. As the difference was too small, the researchers failed to reject the null hypothesis: that taking Vioxx over naproxen has no significant effect on the likelihood of having a heart attack. However, after the drug was put on the market, it was later revealed that Vioxx did increase the risk of heart attack. Thousands of people were harmed, and the company producing Vioxx faced over four thousand lawsuits.¹⁸ 

This example shows that when there is an abundance of caution for proving a relationship, we risk dismissing early warning signs—sometimes with devastating consequences. Eliminating type I and type II errors poses a challenge, as minimizing the risk of one tends to increase the chances of committing the other. To balance out this risk, the best practice is to fix the significance level at 0.05 and then maximize power.

Stifling innovation

One way to reduce type II errors is to increase the sample size, which decreases the beta value. However, there are obstacles to conducting experiments with large sample sizes: it can be costly and time-intensive. When researchers are unable to show a statistically significant relationship between a variable and an outcome in early and exploratory stages of the research process, they struggle to gain funding and risk being shut down. This leads to many promising ideas being abandoned too early. 

In the late 1990s, clinical trials were conducted to test the effectiveness of high doses of interleukin-2, an immunotherapy drug that kills cancer cells, in the treatment of advanced skin cancer. The results showed that the drug did have positive effects on patients, leading to remission for some, but the sample size was small. Researchers advocated for subsequent larger studies to follow, but it took time for people to pay attention to the drug. Only 8% of patients experienced a complete response to interleukin-2—meaning their cancer completely went away—while 18% experienced partial positive effects, such as their tumours shrinking. These results were not statistically significant, resulting in a delay for later studies. Thankfully, larger trials eventually were conducted years later, and interleukin-2 ended up playing a big role in the development of cancer immunotherapies.¹⁹

This case illustrates how the fear of committing a Type II error discourages investment in new and innovative approaches. When combined with a publication culture that disproportionately rewards "significant" results, the scientific process can become risk-averse, favoring safe bets over bold ideas. In the long run, this not only slows progress but may prevent life-changing breakthroughs from ever making it out of the lab.

Claiming zero effect when statistical significance is missing

A type II error occurs when a study fails to show a statistically significant relationship between a variable and an outcome and incorrectly fails to reject the null hypothesis that there is no effect. However, just because the relationship is statistically insignificant, that doesn’t mean there’s no effect at all, and claiming so may be misleading.

In 2002, journalists and researchers Stephen Balkin and Keith Ord conducted a study to understand the impact of increased speed limits on U.S. interstate highways. In 1987, speed limits were increased to 65 mph on rural interstates. In 1995, the federal government allowed states to set their own speed limits, and many increased the limit once again. Balkin and Ord wanted to see whether the changes in speed limits led to increased fatal crashes and focused on these two time periods. 

After reviewing data, Balkin and Ord concluded that nineteen out of forty states experienced a significant increase in fatal crashes following the 1987 speed limit increase, but this relationship was only seen in ten out of thirty-six states after 1995. What this suggests is that for twenty-six states, the speed limit had no effect on the number of fatal crashes. However, if you looked at their raw data, some of these states did in fact experience an increased number of fatal crashes: it just wasn’t statistically significant. There’s potential that Balkin and Ord committed a type II error, claiming no relationship, when in fact, their study may just not have had the statistical power required to accept the alternative hypothesis.¹⁶ 

This example highlights a broader issue in statistical analysis: the risk of type II errors, where real effects go undetected due to limitations in study design or data. This can lead to false conclusions that put lives at risk, like the belief that an increase in speed limits beyond 65 mph has no effect on fatal crashes. 

Outdated data leading to type II errors

In recent years, climate change has been at the forefront of the news and politics. However, it’s actually been a concern for many years, with environmental protection agencies setting laws in place to try and reduce the level of pollution.

In the early 2000s, Houston was facing poor air quality challenges. To date, the city has been relying on emission factor policies, forcing industrial factories to stay under a particular limit to prevent pollution. These policies were implemented in 1971 by Jim Southerland, who had been hired by the US Environmental Protection Agency. The emission policies suggested that if industrial companies stayed under it, then there would not be a significant effect on air quality. 

As Houston was facing challenges, it prompted a deeper dive into these emission policies. The Texas Natural Resource Conservation Commission and other environmental agencies conducted a comprehensive study to try and figure out the problem. What they realized was that the levels of volatile organic compounds, which contribute to poor air quality, were much higher than the data estimated. Industrial organizations were either underreporting or misreporting their emission levels, causing a type II error to occur. The null hypothesis that the organizations’ industrial activities had no significant effect on pollution was being incorrectly retained. 

In this instance, a type II error occurred because of outdated and unreliable data. Once the true extent of volatile organic compound emissions was recognized, Houston implemented stricter controls, leading to a 50% reduction in ozone production rates over six years.²⁰

Statistical Significance 

To avoid committing either type I or type II errors, researchers try to determine statistical significance. Statistical significance means that it is unlikely that the data collected occurred due to chance, which allows researchers to either retain the null hypothesis or support the alternative hypothesis with confidence. In this article, our writer Annika Steele describes how statistical significance is calculated using the p-value and explores the impact statistical significance has on the way we interpret data and advance scientific knowledge.

The Mistake that Almost Half of Product Managers Make

Usually, a p-value of less than 0.05 is used as a threshold for determining if results are statistically significant. A 0.05 p-value means that there is just a 5% likelihood that the observed results, or something more extreme, could occur due to chance. While this guideline is a good metric for fields like science and the law, where committing a type I error has significant consequences, should it be used in other fields, like product design? In this article, our director, Turney McKee, suggests that it’s irrational for product managers to ignore results from user testing that produce a p-value above 0.05, and suggests they are making a type II error. 

1. Hayes, A. (2025, May 15). Type II error: What it means in statistics, plus an example. Investopedia. https://www.investopedia.com/terms/t/type-ii-error.asp
2. Whitaker, K. (n.d.). Quotes [Computer software]. GitHub. https://github.com/kwstat/quotes
3. Hayes, A. (2024, September 21). Null hypothesis. Investopedia. https://www.investopedia.com/terms/n/null_hypothesis.asp
4. Shreffler, J., & Huecker, M. R. (2023, March 13). Type I and Type II errors and statistical power. In StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing. Retrieved from https://www.ncbi.nlm.nih.gov/books/NBK557530/
5. Hayes, A. (2022, July 14). Beta risk: What it is, how it works, and example. Investopedia. https://www.investopedia.com/terms/b/beta-risk.asp
6. Stanford University. (2016). Lecture 6 — Simple alternatives and the Neyman-Pearson lemma. STATS 200: Introduction to Statistical Inference. Retrieved from https://web.stanford.edu/class/archive/stats/stats200/stats200.1172/Lecture06.pdf
7. Braithwaite, L. (2024, October 15). Experimental design. The Decision Lab.
8. NCSS, LLC. (2023). Fisher’s exact test for two proportions. PASS Sample Size Software. Retrieved from https://www.ncss.com/wp-content/themes/ncss/pdf/Procedures/PASS/Fishers_Exact_Test_for_Two_Proportions.pdf
9. Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231, 289–337. https://doi.org/10.1098/rsta.1933.0009
10. O'Connor, J. J., & Robertson, E. F. (n.d.). Ronald Aylmer Fisher. MacTutor History of Mathematics Archive. University of St Andrews. Retrieved June 9, 2025, from https://mathshistory.st-andrews.ac.uk/Biographies/Fisher/
11. The Editors of Encyclopaedia Britannica. (2025, April 12). Jerzy Neyman. Encyclopaedia Britannica. Retrieved June 9, 2025, from https://www.britannica.com/biography/Jerzy-Neyman
12. Pearson, E. S. (2017). On the history of the statistical approach to hypothesis testing. Amstat News. Retrieved from https://magazine.amstat.org/wp-content/uploads/2017/08/pearsonegon.pdf
13. Campagna, R., & Vignini, A. (2025). The role of xenobiotic caffeine on cardiovascular health: Promises and challenges. Journal of Xenobiotics, 15(2), 51. https://doi.org/10.3390/jox15020051
14. Lee, S. (2025, March 11). Exploring the effects of Type II error in scientific studies. Number Analytics. Retrieved from https://www.numberanalytics.com/blog/exploring-type-ii-error-scientific-studies
15. Rogers, T. (n.d.). Type I and Type II errors: Making mistakes in the justice system. Intuitor. Retrieved June 9, 2025, from https://www.intuitor.com/statistics/T1T2Errors.html
16. Hauer, E. (2004). The harm done by tests of significance. Accident Analysis and Prevention, 36(3), 495–500. https://doi.org/10.1016/j.aap.2003.09.008
17. Minitab Blog Editor. (2017, March 8). Which statistical error is worse: Type 1 or Type 2? Minitab Blog. Retrieved June 9, 2025, from https://blog.minitab.com/en/understanding-statistics/which-statistical-error-is-worse-type-1-or-type-2
18. Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017). The earth is flat (p > 0.05): Significance thresholds and the crisis of unreplicable research. PeerJ, 5, e3544. https://doi.org/10.7717/peerj.3544
19. Davar, D., Ding, F., Saul, M., Sander, C. S., Tarhini, A. A., Kirkwood, J. M., & Tawbi, H. A. (2017). High-dose interleukin-2 (HD IL-2) for advanced melanoma: A single center experience from the University of Pittsburgh Cancer Institute. Journal for ImmunoTherapy of Cancer, 5, 74. https://doi.org/10.1186/s40425-017-0279-5
20. Leven, R. (2018, February 6). The EPA’s pollution estimates stink. Everyone uses them anyway. WIRED. https://www.wired.com/story/the-epas-pollution-estimates-stink-everyone-uses-them-anyway