Statistical Significance

What is Statistical Significance? 

Statistical significance is a measure used in hypothesis testing to determine if the observed results in a study are unlikely to have occurred due to chance alone, typically assessed using a p-value threshold (e.g., p < 0.05). It provides confidence that the effect or relationship observed in the data is real and not random. Understanding statistical significance is crucial for making informed, evidence-based decisions in research and data analysis.

The Basic Idea

How many times in life have you noticed a strange phenomenon? Maybe the same song you were just humming came on the radio, you bumped into someone from your hometown in a completely different part of the world, or you’ve noticed that every time you have oatmeal for breakfast, you hit all the green lights on the way to work. While each of these could be a coincidence, wouldn’t it be nice to have a way to figure out whether or not they truly were? Or at least determine the odds that these events were due to chance alone? While it may not be entirely possible to calculate whether each of these personal situations was completely coincidental, the concept of assessing the likelihood that there is a force besides chance at play, known as statistical significance, can be applied to many other phenomena in the world. 

Statistical significance is used in many types of research to determine whether the results of a study are likely to be due to a specific factor rather than occurring by chance. Researchers use statistical tests to assess this, including calculating a p-value. The p-value is a measure that helps determine the likelihood of observing the obtained results if the null hypothesis (which suggests no effect or no difference between groups) is true. Basically, the p-value answers the question: “Is this result unlikely to be due to random chance?” If the p-value falls below a predetermined threshold, the result is deemed statistically significant. Usually, the threshold of 0.05 is used (meaning you’re looking for a p-value of 0.05 or lower) because it indicates less than a 5% probability that the observed results are due to chance.1 

However, statistical significance only speaks to the presence of an effect—not its magnitude or practical importance. Findings from a study could be statistically significant but have a negligible effect in real-world applications, either because the methodology was flawed, the effect size was too small, or some other factor. Although researchers should consider statistical significance in any study, many other considerations are needed to understand the practical implications of statistical results.2

“A statistical analysis, properly conducted, is a delicate dissection of uncertainties, a surgery of suppositions.”


— M. J. Moroney, statistician and author of Facts from Figures.

About the Author

A smiling woman with long blonde hair is standing, wearing a dark button-up shirt, set against a backdrop of green foliage and a brick wall.

Annika Steele

Annika completed her Masters at the London School of Economics in an interdisciplinary program combining behavioral science, behavioral economics, social psychology, and sustainability. Professionally, she’s applied data-driven insights in project management, consulting, data analytics, and policy proposal. Passionate about the power of psychology to influence an array of social systems, her research has looked at reproductive health, animal welfare, and perfectionism in female distance runners.

About us

We are the leading applied research & innovation consultancy

Our insights are leveraged by the most ambitious organizations

Image

I was blown away with their application and translation of behavioral science into practice. They took a very complex ecosystem and created a series of interventions using an innovative mix of the latest research and creative client co-creation. I was so impressed at the final product they created, which was hugely comprehensive despite the large scope of the client being of the world's most far-reaching and best known consumer brands. I'm excited to see what we can create together in the future.

Heather McKee

BEHAVIORAL SCIENTIST

GLOBAL COFFEEHOUSE CHAIN PROJECT

OUR CLIENT SUCCESS

$0M

Annual Revenue Increase

By launching a behavioral science practice at the core of the organization, we helped one of the largest insurers in North America realize $30M increase in annual revenue.

0%

Increase in Monthly Users

By redesigning North America's first national digital platform for mental health, we achieved a 52% lift in monthly users and an 83% improvement on clinical assessment.

0%

Reduction In Design Time

By designing a new process and getting buy-in from the C-Suite team, we helped one of the largest smartphone manufacturers in the world reduce software design time by 75%.

0%

Reduction in Client Drop-Off

By implementing targeted nudges based on proactive interventions, we reduced drop-off rates for 450,000 clients belonging to USA's oldest debt consolidation organizations by 46%

Read Next

Notes illustration

Eager to learn about how behavioral science can help your organization?