Why do scientists keep looking for a statistically significant result after failing to find one initially?

The 

Look-elsewhere Effect

, explained.
Bias

What is the Look-elsewhere Effect?

The Look-elsewhere Effect describes how, when scientists analyze the results of their experiments, results that are apparently statistically significant might actually have arisen by chance. One reason why this might happen is that a researcher has ignored a statistically insignificant result that they found previously, choosing to “look elsewhere”—continuing to search for a significant finding instead of accepting their initial results.

Where this bias occurs

Let’s say your friend David is a medical researcher who is trying to develop a drug that will help people recover from colds more quickly. He runs an experiment where he tests his new treatment, collects a bunch of data, and analyzes it using statistical tests. His analysis does not find any significant effect of the treatment on people’s recovery time.

Initially, David is disappointed—but then he decides that maybe the reason he didn’t find a significant result is that he’s just looking in the wrong place. After running a few different tests, he eventually finds a statistically significant effect: the treatment group reported fewer headache symptoms than the control group. Success!

Related Biases

Sources

  1. Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., … & Heikensten, E. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436.
  2. Engber, D. (2019, April 19). Think psychology’s replication crisis is bad? Welcome to the one in medicine. Slate Magazine. https://slate.com/technology/2016/04/biomedicine-facing-a-worse-replication-crisis-than-the-one-plaguing-psychology.html
  3. Goldman, M. (2008). Why is multiple testing a proble,? [PDF]. The University of California, Berkeley. https://www.stat.berkeley.edu/~mgoldman/Section0402.pdf
  4. Koehrsen, W. (2018, February 7). The misleading effect of noise: The multiple comparisons problem. Medium. https://towardsdatascience.com/the-multiple-comparisons-problem-e5573e8b9578
  5. Kahneman, D. (2011). Thinking, fast and slow. Macmillan.
  6. Ackerman, C. E. (2016, September 1). Pollyanna principle: The psychology of positivity bias. PositivePsychology.com. https://positivepsychology.com/pollyanna-principle/
  7. Lovallo, D., & Kahneman, D. (2003, July). Delusions of success: How optimism undermines executives’ decisions. Harvard Business Review. https://hbr.org/2003/07/delusions-of-success-how-optimism-undermines-executives-decisions
  8. Aronson, E., & Mills, J. (1959). The effect of severity of initiation on liking for a group. The Journal of Abnormal and Social Psychology, 59(2), 177-181. https://doi.org/10.1037/h0047195
  9. Larson, R. C., Ghaffarzadegan, N., & Xue, Y. (2014). Too many PhD graduates or too few academic job openings: the basic reproductive number R0 in academia. Systems research and behavioral science, 31(6), 745-750.
  10. Inzlicht, M. (2020, June 26). The replication crisis is not over. Michael Inzlicht. https://michaelinzlicht.com/getting-better/2020/6/26/the-replication-crisis-is-not-over
  11. Center for Open Science. (n.d.). Preregistration. https://www.cos.io/initiatives/prereg
  12. In praise of replication studies and null results, Nature 578, 489-490 (2019).
  13. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.
  14. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524-532.
  15. Yong, E. (2018, November 19). Psychology’s replication crisis is running out of excuses. The Atlantic. https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/
  16. Flender, S. (2019, July 28). The statistics of the improbable. Medium. https://towardsdatascience.com/the-statistics-of-the-improbable-cec9a754e0ff
  17. Dawid, R. (2015). Higgs discovery and the look elsewhere effect. Philosophy of Science, 82(1), 76-96.

About the Authors

A man in a blue, striped shirt smiles while standing indoors, surrounded by green plants and modern office decor.

Dan Pilat

Dan is a Co-Founder and Managing Director at The Decision Lab. He is a bestselling author of Intention - a book he wrote with Wiley on the mindful application of behavioral science in organizations. Dan has a background in organizational decision making, with a BComm in Decision & Information Systems from McGill University. He has worked on enterprise-level behavioral architecture at TD Securities and BMO Capital Markets, where he advised management on the implementation of systems processing billions of dollars per week. Driven by an appetite for the latest in technology, Dan created a course on business intelligence and lectured at McGill University, and has applied behavioral science to topics such as augmented and virtual reality.

A smiling man stands in an office, wearing a dark blazer and black shirt, with plants and glass-walled rooms in the background.

Dr. Sekoul Krastev

Sekoul is a Co-Founder and Managing Director at The Decision Lab. He is a bestselling author of Intention - a book he wrote with Wiley on the mindful application of behavioral science in organizations. A decision scientist with a PhD in Decision Neuroscience from McGill University, Sekoul's work has been featured in peer-reviewed journals and has been presented at conferences around the world. Sekoul previously advised management on innovation and engagement strategy at The Boston Consulting Group as well as on online media strategy at Google. He has a deep interest in the applications of behavioral science to new technology and has published on these topics in places such as the Huffington Post and Strategy & Business.

About us

We are the leading applied research & innovation consultancy

Our insights are leveraged by the most ambitious organizations

Image

“

I was blown away with their application and translation of behavioral science into practice. They took a very complex ecosystem and created a series of interventions using an innovative mix of the latest research and creative client co-creation. I was so impressed at the final product they created, which was hugely comprehensive despite the large scope of the client being of the world's most far-reaching and best known consumer brands. I'm excited to see what we can create together in the future.

Heather McKee

BEHAVIORAL SCIENTIST

GLOBAL COFFEEHOUSE CHAIN PROJECT

OUR CLIENT SUCCESS

$0M

Annual Revenue Increase

By launching a behavioral science practice at the core of the organization, we helped one of the largest insurers in North America realize $30M increase in annual revenue.

0%

Increase in Monthly Users

By redesigning North America's first national digital platform for mental health, we achieved a 52% lift in monthly users and an 83% improvement on clinical assessment.

0%

Reduction In Design Time

By designing a new process and getting buy-in from the C-Suite team, we helped one of the largest smartphone manufacturers in the world reduce software design time by 75%.

0%

Reduction in Client Drop-Off

By implementing targeted nudges based on proactive interventions, we reduced drop-off rates for 450,000 clients belonging to USA's oldest debt consolidation organizations by 46%

Let's chat

Read Next

Einstellung Effect

Why do our past experiences prevent us from reaching the best possible outcome?

Why do our past experiences prevent us from reaching the best possible outcome?

Learn More

Cognitive Dissonance

Why is it so hard to change someone's beliefs?

Why is it so hard to change someone's beliefs?

Learn More

False Consensus Effect

Why do we overestimate agreement?

Why do we overestimate agreement?

Learn More
See All
Notes illustration

Eager to learn about how behavioral science can help your organization?

Contact us