Tablet displays a presentation while a notebook with handwritten notes and diagrams lies beside it on a wooden table. Two black pens are placed near the tablet and notebook.

The Pitfalls of Behavioral Science: How to avoid untrustworthy data

There is a crisis in the field of behavioral science

The field of behavioral science has grown from obscure to mainstream. Like anything, it’s had growing pains. While the discipline has taken huge strides from its origins, the current crisis is one of evidence.

What does an evidence crisis look like? Well, some behavioral studies only use a narrow range of stimuli to deduce their findings. Others neglect whether their stimuli are generalizable to the real world, outside of clinical trials.1 And others still may have used methods of measurement that are later proven invalid. 

Publication issues in behavioral science

Research paper retractions

The media star of the newest behavioral science crisis is the retraction of research papers. Retractions can occur for a number of reasons, including:2

  • Mistakes (whether honest or careless) in the data collection process or research methods
  • Results that cannot be replicated
  • Misconduct, e.g. fabricating results

Retractions in behavioral science have been on the rise - with a faster growth rate than other scientific publications.5 Before their retractions, these studies may have been used as evidence to implement interventions or guide further studies. 

The optimistic interpretation: it’s plausible that the reason more articles are being retracted is due to a greater emphasis on flagging errors, misconduct, and faulty research practices in recent years.

References

  1. Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54–69. https://doi.org/10.1037/a0028347
  2. Hantula, D. A. (2019). Editorial: Replication and Reliability in Behavior Science and Behavior Analysis: A Call for a Conversation. Perspectives on Behavior Science, 42(1), 1–11. https://doi.org/10.1007/s40614-019-00194-2
  3. Lilienfeld, S. (2017). Psychology’s Replication Crisis and the Grant Culture: Righting the Ship. Perspectives on Psychological Science, 12(4). https://journals.sagepub.com/doi/full/10.1177/1745691616687745
  4. Henrich, J., Heine, S. J., & Norenzayan, A. (2010b). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X
  5. Heine, S. J. (2010). Cultural psychology. In Handbook of social psychology, Vol. 2, 5th ed (pp. 1423–1464). John Wiley & Sons, Inc. https://doi.org/10.1002/9780470561119.socpsy002037
  6. Henrich, J., Heine, S. J., & Norenzayan, A. (2010a). Beyond WEIRD: Towards a broad-based behavioral science. Behavioral and Brain Sciences, 33(2–3), 111–135. https://doi.org/10.1017/S0140525X10000725
  7. Pashler, H., & Harris, C. R. (2012). Is the Replicability Crisis Overblown? Three Arguments Examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401
  8. Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202–217. https://doi.org/10.1037/teo0000137
  9. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157
  10. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
  11. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31(2), 107–112. https://doi.org/10.1111/j.2044-8317.1978.tb00578.x
  12. Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015). The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates. Review of General Psychology, 19(2), 172–182. https://doi.org/10.1037/gpr0000034
  13. IJzerman, H., Lewis, N. A., Przybylski, A. K., Weinstein, N., DeBruine, L., Ritchie, S. J., Vazire, S., Forscher, P. S., Morey, R. D., Ivory, J. D., & Anvari, F. (2020). Use caution when applying behavioural science to policy. Nature Human Behaviour, 4(11), 1092–1094. https://doi.org/10.1038/s41562-020-00990-w
  14. Technology Readiness Level. (2021, April 1). NASA; Brian Dunbar.http://www.nasa.gov/directorates/heo/scan/engineering/technology/technology_readiness_level

About the Authors

Lindsey Turk's portrait

Lindsey Turk

Lindsey Turk is a Summer Content Associate at The Decision Lab. She holds a Master of Professional Studies in Applied Economics and Management from Cornell University and a Bachelor of Arts in Psychology from Boston University. Over the last few years, she’s gained experience in customer service, consulting, research, and communications in various industries. Before The Decision Lab, Lindsey served as a consultant to the US Department of State, working with its international HIV initiative, PEPFAR. Through Cornell, she also worked with a health food company in Kenya to improve access to clean foods and cites this opportunity as what cemented her interest in using behavioral science for good.

A smiling man stands in an office, wearing a dark blazer and black shirt, with plants and glass-walled rooms in the background.

Dr. Sekoul Krastev

Sekoul is a Co-Founder and Managing Director at The Decision Lab. He is a bestselling author of Intention - a book he wrote with Wiley on the mindful application of behavioral science in organizations. A decision scientist with a PhD in Decision Neuroscience from McGill University, Sekoul's work has been featured in peer-reviewed journals and has been presented at conferences around the world. Sekoul previously advised management on innovation and engagement strategy at The Boston Consulting Group as well as on online media strategy at Google. He has a deep interest in the applications of behavioral science to new technology and has published on these topics in places such as the Huffington Post and Strategy & Business.

A woman is smiling, wearing a blue cardigan over a striped shirt, standing against a concrete wall with green plants and dappled sunlight in the background.

Sarah Chudleigh

Sarah Chudleigh is passionate about the accessible distribution of academic research. She has had the opportunity to practice this as an organizer of TEDx conferences, editor-in-chief of her undergraduate academic journal, and lead editor at the LSE Social Policy Blog. Sarah gained a deep appreciation for interdisciplinary research during her liberal arts degree at Quest University Canada, where she specialized in political decision-making. Her current graduate research at the London School of Economics and Political Science examines the impact of national values on motivations to privately sponsor refugees, a continuation of her interest in political analysis, identity, and migration policy. On weekends, you can find Sarah gardening at her local urban farm.

About us

We are the leading applied research & innovation consultancy

Our insights are leveraged by the most ambitious organizations

Image

I was blown away with their application and translation of behavioral science into practice. They took a very complex ecosystem and created a series of interventions using an innovative mix of the latest research and creative client co-creation. I was so impressed at the final product they created, which was hugely comprehensive despite the large scope of the client being of the world's most far-reaching and best known consumer brands. I'm excited to see what we can create together in the future.

Heather McKee

BEHAVIORAL SCIENTIST

GLOBAL COFFEEHOUSE CHAIN PROJECT

OUR CLIENT SUCCESS

$0M

Annual Revenue Increase

By launching a behavioral science practice at the core of the organization, we helped one of the largest insurers in North America realize $30M increase in annual revenue.

0%

Increase in Monthly Users

By redesigning North America's first national digital platform for mental health, we achieved a 52% lift in monthly users and an 83% improvement on clinical assessment.

0%

Reduction In Design Time

By designing a new process and getting buy-in from the C-Suite team, we helped one of the largest smartphone manufacturers in the world reduce software design time by 75%.

0%

Reduction in Client Drop-Off

By implementing targeted nudges based on proactive interventions, we reduced drop-off rates for 450,000 clients belonging to USA's oldest debt consolidation organizations by 46%

Read Next

A group of people sitting around a wooden table during a meeting, with notepads and pens in hand, engaged in discussion.
Insight

Blue Cars, Robots, and How Your Brain Chooses to See

Drawing from a childhood anecdote about picking a unique blue car color, this story illustrates how our minds spotlight what we focus on, making the seemingly rare feel common. Discover how our perception shapes reality in this engaging exploration of top-down attention.

man facing a cork board filled with paper
Insight

TDL Perspectives: Becoming A Behavioral Scientist

How does one break into the field of applied behavioral science? This perspective piece explains what applied behavioral scientists actually do on a daily basis among other discussion points related to entering the industry.

Insight

Scaling Up Housing Innovation: Insights from Developers in Canada

How can housing innovations like prefabrication and modular construction truly scale up if developers are still hindered by high costs, perceived risks, and regulatory challenges? And with 3.5 million additional housing units needed by 2030, what will it take for policymakers and industry leaders to shift the tide and make these solutions viable?

Notes illustration

Eager to learn about how behavioral science can help your organization?