The Pitfalls of Behavioral Science: How to avoid untrustworthy data
There is a crisis in the field of behavioral science
The field of behavioral science has grown from obscure to mainstream. Like anything, it’s had growing pains. While the discipline has taken huge strides from its origins, the current crisis is one of evidence.
What does an evidence crisis look like? Well, some behavioral studies only use a narrow range of stimuli to deduce their findings. Others neglect whether their stimuli are generalizable to the real world, outside of clinical trials.1 And others still may have used methods of measurement that are later proven invalid.
Publication issues in behavioral science
Research paper retractions
The media star of the newest behavioral science crisis is the retraction of research papers. Retractions can occur for a number of reasons, including:2
- Mistakes (whether honest or careless) in the data collection process or research methods
- Results that cannot be replicated
- Misconduct, e.g. fabricating results
Retractions in behavioral science have been on the rise - with a faster growth rate than other scientific publications.5 Before their retractions, these studies may have been used as evidence to implement interventions or guide further studies.
The optimistic interpretation: it’s plausible that the reason more articles are being retracted is due to a greater emphasis on flagging errors, misconduct, and faulty research practices in recent years.
References
- Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54–69. https://doi.org/10.1037/a0028347
- Hantula, D. A. (2019). Editorial: Replication and Reliability in Behavior Science and Behavior Analysis: A Call for a Conversation. Perspectives on Behavior Science, 42(1), 1–11. https://doi.org/10.1007/s40614-019-00194-2
- Lilienfeld, S. (2017). Psychology’s Replication Crisis and the Grant Culture: Righting the Ship. Perspectives on Psychological Science, 12(4). https://journals.sagepub.com/doi/full/10.1177/1745691616687745
- Henrich, J., Heine, S. J., & Norenzayan, A. (2010b). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X
- Heine, S. J. (2010). Cultural psychology. In Handbook of social psychology, Vol. 2, 5th ed (pp. 1423–1464). John Wiley & Sons, Inc. https://doi.org/10.1002/9780470561119.socpsy002037
- Henrich, J., Heine, S. J., & Norenzayan, A. (2010a). Beyond WEIRD: Towards a broad-based behavioral science. Behavioral and Brain Sciences, 33(2–3), 111–135. https://doi.org/10.1017/S0140525X10000725
- Pashler, H., & Harris, C. R. (2012). Is the Replicability Crisis Overblown? Three Arguments Examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401
- Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202–217. https://doi.org/10.1037/teo0000137
- Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20. https://doi.org/10.1037/h0076157
- Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638
- Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31(2), 107–112. https://doi.org/10.1111/j.2044-8317.1978.tb00578.x
- Nuijten, M. B., van Assen, M. A. L. M., Veldkamp, C. L. S., & Wicherts, J. M. (2015). The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates. Review of General Psychology, 19(2), 172–182. https://doi.org/10.1037/gpr0000034
- IJzerman, H., Lewis, N. A., Przybylski, A. K., Weinstein, N., DeBruine, L., Ritchie, S. J., Vazire, S., Forscher, P. S., Morey, R. D., Ivory, J. D., & Anvari, F. (2020). Use caution when applying behavioural science to policy. Nature Human Behaviour, 4(11), 1092–1094. https://doi.org/10.1038/s41562-020-00990-w
- Technology Readiness Level. (2021, April 1). NASA; Brian Dunbar.http://www.nasa.gov/directorates/heo/scan/engineering/technology/technology_readiness_level
About the Authors
Lindsey Turk
Lindsey Turk is a Summer Content Associate at The Decision Lab. She holds a Master of Professional Studies in Applied Economics and Management from Cornell University and a Bachelor of Arts in Psychology from Boston University. Over the last few years, she’s gained experience in customer service, consulting, research, and communications in various industries. Before The Decision Lab, Lindsey served as a consultant to the US Department of State, working with its international HIV initiative, PEPFAR. Through Cornell, she also worked with a health food company in Kenya to improve access to clean foods and cites this opportunity as what cemented her interest in using behavioral science for good.
Dr. Sekoul Krastev
Sekoul is a Co-Founder and Managing Director at The Decision Lab. He is a bestselling author of Intention - a book he wrote with Wiley on the mindful application of behavioral science in organizations. A decision scientist with a PhD in Decision Neuroscience from McGill University, Sekoul's work has been featured in peer-reviewed journals and has been presented at conferences around the world. Sekoul previously advised management on innovation and engagement strategy at The Boston Consulting Group as well as on online media strategy at Google. He has a deep interest in the applications of behavioral science to new technology and has published on these topics in places such as the Huffington Post and Strategy & Business.
Sarah Chudleigh
Sarah Chudleigh is passionate about the accessible distribution of academic research. She has had the opportunity to practice this as an organizer of TEDx conferences, editor-in-chief of her undergraduate academic journal, and lead editor at the LSE Social Policy Blog. Sarah gained a deep appreciation for interdisciplinary research during her liberal arts degree at Quest University Canada, where she specialized in political decision-making. Her current graduate research at the London School of Economics and Political Science examines the impact of national values on motivations to privately sponsor refugees, a continuation of her interest in political analysis, identity, and migration policy. On weekends, you can find Sarah gardening at her local urban farm.
About us
We are the leading applied research & innovation consultancy
Our insights are leveraged by the most ambitious organizations
“
I was blown away with their application and translation of behavioral science into practice. They took a very complex ecosystem and created a series of interventions using an innovative mix of the latest research and creative client co-creation. I was so impressed at the final product they created, which was hugely comprehensive despite the large scope of the client being of the world's most far-reaching and best known consumer brands. I'm excited to see what we can create together in the future.
Heather McKee
BEHAVIORAL SCIENTIST
GLOBAL COFFEEHOUSE CHAIN PROJECT
OUR CLIENT SUCCESS
$0M
Annual Revenue Increase
By launching a behavioral science practice at the core of the organization, we helped one of the largest insurers in North America realize $30M increase in annual revenue.
0%
Increase in Monthly Users
By redesigning North America's first national digital platform for mental health, we achieved a 52% lift in monthly users and an 83% improvement on clinical assessment.
0%
Reduction In Design Time
By designing a new process and getting buy-in from the C-Suite team, we helped one of the largest smartphone manufacturers in the world reduce software design time by 75%.
0%
Reduction in Client Drop-Off
By implementing targeted nudges based on proactive interventions, we reduced drop-off rates for 450,000 clients belonging to USA's oldest debt consolidation organizations by 46%
