Questionable Research Practices in Behavioral Science, and How to Fix Them
In 2012, Shu, Mazar, Gino, Ariely, and Bazerman published a three-study paper titled, “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end.”1 The paper demonstrated that asking people to sign a statement of honest intent before providing information—for example, when submitting an insurance claim—can significantly decrease dishonesty, compared to when people sign such a statement after providing information.
Since its publication, the paper has received hundreds of citations, becoming influential in the study of dishonest behavior. However, a failed 2020 replication and a viral article posted on the behavioral science blog Data Colada in August 2021 had people questioning the results of the studies and the integrity of the researchers.2
Unsurprisingly, these revelations sent shockwaves across the entirety of behavioral science. This was because the article accused Dan Ariely, one of the leading academics in the field, of data fabrication (an allegation which he has strongly denied).
Why did the authors of the Data Colada post accuse only Ariely of data fabrication, even though he had four other co-authors? This was because Ariely was the only author responsible for the study in question (Study 3). They came to this conclusion because of a number of anomalies in Study 3’s original data set that are difficult to explain as results of anything but deliberate manipulation.
One notable discovery was that Study 3’s data, which purported to look at the number of miles driven by auto insurance customers, had a distribution so uniform as to be statistically impossible. The researchers behind the blog post also examined the spreadsheet containing Study 3’s data and concluded that many data points had been duplicated and lightly edited.
We still don’t know for sure what happened in this case, and all of the paper’s authors have stated that they were unaware of these anomalies and do not know how they came to be. The purpose of this article is not to summarize everything that has come to light through the Data Colada post, but rather to examine why this type of fraud (if indeed it is fraud) happens in academia, and to explore some structural changes that could improve research practices everywhere.
References
- Shu, L. L., Mazar, N., Gino, F., Ariely, D., & Bazerman, M. H. (2012). Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end. Proceedings of the National Academy of Sciences, 109(38), 15197-15200.
- Simonsohn, U., Simmons, J., & Nelson, L. (2021, August 17). [98] Evidence of Fraud in an Influential Field Experiment About Dishonesty. Data Colada. https://datacolada.org/98
undefined - Warner, J., & Clauset, A. (2015, February 23). The Academy’s Dirty Secret. Slate Magazine. Retrieved November 4, 2021, from https://slate.com/human-interest/2015/02/university-hiring-if-you-didn-t-get-your-ph-d-at-an-elite-university-good-luck-finding-an-academic-job.html
- Rawat, S., & Meena, S. (2014). Publish or perish: Where are we heading?. Journal of research in medical sciences: the official journal of Isfahan University of Medical Sciences, 19(2), 87.
- Hong, M., & Moran, A. (2019, February). An introduction to open science. American Psychological Association. Retrieved October 8, 2021, from https://www.apa.org/science/about/psa/2019/02/open-science.
- van der Zee, T., & Reich, J. (2018). Open Education Science. AERA Open.
- Aitkenhead, D. (2013, December 6). Peter Higgs: I wouldn’t be productive enough for today’s academic system. The Guardian. Retrieved October 8, 2021, from https://www.theguardian.com/science/2013/dec/06/peter-higgs-boson-academic-system.
About the Author
Shi Shi Li
Shi Shi is currently a graduate student studying behavioral and decision sciences at the University of Pennsylvania. She is interested in using insights from behavioral science to solve a wide range of problems facing our society today. When she is not reading the latest behavioral science papers, she enjoys painting and playing video games with her friends. She also holds bachelor’s degrees in economics and psychology from the University of Southern California.
About us
We are the leading applied research & innovation consultancy
Our insights are leveraged by the most ambitious organizations
“
I was blown away with their application and translation of behavioral science into practice. They took a very complex ecosystem and created a series of interventions using an innovative mix of the latest research and creative client co-creation. I was so impressed at the final product they created, which was hugely comprehensive despite the large scope of the client being of the world's most far-reaching and best known consumer brands. I'm excited to see what we can create together in the future.
Heather McKee
BEHAVIORAL SCIENTIST
GLOBAL COFFEEHOUSE CHAIN PROJECT
OUR CLIENT SUCCESS
$0M
Annual Revenue Increase
By launching a behavioral science practice at the core of the organization, we helped one of the largest insurers in North America realize $30M increase in annual revenue.
0%
Increase in Monthly Users
By redesigning North America's first national digital platform for mental health, we achieved a 52% lift in monthly users and an 83% improvement on clinical assessment.
0%
Reduction In Design Time
By designing a new process and getting buy-in from the C-Suite team, we helped one of the largest smartphone manufacturers in the world reduce software design time by 75%.
0%
Reduction in Client Drop-Off
By implementing targeted nudges based on proactive interventions, we reduced drop-off rates for 450,000 clients belonging to USA's oldest debt consolidation organizations by 46%
