Why do we overestimate the probability of success?
Optimism Bias, explained.
What is Optimism Bias?
The optimism bias refers to our tendency to overestimate our likelihood of experiencing positive events and underestimate our likelihood of experiencing negative events.
Where it occurs
Consider the following scenario: Tom is a bright and motivated entrepreneur who is in the midst of starting his own restaurant. There had previously been six failed business attempts in the building he purchased. Each venture was unable to get the necessary returns to stay afloat and were forced to close. However, Tom felt as though he had what it took to make this restaurant succeed. He was at the top of his class, full of big ideas, and understood the pulse of young city-goers.
Tom pours his time and financial resources into the venture, refusing to accept failure as an option or yield to potential shortcomings. A friend even tells him that the street layout and surrounding competition simply made it difficult for pedestrians to be drawn into the area. Yet Tom still feels that with his know-how, he can circumvent these issues.
However, like the businesses before his, Tom’s restaurant is not cultivating enough business. In this scenario, Tom exhibits the optimism bias. Even though the past six businesses failed and environmental factors were working against him, Tom overestimated his chances for success and showed overconfidence in his abilities as an entrepreneur.
Debias Your Organization
Most of us work & live in environments that aren’t optimized for solid decision-making. We work with organizations of all kinds to identify sources of cognitive bias & develop tailored solutions.
Research consistently supports that most of the population (estimated around 80%) exhibit an optimistic bias that arises in a variety of circumstances.1 However, as seen in Tom’s case, there are large costs to the optimism bias in our personal and professional lives. We overestimate our chances of success. We assume that projects will be completed in shorter amounts of time than they actually are. We assume our relationships will last longer than they realistically might. We assume that we will make more money than others. The optimism bias can encourage risky behaviors, like smoking, by causing us to ignore the potential for unwanted outcomes.2 It also can stop us from taking preventative measures, like buying insurance or using contraceptives.
It is necessary to have some optimism. Optimism encourages us to persevere, even in the face of hardship or rejection. It pushes us to believe in our own abilities. It nudges us towards focusing on the positive without becoming preoccupied with the negative. Yet, overall it is important to be aware of how our optimism can blind us to negative outcomes and result in poor decision making.
While over-optimism can negatively impact us each individually, aggregation of overoptimism on a larger level can have an exponentially devastating impact. We can see this in how optimism bias impacts financial markets. Cognitive neuroscientist and optimism expert Tali Sharot posits that the optimism bias was “one of the core causes of the financial downfall in 2008”.3 Financial analysts and investors had unrealistic expectations of financial growth and success. Banks continued to engage in high-risk decision making and contributed to the growing economic bubble and its ultimate crash.
The optimism bias also impacts our global response to climate change.4 We tend to focus on things we are looking forward to rather than negative events. We also tend to feel like bad things may happen to others, but not ourselves. With climate change, this can allow us to feel like the consequences of environmental disaster will not affect us personally.
We might acknowledge the imminence of climate change, but our ability to assume that we will come out unscathed often prevents serious action. This was proved in a study where psychology researchers found that generally, the optimism bias resulted in less environmental concern. They also found that in climate skeptics, optimism bias resulted in “less guilt, less perceived responsibility, and lower behavioral intentions”.5
Why it happens
In order to understand the effects of the optimism bias, it is important to understand where it comes from and why. By breaking down biases into their cognitive processes and exploring their harms and benefits, we have a better chance of learning how to avoid costly decision-making.
The optimism bias instills feelings of control. We generally want to feel as if we have control over our lives and our fates. Negative events like illness, divorce, or financial loss often threaten our plans or derail the predictions we have about ourselves. Optimism prevents us from lingering in these negative outcomes.
The optimism bias has evolutionarily adaptive functions
For many traits in humans, there are adaptive functions that can be traced back to the primitive evolutionary environment. If a trait promoted survival and furthered chances for reproduction, it would continue to be passed onto offspring. It might seem as though a realist might be more successful than an optimist, yet the widespread optimism in humans shows a clear functional benefit. Economists Heifetz and Spiegel simulated interactions in our evolutionary environment between individuals with different degrees of optimism.6 They concluded that when someone with a lack of optimism and someone with optimism are in conflict, the optimist drives the interaction. Their optimistic bias towards their outcome results in more aggression and dominance in the conflict.
Another area in which the optimism bias is adaptive is health, both physical and mental. Quantitative data on levels of optimism and pessimism in depressive patients found that the optimism bias was positively associated with low levels of depression.7 As chronic mental stress can take a physical toll on our bodies by overworking the nervous system, the optimism bias is associated with physical health as well. Additionally, optimism has been shown to encourage healthy eating and exercise.8 If we are more focused on the benefits of healthy habits (i.e. changes in physical appearance, better immune system, overall improved mood) and feel we are likely to meet our goals, it is easier to stay health motivated.
We adjust our beliefs more in response to positive occurrences
So how do we stay optimistic in the face of information that tells us our beliefs are false? Sharot and her colleagues sought out to answer this question. They performed a study where they asked participants to “estimate their likelihood of encountering different adverse life events (such as Alzheimer’s disease and burglary)”.9 After the participants gave their estimate, they were given the statistical likelihood and then asked to re-estimate their likelihood. The researchers found that if a participant’s initial estimate was lower than the true likelihood, their revised estimate would barely change. If a participant’s initial estimate was higher than the true likelihood, their revised estimate would go down a large amount. For example, if the subject said they were 10% likely to get cancer and the true statistic was 25%, they would barely change their answer. But, if the subject said they were 20% likely to get robbed and the statistic was 10%, they might change their answer to 5%. The subjects demonstrated how the optimism bias skews our judgment.
With this information, Sharot concluded that we have the tendency to update our beliefs regarding positive information much more than we do with negative information. Through selectively calibrating our expectations with positive events, we can maintain optimism even in the face of negativity.
Why it is important
All of us, including experts, are susceptible to the optimism bias. In situations where we have a lot at stake surrounding our success, the optimism bias can cause us to ignore important information that can make or break our outcome. It is necessary to have some optimism. However, the fate of our projects, our businesses, our economy, and our planet can be put at risk by the optimism bias. We must be able to recognize when our optimism is clouding our judgment at major costs, and there are tangible ways to do so.
How to avoid it
Nobel Prize-winning economist Daniel Kahneman has widely researched the optimism bias and proposes two different ways of mitigating its influence on our decision making: taking an outside view and a post mortem approach.10
Seeing scenarios from an outsider’s perspective
The optimism bias often causes us to overestimate our abilities or our control over our environment. We can all relate to what Kahneman labels “the planning fallacy”, where we assume that we will finish something much quicker than we actually do. He suggests combatting this tendency by taking ourselves outside of the scenario and looking for “base rates”. Base rates are existing statistics from relevant situations that provide quantitative data to anchor our judgements. Base rates can be the probability of an event occurring, the average time something takes, or whatever figure fits the situation–as long as the base rate is from existing data. Kahneman proposes taking an outside perspective through the following three steps:
- “Identify an appropriate reference class” – Look for a general category to put your task in. This could be grocery shopping, home remodeling, or a professional project.
- “Obtain statistics for this reference class” – Look for statistics on how long it takes to complete the type of task on average. These are your “base rates”
- “Use specific information about the case to adjust the baseline prediction” – If there are certain concrete things that you think are worth changing your predictions for, use your judgment to make prediction adjustments.
While these steps apply most directly to time management and planning, the use of base rates can be a major tool in grounding your expectations and combatting optimism bias.
Start at the end
Kahneman suggests the “premortem approach” as a tool for organizations to overcome the optimism bias. The premortem approach is an exercise for teams to predict potential areas of failure when beginning a project. Everyone on the team is instructed to imagine it is a year from the present and the project has failed. They are then instructed to write out what has gone wrong and why. By forcing team members to consider negative outcomes, we can resist the shortsightedness of overconfidence.
How it all started
The optimism bias was first encountered in a 1980 study by psychologist Neil Weinstein, although it was then labeled “unrealistic optimism”.11 Weinstein conducted an experiment with over 200 college students testing the following statement:
“People believe that negative events are less likely to happen to them than to others, and they believe that positive events are more likely to happen to them than to others.”
He tested this by asking the students to rate how much their chances of experiencing a certain event differed from their classmates. If more than half rated themselves below average for a negative event or above average for a positive event, it would serve as evidence for a widespread optimism bias.
Weinstein’s results supported his hypothesis. There were varying magnitudes for different questions, yet the overall conclusion was strong: We overestimate our chances of achieving things we want and underestimate our chances of experiencing misfortune. He posited that there were both cognitive and motivational explanations for these results. His cognitive explanation was that optimism is a mental protector from anxiety or constant distress about the future. His motivational explanation was that optimism served as an indicator of one’s desire to achieve or avoid certain outcomes, pushing individuals to act accordingly.
Example 1 - Clinical research
In clinical research, the optimism bias can lead to unwarranted belief in the efficacy of new therapies. Research shows that the optimism bias manifests in clinical research citations as studies mostly cite positive past research and under-represent research proving the treatment ineffective. Health science researchers Iaian Chalmers and Robert Matthews also suggest that the optimism bias is reflected in selectively reporting results of studies that shine a positive light on treatments, as well as the “early stopping of studies” that might have negative results.12 This is a dangerous trend. Biased results give patients, and their doctors, unrealistic health expectations.
Chalmers and Matthews discuss a study in 1990 that tested a new radiotherapy treatment for head and neck cancer. They surveyed clinicians on their expected outcomes. The clinicians unanimously believed there would be a reduced mortality rate of 30%. However, the results of the trial showed no evidence for a reduced mortality rate. Here we can see how the optimism bias can affect clinical research and lead to poor judgment, even in experts.
On the production side, corporate-funded research has been found to encourage optimism bias more-so than publicly funded research.13 As individuals, it is important to be wary of our own optimism bias, as well as the research industry’s optimism bias, when participating in medical research.
Example 2 - Student debt accumulation
It makes sense to not want to imagine a future full of debt. Consequently, we might underestimate how long it will take to pay off our student loans. Psychology researchers Seaward and Kemp conducted a study to investigate the rise in student debts in New Zealand.14
Seaward and Kemp interviewed over 200 psychology students on their expected post-graduation incomes, their expected debts, and their expected pay-back period. They found that the students, on average, expected to pay back their loans in 10 years yet government statistics showed that it generally took considerably more time. Students typically assessed their own chances of getting higher-paying jobs as much more than their classmates.
Through analysis of these results, Seaward and Kemp suggested that the optimism bias causes students to take on higher amounts of student loans, as the students expect higher incomes than they receive after graduation.
What it is
The optimism bias refers to our tendency to overestimate our chances of positive experiences and underestimate our chances of negative experiences. This can cause overconfidence in our professional ventures and personal life. It also contributes to global issues like the 2008 market crash and failure to act against climate change.
Why it happens
Optimism allows us to feel in control of our environment and in range of our goals. It yields adaptive advantages in conflict and health. We maintain unrealistic optimism even when the world tells us otherwise because we update our beliefs when we are exposed to positive events than with negative events.
Example 1 – How the optimism bias can affect clinical research
Clinical research, especially private-sponsored research, exhibits the optimism bias by failing to cite negative results. This causes clinicians and patients to have unrealistic expectations about the efficacy of new treatments.
Example 2 – How the optimism bias contributes to the accumulation of student debt
A study by Seaward and Kemp demonstrated that New Zealand college students had considerably higher expectations of their post-graduate incomes, and lower expectations for the length of time it would take to pay off their debt than the national averages. They posit that this optimism bias contributes to students amassing large amounts of debt
How to avoid it
Kahneman suggests two approaches to combatting the optimism bias. One approach is to take an outside view, meaning we should look at base rates for our estimates as if we are looking at someone else’s chances. Another approach for organizations is the premortem approach, where team members predict how a project could fail and then work backward to prevent these issues.