In the modern era, we often use automated tools on a day-to-day basis. When we get in a car to drive somewhere, most of us are quick to turn on Google Maps. We may ask our Alexa what the weather is outside rather than stepping out ourselves. We often turn to ChatGPT to help answer questions. Anytime we unquestioningly trust and follow the recommendations or decisions of automated tools, we may fall victim to automation bias. 

When we outsource our judgment, we can end up making bad decisions. While it can be relatively harmless to use automated tools to answer general knowledge questions, there are instances in which it can have detrimental effects. People often turn to Google to address health concerns—in 2024, one in three Americans searched symptoms online, and more than half were convinced by the diagnosis without seeking healthcare advice.²

With advancements in technology, you can even upload images to large language models and ask them to provide a diagnosis. Imagine uploading an image of a mole and being told that it’s a regular mole and there is no cause for concern, only to find out months later, during a regularly scheduled doctor’s visit, that it is cancerous. The automated tool provides this output based on the data it has available, such as other images of regular versus irregular moles, but it can’t match the quality of care that a practitioner can offer because it doesn’t know the specifics of your situation. You may not be asked if melanoma runs in your family, or be prompted to think about whether the mole has changed shape over the past couple of months. In this case, automation bias can prevent you from getting the diagnosis and treatment you need. 

While using automated tools can support decision-making, we must also verify outputs through human judgment and critical thinking. Automation has become increasingly intelligent with technological advancements, but human clinicians still draw on forms of judgment and pattern recognition that machines can’t easily replicate. When we encounter contradictory information from another source or something feels off in our gut, we shouldn’t ignore it. 

As more industries adopt artificial intelligence and automated decision-making, the consequences of overreliance on these tools ripple outward. When people in powerful or high-stakes roles defer to automated systems without sufficient scrutiny, the resulting errors can affect not only single outcomes but also institutional trust, equity, and safety. From healthcare and defense to finance, automation bias can quietly embed itself into the structures we depend on most, reinforcing existing inequalities and amplifying mistakes at scale.

Healthcare

In recent years, AI has been integrated into many aspects of healthcare to aid practitioners. Natural language processing tools can analyze medical records to prescribe medications, and other tools can help interpret complex medical images. While AI has the potential to reduce the impact of human error, make treatment more effective, and reduce the workload for doctors, it also carries the risk of automation bias. 

In healthcare, there are two types of errors: omission errors, where a necessary action is not taken, and commission errors, where an incorrect action is taken. Both can be dangerous and harmful to patients. If doctors come to rely on automated tools, they can make either one of these mistakes, as they ignore evidence to the contrary. 

In a 2017 study exploring the impact of using clinical decision support systems (CDSS), software that analyzes data from electronic health records, researchers tested how these systems influence decision-making. The CDSS would intentionally output incorrect answers so that researchers could determine whether medical students would question its output or follow its advice. They found that participants were more likely to make mistakes in prescribing when they used the CDSS compared to when they didn’t. Using a clinical decision support system increased prescribing errors by 56.9% when the system gave incorrect recommendations, compared to when clinicians relied on their own judgment without the tool. Although other research shows that CDSS tools can be more accurate than human decisions, when they do make a mistake, it is up to the medical professional to intervene and actively work against automation bias by questioning the output.³

Military & Defense

The military utilizes various automated systems to inform its decision-making. Tools are used to detect, track, and engage targets, thereby speeding up response time. This technology also enables human personnel to remain safe in dangerous environments, as they don’t need to be on the scene. While people often overlook these systems, due to automation bias, they may not override the system even when they think something may be wrong.⁴ 

Within any system, there is inherent risk. These tools may incorrectly identify targets and deploy force, or they can fail to detect danger and not trigger the necessary alerts and procedures. There are several documented cases where drones, which use automated systems to navigate and sensors to track targets, have erroneously killed civilians. Although these weapons require a human to make a final decision, operators often trust the sensors.⁵

Automated tools are also frequently used by the police to aid in identifying criminals. Surveillance tools often capture video or photo images of a crime, which can be input into facial recognition technology to return a match. Sometimes, it can pull up incorrect matches. Such was the case in 2018, when the Detroit Police Department arrested Robert Williams for shoplifting based on the facial recognition technology search, despite the fact that Williams was not near the store at the time of the incident. This case of automation bias highlights the importance of seeking additional evidence beyond technology to confirm or refute its outputs.⁶ 

Financial Forecasting

Finance is another sector where automated systems are often used. Automated tools help analyze vast amounts of data, including stock prices, consumer behavior, market indicators, and global events, to identify trends and predict future outcomes. These predictions are then used to guide investment and trading decisions.

Although automated tools enable a greater amount of data to be analyzed at a rapid pace, the algorithms are often trained on biased data, which can lead to incorrect or unfair assumptions. For example, research has shown that Black credit applicants often require a credit score approximately 120 points higher than white applicants to be approved by an AI tool. That’s because historically, Black applicants were more likely to be unjustifiably denied loans due to discriminatory practices, and as the AI tool is making predictions based on past data, it identifies that people of certain demographics as riskier borrowers.⁷

That’s why financial experts should not assume that AI predictions are objective. Instead, they should investigate how the tool arrives at its decision and examine the quality of its input data. There are many tests that financial institutions can use to check for bias, including conducting regular audits, to mitigate the risks of automation bias.

Ultimately, automation bias exists because we trust the output of automated technology too much. It’s not surprising that we trust automation—in most cases, it is more reliable. 

Humans are also prone to biases in decision-making, which can impact the quality of our decisions. High cognitive load, emotions, and fatigue can all impact our decisions; however, we often perceive automated tools as being free of these variables. But, automated systems aren’t immune to error—they reflect the limitations, assumptions, and biases of the data and people who design them. When we fail to recognize that these tools can make mistakes or inherit human bias, our trust in them becomes misplaced.

Often, people perceive automated systems as being superior to humans, which can lead us to view these tools as a kind of authority. Due to authority bias, we tend to follow instructions or accept information as true if we perceive it as coming from a source that is viewed as an expert.

Anchoring bias can also contribute to automation bias. We tend to rely heavily on the first piece of information we encounter as a starting point, which skews how we interpret subsequent information. We often use automated systems to generate initial recommendations, but this can become a psychological anchor that we hold onto even when new information contradicts it. Moreover, confirmation bias can also exacerbate automation bias. We tend to focus on information that confirms our preexisting beliefs. If automated systems generate our initial belief about a particular issue or challenge, even if we try to seek further evidence, we’ll give greater weight to information that confirms what the system generated. 

There’s also a sense of safety in following the advice provided by an automated tool. In high-stakes environments, such as in healthcare or military defense, automation is often relied on, and people do not want to be responsible for making a mistake. If they choose to override the system’s recommendation, they are likely to feel a much greater sense of responsibility if it was the wrong decision than if they just followed the tool’s recommendation. 

Moreover, many people do not fully understand how automated systems work or arrive at their outputs. It’s hard to understand where there may be errors in the system, its limitations, or what biases are embedded in its processes. If we don’t fully understand the system, it’s difficult to identify where it went wrong.⁸

While using automated systems to simplify complex decisions and help us overcome some of our own cognitive shortcuts can be effective, when this shifts towards an illusion of certainty, we overlook the potential flaws and fail to verify outputs. 

As automated tools become increasingly integrated into various sectors, including those like healthcare and aviation, where decisions can have life-altering consequences, automation bias can lead to serious consequences. Our undue trust in automated tools causes us to become complacent and not take the time to double-check their outputs—which can cause a misdiagnosis, a plane to approach a runway dangerously, or a drone to attack a civilian.

Automation bias causes us to let these tools guide us rather than assist our decision-making, eroding our sense of agency. If it occurs for a prolonged period, it can lead to skill degradation and a reduced ability to think critically.¹ Already, we can see ways that our society’s reliance on technology has impacted our cognitive capabilities. The Google effect describes our tendency to forget information that is readily available on the internet, as we don’t take the time to commit it to memory since it’s easy to access online. While that may not seem like a big deal, what happens when technology fails? If a doctor relies on CDSS too often, they may lose the ability to review health records to make decisions.

On a systemic level, automation bias can reinforce inequalities, as we explored when credit-scoring algorithms replicate historical patterns of racial discrimination, or surveillance tools unfairly identify people from particular neighborhoods or income brackets. Bias is embedded within the data that automated tools use to make predictions, which means it’s up to us to adjust the algorithms and their outputs to account for progress. 

Recognizing that automated systems are tools, not infallible experts, is key to balancing efficiency with accountability and ensuring that human judgment remains an active part of decision-making.

Understanding automation bias is not about rejecting technology—it’s about using it wisely. Automated systems are designed to enhance human performance, not replace it. 

Critical thinking and skepticism should be encouraged in the workplace. Leaders should encourage team members to speak up if a certain output doesn’t feel right to them. They should also seek out diverse perspectives. Just as people often seek a second opinion in the medical field to confirm a diagnosis, humans should provide a “second opinion” to automated outputs.⁸

Transparency is also an important factor in reducing the risk of automation bias. While it’s possible to explain to users how an AI system arrives at its output, we can also build more transparent automated tools. Although there were a lot of challenges with IBM’s AI CDSS, which caused them to retire it, one thing that it did well was offering detailed explanations for its treatment recommendations. Clinicians could therefore evaluate its inputs and processes to determine if the output made sense.⁸

The evaluation of automated outputs should also be built into a review process. While it may be inefficient to review every piece of data an automated tool produces, audits should be conducted regularly. Audits can help expose hidden errors and biases, especially for “black box” automated systems where there is a lack of transparency about how they arrive at a decision. If an error is identified, adjustments can be made to the tool to improve its reliability. 

Overall, it’s important to balance trust and skepticism when it comes to our use of automated tools. If we are too skeptical, we’ll fall victim to algorithm aversion, where we reject decisions made by algorithms completely. This can lead to missed opportunities for more reliable and efficient decision-making processes. On the other hand, if we trust them too much, then we risk being blinded by automation bias and failing to identify errors. Achieving this balance means staying engaged—using automated tools as partners in decision-making rather than unquestioned authorities, and combining their efficiency with our own critical judgment.

Automation bias was first identified in research in the 1990s, examining its effects on industries such as aviation and nuclear energy. While the tools helped to reduce human error and, therefore, the overall volume of errors, the number of mistakes increased in tasks where humans interacted with the tools. While automation did a great job of taking over routine tasks, when it was applied to more complex problems involving human decision-making, people were more likely to make mistakes because they trusted the automation too much.⁴ 

In 1999, researchers Mica R. Mosier and Linda J. Skitka coined the term automation bias after a series of flight simulation studies revealed that pilots performed worse when using an automated tool that provided recommendations than when they relied on their own knowledge and expertise. Eighty undergraduate students participated in tasks that simulated the types of monitoring and tracking tasks involved in flying a commercial aircraft, such as location monitoring or maintaining their aircraft's alignment with a moving target. One group of students had an automated computer system that made recommendations, while the other did not. 

Mosier and Skitka found that participants in the automated group missed things that the system didn’t alert them to, resulting in only 59% task accuracy, compared to a 97% accuracy for participants in the non-automated group. Participants in the automated group also made commission errors, where they followed the computer’s advice even though it was wrong. On average, each participant made 3.92 commission errors across six tasks, and 23.1% of participants made a commission error in each task.⁹

Automated systems are trained on data, which means that their outputs will carry forward any flaws in their inputs. If people are not reviewing both their inputs and outputs, then mistakes in the system get reinforced instead of corrected.

For example, if a self-driving car that relies on an automated system misclassifies certain types of objects—for example, mistaking a paper bag for a small animal—it would probably respond by reducing speed or braking. If engineers don’t notice this error and fix it, then when the self-driving car hits the market, it would continue this response to its input. Alternatively, if engineers notice the error, they can adjust the algorithm to correctly identify paper bags and ensure the car adjusts accordingly. 

Many cars today are equipped with automated elements such as adaptive cruise control, lane departure warnings, and parking assists. These tools are supposed to help a driver be safe on the roads. However, if people come to rely on these tools too much, they may not pay attention and fail to intervene when they malfunction. 

In 2023, researchers tested how often people experienced automation bias when partially automated car systems malfunctioned. Thirty-two participants were tested in two situations:

• Cornering with system limits: the car automatically started turning its wheel, causing the car to deviate from the lane into an oncoming lane, while on a curving country road. 
• Phantom braking with automatic emergency braking: the car would brake as if a collision was coming, but in these scenarios, there was no danger of collision

In the first scenario, only one out of the 32 participants deactivated the automatic steering wheel, showing that the other participants placed too much trust in the system. Although the other participants did try to overtake steering manually, they only did so after the car crossed the center line and only once a system warning message was given. This demonstrates that because the participants trusted the system, they didn’t pay enough attention to correct its error as soon as it started happening. 

In the second scenario, when the car braked for no reason, 28% of participants did not take any action. A total of 22% of participants pushed the brake, 28% put their foot on the gas but didn’t press it, and 22% actually pressed the accelerator to counteract the phantom braking. In this scenario, automation bias resulted in a commission error. When participants were later asked how they realized they should take action, 81% said it was because of a visual or sound warning—once again relying on automated tools.¹⁰

In 2010, Enbridge experienced an oil spill in Michigan, which resulted in crude oil entering the Kalamazoo River. Evidence suggests that this mistake may have occurred due to automation bias. 

In July 2010, operators in Edmonton ignored alarms that signaled to them that they should shut down the pipeline. On their supervisory control system, the situation was displayed as normal. The operators trusted the system and guessed that they were false alarms, continuing to pump oil. It took 17 hours for the operators to hear that oil was being spilled after reports came in that the area smelled like oil. By that time, almost 1 million gallons of crude oil had spilled into nearby wetlands and rivers, causing significant damage to the ecosystem. 

People have suggested that the spill occurred due to automation bias. Operators did not critically evaluate all signals, relying too much on and trusting the supervisory control system. This caused them to ignore the alarms, which were telling them to shut down the pipeline due to a rupture.¹¹ 

What it is

Automation bias is our tendency to trust outputs from automated systems, like AI tools or decision-making software, over our own judgment—even when these outputs are incorrect.

Why it happens

It occurs because people perceive automated systems as more reliable and authoritative than humans, and cognitive biases like authority, anchoring, and confirmation bias make us less likely to question them.

Example #1 – Driving with Partially Automated Systems

In tests with partially automated cars, drivers often failed to correct the system when it malfunctioned, such as during lane deviation or phantom braking, waiting instead for warnings before taking action. 

Example #2 – Automation Bias Led to Pipeline Disaster

In 2010, Enbridge operators ignored alarms indicating a pipeline problem because the supervisory system showed a normal status. Their overreliance on automation delayed intervention, resulting in the spill of nearly 1 million gallons of crude oil into Michigan’s rivers and wetlands.

How to avoid it

We should treat automation as a tool rather than a guide, maintain a healthy amount of skepticism, apply our critical thinking skills to evaluate its outputs, and regularly audit systems to identify errors and biases.

How to Preserve Agency in an AI-Driven Future

Not only is agency important to avoid automation bias, but also provides a sense of fulfillment. Our ability to make meaningful decisions is what makes us human. In this article, our writer Dr. Sekoul Krastev explores why agency is important to our well-being and how it can be preserved as automation becomes more and more embedded in our society. 

Why Machines Will Not Replace Us

Automated tools and systems can still make mistakes, which is why “human” skills like critical thinking are so important. Although people are worried that AI is going to take over our jobs—and the world—in this article, our writers Danny Goh, Terence Tse, and Mark Esposito explore why machines won’t replace us. Humans are unique and have unique capabilities that will continue to be valued.