Calibrated Trust in AI: How to Measure Appropriate Reliance

Last updated November 8, 2026.

KPMG and the University of Melbourne surveyed more than 48,000 people in 47 countries for their 2025 global study of trust in AI. Fewer than half of respondents, 46%, said they were willing to trust AI systems. In the same study's section on work, about 2 in 3 employees said they rely on AI output without checking whether it is accurate.

Those two figures describe different things. Trust is what people believe about an AI system, and reliance is what they actually do with its output. At The Decision Lab, we measure the two separately when we evaluate an AI product or an AI program, because the gap between them shows a team where to act. The goal is calibrated trust: beliefs and behavior that match how well the system performs on a given task.

Our work on trust in AI products goes back to 2020, when we tracked Canadians' trust in contact tracing apps through weekly surveys for the COVI app built with Mila. This guide sets out the framework we use today.

Calibrated trust in AI, defined

Trust is a person's belief that an AI system is reliable enough to use for a given task. Reliance is what the person does with the AI's output: accepting it, checking it, changing it or setting it aside. Calibrated trust occurs when beliefs and behavior both line up with the system's real performance, so people rely on the AI where it is accurate and question it where it is not.

The two often move apart: a user may report high trust and still avoid the tool, or use it heavily without understanding where it fails. A survey would catch the first case and miss the second entirely.

Trust versus reliance

  • Trust: what a person believes about the AI, such as how accurate and how predictable they think it is.
  • Reliance: how often the person accepts or acts on the AI's output.
  • Appropriate reliance: reliance that matches the AI's real performance on a specific task, in a specific context.

Spencer Kohn and colleagues' review of how trust in automation is measured, published in Frontiers in Psychology in 2021, draws the same line. They treat reliance as a form of risk-taking that can happen without trust, for example when someone simply judges the odds. In the studies they reviewed, self-reported trust and observed behavior both responded to changes in how reliable a system was, but the two were not well calibrated with each other.

Appropriate reliance can mean any of four responses to an AI output, depending on the task:

  • Accept. An assistant summarizes an order history the customer can see on the same screen. The output is low-risk and easy to check at a glance.
  • Verify. An AI tool flags a clause in a supplier contract as non-standard. The flag is useful, and a reviewer should read the clause before acting on it.
  • Override. An AI triage tool ranks a case as low priority, but the case worker knows about a recent change the system has no data on. The person holds information the model lacks.
  • Escalate. An AI assistant recommends denying an insurance claim. The decision is consequential and hard to reverse, so it goes to a qualified reviewer whatever confidence the AI reports.

Where trust comes from: the precursors

Before trust forms, people judge whether the system deserves it. Roger Mayer, James Davis and David Schoorman's 1995 model of organizational trust, which much of the trust-in-AI research builds on, describes those judgments as perceptions of ability, benevolence and integrity. Work on trust in automation, including John Lee and Katrina See's review of designing for appropriate reliance in Human Factors, adds how predictable the system's behavior is. Kohn and colleagues use the Mayer model to keep these perceptions apart from trust itself and from the behavior that follows.

The person and the situation shape trust as well. A user's general willingness to trust technology, their expertise in the task and their past experience with the system all affect how much they rely on it, and so do the stakes of the decision. When we measure precursors, we capture both sides: what users believe about the system, and who those users are.

Those beliefs then have to be checked against the system's tested performance, since calibration is a comparison between the two. The US National Institute of Standards and Technology's AI Risk Management Framework lists the characteristics of a trustworthy AI system, starting with validity and reliability. Where a model reports its own confidence, a useful test is whether answers given at 80% confidence turn out to be right about 80% of the time. If they do not, even well-calibrated users are working from a misleading signal.

A measurement framework for calibrated trust

We evaluate calibrated trust across four dimensions. Each needs its own measures, and the most useful findings usually come from comparing them.

1. User beliefs

What to measure: perceived accuracy, predictability, transparency, fairness and usefulness.

How: validated survey scales, repeated over time. Kohn and colleagues identify the 12-item checklist for trust developed by Jiun-Yin Jian, Ann Bisantz and Colin Drury in 2000 as the most frequently cited self-report measure of trust in automation. They recommend validated instruments over custom questionnaires, and repeated measurement, since trust changes with experience. A reading taken just after users first see the system make a visible error is especially informative, since beliefs are most likely to shift then.

Beliefs also vary by group in ways demographics do not predict. For a leading online learning platform, we surveyed more than 800 students about how they use study tools as AI changes how they learn. Willingness to subscribe, an attitude, produced a cleaner and more predictive segmentation than any demographic. We segment users by how they approach an AI system before reading their trust scores as one average.

2. Observed reliance behavior

What to measure: acceptance rates, override rates, verification behavior, escalation behavior and how often users review outputs independently.

How: behavioral data from the product or from a test, always split by whether the AI was right. An overall acceptance rate cannot separate good reliance from bad. Two rates do most of the work:

  • Over-reliance rate: the share of incorrect AI outputs that users accepted. Lower is better.
  • Under-reliance rate: the share of correct AI outputs that users rejected. Lower is better.

Max Schemmer and colleagues at the Karlsruhe Institute of Technology proposed a more precise version in a 2022 paper on measuring appropriate reliance. Participants answer first on their own, then see the AI's advice. The measures track how often people switch to correct advice when their own answer was wrong, and how often they keep their own correct answer when the advice is wrong.

Splitting by correctness matters because some design features raise acceptance across the board. In experiments on AI explanations and team performance, Gagan Bansal and colleagues found that explanations increased the chance people accepted the AI's recommendation whether or not it was correct. A rising acceptance rate after a design change can therefore mean worse calibration.

3. Decision quality

What to measure: accuracy, error detection, consistency, speed, downstream outcomes and harm reduction.

How: compare AI-supported decisions with two baselines, the person working alone and the AI working alone. Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone's meta-analysis of human and AI combinations, published in Nature Human Behaviour, pooled 106 experiments. On average, people working with AI did better than people alone but worse than whichever of the two performed best by itself. A team that only compares against the human baseline would report a gain where the combination was actually losing accuracy.

4. Context and task risk

What to measure: the stakes of the decision, how reversible it is, the user's expertise, the consequence of an error and whether human support is available.

How: tag every task and decision in the evaluation with its risk level, then report the other three dimensions by tier. An over-reliance rate of 1 in 20 may be acceptable when users are checking an order status and unacceptable when they are approving a payment.

What to test before scale

Before an AI system reaches more users, we check six things:

  • Whether users catch incorrect AI output. Seeding known errors into a test set shows how many users detect them.
  • Whether users push back when the evidence behind a recommendation is weak.
  • Whether less experienced users over-rely more than experts. In a 2021 experiment on cognitive forcing, Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos found that designs asking people to decide before seeing the AI's recommendation reduced over-reliance compared with simple explanations, but the benefits went mainly to people who enjoy effortful thinking.
  • Whether users reject useful outputs because they do not understand the system. Berkeley Dietvorst, Joseph Simmons and Cade Massey's study of algorithm aversion found that people who saw a forecasting algorithm make mistakes became less willing to use it than a human forecaster, even after seeing it outperform the human.
  • Whether reliance patterns differ across user groups or decision contexts.
  • Whether final decisions improve, measured separately from speed and engagement.

A six-step evaluation sequence

  1. Define the task and the role the AI is allowed to play in it.
  2. Establish a human-only baseline on a fixed set of cases.
  3. Test AI-supported performance on the same cases, including cases where the AI is wrong.
  4. Measure trust, reliance, overrides and errors.
  5. Identify over-reliance and underuse patterns, broken down by user group and task risk.
  6. Redesign the experience and retest before wider deployment.

We build comparison groups into this sequence wherever we can. When we evaluated AI adoption programs for a Fortune 500 HR technology company, we piloted eight programs for one month with more than 100 employees against a control group. Employees' confidence in exploring AI tools, measured before and after the pilot, rose 41% in the pilot group and fell 26% in the control group over the same period.

Steps 3 to 6 are cheapest before launch. For a global consumer electronics company, every target behavior for a new mindfulness service was tested through experimentation before any production code was written, and design time fell 75% even though the process added research steps. For a digital wellness platform serving 25 million members, we ran the full recommendation engine against synthetic members in a validation harness, so design failures surfaced in an afternoon instead of months into production.

For practical ways to design transparency, user control, uncertainty cues and error recovery into customer-facing AI, read our product-design guide to building calibrated trust in AI products. To test how a model behaves under adversarial use before people rely on it, see our guide to red teaming a large language model, and for team-level decisions, our guide to evaluating human-AI collaboration before you scale.

Frequently asked questions

What is calibrated trust in AI?

Calibrated trust in AI means a person's beliefs about an AI system, and how much they rely on it, match what the system can actually do on a given task. People accept its output where it performs well and question or escalate it where it does not. The aim is accurate trust, and sometimes that means lower trust.

What is the difference between trust and reliance on AI?

Trust is what a person believes about an AI system, such as whether it is accurate and predictable. Reliance is what they do with its output, such as accepting a recommendation or overriding it. The two can diverge: people report low trust and still accept outputs unchecked, or report high trust and rarely use the tool.

How do you measure trust and its precursors in an AI model?

Measure precursors with validated surveys of how users perceive the system's ability, integrity, benevolence and predictability, alongside user traits such as expertise. Measure trust itself with repeated self-report scales. Then measure reliance through behavior, split by whether the AI was correct, and compare all of it with the system's tested performance.

How do you measure over-reliance on AI?

Track the share of incorrect AI outputs that users accept without changing them. To get enough incorrect cases, seed known errors into a test set. Report the rate by task risk and user group, and pair it with the under-reliance rate, the share of correct outputs users reject, so a fix for one does not hide the other.

Is high trust in AI a good sign?

High trust is a good sign only when the system has earned it for that task. In a system that is often wrong, high trust lets errors pass unchecked into decisions. We judge an AI product by whether acceptance tracks accuracy, so a lower average trust score paired with better error detection can be an improvement.

Sources

  1. Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. CHI '21.
  2. Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction.
  3. Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126.
  4. Gillespie, N., Lockey, S., et al. (2025). Trust, attitudes and use of artificial intelligence: A global study 2025. University of Melbourne and KPMG.
  5. Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000). Foundations for an empirically determined scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1), 53-71.
  6. Kohn, S. C., de Visser, E. J., Wiese, E., Lee, Y.-C., & Shaw, T. H. (2021). Measurement of trust in automation: A narrative review and reference guide. Frontiers in Psychology.
  7. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1).
  8. Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20(3), 709-734.
  9. National Institute of Standards and Technology (2023). AI Risk Management Framework (AI RMF 1.0).
  10. Schemmer, M., Hemmer, P., Kühl, N., Benz, C., & Satzger, G. (2022). Should I follow AI-based advice? Measuring appropriate reliance in human-AI decision-making. CHI '22 Workshop on Trust and Reliance in AI-Human Teams.
  11. Vaccaro, M., Almaatouq, A., & Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour.
  12. The Decision Lab. Changing how billions of phones communicate risk (COVI case study).
  13. The Decision Lab. Keeping students learning in the age of AI through behavioral segmentation.
  14. The Decision Lab. Increasing AI adoption across a 20,000-person workforce.
  15. The Decision Lab. Reducing product feature design time using simulation-first validation.
  16. The Decision Lab. How do you get 25 million people to follow clinically sound recommendations?

About the Authors

A man in a blue, striped shirt smiles while standing indoors, surrounded by green plants and modern office decor.

Dan Pilat

Managing Director

Dan is a Co-Founder and Managing Director at The Decision Lab. He is a bestselling author of Intention - a book he wrote with Wiley on the mindful application of behavioral science in organizations. Dan has a background in organizational decision making, with a BComm in Decision & Information Systems from McGill University. He has worked on enterprise-level behavioral architecture at TD Securities and BMO Capital Markets, where he advised management on the implementation of systems processing billions of dollars per week. Driven by an appetite for the latest in technology, Dan created a course on business intelligence and lectured at McGill University, and has applied behavioral science to topics such as augmented and virtual reality.

A smiling man stands in an office, wearing a dark blazer and black shirt, with plants and glass-walled rooms in the background.

Dr. Sekoul Krastev

Managing Director & Co-Founder

Dr. Sekoul Krastev is a decision scientist and Co-Founder of The Decision Lab, one of the world's leading behavioral science consultancies. His team works with large organizations—Fortune 500 companies, governments, foundations and supernationals—to apply behavioral science and decision theory for social good. He holds a PhD in neuroscience from McGill University and is currently a visiting scholar at NYU. His work has been featured in academic journals as well as in The New York Times, Forbes, and Bloomberg. He is also the author of Intention (Wiley, 2024), a bestselling book on the science of human agency. Before founding The Decision Lab, he worked at the Boston Consulting Group and Google.

Notes illustration

Eager to learn about how behavioral science can help your organization?