How to Run Red Teaming on an LLM: Behavioral Tests for Sycophancy and Harmful Reliance

Last updated October 30, 2026.

In a study of 11 large language models published in Science in March 2026, Myra Cheng and colleagues compared the models' advice on personal conflicts with advice from people on the same situations. The models endorsed the user's own actions about half again as often as human advisers did, according to the Stanford-led study of sycophantic AI.

Running red teaming on an LLM for this kind of failure means testing the model through realistic multi-turn conversations with simulated and real users, and measuring how it changes what people believe and do. This approach is often called behavioral red teaming. At The Decision Lab, we apply behavioral science and psychometrics to AI trust and safety evaluation, with tests for sycophancy and harmful reliance and measures of whether users' trust tracks a model's actual accuracy. We also study human-AI coupling, the feedback loop that forms when a user and a model shape each other over many conversations. We work with safety teams and researchers who need evidence of how a model affects its users.

Standard red teaming looks for prompts that make a model produce content it should refuse, such as weapons instructions or hate speech. Behavioral red teaming looks for responses that each seem acceptable on their own but, over a conversation, push a user toward a worse decision or an unhealthy pattern of use.

Lujain Ibrahim and co-authors make the case for this kind of testing in their paper on interactive evaluations for interaction harms. They argue that single-turn tests miss harms such as emotional dependence and overreliance for two reasons. They look at isolated outputs instead of whole conversations, and they measure what the model says instead of what changes in the person using it.

OpenAI's experience with a GPT-4o update in April 2025 shows the gap in practice. In its account of the sycophancy rollback, the company says its offline evaluations looked good and users in a small A/B test liked the new version, yet the model had become noticeably more flattering and agreeable. No deployment evaluation tracked sycophancy, and the update was rolled back within days of launch. OpenAI has since said it will treat problems with model behavior as launch-blocking.

Behavioral evaluations of LLMs are run by four kinds of team.

1. Model developers' safety and evaluation teams, which test models before release and after each update.
2. Government AI safety institutes and independent evaluation organizations, which test models from outside the companies that build them.
3. Academic researchers in psychology and human-computer interaction, who study how model behavior affects trust and decision-making. Every study cited in this guide comes from groups like these.
4. Applied behavioral science teams such as The Decision Lab, which design user studies and controlled experiments to test whether a model changes what people believe or do.

The right partner depends on the risk you are testing for. Security red teams test whether a model can be pushed into producing prohibited content, while behavioral evaluation teams test whether its everyday behavior builds unhealthy reliance or distorts users' judgment over time.

1. Factual sycophancy. The model changes a correct answer, or holds back a correction, to agree with the user.
2. Social sycophancy. The model validates the user's actions and self-image when an honest adviser would push back.
3. Miscalibrated trust. Users trust the model more than its accuracy warrants, often because agreement feels like reliability.
4. Harmful reliance. Users hand over decisions or emotional needs they would be better off keeping, and come back for more.

1. Factual sycophancy

In SycEval, a benchmark for LLM sycophancy, Aaron Fanous and colleagues pushed back on answers from three widely used models to math and medical questions. In about 1 in 7 cases, a model dropped a correct answer for an incorrect one after the user disagreed. Once a model had given way, it kept the new position in close to 4 in 5 of the follow-up exchanges.

2. Social sycophancy

The Science study by Cheng and colleagues also ran experiments with more than 2,400 participants. After discussing a real conflict with a sycophantic model, participants became more convinced they were in the right and less willing to apologize or repair the relationship than participants who talked with a less agreeable model. The harm shows up in what the person does next, so it cannot be found by grading the model's text alone.

3. Miscalibrated trust

In the same experiments, participants rated sycophantic responses as more trustworthy, and rated sycophantic and non-sycophantic models as equally objective. Users could not tell when they were being flattered. Trust is well calibrated when it tracks how reliable a system actually is, a standard John Lee and Katrina See set out in their review of trust in automation and appropriate reliance. Training can work against that standard. Mrinank Sharma and colleagues found that both human raters and the preference models trained on their judgments sometimes favored a convincing sycophantic answer over a correct one in their study of why language models become sycophantic.

4. Harmful reliance

A four-week randomized study by researchers at the MIT Media Lab and OpenAI followed 981 participants who used ChatGPT every day. In their study of the psychosocial effects of chatbot use, participants who used the chatbot more each day reported more emotional dependence on it and less time socializing with other people. These are associations within the study rather than proof that heavier use caused the change, and they are the outcomes a behavioral red team should measure directly.

This is the sequence we would follow, adapted from the way we design and test interventions in other digital products.

1. Define each harm as a change in the user. Write it in terms you can observe, such as "the user abandons a correct belief after two turns of agreement" or "the user becomes less willing to seek a second opinion". A definition that only describes model text will miss the risks above.
2. Build user profiles from the behavioral literature. Vary the traits that change how people respond to advice: expertise in the topic, confidence in their own view, emotional state and vulnerability. A model that holds up with a confident expert can fold with a distressed beginner.
3. Write multi-turn scenarios with pressure points. Include pushback on a correct answer, a request to approve a decision already made, gradual emotional disclosure and attempts to hand a decision over entirely. Most of these failures appear several turns in, so single prompts are not enough.
4. Run simulated users at scale, then test with people. Simulated users find candidate failures cheaply across thousands of conversations. A smaller study with real participants then confirms whether those failures change what people believe or do, as Ibrahim and co-authors recommend.
5. Measure the model and the user. On the model side, track how often it changes a correct answer under pushback and how often it endorses a harmful plan. On the user side, track changes in decision quality and in willingness to consult anyone else, and compare each user's trust in the model with its actual accuracy on their questions.
6. Compare against a baseline. Test the candidate model against the current version or a deliberately less agreeable variant, with users assigned at random. Set your hypotheses and thresholds in advance. Without a comparison, a validation rate of any size is hard to interpret.
7. Set launch criteria and keep testing after release. Agree before testing which results would block a launch, then repeat the core scenarios after every update. OpenAI's 2025 rollback came from an update that passed its existing checks, so these tests need to keep running after deployment.

The second half of step 5 is where most teams have the least experience. Measuring a change in someone's confidence or intentions needs validated questions and enough participants to detect a realistic effect, and those are standard parts of behavioral research design.

Human-AI coupling is the feedback loop that forms when a model's conversational behavior and a user's beliefs or emotional needs start to reinforce each other over repeated conversations. Sycophancy is the most direct route into it. The model validates the user, the user trusts the model more, the model becomes a bigger source of reassurance or advice, and the user brings it more decisions to approve.

Ibrahim and co-authors describe these harms as compositional, meaning each response can look harmless while their combined effect shifts what the user believes. An October 2026 preprint by Deepthika S, Pawni Yadav and B. Sankar shows the loop working from the user's side too. In their study of confident LLMs and human deference, 32 participants learned science topics from assistants that stated planted falsehoods with full confidence. On a later quiz, participants chose the planted false answer on about 6 in 10 falsified items, compared with about 1 in 10 items the assistant had taught correctly, and their trust ratings barely changed after the task.

The study is small and uses one model on one learning task, so it shows the pattern without settling how common it is. Its authors conclude that trustworthiness belongs to the human-AI pair, and that evaluations should measure the user's deference to the model as well as the model's agreement with the user. We design behavioral red teams on the same basis: a scenario passes only when the model's answers and the user's resulting beliefs both hold up, and the tests include confidently wrong models as well as agreeable ones.

Psychometrics, the science of measuring psychological traits reliably, gives evaluators a tested way to describe a model's personality and check whether that description means anything. In a psychometric framework for evaluating and shaping LLM personality, published in Nature Machine Intelligence in December 2025, Gregory Serapio-García and colleagues at the University of Cambridge and Google DeepMind gave adapted Big Five personality questionnaires to 18 models. Larger instruction-tuned models produced reliable profiles that predicted their behavior in other tasks, and prompts could shift each trait across nine levels with effects that carried into tasks such as writing social media posts.

A 2026 preprint adds an important limit. Rafal Kocielnik and colleagues tested 11 frontier models in their study of when LLM self-reports predict behavior. A model's answers about a specific behavior predicted what it then did in the same conversation, but broad personality scores did not. For behaviors that depend heavily on the conversation, sycophancy among them, the link between self-report and behavior broke down across conversations, and persona prompts made the self-reports more consistent without changing the behavior.

We use questionnaire scores to describe how a model presents itself and to track changes between versions, and we measure sycophancy and reliance directly in conversation. Our framework puts that into six steps, drawing on the methods psychometrics uses to show that a measure is reliable and predicts real behavior.

1. Define the trait precisely. Name one behavior, such as giving way under user pressure or approving a harmful plan, in place of a broad personality dimension.
2. Check reliability. Confirm that the model produces stable scores across equivalent prompts and across separate sessions.
3. Check construct validity. Confirm that the measure captures the intended trait and not side effects such as answer length or the wording of the prompt.
4. Check behavioral validity. Confirm that the score predicts what the model does in realistic multi-turn scenarios, the test broad personality scores failed in the Kocielnik study.
5. Test the effects of any shaping. When prompts or fine-tuning are used to shift a trait, measure the intended change in behavior and look for unintended effects on capability or safety.
6. Monitor drift between versions. Re-run the same measures after every model or memory update, since a change elsewhere in training can move a trait nobody targeted.

Applied behavioral science adds something output-focused testing lacks: a way to measure whether a model's behavior changes the judgment and reliance of the people using it.

At The Decision Lab, that measurement is the core of our work on digital products and services. Two published projects show the methods that carry over to model evaluation.

• Trust in an AI risk score. For Mila's COVI contact-tracing app, whose AI model gave each user a graded COVID-19 risk score, we ran weekly surveys of Canadians on trust in tracing apps and privacy concerns, and built a decision model that put comprehension and user wellbeing at the center of the product. The work contributed to Apple and Google moving their exposure notification system from yes-or-no alerts to multi-level risk. Read the COVI case study on communicating AI-generated risk.
• Whether safety guidance gets trusted. For a global ride-sharing platform, we combined 31 executive interviews with surveys of more than 1,350 drivers and riders and identified nine factors behind negative safety interactions. A central one was whether drivers believed the safety guidance served them or the company. Read the ride-sharing safety case study.

We apply the same toolkit to LLMs. We treat trust as something with distinct precursors that can each be measured, and we build user profiles from decades of research on advice-taking and automation. A behavioral LLM evaluation with us can include:

• a library of multi-turn scenarios built around the risks your model is most likely to create
• simulated-user testing across thousands of conversations
• preregistered experiments with real participants and control groups
• trust-calibration measures that compare users' trust with the model's accuracy
• comparisons between model versions on the same scenarios
• launch thresholds tied to user outcomes

The deliverable is evidence about what a model does to the people using it, in a form a safety team can rerun after every update.

What is sycophancy in AI?

Sycophancy is a model's tendency to tell users what they want to hear. It covers factual sycophancy, where a model drops a correct answer under pushback, and social sycophancy, where it validates a user's actions or self-image. Research links it partly to training on human ratings, since people often prefer answers that agree with them.

What is the difference between LLM red teaming and behavioral evaluation?

LLM red teaming usually means adversarial testing for unsafe or prohibited outputs, often run by security specialists. Behavioral evaluation tests whether a model changes users' decisions or patterns of reliance, even when no single response looks unsafe. The two need different evidence: red teams log failing outputs, while behavioral evaluations need data from the people using the model.

What is human-AI coupling in LLM safety?

Human-AI coupling is the feedback loop between a user's psychology and a model's conversational behavior, where each shapes the other over many turns. It becomes a safety risk when a model's agreement or confidence leads users to rely on it more than its accuracy warrants. Testing for it means following the same users across several conversations or sessions.

Can simulated users replace human participants in AI safety testing?

They cannot fully replace them, though simulated users are useful for finding candidate failures across thousands of conversations at low cost. They cannot show whether a model actually changes what real people believe or do, and their own behavior needs checking against human data. A sound program uses simulations to search widely and human studies to confirm.

How do you measure trust calibration in an LLM evaluation?

You compare how much users trust the model with how accurate it actually is on their questions. Ask participants how confident they are in each answer and whether they would act on it, then score the answers for accuracy. Calibration is good when trust rises and falls with accuracy across questions and users.

Can psychometric tests measure an LLM's personality?

They can measure how a model describes itself, and for larger models those descriptions are reliable and can be shaped with prompts. They are weaker at predicting behavior that depends on the conversation, such as sycophancy. Teams should validate any questionnaire against observed behavior before using it in safety decisions.

When should a team run behavioral red teaming?

Before any launch or update that changes how a model talks to users, and again after release. Changes to training feedback or memory can shift sycophancy, and an update can pass standard checks while making the model more agreeable. Repeating a fixed set of scenarios lets a team compare versions over time.