Last updated October 25, 2026
Only 1 to 2 in 10 of the controlled experiments run at Google and Bing produce a positive result over the existing version, according to Ron Kohavi and Stefan Thomke's Harvard Business Review analysis of online experiments.
Companies build a behavioral experimentation program by setting up a repeatable system that picks a high-value customer behavior, diagnoses what blocks it and tests a fix against a control group, measuring harms alongside outcomes and scaling only what keeps working. At The Decision Lab, we design and run these programs with clients, from simulated shopping journeys for Pinterest to a sales-call pilot that lifted a Canadian insurer's sales 12% above a control group, and we build each one so the client's own team can keep running it after we leave.
How companies build a behavioral experimentation program
A behavioral experimentation program ties a series of tests to one priority customer behavior, with a named owner and a standing process for acting on what each test shows. We set these programs up with clients in six steps, and the first three happen before any customer sees a change.
- Choose one behavior. Name the action you want more customers to take and how you will observe it.
- Diagnose the barrier. Find out why customers who could act do not, before anyone designs a variant.
- Define success and guardrails. Put the primary measure and the harms you will watch for in writing before launch.
- Pre-test cheaply. Use online experiments or simulated environments to narrow many ideas down to a few.
- Run a field pilot against a control group. Test the shortlist with real customers in the setting where the behavior happens.
- Scale in stages and keep measuring. Roll out gradually against a comparison group and check that the effect holds across the full customer base.
The difference between A/B testing and behavioral experimentation
A/B testing is a randomized method for comparing two or more versions of an experience. Behavioral experimentation is a broader program that uses methods such as A/B tests, randomized field trials, simulations and phased rollouts to understand and change one real behavior. Both rely on random assignment. They differ mainly in how deeply the team diagnoses the barrier before testing, and in whether each result explains why a change worked well enough to design the next one.
| Compared on | A/B testing | Behavioral experimentation |
|---|---|---|
| Role | A method for comparing variants | A program for diagnosing barriers, testing changes, learning from results and scaling what works |
| Methods | Randomized comparison of versions | A/B tests, randomized field trials, simulations and phased rollouts |
| Main question | Which option performs better on the chosen metric | What barrier is blocking a behavior, and which change removes it safely |
| Typical learning | How the variants performed against each other | How the variants performed, plus evidence about the mechanism that shapes the next intervention |
| Scope | Anything from a single test to a large program | Usually several connected tests tied to one priority behavior and an operating process |
Why experimentation has to be a program
Most ideas fail outside search too: across Microsoft's products as a whole, about 1 in 3 experiments improved their target metric over the existing version, and the rest split evenly between no measurable effect and a negative one. Kohavi and Thomke report the Google and Bing figure separately and do not break the company-wide figure down by product.
Expert judgment is a weak substitute for testing. In a 2021 megastudy published in Nature, Katherine Milkman and colleagues tested 54 four-week programs designed to get more than 61,000 gym members to exercise more often. About 9 in 20 of the programs raised weekly gym visits above a control group, by between 9% and 27%. When experts and members of the public were asked to predict the results, their forecasts showed no reliable relationship with what happened, and on average they expected benefits about nine times larger than the real ones.
The payoff also comes from repetition. A study of more than 35,000 technology startups by Rembrand Koning, Sharique Hasan and Aaron Chatterji, published in Management Science, compared startups that adopted A/B testing tools with similar startups that did not. As summarized by Harvard Business School Working Knowledge, adopters had about 10% more weekly page views and launched 9% to 18% more products, and those gains typically took around six months of regular testing to appear. Adopters were also more likely both to shut down and to reach very high traffic, in line with the authors' finding that testing helped startups scale and fail faster.
How to run behavioral experiments in a business
Running a behavioral experiment in a business means turning one hypothesis about why customers are not acting into a test with a control group, a primary outcome and guardrails agreed in advance. In our projects, each experiment moves through the six steps below, and most of the work that decides whether it succeeds happens before launch.
Step 1: Choose one behavior to change
Programs stall when the goal is a vague outcome like "engagement" or "loyalty". A usable target behavior names a specific group of customers and the action you want from them, for example "first-time customers who reach the shipping page complete their purchase within 24 hours". If you cannot observe the behavior in your data, you cannot run an experiment on it yet.
When we helped one of North America's largest insurers set up its own behavioral science team, we ranked candidate pilots by expected return and feasibility. We also weighed spillover, meaning whether a win in one area would carry over to others. Agent sales calls came out on top, so the first experiment targeted what agents say at the moments when calls tend to go wrong.
Step 2: Diagnose why the behavior is not happening
Diagnosis comes before design, because in our experience many failed experiments test a solution to the wrong problem. A diagnosis maps the steps a customer takes toward the behavior and finds where and why they drop out, using journey data, interviews, observation and the behavioral frameworks described in our behavioral science consulting methodology.
For the insurer, that meant mapping more than 170,000 hours of recorded sales calls to find where conversations went wrong. For Pinterest, it meant watching real purchases in live shop-alongs before we designed a single experiment.
Step 3: Define success and guardrails before launch
Agree on one primary measure before the experiment starts. Kohavi and Thomke call this an overall evaluation criterion and recommend revisiting it about once a year. A measure that rewards the wrong behavior will mislead every test built on it.
Guardrail metrics are the measures that must not get worse while the primary measure improves. For customer experiments, these typically include complaints, cancellations, refunds, opt-outs and satisfaction scores, checked separately for vulnerable or high-value groups. We also apply a plain ethical test: we do not run a variant we would be uncomfortable explaining to the customers who saw it.
Write down the sample size, duration, analysis plan and stopping rules in advance. Then check the system itself. Kohavi and Thomke recommend A/A tests, in which both groups receive the identical experience and the system should find no difference, and report that Microsoft found hundreds of invalid experiments this way. If the share of customers in each group does not match the split you planned, treat the result as void until you find out why.
A behavioral experiment brief template
Before an experiment moves to Step 4, we ask client teams to complete a one-page brief. A team that cannot fill in every field is not ready to test.
| Field | What to write |
|---|---|
| Target customer group and behavior | The customer group and the observable action you want from them |
| Behavioral barrier and evidence | The barrier your diagnosis found, and the data or research behind it |
| Hypothesis and intervention | The change you will make and why you expect it to remove the barrier |
| Control condition | What the comparison group experiences, usually the current version |
| Primary outcome metric | The one behavioral measure that decides success, with its current baseline |
| Guardrail metrics | The measures that must not get worse, with the threshold that would stop the test |
| Unit of randomization | Whether you assign individual customers, accounts, stores or regions |
| Sample size | The number per group needed to detect the smallest effect worth acting on |
| Duration and stopping rules | How long the test runs and the conditions for ending it early |
| Decision rule | The result that triggers a scale-up, and the result that sends the idea back for revision or retirement |
| Owner and review date | Who is accountable for the decision, and when the results will be reviewed |
Step 4: Pre-test cheaply before going live
Because forecasts are unreliable, test a long list of ideas with real people in a controlled setting before any live rollout, and carry only the strongest few into the field. Online experiments with a representative sample can compare several versions quickly, and simulated environments go further by recreating the setting where the decision happens.
For Pinterest, we built working replicas of Pinterest and two competing platforms and ran five behavioral experiments across 2,400 simulated shopping sessions. The experiments identified 29 Pinterest strengths grounded in observed behavior rather than self-report. We applied the same principle in a 400-person framing experiment for Money Guided, a financial wellbeing product, and in simulation-first design work for a global consumer electronics company. Our guide to behavioral research methods, from surveys to simulations covers when each method fits.
Step 5: Run a field pilot against a control group
The field pilot tests the shortlist with real customers, in the real channel, against a control group that keeps the current experience. Assign customers at random, run the pilot long enough to cover normal weekly and monthly cycles, and expect early results to drift as the novelty of a change wears off.
For the insurer, we turned the call analysis into targeted script changes and agent training, then piloted them against a control group. Sales rose 12% above the control group in that first pilot, with a projected increase of $35 million in annual revenue. In our work with the World Bank in Uganda, we used randomized trials delivered by text message to test ways of increasing the use of clean cookstoves.
Use random assignment whenever it is practical, and fall back to a phased rollout or a matched comparison group only when randomization is not feasible, for example when a change has to reach everyone at once or the population is too small to split. Report that evidence as weaker, and say so to the people relying on it.
Step 6: Scale in stages and keep measuring
Across 74 nudge interventions in academic journals, take-up of the target behavior rose by an average of 8.7 percentage points over control groups. In 126 randomized trials run at scale by two US government nudge units, covering about 23 million people, the average rise was 1.4 points, about a sixth as large, according to Stefano DellaVigna and Elizabeth Linos's study in Econometrica.
Roll out to a growing share of customers, keep a small holdout group on the old experience, watch the guardrail metrics at each stage, and check whether the effect holds for each major customer segment. Retire versions whose effect fades. Every result, including the ones that failed, goes into a shared log so the next team does not repeat a test the company has already run.
How to keep the program running after the first win
Kohavi and Thomke describe companies that each run more than 10,000 controlled experiments a year, and Microsoft's experimentation team alone has more than 80 people. Most organizations do not need that scale to benefit. They need a named owner for the program, a backlog of hypotheses ranked by expected value, reliable data on the target behaviors and a regular review where results, including null results, change what gets built next.
Ownership matters most. Our engagement with the insurer ended with a permanent in-house behavioral science team that was fully staffed and had already proven itself on a live commercial problem. The consumer electronics team now runs its behavioral design process without us. In our AI adoption work with a Fortune 500 HR technology company, each piloted program came with the protocols and guides its teams need to run it on their own.
Frequently asked questions
What is a behavioral experimentation program?
A behavioral experimentation program is a repeatable process for testing changes designed to shift what customers actually do. It pairs a diagnosis of why people are not acting with controlled experiments that compare new versions against the current experience. Only what works gets scaled, and each test builds on what earlier tests revealed.
How long does a behavioral experiment take to run?
A pre-test with an online sample can return results within days. A field pilot usually needs several weeks so it covers normal weekly cycles and lets novelty effects fade. In our Fortune 500 AI adoption work, pilots ran for one month. Behaviors that repeat slowly, such as renewals, need longer follow-up.
How many customers do you need for a behavioral experiment?
It depends on how common the behavior is and how small an effect you need to detect. Rare behaviors and small expected effects need much larger samples. Calculate the sample size before launch from your baseline rate, and if you cannot reach it, test a bigger change or a more frequent behavior instead.
What are guardrail metrics in experimentation?
Guardrail metrics are measures that must not get worse while an experiment improves its main target. For customer experiments, common guardrails include complaints, refunds, cancellations, opt-outs and satisfaction, checked across customer groups. A version that lifts sales while raising complaints has not passed, even if the primary measure looks strong.
Is it ethical to run experiments on customers?
Experiments are often more responsible than untested changes, because untested changes affect every customer at once with no evidence. Ethical programs test only versions the company would be willing to ship, avoid deceptive designs, protect vulnerable groups with guardrails and follow applicable privacy and consent rules. Some sectors also require independent ethics review.
Can the same approach change employee behavior?
Yes. The six steps apply to employees as well as customers. With a Fortune 500 HR technology company, we piloted eight AI adoption programs with more than 100 employees against a control group. Self-reported confidence in exploring AI tools rose 41% among participants over one month and fell 26% in the control group.
Where The Decision Lab fits
We are an applied behavioral science firm, and our experimentation work covers the full sequence above, from diagnosis and experiment design to field pilots and handover. Our more than 60 published case studies include randomized field trials and simulated decision environments, along with the setup of in-house behavioral teams.
If your main need is software to run thousands of web A/B tests, a dedicated experimentation platform is the better purchase. We work alongside those platforms when the open question is which behavior to target and why customers are not doing it yet.
