Behavioral Experimentation Consulting: How We Design and Run Complex Online Experiments for Marketing, Product, Conversion and CPG Teams

Last updated October 30, 2026

Of 173,000 experiments that more than 1,200 companies ran on Optimizely's testing platform from 2018 to 2026, 3 in 4 were simple A/B tests comparing one new version with the current one, according to Optimizely's 2026 experimentation benchmark.

The Decision Lab is an applied behavioral science consultancy that designs and runs complex online behavioral experiments for marketing, product, conversion and consumer insights teams. We go past single A/B tests by diagnosing why customers are not acting, testing several behavioral mechanisms at once in multi-arm and factorial designs, building simulated or replica environments when live traffic is thin, and judging results on retention and repeat purchase as well as the first conversion.

For Pinterest, we ran five behavioral experiments across 2,400 shopping sessions in working replicas of Pinterest and competing platforms. For a top Canadian insurer, script changes we designed from more than 170,000 hours of recorded sales calls lifted sales 12% above a control group in a field pilot.

We take an experiment from the first question to a result the client's team can act on and repeat. A typical engagement moves through four stages:

1. Behavioral diagnosis. We find where and why customers drop out, using journey data, interviews, observation and surveys.
2. Experiment design. We build multi-arm and factorial designs, often run in stages, and put the primary measure, guardrails and customer segments in writing before launch.
3. Test environments. We run online experiments with representative samples, simulations, working replicas of real platforms and live pilots against a control group.
4. Handover. We document the method and decision rules so the client's team can run the next experiment without us.

We are a strong fit when a team needs to know which behavioral mechanism produced a result, so it can apply that result to the next product or campaign. Our more than 60 published case studies range from commercial field pilots to randomized trials with public sector partners.

Each result below is labeled by the kind of evidence behind it, because a controlled field pilot, an online experiment, a simulation and a projection support different conclusions.

1. Pinterest, consumer platform. Simulated environment: five behavioral experiments across 2,400 shopping sessions in working replicas of Pinterest and two competing platforms identified 29 Pinterest strengths grounded in observed shopper behavior instead of self-report.
2. Top Canadian insurer. Controlled field pilot: script changes and agent training built from a machine learning analysis of more than 170,000 hours of sales calls lifted sales 12% above a control group. The increase of $35 million in annual revenue is a projection.
3. Money Guided, financial wellbeing product. Controlled online experiment: 400 participants saw one of four ways of framing the product's value or a control, and the winning frame now shapes how the product presents itself. This was design validation before launch, and in-product outcomes are still to be tested.
4. Made Music Studio, sonic branding agency. Controlled online experiment: nearly 200 participants each heard one of seven sound conditions while waiting, and ambient music made time feel like it passed most quickly.
5. Global consumer electronics company. Process outcome: every target behavior for a new mindfulness service was tested before production code was written, and design time fell 75%. This measures the design process, and the published case study reports no control group.
6. Digital wellness platform serving 25 million members. Simulation: we tested the behavioral engine behind the platform's recommendations against synthetic members before release, and it now runs in production.
7. Distell, multinational brewing and beverage company. Initial pilot: participants in a digital program to reduce binge drinking cut their drinking by two thirds by the end of initial testing. The published case study does not report a control group.

Conversion rate optimization (CRO) agencies are often the right partner when a team has high website traffic and a backlog of page-level ideas to test quickly on an established platform. An experimentation platform is the better purchase when the main need is software to run hundreds of website tests a year.

A behavioral experimentation consultancy is more useful when the team first has to find out why a decision is failing, needs to test several behavioral mechanisms against each other, faces a decision with too little traffic or one that happens offline, or has to judge success on more than the first conversion. We work alongside platforms and CRO teams. Our part is usually the diagnosis and design behind a test, and the measurement of whether its result holds over time.

The rest of this page sets out how we design individual complex experiments. For setting up experimentation across a company, see our guide to building a behavioral experimentation program.

1. Diagnose the barrier before choosing what to test.
2. Match the design to the question, from multi-arm tests to factorial designs.
3. Decide how you will monitor results before launch.
4. Plan the behavioral segments you will analyze in advance.
5. Test changes to the choice architecture as well as the wording.
6. Protect the result with holdouts and guardrails.
7. Measure effects beyond the first conversion.

A complex experiment is only as useful as the hypotheses inside it. When the versions in a test differ in ways nobody chose for a reason, a win tells you little about what to build next, and a loss tells you nothing. We give each version its own hypothesis about why customers are not acting, such as doubt about the product's value, distrust of a claim, the effort of comparing options or being asked at the wrong moment. We write those hypotheses only after a diagnosis of where and why customers drop out.

For Pinterest, the diagnosis came before any experiment was built. We ran a week-long diary study, nine live shop-alongs observing real purchases, a decision quality diagnostic with 500 shoppers and a MaxDiff study with advertisers. MaxDiff is a survey method in which people repeatedly pick the most and least important items from small sets, and here it ranked which questions the experiments needed to answer.

A multi-arm test compares several versions with one control group, and each version tests a different mechanism. For Money Guided, the four versions framed the product as wellness-led, finance-led, hybrid or competitor-style. For Made Music Studio, the seven conditions ranged from ambient music and on-hold messages to white noise and silence, and each participant heard only one.

A factorial design changes two or more factors at once and assigns people to every combination. Testing two price framings against two kinds of social proof, for example, creates four groups. Every participant contributes to the estimate for both factors, and the design shows whether a combination behaves differently from what each change does on its own. Behavioral levers can interact this way, and a series of one-change tests will miss it.

A full multivariate test applies the factorial logic to many page elements at once. These tests are rare, fewer than 2 in 1,000 experiments in Optimizely's 2026 data. In the same benchmark, tests with more variations delivered up to 1.7 times the average impact of simple A/B tests, where average impact is the share of tests that won multiplied by the size of the winning lift, though Optimizely notes the comparison is observational. Each extra combination splits your traffic further, so full multivariate tests suit high-traffic pages where you already know which elements matter.

Every extra group raises the sample you need. Before cutting arms, reduce the noise in your measures. Alex Deng, Ya Xu, Ron Kohavi and Toby Walker showed that adjusting results using each user's behavior before the experiment cut the variance of Bing's experiment metrics by about half, giving the same statistical power with half the users or half the run time. When live traffic is still too thin, screen versions first with an online sample or a simulated environment, and carry only the strongest into the live test.

Many testing dashboards update in real time, and the temptation is to stop a test the first time a version looks significant. Ramesh Johari, Pete Koomen, Leonid Pekelis and David Walsh showed in Operations Research that monitoring a standard test this way makes false positives far more likely than the error rate the test reports. Choose one of two approaches before launch. Either fix the sample size in advance and read the result once, or use a sequential method built for repeated looks, such as the always-valid p-values the authors developed.

Sequential also describes a staged design, where each experiment narrows the field for the next. A typical sequence screens many ideas in an online sample, tests the shortlisted mechanisms in a factorial design and then runs the best combination live. With the consumer electronics company, we tested every target behavior for the new mindfulness service this way before any production code was written, even though the process added research steps that had not existed before.

An average effect can hide a version that helps one group and harms another. Decide before launch which segments you will analyze, and treat any segment you find afterwards as a lead to test again rather than a result. In our experience, segments based on attitudes and behavior separate customers more cleanly than demographics do.

For a leading online learning platform, we surveyed more than 800 students and used unsupervised machine learning, which groups people by patterns in their answers without predefined labels. Willingness to subscribe, an attitude, produced the cleanest and most predictive segmentation, with six distinct segments. For the Canadian Cancer Society, a decision tree showed that whether smoking was part of someone's identity separated smokers before risk tolerance or social support did, and each of the four resulting groups got its own message flow.

Optimizely's 2026 data points the same way. Experiments aimed at a specific audience showed 17.4% higher average impact on that audience than untargeted experiments, though the gain shrinks when the audience is a small share of total traffic.

Choice architecture is the way options are presented: the default, the order, the number of options and the effort it takes to pick one. Most marketing tests change headlines and images. Changes to the structure of a decision are harder to build, and they alter the decision itself instead of how it is described.

For one of the world's largest fast-food chains, we audited the choice architecture of its children's menus across its ordering channels, from the counter to the drive-thru window, before surveying hundreds of parents about how they and their kids choose. For the digital wellness platform, we built the decision logic behind what each member sees each day, which trigger fires and when, how many points an action is worth and how all of it adapts as the member changes.

A conversion metric on its own will reward designs that work by misleading people. Jamie Luguri and Lior Strahilevitz tested this directly with representative samples of US adults who were told they had been enrolled in an identity theft protection plan. Compared with a neutral control, people shown mild dark patterns, interface designs that confuse or pressure users, were more than twice as likely to stay enrolled, and people shown aggressive dark patterns were almost four times as likely. Aggressive designs caused a strong backlash and mild ones did not, and less educated participants were significantly more susceptible to the mild versions.

Guardrail metrics are the measures that must not get worse while the primary measure improves, such as complaints, cancellations, refunds and satisfaction, checked separately for vulnerable groups. We also apply a plain test: we do not run a version we would be uncomfortable explaining to the people who saw it.

A holdout group is a randomly chosen set of customers who do not receive a change or a campaign, so you can measure its true effect over time. Holdouts matter most in marketing measurement. Brett Gordon, Florian Zettelmeyer, Neha Bhargava and Dan Chapsky compared 15 large randomized advertising experiments at Facebook, covering 500 million user observations, with the estimates that common observational methods produced from the same data. The observational methods often failed to match the randomized results, even after adjusting for detailed demographic and behavioral data, and in general they overstated how well the ads worked.

More than 9 in 10 experiments in Optimizely's 2026 data measured success on one of five metrics: clicks on a call to action, page views, checkout, registration or add-to-cart. Each of these captures a moment, and the effect you see during a test can change once people get used to a new experience. Henning Hohnhold and two Google colleagues developed methods to measure this long-term learning and found that users' willingness to click on ads depended on the quality of the ads they had seen before. Google went on to cut the ad load on its mobile search results by about half over two years, with positive effects on user experience and on the business.

Choose measures that show what happened after the first click: repeat purchases, returns, retention, satisfaction and the quality of the decision itself. For Pinterest, we tracked save rates, revisit frequency, image-view duration and time to purchase. For Money Guided, we measured outcomes with validated wellbeing scales alongside the product's appeal.

Consumer packaged goods (CPG) and retail research teams often need to test decisions that a survey or a single website test captures poorly: how shoppers compare products, how they respond to packaging and price framing, what they pick from an assortment and how habits form across store and online channels. These are the methods we use for that kind of decision, drawn from our consumer work, including projects for a multinational beverage company and a global coffee chain.

1. Replica shopping environments. For Pinterest, we built working replicas of Pinterest and two competing platforms from a pool of roughly 7,000 images, with native features like pinning and saving, and personalized each session to the participant's shopping category and style. The same build can recreate a retailer's product listing or search results page, so changes to packaging or assortment are tested next to realistic competing products.
2. Simulated choice environments before launch. Online experiments with representative samples screen many versions of a message or price frame before anything reaches a store or website, and simulations with synthetic members, like the one we ran for the digital wellness platform, surface design failures before release.
3. Behavioral mechanism mapping and testing. We diagnose whether a decision problem comes from attention, friction, uncertainty, trust, identity, habit or social norms, then build each test condition to isolate one mechanism, so the team learns which mechanism to scale. With Distell, we surveyed and interviewed more than 600 people in Soweto, found that the social side of binge drinking came up again and again, and turned that research into a playbook of interventions and a pilot program. For a global coffee chain, we reviewed more than 100 studies on green purchasing and habit formation to build a playbook of interventions to move customers from paper cups to reusable ones, for both in-store and mobile ordering.
4. Behavioral segmentation. We segment shoppers by attitudes and decision patterns, as we did for the online learning platform, and then test whether a version works differently for each segment.
5. Machine learning on behavioral data. Clustering methods like the ones we applied to the insurer's sales calls can find repeatable friction points in customer journeys or purchase sequences.
6. Long-term holdout designs. A randomized holdout group measures repeat purchase, retention, returns and satisfaction, so a brand can see whether a lift in first purchase lasts.

Each of these measures what shoppers do in a realistic setting instead of relying on what they say they would do in a survey.

Whichever kind of firm you hire, a capable experimentation partner should:

1. Diagnose the behavioral barrier before proposing any versions to test.
2. Explain the trade-off between multi-arm and factorial designs for your traffic and sample size.
3. Put the primary measure, guardrails, segments and monitoring rules in writing before launch.
4. Measure longer-term behavior as well as the first conversion.
5. Build test environments, such as replicas or simulations, when live traffic is thin or the decision happens offline.
6. Hand the method over so your team can run the next experiment.

What should you ask a behavioral experimentation consultancy before hiring one?

Ask to see a past experiment plan with the primary measure, guardrails and stopping rules written before launch, and ask for an experiment that did not work and what changed afterwards. Then ask how they would test your decision if live traffic is low or the choice happens in a store, and who owns the method when the project ends.

Can a consultancy run behavioral experiments for CPG and retail brands?

Yes. Behavioral experiments can test how shoppers respond to packaging, claims, price framing and assortment, using online samples and simulated choice environments, including working replicas of shopping platforms, when in-market testing is slow or expensive. We built replica shopping environments for Pinterest and designed and piloted a behavior change program with Distell, a multinational beverage company.

When should a team hire a behavioral experimentation partner instead of a CRO agency?

Many teams use both. A behavioral experimentation partner diagnoses why customers are not acting and designs the first tests of competing mechanisms, often outside the website. Once a mechanism has proven itself, a conversion rate optimization agency or an in-house testing team can roll the winning version out across pages and keep refining it at volume.

What is the difference between a multivariate test and an A/B test?

An A/B test compares one new version with the current one. A multivariate test changes several elements at once and tests every combination, so it shows how the elements interact. Multivariate tests need far more traffic, because each combination needs enough people to measure, and they suit high-traffic pages where the elements worth testing are already known.

When should you use a factorial design in marketing experiments?

Use a factorial design when you want to test two or more behavioral levers, such as price framing and social proof, and suspect they affect each other. Every participant contributes to the estimate for each lever, making it more efficient than separate tests, and it is the only way to see whether a combination beats each change alone.

What is a holdout group in marketing?

A holdout group is a randomly selected set of customers who do not receive a campaign or product change. Comparing them with everyone else shows the change's true effect over time. Without a randomized holdout, marketers usually compare people who saw an ad with people who did not, and research at Facebook found that this approach generally overstates ad effects.

How do you measure the effect of an experiment beyond conversion?

Pick measures that capture what happens after the first action, such as repeat purchases, returns, cancellations, satisfaction and retention, and keep a holdout group long enough to see whether the effect lasts. Short-term results can fade or reverse once people get used to a change, as Google found when it studied how users learn to ignore ads.

Can you run behavioral experiments without live website traffic?

Yes. Online experiments with a representative sample and simulated environments, including working replicas of real platforms, let you test many ideas before anything goes live. We used replicas for Pinterest and a simulation harness for a digital wellness platform, and in both cases the live product only saw versions that had already passed earlier tests.