What Is Dilution in A/B Testing and How it Affects Power

When running an A/B test, data scientists often focus on metrics like sample size, statistical significance, and p-values. However, one subtle but critical concept that can silently weaken your experiment’s ability to detect real differences is dilution.

In this post, we’ll break down what dilution means in the context of A/B testing, why it happens, and how it reduces the statistical power of your experiment.

What Is Dilution?

Dilution occurs when the effect of a treatment (the “B” variant) is spread out or averaged down by users, events, or metrics that are not directly exposed to the treatment change.

In simpler terms, dilution happens when the test includes noise from unexposed or weakly exposed data, which makes the observed effect size smaller than the true impact of the treatment.

Example

Suppose you run an experiment on the checkout page of an e-commerce site to test a new “express checkout” button. However, your overall experiment metric is total revenue per user - calculated across all users, including those who never reach checkout.

Only 10% of users see the checkout page, meaning your change affects only 10% of users. Even if the treatment increases checkout conversion by 10%, the overall impact on total revenue per user will appear as:

Observed Lift = 0.10 × 0.10 = 0.01

That’s a tenfold dilution of your true effect.

Common Sources of Dilution

Partial exposure to treatment

Only a subset of users or sessions experiences the change (e.g., page-level tests, not user-level).

Metric misalignment

The chosen metric aggregates across unrelated areas. For instance, measuring “session duration” sitewide when the change only affects one feature page.

Event-based measurement

When metrics are computed per event rather than per user, repeated exposure to the control variant can dilute the treatment effect.

Cross-contamination

Users see both variants during the experiment (due to cookie resets, shared devices, or incorrect bucketing).

How Dilution Reduces Statistical Power

Statistical power is the probability that an experiment correctly detects a real difference when one exists. Power depends on three key factors:

Power ∝ (True Effect Size)² / (Variance/n)

When dilution occurs, the observed effect size shrinks, while variance remains roughly the same. This drastically lowers the power - meaning your experiment is less likely to detect meaningful improvements.

For example:

True effect = 10% lift
Dilution factor = 0.1 (only 10% exposed)
Observed effect = 1% lift

To detect the same 10% improvement diluted down to 1%, you would need 100× more samples to achieve the same power.

How to Mitigate Dilution

Here are a few ways to reduce dilution and design stronger experiments:

Target the right population

Run the test only on users who are actually exposed to the treatment. → e.g., restrict analysis to users who reached the checkout page.

Align metrics with exposure

Use metrics that reflect the change’s direct impact (e.g., “checkout conversion rate” instead of “total site revenue”).

Use triggered analysis

Start measuring from the first exposure to the feature rather than from all sessions.

Prevent cross-contamination

Use user-level bucketing, persistent IDs, and feature flags to ensure consistent variant exposure.

📊 Takeaway

Dilution doesn’t mean your experiment failed - it means your analysis included too much irrelevant data. By recognizing and accounting for dilution:

You get clearer signals of true effects.
You reduce wasted sample size.
You make better data-driven decisions faster.

In A/B testing, the key to power isn’t always more data - it’s cleaner data. Make sure your metrics and exposure align perfectly, and you’ll see the real impact your experiments are driving.

How to Use This Page as a Prep Plan

Do not treat this as passive reading. Convert the ideas in this page into a short weekly loop: learn one idea, practice it under interview conditions, then write down what changed. That is the fastest way to turn advice into visible interview behavior.

Prep area	What you need to prove	Practice artifact
Understand	Turn the prompt into a concrete goal.	Clarifying questions and success criteria.
Practice	Use realistic constraints and timed reps.	Worked examples with edge cases.
Explain	Make reasoning visible.	Tradeoffs, assumptions, and test strategy.
Improve	Review misses quickly.	A short feedback log and next action.

For What Is Dilution in A/B Testing and How it Affects Power, the strongest candidates usually do three things well: they make their assumptions explicit, they use concrete examples instead of vague claims, and they review mistakes quickly enough that the next practice rep is better than the last one.

Video Walkthrough

This verified YouTube video gives a second pass on the same preparation area. Use it after reading the guide, then come back and turn the advice into a practice artifact.

FAQ

How should I use this guide?

Read it once for the structure, then turn each section into a practice task with a visible artifact.

What should I do if I am short on time?

Prioritize the skills most likely to be tested, then do one mock or timed drill to expose the largest gap.

How do I know I am ready?

You can explain your approach clearly, recover from hints, and name tradeoffs without relying on memorized wording.