1.1 What A/B Testing Is Really Testing

Concept overview

What A/B Testing Is Really Testing matters because an experiment is only useful when it changes a product decision. The statistics are there to protect the decision from noise. They do not replace judgment.

In a feed feature experiment, the team is not asking "did a number move?" The real question is whether the change creates enough product value to ship, whether the measurement is trustworthy, and whether any guardrail says the win is too expensive. Good A/B testing answers make that chain explicit: decision, hypothesis, metric, validity, interpretation, recommendation.

Read this lesson as a decision framework. When you answer in an interview, keep returning to the action the team will take after the result. That is what separates a strong data science answer from a formula recap.

Mental model

Think of What A/B Testing Is Really Testing as a chain of evidence.

First, define the decision. A team may launch, roll back, continue testing, segment the result, or redesign the feature. If the decision is vague, every metric debate becomes vague too.

Second, define the unit of exposure. Users, sessions, devices, listings, stores, queries, and conversations behave differently. Many experiment mistakes happen because the randomization unit and the product effect do not match.

Third, decide what can invalidate the result. That includes peeking, novelty effects, sample ratio mismatch, interference, logging bugs, delayed effects, and guardrail regressions. Interviewers care about these checks because real experiments fail quietly before they fail mathematically.

Worked example: Feed feature experiment

Suppose a social feed team tests a new creator card and needs to decide whether to launch. A weaker answer starts by choosing a p-value threshold or listing common metrics. A stronger answer starts with the product decision: "If the new experience improves completed purchases without increasing payment errors or support contacts, we will roll it out to all users."

Now the hypothesis has a job. The primary metric might be completed checkout rate among exposed users. Guardrails might include payment failure rate, refund rate, latency, support tickets, and revenue per buyer. Diagnostic metrics can explain movement, for example add-to-cart rate, payment-method selection, or retry behavior.

The design still needs validity checks. The randomization unit should match the exposure point. If users can switch devices or interact with both variants, the result may be diluted. If the team checks results every hour and stops on the first green number, the false-positive rate rises. If the treatment affects sellers, inventory, or other users, a normal user-level test may understate spillovers.

The recommendation should sound like a decision, not a statistical ritual. "The lift is positive, the confidence interval excludes zero, and guardrails are flat" is only part of the answer. Finish with what you would do next: launch, ramp gradually, extend the test, segment the result, or redesign the feature.

Tradeoffs and failure modes

The main tradeoff in What A/B Testing Is Really Testing is speed versus trust. Teams want a decision quickly, but faster reads of the data increase the chance of acting on noise or biased measurement.

Common failure modes include unclear hypotheses, mismatched randomization units, too many primary metrics, missing guardrails, peeking, novelty effects, logging bugs, and interference between users. The fix is not to recite every possible issue. The fix is to name the two or three risks most likely for the product scenario and explain how you would detect them.

Interview prep layer

What interviewers are testing

Interviewers are testing how you connect experimental design to product decisions, metric quality, and validity risks. They are less interested in whether you memorized a framework and more interested in whether you can apply it cleanly when the prompt is ambiguous.

Short model answer

"For What A/B Testing Is Really Testing, I would first clarify the decision the experiment supports, then define the hypothesis, exposure point, randomization unit, primary metric, and guardrails. After that I would check validity risks like logging quality, sample ratio mismatch, peeking, novelty, and interference before making a launch recommendation."

Common traps

Starting with p-values before the decision is clear.
Treating every metric as a primary metric.
Ignoring guardrails or validity checks because the primary metric moved.

Practice drill

Take feed feature experiment and explain the lesson in three minutes. Spend the first minute on the concept, the second minute on the design or analysis choices, and the third minute on tradeoffs, risks, and what you would do next.

Interview-ready detail

For What A/B Testing Is Really Testing, the safest way to sound senior is to separate four layers that candidates often mix together: the product goal, the statistical question, the operational risk, and the launch decision. The product goal explains why the team cares. The statistical question explains what evidence would change the team's mind. The operational risk explains what can go wrong even if the metric looks good. The launch decision explains what action follows from each result.

In a feed feature experiment, write the hypothesis as a sentence that contains a user behavior and a decision. For example: "If the new experience helps users understand the value of the feature earlier, then eligible exposed users should complete the target action more often, and we should launch if guardrails stay neutral." That sentence is stronger than "increase conversion" because it tells the interviewer what the team believes and what evidence would matter.

The next move is to choose the unit of analysis carefully. If users can see the treatment many times, user-level analysis may be better than session-level analysis. If stores, sellers, teams, or creators influence each other, simple user randomization can hide network effects. If the feature is triggered only for a subset of users, define the denominator before the test starts. Many experiment answers lose credibility because the metric sounds plausible but the denominator is unclear.

Finally, explain how you would read the result. A statistically significant lift is not automatically a launch. A non-significant result is not automatically a failure. Look at the confidence interval, guardrails, sample ratio, instrumentation, exposure distribution, and the practical size of the effect. Then state the recommendation in product language: launch, ramp, extend, segment, or redesign.

Follow-up pressure tests

Expect the interviewer to push on ambiguity. If the primary metric improves but a guardrail regresses, do not average them into a single score unless the business already has an agreed utility function. Explain the tradeoff directly and ask which risk the organization is willing to accept. For a payments change, a tiny conversion lift may not justify a large error-rate increase. For a discovery change, a small latency increase may be acceptable if long-term retention improves.

If the test is underpowered, say what you would do before collecting more data. You might widen the eligible population, choose a more sensitive metric, run longer, reduce metric variance, or use a higher-signal proxy. If the result is heterogeneous, separate exploratory segments from pre-registered decision segments. Segment analysis can explain a result, but it can also create false confidence if every slice is treated like a new primary test.

If the interviewer asks about peeking, answer practically. Teams can monitor safety metrics continuously, but they should avoid repeatedly making launch decisions on the primary metric without correction. If early stopping is part of the plan, use a sequential design or an agreed decision rule. The key is not to pretend teams never look at dashboards. The key is to protect the final decision from accidental cherry-picking.

Quick self-check

Before moving on, answer these out loud:

What decision would this experiment change?
What is the exposure point and randomization unit?
Which guardrail could block launch even if the primary metric improves?
What validity risk would you check first?

Topic walkthrough

The real interview signal

Most candidates do not fail A/B testing interviews because they forget a statistic. They fail because they treat the prompt like a statistics quiz.

Interviewers are usually asking a broader question:

Can this person own an experiment from product goal to launch decision?

That means your answer needs more than math. A strong answer shows judgment about:

The product goal and hypothesis
The right users or units to randomize
Success metrics and guardrails
Experiment validity and common failure modes
How to interpret the result without overclaiming
Whether to ship, iterate, stop, or run a follow-up test

Statistics are the tool. Judgment is the signal.

Figure: A strong interview answer moves through the full decision loop instead of jumping straight into calculation.

What interviewers mean by "design an A/B test"

When an interviewer asks, "How would you design an A/B test for X?", they are not mainly checking whether you can run a t-test. That is table stakes.

They want to see whether you can cover the full experiment path:

Clarify the product change and the decision being made.
State the hypothesis.
Define the control and treatment.
Choose the randomization unit and exposure point.
Pick a primary metric, supporting metrics, and guardrails.
Discuss sample size, duration, and power at the right level of detail.
Check experiment health before trusting the result.
Interpret statistical and practical significance.
Make a recommendation and name the follow-up.

Helpful phrasing: "Before choosing metrics, I want to clarify the decision this experiment is supposed to inform. Are we deciding whether to launch this change broadly, or are we comparing several possible designs?"

A/B testing in one sentence

An A/B test is a randomized controlled experiment that estimates the causal effect of a change by comparing outcomes between a treatment group and a control group under similar conditions.

Four words in that sentence matter:

Word	Why it matters in an interview
Randomized	Random assignment makes the groups comparable.
Controlled	The control group estimates what would have happened without the change.
Causal	The goal is to estimate the effect of the intervention, not just observe a relationship.
Counterfactual	The control group is the best available stand-in for the untreated outcome.

Why randomization matters

Correlation is not enough for product decisions. Two things can move together because a third factor affects both.

A common example is ice cream sales and shark attacks. They rise at the same time, but ice cream does not cause shark attacks. Hot weather drives both: people buy more ice cream and spend more time in the ocean.

Randomization helps break that link. If users are randomly assigned to control and treatment, known and unknown factors should be balanced in expectation. Then a systematic difference in outcomes is more credibly tied to the product change.

Figure: Observational comparisons are vulnerable to hidden factors. Random assignment gives the control group a fair counterfactual role.

Interview tip: Do not just say "randomize users." Say what unit you would randomize and why. User-level, session-level, device-level, household-level, market-level, and seller-level randomization can lead to different answers.

What A/B testing is not

Strong candidates define boundaries. A/B testing is not:

A before-and-after comparison
A correlation analysis
Offline model evaluation
A check that one metric moved up
A way to avoid product judgment
A guarantee that the short-term winner is the long-term right answer

Before-and-after comparisons are especially risky. A metric can change because of seasonality, traffic mix, marketing campaigns, pricing changes, outages, or regression to the mean. Without a fair control group, it is hard to isolate the product change.

What an A/B test can do

A/B tests are useful when you already have a specific change and need evidence about its impact.

They can help you:

Compare control and treatment fairly
Reduce risk before a full launch
Measure real user behavior in production
Make metric trade-offs visible
Align a team around structured evidence

Example: suppose a checkout redesign increases conversion but also increases refund rate. A good A/B test answer would not stop at "conversion went up." It would weigh the primary metric against guardrails and ask whether the net effect supports launch.

What an A/B test cannot do

A/B tests are powerful, but they do not answer every product question.

They cannot define product strategy. They only evaluate options you chose to test.

They may miss long-term effects if the test runs briefly. User learning, fatigue, retention, trust, and marketplace health can take longer to show up.

They can break down when users affect each other. This matters in marketplaces, social networks, ads auctions, supply-constrained systems, and networked products.

They do not automatically explain why a metric changed. For that, you need funnel cuts, segment analysis, user research, logging, and follow-up experiments.

They do not replace ethics or business constraints. Even a statistically significant winner may be the wrong launch if it harms trust, fairness, safety, or long-term ecosystem health.

Helpful phrasing: "The experiment can tell us whether the treatment moved the metric. It does not, by itself, explain the mechanism, so I would pair the result with funnel and segment analysis."

How to start a strong interview answer

Use this opening structure when you get an A/B testing prompt:

"First I would clarify the product goal and the decision we need to make."
"Then I would state the hypothesis in terms of user behavior."
"I would define control, treatment, exposure point, and randomization unit."
"Next I would choose one primary metric, supporting diagnostics, and guardrails."
"After that I would discuss sample size, duration, experiment health, and interpretation."

That opening shows the interviewer that you know the full shape of the problem before you start calculating.