1.2 Define the Problem and Goal, Then Forming a Hypothesis

Concept overview

Define the Problem and Goal, Then Forming a Hypothesis matters because an experiment is only useful when it changes a product decision. The statistics are there to protect the decision from noise. They do not replace judgment.

In a feed feature experiment, the team is not asking "did a number move?" The real question is whether the change creates enough product value to ship, whether the measurement is trustworthy, and whether any guardrail says the win is too expensive. Good A/B testing answers make that chain explicit: decision, hypothesis, metric, validity, interpretation, recommendation.

Read this lesson as a decision framework. When you answer in an interview, keep returning to the action the team will take after the result. That is what separates a strong data science answer from a formula recap.

Mental model

Think of Define the Problem and Goal, Then Forming a Hypothesis as a chain of evidence.

First, define the decision. A team may launch, roll back, continue testing, segment the result, or redesign the feature. If the decision is vague, every metric debate becomes vague too.

Second, define the unit of exposure. Users, sessions, devices, listings, stores, queries, and conversations behave differently. Many experiment mistakes happen because the randomization unit and the product effect do not match.

Third, decide what can invalidate the result. That includes peeking, novelty effects, sample ratio mismatch, interference, logging bugs, delayed effects, and guardrail regressions. Interviewers care about these checks because real experiments fail quietly before they fail mathematically.

Worked example: Feed feature experiment

Suppose a social feed team tests a new creator card and needs to decide whether to launch. A weaker answer starts by choosing a p-value threshold or listing common metrics. A stronger answer starts with the product decision: "If the new experience improves completed purchases without increasing payment errors or support contacts, we will roll it out to all users."

Now the hypothesis has a job. The primary metric might be completed checkout rate among exposed users. Guardrails might include payment failure rate, refund rate, latency, support tickets, and revenue per buyer. Diagnostic metrics can explain movement, for example add-to-cart rate, payment-method selection, or retry behavior.

The design still needs validity checks. The randomization unit should match the exposure point. If users can switch devices or interact with both variants, the result may be diluted. If the team checks results every hour and stops on the first green number, the false-positive rate rises. If the treatment affects sellers, inventory, or other users, a normal user-level test may understate spillovers.

The recommendation should sound like a decision, not a statistical ritual. "The lift is positive, the confidence interval excludes zero, and guardrails are flat" is only part of the answer. Finish with what you would do next: launch, ramp gradually, extend the test, segment the result, or redesign the feature.

Tradeoffs and failure modes

The main tradeoff in Define the Problem and Goal, Then Forming a Hypothesis is speed versus trust. Teams want a decision quickly, but faster reads of the data increase the chance of acting on noise or biased measurement.

Common failure modes include unclear hypotheses, mismatched randomization units, too many primary metrics, missing guardrails, peeking, novelty effects, logging bugs, and interference between users. The fix is not to recite every possible issue. The fix is to name the two or three risks most likely for the product scenario and explain how you would detect them.

Interview prep layer

What interviewers are testing

Interviewers are testing how you connect experimental design to product decisions, metric quality, and validity risks. They are less interested in whether you memorized a framework and more interested in whether you can apply it cleanly when the prompt is ambiguous.

Short model answer

"For Define the Problem and Goal, Then Forming a Hypothesis, I would first clarify the decision the experiment supports, then define the hypothesis, exposure point, randomization unit, primary metric, and guardrails. After that I would check validity risks like logging quality, sample ratio mismatch, peeking, novelty, and interference before making a launch recommendation."

Common traps

Starting with p-values before the decision is clear.
Treating every metric as a primary metric.
Ignoring guardrails or validity checks because the primary metric moved.

Practice drill

Take feed feature experiment and explain the lesson in three minutes. Spend the first minute on the concept, the second minute on the design or analysis choices, and the third minute on tradeoffs, risks, and what you would do next.

Interview-ready detail

For Define the Problem and Goal, Then Forming a Hypothesis, the safest way to sound senior is to separate four layers that candidates often mix together: the product goal, the statistical question, the operational risk, and the launch decision. The product goal explains why the team cares. The statistical question explains what evidence would change the team's mind. The operational risk explains what can go wrong even if the metric looks good. The launch decision explains what action follows from each result.

In a feed feature experiment, write the hypothesis as a sentence that contains a user behavior and a decision. For example: "If the new experience helps users understand the value of the feature earlier, then eligible exposed users should complete the target action more often, and we should launch if guardrails stay neutral." That sentence is stronger than "increase conversion" because it tells the interviewer what the team believes and what evidence would matter.

The next move is to choose the unit of analysis carefully. If users can see the treatment many times, user-level analysis may be better than session-level analysis. If stores, sellers, teams, or creators influence each other, simple user randomization can hide network effects. If the feature is triggered only for a subset of users, define the denominator before the test starts. Many experiment answers lose credibility because the metric sounds plausible but the denominator is unclear.

Finally, explain how you would read the result. A statistically significant lift is not automatically a launch. A non-significant result is not automatically a failure. Look at the confidence interval, guardrails, sample ratio, instrumentation, exposure distribution, and the practical size of the effect. Then state the recommendation in product language: launch, ramp, extend, segment, or redesign.

Follow-up pressure tests

Expect the interviewer to push on ambiguity. If the primary metric improves but a guardrail regresses, do not average them into a single score unless the business already has an agreed utility function. Explain the tradeoff directly and ask which risk the organization is willing to accept. For a payments change, a tiny conversion lift may not justify a large error-rate increase. For a discovery change, a small latency increase may be acceptable if long-term retention improves.

If the test is underpowered, say what you would do before collecting more data. You might widen the eligible population, choose a more sensitive metric, run longer, reduce metric variance, or use a higher-signal proxy. If the result is heterogeneous, separate exploratory segments from pre-registered decision segments. Segment analysis can explain a result, but it can also create false confidence if every slice is treated like a new primary test.

If the interviewer asks about peeking, answer practically. Teams can monitor safety metrics continuously, but they should avoid repeatedly making launch decisions on the primary metric without correction. If early stopping is part of the plan, use a sequential design or an agreed decision rule. The key is not to pretend teams never look at dashboards. The key is to protect the final decision from accidental cherry-picking.

How to practice this lesson

Practice by writing a one-page experiment brief. Include the product decision, hypothesis, target population, randomization unit, exposure event, primary metric, two guardrails, expected effect size, minimum detectable effect, and launch rule. Keep the brief short enough that someone could challenge every line in an interview.

Then rehearse the answer in two passes. In the first pass, explain the clean version. In the second pass, add two realistic complications: a logging bug, a sample ratio mismatch, novelty effects, interference, or a guardrail regression. This trains you to move from textbook statistics to production judgment.

The final practice step is to speak the recommendation out loud. Strong candidates do not end with "the test is significant." They say, "I would ramp to 25 percent while monitoring errors," or "I would not launch yet because the confidence interval includes effects too small to matter," or "I would rerun with cluster randomization because the current design has spillover." That final sentence is where interview signal appears.

Common interviewer variants

Interviewers often vary Define the Problem and Goal, Then Forming a Hypothesis by changing what counts as exposure. A banner experiment may expose users on page load. A ranking experiment may expose users only when the ranked surface is viewed. A notification experiment may expose users when a notification is sent, opened, or delivered. Choose one exposure event and explain why. If the event is too early, the denominator includes users who never had a chance to respond. If it is too late, the metric can condition on behavior affected by treatment.

Another variant is delayed impact. Some features create immediate conversion changes but affect retention, trust, or marketplace balance later. In that case, separate the launch metric from the follow-up metric. You might launch on short-term conversion only if quality guardrails are stable, then continue watching retention, repeat purchase, seller churn, or complaint rate after rollout. That answer shows that you understand experiments as part of a product lifecycle, not a single dashboard read.

A third variant is conflicting evidence. Imagine the primary metric improves, the confidence interval is wide, and one important segment gets worse. Do not hide behind the average. Explain whether the segment was pre-planned, whether the sample size is large enough to trust, and whether the product can target rollout safely. If the segment is important and the harm is plausible, a careful recommendation may be to hold launch, investigate mechanism, and run a focused follow-up.

The last variant is instrumentation risk. If the event fires from the frontend, ad blockers, retries, and page lifecycle can distort counts. If it fires from the backend, it may miss user intent or UI friction. Good answers name the logging source, add reconciliation where possible, and include an A/A test or sample ratio check when the risk is high.

Answer template

Use this structure when Define the Problem and Goal, Then Forming a Hypothesis appears inside a live interview:

"I will first clarify the product decision, because the experiment should answer a launch question rather than produce a dashboard number."
"The hypothesis is that this change improves a specific user behavior for a defined exposed population."
"I would randomize at the level that matches exposure and avoids obvious spillover."
"The primary metric should capture the behavior closest to the hypothesis. Guardrails protect quality, reliability, revenue, and user trust."
"Before reading the result, I would check sample ratio, logging consistency, novelty, peeking, and whether the treatment could affect users outside the assigned group."
"My recommendation would depend on practical effect size, confidence interval, guardrails, and whether the mechanism matches the hypothesis."

This template is intentionally plain. The interviewer should be able to interrupt after any sentence and ask for detail. If you can defend each line with the product scenario, the answer will feel structured without sounding memorized.

Launch decision notes

When the result is ready, translate the analysis into a decision table. If the primary metric improves, guardrails are stable, and the effect is large enough to matter, recommend a staged rollout with monitoring. If the primary metric is flat but guardrails improve, consider whether the product goal was wrong or whether the feature is useful for a narrower segment. If the primary metric improves but a serious guardrail regresses, recommend holding launch unless the team has a clear business reason and a mitigation plan.

If the result is inconclusive, avoid saying only "run longer." Explain why more data would help. More data helps when the effect may be real but the interval is wide. It does not fix broken logging, wrong randomization, spillover, novelty effects, or a metric that does not match the hypothesis. In those cases, redesign the measurement first.

The cleanest final sentence is a product recommendation backed by evidence: "I would launch gradually because the lift is practical, the interval is narrow enough, and guardrails are neutral," or "I would not launch because the observed lift is smaller than the cost of the latency regression." That is the level of clarity interviewers are looking for.

When you practice, force yourself to choose one of four endings: ship, ramp, rerun, or redesign. That habit keeps the answer decisive, practical, specific, and tied to the product decision instead of drifting into abstract statistics.

Quick self-check

Before moving on, answer these out loud:

What decision would this experiment change?
What is the exposure point and randomization unit?
Which guardrail could block launch even if the primary metric improves?
What validity risk would you check first?