Decide whether to ship a redesigned signup flow from an A/B results file with signups, 7-day activation, p95 latency, support tickets, refunds and 30-day revenue per variant. Cover data validity checks, metric choice and intervals, power and stopping rules, and how to weigh activation gains against latency and support costs.
You receive results for an A/B experiment on a redesigned user signup flow. Data file columns: variant (A/B), sessions, signups, activation_d7, p95_latency_ms, support_tickets, refund_rate, and revenue_d30. Describe exactly how you would: (1) validate data quality (SRM test, bucketing integrity, exposure logs vs analytics counts); (2) choose primary and guardrail metrics and justify them; (3) compute effects with CIs (include ratio metrics and non-parametric options if skewed), and apply variance reduction (e.g., CUPED) if appropriate; (4) check power/min detectable effect and whether the observed duration met the pre-registered stopping rule; (5) evaluate novelty and learning effects (time-sliced and cohort views); (6) make the ship/no-ship call with a concrete decision framework that balances activation gains vs increased latency/support tickets; (7) list at least three additional insights to extract beyond the ship decision (e.g., segment heterogeneity by traffic source/device, step-drop analysis, form-field sensitivity). Be precise about formulas, tests, and thresholds you would use.
Overview: Decide whether to ship a redesigned signup flow from an A/B results file with signups, 7-day activation, p95 latency, support tickets, refunds and 30-day revenue per variant. Cover data validity checks, metric choice and intervals, power and stopping rules, and how to weigh activation gains against latency and support costs.
You are given the results of an A/B experiment on a redesigned user signup flow and asked whether to ship the redesign. The data file has these columns:
Column
Description
variant
A or B
sessions
Sessions exposed to the variant
signups
Completed signups
activation_d7
Users who activated within 7 days of signing up
p95_latency_ms
95th-percentile request latency in the signup flow
support_tickets
Support tickets attributed to the signup and onboarding experience
refund_rate
Refund rate
revenue_d30
Revenue within 30 days
Describe exactly how you would get from this file to a ship / no-ship recommendation, and what else you would learn from the experiment. Be precise about the formulas, statistical tests and thresholds you would use.
Constraints & Assumptions
The grain of the file is not stated: it could be one row per user, one row per variant per day, or one row per variant. Say which grain each step of your plan needs, and what you would do if only aggregates are available.
A is the current flow (control) and B is the redesign. Assignment was meant to be random and sticky per user, with a planned split; state the split you assume.
A stopping rule (duration or sample size, plus any interim looks) was pre-registered before launch.
No baselines, sample sizes, costs or business values are given. Wherever your plan needs a number (a threshold, an MDE, a decision weight), state the assumption and show how the number would be computed.
activation_d7
and
revenue_d30
are only complete 7 and 30 days after the event that starts their window.
Clarifying Questions to Ask Guidance
What is the grain of the data file, and can I get the user-level data behind it (and request-level data for latency)?
What was randomized (user, device or session), at what split, and where does the exposure event fire: at assignment or when the signup page renders?
How is "activation" defined, and what is the denominator of
refund_rate
(all signups, paying users or transactions)?
What did the pre-registration specify: primary metric, minimum effect of interest,
α
, duration and interim looks?
How are support tickets attributed to the signup flow, and can each ticket be linked to the user's variant?
Did anything else change during the test window, such as marketing campaigns, other launches or outages?
Part 1: Data quality
Before reading any effect, how do you validate the data? Cover a sample ratio mismatch (SRM) test, bucketing integrity, and a reconciliation of exposure logs against analytics counts. Give the test statistic and threshold you would use, and say what you do if a check fails.
What This Part Should Cover Guidance
An SRM test on the randomization unit, with an explicit statistic and threshold, repeated by day and by segment.
Bucketing checks: reproducible assignment, stickiness and crossovers, and interaction with other experiments.
Reconciliation of assignment, exposure, analytics and backend counts, compared between arms.
A clear rule for what a failed check means for the readout.
Part 2: Primary and guardrail metrics
Choose the primary metric and the guardrail metrics, from the columns given and any you would derive from them. For each, give the unit, numerator, denominator and time window, and justify the choice.
What This Part Should Cover Guidance
One primary metric tied to business value, measured per randomized unit, with its window.
Guardrails for performance, support load and money (refunds and revenue), each with a margin set before the test and a reason for it.
A clear separation between diagnostic metrics (conditional rates, funnel steps) and decision metrics.
Part 3: Effect estimates and confidence intervals
For each metric, how do you estimate the treatment effect and its confidence interval? Include ratio metrics and a non-parametric option for skewed metrics, and say whether, when and how you would apply variance reduction such as CUPED.
What This Part Should Cover Guidance
The right estimator and interval for each metric type (proportion, mean, ratio, quantile), as absolute and relative effects.
Variances that match the randomization unit, with the method chosen for each metric type.
Treatment of skew with the estimand stated (mean, winsorized mean or distribution shift).
Variance reduction with valid pre-treatment covariates and a realistic view of how much it helps here.
Part 4: Power and the stopping rule
Check whether the test had enough power: compute the minimum detectable effect (MDE) for the primary metric at the achieved sample size, and check whether the test's actual duration and number of looks met the pre-registered stopping rule.
What This Part Should Cover Guidance
The MDE formula with numbers plugged in, compared with the smallest effect that matters to the business.
Accounting for the matured sample behind the 7-day and 30-day metrics.
An audit of the stopping rule (full weekly cycles, planned sample reached, peeking) and what to do if it was broken.
Part 5: Novelty and learning effects
How do you check whether the measured effect is a novelty or learning effect rather than a steady-state effect? Use both time-sliced and cohort views.
What This Part Should Cover Guidance
A cohort analysis by first-exposure date on matured outcomes, with a test for a trend.
A split of the effect by visitor type, with a reason for each group's expected behavior.
Operational learning visible in latency and tickets over calendar time.
How the decision changes if the effect is decaying.
Part 6: The ship / no-ship decision
Make the ship / no-ship call with a concrete decision framework that balances activation gains against increased latency and support tickets.
Clarifying Questions for this Part Guidance
Is there a hard latency SLO, or a cap on support capacity, that would block a launch whatever the value?
What is an activated user worth, what does a support ticket cost, and who owns those numbers?
Is a partial launch (a holdback, or a launch to some segments only) an option?
What This Part Should Cover Guidance
An ordered decision framework that separates what can veto a launch from what can be traded off, and says what happens in every outcome of the primary metric.
A value model in one currency, with its uncertainty and a break-even sensitivity.
A clear recommendation with next steps (mitigation, holdback or rerun).
Part 7: Insights beyond the ship decision
List at least three additional insights you would extract from this experiment beyond the ship decision (for example heterogeneity by traffic source or device, step-drop analysis, form-field sensitivity), and how you would compute each.
Clarifying Questions for this Part Guidance
Which step-level and field-level events are logged (step views, field focus, validation errors), and are they logged the same way in both variants?
What This Part Should Cover Guidance
At least three distinct, actionable insights, each with its method.
Statistical discipline for exploratory cuts: interaction tests and multiple-comparison control.
A link from each insight to a product or engineering action.
What a Strong Answer Covers Guidance
Across all parts:
A consistent unit of analysis: every variance matches how users were randomized.
A firm line between the pre-registered decision metrics and exploratory or diagnostic cuts.
Concrete formulas, tests and thresholds, with any assumed numbers flagged as assumptions.
Discipline about outcome windows (7-day and 30-day maturation) in every step.
A recommendation a mixed audience can act on, with its uncertainty stated plainly.
Follow-up Questions Guidance
Signups rose 6% in B but activation per exposed user is flat. What happened, and what do you recommend?
If the file has only one row of totals per variant, which of your intervals can you still compute, and which can you not?
The SRM test gives
p=0.0002
, yet the activation lift is large and significant. What do you do?
You only observe 30 days of revenue. How would you estimate the long-term value of the extra activations, and check that they are worth as much as typical ones?