Compute p-values for 2 variants vs control
Company: Gusto
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## A/B/n test: compute p-values and make a ship decision
You ran an online A/B/n experiment with **1 control** and **2 treatment variants** (A/B/C).
You are given a table of aggregated results with one row per group:
**Table: `ab_results`**
- `group` (STRING): one of `control`, `variant_1`, `variant_2`
- `users` (INT): number of unique users exposed to the group
- `conversions` (INT): number of users who converted (binary outcome)
Assume:
- Users are independently assigned and each user appears in exactly one group.
- The metric is **conversion rate** = `conversions / users`.
- You want to test whether each variant changes conversion rate vs control.
- Use a two-sided test unless you justify a one-sided test.
### Tasks
1. Using Python, compute the **p-value** for:
- `variant_1` vs `control`
- `variant_2` vs `control`
(State what statistical test you chose and why.)
2. Provide **95% confidence intervals** for the lift (difference in conversion rates) for each variant vs control.
3. Because there are **two** comparisons vs the same control, explain how you would handle **multiple testing** (e.g., Bonferroni, Holm, FDR), and how that affects your decision.
4. Interpret the results in plain language and recommend **ship / no-ship** (and under what conditions you’d run a follow-up experiment).
Include any assumptions or caveats (e.g., sample size adequacy, novelty effects, missing data, metric definition issues).
Quick Answer: This question evaluates a candidate's competence in statistical hypothesis testing and experiment analysis, focusing on A/B/n comparison of conversion rates, p-value computation, confidence interval estimation for lift, multiple-testing considerations, and interpretation of results.