Compute p-values for 2 variants vs control

A/B/n test: compute p-values and make a ship decision

You ran an online A/B/n experiment with 1 control and 2 treatment variants (A/B/C).

You are given a table of aggregated results with one row per group:

Table: ab_results

• group (STRING): one of control , variant_1 , variant_2
• users (INT): number of unique users exposed to the group
• conversions (INT): number of users who converted (binary outcome)

Assume:

• Users are independently assigned and each user appears in exactly one group.
• The metric is conversion rate = conversions / users .
• You want to test whether each variant changes conversion rate vs control.
• Use a two-sided test unless you justify a one-sided test.

Tasks

1. Using Python, compute the p-value for:
   • variant_1 vs control
   • variant_2 vs control (State what statistical test you chose and why.)
2. Provide 95% confidence intervals for the lift (difference in conversion rates) for each variant vs control.
3. Because there are two comparisons vs the same control, explain how you would handle multiple testing (e.g., Bonferroni, Holm, FDR), and how that affects your decision.
4. Interpret the results in plain language and recommend ship / no-ship (and under what conditions you’d run a follow-up experiment).

Include any assumptions or caveats (e.g., sample size adequacy, novelty effects, missing data, metric definition issues).