PracHub
QuestionsPremiumCoachesLearningGuidesInterview Prep
|Home/Analytics & Experimentation/Gusto

How do you test two variants vs control?

Last updated: Apr 15, 2026

Quick Overview

This question evaluates proficiency in A/B/n experimentation, hypothesis testing for binomial proportions, multiple-comparison correction methods, confidence interval estimation for conversion uplift, and experiment-related decision-making in the Analytics & Experimentation domain at an applied/intermediate statistical-analysis level.

  • medium
  • Gusto
  • Analytics & Experimentation
  • Data Scientist

How do you test two variants vs control?

Company: Gusto

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

You ran an A/B/n experiment with 1 control and 2 treatment variants. The primary metric is **conversion rate** (each user either converts or not within the experiment window). Users are independently assigned to groups. You are given the following aggregated results: | group | users (n) | conversions (x) | |---|---:|---:| | control | 50,000 | 5,000 | | variant_a | 50,000 | 5,250 | | variant_b | 50,000 | 5,400 | Assumptions: - Two-sided tests unless specified. - Significance level b1 = 0.05. - Metric is a binomial proportion; large-sample normal approximations are acceptable. Tasks: 1) In **Python**, compute the **p-value** for each variant vs control using an appropriate statistical test for proportions. 2) Because there are two variants, address **multiple comparisons** (e.g., Bonferroni or FDR). Report adjusted significance conclusions. 3) Provide 95% confidence intervals for the uplift (absolute and/or relative) for each variant vs control. 4) Make a **ship / no-ship** recommendation. Explain what additional checks you would do before shipping (e.g., guardrail metrics, segmentation concerns, SRM, novelty effects), and how you would communicate uncertainty to stakeholders. (You may use `statsmodels`/`scipy` or implement the formulas directly.)

Quick Answer: This question evaluates proficiency in A/B/n experimentation, hypothesis testing for binomial proportions, multiple-comparison correction methods, confidence interval estimation for conversion uplift, and experiment-related decision-making in the Analytics & Experimentation domain at an applied/intermediate statistical-analysis level.

Related Interview Questions

  • Define Churn and Design Onboarding Experiment - Gusto (medium)
  • Define Product Health and Experiment Design - Gusto (medium)
  • Compute p-values for 2 variants vs control - Gusto (easy)
  • Identify Growth Opportunities for New Payroll Feature Launch - Gusto (medium)
  • Investigate Causes of Increased Payroll Processing Time - Gusto (medium)
Gusto logo
Gusto
Nov 30, 2025, 12:00 AM
Data Scientist
Technical Screen
Analytics & Experimentation
5
0

You ran an A/B/n experiment with 1 control and 2 treatment variants. The primary metric is conversion rate (each user either converts or not within the experiment window). Users are independently assigned to groups.

You are given the following aggregated results:

groupusers (n)conversions (x)
control50,0005,000
variant_a50,0005,250
variant_b50,0005,400

Assumptions:

  • Two-sided tests unless specified.
  • Significance level b1 = 0.05.
  • Metric is a binomial proportion; large-sample normal approximations are acceptable.

Tasks:

  1. In Python , compute the p-value for each variant vs control using an appropriate statistical test for proportions.
  2. Because there are two variants, address multiple comparisons (e.g., Bonferroni or FDR). Report adjusted significance conclusions.
  3. Provide 95% confidence intervals for the uplift (absolute and/or relative) for each variant vs control.
  4. Make a ship / no-ship recommendation. Explain what additional checks you would do before shipping (e.g., guardrail metrics, segmentation concerns, SRM, novelty effects), and how you would communicate uncertainty to stakeholders.

(You may use statsmodels/scipy or implement the formulas directly.)

Solution

Show

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Gusto•More Data Scientist•Gusto Data Scientist•Gusto Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.