How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Gusto.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Gusto during technical interviews.

Compute p-values for 2 variants vs control

Quick Overview

This question evaluates a candidate's competence in statistical hypothesis testing and experiment analysis, focusing on A/B/n comparison of conversion rates, p-value computation, confidence interval estimation for lift, multiple-testing considerations, and interpretation of results.

A/B/n test: compute p-values and make a ship decision

You ran an online A/B/n experiment with 1 control and 2 treatment variants (A/B/C).

You are given a table of aggregated results with one row per group:

Table: ab_results

group (STRING): one of control , variant_1 , variant_2
users (INT): number of unique users exposed to the group
conversions (INT): number of users who converted (binary outcome)

Assume:

Users are independently assigned and each user appears in exactly one group.
The metric is conversion rate = conversions / users .
You want to test whether each variant changes conversion rate vs control.
Use a two-sided test unless you justify a one-sided test.

Tasks

Using Python, compute the p-value for:
- variant_1 vs control
- variant_2 vs control (State what statistical test you chose and why.)
Provide 95% confidence intervals for the lift (difference in conversion rates) for each variant vs control.
Because there are two comparisons vs the same control, explain how you would handle multiple testing (e.g., Bonferroni, Holm, FDR), and how that affects your decision.
Interpret the results in plain language and recommend ship / no-ship (and under what conditions you’d run a follow-up experiment).

Include any assumptions or caveats (e.g., sample size adequacy, novelty effects, missing data, metric definition issues).

Quick Overview

A/B/n test: compute p-values and make a ship decision

You ran an online A/B/n experiment with 1 control and 2 treatment variants (A/B/C).

You are given a table of aggregated results with one row per group:

Table: ab_results

group (STRING): one of control , variant_1 , variant_2

users (INT): number of unique users exposed to the group

conversions (INT): number of users who converted (binary outcome)

Assume:

Users are independently assigned and each user appears in exactly one group.

The metric is conversion rate = conversions / users .

You want to test whether each variant changes conversion rate vs control.

Use a two-sided test unless you justify a one-sided test.

Tasks

Using Python, compute the p-value for:

variant_1 vs control
variant_2 vs control (State what statistical test you chose and why.)

Provide 95% confidence intervals for the lift (difference in conversion rates) for each variant vs control.

Because there are two comparisons vs the same control, explain how you would handle multiple testing (e.g., Bonferroni, Holm, FDR), and how that affects your decision.

Interpret the results in plain language and recommend ship / no-ship (and under what conditions you’d run a follow-up experiment).

Include any assumptions or caveats (e.g., sample size adequacy, novelty effects, missing data, metric definition issues).

Compute p-values for 2 variants vs control

Quick Overview

A/B/n test: compute p-values and make a ship decision

Tasks

Solution

Submit Your Answer to Earn 20XP

Compute p-values for 2 variants vs control

Quick Overview

A/B/n test: compute p-values and make a ship decision

Tasks

Solution

Submit Your Answer to Earn 20XP