##### Scenario Two weeks of experiment data are now available and you must report statistical significance of the new algorithm. ##### Question Which statistical test is most appropriate for comparing mean active minutes between control and treatment? State the assumptions and how you would validate them. Calculate the p-value and 95% confidence interval; interpret both. Discuss Type I and Type II errors in this context and how you would adjust for multiple comparisons if the team also tracked five secondary metrics. ##### Hints Mention t-test vs. non-parametric options; Bonferroni/Holm corrections; power calculations.

Pinterest statistics prompt on comparing mean active minutes in an A/B test, covering Welch's t-test, assumptions, p-values, confidence intervals, Type I and Type II errors, and multiple-comparison adjustments.

How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Onsite rounds at Pinterest.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Pinterest during technical interviews.

Determine Appropriate Statistical Test for Comparing Means

Statistical Test for Comparing Mean Active Minutes

You have two weeks of experiment data for a new algorithm. The primary metric is user active minutes. Each user is assigned to control or treatment for the full duration.

Analyze the per-user total or average active minutes over the two-week window.

Constraints & Assumptions

The unit of analysis should match the randomization unit.
Active minutes are nonnegative and likely right-skewed.
Explain assumptions and validation checks.
Discuss multiple comparisons for five secondary metrics.

Clarifying Questions to Ask

Are users randomized independently?
Do we have one row per user or repeated daily rows?
Are group sizes and variances equal?
Is the business interested in means, medians, or distributional effects?

What a Strong Answer Covers

Recommended test: Welch's two-sample t-test on per-user aggregated active minutes for mean comparison.
Alternatives: permutation test for the mean, bootstrap confidence intervals, Mann-Whitney as a distributional/median robustness check, and cluster-robust or mixed models for repeated daily data.
Assumptions: stable randomization, independent users, no major interference, correct unit of analysis, enough sample size for CLT, and no severe data-quality issues.
Validation: SRM test, covariate balance, histogram/tails, variance comparison, time-series by arm, and instrumentation checks.
p-value and 95% CI computation using mean difference, standard error, Welch-Satterthwaite degrees of freedom, and business interpretation.
Type I and Type II error explanation in the product context.
Multiple-comparison adjustment using Holm-Bonferroni, Benjamini-Hochberg, or pre-specified metric hierarchy.

Follow-up Questions

Why use Welch's t-test instead of Student's t-test?
What if one user has extreme minutes?
How would you analyze daily repeated measurements?
How would you communicate a statistically significant but tiny effect?

Statistical Test for Comparing Mean Active Minutes

You have two weeks of experiment data for a new algorithm. The primary metric is user active minutes. Each user is assigned to control or treatment for the full duration.

Analyze the per-user total or average active minutes over the two-week window.

Constraints & Assumptions

The unit of analysis should match the randomization unit.

Active minutes are nonnegative and likely right-skewed.

Explain assumptions and validation checks.

Discuss multiple comparisons for five secondary metrics.

Clarifying Questions to Ask

Are users randomized independently?

Do we have one row per user or repeated daily rows?

Are group sizes and variances equal?

Is the business interested in means, medians, or distributional effects?

What a Strong Answer Covers

Recommended test: Welch's two-sample t-test on per-user aggregated active minutes for mean comparison.

Alternatives: permutation test for the mean, bootstrap confidence intervals, Mann-Whitney as a distributional/median robustness check, and cluster-robust or mixed models for repeated daily data.

Assumptions: stable randomization, independent users, no major interference, correct unit of analysis, enough sample size for CLT, and no severe data-quality issues.

Validation: SRM test, covariate balance, histogram/tails, variance comparison, time-series by arm, and instrumentation checks.

p-value and 95% CI computation using mean difference, standard error, Welch-Satterthwaite degrees of freedom, and business interpretation.

Type I and Type II error explanation in the product context.

Multiple-comparison adjustment using Holm-Bonferroni, Benjamini-Hochberg, or pre-specified metric hierarchy.

Follow-up Questions

Why use Welch's t-test instead of Student's t-test?

What if one user has extreme minutes?

How would you analyze daily repeated measurements?

How would you communicate a statistically significant but tiny effect?

Determine Appropriate Statistical Test for Comparing Means

Quick Overview

Determine Appropriate Statistical Test for Comparing Means

Statistical Test for Comparing Mean Active Minutes

Constraints & Assumptions

Clarifying Questions to Ask

What a Strong Answer Covers

Follow-up Questions

Write your answer

Determine Appropriate Statistical Test for Comparing Means

Quick Overview

Determine Appropriate Statistical Test for Comparing Means

Statistical Test for Comparing Mean Active Minutes

Constraints & Assumptions

Clarifying Questions to Ask

What a Strong Answer Covers

Follow-up Questions

Write your answer