Determine Appropriate Statistical Test for Comparing Means

Quick Overview

Pinterest statistics prompt on comparing mean active minutes in an A/B test, covering Welch's t-test, assumptions, p-values, confidence intervals, Type I and Type II errors, and multiple-comparison adjustments.

Determine Appropriate Statistical Test for Comparing Means

Company: Pinterest

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Onsite

##### Scenario Two weeks of experiment data are now available and you must report statistical significance of the new algorithm. ##### Question Which statistical test is most appropriate for comparing mean active minutes between control and treatment? State the assumptions and how you would validate them. Calculate the p-value and 95% confidence interval; interpret both. Discuss Type I and Type II errors in this context and how you would adjust for multiple comparisons if the team also tracked five secondary metrics. ##### Hints Mention t-test vs. non-parametric options; Bonferroni/Holm corrections; power calculations.

Quick Answer: Pinterest statistics prompt on comparing mean active minutes in an A/B test, covering Welch's t-test, assumptions, p-values, confidence intervals, Type I and Type II errors, and multiple-comparison adjustments.

|Home/Statistics & Math/Pinterest
Pinterest logo
Pinterest
Jul 12, 2025, 6:59 PM
mediumData ScientistOnsiteStatistics & Math
82
0

Statistical Test for Comparing Mean Active Minutes

You have two weeks of experiment data for a new algorithm. The primary metric is user active minutes. Each user is assigned to control or treatment for the full duration.

Analyze the per-user total or average active minutes over the two-week window.

Constraints & Assumptions

  • The unit of analysis should match the randomization unit.
  • Active minutes are nonnegative and likely right-skewed.
  • Explain assumptions and validation checks.
  • Discuss multiple comparisons for five secondary metrics.

Clarifying Questions to Ask Guidance

  • Are users randomized independently?
  • Do we have one row per user or repeated daily rows?
  • Are group sizes and variances equal?
  • Is the business interested in means, medians, or distributional effects?

What a Strong Answer Covers Guidance

  • Recommended test: Welch's two-sample t-test on per-user aggregated active minutes for mean comparison.
  • Alternatives: permutation test for the mean, bootstrap confidence intervals, Mann-Whitney as a distributional/median robustness check, and cluster-robust or mixed models for repeated daily data.
  • Assumptions: stable randomization, independent users, no major interference, correct unit of analysis, enough sample size for CLT, and no severe data-quality issues.
  • Validation: SRM test, covariate balance, histogram/tails, variance comparison, time-series by arm, and instrumentation checks.
  • p-value and 95% CI computation using mean difference, standard error, Welch-Satterthwaite degrees of freedom, and business interpretation.
  • Type I and Type II error explanation in the product context.
  • Multiple-comparison adjustment using Holm-Bonferroni, Benjamini-Hochberg, or pre-specified metric hierarchy.

Follow-up Questions Guidance

  • Why use Welch's t-test instead of Student's t-test?
  • What if one user has extreme minutes?
  • How would you analyze daily repeated measurements?
  • How would you communicate a statistically significant but tiny effect?
Loading comments...

Browse More Questions

More Statistics & Math•More Pinterest•More Data Scientist•Pinterest Data Scientist•Pinterest Statistics & Math•Data Scientist Statistics & Math