Determine Appropriate Statistical Test for Comparing Means

Q: Determine Appropriate Statistical Test for Comparing Means

This question evaluates statistical inference and experimental-design competencies for A/B testing—specifically the selection and validation of hypothesis tests, estimation and interpretation of p-values and confidence intervals, characterization of Type I and Type II errors, power and minimal detectable effect calculations, and approaches for multiple comparisons control, and is commonly asked to assess a candidate's ability to draw reliable conclusions from randomized experiments. It is in the Statistics & Math category and emphasizes practical application of statistical concepts (conceptual understanding of assumptions plus applied interpretation of results) rather than purely theoretical derivations.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

A/B Test: Active Minutes After Two Weeks

Scenario

You ran a two-week A/B experiment on a new algorithm. The primary metric is user active minutes. Assume the unit of randomization is the user, and each user is assigned to control or treatment for the full duration. You will analyze the per-user total (or average) active minutes over the two weeks.

Tasks

Choose the most appropriate statistical test to compare mean active minutes between control and treatment. State the assumptions and how you would validate them.
Calculate (show how to compute) the p-value and a 95% confidence interval for the difference in means; interpret both in business terms.
Explain Type I and Type II errors for this experiment, and how you would adjust for multiple comparisons if tracking five secondary metrics.

Hints

Consider Welch's t-test vs. non-parametric options (Mann–Whitney U, permutation test) and bootstrap CIs.
For multiple comparisons, consider Holm–Bonferroni or Benjamini–Hochberg.
Discuss power calculations and minimal detectable effect (MDE).

Determine Appropriate Statistical Test for Comparing Means

A/B Test: Active Minutes After Two Weeks

Scenario

Tasks

Hints

Solution

Comments (0)