Test two models' proportions for significance

Quick Overview

This question evaluates proficiency in statistical inference for proportions, covering hypothesis testing, confidence intervals, power and sample-size calculations, multiple-testing correction, and comparison of frequentist versus Bayesian approaches within the Statistics & Math domain for data scientist roles.

Test two models' proportions for significance

Company: Meta

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Onsite

Two search models, A and B, were each used once by 100 distinct users (one query per user). Success is defined per query by your composite metric (success=1, failure=0). Model A had 90 successes, Model B had 85. Using a two-sided test at alpha=0.05: 1) State H0 and H1, choose the appropriate test (pooled two-proportion z-test), compute the test statistic and p-value, and conclude whether A outperforms B. 2) Compute a 95% confidence interval for pA−pB and interpret it for practical significance. 3) What per-arm sample size is needed to detect a +5 percentage-point uplift (baseline 85%) with 80% power at alpha=0.05? Show formulas/inputs. 4) If you simultaneously test these two models across 10 independent intents, apply a Bonferroni correction and say whether your conclusion changes. 5) Briefly explain when you would prefer an exact test or a Bayesian comparison and what you would report in each case.

Quick Answer: This question evaluates proficiency in statistical inference for proportions, covering hypothesis testing, confidence intervals, power and sample-size calculations, multiple-testing correction, and comparison of frequentist versus Bayesian approaches within the Statistics & Math domain for data scientist roles.

|Home/Statistics & Math/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
mediumData ScientistOnsiteStatistics & Math
4
0

Two search models, A and B, were each used once by 100 distinct users (one query per user). Success is defined per query by your composite metric (success=1, failure=0). Model A had 90 successes, Model B had 85. Using a two-sided test at alpha=0.05: 1) State H0 and H1, choose the appropriate test (pooled two-proportion z-test), compute the test statistic and p-value, and conclude whether A outperforms B. 2) Compute a 95% confidence interval for pA−pB and interpret it for practical significance. 3) What per-arm sample size is needed to detect a +5 percentage-point uplift (baseline 85%) with 80% power at alpha=0.05? Show formulas/inputs. 4) If you simultaneously test these two models across 10 independent intents, apply a Bonferroni correction and say whether your conclusion changes. 5) Briefly explain when you would prefer an exact test or a Bayesian comparison and what you would report in each case.

Loading comments...