Determine Superiority of Model A Using Hypothesis Testing

Q: Determine Superiority of Model A Using Hypothesis Testing

Meta statistics prompt on testing whether Model A beats Model B using a two-proportion z-test, covering pooled variance, p-value, confidence interval, power, and practical significance.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Hypothesis Test: Is Model A Better Than Model B?

A search feature marks a session as successful only when both relevancy and accuracy binary flags equal 1.

Two ranking models were tested independently with equal traffic:

Model A: 100 users, 90 successes
Model B: 100 users, 85 successes

Using only these data, determine whether you can conclude that Model A is better than Model B at a 5% significance level.

Constraints & Assumptions

Treat each user outcome as an independent binary success/failure.
Use a two-proportion test.
State whether the test is one-sided or two-sided.
Include statistical and practical interpretation.

Clarifying Questions to Ask Guidance

Was traffic randomized between the models?
Are users independent, or can one user appear multiple times?
Is "better" one-sided, or do we need to detect any difference?
What minimum detectable effect would justify shipping Model A?

What a Strong Answer Covers Guidance

Hypotheses for a two-proportion z-test, typically H0: p_A = p_B and H1: p_A > p_B for a one-sided superiority test.
Sample proportions: 0.90 for A and 0.85 for B.
Pooled proportion, pooled standard error, z-statistic, and p-value.
95% confidence interval for the difference using an unpooled standard error.
Conclusion that the observed 5 percentage-point lift is not statistically significant with 100 users per arm at 5%.
Comment that the test is underpowered for a 5 percentage-point effect at these sample sizes.
Practical significance and decision guidance: do not conclude superiority from these data alone; collect more data or evaluate business risk.

Follow-up Questions Guidance

How many users per arm would you need to detect a 5 percentage-point lift?
What changes if you use a two-sided test?
How would correlated user sessions affect the test?
How would you combine relevancy and accuracy if they are separate outcomes?

Determine Superiority of Model A Using Hypothesis Testing

Quick Overview

Determine Superiority of Model A Using Hypothesis Testing

Hypothesis Test: Is Model A Better Than Model B?

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer