##### Scenario You compared two large-language models in an A/B test: Model A success = 70 % (700/ 1000), Model B = 80 % (800/ 1000). ##### Question State hypotheses and compute a two-proportion z-statistic and p-value to decide if Model B is significantly better at α = 0.05; also give the 95 % confidence interval of the lift. ##### Hints p̂ = (x1/n1 – x2/n 2) / sqrt(p*(1-p)*(1/n1+1/n 2)), where p is pooled proportion.

This question evaluates a data scientist's competency in statistical hypothesis testing for proportions, specifically two-proportion z-tests and confidence interval estimation within A/B experimentation.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Determine Significance of Model B's Performance Improvement

A/B Test: Two-Proportion Z-Test for Success Rates

Scenario

You ran an A/B test comparing two large language models (LLMs):

Model A: 700 successes out of 1000 trials (p_A = 0.70)
Model B: 800 successes out of 1000 trials (p_B = 0.80)

Task

State the hypotheses to test whether Model B is better than Model A at α = 0.05.
Compute the two-proportion z-statistic (using the pooled standard error) and the corresponding p-value.
Decide if Model B is significantly better at α = 0.05.
Compute the 95% confidence interval for the lift (assume lift = p_B − p_A, the absolute difference in success rates).

Hint (pooled z-test):

z = (p_B − p_A) / sqrt(p*(1 − p)*(1/n_A + 1/n_B)), where p is the pooled proportion p = (x_A + x_B)/(n_A + n_B).

A/B Test: Two-Proportion Z-Test for Success Rates

Scenario

You ran an A/B test comparing two large language models (LLMs):

Model A: 700 successes out of 1000 trials (p_A = 0.70)
Model B: 800 successes out of 1000 trials (p_B = 0.80)

Task

State the hypotheses to test whether Model B is better than Model A at α = 0.05.
Compute the two-proportion z-statistic (using the pooled standard error) and the corresponding p-value.
Decide if Model B is significantly better at α = 0.05.
Compute the 95% confidence interval for the lift (assume lift = p_B − p_A, the absolute difference in success rates).

Hint (pooled z-test):

z = (p_B − p_A) / sqrt(p*(1 − p)*(1/n_A + 1/n_B)), where p is the pooled proportion p = (x_A + x_B)/(n_A + n_B).

Determine Significance of Model B's Performance Improvement

Quick Overview

A/B Test: Two-Proportion Z-Test for Success Rates

Scenario

Task

Solution

Submit Your Answer to Earn 20XP

Determine Significance of Model B's Performance Improvement

Quick Overview

A/B Test: Two-Proportion Z-Test for Success Rates

Scenario

Task

Solution

Submit Your Answer to Earn 20XP