How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Determine Significance of Model B's Performance Improvement | Meta Interview Question

Determine Significance of Model B's Performance Improvement

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a data scientist's competency in statistical hypothesis testing for proportions, specifically two-proportion z-tests and confidence interval estimation within A/B experimentation.

A/B Test: Two-Proportion Z-Test for Success Rates

Scenario

You ran an A/B test comparing two large language models (LLMs):

Model A: 700 successes out of 1000 trials (p_A = 0.70)
Model B: 800 successes out of 1000 trials (p_B = 0.80)

Task

State the hypotheses to test whether Model B is better than Model A at α = 0.05.
Compute the two-proportion z-statistic (using the pooled standard error) and the corresponding p-value.
Decide if Model B is significantly better at α = 0.05.
Compute the 95% confidence interval for the lift (assume lift = p_B − p_A, the absolute difference in success rates).

Hint (pooled z-test):

z = (p_B − p_A) / sqrt(p*(1 − p)*(1/n_A + 1/n_B)), where p is the pooled proportion p = (x_A + x_B)/(n_A + n_B).

Solution

Show

Comments (0)

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation