Determine Significance of Model B's Performance Improvement

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Significance of Model B's Performance Improvement states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Determine Significance of Model B's Performance Improvement

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: easy

Interview Round: Onsite

##### Scenario You compared two large-language models in an A/B test: Model A success = 70 % (700/ 1000), Model B = 80 % (800/ 1000). ##### Question State hypotheses and compute a two-proportion z-statistic and p-value to decide if Model B is significantly better at α = 0.05; also give the 95 % confidence interval of the lift. ##### Hints p̂ = (x1/n1 – x2/n 2) / sqrt(p*(1-p)*(1/n1+1/n 2)), where p is pooled proportion.

Overview: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Significance of Model B's Performance Improvement states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Aug 4, 2025
easyData ScientistOnsiteAnalytics & Experimentation
5
0

Determine Significance of Model B's Performance Improvement

A/B Test: Two-Proportion Z-Test for Success Rates

Scenario

You ran an A/B test comparing two large language models (LLMs):

  • Model A: 700 successes out of 1000 trials (p_A = 0.70)
  • Model B: 800 successes out of 1000 trials (p_B = 0.80)

Task

  1. State the hypotheses to test whether Model B is better than Model A at α = 0.05.
  2. Compute the two-proportion z-statistic (using the pooled standard error) and the corresponding p-value.
  3. Decide if Model B is significantly better at α = 0.05.
  4. Compute the 95% confidence interval for the lift (assume lift = p_B − p_A, the absolute difference in success rates).

Hint (pooled z-test):

  • z = (p_B − p_A) / sqrt(p*(1 − p)*(1/n_A + 1/n_B)), where p is the pooled proportion p = (x_A + x_B)/(n_A + n_B).

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation