Interpret Results and Address Multiple Testing Concerns

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Interpret Results and Address Multiple Testing Concerns states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Interpret Results and Address Multiple Testing Concerns

Company: Attentive

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

##### Scenario A company runs an A/B experiment on a new message-sending method across 30 independent brands; each brand splits users 50/50 into control and test with α = 0.05. ##### Question Given that 2 brands show a statistically significant lift, 1 shows a statistically significant drop, and the remaining 27 show no significance, what conclusion(s) can you draw? How would you account for multiple testing in your answer? ##### Hints Think about expected false positives at α=0.05, family-wise error rate vs. FDR, and whether observed significant results exceed chance.

Overview: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Interpret Results and Address Multiple Testing Concerns states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Attentive
Attentive logo
Attentive
Aug 4, 2025
mediumData ScientistTechnical ScreenAnalytics & Experimentation
4
0

Interpret Results and Address Multiple Testing Concerns

Experiment Interpretation with Multiple Testing

Context

  • An A/B experiment is run independently across 30 brands.
  • Within each brand, users are split 50/50 into Control vs Test.
  • Per-brand hypothesis tests use α = 0.05 (assume two-sided tests and independence across brands).

Question

You observe:

  • 2 brands show a statistically significant lift,
  • 1 brand shows a statistically significant drop,
  • 27 brands show no statistically significant difference.

What conclusions can you draw from these results? How would you account for multiple testing in your answer?

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Attentive•More Data Scientist•Attentive Data Scientist•Attentive Analytics & Experimentation•Data Scientist Analytics & Experimentation