PracHub
QuestionsPremiumCoachesLearningGuidesInterview Prep
|Home/Statistics & Math/Meta

Control error under multiple testing

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a candidate's understanding of multiple hypothesis testing, sequential monitoring, and error-rate control—specifically familywise error rate (FWER) and false discovery rate (FDR)—when handling correlated primary and secondary metrics in experiments.

  • hard
  • Meta
  • Statistics & Math
  • Data Scientist

Control error under multiple testing

Company: Meta

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Onsite

You have 1 primary and 12 secondary metrics with correlations, and you plan 4 weekly looks. a) Propose a hierarchy that preserves strong FWER control for confirmatory endpoints and FDR control for exploratory ones. b) If there are k=5 confirmatory metrics at α=0.05, compute the Holm–Bonferroni adjusted threshold for the smallest p‑value. c) For the remaining k=8 exploratory metrics, show how to apply BH at q=0.10 and discuss how to combine with group‑sequential alpha spending or always‑valid p‑values to avoid double‑counting looks.

Quick Answer: This question evaluates a candidate's understanding of multiple hypothesis testing, sequential monitoring, and error-rate control—specifically familywise error rate (FWER) and false discovery rate (FDR)—when handling correlated primary and secondary metrics in experiments.

Related Interview Questions

  • Compute probability an account is fake - Meta (easy)
  • Compute Bayes probability for fake accounts - Meta (easy)
  • Compute probabilities for chatbot response quality - Meta (easy)
  • Compute posterior fake probability using Bayes' rule - Meta (medium)
  • Estimate bots and CI from DAU spike - Meta (medium)
Meta logo
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Statistics & Math
1
0
Loading...

Multiple Testing and Sequential Monitoring with 1 Primary and 12 Secondary Metrics

Context

You are monitoring an A/B experiment over 4 weekly looks. There is 1 primary metric and 12 secondary metrics, and many are correlated. You need strong familywise error rate (FWER) control for confirmatory claims and false discovery rate (FDR) control for exploratory findings.

Assume 5 metrics are pre-specified as confirmatory (including the primary), and the remaining 8 are exploratory.

Tasks

(a) Propose a hierarchy that preserves strong FWER control for confirmatory endpoints and FDR control for exploratory endpoints across 4 looks.

(b) If there are k = 5 confirmatory metrics at α = 0.05, compute the Holm–Bonferroni adjusted threshold for the smallest p-value.

(c) For the remaining k = 8 exploratory metrics, show how to apply the Benjamini–Hochberg (BH) procedure at q = 0.10, and discuss how to combine this with group-sequential alpha spending or always-valid p-values so repeated looks do not inflate error rates.

Solution

Show

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More Statistics & Math•More Meta•More Data Scientist•Meta Data Scientist•Meta Statistics & Math•Data Scientist Statistics & Math
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.