Design Metrics to Measure Inappropriate Content Severity and Prevalence
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
##### Scenario
Product team is launching a new harmful-content detection system and needs a comprehensive measurement plan.
##### Question
Propose metrics to measure the severity and prevalence of inappropriate content. Explain why each is chosen and list pros/cons.
Define and justify the View Prevalence metric.
Design an online A/B experiment to evaluate the new model: state hypothesis, primary success metric, guardrails, sample-size and runtime estimations, and steps to interpret results.
##### Hints
Tie metrics to user harm; weigh severity vs frequency; outline power analysis, segment checks, and risk mitigations.
Quick Answer: This question evaluates a data scientist's ability to design measurement and experimentation for harmful-content detection, including selecting metrics for severity and prevalence, defining a view-prevalence metric, and constructing online A/B tests with power analysis, guardrails, and segment checks.