How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Choose KPIs for short-video recommendations

Quick Overview

This question evaluates a data scientist's ability to define precise product metrics, set guardrails, design and power A/B tests, and apply weighted decision frameworks for a short‑video recommender, covering metric definition, statistical analysis, experiment logistics, and escalation rules.

Instagram Short‑Video Recommender: Metrics, Decisioning, and Experiment Design

Task

Instagram launched a new short‑video recommender. Do the following:

Choose one primary success metric and define it precisely (formula, numerator/denominator, unit of analysis, time window, outlier handling).
List at least three guardrail metrics and explain why they matter.
You observe these A/B test deltas: mean watch time per session = +3%, session starts per user = −2%. Decide whether to ship using a weighted decision framework and justify.
Design the A/B test: assignment (user- or session-level), novelty warmup, SRM checks, bucketing, holdout length, and seasonality controls.
Powering: Target MDE is a +2% relative lift on mean watch time per session; baseline mean = 120s, SD = 180s, alpha = 0.05, power = 0.8, 50/50 split, 10M DAU. Estimate required sample size per arm, expected runtime, and key failure modes (e.g., creator‑side regressions, regressions in content diversity).
Propose tie‑breaker rules and escalation criteria when "metric A up, metric B down."

Quick Overview

Task

Instagram launched a new short‑video recommender. Do the following:

Choose one primary success metric and define it precisely (formula, numerator/denominator, unit of analysis, time window, outlier handling).

List at least three guardrail metrics and explain why they matter.

You observe these A/B test deltas: mean watch time per session = +3%, session starts per user = −2%. Decide whether to ship using a weighted decision framework and justify.

Design the A/B test: assignment (user- or session-level), novelty warmup, SRM checks, bucketing, holdout length, and seasonality controls.

Powering: Target MDE is a +2% relative lift on mean watch time per session; baseline mean = 120s, SD = 180s, alpha = 0.05, power = 0.8, 50/50 split, 10M DAU. Estimate required sample size per arm, expected runtime, and key failure modes (e.g., creator‑side regressions, regressions in content diversity).

Propose tie‑breaker rules and escalation criteria when "metric A up, metric B down."

Choose KPIs for short-video recommendations

Quick Overview

Instagram Short‑Video Recommender: Metrics, Decisioning, and Experiment Design

Task

Solution

Submit Your Answer to Earn 20XP

Choose KPIs for short-video recommendations

Quick Overview

Instagram Short‑Video Recommender: Metrics, Decisioning, and Experiment Design

Task

Solution

Submit Your Answer to Earn 20XP