Design A/B Test to Evaluate New Video-Feed Feature

Q: Design A/B Test to Evaluate New Video-Feed Feature

This question evaluates a candidate's skills in experimental design, statistical inference, metric definition, and product analytics for measuring a new short-video feed feature, with attention to randomization, cross-device consistency, sample size, guardrail metrics, and interference risks.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

A consumer social-media app is launching a short‑video feed (TikTok-style). A newly added feed feature (e.g., UI change, ranking tweak, or interaction control) may alter how users consume and engage with videos.

Task

Design an end-to-end A/B test to measure the feature's impact. Address:

Experiment unit and eligibility criteria.
Randomization and bucketing (including cross-device consistency).
Primary and secondary success metrics with definitions and rationale.
Guardrail/health metrics and thresholds.
Sample-size and test-duration needs; state assumptions and how you would compute MDE/sample size.
Handling novelty effects and seasonality.
Monitoring, analysis plan, and decision criteria.
Risks (e.g., interference/network effects) and mitigations.

Hints: Consider activation, retention, engagement, and negative outcomes. Outline the experiment design and what would make you ship, iterate, or roll back.

Design A/B Test to Evaluate New Video-Feed Feature

Scenario

Task

Solution

Comments (0)