Design A/B Test for New Recommendation Algorithm Launch

Quick Overview

Design A/B Test for New Recommendation Algorithm Launch evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design A/B Test for New Recommendation Algorithm Launch

Company: Snapchat

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

##### Scenario Product team wants to launch a new recommendation algorithm and asks for a rigorous A/B test plan. ##### Question Design an A/B test to measure the uplift of the new recommender on GMV. Which primary and guardrail metrics will you track and why? Compute required sample size assuming 3% baseline conversion, 7% relative lift, α=0.05, power=0.8. How would you address novelty effect and uneven seasonality across groups? Explain how you would interpret results if the primary metric is flat but secondary engagement metrics improve. ##### Hints Talk randomization, CUPED, sequential testing, and post-test segmentation.

Quick Answer: Design A/B Test for New Recommendation Algorithm Launch evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Snapchat
Snapchat logo
Snapchat
Aug 4, 2025, 10:55 AM
mediumData ScientistOnsiteAnalytics & Experimentation
82
0

Design A/B Test for New Recommendation Algorithm Launch

A/B Test Design: New Recommendation Algorithm

Objective

Design a rigorous A/B test to estimate the incremental impact of a new recommendation algorithm on gross merchandise value (GMV).

Tasks

  1. Define the experimental design and randomization strategy.
  2. Specify the primary metric and guardrail metrics, and justify each choice.
  3. Compute the per-variant sample size given:
    • Baseline conversion rate: 3%
    • Expected relative lift: 7%
    • Significance level: α = 0.05 (two-sided)
    • Power: 0.8
  4. Explain how to handle novelty effects and uneven seasonality across groups.
  5. Describe how you would interpret results if the primary GMV metric is flat but secondary engagement metrics improve.

Hints

  • Discuss randomization strategy, CUPED or other variance reduction, sequential testing, and post-test segmentation.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Snapchat•More Data Scientist•Snapchat Data Scientist•Snapchat Analytics & Experimentation•Data Scientist Analytics & Experimentation