PracHub
QuestionsCoachesLearningGuidesInterview Prep
|Home/Analytics & Experimentation/Snapchat

Design A/B Test for New Recommendation Algorithm Launch

Last updated: Mar 29, 2026

Quick Overview

Design A/B Test for New Recommendation Algorithm Launch evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

  • medium
  • Snapchat
  • Analytics & Experimentation
  • Data Scientist

Design A/B Test for New Recommendation Algorithm Launch

Company: Snapchat

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

##### Scenario Product team wants to launch a new recommendation algorithm and asks for a rigorous A/B test plan. ##### Question Design an A/B test to measure the uplift of the new recommender on GMV. Which primary and guardrail metrics will you track and why? Compute required sample size assuming 3% baseline conversion, 7% relative lift, α=0.05, power=0.8. How would you address novelty effect and uneven seasonality across groups? Explain how you would interpret results if the primary metric is flat but secondary engagement metrics improve. ##### Hints Talk randomization, CUPED, sequential testing, and post-test segmentation.

Quick Answer: Design A/B Test for New Recommendation Algorithm Launch evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Related Interview Questions

  • Design an experiment for spam filtering impact - Snapchat (hard)
  • Decide whether to launch Group Story - Snapchat (Medium)
  • Design and analyze a banner A/B test - Snapchat (hard)
  • Design A/B Tests for Banner Ad and Group-Story Feature - Snapchat (medium)
  • Determine Optimal Energy Project for 10% ROI Target - Snapchat (medium)
|Home/Analytics & Experimentation/Snapchat

Design A/B Test for New Recommendation Algorithm Launch

Snapchat logo
Snapchat
Aug 4, 2025, 10:55 AM
mediumData ScientistOnsiteAnalytics & Experimentation
81
0

Design A/B Test for New Recommendation Algorithm Launch

A/B Test Design: New Recommendation Algorithm

Objective

Design a rigorous A/B test to estimate the incremental impact of a new recommendation algorithm on gross merchandise value (GMV).

Tasks

  1. Define the experimental design and randomization strategy.
  2. Specify the primary metric and guardrail metrics, and justify each choice.
  3. Compute the per-variant sample size given:
    • Baseline conversion rate: 3%
    • Expected relative lift: 7%
    • Significance level: α = 0.05 (two-sided)
    • Power: 0.8
  4. Explain how to handle novelty effects and uneven seasonality across groups.
  5. Describe how you would interpret results if the primary GMV metric is flat but secondary engagement metrics improve.

Hints

  • Discuss randomization strategy, CUPED or other variance reduction, sequential testing, and post-test segmentation.

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Snapchat•More Data Scientist•Snapchat Data Scientist•Snapchat Analytics & Experimentation•Data Scientist Analytics & Experimentation

Write your answer

Your first approved answer each day earns 20 XP.

Sign in to write your answer.
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • AI Coding Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.