Define Success Metrics and Experiment Plan for Product Development
Quick Overview
Evaluates product-planning metrics and experiment design for a two-sided marketplace booking funnel. Strong answers define a primary booking metric, diagnostic and guardrail metrics, randomization unit, sample size, marketplace spillovers, and launch criteria before development.
Define Success Metrics and Experiment Plan for Product Development
Company: Airbnb
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
During a product-planning session you must define success criteria before development begins.
##### Question
Design the key metrics, guardrails, and experiment plan you would propose, including unit of randomization, sample-size calculation, and runtime monitoring.
##### Hints
Cover north-star vs. diagnostic metrics, power analysis, and data-quality checks.
Overview: Evaluates product-planning metrics and experiment design for a two-sided marketplace booking funnel. Strong answers define a primary booking metric, diagnostic and guardrail metrics, randomization unit, sample size, marketplace spillovers, and launch criteria before development.
Define Success Metrics and Experiment Plan for Product Development
You are in a product-planning session for a new change to the core booking funnel in a two-sided marketplace app where guests book stays from hosts. The feature affects guest search or booking behavior and may indirectly affect host outcomes.
Constraints & Assumptions
Define success criteria before development begins.
Include a primary metric, diagnostic metrics, guardrails, randomization, sample size, and launch criteria.
Account for marketplace effects on guests and hosts.
Specify measurement windows and units clearly.
Clarifying Questions to Ask Guidance
What part of the booking funnel changes: search, listing page, checkout, messaging, or pricing?
Is the main goal conversion, booking value, retention, host quality, or trust?
Does the feature affect host exposure or supply-side outcomes?
What traffic volume and baseline conversion rate are available?
Part 1 - Success Metrics
Define a single primary north-star metric and supporting diagnostic metrics.
What This Part Should Cover Guidance
Primary metric such as bookings per exposed user or booking conversion rate.
Supporting metrics for search, listing views, checkout starts, completed bookings, GBV, cancellations, and repeat usage.
Unit and window for each metric.
Part 2 - Guardrails
List key guardrail metrics and suggested non-degradation thresholds.
What This Part Should Cover Guidance
Host outcomes, cancellations, support contacts, refund rate, guest complaints, latency, fairness, search quality, and long-term retention.
Thresholds and escalation rules.
Part 3 - Experiment Plan
Design the experiment plan, including unit of randomization, sample size, duration, and analysis.
What This Part Should Cover Guidance
User or session randomization, exposure definition, MDE, power, CUPED or variance reduction, segment analysis, and monitoring.
Marketplace spillover considerations.
Part 4 - Launch Decision
How would you present launch criteria to stakeholders?
What This Part Should Cover Guidance
Predefined decision rules, practical significance, uncertainty, guardrails, and staged rollout.
What to do if metrics are mixed.
What a Strong Answer Covers Guidance
A strong answer aligns metrics with product goals, protects both sides of the marketplace, and lays out an experiment plan stakeholders can evaluate before engineering work begins.
Follow-up Questions Guidance
What if bookings increase but host cancellations also increase?
How would you handle seasonality in booking behavior?