##### Scenario During a product-planning session you must define success criteria before development begins. ##### Question Design the key metrics, guardrails, and experiment plan you would propose, including unit of randomization, sample-size calculation, and runtime monitoring. ##### Hints Cover north-star vs. diagnostic metrics, power analysis, and data-quality checks.

Evaluates product-planning metrics and experiment design for a two-sided marketplace booking funnel. Strong answers define a primary booking metric, diagnostic and guardrail metrics, randomization unit, sample size, marketplace spillovers, and launch criteria before development.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Airbnb.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Airbnb during technical interviews.

Define Success Metrics and Experiment Plan for Product Development

You are in a product-planning session for a new change to the core booking funnel in a two-sided marketplace app where guests book stays from hosts. The feature affects guest search or booking behavior and may indirectly affect host outcomes.

Constraints & Assumptions

Define success criteria before development begins.
Include a primary metric, diagnostic metrics, guardrails, randomization, sample size, and launch criteria.
Account for marketplace effects on guests and hosts.
Specify measurement windows and units clearly.

Clarifying Questions to Ask

What part of the booking funnel changes: search, listing page, checkout, messaging, or pricing?
Is the main goal conversion, booking value, retention, host quality, or trust?
Does the feature affect host exposure or supply-side outcomes?
What traffic volume and baseline conversion rate are available?

Part 1 - Success Metrics

Define a single primary north-star metric and supporting diagnostic metrics.

What This Part Should Cover

Primary metric such as bookings per exposed user or booking conversion rate.
Supporting metrics for search, listing views, checkout starts, completed bookings, GBV, cancellations, and repeat usage.
Unit and window for each metric.

Part 2 - Guardrails

List key guardrail metrics and suggested non-degradation thresholds.

What This Part Should Cover

Host outcomes, cancellations, support contacts, refund rate, guest complaints, latency, fairness, search quality, and long-term retention.
Thresholds and escalation rules.

Part 3 - Experiment Plan

Design the experiment plan, including unit of randomization, sample size, duration, and analysis.

What This Part Should Cover

User or session randomization, exposure definition, MDE, power, CUPED or variance reduction, segment analysis, and monitoring.
Marketplace spillover considerations.

Part 4 - Launch Decision

How would you present launch criteria to stakeholders?

What This Part Should Cover

Predefined decision rules, practical significance, uncertainty, guardrails, and staged rollout.
What to do if metrics are mixed.

What a Strong Answer Covers

A strong answer aligns metrics with product goals, protects both sides of the marketplace, and lays out an experiment plan stakeholders can evaluate before engineering work begins.

Follow-up Questions

What if bookings increase but host cancellations also increase?
How would you handle seasonality in booking behavior?
When would you choose geo-level randomization?