Design and Analyze Airbnb Locker Experiment
Company: Airbnb
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Airbnb is considering launching a locker feature that lets guests store luggage before their scheduled check-in time. Design an experiment to decide whether Airbnb should launch this feature.
Address the following:
1. Should the experiment be randomized on the guest side, reservation side, host/listing side, or market side? Justify your choice.
2. Define the eligible population, treatment, and control experience.
3. Choose a primary metric, secondary metrics, and guardrail metrics. Discuss tradeoffs among guest experience, host experience, operational cost, safety, and marketplace revenue.
4. Explain how you would calculate required sample size and decide experiment duration.
5. Describe the analysis plan for estimating the treatment effect and computing a p-value.
6. Suppose you are given a reservation-level experiment table with the following columns: unit_id string, guest_id string, host_id string, listing_id string, market string, assignment_variant string, assigned_at timestamp in UTC, checkin_at timestamp in UTC, checkout_at timestamp in UTC, booked integer, used_locker integer, support_contacted integer, post_stay_rating float, gross_booking_value float, pre_assignment_trips integer, pre_assignment_support_rate float, and is_aa_test boolean. Explain how you would run an AB test and an AA test to check for bias or instrumentation issues.
7. Discuss risks such as selection bias, interference between treatment and control, locker capacity constraints, host-side spillovers, and heterogeneous effects across markets.
Quick Answer: This question evaluates experimental design, causal inference, A/B testing, metric selection (primary, secondary, guardrail), power/sample-size calculation, instrumentation validation, and analysis of bias, interference, and heterogeneous effects for a Data Scientist in the Analytics & Experimentation domain.