Analyze A/B Test Results to Inform Stakeholder Decisions
Quick Overview
Evaluates end-to-end A/B test analysis from raw log data. Strong answers clean events into user-level exposure data, check assignment and sample-ratio mismatch, compute conversion lift, p-values and confidence intervals, visualize uncertainty, and provide a stakeholder-ready recommendation.
Analyze A/B Test Results to Inform Stakeholder Decisions
Company: Airbnb
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
You receive raw log-level data from an A/B test and must convince stakeholders of the result.
##### Question
Clean the data in Python, compute primary metrics (e.g., conversion, lift, p-value), produce at least one visualization that supports your conclusion, and clearly interpret the outcome for the business.
##### Hints
Use pandas for ETL, seaborn/matplotlib for plots, and a two-sample t-test or proportion z-test for significance.
Overview: Evaluates end-to-end A/B test analysis from raw log data. Strong answers clean events into user-level exposure data, check assignment and sample-ratio mismatch, compute conversion lift, p-values and confidence intervals, visualize uncertainty, and provide a stakeholder-ready recommendation.
Analyze A/B Test Results to Inform Stakeholder Decisions
You receive raw log-level event data from an A/B test on a consumer booking funnel. You must clean the data, compute primary experiment metrics, visualize the outcome, and communicate the result to stakeholders.
Constraints & Assumptions
Assume rows are user events during the experiment window.
Typical columns include
user_id
,
variant
,
event
,
ts
,
revenue
,
bot
, and optional covariates.
Analyze unique exposed users and avoid double-counting repeated events.
Use a two-sided significance test unless the experiment plan states otherwise.
Clarifying Questions to Ask Guidance
What is the primary metric and decision threshold?
What counts as exposure and conversion?
Are users assigned persistently to variants?
Are bot traffic, duplicate events, or cross-device users present?
Part 1 - Clean the Data
How would you clean the raw log-level data in Python?
What This Part Should Cover Guidance
Remove bots and invalid rows, parse timestamps, deduplicate users, verify exposure, enforce variant consistency, and filter events after exposure.
Check sample-ratio mismatch and missing data.
Part 2 - Compute Metrics
How would you compute primary metrics such as conversion, lift, p-value, and confidence interval?
What This Part Should Cover Guidance
User-level conversion rates for treatment and control.
Absolute and relative lift.
Two-proportion z test or regression, confidence intervals, and revenue metrics if relevant.
Part 3 - Visualize Results
What visualization would you produce?
What This Part Should Cover Guidance
Bar chart or point estimate with confidence intervals, funnel chart, distribution of revenue, cumulative metric over time, or segment view.
Clear labeling for stakeholders.
Part 4 - Interpret for Stakeholders
How would you explain the result and recommendation?
What This Part Should Cover Guidance
Statistical significance, practical significance, guardrails, risk, and decision options.
Caveats and follow-up analysis.
What a Strong Answer Covers Guidance
A strong answer converts logs to user-level experiment data, validates assignment and exposure, computes statistically sound metrics, visualizes uncertainty, and makes a business recommendation.
Follow-up Questions Guidance
What if sample-ratio mismatch is detected?
How would you handle revenue outliers?
What if conversion lift is significant but cancellation rate worsens?