##### Scenario You receive raw log-level data from an A/B test and must convince stakeholders of the result. ##### Question Clean the data in Python, compute primary metrics (e.g., conversion, lift, p-value), produce at least one visualization that supports your conclusion, and clearly interpret the outcome for the business. ##### Hints Use pandas for ETL, seaborn/matplotlib for plots, and a two-sample t-test or proportion z-test for significance.

Evaluates end-to-end A/B test analysis from raw log data. Strong answers clean events into user-level exposure data, check assignment and sample-ratio mismatch, compute conversion lift, p-values and confidence intervals, visualize uncertainty, and provide a stakeholder-ready recommendation.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Airbnb.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Airbnb during technical interviews.

Analyze A/B Test Results to Inform Stakeholder Decisions

You receive raw log-level event data from an A/B test on a consumer booking funnel. You must clean the data, compute primary experiment metrics, visualize the outcome, and communicate the result to stakeholders.

Constraints & Assumptions

Assume rows are user events during the experiment window.
Typical columns include user_id , variant , event , ts , revenue , bot , and optional covariates.
Analyze unique exposed users and avoid double-counting repeated events.
Use a two-sided significance test unless the experiment plan states otherwise.

Clarifying Questions to Ask

What is the primary metric and decision threshold?
What counts as exposure and conversion?
Are users assigned persistently to variants?
Are bot traffic, duplicate events, or cross-device users present?

Part 1 - Clean the Data

How would you clean the raw log-level data in Python?

What This Part Should Cover

Remove bots and invalid rows, parse timestamps, deduplicate users, verify exposure, enforce variant consistency, and filter events after exposure.
Check sample-ratio mismatch and missing data.

Part 2 - Compute Metrics

How would you compute primary metrics such as conversion, lift, p-value, and confidence interval?

What This Part Should Cover

User-level conversion rates for treatment and control.
Absolute and relative lift.
Two-proportion z test or regression, confidence intervals, and revenue metrics if relevant.

Part 3 - Visualize Results

What visualization would you produce?

What This Part Should Cover

Bar chart or point estimate with confidence intervals, funnel chart, distribution of revenue, cumulative metric over time, or segment view.
Clear labeling for stakeholders.

Part 4 - Interpret for Stakeholders

How would you explain the result and recommendation?

What This Part Should Cover

Statistical significance, practical significance, guardrails, risk, and decision options.
Caveats and follow-up analysis.

What a Strong Answer Covers

A strong answer converts logs to user-level experiment data, validates assignment and exposure, computes statistically sound metrics, visualizes uncertainty, and makes a business recommendation.

Follow-up Questions

What if sample-ratio mismatch is detected?
How would you handle revenue outliers?
What if conversion lift is significant but cancellation rate worsens?