Analyze A/B test with revenue–cost tradeoffs
Overview
This question evaluates a data scientist's skills in experimental design and statistical inference for A/B testing, revenue-impact modeling with skewed and zero-inflated metrics, segmentation for heterogeneous treatment effects, and constrained optimization to balance contribution and operational risk.
A/B Test: Same‑Day Delivery Checkout Change
You are evaluating a checkout UI change that promotes same‑day delivery. The experiment is a standard two‑arm A/B test with ≈50,000 users per arm. Primary metrics:
- Conversion rate (CR)
- Average order value (AOV)
- Take rate (platform % of GMV)
- Delivery cost per order
Data characteristics and wrinkles:
- AOV and delivery cost are right‑skewed.
- Unequal variances between arms (variance ratio ≈ 2).
- 15% of orders have zero delivery cost (promotions/free delivery).
Tasks
a) For each metric, choose and justify appropriate hypothesis test(s) (e.g., Welch’s t‑test vs classic t‑test vs nonparametric/bootstrapping vs z‑test). State the key distributional assumptions and how you would validate them.
b) Quantify net revenue impact. Define contribution per order = GMV × take_rate − delivery_cost. Allow delivery_cost to increase with order volume (operational scaling) and take_rate to saturate at high discounts. Show how you’d model cost scaling (e.g., piecewise/queueing‑informed function) and diminishing take_rate, then propagate uncertainty to a final decision.
c) Propose a segmentation plan (e.g., market, daypart, basket size, new vs repeat) to detect heterogeneous treatment effects. Explain multiplicity control and how to avoid overfitting while surfacing actionable segments.
d) Given operational risk of manually increasing shopper supply, propose an optimization objective and constraints that balance short‑term contribution lift with service‑level metrics (SLAs, cancellation risk).
Solution
Login required