A/B Test Design: New Matching Model for a Two‑Sided Marketplace

Context

You are testing a new matching/ranking model that determines which providers are surfaced/notified for each customer request in a two‑sided services marketplace. The model may change who gets contacted, how quickly customers receive responses, and ultimately whether a booking occurs. Your design must measure impact on both customers (demand) and providers (supply) while handling interference common to marketplace experiments.

Task

Design an A/B test plan and analysis that covers:

1. Metrics
   • Specify a single primary metric and guardrail metrics for both sides of the marketplace. Examples: booking conversion, provider response rate, time‑to‑first‑response, cancellation rate.
2. Randomization and Interference Control
   • Choose the unit of randomization and describe how you will prevent interference/spillovers. Examples: geo or time bucketing, cluster randomization, provider saturation caps.
3. Power Analysis and Duration
   • Provide baseline rates, minimum detectable effect (MDE), variance estimates, and how you determine the test duration.
4. Estimation and Monitoring
   • Plans for CUPED or covariate adjustment; sample‑ratio‑mismatch (SRM) checks; and sequential monitoring boundaries.
5. Pre‑Registration and Ops
   • A pre‑registration outline with stop/go criteria and a rollback plan.
6. Heterogeneity Without p‑Hacking
   • How to interpret heterogeneous lift by region and job_category while controlling false discoveries.