Design and evaluate an A/B test for launch

Quick Overview

This question evaluates competency in experimental design and causal inference for two-sided marketplaces, covering metric selection, interference control, randomization strategy, power analysis, monitoring, pre-registration, and heterogeneity analysis.

Design and evaluate an A/B test for launch

Company: Thumbtack

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

A new matching model is ready for launch. Design an A/B test to determine success. Specify: (1) primary metric and guardrails for both sides of the marketplace (e.g., booking conversion, provider response rate, time-to-first-response, cancellation rate); (2) unit of randomization and how you will prevent interference/spillovers (e.g., geo or time bucketing, provider saturation caps); (3) power analysis with baseline rates, MDE, variance estimates, and test duration; (4) plans for CUPED or covariate adjustment, sample-ratio-mismatch checks, and sequential monitoring boundaries; (5) a pre-registration doc with stop/go criteria and a rollback plan; and (6) how to interpret heterogeneous lift by region and job_category without p-hacking.

Quick Answer: This question evaluates competency in experimental design and causal inference for two-sided marketplaces, covering metric selection, interference control, randomization strategy, power analysis, monitoring, pre-registration, and heterogeneity analysis.

|Home/Analytics & Experimentation/Thumbtack
Thumbtack logo
Thumbtack
Oct 13, 2025
hardData ScientistOnsiteAnalytics & Experimentation
7
0

A/B Test Design: New Matching Model for a Two‑Sided Marketplace

Context

You are testing a new matching/ranking model that determines which providers are surfaced/notified for each customer request in a two‑sided services marketplace. The model may change who gets contacted, how quickly customers receive responses, and ultimately whether a booking occurs. Your design must measure impact on both customers (demand) and providers (supply) while handling interference common to marketplace experiments.

Task

Design an A/B test plan and analysis that covers:

  1. Metrics
    • Specify a single primary metric and guardrail metrics for both sides of the marketplace. Examples: booking conversion, provider response rate, time‑to‑first‑response, cancellation rate.
  2. Randomization and Interference Control
    • Choose the unit of randomization and describe how you will prevent interference/spillovers. Examples: geo or time bucketing, cluster randomization, provider saturation caps.
  3. Power Analysis and Duration
    • Provide baseline rates, minimum detectable effect (MDE), variance estimates, and how you determine the test duration.
  4. Estimation and Monitoring
    • Plans for CUPED or covariate adjustment; sample‑ratio‑mismatch (SRM) checks; and sequential monitoring boundaries.
  5. Pre‑Registration and Ops
    • A pre‑registration outline with stop/go criteria and a rollback plan.
  6. Heterogeneity Without p‑Hacking
    • How to interpret heterogeneous lift by region and job_category while controlling false discoveries.
Loading comments...