Define success metrics for Instant Book
Company: Thumbtack
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
Thumbtack is considering an "Instant Book" feature that lets customers immediately book a pro at a pre-agreed price/time without waiting for quotes.
Answer:
1) North-star and objectives: Propose a single north-star metric for marketplace health and two secondary objectives capturing both demand (customer) and supply (pro) perspectives. Define exact formulas and units.
2) Diagnostic metrics: List at least six diagnostics spanning liquidity (time-to-first-commit), fulfillment rate, cancellations/reschedules, refund rate, average booking value, pro utilization, and fairness (e.g., Gini of bookings across pros within category-region-week). Include target directions and acceptable short-term regressions.
3) Measurement plan: Describe instrumentation/events, deduping rules, and attribution windows. Clarify how you’ll separate organic bookings from Instant Book, handle multi-device sessions, and prevent double-counting when a request also receives quotes.
4) Rollout strategy: Recommend a staged launch (e.g., supply-gated by category-region with minimum active pros and SLOs), with a holdback for long-term effects. Explain quota controls to avoid starving non-Instant-Book requests.
5) Success criteria and decision tree: Specify quantitative launch gates after 2 and 6 weeks, including guardrails that must not worsen by more than X%.
6) Risks and mitigations: Identify at least five risks (e.g., adverse selection, schedule conflicts, price anchoring, unfair exposure, fraud) and propose concrete mitigations and monitoring.
Quick Answer: This question evaluates a data scientist's competency in defining north-star and diagnostic metrics, designing instrumentation and attribution, planning staged rollouts, and identifying operational risks for a two-sided marketplace feature (metrics definition, experimentation design, measurement, governance, and risk mitigation).