PracHub
QuestionsPremiumLearningGuidesInterview PrepNEWCoaches
|Home/Analytics & Experimentation/Thumbtack

Define success metrics for Instant Book

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a data scientist's competency in defining north-star and diagnostic metrics, designing instrumentation and attribution, planning staged rollouts, and identifying operational risks for a two-sided marketplace feature (metrics definition, experimentation design, measurement, governance, and risk mitigation).

  • hard
  • Thumbtack
  • Analytics & Experimentation
  • Data Scientist

Define success metrics for Instant Book

Company: Thumbtack

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Thumbtack is considering an "Instant Book" feature that lets customers immediately book a pro at a pre-agreed price/time without waiting for quotes. Answer: 1) North-star and objectives: Propose a single north-star metric for marketplace health and two secondary objectives capturing both demand (customer) and supply (pro) perspectives. Define exact formulas and units. 2) Diagnostic metrics: List at least six diagnostics spanning liquidity (time-to-first-commit), fulfillment rate, cancellations/reschedules, refund rate, average booking value, pro utilization, and fairness (e.g., Gini of bookings across pros within category-region-week). Include target directions and acceptable short-term regressions. 3) Measurement plan: Describe instrumentation/events, deduping rules, and attribution windows. Clarify how you’ll separate organic bookings from Instant Book, handle multi-device sessions, and prevent double-counting when a request also receives quotes. 4) Rollout strategy: Recommend a staged launch (e.g., supply-gated by category-region with minimum active pros and SLOs), with a holdback for long-term effects. Explain quota controls to avoid starving non-Instant-Book requests. 5) Success criteria and decision tree: Specify quantitative launch gates after 2 and 6 weeks, including guardrails that must not worsen by more than X%. 6) Risks and mitigations: Identify at least five risks (e.g., adverse selection, schedule conflicts, price anchoring, unfair exposure, fraud) and propose concrete mitigations and monitoring.

Quick Answer: This question evaluates a data scientist's competency in defining north-star and diagnostic metrics, designing instrumentation and attribution, planning staged rollouts, and identifying operational risks for a two-sided marketplace feature (metrics definition, experimentation design, measurement, governance, and risk mitigation).

Related Interview Questions

  • Explain power drivers and resolve unexpected A/B results - Thumbtack (medium)
  • Design a robust pro-ranking A/B test - Thumbtack (hard)
  • Design and evaluate an A/B test for launch - Thumbtack (hard)
Thumbtack logo
Thumbtack
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
1
0

Instant Book: Metrics, Measurement, Rollout, and Risk Plan

Context

You are evaluating an "Instant Book" feature that allows customers to immediately book a pro for a pre-set price and time, without waiting for quotes. Design a comprehensive plan to measure, roll out, and govern the feature across a two-sided marketplace.

Tasks

  1. North-star and objectives
    • Propose one north-star metric for marketplace health and two secondary objectives (one demand-side, one supply-side).
    • Provide exact formulas and units for each metric.
  2. Diagnostic metrics
    • List at least six diagnostics covering: liquidity (time to first commit), fulfillment rate, cancellations/reschedules, refund rate, average booking value, pro utilization, and fairness (e.g., Gini of bookings across pros within category–region–week).
    • For each, specify target direction and what short-term regression (if any) is acceptable during ramp.
  3. Measurement plan
    • Define instrumentation/events, deduping rules, and attribution windows.
    • Explain how you will separate organic bookings from Instant Book, handle multi-device sessions, and prevent double-counting when a request also receives quotes.
  4. Rollout strategy
    • Recommend a staged launch (e.g., supply-gated by category–region with minimum active pros and SLOs), plus a holdback for long-term effects.
    • Describe quota controls to avoid starving non–Instant Book requests.
  5. Success criteria and decision tree
    • Specify quantitative launch gates after 2 and 6 weeks, including guardrails that must not worsen by more than X%.
  6. Risks and mitigations
    • Identify at least five risks (e.g., adverse selection, schedule conflicts, price anchoring, unfair exposure, fraud) and propose concrete mitigations and monitoring.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Thumbtack•More Data Scientist•Thumbtack Data Scientist•Thumbtack Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.