How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Thumbtack.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Thumbtack during technical interviews.

Design and evaluate an A/B test for launch | Thumbtack Interview Question

Quick Overview

This question evaluates competency in experimental design and causal inference for two-sided marketplaces, covering metric selection, interference control, randomization strategy, power analysis, monitoring, pre-registration, and heterogeneity analysis.

A/B Test Design: New Matching Model for a Two‑Sided Marketplace

Context

You are testing a new matching/ranking model that determines which providers are surfaced/notified for each customer request in a two‑sided services marketplace. The model may change who gets contacted, how quickly customers receive responses, and ultimately whether a booking occurs. Your design must measure impact on both customers (demand) and providers (supply) while handling interference common to marketplace experiments.

Task

Design an A/B test plan and analysis that covers:

Metrics
- Specify a single primary metric and guardrail metrics for both sides of the marketplace. Examples: booking conversion, provider response rate, time‑to‑first‑response, cancellation rate.
Randomization and Interference Control
- Choose the unit of randomization and describe how you will prevent interference/spillovers. Examples: geo or time bucketing, cluster randomization, provider saturation caps.
Power Analysis and Duration
- Provide baseline rates, minimum detectable effect (MDE), variance estimates, and how you determine the test duration.
Estimation and Monitoring
- Plans for CUPED or covariate adjustment; sample‑ratio‑mismatch (SRM) checks; and sequential monitoring boundaries.
Pre‑Registration and Ops
- A pre‑registration outline with stop/go criteria and a rollback plan.
Heterogeneity Without p‑Hacking
- How to interpret heterogeneous lift by region and job_category while controlling false discoveries.

Quick Overview

Context

Task

Design an A/B test plan and analysis that covers:

Metrics

Specify a single primary metric and guardrail metrics for both sides of the marketplace. Examples: booking conversion, provider response rate, time‑to‑first‑response, cancellation rate.

Randomization and Interference Control

Choose the unit of randomization and describe how you will prevent interference/spillovers. Examples: geo or time bucketing, cluster randomization, provider saturation caps.

Power Analysis and Duration

Provide baseline rates, minimum detectable effect (MDE), variance estimates, and how you determine the test duration.

Estimation and Monitoring

Plans for CUPED or covariate adjustment; sample‑ratio‑mismatch (SRM) checks; and sequential monitoring boundaries.

Pre‑Registration and Ops

A pre‑registration outline with stop/go criteria and a rollback plan.

Heterogeneity Without p‑Hacking

How to interpret heterogeneous lift by region and job_category while controlling false discoveries.

Design and evaluate an A/B test for launch

Quick Overview

A/B Test Design: New Matching Model for a Two‑Sided Marketplace

Context

Task

Solution

Comments (0)

Design and evaluate an A/B test for launch

Quick Overview

A/B Test Design: New Matching Model for a Two‑Sided Marketplace

Context

Task

Solution

Comments (0)