Design metrics and experiment
Overview
This question evaluates a candidate's competency in experimental design, success-metric definition, event instrumentation, causal inference for heterogeneous treatment effects, and statistical power and sample-size calculations for subscription-product analytics.
Other
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Context
You are the data scientist designing success metrics and an experiment for a new subscriber-only feature in a consumer subscription product (e.g., media, productivity, fitness). The feature is visible only to paying subscribers, with the business goal of improving subscriber retention and long‑term revenue. Where necessary, make minimal, explicit assumptions to fully specify the design.
Task
Design success metrics and an experiment to achieve your chosen goal. Provide:
- Primary metric(s) with precise definitions and units (e.g., 28‑day subscriber churn; free→paid conversion within 14 days; ARPU net of discounts).
- Guardrail metrics (e.g., complaint rate, time-to-render, refund rate, non-subscriber engagement) and thresholds.
- Segmentation plan (e.g., new vs existing subscribers, price tiers, geos) and how you will detect and handle heterogeneous treatment effects (HTE).
- Event instrumentation plan and attribution windows.
- Experiment unit, randomization scheme, sample size and power for your chosen minimum detectable effect (MDE), and a staged ramp plan.
- Policy for non‑subscribers who upgrade mid‑test (intention‑to‑treat vs per‑protocol), crossovers, and novelty effects.
- Approach to detect and cap cannibalization of other revenue streams.
- A difference‑in‑differences or synthetic control fallback if an RCT is infeasible.
Solution
Login required
Show