Design metrics and experiment

Quick Overview

This question evaluates a candidate's competency in experimental design, success-metric definition, event instrumentation, causal inference for heterogeneous treatment effects, and statistical power and sample-size calculations for subscription-product analytics.

Design metrics and experiment

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Design success metrics and an experiment for the new subscriber-only feature to achieve your chosen goal. Specify: (1) precise primary metric(s) and units (e.g., 28‑day subscriber churn, free→paid conversion within 14 days, ARPU net of discounts); (2) guardrails (e.g., complaint rate, time-to-render, refund rate, non-subscriber engagement); (3) segmentation (new vs existing subscribers, price tiers, geos), plus how you’ll handle heterogeneous treatment effects; (4) event instrumentation and attribution windows; (5) experiment unit, randomization, sample size and power for your chosen MDE, and a staged ramp plan; (6) how you’ll treat non-subscribers who upgrade mid-test (intention-to-treat vs per-protocol), crossovers, and novelty effects; (7) how to detect and cap cannibalization of other revenue; (8) a difference-in-differences or synthetic control fallback if an RCT is infeasible.

Quick Answer: This question evaluates a candidate's competency in experimental design, success-metric definition, event instrumentation, causal inference for heterogeneous treatment effects, and statistical power and sample-size calculations for subscription-product analytics.

|Home/Analytics & Experimentation
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
2
0

Context

You are the data scientist designing success metrics and an experiment for a new subscriber-only feature in a consumer subscription product (e.g., media, productivity, fitness). The feature is visible only to paying subscribers, with the business goal of improving subscriber retention and long‑term revenue. Where necessary, make minimal, explicit assumptions to fully specify the design.

Task

Design success metrics and an experiment to achieve your chosen goal. Provide:

  1. Primary metric(s) with precise definitions and units (e.g., 28‑day subscriber churn; free→paid conversion within 14 days; ARPU net of discounts).
  2. Guardrail metrics (e.g., complaint rate, time-to-render, refund rate, non-subscriber engagement) and thresholds.
  3. Segmentation plan (e.g., new vs existing subscribers, price tiers, geos) and how you will detect and handle heterogeneous treatment effects (HTE).
  4. Event instrumentation plan and attribution windows.
  5. Experiment unit, randomization scheme, sample size and power for your chosen minimum detectable effect (MDE), and a staged ramp plan.
  6. Policy for non‑subscribers who upgrade mid‑test (intention‑to‑treat vs per‑protocol), crossovers, and novelty effects.
  7. Approach to detect and cap cannibalization of other revenue streams.
  8. A difference‑in‑differences or synthetic control fallback if an RCT is infeasible.
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Data Scientist•Data Scientist Analytics & Experimentation