How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Other.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Other during technical interviews.

Design metrics and experiment | Other Interview Question

Quick Overview

This question evaluates a candidate's competency in experimental design, success-metric definition, event instrumentation, causal inference for heterogeneous treatment effects, and statistical power and sample-size calculations for subscription-product analytics.

Context

You are the data scientist designing success metrics and an experiment for a new subscriber-only feature in a consumer subscription product (e.g., media, productivity, fitness). The feature is visible only to paying subscribers, with the business goal of improving subscriber retention and long‑term revenue. Where necessary, make minimal, explicit assumptions to fully specify the design.

Task

Design success metrics and an experiment to achieve your chosen goal. Provide:

Primary metric(s) with precise definitions and units (e.g., 28‑day subscriber churn; free→paid conversion within 14 days; ARPU net of discounts).
Guardrail metrics (e.g., complaint rate, time-to-render, refund rate, non-subscriber engagement) and thresholds.
Segmentation plan (e.g., new vs existing subscribers, price tiers, geos) and how you will detect and handle heterogeneous treatment effects (HTE).
Event instrumentation plan and attribution windows.
Experiment unit, randomization scheme, sample size and power for your chosen minimum detectable effect (MDE), and a staged ramp plan.
Policy for non‑subscribers who upgrade mid‑test (intention‑to‑treat vs per‑protocol), crossovers, and novelty effects.
Approach to detect and cap cannibalization of other revenue streams.
A difference‑in‑differences or synthetic control fallback if an RCT is infeasible.

Quick Overview

Context

Task

Design success metrics and an experiment to achieve your chosen goal. Provide:

Primary metric(s) with precise definitions and units (e.g., 28‑day subscriber churn; free→paid conversion within 14 days; ARPU net of discounts).

Guardrail metrics (e.g., complaint rate, time-to-render, refund rate, non-subscriber engagement) and thresholds.

Segmentation plan (e.g., new vs existing subscribers, price tiers, geos) and how you will detect and handle heterogeneous treatment effects (HTE).

Event instrumentation plan and attribution windows.

Experiment unit, randomization scheme, sample size and power for your chosen minimum detectable effect (MDE), and a staged ramp plan.

Policy for non‑subscribers who upgrade mid‑test (intention‑to‑treat vs per‑protocol), crossovers, and novelty effects.

Approach to detect and cap cannibalization of other revenue streams.

A difference‑in‑differences or synthetic control fallback if an RCT is infeasible.

Design metrics and experiment

Quick Overview

Context

Task

Solution

Comments (0)

Design metrics and experiment

Quick Overview

Context

Task

Solution

Comments (0)