Drive product decisions with causal product sense

Q: Drive product decisions with causal product sense

This question evaluates a data scientist's competency in causal inference and product experimentation design, specifically handling interference/spillovers, defining success metrics and guardrails, applying variance reduction techniques, computing power and MDE for cluster RCTs, planning sequential monitoring, and framing rollout decisions.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at TikTok.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at TikTok during technical interviews.

Question

Experimenting on a New Paywall with Likely Spillovers

Context

You are designing an experiment to evaluate a new paywall on a social/content app where users frequently share links with each other and externally. Because of sharing and algorithmic amplification, user outcomes may be affected by other users' treatment assignments (interference/spillovers), violating standard A/B test assumptions.

Task

Define success metrics and choose an experimentation strategy that handles interference. Address all parts concisely and concretely.

Requirements

(a) Interference and mitigation

Identify where interference is most probable (e.g., sharing links, followers/creators, households, algorithm training).
Propose mitigation options (e.g., cluster randomization by social network or geo, randomized saturation, exposure models).

(b) Metrics

Define a primary metric, a north-star metric, and guardrails (e.g., retention, complaints), including measurement windows and units of analysis.

(c) Variance reduction

Describe how you would use CUPED or pre-exposure covariates, and at what level (user/cluster), without inducing bias.

(d) Power and MDE for a cluster RCT

With ICC = 0.05 and 30 clusters total, state your assumptions and show the MDE calculation for a binary primary outcome. Show formulas and at least one numeric example.

(e) Sequential monitoring

Outline an alpha-spending approach, risks of peeking, and how many looks you would plan.

(f) Rollout decision

Propose a decision framework balancing short-term revenue lift against retention risk and potential long-term effects (network health, creator reactions), including thresholds or expected-value logic.

Drive product decisions with causal product sense

Quick Overview

Drive product decisions with causal product sense

Experimenting on a New Paywall with Likely Spillovers

Context

Task

Requirements

Write your answer