Handle novelty and residual effects

Q: Handle novelty and residual effects

This question evaluates a data scientist's competency in experiment design and causal inference for online metrics under temporal dynamics, specifically assessing understanding of novelty-decay, residual (carryover) effects, cohort/time-since-exposure analysis, and the implications for power and sample sizing.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Loading...

Experiment design under novelty-decay and residual (carryover) effects

Context

You are testing a new UI that creates a novelty spike that decays over roughly 21 days. The previous release may also leave residual (carryover) effects on users. You need to design an experiment that separately estimates novelty versus steady-state impact, while guarding against contamination from prior releases.

Assume:

User-level randomization is available and can be made sticky for weeks.
You can create a persistent holdback that never receives the new UI during the experiment.
You can ramp traffic in stages and instrument the timestamp of a user’s first exposure to the new UI.
Primary metrics are user-level or user-day outcomes (e.g., sessions, time spent, CTR), and you can analyze them by time-since-first-exposure.

Tasks

(a) Design a ramp plan and a persistent holdback to estimate novelty separately from steady-state.

(b) Specify a cohort analysis by time-since-first-exposure and a model that captures decay (e.g., spline or exponential).

(c) Propose a cool-off window and rehash strategy to address residual effects from the previous release.

(d) Determine experiment duration and sample size under decay, and explain how decay affects power.

Handle novelty and residual effects

Quick Overview

Experiment design under novelty-decay and residual (carryover) effects

Context

Tasks

Solution

Comments (0)