Design experiment with network and novelty effects

Q: Design experiment with network and novelty effects

This question evaluates a data scientist's competence in experimental design and causal inference under network interference and novelty effects, covering skills such as choice of randomization unit, estimand specification, exposure mapping, and bias-aware inference for spillovers and temporal heterogeneity.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Loading...

Experiment Design: New Calling Feature with Network Interference and Novelty Effects

Context: You are launching a new calling feature on a large social platform where calls occur along a known social graph G = (V, E). A user's outcome (e.g., calls placed, calls received, call minutes, call acceptance rate) may be affected by both their own assignment and their neighbors' assignments (network interference). The feature is also expected to have strong novelty effects (initial excitement/learning curves).

Task: Design an experiment and analysis plan that measures product impact while addressing both interference and novelty. Specifically describe:

Choice of randomization unit
- Choose among: user, pair/edge, or cluster/ego-network. Justify the choice, including trade-offs.
Handling treated callers contacting control users
- Propose either a two-stage randomization (cluster-level coverage + within-cluster user assignment) or cluster/ego-network holdouts. Specify how calls between treated–control pairs are handled.
Interference and estimands
- Explain what happens to control-group outcomes under interference.
- Define and specify estimands for the total effect vs. direct effect (and indirect/spillover, if used). Include notation.
Detecting and mitigating novelty effects
- Describe ramp schedules and how you would detect novelty over time.
- Describe analytical adjustments (e.g., CUPED, event-study/temporal heterogeneity modeling) to isolate steady-state effects.
Analysis plan with spillovers into control
- If some control users receive spillovers, describe how you will estimate effects and uncertainty without bias (e.g., exposure mapping, IPW/HT estimators, IV, randomization inference).

Include simple diagrams/notation as helpful and be explicit about assumptions and estimands.

Design experiment with network and novelty effects

Quick Overview

Experiment Design: New Calling Feature with Network Interference and Novelty Effects

Solution

Comments (0)