Design Experiments Under Network Interference
Company: Playstation
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
You are a Senior Data Scientist for a gaming platform such as PlayStation.
Answer the following experimentation questions.
1. **A/A test validation:** The experimentation platform ran an A/A test to validate randomization and logging. The intended split was 50% control and 50% treatment, but the observed assignment ratio was 51% control and 49% treatment.
- Is this a problem?
- How would you test whether the imbalance is statistically meaningful?
- What root causes would you investigate before trusting future A/B test results?
2. **Experiment design with social network effects:** The product team wants to launch a new feature that lets friends send virtual gifts to each other. The hypothesis is that gifting will increase user engagement.
- Define the primary metric, guardrail metrics, and success criteria.
- How would you design an experiment to estimate the feature's causal impact?
- What randomization unit would you choose: user-level, friend-cluster-level, region-level, or something else?
- How would your design change if users' friendship graph is highly connected, such that almost every user is connected directly or indirectly to almost every other user?
- What biases or validity threats would you worry about?
Quick Answer: This question evaluates skills in experimental design and causal inference under network interference, including A/A test validation, randomization diagnostics, metric definition, and bias identification within the Analytics & Experimentation domain for data scientists.