Design Experiment to Measure Shopping Feature Impact
Overview
This question evaluates experimental design and causal inference skills for product analytics, focusing on metric selection, handling interference and non-compliance, and analysis of heavy-tailed and delayed conversion outcomes within the Analytics & Experimentation domain.
Meta
Jul 12, 2025, 6:59 PM
Data Scientist
Onsite
Analytics & Experimentation
5
0
Experiment Design: Measuring Instagram Shopping's Impact
Scenario
Instagram is launching an in-app Shopping feature (e.g., product tags, shop surfaces, in-app checkout). Leadership wants to quantify its incremental impact on user engagement and revenue.
Task
Design an experiment to measure the impact of the Shopping feature on engagement and revenue.
Address:
- What experimental design(s) would you use and why (e.g., A/B vs. geo hold-out; rollout strategy)?
- Which metrics (north-star and guardrails) would you monitor, with success criteria?
- How would you mitigate and measure selection bias and network effects (spillovers across the social graph)?
Considerations
- Users can influence each other via follows/likes/shares (potential interference).
- Some users may be eligible but never use the feature (non-compliance).
- Rare, heavy-tailed outcomes (purchases/GMV) and delayed conversions.
- Hints: A/B vs. hold-out, north-star metrics, guard-rails, diff-in-diff.
Solution
Login required
Show