Design Experiment to Measure Shopping Feature Impact

Q: Design Experiment to Measure Shopping Feature Impact

This question evaluates experimental design and causal inference skills for product analytics, focusing on metric selection, handling interference and non-compliance, and analysis of heavy-tailed and delayed conversion outcomes within the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Experiment Design: Measuring Instagram Shopping's Impact

Scenario

Instagram is launching an in-app Shopping feature (e.g., product tags, shop surfaces, in-app checkout). Leadership wants to quantify its incremental impact on user engagement and revenue.

Task

Design an experiment to measure the impact of the Shopping feature on engagement and revenue.

Address:

What experimental design(s) would you use and why (e.g., A/B vs. geo hold-out; rollout strategy)?
Which metrics (north-star and guardrails) would you monitor, with success criteria?
How would you mitigate and measure selection bias and network effects (spillovers across the social graph)?

Considerations

Users can influence each other via follows/likes/shares (potential interference).
Some users may be eligible but never use the feature (non-compliance).
Rare, heavy-tailed outcomes (purchases/GMV) and delayed conversions.
Hints: A/B vs. hold-out, north-star metrics, guard-rails, diff-in-diff.

Design Experiment to Measure Shopping Feature Impact

Experiment Design: Measuring Instagram Shopping's Impact

Scenario

Task

Considerations

Solution

Comments (0)