##### Scenario Instagram is launching an in-app Shopping feature. Leadership wants to understand the business impact after release. ##### Question How would you design an experiment to measure the impact of the Shopping feature on user engagement and revenue? Which metrics and success criteria would you monitor and why? How would you deal with potential selection bias or network effects? ##### Hints Think A/B vs. hold-out, north-star metrics, guard-rails, diff-in-diff.

Meta data scientist experimentation prompt on measuring Instagram Shopping impact with A/B tests, geo or cluster holdouts, engagement and revenue metrics, guardrails, selection bias, network effects, and rollout decisions.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design Experiment to Measure Shopping Feature Impact

Experiment Design: Measure Instagram Shopping Impact

Instagram is launching an in-app Shopping feature, such as product tags, shop surfaces, or in-app checkout. Leadership wants to quantify the feature's incremental impact on user engagement and revenue.

Constraints & Assumptions

Measure causal impact, not just post-launch correlation.
Include engagement, revenue, and app-health guardrails.
Address selection bias, non-compliance, delayed conversions, heavy-tailed purchase outcomes, and network effects.
Explain when user-level A/B, cluster randomization, geo holdout, or difference-in-differences would be appropriate.

Clarifying Questions to Ask Guidance

Which Shopping surface is launching, and who is eligible?
Is the goal GMV, revenue, buyer engagement, creator/merchant value, or overall app engagement?
Can we randomize users, clusters, geographies, or rollout timing?
What spillovers are likely through follows, shares, creators, or merchants?

What a Strong Answer Covers Guidance

Clear treatment, control, eligibility, exposure logging, and intent-to-treat analysis.
Experimental options: user-level A/B for clean individual effects, cluster randomization for social interference, geo holdouts for ecosystem effects, and staged ramps for safety.
Primary metrics such as incremental buyers, purchases, GMV, revenue, product-detail engagement, or checkout conversion.
Guardrails including retention, session quality, ads revenue, latency, crashes, support, feed engagement, hide/report rates, and merchant or creator fairness.
Methods for selection bias and non-compliance: random assignment, ITT/TOT framing, covariate balance checks, CUPED, and propensity or diff-in-diff only when randomization is limited.
Power, duration, minimum detectable effect, delayed conversion windows, and heavy-tail handling.
Decision rules for launch, iteration, targeting, or rollback.

Follow-up Questions Guidance

How would you measure impact if users can share Shopping content with control users?
What if GMV rises but core feed engagement falls?
How would you handle purchase outcomes with high variance?
What would you monitor during a 1% ramp?

Experiment Design: Measure Instagram Shopping Impact

Constraints & Assumptions

Measure causal impact, not just post-launch correlation.

Include engagement, revenue, and app-health guardrails.

Address selection bias, non-compliance, delayed conversions, heavy-tailed purchase outcomes, and network effects.

Explain when user-level A/B, cluster randomization, geo holdout, or difference-in-differences would be appropriate.

Clarifying Questions to Ask Guidance

Which Shopping surface is launching, and who is eligible?

Is the goal GMV, revenue, buyer engagement, creator/merchant value, or overall app engagement?

Can we randomize users, clusters, geographies, or rollout timing?

What spillovers are likely through follows, shares, creators, or merchants?

What a Strong Answer Covers Guidance

Clear treatment, control, eligibility, exposure logging, and intent-to-treat analysis.

Experimental options: user-level A/B for clean individual effects, cluster randomization for social interference, geo holdouts for ecosystem effects, and staged ramps for safety.

Primary metrics such as incremental buyers, purchases, GMV, revenue, product-detail engagement, or checkout conversion.

Guardrails including retention, session quality, ads revenue, latency, crashes, support, feed engagement, hide/report rates, and merchant or creator fairness.

Methods for selection bias and non-compliance: random assignment, ITT/TOT framing, covariate balance checks, CUPED, and propensity or diff-in-diff only when randomization is limited.

Power, duration, minimum detectable effect, delayed conversion windows, and heavy-tail handling.

Decision rules for launch, iteration, targeting, or rollback.

Follow-up Questions Guidance

How would you measure impact if users can share Shopping content with control users?

What if GMV rises but core feed engagement falls?

How would you handle purchase outcomes with high variance?

What would you monitor during a 1% ramp?

Design Experiment to Measure Shopping Feature Impact

Quick Overview

Design Experiment to Measure Shopping Feature Impact

Experiment Design: Measure Instagram Shopping Impact

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer

Design Experiment to Measure Shopping Feature Impact

Quick Overview

Design Experiment to Measure Shopping Feature Impact

Experiment Design: Measure Instagram Shopping Impact

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer