How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design A/B test and success metrics for new feature

Quick Overview

This question evaluates a data scientist's competency in experimental design, metric selection and guardrails, power and sample-size calculations, logging and attribution specifications, and revenue impact estimation for a new feature on a social media platform.

Instagram Collections 2.0 — Define Success, Experiment Design, and Measurement

Context: You are proposing a new Instagram feature, Shareable Collections 2.0, intended to increase creator activity and monetization. Design how you would define success, run the experiment, and decide whether to launch.

1) Define Success and Guardrails

State the mission and choose primary North Star metrics (one engagement metric, one revenue metric).
List at least three counter metrics to protect user experience and integrity.

2) Design the Experiment

Choose user-level A/B test or geo-clustered A/B test and justify your choice considering network effects and contamination.
Propose a stratification scheme using deciles of an engagement score. Outline randomization and a ramp plan.
With baseline D30 retention = 30% and a target 1% relative lift, compute the required sample size per arm at 90% power, alpha = 0.05. State assumptions and the formula you use.
Define stopping rules, pre-registration plan, and guardrail thresholds (e.g., crash rate, time spent).

3) Measurement and Decisioning

Specify the exact logging spec (key events and fields) and how you would attribute revenue.
Given: average 4 ad impressions per DAU, CTR = 1%, revenue per ad click = $0.50. If the feature increases ad clicks by 2% on 100M DAUs, estimate weekly incremental revenue. Include sensitivity and uncertainty bands.
List reasons not to launch even if the test shows statistically significant wins (cover technical feasibility and long-term costs).

Quick Overview

2) Design the Experiment

Choose user-level A/B test or geo-clustered A/B test and justify your choice considering network effects and contamination.

Propose a stratification scheme using deciles of an engagement score. Outline randomization and a ramp plan.

With baseline D30 retention = 30% and a target 1% relative lift, compute the required sample size per arm at 90% power, alpha = 0.05. State assumptions and the formula you use.

Define stopping rules, pre-registration plan, and guardrail thresholds (e.g., crash rate, time spent).

3) Measurement and Decisioning

Specify the exact logging spec (key events and fields) and how you would attribute revenue.

Given: average 4 ad impressions per DAU, CTR = 1%, revenue per ad click = $0.50. If the feature increases ad clicks by 2% on 100M DAUs, estimate weekly incremental revenue. Include sensitivity and uncertainty bands.

List reasons not to launch even if the test shows statistically significant wins (cover technical feasibility and long-term costs).

Design A/B test and success metrics for new feature

Quick Overview

Instagram Collections 2.0 — Define Success, Experiment Design, and Measurement

1) Define Success and Guardrails

2) Design the Experiment

3) Measurement and Decisioning

Solution

Submit Your Answer to Earn 20XP

Design A/B test and success metrics for new feature

Quick Overview

Instagram Collections 2.0 — Define Success, Experiment Design, and Measurement

1) Define Success and Guardrails

2) Design the Experiment

3) Measurement and Decisioning

Solution

Submit Your Answer to Earn 20XP