Evaluates experiment design for restaurant recommendations inside a social feed. Strong answers define treatment, control, metrics, guardrails, power, rollout, and decision criteria while accounting for feed-health trade-offs.
##### Scenario
Facebook considering launching restaurant recommendations in the News Feed.
##### Question
Design an experiment to evaluate the feature: define treatment, control, success metrics, guardrail metrics, sample-size approach, and rollout criteria.
##### Hints
Focus on incremental engagement, CTR, downstream orders; watch for feed scroll depth and churn.
Overview: Evaluates experiment design for restaurant recommendations inside a social feed. Strong answers define treatment, control, metrics, guardrails, power, rollout, and decision criteria while accounting for feed-health trade-offs.
Experiment Design: Restaurant Recommendations in Facebook News Feed
Facebook is considering restaurant recommendation units inside News Feed, such as cards showing nearby restaurants with images, ratings, and a call to action. The product goal is to drive meaningful engagement and downstream restaurant activity without harming feed health.
Design an experiment to evaluate whether the feature should launch.
Constraints & Assumptions
Preserve causal interpretation by defining treatment, control, exposure, and randomization before looking at outcomes.
Avoid confounding the restaurant card with feed position, eligibility, geography, or supply availability.
Include both user-side outcomes and feed-health guardrails.
Make sample-size and rollout recommendations explicit enough for a production experiment review.
Clarifying Questions to Ask Guidance
Which markets and users are eligible for restaurant recommendations?
Is the module inserted in a fixed slot, ranked organically, or shown only when a restaurant candidate is available?
What downstream action is most valuable: click, save, order intent, reservation, or completed order?
Are restaurant partners, ad revenue, or local business outcomes part of the launch decision?
Part 1 - Treatment, Control, and Randomization
Define the experimental arms, the randomization unit, eligibility, and exposure logging.
What This Part Should Cover Guidance
Prefer user-level randomization with sticky assignment and intent-to-treat analysis.
Explain how control users are handled when the treatment would show a restaurant module.
Log eligibility, actual exposure, feed position, and module interactions.
Identify contamination risks such as cross-device behavior, marketplace supply limits, or shared social context.
Part 2 - Metrics and Guardrails
Define the primary success metric, secondary metrics, and guardrails.
What This Part Should Cover Guidance
Include restaurant-card engagement, downstream conversion, and durable user engagement where applicable.
Use feed-health guardrails such as hide/report, session quality, time spent quality, retention, and ad revenue.
Separate per-exposed metrics from user-level intent-to-treat metrics.
Include restaurant-side or merchant outcomes if the product decision depends on them.
Part 3 - Power, Duration, and Decisioning
Describe how you would size the experiment, run it safely, and decide whether to launch.