How to evaluate a similar-listing notifications feature
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
##### Question
You are a Data Scientist on a US C2C marketplace app (like Facebook Marketplace) where users buy and sell second-hand products.
**Current product behavior**
- Users browse product listings.
- If a buyer is interested in a listing, they can click **“Send message”** to contact the seller.
- Each message sent counts as **one listing interaction**.
**Proposed feature**
On a product listing, buyers can opt into **reminders/notifications** for **“similar listings you may like.”** When similar products become available, the buyer receives a notification.
Answer the following:
1. **Pre-launch / decision framing.** How would you decide whether this feature is a good idea for the product? Cover:
- The user problem and hypothesis you are testing.
- What success metrics you would expect to move (and why), and how you would distinguish primary vs. diagnostic vs. guardrail metrics.
- Key tradeoffs and risks (e.g., notification fatigue, adverse selection, cannibalization of search).
- What data you would analyze *before* building to validate demand and size the opportunity (e.g., backtesting against historical logs).
- What MVP / phased rollout plan you would propose if you were uncertain.
2. **Post-implementation impact evaluation.** Assume engineers have shipped the functionality (or it can be enabled for some users). How would you measure its impact and determine whether it is successful? Be specific about:
- The recommended experiment or causal design, the unit of randomization, control vs. treatment, and duration.
- Primary success metric(s) vs. secondary/diagnostic metrics vs. guardrail metrics.
- Key pitfalls (opt-in selection bias, notification fatigue, interference/network effects, seasonality, attribution) and how you would handle them.
- How you would interpret results and decide to iterate, roll out, or roll back.
Overview: A Meta Data Scientist analytics-and-experimentation interview question: a US C2C second-hand marketplace is considering opt-in “similar listings” notifications. You must (1) decide whether to build it — hypotheses, opportunity sizing, risks, MVP plan — and (2) design the post-launch measurement: a user-randomized A/B test with a primary/diagnostic/guardrail metric stack, handling of opt-in selection bias, and a clear launch/rollback rule.
Community answers
Answer by SS
The Core Idea
You want to test: "Does offering notifications help buyers?"
But you have a trap: some people will opt in, some won't. You can't just compare "people who opted in" vs. "people who didn't" because they're different types of people.
Solution: Randomize who gets the OFFER, not who uses it.
Unit of Randomization: User-Level (Not Event-Level)
What does this mean?
WRONG: Randomize at notification event level
Buyer searches for "used phones"
Similar phone listed
→ Flip a coin: should we notify them?
YES → send notification
NO → don't send notification
Problem: Same buyer might get notified about one phone
but not notified about another similar phone.
They start comparing → confused about what the feature does.
RIGHT: Randomize at user level
Buyer signs up
→ Flip a coin for THAT USER
YES → They can opt into notifications (and will see them forever)
NO → They can't opt in (control group)
Consistency: Each user has one experience throughout the test.
Treatment vs. Control (The Setup)
TREATMENT Group
User sees prompt: "Get notified when similar listings appear?"
If they click YES → they get notifications
If they click NO → they don't get notifications (their choice)
Key point: They had the OPTION
CONTROL Group
User never sees the prompt.
They can't opt in.
No notifications.
The Selection Bias Problem
The trap:
Suppose 40% of treatment users opt in.
Suppose 50% of treatment users who opted in made a purchase.
Question: Did notifications help?
Naive answer: "Users who opted