Evaluate Success of 'Similar Listings' Notification Feature
Overview
This question evaluates a data scientist's competency in product analytics, causal inference, experimental design, and metric-definition for a notification feature that surfaces listings similar to items a user has viewed or saved, situated in the Analytics & Experimentation domain.
Meta
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Marketplace Analytics Case: "Similar Listings You May Like" Notifications
Context
You work on a marketplace where buyers browse and purchase listings. The team is considering a new notification feature (push/email/in-app) that alerts users when new listings similar to items they've viewed or saved become available. The goal is to increase user engagement and conversion without causing notification fatigue or harming overall marketplace health.
Tasks
- Decision framework: How would you decide whether introducing this notification feature is a good idea? State your hypotheses, expected user/business value, risks, and prerequisites.
- Experiment design: Propose an A/B test to evaluate the feature. Specify unit of randomization, eligibility, randomization scheme, sample size/power, duration, attribution windows, and a roll-out plan (ramp/holdouts).
- Success criteria: Define and justify primary, secondary, and guardrail metrics (e.g., CTR, conversion, retention, user satisfaction), including precise metric definitions and evaluation windows.
Solution
Login required
Show