Assessing whether a new metric A is meaningful for News Feed
Overview
This question evaluates a data scientist's ability to validate a proposed proxy metric for product goals, testing skills in metrics validation, predictive reasoning, experiment design, and instrumentation within the Analytics & Experimentation domain.
Meta
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Analytics & Experimentation
21
0
Evaluating a Proposed Proxy Metric for News Feed
Context
A partner team proposes a new proxy metric, metric A, intended to capture "meaningful interactions" in News Feed. Before adopting it, you need to evaluate whether it is a good proxy for the underlying product goals (e.g., user value, long-term engagement, satisfaction).
Task
Assess whether metric A is meaningful for News Feed along three dimensions:
- Predictive power: Does A reliably predict the outcomes we care about, ahead of time?
- Actionability: Can the product/ranking system move A in the right direction, and will doing so improve the true goals without perverse incentives?
- Collection cost: What is the engineering, infra, latency, privacy, and experimentation cost to log and use A at scale?
Provide a plan, tests, and decision criteria for each dimension.
Solution
Login required
Show