How should you evaluate unconnected content?
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
A social media platform has launched a feed feature that increases the share of unconnected content, meaning posts from creators who do not have an existing social relationship with the viewer. The stated goal is to improve engagement, but the interviewer wants a deeper product and experimentation framework rather than a single engagement-rate metric.
Answer the following as a data scientist:
1. How would you define success for this launch?
- Distinguish between short-term engagement, meaningful social interaction, long-term retention, and creator or ecosystem health.
- Explain which metric you would choose as the primary decision metric and why.
2. What metric framework would you use?
- Propose a funnel from impression to consumption to interaction to downstream value.
- Include primary metrics, secondary metrics, and guardrail metrics.
- Discuss tradeoffs between rate-based metrics and absolute metrics.
3. If unconnected content does not directly increase social interaction between friends, why might the company still want to launch it?
- Provide product, business, and user-value hypotheses.
4. What are the main risks of this launch?
- Consider cannibalization of friend content, low-quality recommendations, safety concerns, creator concentration, heterogeneous user effects, and misleading aggregate metrics.
5. How would you design an experiment to evaluate this launch?
- Specify the unit of randomization, treatment definition, duration, power or MDE considerations, segmentation strategy, and how you would handle interference or network effects.
- Explain what could go wrong in measurement and interpretation.
Quick Answer: This question evaluates a data scientist's competency in product analytics, experimentation design, metric engineering, causal inference, and risk assessment for feed and recommendation changes.