Determine Success Metrics for Circle Feature Optimization
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Meta is evaluating a new social feature called **Circle** (similar to Facebook Groups), where members join a group and then post and comment within it. You own product analytics for Circle: you have to decide how to measure it, how to evaluate it under a fixed engineering budget, which kind of group to optimize for, and how to read the charts you are handed.
##### Question
Concentrate on **engagement and retention** as the objective. Name guardrails where they matter, but they are not the focus. Answer the following:
1. **Success metrics.** What metrics would you track to measure the success of Circle, and how do they differ from the metrics you would use for regular (non-Circle) posts? Define a primary north-star metric and the supporting metrics around it.
2. **Experiment design under a resource constraint.** Design an experiment to evaluate Circle, first assuming a **small** engineering allocation and then assuming a **large** one. What changes between the two designs, and what trade-offs are you accepting in each?
3. **Small vs. large groups (retrospective only).** Using only retrospective (observational) data — no live experiment — how would you decide whether Circle should be optimized for **small private groups** (≈5–6 members) or **large public groups** (tens to hundreds of members)? Detail the analysis, the confounders you would control for, and the decision rule.
4. **Comparing the charts.** You are shown three time-series plots of `total comments / total posts` — one for Circle and two for other surfaces with different user bases (for example business posts and friends posts). Can these lines be compared directly? What insights or hypotheses would you draw from them?
##### Hints
- Pick a single north-star metric that captures reciprocal conversation, then normalize it per active member; segment by group size, privacy, and cohort.
- The engineering budget decides your randomization unit: what can you gate cheaply (member-level access) versus what needs real infra (graph-cluster or factorial designs with delivery gating).
- Retrospective data is observational — beware reverse causality (good groups grow) and survivorship; use within-unit fixed effects, threshold/event studies, or matching.
- The comment/post ratio is sensitive to user-base composition, heavy tails, and measurement windows; normalize before comparing.
Overview: A four-part Meta data scientist onsite case on Circle, a Facebook Groups-style feature: define a north-star success metric and its supporting tree, design an evaluation experiment under both a small and a large engineering allocation, and decide from retrospective data alone whether to optimize for small private or large public groups. The final part asks whether three time series of total comments divided by total posts can be compared directly, and a strong answer explains why normalization, cohorting, and composition reweighting come first.