Analytics & Experimentation: Facebook Groups
Context: You are evaluating how to measure and improve Facebook Groups. Assume access to standard product analytics stacks, experiment frameworks, and integrity signals. Answer all parts concisely with formulas and concrete decision rules.
1) Measure Success: KPI Hierarchy, Definitions, Guardrails
Propose a KPI hierarchy that captures member value, creator value, and platform health. Provide exact metric definitions (e.g., weekly commenter rate, median time-to-first-reply, churn hazard) and concrete guardrails (e.g., cross-surface cannibalization: News Feed views per DAU).
2) Resource Prioritization: Small vs. Large Groups
You can improve either small groups (<100 monthly active members) or large groups (≥1000). Build a marginal ROI framework that defines incremental value per engineering week by combining uplift to retention and quality-adjusted interactions. Show how you would estimate diminishing returns and uncertainty, then choose which segment to fund under a fixed quarterly budget.
3) Observation → Causality Plan
Observed: Posts in Groups receive more comments than posts from friends/business accounts.
-
List at least 3 plausible causal mechanisms and 3 confounders.
-
Design a verification plan distinguishing causality from selection: include stratified matching by author and audience size, within-user cross-surface randomized holdouts (if available), and a DID exploiting group join dates.
-
Specify the exact regression(s) you would run, including fixed effects and key interaction terms, and how you would validate common trends.
4) Decision Under Mixed Evidence
Given mixed evidence (Groups +8% comments but −2% reach on Feed), state the product decision you recommend, thresholds under which you would reverse it, and what additional logging you require to monitor long-term community health.