Define Success Metrics for Circle Feature Evaluation
Overview
This question evaluates a data scientist's competency in defining product success metrics, designing experiments under varying engineering-resource constraints, applying normalization and cohorting for comparative analytics, and reasoning about causality and interference within the Analytics & Experimentation domain.
Scenario
Measuring success and allocating resources for a new "Circle" posting feature in a social app. Circle lets a creator share posts with a smaller, selected audience (e.g., close friends), alongside existing posting types like friends and business/public posts.
Task
- Define success metrics for Circle relative to regular posts (friends/business). Include primary, secondary, and guardrail metrics.
-
Design an experiment to evaluate Circle under two engineering-resource settings:
- Small allocation (minimal ability to change delivery/ranking/instrumentation)
- Large allocation (can change delivery, ranking, and logging; can enforce exposure rules) Discuss trade-offs for each.
- You are given three line charts (over time) of metric = total comments / total posts for Circle, business posts, and friends posts. Can we compare these lines directly? Why or why not? What additional normalizations or cohorting would you require? What hypotheses or insights might you form?
Hints: Address normalization (per exposure, user-day), cohort selection (post age, creator/viewer cohorts), variance/power, resource constraints, and causality/interference.
Solution
Login required