Define Success Metrics for Circle Feature Evaluation

Q: Define Success Metrics for Circle Feature Evaluation

This question evaluates a data scientist's competency in defining product success metrics, designing experiments under varying engineering-resource constraints, applying normalization and cohorting for comparative analytics, and reasoning about causality and interference within the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

Measuring success and allocating resources for a new "Circle" posting feature in a social app. Circle lets a creator share posts with a smaller, selected audience (e.g., close friends), alongside existing posting types like friends and business/public posts.

Task

Define success metrics for Circle relative to regular posts (friends/business). Include primary, secondary, and guardrail metrics.
Design an experiment to evaluate Circle under two engineering-resource settings:
- Small allocation (minimal ability to change delivery/ranking/instrumentation)
- Large allocation (can change delivery, ranking, and logging; can enforce exposure rules) Discuss trade-offs for each.
You are given three line charts (over time) of metric = total comments / total posts for Circle, business posts, and friends posts. Can we compare these lines directly? Why or why not? What additional normalizations or cohorting would you require? What hypotheses or insights might you form?

Hints: Address normalization (per exposure, user-day), cohort selection (post age, creator/viewer cohorts), variance/power, resource constraints, and causality/interference.

Define Success Metrics for Circle Feature Evaluation

Scenario

Task

Solution

Comments (0)