Diagnose Decline in User Engagement and Experience Quality
Overview
This question evaluates a data scientist's competency in defining product metrics, diagnosing declines in user engagement through retention, cohort and segmentation analysis, designing experiments or analyses, and applying causal-inference reasoning with observational data.
Product Metrics Deep-Dive and Causal Inference (TikTok)
Context
You are a data scientist working on TikTok’s core product. Over several weeks, daily active users (DAU) have been declining. Separately, a manager wants to understand how users’ network speed affects TikTok usage, but only observational data are available.
Tasks
a) DAU has been steadily declining. Which metrics would you define, and what experiment(s) or analysis would you design to diagnose the issue?
b) To quantify the causal impact of users’ network speed on TikTok usage with observational data, describe a suitable causal-inference approach and how you would explain its validity to a non-technical stakeholder.
Hints: Consider retention, cohorts, segmentation, and methods like instrumental variables (IV) or propensity weighting. Communicate assumptions clearly.
Solution
Login required