Define Success with Contact Syncing for Growth and Evaluation
Overview
This question evaluates a data scientist's competency in growth metric definition, experiment evaluation, and causal inference within product analytics, and is commonly asked because interviewers need to assess the ability to align proxy metrics with true‑north outcomes, set realistic targets, diagnose when A/B test changes fail to move business metrics, and choose valid causal‑inference approaches when randomization is infeasible. It is categorized under Analytics & Experimentation and tests a mix of conceptual understanding (metric alignment and causal reasoning) and practical application (target-setting and A/B test diagnosis) for mid-to-senior product-focused data scientists.
Using "% of users with contacts synced" as a growth driver
Context
You are a data scientist at a consumer fintech app with strong network effects in peer-to-peer interactions. Leadership wants to use the metric "% of users with contacts synced" to drive growth and evaluate experiments. Assume:
- Contacts synced means the user has granted permission and at least one contact has been successfully uploaded and processed within a recent window (for example, last 90 days).
- The true-north business metric (TN) is a volume/engagement outcome such as active transacting users or payment volume.
Question
- How would you position this percentage metric as a meaningful goal for stakeholders?
- Describe a framework to set a realistic 2025 target for this metric.
- During an A/B test the percentage increases but the true-north business metric does not—how would you investigate and respond?
- If an A/B test is infeasible, what causal-inference approach(es) would you use to estimate the impact of contact syncing?
Solution
Login required