Investigate Declining ROI and Propose Effective Solutions
Overview
This question evaluates a data scientist's skills in advertising effectiveness measurement, causal inference, experiment design, and diagnostic analytics for two-sided marketplaces, covering north-star and guardrail metrics, attribution, and root-cause analysis for declining ROI.
TikTok
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
E-commerce Ads Effectiveness and Diagnostics (Analytics & Experimentation)
Context
You are a data scientist on an e-commerce platform responsible for measuring advertising effectiveness and diagnosing performance issues across a two-sided marketplace (users and merchants). You work with paid placements/ads that affect merchant GMV and ad ROI.
Tasks
- Resume Project Walkthrough
-
Describe one relevant project end-to-end:
- Objectives and success metrics (north-star and supporting/guardrail metrics).
- How you evaluated success (incrementality, attribution, and statistical evaluation).
- Key details of your A/B experiment design (unit of randomization, power/MDE, duration, interference handling, guardrails).
- Case 1 — Grow Merchant Ad Revenue or Merchant GMV
-
The business wants to increase either merchant advertising revenue (ad spend on the platform) or merchant GMV. How would you:
- Frame objectives and select metrics.
- Analyze current performance (funnel, segments, cohorts).
- Propose data-driven actions and experiments.
- Case 2 — ROI on Ads Has Declined
-
ROI/ROAS has recently fallen. Walk through how you would:
- Diagnose root causes with a structured metrics tree and analyses (segmentation, cohorts, time-series, experiment logs).
- Recommend short-term mitigations and longer-term fixes.
Consider Including
- Clear north-star metric and supporting metrics.
- Experiment setup for marketplaces with potential auction/interference effects.
- Funnel, cohort, and segmentation analyses.
- A hypothesis-driven root-cause framework and validation plan.
Solution
Login required
Show