E-commerce Ads Effectiveness and Diagnostics (Analytics & Experimentation)

Context

You are a data scientist on an e-commerce platform responsible for measuring advertising effectiveness and diagnosing performance issues across a two-sided marketplace (users and merchants). You work with paid placements/ads that affect merchant GMV and ad ROI.

Tasks

1. Resume Project Walkthrough

• Describe one relevant project end-to-end:
  • Objectives and success metrics (north-star and supporting/guardrail metrics).
  • How you evaluated success (incrementality, attribution, and statistical evaluation).
  • Key details of your A/B experiment design (unit of randomization, power/MDE, duration, interference handling, guardrails).

2. Case 1 — Grow Merchant Ad Revenue or Merchant GMV

• The business wants to increase either merchant advertising revenue (ad spend on the platform) or merchant GMV. How would you:
  • Frame objectives and select metrics.
  • Analyze current performance (funnel, segments, cohorts).
  • Propose data-driven actions and experiments.

3. Case 2 — ROI on Ads Has Declined

• ROI/ROAS has recently fallen. Walk through how you would:
  • Diagnose root causes with a structured metrics tree and analyses (segmentation, cohorts, time-series, experiment logs).
  • Recommend short-term mitigations and longer-term fixes.

Consider Including

• Clear north-star metric and supporting metrics.
• Experiment setup for marketplaces with potential auction/interference effects.
• Funnel, cohort, and segmentation analyses.
• A hypothesis-driven root-cause framework and validation plan.