Ads, Revenue, And Marketplace Analytics

What's being tested
Ability to define and measure ad marketplace metrics, design rigorous experiments or causal analyses, and reason about auction dynamics, advertiser responses, and privacy-driven measurement constraints. Interviewers want clear framing of revenue impact, bias sources, and actionable tradeoffs.

Core knowledge

• Key metrics: impressions, reach, frequency, CTR, CVR, CPM, CPC, eCPM, CPA, ROAS, ARPU, LTV.
• Auction model: value-based/auction ranking (generalized second-price variants), ranking = bid × estimated action rate.
• Experimentation: user-level randomization, cluster/geo experiments, interference, SUTVA violations, pre-period balance.
• Causal methods: randomized A/B, diff-in-diff, instrumental variables, regression discontinuity, propensity scores, uplift modeling.
• Attribution issues: lookback windows, multi-touch vs last-touch, deterministic vs probabilistic matching, SKAdNetwork/aggregate privacy limits.
• Data systems & tooling: Spark/Presto/Hive for logs, Airflow orchestration, SQL-based ad-hoc analysis, propensity or causal ML libs.
• Common biases: selection bias, advertiser bid response, seasonality, post-treatment changes in supply or demand.

Worked example — "Evaluate revenue impact of a new auction ranking algorithm"
Start by defining primary KPI (incremental net revenue per DAU and advertiser ROI) and guardrail metrics (user engagement, CTR, ad quality). Propose an experiment: user-level randomization with sufficient traffic slices, stratify by geography/time, and log auction-level variables (winning bids, pctr, ad quality, payments). Plan to measure short-term incremental revenue and model long-term bidder response: estimate immediate delta via A/B, then build a dynamic model or run longer rollout to capture bid adaptation and supply effects. If randomization isn't possible, propose an IV leveraging rollout timing or eligibility thresholds.

A common pitfall
The tempting quick answer is "run a simple pre/post uplift on revenue." That ignores seasonality, bidder adaptation, and interference across users/auctions, producing biased estimates. Similarly, optimizing only for short-term revenue can worsen user engagement or advertiser ROI; failing to log auction-level granularity prevents diagnosing why revenue changed.

Further reading

• Kohavi et al., "Trustworthy Online Controlled Experiments" (practical experimentation pitfalls).
• Edelman, Ostrovsky, & Schwarz, "Internet Advertising and the Generalized Second-Price Auction" (auction fundamentals).