Evaluate and safely deploy a CVR model online
Company: Tradedesk
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
You have built a new CVR (conversion rate) prediction model for an RTB bidding system at a DSP. Offline metrics improved (e.g., lower log loss and higher PR-AUC), but you must validate that the change improves business outcomes under real auction dynamics.
Design an online evaluation and rollout plan.
Include:
1) What primary and guardrail metrics you would monitor (e.g., CPA, ROAS, conversions, spend, win rate, latency).
2) An experimental design appropriate for RTB (A/B test vs shadow mode vs interleaving). Define the unit of randomization and how to avoid interference.
3) How you would handle budget constraints/pacing and non-stationarity during the test.
4) What failure modes you would watch for (e.g., calibration shifts, bidder feedback loops, overbidding) and what rollback/ramp-up strategy you would use.
Quick Answer: This question evaluates a data scientist's competence in online experimentation, model deployment safety, and metrics-driven monitoring for real-time bidding systems, including metric selection, experimental design, budget and pacing controls, and detection of operational failure modes.