Evaluate and safely deploy a CVR model online
Quick Overview
This question evaluates a data scientist's competence in online experimentation, model deployment safety, and metrics-driven monitoring for real-time bidding systems, including metric selection, experimental design, budget and pacing controls, and detection of operational failure modes.
Tradedesk
Dec 10, 2025, 12:00 AM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
You have built a new CVR (conversion rate) prediction model for an RTB bidding system at a DSP. Offline metrics improved (e.g., lower log loss and higher PR-AUC), but you must validate that the change improves business outcomes under real auction dynamics.
Design an online evaluation and rollout plan.
Include:
- What primary and guardrail metrics you would monitor (e.g., CPA, ROAS, conversions, spend, win rate, latency).
- An experimental design appropriate for RTB (A/B test vs shadow mode vs interleaving). Define the unit of randomization and how to avoid interference.
- How you would handle budget constraints/pacing and non-stationarity during the test.
- What failure modes you would watch for (e.g., calibration shifts, bidder feedback loops, overbidding) and what rollback/ramp-up strategy you would use.
Solution
Show