Evaluate New Ad Model with A/B Testing Experiment

Q: Evaluate New Ad Model with A/B Testing Experiment

This question evaluates experimental-design and product-analytics competencies—A/B testing, traffic allocation and ramping, metric definition and guardrails, statistical power and significance, and handling marketplace/auction interference—within the Analytics & Experimentation domain for a Data Scientist role.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Evaluate a New Ads Recommendation Model via Online Experimentation

Scenario

You have trained a new ad-recommendation model and must decide whether it should replace the incumbent model that currently ranks/serves ads in a large-scale, auction-based ads system.

Task

Design an experiment to evaluate the new model against the incumbent and answer:

Experiment design: unit of randomization, traffic split/ramp, duration, and how to handle auction/marketplace interference.
Metrics: define primary success metric(s) and guardrails (revenue, CTR, ROI, user retention, latency, etc.).
Stakeholder readouts: how the story and metrics differ for a CFO vs a CGO (growth).
Decision framework: launch or rollback criteria, with monitoring and risk mitigation.

Requirements

Cover A/B design, traffic allocation, expected duration, statistical significance/power, and trade-offs between finance and growth.
Include clear metric definitions and assumptions where needed.

Evaluate New Ad Model with A/B Testing Experiment

Evaluate a New Ads Recommendation Model via Online Experimentation

Scenario

Task

Requirements

Solution

Comments (0)