Design an Experiment to Evaluate New ML Model

Q: Design an Experiment to Evaluate New ML Model

This question evaluates a data scientist's competencies in online experiment design, causal inference, metric selection, assessment of statistical and practical significance, identification and handling of heterogeneous treatment effects, data-quality and marketplace interference mitigation, and ethical rollout decision-making.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Experiment Design: Validating a New Ads Ranking Model

Context

You operate an ads platform with an existing recommender/ranking model. Engineers built a new ML ranker that is hypothesized to improve outcomes. You need to run an online controlled experiment (A/B) to validate performance, choose appropriate metrics, assess significance, and make a rollout decision, including handling heterogeneous effects and ethics.

Tasks

Experiment design
- What is your randomization unit, ramp plan, duration, and sample size approach?
- How will you guard against marketplace interference and data-quality issues?
Metrics
- Define 2–3 primary KPIs and a set of guard-rail metrics. Explain why each matters for users, advertisers, and the platform.
Practical significance
- An A/B test shows a +5% lift in CTR. How do you judge practical (business) significance vs. statistical significance?
Heterogeneous effects
- CTR doubles for Indian males aged 18–55. What might this indicate, and what next steps do you take to validate and respond?
Rollout decision
- If the test shows +5% CTR and +5% revenue, do you roll out globally? Explain your decision process, including trade-offs and ethical checks.

Design an Experiment to Evaluate New ML Model

Experiment Design: Validating a New Ads Ranking Model

Context

Tasks

Solution

Comments (0)