Model wins offline but regresses online right before launch: what do you do?
Company: Yahoo
Role: Machine Learning Engineer
Category: Behavioral & Leadership
Difficulty: easy
Interview Round: Onsite
In a behavioral round with a product manager, you are given this scenario: your model wins offline, but online testing or a partner team's validation on production data shows a quality regression against the product's success criteria, for example precision or recall. Launch is very near. As a senior ML engineer, what would you do, how would you communicate it, and what is your plan of action?
### Clarifying Questions
- How large is the regression relative to the success criteria, and how confident is the measurement (number of examples, duration)?
- Is the launch date tied to an external commitment, or can it move?
- Is the current production model, or the current non-ML behavior, still available as a fallback?
- Did the partner team measure with the same label definition, threshold and traffic as our offline evaluation?
### Part 1 — What you do first
"What would you do?" Describe your immediate response and how you would diagnose the gap.
```hint Trust neither number yet
Before choosing between the offline result and the online one, ask what would have to be true for both to be correct at the same time.
```
#### What This Part Should Cover
- Verifying that the regression is real and that the comparison is like for like
- The likely causes of an offline-online gap, and how to test each
- What happens to the launch while the diagnosis runs
### Part 2 — Communication
"How would you communicate it?"
```hint Known, unknown, next update
Think about who must hear first, how to separate what is verified from what is suspected, and when the next update will come.
```
#### What This Part Should Cover
- Who is told, how quickly, and in what form
- Facts, unknowns, options and a recommendation
- Working with the partner team without blame
### Part 3 — Plan of action
"What is your plan of action?"
```hint More than launch or delay
List the options between a full launch and a full delay, and ask which of them still meet the agreed success criteria.
```
#### What This Part Should Cover
- Options with their risks, decided against the agreed success criteria
- Concrete fix and re-validation steps with owners and dates
- Process changes so that such a gap is caught earlier next time
### What a Strong Answer Covers
- Treats the agreed success criteria as the bar, not the offline win
- A systematic diagnosis of offline-online gaps rather than guesswork
- Early, factual communication with clear options and a named decision owner
- The ownership expected of a senior engineer: drives the plan through to the follow-up
### Follow-up Questions
- The product manager wants to launch anyway because the date is tied to an announcement. What do you do?
- How do you tell training-serving skew apart from a genuine shift in the production data?
- The partner team's labels disagree with your labeling guidelines on many examples. How do you settle whose labels are right?
- What would you change in the evaluation process so this is caught weeks before launch?
Overview: Scenario for a senior machine learning engineer: a model that wins offline shows a precision or recall regression in online testing or a partner team's validation shortly before launch. Tests diagnosis of offline-online gaps, stakeholder communication, and a concrete plan of action.
Read the full Yahoo Machine Learning Engineer interview experience this question came from