Debug a Transformer bug in an unfamiliar repo
Company: Lila
Role: Machine Learning Engineer
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Onsite
You are given an unfamiliar GitHub repository that implements a Transformer model. The interviewer claims there is a **bug** causing one of the following symptoms:
- training loss diverges or becomes NaN,
- model quality is far worse than expected,
- outputs violate causality for an autoregressive model,
- shapes/broadcasting silently behave incorrectly.
You cannot rewrite the whole project; your task is to **find and fix the bug efficiently**.
Explain:
1. The step-by-step debugging strategy you would use.
2. What minimal experiments/tests you would run to localize the issue.
3. The most common Transformer implementation bugs you would check first.
4. How you would validate the fix and prevent regressions.
Quick Answer: This question evaluates proficiency in debugging Transformer implementations, encompassing codebase navigation, tensor shape and broadcasting reasoning, numerical stability, masking/causality checks, and model validation competencies.