Debug a Small PyTorch Transformer with Four Bugs, Then Add a KV Cache
Company: OpenAI
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: hard
Interview Round: Onsite
Overview: Debug a small PyTorch transformer language model that hides four bugs, some of which crash and some of which silently ruin training, then add a key-value cache so generation computes only the newest token each step. It tests causal masking, positional encoding, training-loop order and cached decoding.