Debug a Broken Transformer
Company: OpenAI
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
You are given a Transformer model implementation that does not train correctly. Describe how you would debug it systematically from data input to optimization. Your answer should cover common failure points such as tokenization and labels, tensor shapes, attention masking, positional encoding, train versus eval mode, loss function setup, gradient flow, and optimizer configuration. As a follow-up, explain how to convert the model into a classification model and verify the change by running a single forward and backward pass successfully.
Quick Answer: This question evaluates a candidate's practical debugging and model engineering skills for Transformer implementations, covering tokenization and labels, tensor shapes, attention masking, positional encodings, train versus eval modes, loss and gradient flow, optimizer configuration, and converting a model to classification within the Machine Learning/deep learning (NLP) domain. It is commonly asked to assess end-to-end troubleshooting and implementation-level reasoning, and its level of abstraction is primarily practical application with some required conceptual understanding of model internals.