Debug a Transformer Implementation and Implement a KV Cache for Decoding
Company: OpenAI
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
Overview: Practice ML coding question on Transformer language models: find and fix several deliberately planted bugs in a decoder-only Transformer implementation, then implement key-value caching from scratch for autoregressive generation. Tests attention mechanics, tensor-shape reasoning, causal masking, position handling, and verification against full recomputation.
Read the full OpenAI Machine Learning Engineer interview experience this question came from