Debug a broken Transformer implementation

Quick Overview

This question evaluates a candidate's ability to debug and validate a Transformer implementation, focusing on attention masking, parameter initialization, loss alignment, and other implementation-level correctness issues.

Debug a broken Transformer implementation

Company: OpenAI

Role: Machine Learning Engineer

Category: Machine Learning

Difficulty: hard

Interview Round: Onsite

You are given a small Transformer model implementation (e.g., in PyTorch) plus a tiny training script. The code executes, but the model does not match a reference implementation: unit tests that check (1) the forward-pass output for a fixed input/seed and (2) the training loss for one step either fail or are inconsistent. Task: Debug the model so that it runs end-to-end and matches the expected outputs/loss. The buggy code contains multiple independent issues, including: 1) An error in the attention mask (shape/broadcasting or causal/padding masking is applied incorrectly). 2) Incorrect parameter initialization (some weights are initialized with the wrong distribution/scale or not initialized at all). 3) A bug in the loss computation due to misaligned positions (e.g., logits/labels are shifted incorrectly for next-token prediction). 4) One additional hidden bug of similar difficulty (e.g., wrong softmax dimension, missing attention scaling by sqrt(d_k), wrong dtype/device handling, dropout/eval-mode misuse, or an off-by-one in sequence lengths). Explain how you would systematically find and fix these issues, and what the correct implementations should look like.

Quick Answer: This question evaluates a candidate's ability to debug and validate a Transformer implementation, focusing on attention masking, parameter initialization, loss alignment, and other implementation-level correctness issues.

|Home/Machine Learning/OpenAI
OpenAI logo
OpenAI
Jan 21, 2026, 12:00 AM
hardMachine Learning EngineerOnsiteMachine Learning
170
0

You are given a small Transformer model implementation (e.g., in PyTorch) plus a tiny training script. The code executes, but the model does not match a reference implementation: unit tests that check (1) the forward-pass output for a fixed input/seed and (2) the training loss for one step either fail or are inconsistent.

Task: Debug the model so that it runs end-to-end and matches the expected outputs/loss. The buggy code contains multiple independent issues, including:

  1. An error in the attention mask (shape/broadcasting or causal/padding masking is applied incorrectly).
  2. Incorrect parameter initialization (some weights are initialized with the wrong distribution/scale or not initialized at all).
  3. A bug in the loss computation due to misaligned positions (e.g., logits/labels are shifted incorrectly for next-token prediction).
  4. One additional hidden bug of similar difficulty (e.g., wrong softmax dimension, missing attention scaling by sqrt(d_k), wrong dtype/device handling, dropout/eval-mode misuse, or an off-by-one in sequence lengths).

Explain how you would systematically find and fix these issues, and what the correct implementations should look like.

Loading comments...