Implement multi-head self-attention correctly

Quick Overview

This question evaluates a candidate's practical implementation skills and conceptual understanding of multi-head self-attention, including query/key/value projections, head-wise tensor reshaping, masking behavior, and considerations for numerical stability and computational complexity.

Implement multi-head self-attention correctly

Company: Apple

Role: Software Engineer

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

Implement multi-head self-attention from scratch in PyTorch or NumPy. Given input X with shape (batch_size, seq_len, d_model), implement linear projections for Q, K, and V; split into h heads (d_k = d_model / h); compute scaled dot-product attention with optional padding and causal masks; then concatenate heads and apply an output projection. Ensure all tensor shapes and transpositions are correct. Provide the forward pass, explain the shape of each intermediate tensor, analyze time and memory complexity, and discuss numerical stability considerations (e.g., softmax scaling and mixed precision).

Quick Answer: This question evaluates a candidate's practical implementation skills and conceptual understanding of multi-head self-attention, including query/key/value projections, head-wise tensor reshaping, masking behavior, and considerations for numerical stability and computational complexity.

|Home/Machine Learning/Apple
Apple logo
Apple
Sep 6, 2025, 12:00 AM
hardSoftware EngineerTechnical ScreenMachine Learning
27
0

Implement Multi-Head Self-Attention (from scratch)

Context

You are given an input tensor X with shape (batch_size, seq_len, d_model). Implement a multi-head self-attention layer (forward pass) using PyTorch or NumPy that:

  • Projects inputs into queries (Q), keys (K), and values (V).
  • Splits into h heads with per-head dimension d_k = d_model / h.
  • Computes scaled dot-product attention with optional padding and causal masks.
  • Concatenates heads and applies an output projection.

Assume d_model is divisible by h.

Requirements

  1. Implement the forward pass with correct tensor shapes and transpositions.
  2. Support optional masks:
    • Padding mask (e.g., shape (batch_size, seq_len) or broadcastable variants).
    • Causal mask (prevent attending to future positions).
  3. Explain the shape of each intermediate tensor.
  4. Analyze time and memory complexity.
  5. Discuss numerical stability (e.g., scaling, masking, softmax stability, mixed precision).
Loading comments...