Implement correct attention masking

Q: Implement correct attention masking

This question evaluates a candidate's competency in attention mechanisms for autoregressive Transformers, specifically the implementation of causal and padding masks, their combination in multi-head attention, and the ability to diagnose masking-related training anomalies.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Machine Learning question, commonly asked during Technical Screen rounds at Applied Intuition.

Q: What role is this question designed for?

This question is commonly asked for Machine Learning Engineer candidates at Applied Intuition during technical interviews.

Question

Autoregressive Transformer: Correct Attention Masking with Padding

Context: You are implementing decoder self-attention for an autoregressive Transformer where input sequences in a batch are right-padded to a common length. You must prevent attention to future tokens and exclude padded positions.

Task:

Implement a causal mask that prevents attending to future tokens.
Implement a padding mask that prevents attending to padded positions.
Show how to combine and apply these masks in multi-head attention.
Explain common bugs, how they appear in loss/accuracy, and how to test for them.

Assumptions:

Batch-first tensors: shape (B, T, D).
Right padding. An attention_mask of shape (B, T) uses 1 for real tokens and 0 for padding (or equivalently, you have pad_token_id and input_ids).
PyTorch is available.

Implement correct attention masking

Quick Overview

Implement correct attention masking

Autoregressive Transformer: Correct Attention Masking with Padding

Write your answer