How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a medium difficulty Machine Learning question, commonly asked during Technical Screen rounds at Applied Intuition.

What role is this question designed for?

This question is commonly asked for Machine Learning Engineer candidates at Applied Intuition during technical interviews.

Implement correct attention masking

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a candidate's competency in attention mechanisms for autoregressive Transformers, specifically the implementation of causal and padding masks, their combination in multi-head attention, and the ability to diagnose masking-related training anomalies.

|Home/Machine Learning/Applied Intuition

Implement correct attention masking

Applied Intuition

Sep 6, 2025, 12:00 AM

mediumMachine Learning EngineerTechnical ScreenMachine Learning

Autoregressive Transformer: Correct Attention Masking with Padding

Context: You are implementing decoder self-attention for an autoregressive Transformer where input sequences in a batch are right-padded to a common length. You must prevent attention to future tokens and exclude padded positions.

Task:

Implement a causal mask that prevents attending to future tokens.
Implement a padding mask that prevents attending to padded positions.
Show how to combine and apply these masks in multi-head attention.
Explain common bugs, how they appear in loss/accuracy, and how to test for them.

Assumptions:

Batch-first tensors: shape (B, T, D).
Right padding. An attention_mask of shape (B, T) uses 1 for real tokens and 0 for padding (or equivalently, you have pad_token_id and input_ids).
PyTorch is available.

Loading comments...

Browse More Questions

More Machine Learning•More Applied Intuition•More Machine Learning Engineer•Applied Intuition Machine Learning Engineer•Applied Intuition Machine Learning•Machine Learning Engineer Machine Learning