Implement multi-head attention and LLM sampling

Q: How do I practice coding and algorithm questions?

Use PracHub's coding console to write, test, and debug your solutions in Python or JavaScript. View hints, test against sample inputs, and compare with official solutions.

Q: What difficulty level is this coding question?

This is a easy difficulty Coding & Algorithms question, commonly asked during Onsite rounds at Scale AI.

Q: What role is this question designed for?

This question is commonly asked for Machine Learning Engineer candidates at Scale AI during technical interviews.

Question

Implement multi-head attention and LLM sampling

PracHub · Accepted Answer

This question evaluates implementation and conceptual understanding of Transformer multi-head scaled dot-product attention and next-token sampling for LLMs, including attention masking semantics, softmax numerical stability, time and space complexity, and top-k/top-p with temperature-based sampling.

Quick Overview

Implement multi-head attention and LLM sampling