How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Technical Screen rounds at Amazon.

What role is this question designed for?

This question is commonly asked for Software Engineer candidates at Amazon during technical interviews.

Explain attention and Transformers | Amazon Interview Question

Quick Overview

Explain attention and Transformers evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Explain attention and Transformers

Scaled Dot-Product Self-Attention, Transformer Architecture, and BERT vs GPT

You are interviewing for a software engineer role focused on machine learning. Explain the core math and design choices behind Transformers and how they translate to practical trade-offs in transfer learning and inference.

1) Scaled Dot-Product Self-Attention

Derive and define the following:

Queries (Q), Keys (K), Values (V) and how they are computed from inputs
The scaling factor and why it is needed
Masking (padding and causal)
Softmax over attention logits
Time and memory complexity (including multi-head and autoregressive decoding)

2) Transformer Architecture

Explain the encoder–decoder Transformer architecture, including:

Encoder vs decoder stacks and their sublayers
Residual connections and the normalization strategy (pre-norm vs post-norm)
Positional encoding (sinusoidal and alternatives)

3) BERT vs GPT

Compare BERT and GPT in terms of:

Pretraining objectives
Architectural differences
Typical downstream usage
How these choices affect transfer learning and inference behavior/performance

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

Clarify the task, data shape, labels, constraints, and evaluation metric.
State assumptions behind the math or modeling technique you choose.
Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

Correct definitions and formulas where the prompt requires them.
A practical explanation of how the method behaves on real data.
Trade-offs, failure modes, diagnostics, and mitigation strategies.
Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

How would noisy labels, class imbalance, or distribution shift affect the answer?
What would you monitor after deployment?
Which baseline would you compare against first?

Quick Overview

3) BERT vs GPT

Compare BERT and GPT in terms of:

Pretraining objectives

Architectural differences

Typical downstream usage

How these choices affect transfer learning and inference behavior/performance

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.

If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.

Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

Clarify the task, data shape, labels, constraints, and evaluation metric.

State assumptions behind the math or modeling technique you choose.

Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

Correct definitions and formulas where the prompt requires them.

A practical explanation of how the method behaves on real data.

Trade-offs, failure modes, diagnostics, and mitigation strategies.

Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

How would noisy labels, class imbalance, or distribution shift affect the answer?

What would you monitor after deployment?

Which baseline would you compare against first?

Explain attention and Transformers

Quick Overview

Explain attention and Transformers

Explain attention and Transformers

Scaled Dot-Product Self-Attention, Transformer Architecture, and BERT vs GPT

1) Scaled Dot-Product Self-Attention

2) Transformer Architecture

3) BERT vs GPT

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer

Explain attention and Transformers

Quick Overview

Explain attention and Transformers

Explain attention and Transformers

Scaled Dot-Product Self-Attention, Transformer Architecture, and BERT vs GPT

1) Scaled Dot-Product Self-Attention

2) Transformer Architecture

3) BERT vs GPT

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer