Why do transformers struggle with long context?
Company: Dandy
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: HR Screen
In a transformer-based model, why is it difficult to process very long input context? Explain the main challenges in terms of computation, memory usage, attention behavior, and training or inference efficiency. Briefly mention common techniques used to reduce these problems.
Overview: This question evaluates understanding of transformer architectures, attention mechanisms, and scalability trade-offs, assessing competencies in computational complexity, memory usage, and system-level performance implications.
Community answers
Answer by payal27roy
Transformers struggle with long context mainly because
full self-attention is O(N^2) in compute and memory
, which hurts GPU utilization, batch size, and latency additionally, the model can have behavioral degradation (retrieval/positional/generalization) at extreme lengths. Practical systems combine
efficient attention kernels sparsity/approximation KV-cache optimizations
, and often RAG/chunking to keep context manageable.