Why do transformers struggle with long context?
Company: Dandy
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: HR Screen
In a transformer-based model, why is it difficult to process very long input context? Explain the main challenges in terms of computation, memory usage, attention behavior, and training or inference efficiency. Briefly mention common techniques used to reduce these problems.
Quick Answer: This question evaluates understanding of transformer architectures, attention mechanisms, and scalability trade-offs, assessing competencies in computational complexity, memory usage, and system-level performance implications.