How Context Window Length Shapes LLM Quality, Cost and Latency

What changes when a large language model's context window is made larger or smaller? ```hint Follow each extra token Consider what the model has to compute and hold in memory for every token it can see, and whether a model reliably uses everything placed in front of it. ``` ### Clarifying Questions - Do you mean changing the maximum length the model is built and trained to handle, or changing how much of an existing window an application chooses to fill? - Is there a use case in mind, such as analyzing long documents, long multi-turn chats, or answers grounded in retrieved passages, where the trade-offs differ? - Should the answer cover serving cost, latency and price as well as output quality? ### What a Strong Answer Covers - What the context window is and everything that counts against it, including the model's own output - How more or less context changes answer quality, including how well models use information at different positions in a long input - How compute, memory, latency and price scale with context length - What extending a model's window requires, and what a smaller window forces an application to do - How a product team would pick a context length and when to prefer retrieval ### Follow-up Questions - A competitor launches a model with a much longer context window. How would you decide whether your product needs to match it? - Users report that the assistant ignores instructions given early in long conversations. What is likely happening, and what would you change? - When would you still use retrieval instead of putting every document directly into a very long context?