Design How an LLM Detects Questions Beyond Its Training Cutoff and Says So

When a chat assistant such as ChatGPT-3.5 is asked about something recent, it may reply that its training data only goes up to a certain point (for example, 2022) and that it therefore cannot answer. How would you implement this behavior in a large language model you are building? ```hint Two separate problems Knowing where your own knowledge ends is a different problem from recognizing that a particular question depends on something after that point. Solve them separately. ``` ```hint Where the behavior lives Consider each place the behavior could come from: the prompt at serving time, the training data, a separate component in front of the model, or a tool. Each has a different cost when the cutoff moves. ``` ### Constraints and Clarifications - Assume you control the model's training pipeline, including a pretraining corpus in which most documents carry a date and the later fine-tuning stages, as well as the serving stack. - The model is retrained periodically, so the cutoff date will change. ### Clarifying Questions - Should the model refuse outright, or answer with what it knew as of its cutoff and a clear caveat? - Is the product allowed to use retrieval or web search, which could answer the question instead of declining it? - Does the model receive the current date at inference time? - Which failure is more costly for this product: declining a question it could have answered, or confidently answering with outdated information? ### What a Strong Answer Covers - A framing that separates knowing the cutoff, knowing the current date, and detecting time-dependent questions - How training data for the behavior is constructed, including examples where the model should answer normally - Where the logic lives, and the trade-offs between prompting, fine-tuning, a separate classifier and retrieval - Evaluation of both failure directions, with concrete metrics - Keeping the behavior correct when the model is retrained with a newer cutoff ### Follow-up Questions - A model's knowledge usually thins out gradually over the last months before its nominal cutoff. How does that affect the date you tell it and its answers about that period? - How would you measure over-refusal on questions about stable facts that merely mention a recent year? - If you later add web search, how does this behavior change, and what should the model say when the search fails?