Training LLM Agents for Long-Context, Long-Horizon Tasks: Environments and Rewards

Discuss how you would train a language-model agent to perform well on **long-context, long-horizon tasks**: tasks in which the model must work over a large amount of input and complete many dependent steps, such as tool calls, reads and intermediate drafts, before the final outcome can be judged. The discussion then turns to the reinforcement-learning side: how you would design the environments the model trains in, and the rewards it learns from. The report does not name a specific task, model size or compute budget. Choose one concrete long-horizon task to anchor your answer (for example, answering a question that requires locating and reconciling evidence across a large collection of long documents), and state your assumptions about the base model and resources. ### Clarifying Questions - What does the base model start with: its context length, and whether it is already instruction-tuned and able to call tools? - Can the final outcome of a task be checked automatically, or does judging it require expert review? - How much expert time is available for demonstrations, preference labels or rubric writing, compared with compute? - Are there inference-time limits on latency or cost that cap how many steps or tokens an episode may use? ### Part 1 — Training approach for long context and long horizons Describe the training recipe: which stages you would run, what data each stage needs, and how you would address two different difficulties: using very long inputs well, and staying on track over many steps. ```hint Split the two problems A model can fail because it cannot find or use information deep in its input, or because it loses the plan after many steps. Ask which training signal targets each kind of failure. ``` #### What This Part Should Cover - Stages from long-context adaptation through supervised trajectories to reinforcement learning, and what each contributes. - Data sources and synthesis for long inputs and for multi-step demonstrations. - Evaluation that separates long-context use from long-horizon completion, so you know which stage to improve. ### Part 2 — Environment design Design the environment in which an agent would train on your chosen task: what it observes, which actions or tools it has, how an episode starts and ends, and how tasks are generated at scale. ```hint Think about what makes an episode reproducible RL needs many rollouts of the same task and a fair comparison between them. Consider what must be fixed, sandboxed or snapshotted for that to hold. ``` #### What This Part Should Cover - Observation and action spaces, including how the agent manages a trajectory longer than its context window. - Task generation with controllable difficulty and a held-out split. - Determinism, isolation and throughput of rollouts. ### Part 3 — Reward design Design the rewards: what is rewarded at the end of an episode, whether and how intermediate steps are rewarded, and how you would stop the policy from exploiting the reward. ```hint Assume the policy will find the loophole For each reward term you propose, ask what the cheapest behavior is that scores well without solving the task. ``` #### What This Part Should Cover - Outcome rewards versus step-level rewards, and verifiable checks versus model-graded rubrics. - Credit assignment over long trajectories and the handling of sparse rewards. - Concrete reward-hacking risks for the chosen task, their mitigations, and how you would confirm that the reward tracks real quality. ### What a Strong Answer Covers - One concrete task carried through training, environment and reward, rather than three disconnected surveys. - A clear separation of long-context failures from long-horizon failures, with an evaluation for each. - Trade-offs between expert data, synthetic data and RL compute. - Specific reward-hacking and evaluation-leakage risks, with detection and mitigation. ### Follow-up Questions - How would you decide whether a failure should be fixed with more supervised trajectories or with more RL? - How would you train on a task whose outcome only an expert can judge, at a volume far below what RL needs? - If rollouts became too long to fit in memory during training, what would you change in the agent's context management or in the training setup? - How would you detect that a model-graded reward has drifted away from what experts actually prefer?