Debug an LLM Agent Codebase: Find and Fix Bugs in the Agent Loop and Its Tools

In this round you are given the codebase of an LLM agent project and asked to find and fix its bugs. The bugs can be in the agent's tools or in the agent code around them. You may use an AI coding assistant during the round. The task is not hard in itself; it tests how well you read unfamiliar code, locate faults and verify your fixes. The actual codebase was not reported. The module below, `agent.py`, is an illustrative stand-in of the same kind: a tool registry, two tools and the agent loop. The model client's behavior is also an assumption, stated here so that the code can be judged against it: - `llm.complete(messages, tools)` takes the conversation and the tool registry and returns a reply with `content`, a string, and `tool_calls`, a list that is empty when the model gives its final answer. - Each tool call has an `id`, a tool `name`, and `arguments`, a JSON-encoded string such as `'{"path": "notes.txt", "start": 1, "end": 20}'`. - After a reply that makes tool calls, the client expects `messages` to contain that reply as `{"role": "assistant", "content": ..., "tool_calls": [...]}`, followed by one `{"role": "tool", "tool_call_id": ..., "content": ...}` message for each call. - The tool descriptions shown to the model say that `read_file` reads a file in the agent's workspace directory using 1-indexed, inclusive line numbers, and that `search` returns the ids of the indexed documents that contain the query. ```python import json TOOLS = {} INDEX = [] # documents with .id and .text, loaded at startup def tool(name): def register(fn): TOOLS[name] = fn return fn return register @tool("read_file") def read_file(path: str, start: int = 1, end: int | None = None) -> str: with open(path) as f: lines = f.readlines() return "".join(lines[start:end]) @tool("search") def search(query: str, results: list = []) -> list: for doc in INDEX: if query in doc.text: results.append(doc.id) return results def run_agent(llm, task: str, max_steps: int = 10) -> str: messages = [{"role": "user", "content": task}] for _ in range(max_steps): reply = llm.complete(messages, tools=TOOLS) if reply.tool_calls is None: return reply.content call = reply.tool_calls[0] try: result = TOOLS[call.name](**call.arguments) except Exception: result = "" messages.append({"role": "tool", "content": str(result)}) return messages[-1]["content"] ``` ### Clarifying Questions - Is there a failing test, a bug report or an example task that reproduces the problem, or must the bugs be found by reading? - Are the bugs only in the tools, only in the loop, or in both, and roughly how many are expected? - May you change the tool descriptions or the client contract, or only the code that must satisfy them? - Can you run the agent against a real model during the round, or only against local fakes? ### Part 1 — Find the bugs Read the module and list every bug you find. For each one, give the line, what is wrong, the symptom it would cause when the agent runs, and how you would confirm it. ```hint Trace one run Walk a single task through `run_agent` by hand: one reply with a tool call, then a reply with the final answer. At each line, compare the value you hold with what the client contract says it is. ``` ```hint Tools against their descriptions Check each tool against the promise its description makes to the model, including what happens when the same tool is called again in the same process, and which inputs the model controls. ``` #### What This Part Should Cover - Bugs in both the agent loop and the tools, each tied to an observable symptom - How one bug can hide another, and which one surfaces first - A systematic reading order rather than scanning at random ### Part 2 — Fix and verify Fix the bugs, then show how you would prove each fix works without calling a real model. Explain how you would use the AI assistant during this work and how you would check what it suggests. ```hint A scripted model `run_agent` depends on the model only through `llm.complete`. Think about what you can test if that object returns replies you wrote yourself. ``` #### Clarifying Questions for this Part - When the agent runs out of steps, should it raise an error or return a best-effort answer? - Should a failing tool call be reported to the model so that it can try again, or should it abort the run? #### What This Part Should Cover - Minimal, correct fixes that satisfy the client contract and the tool descriptions - Deterministic tests that fail before each fix and pass after it - How the AI assistant was used, and how each of its suggestions was checked ### What a Strong Answer Covers - Reads the client contract and the tool descriptions first, then traces data through the loop - Finds bugs in both the loop and the tools and explains each with a concrete symptom - Makes targeted fixes instead of rewriting the module - Verifies with fast, deterministic tests built on a scripted fake model and direct tool calls - Treats model-supplied tool arguments as untrusted input - Uses the AI assistant to speed up reading and test writing while confirming every claim independently ### Follow-up Questions - The model sometimes calls a tool that does not exist or passes arguments of the wrong type. How should the loop respond? - A tool returns a 2 MB file. What happens to the conversation, and how would you bound tool output? - How would you build regression tests for the agent's end-to-end behavior when the real model is not deterministic? - The agent gains a tool that runs shell commands. What changes in how you validate and sandbox tool calls?