Observations, memory, and the agent loop

Trace how observations and stored state enter an agent’s decision loop. Compare session context, persistent records, and external retrieval, then use a support case to diagnose missing information.

The previous lesson separated the model, tools, and instructions. A task spanning several steps also needs a way to carry relevant information from one decision to the next. The application may supply that information in the current context, retrieve it from storage, or observe it again from the environment.

This lesson examines those information paths. The key questions are what the agent can observe, which records it retains, and which of those records actually reach the next model invocation.

By the end of this lesson, we will be able to:

• Identify how memory systems and perception extend basic agent functionality.

• Understand where memory and sensing occur in the agent’s reasoning loop.

• Explain how external data retrieval, such as with vector databases, supports long-term behavior.

• Recognize how real-world agents use these capabilities to respond adaptively and intelligently.

The architecture blueprint

A useful blueprint has four responsibilities:

• Input interface: Receive observations such as text, sensor data, API events, images, or audio.

• Decision-making model: Use the supplied instructions and state to propose a response, plan, or action.

• Tools and actions: Execute validated operations and return their results.

• Memory and context: Store and select relevant history, documents, preferences, or task records.

Instructions guide the model; the input interface and retrieval process determine the evidence available to it. If a relevant fact is stored but omitted from the prompt, the model cannot use that record merely because the database contains it.

Additional components may enforce permissions, inspect outputs, or record traces. Their role should be explicit. A monitoring system observes behavior, while an execution guard can prevent a proposed operation; neither is supplied automatically by a memory store.

The illustration below shows how these elements interact in an agentic system:

The loop receives an observation, selects an action, executes it, and inspects the outcome. State can connect the steps by retaining a plan, an earlier result, or an unresolved question. Updating that state changes what later decisions can use; it does not by itself update the model’s weights.

When evaluating a loop, inspect the information available at each decision. This reveals whether a mistake began in observation, retrieval, context selection, action choice, or execution.

Environmental sensing: obtaining observations

An agent needs observations relevant to its next decision. Those observations can be partial, delayed, or incorrect, so perception is an information-processing step to test rather than a guarantee of grounding.

Environmental sensing includes receiving and processing data from the environment. Classical systems already used cameras, microphones, and other sensors; multimodal input is not unique to LLM agents. The input types required here depend on the application.

Useful observation paths can support:

• Current state: Read a document, device status, or earlier command relevant to the task.

• Multimodal input: Combine speech, text, images, or another supported data type.

• Event-driven behavior: Trigger work when a monitored condition changes, under an explicitly configured schedule or event subscription.

Examples of perception in practice

Examples include:

• A virtual assistant that uses a speech-to-text (STT) model to convert voice commands into text the agent can reason about, and reads text from a calendar.

• A customer support agent that uses a multimodal vision model or optical character recognition (OCR) to extract text and meaning from an uploaded screenshot to understand the issue.

• A factory monitoring agent that receives sensor readings and adjusts machine settings in real time.

• A document-based agent that reads PDFs or spreadsheets to answer user questions.

For a screenshot, OCR may recover the words while a visual model identifies layout or controls. Missing either kind of information can change the next action. Choose the representation according to what the task needs to distinguish.

Where does perception happen in the agent workflow?

Observation can occur at the start of a task and again after each action. A tool result, a changed web page, or a new sensor reading may require the agent to revise its current plan.

This step often involves:

• Converting speech to text.

• Extracting text from an image using OCR.

• Parsing JSON from an API.

• Identifying intent from a natural language prompt.

Separate tools or models may perform these conversions. Retain useful information about their source, time, and errors so downstream decisions can distinguish a recent confirmed result from a failed extraction or an old observation.

Memory systems: retaining and retrieving information

A multistep task often depends on earlier information, such as a user constraint or a completed action. Memory is the application’s mechanism for retaining and making that information available. A simple reactive agent can work without persistent memory, but a task that refers to previous sessions needs some way to recover them.

Memory helps an agent:

• Stay coherent in multi-turn conversations.

• Avoid repeating questions or actions.

• Make decisions based on what it previously learned or observed.

• Maintain a sense of progress toward a goal.

In a hypothetical travel request, “book the same hotel as last time” requires identifying the relevant previous booking. The system might retrieve that record using the authenticated user’s ID, then check the location and dates. A remembered preference does not establish current availability or permission to book.

Retrieving a record is different from learning a new model behavior. Stored experience can influence an answer through context even when every model parameter remains fixed.

Different types of agent memory

Agent memory can be organized by its lifetime and source. The following categories are useful design descriptions rather than a literal model of human memory.

We will distinguish three roles:

Short-term memory

This type of memory holds recent context, typically within a single session or interaction. It allows the agent to keep track of what was just said or done.

For example, in a multi-turn conversation:

• The agent remembers what the user asked two messages ago.

• It tracks follow-up questions or clarifications.

Session history may be held in a buffer and selected for the next context window. Storage and model-visible context are different: truncation or summarization can omit an earlier constraint even while the full transcript remains available elsewhere.

Long-term memory

This memory persists across sessions. It stores important information that the agent might need to refer back to in the future.

Examples include:

• A user’s preferences or settings.

• Notes about past decisions or outcomes.

• Historical interactions or completed tasks.

Persistent memory needs a storage and retrieval mechanism. Decide which records to retain, who may access them, and how corrected or obsolete information is updated. A user preference from last year should not silently override a newer explicit request.

External or knowledge memory

This refers to the agent’s ability to look up relevant information from outside sources, such as documents, knowledge bases, or APIs.

Rather than trying to “remember” everything directly, the agent can:

• Search a knowledge base.

• Retrieve matching documents.

• Pull facts from external systems.

Retrieval-augmented generation (RAG) supplies retrieved material as model context. A vector database can support the search for that material, but similarity alone does not establish relevance or permission.

Vector search retrieves candidates using embedding similarity and can find related wording without an exact keyword match. It does not guarantee that the highest-scoring passage is current or relevant. Exact identifiers, dates, permissions, and structured filters may matter more than semantic similarity for a particular lookup. A past ticket is usually best tied to a customer and ticket ID, not found solely because another customer described a similar problem.

How memory operates across the agent loop

Memory can participate at several points in a loop:

One possible sequence is:

• Retrieve: Select relevant messages, preferences, records, or documents before the decision that needs them. Retrieval itself can be a tool action chosen by an earlier decision.

• Assemble context: Supply the selected evidence to the model along with the current task. Check truncation and conflicting versions.

• Record outcomes: Store confirmed action results or revised task state for later use, with a clear distinction between observations and model guesses.

For example, a failed API call should be recorded as a failure. Storing the intended result as if it occurred would contaminate the next decision and could cause the agent to report completion incorrectly.

Memory quality therefore depends on both what is written and what is retrieved. More storage alone does not solve either problem.

Putting the components into a loop

The responsibilities can now be combined into one illustrative loop:

Follow a single task through these stages:

1. Observe: Receive and interpret the current input or tool result.

2. Recall: Retrieve relevant authorized records when the task needs them.

3. Decide: Use instructions, observations, and selected context to choose the next action.

4. Act: Validate and execute the permitted operation.

5. Update state: Record the observed outcome and progress, then decide whether to continue or stop.

The stages may be combined or repeated. For example, a failed lookup can lead back to another retrieval attempt. The stopping condition should follow task completion, a defined budget, or a problem requiring further input, rather than an assumption that every loop improves the result.

Case study: diagnosing a support agent

Consider a hypothetical support agent for a subscription-based SaaS platform. It handles several messages in one conversation and may need authorized records from previous tickets.

The agent must:

• Answer billing and account questions.

• Retrieve relevant previous tickets and knowledge-base passages.

• Track information already supplied in the current conversation.

• Apply a defined escalation policy when dissatisfaction or another escalation condition is detected.

Suppose the application has an LLM, support APIs, and instructions, but recorded conversations reveal four problems:

1. It repeats questions that the customer answered earlier.

2. It overlooks a recent ticket.

3. It gives a generic answer despite a relevant FAQ.

4. It fails to escalate after the policy’s dissatisfaction condition is met.

Before changing the model, trace one failing decision. Check which history reached it, which ticket lookup ran, which FAQ passages were returned, and which escalation signal was present. These symptoms suggest a context problem, but they do not prove that memory is the only failing component.