Key challenges and design strategies in agentic AI systems

Learn how latency, unreliable outputs, shared state, external services, and human review affect an agent’s design. For each problem, this lesson connects a design choice to the failure it addresses and the cost it introduces.

The previous lessons introduced the agent loop, its components, orchestration, and guardrails. Putting those components into production raises another set of questions: how long a task takes, what happens when a tool fails, which facts remain current, and how to tell whether the system completed the work correctly.

Use the following challenges to examine an architecture before scaling it. A strategy that helps one dimension can hurt another: an extra review call may catch errors while adding time and cost, for example.

High inference latency

An agent’s latency includes model inference, tool execution, network delays, queueing, and any human review. Multiple dependent calls add to the time before the user gets a result. Measure those stages separately so that an inference optimization does not distract from a slower external service.

Latency and cost are related but distinct. More compute may reduce inference time while raising cost; waiting for a remote API may be slow without using much compute. Set a response-time target for the user’s task, then measure the cost of meeting it.

The following are some of the design strategies that can help reduce this issue:

• Model selection and routing: Evaluate smaller or specialized models for bounded tasks such as entity extraction. Route more demanding work to a larger model when the quality improvement justifies it. In a hypothetical extraction-and-copywriting workflow, the two stages may need different models, but that choice should follow measured results.

• Model optimization: Quantization, pruning, and distillation can reduce resource requirements. Check their effect on task quality and actual latency on the deployment hardware.

• Caching: Reuse results only when the inputs, permissions, and freshness requirements allow it. Define expiration and invalidation rules. A repeated read may return changed data, and an idempotent operation does not necessarily have a cacheable response.

• Parallel operations: Run independent calls concurrently. Calls that depend on each other’s results or modify shared state need coordination.

• Asynchronous processing: Put long-running reports or background work in a job workflow so other requests can proceed. Let users check progress and retrieve the result.

• Hardware and deployment: Evaluate GPUs, TPUs, or local deployment against the model’s requirements and the measured bottleneck. Moving inference nearer the input can reduce network time, but hardware capacity still matters.

Output uncertainty and hallucination

An LLM can produce an answer that reads well but contains unsupported or incorrect claims. Other quality failures include biased treatment, irrelevant content, or invalid formatting. These failures need different checks, especially when an answer will guide a consequential action.

Validation also has a cost: it can reject valid answers, delay a response, or introduce another model’s errors. Evaluate the complete workflow rather than assuming each added check improves it.

The following are some of the design strategies that can help reduce this issue:

• Retrieval-augmented generation (RAG): Supply relevant evidence from an appropriate source. Check retrieval quality and whether the answer uses that evidence correctly.

• Output validation: Use schemas and rules for mechanical requirements, and source comparisons or review for factual claims. Valid JSON can still contain a false statement.

• Ensembling and voting: Compare candidates when useful, while testing for shared errors. Agreement is evidence to assess, not a correctness guarantee.

• Confidence estimates: Evaluate whether a score predicts errors on representative tasks before using a threshold for escalation. An unsupported self-rating is a weak basis for an important decision.

• Clear instructions: Specify the task, required output, available evidence, and what to do when information is missing.

Memory management and consistency

Memory management means deciding which context to keep, how to retrieve it, and how to handle updates. A preference saved last month may conflict with today’s request. Two agents may also attempt to update the same record.

Poor memory management makes an agent repeat questions or act on outdated facts. More storage alone does not solve this: the system needs ownership, timestamps or versions where useful, and rules for resolving conflicts.

Choose the storage and update rules for each kind of state:

• Separate memory by purpose: Keep current dialogue context, durable user facts, and external documents in appropriate stores. A structured profile suits exact preference lookup; vector search suits finding semantically relevant passages.

• Access and update policies: Define who may read or modify each record. Use the storage system’s concurrency controls for competing writes.

• Selective retention: Store information because a future task needs it, with expiry and removal rules for sensitive or temporary data.

• Versioning and audit: Record important changes and their origin so a later reader can distinguish current facts from superseded ones. Logs support investigation but do not themselves prevent conflicting writes.

Scalability

Scaling changes the workload as well as the request count. More users, longer tasks, and more agents can increase model calls, tool traffic, stored state, and review queues at different rates.

Load-test the expected workload and identify the limiting resource. Adding workers helps only if the model endpoint, database, external API, or approval queue can support the additional work.

Address the measured bottleneck:

• Modular architecture: Give components clear interfaces so that a busy retrieval service or worker pool can scale independently.

• Orchestration: Distribute independent tasks when the work supports it. Multiple agents also add coordination and model-call overhead, so compare them with a simpler worker design.

• Resource allocation: Balance load, scale capacity, and bound queues according to dependency limits.

• Tool efficiency: Remove redundant calls and batch compatible operations. Account for the external service’s throughput limits instead of assuming it will remain fast under load.

Integration complexity

An agent may connect to modern APIs, legacy services, and several data stores. Each has its own authentication, schema, failure behavior, and version changes. The integration layer must translate those contracts into tools the agent can use reliably.

A thin prototype wrapper may hide important distinctions, such as a failed request versus an action that succeeded but returned no response. Make those states explicit before relying on automatic recovery.

Make the service contract explicit:

• Tool contracts: Define argument and result schemas, permissions, side effects, and error states.

• API adapters: Keep service-specific details in a wrapper so the agent’s task logic does not depend on every provider convention.

• Error handling: Use try...catch or the language’s equivalent to distinguish validation failures, transient failures, and uncertain outcomes. Retry only when the operation’s contract makes repetition safe.

• Handoff messages: Specify the task, supplied evidence, expected output, and completion status when one agent delegates to another.

Security and privacy vulnerabilities

An LLM agent can encounter hostile instructions in user messages, retrieved documents, or tool results. Prompt injection tries to make that content redirect the agent’s behavior. Attempts to bypass content restrictions and accidental disclosure of sensitive data create additional risks.

Treat external content as data with a source and a trust level. Access controls must still apply when the model has been persuaded to request something it should not obtain.

Apply controls to the data and action paths:

• Checks at multiple boundaries: Inspect suspicious input, minimize sensitive fields, validate tool calls, and check outgoing content. Test these controls together; a filter can miss an attack.

• Authentication and authorization: Establish the caller’s identity and enforce access to each resource or action.

• Secure deployment: Protect credentials, network connections, and stored data, and review the deployment’s security configuration.

• Differential privacy: For suitable data-analysis or training workflows, consider a formally specified mechanism that quantifies privacy loss. This requires careful design and evaluation; it is not a general filter that makes arbitrary agent outputs private.

Lack of standardized evaluation metrics

An agent changes its environment over several steps, so a fluent final answer is an incomplete measure of success. Evaluation needs to check the requested outcome, the path taken, and the resources and permissions used.

Choose metrics for the application before comparing architectures. A system that appears more capable may simply spend more calls or rely on more human help.

Build an evaluation around the task:

• Task outcomes: Measure completion and correctness against observable results. Also track cost, time, turns, and user satisfaction. The upcoming MACRS case study uses success rate, hit ratio at K, and average turns for a recommendation task.

• Simulation: Use controlled scenarios to reproduce failures and test edge cases. Then assess how well the simulator represents real users and tools.

• Human evaluation: Use expert review, user feedback, and appropriately designed comparisons to assess qualities automatic metrics miss.

• Traces and logs: Record requests, available context, model outputs, tool calls, and observed results. These are observable artifacts, not a complete record of the model’s internal reasoning.

Human-in-the-loop overhead

A human approval step needs staffing, context, and a response-time expectation. If reviews arrive faster than people can handle them, tasks accumulate even when the automated components have spare capacity.

Measure queue time and reviewer workload as part of end-to-end performance. Review quality can fall when people repeatedly receive low-value alerts or incomplete evidence.

Design the review path as part of the workflow:

• Review placement: Require approval where policy or the action’s consequences call for it, and escalate unresolved ambiguity or conflicting evidence. Use evaluated signals rather than the model’s self-confidence alone.

• Reviewer interface: Show the exact proposed action, relevant source data, checks already performed, and what remains uncertain.

• Summaries with evidence: A generated summary can orient the reviewer, who should also be able to inspect the underlying records.

• Feedback: Use reviewed cases to improve instructions, tests, or a separately evaluated training process. Reflection changes context; it does not automatically update model weights or remove an approval requirement.

Fault tolerance and failure recovery

A tool, model endpoint, or worker can fail partway through a task. Recovery is hardest when the caller does not know whether an external action completed. Continuing from an incorrect assumption can leave inconsistent state or duplicate an action.

Consider a hypothetical payment request that times out after submission. The payment might have succeeded. Check its status or use the service’s supported idempotency mechanism before trying again; treating every timeout as “nothing happened” can cause a duplicate payment.

Define recovery behavior before a failure occurs:

• Graceful degradation: Continue with unaffected capabilities when appropriate. If product lookup is unavailable, a support agent can answer general questions while clearly identifying what it cannot verify.

• Bounded retries: Retry transient failures with a limit and backoff, only when repetition is safe. Preserve the request identity for services that support idempotent retries.

• Checkpoints: Save workflow progress and relevant state. On resume, reconcile external actions rather than assuming restoring local state reverses them.

• Failure handoffs: Escalate when recovery rules are exhausted, carrying the attempted actions and known results to the next operator.

• Isolation: Contain failures with appropriate process or service boundaries and limits on shared resources.

Reviewing the complete system

These challenges are connected. A longer reflection loop changes latency and cost; shared memory changes privacy and consistency requirements; an approval gate changes throughput. Test the whole task with these interactions in mind, including realistic failure cases.

Before deployment, review the following questions using measured results and failure traces:

• Performance: Does the system meet the response-time and cost targets under expected and peak load? Which stage limits it?

• Correctness: Are important outcomes checked against evidence or system state? Have missed errors and false rejections both been evaluated?

• Recovery: Are retries bounded and safe? Can the workflow detect uncertain outcomes and reconcile external actions?

• Memory: Are access, concurrent updates, stale records, and retention handled explicitly?

• Security: Are permissions enforced at resource and tool boundaries, including when the model receives hostile content?

• Human review: Are approval points, staffing, queue behavior, and reviewer evidence defined?

• Observability: Can an operator reconstruct the inputs, calls, and observed effects without collecting unnecessary sensitive data?

• Maintenance: Are dependencies, tool contracts, deployment procedures, and ownership documented and tested?