Building trustworthy agents: guardrails and human oversight

Design checks for agent inputs, outputs, tools, and memory, then decide where people need to approve or supervise the work. This lesson explains what each safeguard can catch, where it can fail, and how to evaluate the trade-off between protection and useful automation.

An agent can change a user’s account, send money, or disclose information. A mistaken action can therefore persist after the conversation ends. Designing such a system requires clear permissions, checks at the point of execution, and a way to detect and respond to failures.

We will start with guardrails: checks and constraints that reduce specific failure risks. We will then place human review at points where the system needs authorization, domain judgment, or help resolving uncertainty.

A useful design states both what the agent may do and what evidence it needs before proceeding. That makes the limits testable.

By the end of this lesson, you will be able to:

• Explain why trust and safety are essential in agentic systems.

• Identify different types of guardrails and what risks they help prevent.

• Apply best practices for designing guardrails that balance safety with flexibility.

• Recognize when and where human oversight is necessary in agent workflows.

• Understand how to combine guardrails and human review into a layered safety strategy.

Guardrails

Guardrails are checks and constraints on agent behavior. They can validate inputs, inspect generated outputs, limit data access, or reject a tool call. Instructions can guide the model, but permissions and other mandatory conditions should also be enforced by application code or the receiving service. The location of a check determines what it can prevent:

• Input guardrails: Inspect incoming requests or other observations. A check intended to block processing must finish before the dependent work begins.

• Output guardrails: Inspect a generated answer or plan before it is delivered or used.

• Tool-use guardrails: Validate a proposed call immediately before execution, and inspect the result before passing it onward when needed.

For example, an output filter can catch private data in a drafted message. It cannot recover that data if an earlier tool call has already sent it to an unauthorized destination. The relevant check must sit before that transmission.

In a multi-agent system, apply the same reasoning to messages and shared memory. Passing information to another agent is a data transfer with its own permissions and exposure risks.

The following categories address different failure modes. Several may be needed for a single action.

Types of guardrails

Choose safeguards based on the agent’s task, the data it handles, and the consequences of its actions. A required JSON field and an unauthorized payment need different checks.

Here are some of the most commonly used types of guardrails:

Contextual grounding checks

A relevance check asks whether the response addresses the task. A grounding check asks whether its claims are supported by the supplied evidence. A document summary can fail either test: it might offer unrelated commentary, or it might discuss the right subject while inventing a result the document never states.

The following are some ways to implement contextual grounding checks:

• Retrieve relevant source material, as in RAG, and give the model enough context to support its answer. Retrieval supplies evidence; the generated answer still needs checking.

• Instruct the model to use the supplied context and identify missing information.

• Compare important claims with the source material. Rules can check structured facts; classifiers or model-based checks can help assess meaning, with errors that need evaluation.

This type of guardrail is especially important in agents that work with long documents, open-ended queries, or search-driven workflows.

Safety and moderation filters

Safety and moderation filters identify content that conflicts with the application’s policies. Depending on the task, the system may block it, redact it, route it for review, or process it in a restricted workflow.

Apply the policy to both user-facing responses and intermediate content sent to tools or other agents. A backend processor can cause harm through downstream actions even when users never see its raw output.

The following are some ways to implement safety and moderation filters:

• Use moderation APIs or classifiers for the categories relevant to the application.

• Give the model clear instructions about permitted behavior.

• Define how flagged content should be handled, including legitimate tasks that require analyzing sensitive material.

• Limit sensitive capabilities through permissions and rate limits as well as content checks.

For example, a financial assistant should reject a request to compromise a payment system and have no permission to perform that action. An agent processing confidential legal documents has a different requirement: it may need to read the material, but only for an authorized purpose and with controls on where the contents go.

Evaluate filters on representative allowed and disallowed requests. Blocking every sensitive topic can prevent the agent from completing legitimate work.

PII and data protection filters

Agents may handle personally identifiable information (PII), such as email addresses, along with medical records and other confidential material such as company intellectual property. Detection, redaction, and access controls serve different purposes: detection finds potentially sensitive content, redaction removes selected fields, and access control determines who may obtain the data in the first place.

The following are some ways to implement PII and data protection filters:

• Use regular expressions or trained classifiers to identify likely personal data, and test what they miss.

• Minimize or mask unnecessary fields before sending content to the model.

• Enforce permissions when selecting documents and calling data services.

• Audit sensitive access while avoiding unnecessary copies of private payloads in logs.

For example, a healthcare chatbot might retrieve records through vector search. The retrieval service must restrict the search to records the authenticated user is authorized to access. Similarity alone cannot establish that permission. The response should then include only the fields needed for the authorized task.

These controls reduce exposure, but filters alone do not establish regulatory compliance. The GDPR includes requirements concerning lawful processing, purpose, minimization, retention, and accountability. The HIPAA Security Rule includes administrative, physical, and technical safeguards for regulated entities. Whether either framework applies depends on the system and its use.

Tool use safeguards

A tool call can modify a database, send a message, or control a device. Tool safeguards check whether the proposed action is permitted and valid for the current state. Place these checks in the execution path so that an acceptable-looking model response cannot bypass them.

The following are the ways to implement tool use safeguards:

• Allowlisting: Specify the tools available to each role and task.

• Argument validation: Check types, ranges, identifiers, destinations, and required fields before invocation.

• Preconditions: Verify authorization and current system state with application logic or the receiving service. A model’s explanation is not proof that a condition holds.

• Rate limits: Restrict call frequency to limit abuse or runaway loops. A call within the limit can still be invalid.

• Simulation before execution: Use a supported dry-run mode or inspect a proposed action before applying it. A model’s imagined result does not test the real service.

For an IT support agent resetting a password, check the requester’s identity, authorization for the target account, the account’s current state, and duplicate or excessive requests. For production deployment, verify the permitted environment and required approvals. These checks belong to the tool boundary, even if the agent has already discussed them in its plan.

Return a specific rejection reason when a call fails a check. The agent can then request missing information or escalate instead of repeating the same invalid action.

Rules-based output validation

Output validation checks a response against explicit requirements before delivery. Some requirements are mechanical, such as valid JSON or a required field. Others concern meaning and require source comparisons or human judgment.

Here are some ways to implement output validation:

• Pattern checks: Detect specified strings or structured data patterns. Patterns do not establish whether a statement is true.

• Format checks: Validate a schema, template, or required field.

• Semantic checks: Compare the response with the task and evidence, using a separate model or reviewer where appropriate. Model-based checks remain probabilistic.

• Blocklists and allowlists: Enforce application-specific vocabulary or required wording when that rule has a clear purpose.

For example, a policy may require a disclaimer in a particular email. A rule can check its presence, but the disclaimer does not make the email’s advice accurate. For generated code, static analysis can detect selected defects or prohibited functions; it does not establish that the whole program is correct.

Validate the output before a downstream system acts on it. A final answer check cannot undo a tool action that already happened.

Advanced guardrails and system-level safeguards

For more complex workflows, consider the following controls and their limits:

• Critic or evaluator: Ask a separate model or agent to review an output against explicit criteria and evidence. Test whether its feedback improves the result.

• Voting and ensembling: Compare several candidate decisions. Agreement can be useful evidence, but models may share the same mistake.

• Self-reflection: Give the agent feedback and an opportunity to revise. Recording or rereading feedback changes the available context; it does not by itself train the model.

• Multi-agent oversight: Enforce permissions on messages, shared state, and delegated tools. Resolve conflicting actions before they execute.

• Quantitative monitoring: Track failures such as unauthorized call attempts or unsupported claims, alongside the number and type of opportunities for those failures. Raw counts alone can be misleading as traffic changes.

• Recovery: Define which actions can be rolled back and which need a compensating action or human response. A sent email, for example, may not be retractable.

• Audit trails: Record inputs, decisions, calls, and observed results at an appropriate level of detail. Logs support investigation; they do not by themselves prove compliance or reveal the model’s internal reasoning.

Framework support can help implement these controls, but execution order matters. The OpenAI Agents SDK guardrails documentation distinguishes agent input and output checks from tool checks. Its input guardrails run in parallel by default, so agent work can begin before the check finishes; blocking mode waits for the input check first. Agent input checks apply to the first agent in a run, and output checks to the final agent. Read the configured scope and timing before assuming a check covers every handoff or tool call.

Limitations and trade-offs of guardrails

Guardrails introduce their own failure modes and costs:

• False positives: A check rejects a legitimate request or acceptable output.

• False negatives: A check misses a prohibited action or harmful output.

• Latency and complexity: Checks take time and add dependencies and failure paths to debug.

• User friction: Repeated or poorly explained approval requests can interrupt useful work.

Measure the cost of false rejections as well as missed violations. A useful evaluation set includes valid requests that resemble prohibited ones and failures that look superficially valid.

Best practices for designing guardrails

Build and evaluate safeguards incrementally:

• Place checks where they can prevent the failure: Combine input, tool, and output controls according to the data and action path.

• Start with critical risks: Prioritize unauthorized data access, high-impact actions, and failures that are difficult to reverse.

• Iterate and monitor: Review missed violations and false rejections, then retest changes against both.

• Explain limits: Tell users when a request needs review or cannot proceed, with enough detail to resolve the issue.

• Preserve useful work: Use scoped permissions and targeted checks so the agent can still complete authorized tasks.

Controlling what the agent remembers

Memory lets an agent reuse facts and preferences from earlier interactions. That continuity also creates a retention and access problem: stored information can be stale, irrelevant, or available to the wrong task.

Define what is stored, who may retrieve it, how long it remains useful, and how it can be corrected or removed:

• Retain only the personal data needed for an authorized purpose.

• Filter irrelevant or inappropriate content before storage.

• Set expiration rules and ways to correct outdated facts.

• Apply permissions during recall as well as during storage, including access from other agents.

For example, a saved preference for morning meetings may be useful; a copied payment credential usually is not needed for scheduling. Decide at the field level instead of treating an entire conversation as equally worth retaining.

Human oversight and intervention

Human oversight gives people a defined way to review, monitor, or intervene in agent operation. It is useful when a decision requires authorization or judgment that the automated checks cannot supply. The design must specify what the reviewer sees and whether execution waits for their response.

Consider human review when:

• A financial, medical, legal, deployment, or infrastructure action requires specialist judgment or approval.

• Required facts are missing, evidence conflicts, or a policy does not cover the case.

• Several valid outcomes depend on a user’s preference.

• Monitoring detects a failure pattern that the system cannot resolve reliably.

Types of human oversight

The following terms describe different roles for people in a workflow. Terminology varies across systems, so define the actual responsibilities rather than relying on the label alone.

Consider a hypothetical invoice-processing system:

• Human-in-the-loop (HITL): A reviewer approves a specific invoice cancellation before execution continues.

• Human-on-the-loop (HOTL): An operator monitors ongoing processing and can pause it when something goes wrong.

• Human-above-the-loop (HATL): People set the operating policy and review outcomes and logs to improve it.

Let’s explore each of these in more detail:

Human-in-the-loop (HITL)

In HITL, a person participates before the relevant decision or action completes. Two common mechanisms are:

• Approval gates: Pause before an action requiring approval and show the exact target and consequences. For a subscription cancellation, identify the subscription and effective date. Bind approval to the reviewed action and recheck relevant state before execution if the task has been waiting.

• Escalation triggers: Hand off cases with missing facts, conflicting evidence, unclear policy, or failures beyond the agent’s recovery rules. A model’s self-reported confidence can inform review, but should not be the sole trigger.

Human-on-the-loop (HOTL)

In HOTL, a person monitors operation and can intervene while work is underway:

• Intervention interfaces: Show current state, queued actions, and observed results, with controls to pause, cancel pending work, or change the plan. Make clear which actions have already executed.

Human-above-the-loop (HATL)

Here, HATL covers governance and review across runs: people define policy, assess outcomes, and revise the system. It includes retrospective review rather than approval of every individual action.

• Audit trails and logging: Preserve enough evidence to reconstruct what the system received, attempted, and actually changed. Reviewers can investigate failures and update policies or tests. Logs assist that work but cannot intervene in an action that has already completed.

Other oversight models

These roles can also be combined with:

• User-triggered review: Users flag an unexpected result or request intervention.

• Multiple reviewers: Several people assess an output where the task requires broader judgment or agreement. Their review can occur before an action or after a run.

• Automatic escalation: Monitoring detects a defined condition and routes the case to a person for review.

Challenges and trade-offs of human oversight

Human review adds judgment and control, but it requires an operating plan. Consider:

• Latency: Review can delay time-sensitive work. Define what happens while a task waits.

• Cost: Specialist review takes staff time, especially for complex cases.

• Scalability: Review queues can become a bottleneck as task volume grows.

• Fatigue: Frequent low-value alerts can make important cases harder to notice. Give reviewers relevant evidence and prioritize cases by consequence and urgency.

Trust is a system-level design choice

Trustworthy operation depends on controls across the whole system. For each important action, identify the permission, the evidence, the check, and the person or process responsible when it fails.

• Use guardrails at input, tool, output, and memory boundaries to address specific failures.

• Place human approval before actions that require it, monitoring during operation, and policy review across completed runs.

Evaluate these layers together on realistic tasks. The goal is useful, authorized work with detectable failures and a workable response when something goes wrong.

Quiz: designing a safe agentic workflow

You’re designing a multi-agent financial assistant for enterprise users. It supports tasks like invoice management, expense tracking, and automated report generation. Agents in the system can:

• Retrieve and summarize documents from a knowledge base.

• Call APIs to schedule payments and update records.

• Communicate with other agents and human users.

• Access user profiles and transaction history.

For each scenario, identify the main failure before choosing a control. Some scenarios need more than one layer.