AI agent governance is the set of controls an organization uses to decide which AI agents may run, what each agent is allowed to do, who approved it on what evidence, and how its behavior is checked after launch. In a regulated enterprise it covers an agent inventory, permission and escalation rules, tool-call testing, approvals tied to a specific agent version, production monitoring with human review, and re-evaluation whenever the agent changes.
An AI agent is a system that uses a large language model (LLM) to plan steps and call tools: it looks up an account, opens a case, drafts a letter or routes a claim. That is what makes agents useful, and it is why governing them is different from governing a chatbot. The risk moves from what the system says to what it does.
This guide is for risk, compliance and AI leaders at banks, credit unions, insurers and health systems who have agents in pilot and need a defensible way to put them into production. It covers the controls, the evidence to keep and who owns each decision. For how to evaluate an agent's reasoning, memory and tool use in depth, see our guide to evaluating AI agents; this article does not repeat it.
- Govern agents by what they can do: list every tool, the data it reaches and which actions need a human.
- Keep an agent inventory with an owner, risk tier, tools, permissions, approved version and monitoring owner for each agent.
- Test tool calls, not only final answers: right tool, right arguments, right order, and a refusal or escalation when one is required.
- Tie every approval to an exact version of the model, prompts, tools, retrieval sources and guardrails. A change to any of them reopens review.
- After launch, trace live runs, score them automatically, route samples to human reviewers, and turn confirmed failures into new test cases.
What makes AI agent governance different?
Three things change when an LLM application becomes an agent. It acts through tools, so a wrong step has a direct effect. It takes several steps per request, so a correct final answer can hide a wrong path, such as reading data it should not have touched. And it is built from parts that change on their own schedules: the model provider ships updates, a tool's API changes, a prompt is edited.
The OWASP project names the core risk excessive agency, which it traces to excessive functionality, excessive permissions and excessive autonomy. Each maps to a governance control: limit the tools, limit what each tool can reach, and require human approval for high-impact actions.
Regulation has not caught up evenly. In banking, SR 26-2 places generative and agentic AI outside its scope, as our piece on SR 26-2 and generative AI explains, so banks are setting their own controls. The NIST AI RMF, the EU AI Act and, for insurers, the NAIC AI Model Bulletin all expect documented testing, oversight and monitoring that an agent program has to produce.
What belongs in an AI agent inventory?
You cannot govern agents you cannot list. An agent inventory is a register of every agent in use or in development, with enough detail that a reviewer can see what it does and who answers for it. Include agents bought from vendors and agents embedded in other software, not only those your team built.
| Inventory field | Why it matters |
|---|---|
| Business owner and purpose | Someone is accountable for the outcome, and scope creep is visible |
| Builder or vendor | Shows whether third-party due diligence applies |
| Model, provider and version | A provider update is a change you need to detect |
| Tools and permissions | The list of actions the agent can take, read or write |
| Data it can reach | Personal, account or health data raises the risk tier |
| Autonomy level | Which actions run alone and which need a human |
| Risk tier | Sets how deep testing and review must go |
| Approved version and date | Proves the running agent is the one that was reviewed |
| Monitoring owner and thresholds | Someone watches it after launch, against agreed limits |
Large organizations often keep the enterprise inventory in a dedicated system and the agent-level evidence somewhere else. Our comparison of GenGuardX and IBM watsonx.governance describes one way to split that work.
How do you set agent permissions and escalation?
Start from least privilege. Give the agent only the tools the task needs, and give each tool only the access it needs. Separate read tools from write tools, because reading a balance and reversing a fee carry different risk.
- List every action the agent can take, and mark each as read, write or irreversible.
- For each write or irreversible action, decide whether the agent may act alone, act within a limit, or must hand off to a person.
- Define escalation triggers: a customer asks for a human, a complaint or hardship signal appears, a source is unavailable, or the agent's confidence is low.
- Write the expected behavior for each trigger, so it can be tested.
Example: a credit union's collections agent may look up a member's balance and offer the payment plans in policy. It may not waive fees, and if the member mentions job loss or illness it must stop, acknowledge the situation and transfer to a member rep. This is a fictional example, but each rule in it becomes a test case and a monitoring check. See AI governance for credit unions for the use cases this applies to.
How do you test an agent's tool calls before approval?
Score the path, not only the answer. For each scenario, record which tools the agent should call, with which arguments, in which order, and when it should refuse or escalate instead. Then compare the run against that expectation. Our guide to evaluating AI agents covers trajectory evaluation, tool selection and argument checks in detail.
For approval, three groups of tests matter most:
- Business scenarios. Subject matter experts run the cases they see every day and the awkward ones they remember. Their judgments become the reference answers later tests reuse.
- Risk tests. Accuracy, bias and fair outcomes, toxicity, faithfulness to policy sources, personal data exposure, and prompt injection that tries to make the agent call a tool it should not.
- Escalation tests. Every trigger from the permissions step, checked to confirm the agent hands off instead of acting.
Automated graders, often an LLM judge, can score large volumes, but compare their scores with expert labels on a sample before relying on them. The LLM evaluation guide explains how to pick metrics and thresholds.
How should approvals tie to agent versions?
An approval is only meaningful if you can say exactly what was approved. For an agent, a version is the combination of the model and its version, the system and task prompts, the tool list and each tool's permissions, the retrieval sources, and the guardrails. Record all of it with the approval.
The approval record should hold the test results, open and closed findings, the thresholds used, the names and roles of the approvers, and any conditions, such as a limit on which customer segments the agent may serve. Once approved, lock that version so it cannot be edited in place. A change produces a new version that goes back through review, scaled to the size of the change.
Both the business owner and the risk function should sign. The business confirms the agent does what it should; risk confirms it blocks what it should not. Our article on model risk management for generative AI covers how this fits an existing validation process.
How do you monitor AI agents in production?
Release testing tells you an agent was fit on the cases you tried. Monitoring tells you whether real users are bringing cases you did not try. For agents, monitoring needs the full run, not only the final reply.
- Capture traces. Record each run as a trace: the model calls, tool calls, retrievals and decisions, with timing, cost and errors, linked to the agent version that produced it. Mask personal data before it is stored.
- Score live traffic. Run automated evaluators on live runs or samples, using the same criteria as release testing.
- Set monitors and alerts. Put thresholds on quality, latency, cost and error rates, and watch for patterns such as an agent repeating the same tool call or a step failing silently.
- Route samples to people. Annotation queues send selected conversations, including flagged ones, to business and risk reviewers who label what went wrong.
- Close the loop. Each confirmed failure becomes a finding with an owner, and a new regression test.
Developer observability tools also trace agent runs. LangChain, for example, describes LangSmith as giving "complete visibility into agent behavior," with online LLM-as-judge and code evaluations, per LangChain's site, checked November 2026. Those tools serve the engineering team well. Governance adds what a regulator or auditor asks for on top: the link from a live failure to the approved version, a named owner, a decision and the retest.
When does an agent need re-evaluation?
Any change to the parts that define a version. Keep the saved test suite ready so the rerun is routine, not a project.
| Change | What to rerun |
|---|---|
| Model or provider version | Full suite, including bias and escalation tests |
| Prompt edit | Full suite; a small edit can shift behavior widely |
| New tool or wider permission | Tool-call, escalation and prompt injection tests, plus a new risk review |
| New or updated retrieval sources | Faithfulness and accuracy tests against the new sources |
| Policy or product change | Update reference answers first, then rerun affected scenarios |
| Monitoring breach or confirmed failure | Add the case to the suite, fix, and rerun before closing the finding |
Who owns each agent governance decision?
Agent governance stalls when ownership is vague. Name the people before testing starts.
- Business owner: defines the task, supplies subject matter experts, signs that the agent does what it should.
- AI or engineering lead: builds and versions the agent, fixes findings, keeps traces flowing.
- Model risk or risk management: sets the risk tier, tests and thresholds, and signs that the agent blocks what it should not.
- Compliance: confirms disclosures, fair-outcome rules and regulatory mapping.
- Operations: owns the human side of escalation and reviews annotation queues after launch.
How GenGuardX supports AI agent governance
The GenGuardX governance platform, from Corridor Platforms, is the GenAI and AI agent governance and evaluation platform for regulated enterprises. It gives business, risk and AI teams one shared record for each agent, whoever built it.
- Business testing: a feedback portal where subject matter experts run scenarios against the agent, flag problems and rate answers, which become approved answers for later tests.
- Risk evaluations: standardized reports across agent, prompt, LLM and RAG components, including accuracy, bias and fair lending, toxicity, faithfulness and prompt injection and leakage, plus custom tests and thresholds.
- Findings and approval: findings tracked to closure, and an approval workflow with defined roles that locks the approved version.
- Agent observability: per the GenGuardX docs, traces record every LLM call, tool call, retrieval and agent decision, are tied to the deployment and agent version, and can be sent through the Python SDK, agent frameworks or any OpenTelemetry exporter. Evaluators score live traffic, monitors alert on thresholds, and automated governance signals file findings for risks such as personal data leaks, silent failures, retry loops and evaluator regressions.
- Human review and re-evaluation: annotation queues for live conversations, and saved tests rerun when the model, a prompt or a tool changes.
GenGuardX is SOC 2 Type II certified and deploys as secure SaaS or in your environment on AWS, Microsoft Azure or Google Cloud. It organizes evidence against the NIST AI RMF, the EU AI Act, ISO/IEC 42001, SR 26-2 and the NAIC AI Model Bulletin. Industry pages show how this applies to banks, credit unions, insurers and payers and health systems.
What are the trade-offs of governing AI agents?
Agent governance limits what a mistake can do and proves which version was approved, and it costs slower agent releases, reviewer workload and the effort of rerunning tests after every change.
| What you gain | What it costs |
|---|---|
| Version-tied approvals: the agent in production is the one that was reviewed | Every model, prompt, tool or retrieval change goes back through review, so agent releases slow down |
| Restricted permissions limit the damage a wrong tool call can do | The agent automates less, and operations staff handle more escalations |
| Tool-call and escalation tests before approval | Expected trajectories take time to write, and subject matter experts must supply and rate the cases |
| Live monitoring that links failures to an owner and a version | Traces must be captured and masked, and reviewers have to work through annotation queues on a schedule |
| Saved tests rerun after each change | Each rerun costs model calls and reviewer time, and the suite grows as failures are added |
| One shared record for business, risk and AI teams | A governance platform, GenGuardX included, adds license cost, integration work and onboarding for reviewers |
To keep the costs down, scale each review to the size of the change, as the re-evaluation table above suggests, and reuse the first agent's suite and approval record as the template for the next.
FAQ
How do you govern AI agents in a bank?
List every agent in an inventory with an owner and risk tier, limit each agent's tools and permissions, test tool calls and escalation alongside fair-lending and accuracy tests, have the business owner and model risk approve a specific version, and monitor live runs with human review. Because SR 26-2 leaves agentic AI out of scope, most banks apply their model risk principles through their own GenAI policy. See GenAI governance for banks.
What should an AI agent inventory include?
For each agent: business owner and purpose, builder or vendor, model and version, tools and permissions, data it can reach, autonomy level, risk tier, approved version and date, and who monitors it against which thresholds.
How do you monitor AI agents in production?
Capture each run as a trace of model calls, tool calls and retrievals linked to the agent version, score live traffic with automated evaluators, alert on quality, cost, latency and error thresholds, and send samples to human reviewers. Confirmed failures become findings and new test cases.
Do AI agents need re-approval when the model changes?
Yes, at least a re-evaluation. A model update, prompt edit, new tool or new retrieval source changes the agent's behavior, so rerun the saved tests and have the approvers review the results before the new version serves customers.
What is the difference between agent observability and agent governance?
Observability shows what an agent did in production. Governance decides whether it was allowed to, records who approved which version on what evidence, and makes sure failures reach an owner and a retest. Governance uses observability data as one of its inputs.
How long does it take to set up AI agent governance with GenGuardX?
It depends on scope: the number of agents, how they connect (LLM connectors, API, Python SDK or OpenTelemetry traces), SaaS or in-environment deployment, and SSO. Your team provides access to the agent, representative scenarios and policy documents, subject matter experts to test, and risk owners to set thresholds and approve. Many teams start with one agent. A demo is the place to scope yours.
Start with one agent
Do not try to govern every agent at once. Pick the agent closest to production with the highest consequence, write down its tools and permissions, name its owners, and build the first test suite with the people who will approve it.
If a test fails on an irreversible action, remove the permission or add a human step before release. If the suite passes, record the version, the evidence and the approvers. If production shows something new, add it to the suite. Then do the same for the next agent, reusing what you built.
Want a quick read on where your program stands? Try the two-minute AI trust check.
Resources
Internal resources
- GenGuardX governance platform
- Evaluating AI agents
- LLM evaluation guide
- LLM-as-a-judge evaluation
- Model risk management for generative AI
- SR 26-2 and generative AI
- GenGuardX vs IBM watsonx.governance
- GenAI governance for banks
- AI governance for credit unions
- AI governance for insurers and payers
- AI governance in healthcare