Agent Governance: Permissions, Guardrails, Identity, Audit Trails, and Budgets

An agent that can call tools is an actor in your production environment — it holds credentials, touches systems of record, and spends money on every step. Governing it takes five interlocking controls: identity (who is this agent), permissions (what may it touch), guardrails (which actions need review), audit trails (what did it do and why), and budgets (how much may it spend). None of them is exotic. Most map onto machinery your cloud and model providers already ship — the work is deciding scope, wiring the enforcement points, and writing the policy that makes the whole thing legible to auditors and to your own engineers.

This matters now because the failure modes are asymmetric. A chatbot that hallucinates embarrasses you; an agent with production write access that misfires deletes data, emails customers, or burns a month's inference budget in an afternoon. Anthropic's own guidance on building agents is blunt about the tradeoff: agent autonomy brings "higher costs, and the potential for compounding errors," and the recommended response is "extensive testing in sandboxed environments, along with the appropriate guardrails."^[1] The governance layer is what turns that advice into standing infrastructure.

1. By the numbers

2. The control plane, not the model

The instinct on many teams is to govern the model: better system prompts, refusal training, content filters. Those help, but they govern *outputs*. An enterprise agent program has to govern *actions* — and actions are governed at the infrastructure layer, where the agent's requests cross an enforcement point the model cannot talk its way past. That reframing is the single most useful decision this guide supports: treat agent governance as a control plane wrapped around the agent, with the model treated as an untrusted (if usually well-behaved) component inside it.

If you want a shared vocabulary for the program as a whole, the NIST AI Risk Management Framework — released January 26, 2023 as a voluntary, cross-sectoral framework — organizes the work into four functions: Govern, Map, Measure, and Manage.^[5] The five control layers above are the Manage function made concrete for agents; the usage policy at the end of this guide is the Govern function's paper trail. NIST's Generative AI Profile (AI 600-1, published July 26, 2024) extends the same frame to generative systems specifically and is the companion document to hand your risk team.^[6]

3. Permissions: least privilege for a probabilistic actor

Least privilege for agents means the same thing it has always meant — grant only what the task requires — applied at a finer grain than most IAM programs are used to. The unit of permissioning is not "the agent app"; it is the tool. An agent that reads order status and drafts refund emails needs read access to the order API and send access to a mail relay with a constrained sender domain. It does not need database credentials, filesystem access, or the ability to call arbitrary HTTP endpoints, even if the framework it runs on makes those easy to hand over.

Three practices carry most of the weight. First, decompose before you grant: enumerate the tools the agent's workflows actually invoke, and provision each tool's backing credential with the narrowest scope the provider supports — a scoped API token, a role limited to specific resources, a database account confined to specific views. Second, default deny: any tool, endpoint, or action not explicitly granted is refused at the gateway, so a prompt-injected or confused agent that invents a new capability hits a wall instead of a surprise. Third, separate read from write: many agent tasks are read-heavy with a thin write step at the end, and splitting those into separately permissioned tools lets you run the risky part under tighter guardrails — including human approval — without slowing the rest.

The architectural frame for all of this is zero trust. NIST SP 800-207 (August 2020) describes zero trust as moving "defenses from static, network-based perimeters to focus on users, assets, and resources," with authentication and authorization "performed before a session to an enterprise resource is established."^[7] Agents are the purest case for that model yet: they are non-human, they act at machine speed, and their behavior is probabilistic, so nothing about their network position or past behavior should confer standing trust. Every tool call is a fresh authorization decision against the agent's identity and current policy — not a check done once at deployment.

4. Guardrails: allow lists, deny lists, and the human checkpoint

Permissions define the outer boundary of what an agent *can* do; guardrails shape what it *does* inside that boundary. The workhorse mechanism is the allow/deny rule set evaluated by the orchestrator or gateway on every proposed action: an allow list of approved tools, endpoints, and action patterns, and a deny list of patterns that are refused regardless of what else matches — destructive commands, data-export operations, calls to unknown hosts. Deny should take precedence over allow, and the rules should live in version control with review requirements, because the rule set is itself audit evidence: it is the machine-readable form of your policy.

The critical design decision is *where* the rules are enforced. Instructions in the system prompt are guidance, not enforcement — a sufficiently confused or adversarially prompted model can ignore them. Enforcement belongs in code the model does not control: middleware in the orchestration framework that inspects each tool call before execution, and an API gateway that only routes to allow-listed endpoints. Layering both gives defense in depth; the gateway catches what the middleware misses, and neither depends on the model's cooperation. Anthropic's agent guidance points the same direction, recommending sandboxed testing plus guardrails and noting that agents can "pause for human feedback at checkpoints or when encountering blockers."^[1]

That pause is the third guardrail class: human approval for high-stakes actions. Rather than a binary autonomous-or-supervised choice, tier the agent's action space. Reversible, low-blast-radius actions (read queries, draft generation, internal notifications) run autonomously. Consequential but recoverable actions (sending external email, creating tickets, modifying non-production config) run autonomously with post-hoc sampling review. Irreversible or high-blast-radius actions (payments, deletions, production changes, anything customer-visible at scale) require a named human to approve the specific proposed action — not the workflow in general — before execution. The approval event, approver identity, and the exact action approved all land in the audit trail.

Read that quote as a design constraint, not reassurance: the trust you extend should be earned per action tier, and everything above the trust line gets a checkpoint. Static lists will not catch everything — an agent can reach a harmful outcome through a chain of individually allowed actions — so mature teams pair the rule set with anomaly monitoring on action sequences and red-team the guardrails the way they would any other security control. But the lists come first, because they are cheap, deterministic, and explainable to an auditor.

5. Identity: every agent is a principal

Governance collapses without attribution, and attribution requires that each agent — ideally each agent *deployment* — has its own identity. A shared API key across five agents means five agents' actions are indistinguishable in every downstream log, one leaked credential compromises all five, and revoking access to one agent breaks the other four. The identity mechanisms are the machine-identity patterns your cloud already supports, plus an emerging agent-native layer.

Two decisions follow from the table. First, kill static secrets: wherever the platform can issue the credential — a managed identity, whose credentials "aren't even accessible to you"^[9], a service account attached to the compute resource rather than exported as a key file^[8], or an assumed role — take that over an API key in an environment variable — it removes rotation, leakage, and vault-sprawl problems in one move. Where a static key is unavoidable (most model-provider APIs), issue one key per agent, store it in a secrets manager, and rotate on a schedule. Second, record the delegation chain: an agent acting on a user's request should carry both its own identity and the initiating user's, so the audit trail can answer "which agent did this" and "for whom" as separate questions. The OAuth 2.0 client credentials grant handles the agent's own identity — it is machine-to-machine by design, restricted to confidential clients that can protect their credentials^[10] — while the on-behalf-of context travels in your application layer and must be logged explicitly.

6. Audit trails: reconstruct any action, end to end

The test of an agent audit trail is simple: given any single action the agent took, can you reconstruct — months later, from logs alone — what it did, what inputs it saw, which policy allowed it, and who or what set it in motion? Model-level observability (traces of prompts and completions) is necessary but not sufficient; the governance record also needs the enforcement events around the model. Per agent step, capture:

• Correlation ID — one trace ID threading the user request, every agent step, and every downstream call, so a single action can be walked end to end.
• Agent identity and version — which agent, which build, which prompt and policy version were live at execution time.
• Initiating principal — the human user, schedule, or upstream system on whose behalf the agent acted.
• Tool calls in full — tool name, parameters, target system, and result status for every invocation, not just the final answer.
• Policy decisions — every allow, deny, and human-approval event, including the approver's identity and the exact action approved.
• Model interaction metadata — model ID, key parameters, and token counts (which double as the cost record).
• Outputs and artifacts — what was returned, written, sent, or created, by reference if the payload is large or sensitive.
• Timestamps in UTC — consistently sourced, so distributed steps order correctly during reconstruction.

Operationally, treat the trail like any regulated log: append-only or write-once storage so records are tamper-evident, retention aligned to your regulatory obligations, access controls that separate the people operating agents from the people who can modify their logs, and masking or tokenization for personal data that transits agent context. Volume is the honest cost — logging every step of a chatty agent is not free — so tier the fidelity: full capture for write actions and policy events always, with sampling acceptable only for low-risk read traffic. The audit trail is also your operational early-warning system: the same stream feeds anomaly alerts (an agent suddenly calling a tool it has never used) and the budget dashboards in the next section.

7. Budgets and rate limits: caps you inherit, caps you build

An agent is a loop, and loops can run away — retries, recursive tool calls, or a malformed stop condition can multiply cost with no human in the path. Anthropic's guidance names the tradeoff directly: agentic systems "trade latency and cost for better task performance."^[1] The control response has two distinct instruments, and conflating them is a common gap. Rate limits bound how fast an agent can act; spend limits bound how much it can cost. You want both, because a slow leak passes every rate limit and a burst can do damage before any monthly cap trips.

Start with what the providers already enforce. Anthropic's API distinguishes exactly these two limit types: rate limits measured in requests per minute, input tokens per minute, and output tokens per minute (enforced via a token-bucket algorithm, returning a 429 with a retry-after header on breach), and monthly spend caps per usage tier — $500 on Start, $1,000 on Build, $200,000 on Scale — that pause API usage until the next month once reached, with the option to set your own lower limit and per-workspace spend and rate limits beneath the organization's.^[2] OpenAI's platform similarly meters requests and tokens per minute and per day and gates monthly usage by tier, from a $100 monthly limit at Tier 1 up to $200,000 at Tier 5.^[12] Those workspace- and key-level mechanisms are your per-agent enforcement primitive: one key or workspace per agent, with limits sized to that agent's job.

What you still build yourself is attribution and the kill switch. Attribution: tag every unit of agent spend — API key, workspace, cloud resource — with the agent's identity and owning cost center, so the finance view and the governance view reconcile. The kill switch: a documented, tested procedure (ideally one command) that revokes a specific agent's credentials and halts its runtime, because when a loop is burning money at 2 a.m. the response time that matters is yours, not the monthly cap's. Wire Google Cloud's Pub/Sub budget notifications^[3], AWS budget actions^[13], or your provider's usage webhooks into that switch so the common cases trigger it without a human.

8. The usage policy: writing it down

The written enterprise agent usage policy is what makes the four technical layers governable: it assigns owners, sets thresholds, and gives auditors and new engineers one document to read. It does not need to be long. It needs these sections, each pointing at an enforced mechanism rather than an aspiration:

1. Scope and definitions — what counts as an agent (versus a chat assistant or a script), and which environments and business units the policy covers.
2. Agent classification tiers — capability levels (read-only, supervised write, autonomous write) with the default guardrail and approval posture for each tier.
3. Identity and access requirements — one identity per agent, credential standards (short-lived over static), and the access-review cadence.
4. Approved and prohibited actions — the human-readable counterpart of the allow/deny lists, including the classes of action that always require human approval.
5. Data handling rules — what data classes agents may read and emit, masking requirements, and residency constraints.
6. Logging and retention requirements — the per-action audit fields above, storage guarantees, and retention periods.
7. Budget and limit standards — default per-agent spend caps by tier, who can raise them, and the escalation path when a cap is hit.
8. Incident response — the kill-switch procedure, notification obligations, and post-incident review requirements.
9. Ownership and review cadence — a named owner per control layer, and a scheduled policy review tied to your risk cycle.

Anchor the document to a recognized frame so it composes with the rest of your AI governance program — the NIST AI RMF's Govern function is the natural home, and mapping each policy section to a framework function shortens every subsequent audit conversation.^[5]

9. Honest objections

"This will slow us down before we've proven agents are worth anything." Partly true, and worth steelmanning: a five-layer control plane stood up before the first pilot is over-engineering. The resolution is tiering, not deferral. A read-only pilot agent needs an identity, a scoped credential, and logging — a day of work — while write access, approval workflows, and budget automation arrive with the capability that requires them. What does not work is retrofitting: an agent that shipped with a shared key and broad scopes accumulates dependencies on both, and unwinding them after an incident is far costlier than the day of setup was.

"Our existing IAM and FinOps programs already cover this." Largely right on mechanism — that is this guide's point — but wrong on defaults. Existing programs assume deterministic workloads: a service does what its code says, so provisioning-time review suffices. Agents choose actions at runtime, which is why per-action policy checks, action-tier approvals, and decision-level logging are additions, not duplications. "Determined agents will route around static lists." Also fair — chains of individually allowed actions can compose into a harmful outcome, which is why deny lists are the floor, not the ceiling: anomaly detection on action sequences and periodic red-teaming of the guardrails are the compensating controls. And "full audit logging is expensive" is simply correct; pay it for write actions and policy events, sample the rest, and treat the storage line item as part of the cost of running agents at all.

10. The read

The decision this supports: inherit enforcement wherever a provider ships it, and concentrate your build effort on the thin layer only you can own. Identity comes from your cloud's workload-identity machinery; hard spend stops come from provider tiers, workspace limits, and budget actions; rate limiting comes from the API layer. What you build is the gateway that enforces your allow/deny rules and approval tiers, the audit pipeline that records every decision with attribution, and the policy document that binds it together with named owners. Sequence it that way — identity and logging first, guardrails with the first write-capable agent, budget automation before the first production fleet — and governance becomes the thing that lets you say yes to more autonomous agents, because you can bound the blast radius of each yes.