Agents Change the Threat Model

A model that answers a question has one output: text. A model that can call tools has as many outputs as the tools allow. This is an obvious statement that turns out to have non-obvious consequences, because the security properties of the two systems are not merely different in degree.

When a model only produces text, a successful prompt injection produces wrong text. That is a content problem — serious, but bounded by what the recipient does next. When the same model can issue requests, read internal sources, and write to systems, a successful injection produces actions. The bound is no longer the reader’s judgment. It is whatever the credentials permit.

Untrusted input reaches the control path

Conventional application security rests on a distinction between data and instructions. It is the distinction that SQL parameterisation enforces, that content security policy enforces, and that decades of practice have taught engineers to preserve.

Language models are built to erase it. Their entire utility is that instructions arrive as data and are acted on. A retrieved document, a web page, an email body, a filename, a code comment, or a calendar invitation all enter the same context window as the operator’s instruction, and the model has no reliable channel-level way to distinguish which of them has authority.

This means that any content the agent reads is, in effect, input to its control path. The practical scope of that statement is wider than it first appears: it includes the ticket a customer filed, the README of a dependency, the alt text of an image, and the contents of a shared drive nobody has audited in three years.

The confused deputy, at scale

The classic framing here is the confused deputy — a component with more authority than the party directing it. Agents make this the default architecture rather than an edge case.

An agent typically runs with a service identity provisioned for the union of everything it might need. It reads a mailbox, queries a database, calls an internal API, and posts to a channel. Each permission was justified individually. Together they describe a principal with broader reach than any individual user, driven by input that arrives from outside the trust boundary.

The failure does not require the model to be compromised in any deep sense. It requires only that it be persuaded, once, by content it was designed to read.

New capability creates new exposure. The exposure is usually provisioned in the same change that delivered the capability, and reviewed with less care.

What actually reduces the risk

The mitigations that hold up are architectural rather than behavioural. Instructing a model not to follow instructions found in documents is not a control; it is a request, addressed to the component under attack.

Constrain authority, not intent. The question is not what the agent is meant to do but what it is able to do. Scope credentials to the task, not the role. An agent that summarises tickets does not need write access to the ticket system, however convenient that would be later.

Separate reading from acting. Where a workflow both consumes untrusted content and takes consequential action, those stages benefit from being different principals with different permissions, and from an explicit, inspectable handoff between them.

Make consequential actions reversible or gated. Reversibility is the cheapest safety property available. Where an action cannot be undone — a payment, a deletion, an external message — the appropriate control is a gate, and the gate should be enforced by the system rather than by the agent’s cooperation.

Log the actions, not just the conversation. Many deployments capture prompts and completions and consider themselves observable. What matters during an incident is the sequence of effects: which calls were made, with what parameters, under whose identity. That record is frequently the thing that is missing.

Assume the boundary will be crossed. Injection is not a defect to be patched out; it is a property of building systems that follow instructions in data. Designs that survive assume some fraction of attempts succeed and limit what a success is worth.

The budgeting problem

The reason this tends to go wrong is not that the risks are unknown. Most of them are described in the documentation of the frameworks being used.

It is that autonomy is adopted for the capability, and the capability arrives immediately while the exposure arrives later and lands on a different team. The pilot demonstrates value in a week. The permission model that would contain it is a quarter of work with no visible output, sponsored by no one, and it is therefore deferred.

The organisations that get this right treat the permission model as part of the deliverable rather than as hardening applied afterwards. It is materially cheaper to scope authority correctly at the point the agent is built than to reconstruct what it has been permitted to do across eleven integrations two years later.