Agent Safety › Why Agent Safety Is Different from Model Safety

Before diving into mechanisms, it is worth understanding why agent safety presents qualitatively different challenges from the alignment problems studied for language models in isolation.

A language model in a standard chat interface operates in a sandboxed loop: the user sends a message, the model generates text, and the human reads it and decides what to do next. The human remains in the loop at every step. Even if the model produces harmful output, a human can choose not to act on it. The harm is bounded by the human's willingness to follow bad advice. This is the fundamental safety property of conversational AI: there is always a human review checkpoint before any real-world consequence occurs.

An agent operates differently. It takes actions autonomously, often in long chains where each action depends on the results of the previous ones. The human may be several steps removed from the actual execution of any given action, or may not be involved at all until the task is complete. This structural difference changes the nature of safety problems in several important ways.

Irreversibility is the first concern. Many real-world actions cannot be undone. Deleted files, sent emails, API calls to payment processors, deployed code, and posted content are all difficult or impossible to reverse. A human reviewer who disagrees with a conversational AI's suggestion simply ignores the suggestion. A human reviewer who discovers that an agent already sent an email to a thousand customers faces a fundamentally different situation.

Compounding errors are the second concern. A mistake in step 3 of a 20-step plan can propagate through the remaining steps, each building on the flawed foundation. By step 20, the agent may have taken actions that collectively cause serious harm even though no single step seemed obviously wrong. This is different from the conversational setting where each interaction is largely independent. In an agentic pipeline, errors have memory.

Opacity creates a third problem. Long chains of actions in automated pipelines can be hard to inspect. If the agent does not log its reasoning, it may be impossible to understand why it took a particular action. This matters for debugging and for accountability: if an agent takes a harmful action and you cannot reconstruct its reasoning, you cannot fix the underlying problem.

Prompt injection is a fourth class of risk unique to agents. Agents that read text from external sources, including web pages and documents as well as emails, are vulnerable to adversarial content embedded in those sources that hijacks the agent's behavior. Unlike a conversational model that only processes input from a known user, an agent may process content from thousands of unknown sources, any of which could contain malicious instructions.

Capability overhang completes the picture. As agents gain access to more tools (code execution, file systems, external APIs), each new capability multiplies the surface area for potential harm. The same code execution capability that lets an agent generate data analysis scripts also lets it run malware if its behavior is hijacked.

These properties mean that building safe agents requires careful attention to system architecture as well as model behavior. Safety has to be baked into the design of the agent system from the beginning, not bolted on afterwards.