Agent Safety › Safe Agent Design Patterns
Language AI HandbookAll chapters Agent Safety › Safe Agent Design PatternsOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The individual mechanisms covered so far, constraints and sandboxing together with monitoring and injection defense, are more effective when the agent system is architected with safety in mind from the beginning. Several design patterns support this goal.
The Minimal Footprint Principle
Design agents to accomplish their tasks with the smallest possible footprint: the fewest actions, the least data accessed, the shortest session duration, and the minimum number of side effects. An agent that achieves a goal by reading three files and writing one is safer than one that reads twenty files, modifies several, and spins up a subprocess.
Minimal footprint is partly about permissions (as discussed earlier) but also about the agent's own behavior. An agent can be granted broad permissions but trained or instructed to use only the minimum necessary. Prompt engineering that explicitly instructs the agent to "take the minimum number of actions required to complete the task" and "avoid accessing data outside the immediate task scope" can reduce footprint compared to an agent given the same tools but no such instructions.
Fine-tuning can reinforce minimal footprint as a preference rather than just an instruction. An agent fine-tuned on demonstrations that consistently use minimal tool calls will generalize that preference to novel tasks, even without explicit instructions in the prompt. This is analogous to how RLHF (which we will cover in later chapters) trains models to exhibit certain behaviors consistently rather than requiring those behaviors to be specified every time.
The minimal footprint principle also applies to data retention. An agent that caches data from one session and reuses it in the next accumulates state that can become stale, incorrect, or a security liability. Agents that start each session fresh and discard data that is no longer needed for the immediate task are easier to reason about and less likely to carry forward problems from one session to the next.
Checkpoint and Recovery
Long-running agents should checkpoint their state at regular intervals, saving enough information to resume from a known-good point rather than starting over. This is especially important when later steps depend on the results of earlier ones. If an error occurs in step 15 of 20, a checkpoint at step 10 allows recovery without rerunning the first ten steps.
Checkpointing serves two safety purposes. First, it makes agents more recoverable: when something goes wrong (and eventually something will), the damage can be limited to the work done since the last checkpoint rather than requiring a complete restart. Second, checkpoints create a record of what state the agent was in at each point in time, which is valuable for post-hoc investigation of incidents.
@dataclass
class Checkpoint:
step_index: int
state: Dict[str, Any]
timestamp: float
session_id: str
class CheckpointManager:
def __init__(self, checkpoint_dir: str, session_id: str):
self.checkpoint_dir = checkpoint_dir
self.session_id = session_id
self.checkpoints: List[Checkpoint] = []
def save(self, step_index: int, state: Dict[str, Any]) -> str:
checkpoint = Checkpoint(
step_index=step_index,
state=state,
timestamp=time.time(),
session_id=self.session_id,
)
self.checkpoints.append(checkpoint)
checkpoint_id = f"checkpoint_{step_index:04d}"
return checkpoint_id
def latest(self) -> Optional[Checkpoint]:
return self.checkpoints[-1] if self.checkpoints else None
def at_step(self, step_index: int) -> Optional[Checkpoint]:
matches = [c for c in self.checkpoints if c.step_index == step_index]
return matches[-1] if matches else NoneCheckpoint saved at step 2: checkpoint_0002 Checkpoint saved at step 5: checkpoint_0005 Checkpoint saved at step 8: checkpoint_0008 Latest checkpoint: step 8 State at latest checkpoint: 9 documents processed Recovery from step 9 saves 1 steps
In this example, if the agent fails at step 9, it can recover from the checkpoint at step 8, rerunning only the final step rather than the entire ten-step task. The cost of checkpointing (serializing state to disk or a key-value store) is typically small relative to the cost of rerunning work or investigating an incident without state records.
The frequency of checkpointing involves a similar tradeoff to backup frequency in databases. Checkpointing every step provides maximum recoverability but has overhead. Checkpointing every ten steps reduces overhead but means potentially rerunning up to nine steps on failure. For most agent tasks, checkpointing after each logically complete substage (after retrieving data, after processing it, after writing results) is a natural and effective cadence.
Dual-Layer Architecture
A powerful pattern for high-stakes agents is the dual-layer architecture: an inner agent that proposes actions and an outer safety layer that validates and logs them before execution. The inner agent focuses entirely on reasoning and task completion; the outer layer enforces all safety constraints independently.
This separation has several important properties. First, it keeps safety logic independent of task logic, so you can update safety rules without modifying the agent's core reasoning and vice versa. Second, it makes the safety layer auditable: because it sits between the inner agent and the execution layer, every action that reaches the execution layer has passed through the safety layer, and you can verify this independently of the inner agent's implementation. Third, if the inner agent is compromised (through a prompt injection or a model vulnerability), the outer safety layer continues to enforce its constraints.
The inner agent produces a structured action plan. The outer layer processes the plan through the validator, confirmation gates, and monitor before each action reaches the execution layer. If the safety layer blocks an action, it can provide a structured reason that the inner agent can incorporate into its next reasoning step. This creates a feedback loop where the inner agent learns, within a single session, what actions are permissible and adjusts its plan accordingly.
The chart below shows how each defense layer reduces the probability of a harmful outcome reaching the execution layer. Individually, each layer provides partial protection. Together they create a compounding reduction in risk that makes the system substantially more reliable than any single layer could achieve.
Canary Actions and Dry Run Modes
Two additional patterns are useful in practice. A dry run mode executes the agent's reasoning and planning without executing any actual tool calls, returning instead what actions the agent would have taken and with what arguments. Dry runs are valuable for testing and reviewing agent behavior in new scenarios before allowing the agent to operate live. You can verify that the agent's intended action plan is reasonable before committing to any real-world effects.
Canary actions are real but low-stakes actions that test whether a longer action sequence is likely to succeed. Before an agent executes a 20-step data migration that writes thousands of records, it might execute a canary: a single test write to verify that it has the necessary permissions, the schema is correct, and the target system is available. If the canary fails, the agent aborts before taking any significant action. If the canary succeeds, the agent proceeds with confidence that the environment is as expected.
Both patterns reflect the broader philosophy of building confirmation opportunities into long-running pipelines rather than relying entirely on real-time monitoring and after-the-fact recovery.