Guardrails › Output Guardrails

Output guardrails screen the model's response before it reaches the user. They run after the model has generated text and check whether that text complies with the application's safety and quality requirements. The key motivation for output-side checking, rather than relying solely on input-side checks, is that even a properly constrained input can elicit a problematic output. An adversarial input that evades input guardrails may cause the model to generate harmful content. A benign input that happens to touch a sensitive topic may produce a response that is accurate but inappropriate for the deployment context. Output guardrails provide the final line of defense before content reaches users.

There is also a structural reason to maintain output guardrails independently of input guardrails: the mapping from inputs to outputs is not monotone in safety. A request that passes all input checks can still produce a harmful response if the model generalizes poorly, makes an error in understanding context, or produces a correct-but-inappropriate response for the deployment. Output-side checks give you a second independent chance to catch problems, with visibility into the actual content that users would receive rather than a prediction of what might be generated.

The most fundamental output check re-runs the categories from the content filtering chapter on the generated text: checking for harmful content, hate speech, sexually explicit material, instructions for dangerous activities, and other categories defined by the application's policy. The key difference from input-side filtering is that the model has now processed the input and produced text that could contain harmful content even if the input did not.

A well-aligned model rarely generates harmful content from benign inputs, but it is not perfect. Edge cases exist where benign queries produce problematic outputs. A question about historical events might elicit content that describes atrocities in more detail than appropriate. A question about chemistry might produce information about reactions that have dual-use implications. More importantly, adversarial inputs that evade input guardrails can elicit harmful outputs. Output-side safety classification provides a second chance to catch what input-side checks missed.

The classifier used for output should be calibrated for the application's risk tolerance. High-stakes applications (healthcare, education for minors, legal advice) use conservative thresholds that block borderline content. More general-purpose applications use higher thresholds to avoid over-blocking useful information. The right calibration depends on who uses the system and what harm could result from an incorrectly allowed response. A platform serving children has a fundamentally different risk tolerance than a platform for professional researchers, and both require different output safety thresholds even if they use the same underlying classifier.

One practical consideration is the cost of output-side classification relative to when it runs. Unlike input checks, output checks cannot reduce the cost of the main model call since the model has already run. Output checking adds latency on top of the model's generation time. For streaming applications where the model's response is delivered incrementally as tokens are generated, output safety classification faces a particular challenge: you cannot wait until the full response is complete before starting delivery, but classifying partial outputs introduces uncertainty about whether a response that appears safe at token 50 will remain safe at token 200. Some systems use chunk-based classification, classifying the response in rolling windows as it streams; others use a fast lightweight safety check on the full response before releasing it, sacrificing streaming delivery to maintain safety guarantees.

In retrieval-augmented generation (RAG) systems, the model's response should be grounded in the retrieved documents. If the model claims something not supported by those documents, it is hallucinating. This is a particularly serious failure mode for applications where accuracy matters, such as legal research, medical information, or technical documentation. Faithfulness guardrails check whether each claim in the model's response is supported by the retrieved context.

This is a natural language inference (NLI) problem. Given a claim $c$ from the model's response and a context $d$ composed of retrieved passages, a faithfulness classifier estimates the probability:

$$P(\text{entailed} \mid c, d)$$

where:

• $c$: a claim extracted from the model's response
• $d$: the retrieved document context
• $P(\text{entailed} \mid c, d)$: the probability that the context $d$ logically entails the claim $c$

Responses with low groundedness scores, where $P(\text{entailed} \mid c, d) < \tau_{\text{faithful}}$ for some faithfulness threshold $\tau_{\text{faithful}}$, can be flagged, modified to add uncertainty language, or blocked entirely depending on the application's requirements.

The challenge is that faithfulness checking is expensive. Running an NLI model over a full response requires decomposing the response into individual claims and checking each against retrieved passages. For a response of moderate length with 5-10 individual claims, each checked against multiple passages, this can require dozens of NLI inference calls. For production systems with strict latency requirements, faithfulness checking may only be feasible on a sample of responses, or only when the model's confidence (measured by token log probabilities) is low. A hybrid approach applies lightweight heuristics (are there any sentences in the response that cannot be traced to retrieved passages?) as a quick filter, and runs the full NLI check only on flagged responses.

The definition of "faithfulness" also requires care. A response that correctly summarizes the retrieved context is faithful. A response that correctly states something that happens to not be in the retrieved context but is independently true is technically not grounded in the context, even though it is accurate. For many RAG applications, the requirement is specifically that the model cite and rely on retrieved sources, not just that it say true things. Distinguishing between grounded accuracy (the model says something supported by context) and parametric accuracy (the model says something true from its training knowledge) requires a more context-sensitive faithfulness guardrail than simple NLI entailment.

Many applications require the model to produce structured output: JSON objects, function calls, markdown tables, CSV rows, or other formatted data. Format guardrails validate that the output matches the expected schema before passing it downstream.

Schema validation is deterministic and cheap. A JSON schema validator either accepts or rejects a string in microseconds. If the model produces malformed JSON, the guardrail can trigger a retry with a corrective prompt rather than passing the error to downstream systems. This feedback loop between guardrail and model is a key advantage of output-side validation. Rather than simply blocking a malformed response, the system can ask the model to try again with specific guidance about what went wrong: "Your response was not valid JSON. Please provide a JSON object with keys 'name' and 'value'." Models often succeed on a second attempt with this corrective prompt.

The retry strategy requires careful design to avoid infinite loops. A maximum retry count (typically two or three attempts) prevents the system from getting stuck in a cycle of failed generations. After exhausting retries, the system should fall back gracefully, either to a cached valid response, a hardcoded fallback, or an error message that gives the user useful information. The fallback behavior is as important as the retry logic because it determines what users experience when the model cannot produce valid output.

For applications using tool-calling or function-calling interfaces, output guardrails verify that called functions exist, that arguments are correctly typed, and that required parameters are present. A missing required parameter caught before function execution prevents downstream errors that would otherwise propagate through the entire system. This is especially important for tools that have side effects, such as database writes or external API calls, where a malformed call could corrupt data or trigger unintended actions. A guardrail that validates function calls before executing them is particularly valuable for agentic systems where the model takes sequences of actions; a malformed action early in a sequence can cascade into much larger failures downstream.

The model may reproduce PII from the context window in its response. If the system prompt or retrieved documents contain user information, the model might include names, email addresses, or account numbers in its response even when doing so is not necessary. This can happen inadvertently: a model summarizing a support ticket that includes a user's email might reproduce that email in its summary, even if the summary was only supposed to describe the issue, not the contact details.

Output PII redaction scans responses for personal data patterns and either removes them, replaces them with placeholders, or flags the response for review. This is separate from input-side PII detection. Input-side detection prevents user-provided PII from reaching the model. Output-side detection prevents PII from the context, training data, or model memory from reaching users who should not see it.

A subtler form of this problem arises when the model reproduces PII from its training data. Language models trained on internet-scale data may have memorized specific personal information from public sources. If a user asks about a person and the model happens to have memorized that person's contact information from training data, it might reproduce it in a response. As covered in the upcoming Memorization and Privacy chapter, this memorization behavior creates privacy risks that output-side PII detection helps mitigate.

Output PII detection is also important in multi-user systems where data from one user's session could appear in another user's response. If a RAG system's context contains documents from multiple users, or if conversation history is shared, the model might reproduce one user's personal information in a response to a different user. Access controls and context isolation are the primary defenses here, but output PII scanning provides a safety net for cases where access controls have gaps or misconfigurations.

Beyond safety, output guardrails can enforce quality standards. Responses that are too short may indicate the model refused to provide an adequate answer. A response of five words to a complex technical question is almost certainly unhelpful. Responses that are too long may indicate the model is rambling unproductively. Checking response length against expected ranges for the application catches degenerate outputs.

Toxicity scoring, sentiment analysis, and style checks can verify that responses meet tone requirements for customer-facing applications. A support bot that responds with irritation or condescension to a frustrated customer is producing technically safe but operationally harmful output; quality guardrails catch this class of problem. For branded applications, style guardrails can enforce that the model maintains the appropriate voice and terminology that the organization has defined for its products.

Repetition detection is a quality check worth implementing for generative applications. Some failure modes of language models produce highly repetitive outputs, either because the model is stuck in a generation loop or because the prompt structure inadvertently encourages repetition. A simple check for repeated phrases or repeated sentences catches these degenerate outputs before users see them. Similarly, coherence checks that verify the response's opening and closing are consistent with each other catch cases where the model's generation drifted significantly during production of a long response.

For applications that generate code, syntax validation provides a concrete quality check. Running the model's output through a language parser verifies that the generated code is syntactically valid before it is presented to the user. If the code fails to parse, the guardrail can trigger a retry or add a warning to the response. This is a cheap, deterministic check that dramatically reduces the rate of syntax-broken code reaching users in coding assistant applications.