23Safety, Guardrails, and Moderation

Everything in Part V made the model more capable: a system prompt gave it a role (Chapter 18), tools let it act (Chapter 19), retrieval fed it fresh knowledge (Chapter 21), and agent loops let it pursue goals over many steps (Chapter 22). Each of those additions is also an attack surface. This closing chapter is about the other half of the harness: the machinery that keeps a helpful model from being turned into a harmful one. The single most important idea is that safety is not a feature you switch on but a stack you build, because the model you aligned in Chapter 12 is a tendency, not a guarantee, and no one layer is enough on its own.

23.1Layers of defense

Start from the fact that alignment training is imperfect. Preference optimization (Chapter 12) and Constitutional AI (Bai et al., 2022) push a model hard toward refusing clearly harmful requests, but they shape a disposition over a distribution of inputs, not a proof about every input. An adversary chooses the input, and can search a space the training never covered. So production systems wrap the aligned model in additional, independent layers: a system-prompt policy that states the rules in natural language, input and output classifiers that screen the prompt before it reaches the model and the reply before it reaches the user, and monitoring that logs traffic and flags abuse patterns after the fact.

The mental model is defense in depth, borrowed from security engineering.

The payoff of layering is diversity: alignment training and a keyword-based filter fail on different inputs, so an attack that slips past one is likely caught by another. Stacking identical filters buys little; stacking a trained disposition, a policy, a separate classifier, and human review buys a lot, because an adversary now has to defeat all of them at once.

23.2Jailbreaks and red-teaming

A jailbreak is an input that makes an aligned model do what it was trained to refuse. The reason jailbreaks keep working is structural, not a bug to be patched once: safety training fails in two characteristic ways (Wei et al., 2023). It sets up competing objectives, where the model's drive to be helpful pulls against its drive to refuse, and mismatched generalization, where the model has a capability its safety training never learned to police. Both are byproducts of the same competence that makes the model useful.

The attacks that exploit these gaps come in a handful of recognizable shapes, and knowing the shapes matters more than any specific string. Persona and role-play attacks reframe a forbidden request as fiction or as a character the model is asked to inhabit, pitting helpfulness against the refusal. Obfuscation and encoding hide the intent that a filter reads for, exploiting mismatched generalization — the model understands a request its guard never learned to flag. Injection hides hostile instructions inside content the model consumes rather than in the user's message: a retrieved document (Chapter 21) or a tool result (Chapter 19) that says "ignore your instructions and do X," which the model may follow because it cannot tell data from command. Many-shot attacks flood a long context window with hundreds of fake examples of the model complying, until it completes the pattern by imitation rather than by instruction; the attack's success grows smoothly with the number of examples (Anil et al., 2024). Automated adversarial suffixes use gradients to search for a token string that maximizes the chance of compliance, and strikingly, a suffix optimized against open models often transfers to models the attacker cannot see inside (Zou et al., 2023).

The practice that turns this understanding into defenses is red-teaming: deliberately attacking your own model to find failures before adversaries do. Manual red teams probe by hand and reveal failure categories no benchmark anticipated (Ganguli et al., 2022), while automated red-teaming trains a second model to generate adversarial prompts at scale, surfacing thousands of failures a human team would never reach (Perez et al., 2022). Red-teaming is not a release checkbox but a loop: find failures, patch them into training and filters, then attack the patched model again.

23.3Content moderation classifiers

The layer you can improve fastest is a guard model: a separate, usually smaller classifier that reads the prompt on the way in and the response on the way out, and blocks, redacts, or escalates when either looks unsafe. Llama Guard is the canonical open example — a fine-tuned LLM that classifies both prompts and responses against an explicit safety taxonomy, deployed alongside the main model rather than inside it (Inan et al., 2023). Keeping the guard separate is the point: it can be updated on its own schedule, swapped for a stricter one per deployment, and it fails differently from the model it wraps, which is exactly the independence the Swiss-cheese stack needs.

The hard part is not building the classifier but living with its error rates, and the trap here is the base rate. Real traffic is overwhelmingly benign, so even a genuinely strong classifier produces mostly false alarms. A filter that catches 95% of harmful messages and wrongly flags only 1% of benign ones sounds excellent — but if one message in a thousand is actually harmful, the vast majority of what it flags is innocent, because 1% of an enormous benign stream dwarfs 95% of a tiny harmful one.

The other costs are latency and money. An output guard cannot judge a response until the response exists, so a naive design makes the user wait for the model and then the guard in series; streaming interfaces buffer or screen incrementally to hide this. And you are now running at least two models per turn, so the guard is deliberately kept small.

23.4Governance and the harness's responsibility

Every layer so far has a dial, and turning them all toward maximum safety has a cost that shows up as over-refusal: the model declines requests that were perfectly fine. Over-refusal is not a minor annoyance but the direct price of harmlessness, and it is measurable — a model that refuses "how do I kill a Python process" because it pattern-matched on "kill" has failed its user as surely as one that answered a genuinely dangerous question. The tension is fundamental, and the original RLHF work named it: helpfulness and harmlessness are partly in conflict, and you are always choosing a point on the frontier between them (Bai et al., 2022).

Where does that value judgment live? Not in the model's weights alone. It is set by provider policy — the written rules a lab publishes about what its models will and won't do — and enforced through the layers of this chapter: the disposition baked in by alignment, the system prompt written for a deployment, the classifiers tuned for a context. This is why the same base model behaves differently across products: a coding assistant, a medical tool, and a children's app draw the helpful-harmless line in different places, and they do it in the harness, not by retraining.

That returns us to where the chapter began: safety is not a property of the model but of the whole system around it, and it is never finished, because the adversary adapts. The job of the harness is to make the residual path through the cheese as narrow as it can be, to measure the helpfulness it spends doing so, and to keep watching — which is why the next part of the book turns from building these systems to evaluating them.