NeMo Guardrails is NVIDIA’s open-source toolkit for putting a programmable control layer between your application code and the model. It is Apache 2.0, installs with pip install nemoguardrails, and its distinguishing feature is not filtering — plenty of projects filter — but the fact that it can model the dialog itself. Most guardrail products inspect the two ends of a turn. This one can also decide what conversational move is allowed to happen next.
That extra power is the reason to look at it and the reason it costs more to run than a classifier. This page covers what the rails actually enforce, what each type can and cannot see, and how to verify a configuration rather than trust it.
Five rail types, five different vantage points
Per the project documentation, a guardrails configuration is built from five rail types, each attached to a different point in the request lifecycle. Getting the mapping right is most of the design work, because a rail can only act on what passes through its position.
| Rail type | Sees | Can do | Typical use |
|---|---|---|---|
| Input | The raw user message | Reject or rewrite it | Jailbreak and injection checks, PII masking |
| Dialog | Canonical forms of the exchange | Choose the next step, force a scripted reply, call an action | Topic control, refusal flows, scripted escalation |
| Retrieval | Chunks returned by RAG | Reject or alter a chunk before it reaches the prompt | Poisoned-document and untrusted-source handling |
| Execution | Input and output of tool calls | Gate what an action receives and returns | Constraining agent tool use |
| Output | The generated response | Reject or alter it | Leak checks, hallucination and fact checks, toxicity |
The retrieval and execution rails are the two worth pausing on, because they have no equivalent in a simple input/output filter. Indirect prompt injection arrives inside retrieved content, and consequential agent behaviour happens at the tool boundary. A guardrail layer positioned only at the endpoints of a turn is structurally blind to both, which is the same coverage argument made in Choosing an LLM Guardrail.
Colang, canonical forms, and why dialog rails are different
Rails are written in Colang, a modeling language with Python-like syntax created for describing dialogue flows. The core idea is the canonical form: user messages are normalised into a short intent label, and flows are written against those labels rather than against raw strings. A flow says, in effect, when the user does this, the bot does that.
That normalisation step is what makes dialog rails robust to paraphrase, and it is also where the runtime cost lives. Matching an arbitrary message to a canonical form uses embedding similarity against the examples in your configuration, with a model call to generate a canonical form when nothing matches closely enough. A dialog-rail-heavy configuration therefore adds vector search plus at least one extra inference to a turn, before the answer-generating call has even started.
Two versions of the language are supported. Colang 1.0 remains the default, and Colang 2.0 is available with a redesigned syntax; the repository documents both, and the practical consequence for a new project is that most examples, community configurations, and tutorials you will find are still 1.0.
A configuration is a folder, not a code change: config.yml declares the models and which rails are active, .co files hold the Colang flows, and an optional actions.py supplies custom Python actions. Application code loads it through RailsConfig.from_path() and calls LLMRails.generate() where it used to call the model. The library is async-first, with sync and async variants of the public methods.
The built-in library, and what it borrows
You do not have to write every check yourself. The bundled guardrails library covers, per the documentation, LLM self-checking for input and output moderation, fact-checking and hallucination detection, NVIDIA content-safety and topic-safety models, jailbreak and injection detection, and integrations with third-party services and community models.
Three of those deserve honest framing:
- Self-check rails are prompts, not classifiers. They ask a model whether a message should be allowed. That is flexible and it is also the weakest link in the chain, because the checking model is subject to the same injection pressure as the model it protects.
- Fact-checking rails answer a narrower question than “is this true”. They score grounding against retrieved context. Ungrounded but plausible output from a configuration without retrieval is not what they measure.
- Jailbreak detection heuristics are cheap and evadable. They belong in a stack, layered with a dedicated detector, not on their own. The comparison of purpose-built detectors is in Prompt Injection Detectors: Rebuff vs Vigil compared.
The LangChain integration is opt-in rather than automatic, enabled with an environment variable or an explicit framework call, which is a sensible default for anyone who does not want a second orchestration framework in the dependency tree.
Deployment: library or server
There are two integration shapes, and the choice has security consequences.
As a library, the rails run in your application process. Nothing crosses a network boundary, the configuration ships with your code, and the guardrail’s failure mode is your process’s failure mode.
As a server, nemoguardrails server loads one or more configurations from a folder and exposes an HTTP API, including a /v1/chat/completions endpoint that takes a config_id alongside the messages. A Dockerfile is provided in the repository. This shape suits multiple applications sharing a policy, but it introduces a service that must be authenticated and rate-limited like any other internal API — an unauthenticated guardrails server is an open proxy to whatever models it is configured with.
The CLI also carries chat for interactive sessions, evaluate for the built-in evaluation tasks, and an actions server for running custom actions out of process.
What it costs, in the currency that matters
Every active rail is work added to a turn. Self-check input and output rails are each an extra model call. Dialog rails add embedding search plus, on a miss, a generation call. Fact-checking adds another. A configuration with several rails active can multiply per-turn latency and token spend against the same user-visible response, which is why the production comparison in Llama Guard vs NeMo vs OpenAI Moderation treats latency as a first-class selection criterion rather than a footnote.
The mitigations are ordinary engineering. Use a small, fast model for check rails rather than your answer model. Put cheap deterministic checks first so expensive ones run on less traffic. Keep dialog rails for the flows that genuinely need scripted control instead of applying them globally. And measure refusals, because a rail that fires on legitimate traffic is a product defect with a measurable cost, quantified the way False Positive Cost in Refusal Systems sets out.
Verify the rails, do not trust them
A guardrail configuration is a security control, and untested security controls drift. The failure is quiet: a model upgrade changes phrasing, a canonical-form example set stops covering how real users write, a new flow bypasses a check nobody re-read.
The check is to attack the guarded endpoint rather than the bare model, and NVIDIA’s own evaluation docs do exactly that. They publish garak scan results for the ABC example bot across four configurations — no rails, general instructions in the system prompt, dialog rails, and the full guardrail set — scored as a resilience percentage per probe module.
Two rows in that table are worth more than the headline. knownbadsignatures climbs from 4.0% unprotected to 97.3% on instructions alone and 100% once rails are active, which is the result everyone expects. dan peaks at 61.3% with dialog rails and falls back to 52.7% with the full set: on that module, more rails scored worse than fewer. Rail coverage is not monotonic, guardrail layers interact, and a configuration change that helps one attack class can cost you another. The only way to know which direction yours moved is to run the scan both ways.
Reproduce that shape locally by pointing a probe suite at the guarded endpoint and comparing runs with rails on and off; the workflow is in How to Scan an LLM for Prompt Injection, and the scanner most commonly used for it is described in the garak vulnerability scanner walkthrough.
Two rails deserve targeted tests beyond a generic scan, because generic probe suites rarely reach them: feed a retrieval rail a document with embedded instructions, and drive an execution rail with an argument the tool should refuse. Both are application-specific, which means a red-team framework rather than a fixed probe list. That distinction is covered in Open Source LLM Red Teaming Tools.
When to pick it, and when not to
Pick NeMo Guardrails when the product is conversational and the policy is about conduct rather than only content: staying on topic, following a scripted escalation, refusing categories of request in a specific way, gating tool calls, and handling retrieved content as untrusted. Nothing else in the open-source guardrail field models dialog to the same degree.
Skip it when the requirement is a single content-safety verdict on a message, where a classifier such as Llama Guard is cheaper and simpler, or when the requirement is a pipeline of analysers over input and output text, where a scanner library does the job with less conceptual overhead; LLM Guard’s scanner catalogue is the reference design for that shape (archived July 2026; reference only). The layers are complementary rather than competing: a classifier can be invoked from inside a rail, and frequently should be. For a production review of NeMo Guardrails as a deployed moderation layer, see aimoderationtools.com’s NeMo Guardrails review. Broader defensive tooling in the same space is tracked at guardml.io.
Whichever you choose, place it correctly in the stack. Guardrails are the runtime layer; they seal holes rather than find them. Where each category belongs, and what none of them can catch, is mapped in LLM Security Tools: The 2026 Category Map.
