# Guardrails for RAG

> Put an 8B Nemotron safety classifier in front of your 70B model. Input, retrieval and output rails in NeMo Guardrails, with YAML configs for RAG.

Source: https://krishparekhdev.com/blog/rag-guardrails-nemotron
Published: 2026-07-22
Tags: rag, llm, ai, security, nvidia

---

Recently I have been working on a project where we had to put guardrails into a RAG pipeline, and I spent a good few weeks experimenting with NVIDIA's Nemotron safety models and [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails). NVIDIA has put together a genuinely good suite here, and very little of what I learned was written down in one place. So this is my attempt at that: what the components are and which failure modes each one covers.

If you look for RAG tutorials on YouTube, all of them show up with the same pipeline. Chunk the documents, embed them, push the vectors into a database, add a reranker, pull the top k. That is a real pipeline. I have built it several times, and it is where most tutorials end.

But the issue with that pipeline is that it only ever receives a set of queries you wrote yourself. The moment you deploy a real endpoint for a financial, medical or government department, the question stops being "are the answers good" and becomes "what will this thing refuse, and can you prove it".

A few concerns came up here, mostly because the stakeholders on the project are non-technical and do not have enough trust in the technology yet, so you have to build around that.

1. Someone talks the model into saying something it should never say, either by asking outright or by jailbreaking the system prompt.
2. People burn our compute and resources, because a query that should never have been processed gets processed anyway and starts up our agents.
3. This one I did not think about until later, and it is the one specific to RAG. The retriever does its job correctly and surfaces a chunk this particular user was not supposed to see.

On a side project all of this is fine. In healthcare, finance or government it is not. An unlicensed system giving medical advice is a compliance problem rather than a bug report, and a screenshot of your product saying something indefensible does not get corrected in the next release.

## The one idea

Do not ask your main model to police itself.

My first instinct, and I suspect most people's, was to write the rules into the system prompt. Do not give medical advice, do not discuss competitors, refuse anything unsafe. It fails, and it fails structurally rather than because the prompt was badly worded. The same model is responsible for interpreting the user's request, following the system prompt, and generating the response. If an attacker can influence that reasoning process, they influence both the answer and the policy enforcement. That is the mechanism most jailbreaks rely on. That is the mechanism most jailbreaks rely on.

Splitting the job fixes the problem. You run two models instead of one: a small classifier and your main model. Every query goes to the classifier first. Its only job is to look at that query and say whether there is a problem with it. It does not answer anything, it just labels. If it comes back clean, the query goes on to the main model. If it does not, you stop there. Running an 8B classifier on every request is cheap. Running another 70B model on every request isn't.

## Where you can intervene

NeMo Guardrails calls these rails, and there are five. They fire in a fixed order.

![Fig 1, the five rails](/images/blog/rag-guardrails-nemotron/01.webp)

_Fig 1, the five rails_

Three of them map onto the three concerns above. Content safety on input and output for the first, topic control for the second, a retrieval rail for the third. Dialog and execution rails matter later, once you have rules spanning several turns or tools that can act on the world. I would not start there.

### Concern one: the model says something it should not

Two models in config.yml, then the rails that use them:

```yaml
models:
  - type: main
    engine: nim
    model: meta/llama-3.3-70b-instruct

  - type: content_safety
    engine: nim
    model: nvidia/llama-3.1-nemotron-safety-guard-8b-v3

rails:
  input:
    flows:
      - content safety check input $model=content_safety
  output:
    flows:
      - content safety check output $model=content_safety
```

Most tutorials still reference llama-3.1-nemoguard-8b-content-safety. The current model is [llama-3.1-nemotron-safety-guard-8b-v3](https://huggingface.co/nvidia/llama-3.1-nemotron-safety-guard-8b-v3), released 28 October 2025. Use the v3 name in new configs and ignore the older name wherever you see it.

Check the output, not just the input. Input-only is the common shortcut and I took it at first. It protects you from the user while leaving you completely exposed to your own system. The query can be entirely reasonable and the answer still wrong, either because the retriever surfaced something unhelpful or the generation went somewhere it should not have.

The model returns a verdict plus the categories that were violated:

```json
{
  "User Safety": "unsafe",
  "Response Safety": "safe",
  "Safety Categories": "Criminal Planning, Weapons"
}
```

Log the category list, not just the boolean. This one matters more than it looks, and it comes straight back to the non-technical stakeholders. You will be asked on what basis a request was refused, and probably by someone in compliance rather than someone in engineering.

### Concern two: queries we were never going to answer

Content safety asks whether something is harmful. Topic control asks whether it is your business. Different questions, and they fail in different directions, which is the bit I got wrong initially. "Should I stop taking my medication?" is not unsafe content in the moderation sense at all, so content safety passes it happily. It is still a question an insurance assistant must never answer.

Topic control is not a classifier over fixed labels. You write the scope in the prompt, in plain rules:

```yaml
prompts:
  - task: topic_safety_check_input $model=topic_control
    content: |
      You are a customer service agent for an insurance provider. Answer only from the
      knowledge base and follow these guidelines:
      - Do not give medical, legal, or financial advice.
      - Do not discuss products or services from other companies.
      - Do not answer questions unrelated to company policies.
```

The flow appends the output restriction itself, forcing the model to return on-topic or off-topic. I would add this rail early on any RAG system, because a RAG system always has a scope by definition. It was built over a specific corpus for a specific purpose. Topic control is just where you write that down once instead of restating it in every prompt.

Now the compute part, which is why this rail runs where it runs. An input rail fires before anything else. If it refuses, you never embed the query, never hit the vector store, never start an agent, never spend a token on the large model. Put the same check after retrieval and you have paid for all of it before deciding you were never going to answer. On an agentic pipeline that is worse than it sounds, because by then you may have made several tool calls, and some of those have side effects you cannot take back.

![Fig 2, cost of a refused request](/images/blog/rag-guardrails-nemotron/02.webp)

_Fig 2, cost of a refused request_

Refusing early is cheaper than refusing late, and the gap widens with every agent and tool call you add.

### Concern three: retrieval surfaces the wrong document

This is the rail I did not know existed until I read the source

A retrieval rail runs after your retriever returns chunks and before those chunks reach the prompt. In a standard NeMo Guardrails pipeline, this is the only rail that inspects retrieved chunks before they become prompt context.

```yaml
rails:
  retrieval:
    flows:
      - check retrieval sensitive data
```

Input filtering does not help, because the query was fine. Output filtering is late and lossy, because by then the content has been paraphrased into an answer and you are pattern-matching on prose. If your index holds anything not uniformly visible to every user, this is the rail that matters.

Retrieval rails are not a replacement for authorization. They inspect retrieved content, but sensitive documents should still be filtered by identity and permissions before retrieval whenever possible.

### When a single turn is not enough

Everything so far judges one message at a time. Some rules are about the conversation, and that is what dialog rails are for. They are written in Colang, which reads close enough to Python that I could follow it on sight:

```colang
define user ask for medical advice
  "should I stop taking my medication"
  "is this covered if I have a heart condition"

define bot refuse medical advice
  "I can't advise on medical questions. Please speak to a clinician."

define flow
  user ask for medical advice
  bot refuse medical advice
```

The part worth understanding is that a flow matches on the canonical form of what the user meant, not the words they typed. The runtime resolves the raw utterance into an intent first, then matches the flow against that intent. Which is why a dialog rail survives paraphrasing where a keyword filter falls over. You define the intent, not the phrasings.

LLMRails is the engine underneath all of it, and it is event-driven. A message becomes an event, and the runtime moves that event through the active flows and rails.

One warning, because Colang looks like configuration and is not. It is code. Colang can call Python, so a .co file is executable. Do not load one from a source you have not read.

### What it costs

Latency. You have added a model call in front of a model call, on a pipeline that already spends time embedding the query and hitting the vector store.

I should be straight about the limits of what I can tell you here: I did not benchmark this properly, so I have no measured p50 to give you. What I can say is that the default makes it worse than it needs to be, because rails run sequentially. Three input rails cost you three round trips. One line changes that:

```yaml
rails:
  input:
    parallel: True
    flows:
      - content safety check input $model=content_safety
      - topic safety check input $model=topic_control
```

![Fig 3, sequential vs parallel](/images/blog/rag-guardrails-nemotron/03.webp)

_Fig 3, sequential vs parallel_

Parallel is right for rails that make network calls, which the model-backed ones all do. It is wrong for rails that rewrite the message, PII masking being the obvious one, because two rails mutating the same input concurrently is a race. Keep those sequential.

The trade also runs both ways, which is easy to miss. The rail adds latency to requests you answer and removes it from requests you refuse, along with all the retrieval and generation cost sitting behind them.

### Practical takeaway

Practical takeaway: start with content safety on input and output. It is a few lines of config and it covers the failure that actually gets escalated to someone senior. Add topic control next, since your RAG system already has a scope and this is where you state it. Add a retrieval rail if your index holds anything not everyone should see, and do it before somebody asks you whether it does. Leave dialog rails until you have a rule that genuinely spans turns, because they are the most work to write and maintain. Set parallel: True as soon as you have two rails, and keep it off for anything that rewrites the message.

If I could go back and reorder my own work on that project, I would do all of this before touching chunk size again. These rails decide whether the thing can go in front of real users in an industry that will audit it.

### References I used

- NeMo Guardrails repository and documentation source: [https://github.com/NVIDIA/NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails)
- NeMo Guardrails documentation: [https://docs.nvidia.com/nemo/guardrails/](https://docs.nvidia.com/nemo/guardrails/)
- Llama-3.1-Nemotron-Safety-Guard-8B-v3 model card: [https://huggingface.co/nvidia/llama-3.1-nemotron-safety-guard-8b-v3](https://huggingface.co/nvidia/llama-3.1-nemotron-safety-guard-8b-v3)

---

_Originally published in [Depth First](https://www.linkedin.com/newsletters/depth-first-7485533019647148032/), my LinkedIn newsletter. [Read it on LinkedIn](https://www.linkedin.com/pulse/guardrails-rag-krish-parekh-prozc)._