diff --git a/docs/configure-rails/guardrail-catalog/agentic-security.mdx b/docs/configure-rails/guardrail-catalog/agentic-security.mdx index 03135e409d..f64c52aef5 100644 --- a/docs/configure-rails/guardrail-catalog/agentic-security.mdx +++ b/docs/configure-rails/guardrail-catalog/agentic-security.mdx @@ -131,3 +131,69 @@ Before you begin, install the `yara-python` package or you can install the NeMo *Example Output* + +## Context Bloat Detection + +Context bloat detection guards against context-manipulation attacks. +Attacker-controlled content can be padded, oversized, or repetitively structured to push the system prompt out of the model's attention or exhaust the token budget. +The rail applies to user input and retrieved knowledge-base chunks, the two sources of text an agent does not author. + +The rail is local and model-free. +It does not call an external service and measures text against four thresholds: + +| Check | Configuration Field | Default | What It Catches | +|---|---|---|---| +| Size cap | `max_chars` | `5000` | Oversized input intended to exhaust the token budget | +| Shannon entropy floor | `min_entropy` | `3.5` | Low-information padding. English prose runs about 4.0 to 4.5 bits per character | +| Repeated n-gram ratio | `max_repetition_ratio` | `0.4` | Padding built from a repeated phrase | +| Longest single-character run | `max_run_ratio` | `0.1` | Runs such as `AAAAAAA...` or long whitespace padding | + +Texts shorter than `min_chars` (default `50`) are checked against the size cap only, because entropy and repetition are not meaningful on a short string. + +### Configuring Context Bloat Detection + +Add the flow for each direction you want to guard, and set the thresholds under `rails.config.context_bloat_detection`. +All configuration fields are optional. + +```yaml +rails: + config: + context_bloat_detection: + max_chars: 5000 + min_chars: 50 + min_entropy: 3.5 + max_repetition_ratio: 0.4 + ngram_size: 3 + max_run_ratio: 0.1 + action: reject + + input: + flows: + - context bloat detection on input + + retrieval: + flows: + - context bloat detection on retrieval +``` + +The `action` field selects what happens when a check detects a violation: + +| Value | Behavior | +|---|---| +| `reject` | Block the turn and return a refusal message. This is the default and recommended setting for production. | +| `truncate` | Truncate the text to `max_chars` when the size cap is exceeded, and reject when any other check detects a violation. | +| `warn` | Log the detection only. Do not modify or block the text. | + + + With `action: truncate`, the rail rewrites the text it checked rather than blocking it. + This rewrite affects how the engines schedule the rail. + Refer to [Rail Engine Support](/reference/rail-engine-support) for details. + + +### Engine Support + +`context bloat detection on input` runs on both `LLMRails` and `IORails`. + +`context bloat detection on retrieval` runs on `LLMRails` only, because `IORails` has no retrieval pipeline. +A configuration that declares a `rails.retrieval` section routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. diff --git a/docs/configure-rails/guardrail-catalog/community/alignscore.mdx b/docs/configure-rails/guardrail-catalog/community/alignscore.mdx index d9a03562ba..0b4c997c5c 100644 --- a/docs/configure-rails/guardrail-catalog/community/alignscore.mdx +++ b/docs/configure-rails/guardrail-catalog/community/alignscore.mdx @@ -92,3 +92,13 @@ Installing AlignScore is not supported on Python 3.11. ``` By default, the AlignScore server listens on port `5000`. You can change the port using the `--port` option. By default, the AlignScore server loads only the base model. You can load only the large model using `--models=large` or both models using `--models=base --models=large`. + +## Engine Support + +`alignscore check facts` runs on `LLMRails` only. + +The rail compares the bot response against `relevant_chunks`, a conversation value that the Colang retrieval pipeline builds and `IORails` does not. +A configuration that uses this rail routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/auto-align.mdx b/docs/configure-rails/guardrail-catalog/community/auto-align.mdx index ca714863a7..62f950324f 100644 --- a/docs/configure-rails/guardrail-catalog/community/auto-align.mdx +++ b/docs/configure-rails/guardrail-catalog/community/auto-align.mdx @@ -716,3 +716,21 @@ Specify the fact_check_endpoint to the correct AutoAlign environment. Then set to the corresponding subflows for fact check guardrail. The output of the fact check endpoint provides you with a fact check score that combines the factual correctness of various statements made by the bot response. Then provided with a user set threshold, will log a warning if the bot response is determined to be factually incorrect + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `autoalign check input` | ✓ | ✓ | +| `autoalign check output` | ✓ | ✓ | +| `autoalign factcheck output` | ✓ | ✓ | +| `autoalign groundedness output` | ✓ | ✗ | + +`autoalign groundedness output` reads `relevant_chunks_sep`, a conversation value that the Colang retrieval pipeline builds and `IORails` does not. +A configuration that uses it routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +`autoalign check input` and `autoalign check output` can rewrite the text they check when AutoAlign returns redacted content. +`IORails` runs input and output rails sequentially when a configured rail rewrites content. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/fiddler.mdx b/docs/configure-rails/guardrail-catalog/community/fiddler.mdx index b902b7d67e..d1dbf1e41f 100644 --- a/docs/configure-rails/guardrail-catalog/community/fiddler.mdx +++ b/docs/configure-rails/guardrail-catalog/community/fiddler.mdx @@ -50,3 +50,17 @@ Fiddler Guardrails will not block inputs in the event of any API failure. ## Notes For more information about Fiddler Guardrails, please visit the Fiddler Guardrails [documentation](https://docs.fiddler.ai/protection/guardrails). + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `fiddler user safety` | ✓ | ✓ | +| `fiddler bot safety` | ✓ | ✓ | +| `fiddler bot faithfulness` | ✓ | ✗ | + +`fiddler bot faithfulness` scores the response against `relevant_chunks`, a conversation value that the Colang retrieval pipeline builds and `IORails` does not. +A configuration that uses it routes to `LLMRails` by default, including one that also uses the two safety flows. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/gliner.mdx b/docs/configure-rails/guardrail-catalog/community/gliner.mdx index b1a260db62..95d3ec79ad 100644 --- a/docs/configure-rails/guardrail-catalog/community/gliner.mdx +++ b/docs/configure-rails/guardrail-catalog/community/gliner.mdx @@ -107,6 +107,63 @@ rails: - gliner mask pii on output ``` +### Retrieved Chunks + +Both flows also have a retrieval variant that runs against the knowledge-base chunks retrieved for a turn. +Configure the entities to look for under a `retrieval` key alongside `input` and `output`, then add one of the two retrieval flows. + +Choose the detection flow to block a turn whose retrieved chunks contain PII: + +```yaml +rails: + config: + gliner: + server_endpoint: https://integrate.api.nvidia.com/v1/chat/completions + api_key_env_var: NVIDIA_API_KEY + threshold: 0.5 + retrieval: + entities: + - first_name + - last_name + - email + - phone_number + retrieval: + flows: + - gliner detect pii on retrieval +``` + +Choose the masking flow instead to strip the PII from the chunks and let the turn proceed: + +```yaml +rails: + config: + gliner: + server_endpoint: https://integrate.api.nvidia.com/v1/chat/completions + api_key_env_var: NVIDIA_API_KEY + threshold: 0.5 + retrieval: + entities: + - first_name + - last_name + - email + - phone_number + retrieval: + flows: + - gliner mask pii on retrieval +``` + +Typically, configure only one retrieval flow. +If you configure both flows, list `gliner mask pii on retrieval` before `gliner detect pii on retrieval`. +Retrieval rails run in configuration order, so placing detection first blocks the turn before masking can make the chunks safe. + +### Engine Support + +The input and output flows run on both `LLMRails` and `IORails`. + +`gliner detect pii on retrieval` and `gliner mask pii on retrieval` run on `LLMRails` only, because `IORails` has no retrieval pipeline. +A configuration that declares a `rails.retrieval` section routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + ## Run the Guardrails Chat CLI Start an interactive chat session by pointing `--config` at the subdirectory for the flow you want to test: diff --git a/docs/configure-rails/guardrail-catalog/community/hf-classifier.mdx b/docs/configure-rails/guardrail-catalog/community/hf-classifier.mdx index 0d29b2de23..78906690d8 100644 --- a/docs/configure-rails/guardrail-catalog/community/hf-classifier.mdx +++ b/docs/configure-rails/guardrail-catalog/community/hf-classifier.mdx @@ -276,3 +276,21 @@ rails: ## HF Classifier Rail Behavior When blocked, **input and output rails** respond with `"I'm sorry, I can't respond to that."` and abort. If `enable_rails_exceptions` is set, an `InputRailException` or `OutputRailException` is raised instead. **Retrieval rails** clear all retrieved chunks if any blocked label is detected. With streaming enabled, the output rail checks the accumulated response after streaming completes. + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `hf classifier check input` | ✓ | ✓ | +| `hf classifier check output` | ✓ | ✓ | +| `hf classifier check retrieval` | ✓ | ✗ | + +`hf classifier check retrieval` clears the retrieved chunks when a blocked label is detected, which rewrites `relevant_chunks`. +`IORails` has no retrieval pipeline and nowhere to apply that rewrite, so the flow runs on `LLMRails` only. + +This rail declares `transformers` as an optional dependency for its in-process backend. +`IORails` enforces this dependency only when the configuration selects the local backend. +With a vLLM, KServe, or FMS endpoint, the rail compiles without `transformers` installed. +`LLMRails` imports it lazily and fails on the first request instead. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/patronus-evaluate-api.mdx b/docs/configure-rails/guardrail-catalog/community/patronus-evaluate-api.mdx index 0795d96cb0..fe50d9c3f7 100644 --- a/docs/configure-rails/guardrail-catalog/community/patronus-evaluate-api.mdx +++ b/docs/configure-rails/guardrail-catalog/community/patronus-evaluate-api.mdx @@ -76,3 +76,14 @@ define flow patronus api check output if not $evaluation_passed bot inform answer unknown ``` + +## Engine Support + +`patronus api check output` runs on `LLMRails` only. + +The rail sends the retrieved context alongside the question and answer. +It therefore reads `relevant_chunks`, a conversation value that the Colang retrieval pipeline builds and `IORails` does not. +A configuration that uses this rail routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/patronus-lynx.mdx b/docs/configure-rails/guardrail-catalog/community/patronus-lynx.mdx index 8a00aa332d..631ba8e8f8 100644 --- a/docs/configure-rails/guardrail-catalog/community/patronus-lynx.mdx +++ b/docs/configure-rails/guardrail-catalog/community/patronus-lynx.mdx @@ -159,3 +159,13 @@ define flow patronus lynx check output hallucination bot inform answer unknown stop ``` + +## Engine Support + +`patronus lynx check output hallucination` runs on `LLMRails` only. + +Lynx judges the response against the retrieved context, so the rail reads `relevant_chunks`, a conversation value that the Colang retrieval pipeline builds and `IORails` does not. +A configuration that uses this rail routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/polygraf.mdx b/docs/configure-rails/guardrail-catalog/community/polygraf.mdx index a6a4ce7178..9f427e2528 100644 --- a/docs/configure-rails/guardrail-catalog/community/polygraf.mdx +++ b/docs/configure-rails/guardrail-catalog/community/polygraf.mdx @@ -124,3 +124,20 @@ The integration is fail-closed: a Polygraf failure must not allow potentially-PI - **Malformed entity span** (Polygraf returns an entry without a known `entity_type`, or with non-integer offsets, or with offsets outside `0 <= start < end <= len(text)`). The actions also fail closed: detection blocks the message and masking redacts the whole payload, rather than silently skipping the malformed span and forwarding the rest. - **Default timeout**: `30` seconds per call. Slow or unreachable endpoints cannot hang the rail pipeline. - **Missing API key**: if `POLYGRAF_API_KEY` is not set, the integration logs a warning since cloud endpoints typically reject unauthenticated requests, and proceeds to call the endpoint without an `Authorization` header. + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `polygraf detect pii on input`, `polygraf detect pii on output` | ✓ | ✓ | +| `polygraf mask pii on input`, `polygraf mask pii on output` | ✓ | ✓ | +| `polygraf detect pii on retrieval`, `polygraf mask pii on retrieval` | ✓ | ✗ | + +The retrieval flows run on `LLMRails` only, because `IORails` has no retrieval pipeline. +A configuration that declares a `rails.retrieval` section routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +The masking flows rewrite the text they check rather than blocking it. +`IORails` runs input and output rails sequentially when a configured rail rewrites content, so each mask reaches subsequent rails. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/presidio.mdx b/docs/configure-rails/guardrail-catalog/community/presidio.mdx index 18feb52226..1f474eacbf 100644 --- a/docs/configure-rails/guardrail-catalog/community/presidio.mdx +++ b/docs/configure-rails/guardrail-catalog/community/presidio.mdx @@ -156,3 +156,24 @@ rails: ## Custom Detection If you want to implement a completely different sensitive data detection mechanism, you can override the default actions [`detect_sensitive_data`](https://github.com/NVIDIA-NeMo/Guardrails/tree/develop/nemoguardrails/library/sensitive_data_detection/actions.py) and [`mask_sensitive_data`](https://github.com/NVIDIA-NeMo/Guardrails/tree/develop/nemoguardrails/library/sensitive_data_detection/actions.py). + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `detect sensitive data on input`, `detect sensitive data on output` | ✓ | ✓ | +| `mask sensitive data on input`, `mask sensitive data on output` | ✓ | ✓ | +| `detect sensitive data on retrieval`, `mask sensitive data on retrieval` | ✓ | ✗ | + +The retrieval flows run on `LLMRails` only, because `IORails` has no retrieval pipeline. +A configuration that declares a `rails.retrieval` section routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +This rail declares the `sdd` extra, and the two engines report a missing install differently. +`IORails` refuses to compile the rail at startup and names the extra to install, so the configuration routes to `LLMRails` by default. +`LLMRails` imports Presidio lazily inside the action, so the failure surfaces on the first request that runs the rail. + +The masking flows rewrite the text they check rather than blocking it. +`IORails` runs input and output rails sequentially when a configured rail rewrites content, so each mask reaches subsequent rails. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/privateai.mdx b/docs/configure-rails/guardrail-catalog/community/privateai.mdx index 1816f6eec0..014fd30da2 100644 --- a/docs/configure-rails/guardrail-catalog/community/privateai.mdx +++ b/docs/configure-rails/guardrail-catalog/community/privateai.mdx @@ -93,3 +93,20 @@ If the Private AI detection API request fails, the system will assume PII is pre - The integration currently supports PII detection and masking. For more information on Private AI and its capabilities, please refer to the [Private AI documentation](https://docs.private-ai.com/?utm_medium=github&utm_campaign=nemo-guardrails). + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `detect pii on input`, `detect pii on output` | ✓ | ✓ | +| `mask pii on input`, `mask pii on output` | ✓ | ✓ | +| `detect pii on retrieval`, `mask pii on retrieval` | ✓ | ✗ | + +The retrieval flows run on `LLMRails` only, because `IORails` has no retrieval pipeline. +A configuration that declares a `rails.retrieval` section routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +The masking flows rewrite the text they check rather than blocking it. +`IORails` runs input and output rails sequentially when a configured rail rewrites content, so each mask reaches subsequent rails. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/community/regex.mdx b/docs/configure-rails/guardrail-catalog/community/regex.mdx index b2d83286d9..1b9bd06ea9 100644 --- a/docs/configure-rails/guardrail-catalog/community/regex.mdx +++ b/docs/configure-rails/guardrail-catalog/community/regex.mdx @@ -203,3 +203,16 @@ When a regex rail has a regex pattern match, the following steps happen (dependi 1. Chunks that match any retrieval pattern are **removed** from the retrieval results. Only non-matching chunks are passed to the model as context. **NOTE:** The matched pattern(s) are available in `result["detections"]` for logging and debugging. They are also logged at `INFO` level by the action. + +## Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `regex check input` | ✓ | ✓ | +| `regex check output` | ✓ | ✓ | +| `regex check retrieval` | ✓ | ✗ | + +`regex check retrieval` removes matching chunks from the retrieval results, which rewrites `relevant_chunks`. +`IORails` has no retrieval pipeline and nowhere to apply that rewrite, so the flow runs on `LLMRails` only. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/fact-checking.mdx b/docs/configure-rails/guardrail-catalog/fact-checking.mdx index f46bcb748b..15721b0d8d 100644 --- a/docs/configure-rails/guardrail-catalog/fact-checking.mdx +++ b/docs/configure-rails/guardrail-catalog/fact-checking.mdx @@ -235,3 +235,28 @@ rails: ``` For more details, check out the [Patronus Lynx Integration](/configure-guardrails/guardrail-catalog/third-party/patronus-lynx) page. + +## Engine Support + +Every rail in the following table runs on `LLMRails` only. + +| Flow | Reads | LLMRails | IORails | +|---|---|:---:|:---:| +| `self check facts` | `relevant_chunks` | ✓ | ✗ | +| `self check hallucination` | `_last_bot_prompt` | ✓ | ✗ | +| `alignscore check facts` | `relevant_chunks` | ✓ | ✗ | +| `patronus lynx check output hallucination` | `relevant_chunks` | ✓ | ✗ | + +Each of these rails judges a response against something outside the response itself, and reads a conversation value that the Colang runtime builds and `IORails` does not. + +The three grounding rails read `relevant_chunks`, the retrieved context for the turn. +`self check hallucination` works differently: it reads `_last_bot_prompt`, reruns the prompt at a high temperature, and compares the alternative generations for self-consistency. +`patronus lynx check output hallucination` grounds the response against retrieved context. +In contrast, `self check hallucination` needs no retrieval and is unavailable on `IORails` because the prompt it reruns is Colang state. + +A configuration that uses any of them routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +If you need a grounding check on `IORails`, `autoalign factcheck output` and `cleanlab trustworthiness` both run there, because they judge the response without a retrieval context. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/configure-rails/guardrail-catalog/jailbreak-protection.mdx b/docs/configure-rails/guardrail-catalog/jailbreak-protection.mdx index 858f074b90..4148691f55 100644 --- a/docs/configure-rails/guardrail-catalog/jailbreak-protection.mdx +++ b/docs/configure-rails/guardrail-catalog/jailbreak-protection.mdx @@ -116,10 +116,25 @@ When a configured detector cannot be reached, the rail allows the request instea A connection failure, a timeout, and an error response all produce the same outcome, for both the heuristics server and the jailbreak detection NIM. The rail logs the failure, allows the request, and returns no error to the caller, so a detector outage removes this protection without a visible signal. -`LLMRails` and `IORails` behave identically here. +The rail decides this itself, so it fails open on both engines rather than falling into either engine's error handling. Monitor the availability of your detector if you rely on this rail, and pair it with rails that do not call an external service. +### Engine Support + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `jailbreak detection model` | ✓ | ✓ | +| `jailbreak detection heuristics` | ✓ | ✗ | + +`jailbreak detection heuristics` shares a rail manifest with `jailbreak detection model`, and that manifest declares `torch` and `transformers` as optional dependencies. +`IORails` cannot tell from the manifest alone which of the two flows a configuration is using, and so cannot tell whether those packages are actually required. +Rather than guess, it refuses the flow, and a configuration that uses it routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. +This is tracked in [issue #2285](https://github.com/NVIDIA-NeMo/Guardrails/issues/2285). + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). + ### Setup The recommended way to use the jailbreak detection heuristics in production is to [deploy the jailbreak detection heuristics server](/more-deployment-options/using-docker#jailbreak-detection-heuristics) separately. For a hands-on configuration walkthrough, see [Use Jailbreak Detection Heuristics](/get-started/tutorials/jailbreak-detection-heuristics). diff --git a/docs/configure-rails/guardrail-catalog/pii-detection.mdx b/docs/configure-rails/guardrail-catalog/pii-detection.mdx index f3a630ed39..3d315d17a3 100644 --- a/docs/configure-rails/guardrail-catalog/pii-detection.mdx +++ b/docs/configure-rails/guardrail-catalog/pii-detection.mdx @@ -256,3 +256,23 @@ rails: ``` For more details, check out the [Private AI Integration](/configure-guardrails/guardrail-catalog/third-party/privateai) page. + +## Engine Support + +The input and output flows on this page run on both `LLMRails` and `IORails`. + +The retrieval flows run on `LLMRails` only, because `IORails` has no retrieval pipeline: + +| Flow | LLMRails | IORails | +|---|:---:|:---:| +| `gliner detect pii on retrieval`, `gliner mask pii on retrieval` | ✓ | ✗ | +| `detect sensitive data on retrieval`, `mask sensitive data on retrieval` | ✓ | ✗ | +| `detect pii on retrieval`, `mask pii on retrieval` | ✓ | ✗ | + +A configuration that declares a `rails.retrieval` section routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. + +The masking flows rewrite the text they check rather than blocking it. +`IORails` runs input and output rails sequentially when a configured rail rewrites content, so each mask reaches subsequent rails. + +For the full per-rail matrix, refer to [Rail Engine Support](/reference/rail-engine-support). diff --git a/docs/index.yml b/docs/index.yml index 8d2bd39bd8..74556590f6 100644 --- a/docs/index.yml +++ b/docs/index.yml @@ -155,6 +155,9 @@ navigation: - page: CrowdStrike AIDR path: configure-rails/guardrail-catalog/community/crowdstrike-aidr.mdx slug: crowdstrike-aidr + - page: F5 AI Guardrails + path: configure-rails/guardrail-catalog/community/f5-ai-guardrails.mdx + slug: f5-ai-guardrails - page: Fiddler path: configure-rails/guardrail-catalog/community/fiddler.mdx slug: fiddler @@ -586,6 +589,9 @@ navigation: - page: Engine Feature Support path: reference/engine-feature-support.mdx slug: engine-feature-support + - page: Rail Engine Support + path: reference/rail-engine-support.mdx + slug: rail-engine-support - page: Rail Manifests path: reference/rail-manifests.mdx slug: rail-manifests diff --git a/docs/observability/tracing/content-capture.mdx b/docs/observability/tracing/content-capture.mdx index 7018256f21..92afe72541 100644 --- a/docs/observability/tracing/content-capture.mdx +++ b/docs/observability/tracing/content-capture.mdx @@ -42,9 +42,11 @@ On a blocked request, the CLIENT span records the raw model response while the S Sampling parameters (`gen_ai.request.temperature`, `max_tokens`, and so on) and token-usage attributes are not content, so IORails records them on the CLIENT span whenever tracing is enabled, regardless of the content-capture setting. -A rails-only [`check_async()`](/run-guardrailed-inference/using-python-apis/check-messages) captures the same way: the checked messages land in `guardrails.request.input` and the returned content in `guardrails.request.output`, and every rail that runs records its own `guardrails.rail` attributes. +A rails-only [`check_async()`](/run-guardrailed-inference/using-python-apis/check-messages) uses the same capture behavior. +The checked messages are recorded in `guardrails.request.input`, and the returned content is recorded in `guardrails.request.output`. +Every rail that runs records its own `guardrails.rail` attributes. A check makes no main-model call, so it produces no CLIENT span for the protected model. -IORails captures LLM calls from rail actions as usual. +`IORails` captures LLM calls from rail actions as usual. The synchronous `check()` runs on a short-lived engine with tracing disabled, so it captures nothing. ## Enabling Content Capture diff --git a/docs/observability/tracing/span-reference.mdx b/docs/observability/tracing/span-reference.mdx index 2ed3feee3e..bd347502f2 100644 --- a/docs/observability/tracing/span-reference.mdx +++ b/docs/observability/tracing/span-reference.mdx @@ -226,7 +226,9 @@ guardrails.request SERVER one per generate_async(), stream_ - The main generation call is a child of `guardrails.request`. - An LLM or API call made by a rail action is a child of that `guardrails.action` span. - Streaming requests propagate trace context across the internal task boundary, so streamed spans keep the same parent. -- A [rails-only check](/run-guardrailed-inference/using-python-apis/check-messages) produces the same tree without the top-level `chat {model}` span, because the check makes no main-model call. The LLM and API calls its rail actions make still appear beneath those actions. The synchronous `check()` runs on a short-lived engine with tracing disabled and emits no spans. +- A [rails-only check](/run-guardrailed-inference/using-python-apis/check-messages) produces the same tree without the top-level `chat {model}` span because the check makes no main-model call. + LLM and API calls from rail actions still appear beneath those actions. + The synchronous `check()` runs on a short-lived engine with tracing disabled and emits no spans. Every span sets ERROR status and records the exception when one propagates through it. diff --git a/docs/reference/engine-feature-support.mdx b/docs/reference/engine-feature-support.mdx index 942827a548..3d4df14b64 100644 --- a/docs/reference/engine-feature-support.mdx +++ b/docs/reference/engine-feature-support.mdx @@ -39,8 +39,11 @@ rails = LLMRails(config) `IORails` is optimized for accelerated input and output rail inference. It includes tool-calling rails. -It runs the built-in NeMoGuard safety models (content safety, topic control, and jailbreak detection) and tool validation directly against the model endpoints, with optional parallel rail execution, admission control through an `AsyncWorkQueue`, OpenTelemetry token metrics, and optional speculative generation. -It does not run the Colang dialog runtime, retrieval, custom actions, or accept a custom LLM, and it accepts Colang 1.0 configurations only. +It runs most of the built-in guardrail catalog, including the NeMoGuard safety models and community and third-party integrations. +For each configured flow, it compiles the [rail manifest](/reference/rail-manifests) and calls the rail action directly instead of executing a Colang flow. +It also adds optional parallel rail execution, admission control through an `AsyncWorkQueue`, OpenTelemetry token metrics, and optional speculative generation. +It does not run the Colang dialog runtime, retrieval, or custom actions, and it does not accept a custom LLM. +It accepts Colang 1.0 configurations only. `IORails` has a `start()` and `stop()` lifecycle that initializes and releases the engine's model clients and work queue. `generate_async()`, `stream_async()`, and `check_async()` all call `start()` automatically. @@ -71,10 +74,13 @@ rails = Guardrails(config, require_iorails=True) - No custom `llm` is passed to the constructor. - The configuration is Colang 1.0. -- Only supported rail types and flows are configured (see [Built-in NeMoGuard safety rails](#built-in-nemoguard-safety-rails) and [Tool calling](#tool-calling)). +- The configuration uses only the `input`, `output`, `config`, `tool_input`, and `tool_output` rail sections. +- Every configured input and output flow compiles. The rail must be servable on `IORails`, its optional dependencies must be installed, and any model type it names must be declared under `models`. +- Tool flows are in their own direction and are not duplicated. Refer to [Tool Calling](#tool-calling). Otherwise the facade falls back to `LLMRails` and logs the reason. You can inspect that decision directly with `IORails.unsupported_reason(config, llm)`, which returns the human-readable fallback reason, or `None` when `IORails` can handle the config. +For the per-rail detail behind the third and fourth conditions, refer to [Rail Engine Support](/reference/rail-engine-support). ## Feature Support @@ -91,16 +97,20 @@ Legend: ✓ supported · ✗ not supported · ◐ partial (see notes). | Dialog rails | ✓ | ✗ | Require the Colang runtime, which IORails does not run | | Retrieval (RAG and knowledge base) rails | ✓ | ✗ | LLMRails only | | Execution rails (custom actions) | ✓ | ✗ | LLMRails only | -| Tool rails | ✓ | ✓ | IORails: tool-call and tool-result validation; see [Tool calling](#tool-calling) | +| Tool rails | ✓ | ✓ | IORails validates tool calls and tool results. Refer to [Tool Calling](#tool-calling). | +| Transform (mask, redact, sanitize) rails | ✓ | ✓ | IORails applies them on `user_message` and `bot_message` only | `LLMRails` runs every rail direction through the Colang runtime: input, output, dialog, retrieval, and execution (custom action) rails. Input and output rails wrap the model call, dialog rails drive multi-turn conversation flows, retrieval rails guard a knowledge base, and execution rails run custom Python actions. Execution rails govern those custom actions; validating the model's own tool calls and tool results is covered separately under [Tool calling](#tool-calling). `IORails` runs input, output, and tool rails only, and it does so without the Colang runtime. -Input rails run before the model call and output rails run after it, using a fixed set of built-in flows. +Input rails run before the model call and output rails run after it. +Instead of executing a Colang flow, `IORails` compiles each configured flow directly from its [rail manifest](/reference/rail-manifests) and calls the rail action. Dialog, retrieval, and execution rails are not available on `IORails`; configurations that use them fall back to `LLMRails`. +For the per-rail breakdown of which built-in flows each engine can run, refer to [Rail Engine Support](/reference/rail-engine-support). + ### Colang Language Support | Feature | LLMRails | IORails | Notes | @@ -117,16 +127,20 @@ A Colang 2.x configuration is a fallback condition: `Guardrails` routes it to `L | Feature | LLMRails | IORails | Notes | |---|:---:|:---:|---| -| Content safety | ✓ | ✓ | IORails: input and output | -| Topic control | ✓ | ✓ | IORails: input only | -| Jailbreak detection (NIM) | ✓ | ✓ | IORails: input only | +| Content safety | ✓ | ✓ | Input and output | +| Topic control | ✓ | ✓ | Input only, on both engines | +| Jailbreak detection (NIM) | ✓ | ✓ | `jailbreak detection model`, input only | +| Jailbreak detection (heuristics) | ✓ | ✗ | `jailbreak detection heuristics`. Tracked in [issue #2285](https://github.com/NVIDIA-NeMo/Guardrails/issues/2285). | Both engines support the built-in NeMoGuard safety models: content safety, topic control, and jailbreak detection. -On `LLMRails` these run as Colang flows and can be placed on input or output as the configuration allows. +Content safety and topic control select their model with a `$model=` parameter on the flow. +Both engines reject a flow that names a model type the configuration does not declare. -`IORails` supports a fixed set of these flows per direction. -On input it supports content safety, topic control, and jailbreak detection; on output it supports content safety only. -A topic-control or jailbreak flow on the output rail is a fallback condition. +The one difference is the local heuristics variant of jailbreak detection. +`jailbreak detection heuristics` shares a manifest with `jailbreak detection model`. +As a result, `IORails` cannot determine whether the configuration needs `torch` and `transformers` installed, and it refuses the flow. +A configuration that uses it routes to `LLMRails` by default. +With `require_iorails=True`, `Guardrails` raises `ValueError` instead of falling back. ### Tool Calling @@ -164,7 +178,8 @@ Prompt-based `LLMRails` calls return strings instead. `LLMRails` additionally exposes the event-based API (`generate_events` and `process_events`) and `explain()` for debugging. `IORails` builds its `GenerationResponse` without a Colang runtime, so the fields that depend on Colang behave differently. -It fully supports `response`, `tool_calls`, `reasoning_content`, and `llm_metadata`, and it synthesizes `log.activated_rails`, `log.llm_calls`, and `log.stats` from the record each rail leaves behind. +It fully supports `response`, `tool_calls`, `reasoning_content`, and `llm_metadata`. +It synthesizes `log.activated_rails`, `log.llm_calls`, and `log.stats` from each rail's execution record. The Colang-only options raise rather than returning empty data: `log.internal_events` and `log.colang_history` raise `NotImplementedError`, and `output_vars` and `state` raise `ValueError`. `IORails` also supports selecting individual rails by name through the rail toggles, which `LLMRails` does not. For the field-by-field comparison, refer to [Generation Options: IORails and LLMRails Differences](/run-guardrailed-inference/using-python-apis/generation-options#iorails-and-llmrails-differences). @@ -210,8 +225,8 @@ Parallel streaming output rails, where the output rail validates streamed chunks | Feature | LLMRails | IORails | Notes | |---|:---:|:---:|---| -| Parallel rail execution | ✓ | ✓ | `rails.input.parallel` / `rails.output.parallel` | -| Speculative generation | ✗ | ✓ | Input rails race generation; non-streaming only | +| Parallel rail execution | ✓ | ◐ | `rails.input.parallel` / `rails.output.parallel`. IORails ignores it alongside a transform rail. | +| Speculative generation | ✗ | ◐ | IORails races input rails with generation for non-streaming requests, except when an input rail rewrites content. | | Admission control and concurrency limits | ✗ | ✓ | `AsyncWorkQueue` plus a streaming semaphore | Both engines run multiple rails in the same direction concurrently when `rails.input.parallel` or `rails.output.parallel` is set; the first rail to block short-circuits the result. @@ -222,6 +237,12 @@ Speculative generation (`rails.input.speculative_generation`) runs input rails c For a configuration example, see [Speculative Generation](/configure-guardrails/yaml-schema/guardrails-configuration#speculative-generation). Admission control through an `AsyncWorkQueue` (and a separate semaphore for streaming) bounds the number of in-flight requests and rejects work when the queue is full. +A transform rail disables both `IORails` optimizations because a rewrite must reach the code that reads the text. +Configuring any rail that rewrites content causes both directions to run sequentially on `IORails`, so each rewrite reaches subsequent rails. +A rewriting input rail also disables speculative generation, because the model would otherwise read the text before the rewrite lands. +`IORails` emits a warning in both cases rather than silently dropping the rewrite. +Refer to [Rail Engine Support](/reference/rail-engine-support) for the rails that rewrite content. + ### Reasoning-Model Support | Feature | LLMRails | IORails | Notes | @@ -304,15 +325,31 @@ The knowledge base, embedding providers, and custom embedding or embedding-searc `IORails` does not initialize a knowledge base or embeddings; configurations that rely on retrieval route to `LLMRails`. -### Community and Third-Party Rail Catalog +### Guardrail Catalog Coverage | Feature | LLMRails | IORails | Notes | |---|:---:|:---:|---| -| Community integrations (PII, AlignScore, ActiveFence, and others) | ✓ | ✗ | Run as LLMRails actions and flows | +| Input surfaces (32 in the catalog) | ✓ | ◐ | 31 of 32 run on IORails | +| Output surfaces (35 in the catalog) | ✓ | ◐ | 28 of 35 run on IORails | +| Retrieval surfaces (11 in the catalog) | ✓ | ✗ | IORails has no retrieval pipeline | +| Rails that rewrite content (mask, redact, sanitize) | ✓ | ✓ | IORails runs them sequentially. Refer to [Parallelism and Concurrency](#parallelism-and-concurrency). | + +Every built-in rail under `nemoguardrails/library` declares its behavior in a [rail manifest](/reference/rail-manifests). +The 33 built-in rails declare 78 action-backed surfaces in total, where a surface is the flow name you list in `config.yml`. + +`LLMRails` runs all 78 through Colang flows, in both Colang 1.0 and Colang 2.x. +`IORails` runs 59 surfaces by compiling each manifest and calling the same rail action directly. +As a result, community and third-party integrations such as ActiveFence, Cleanlab, Fiddler, PII detection, and Prompt Security run on both engines. +Support is determined per surface rather than per integration. +For example, a rail can have input and output surfaces that run on `IORails` and a retrieval or groundedness surface that does not. +Check the [per-surface matrix](/reference/rail-engine-support) for the flows you configure. + +The 19 surfaces `IORails` cannot run have manifest contracts that the engine cannot satisfy. +Of these surfaces, 11 read Colang conversation state such as `relevant_chunks`, 7 rewrite retrieved chunks, and 1 is blocklisted pending [issue #2285](https://github.com/NVIDIA-NeMo/Guardrails/issues/2285). -The community and third-party integrations in the [Guardrail Catalog](/configure-guardrails/guardrail-catalog) (for example, PII detection, AlignScore, ActiveFence, Fiddler, Pangea, and others) run as `LLMRails` actions and flows. +`IORails` derives this scope from the manifests rather than from a hardcoded list, so adding a rail to the catalog needs no engine change. -`IORails` ships only the built-in NeMoGuard safety models and tool validation, so catalog integrations route to `LLMRails`. +For the per-surface matrix and the configuration-dependent refusals, refer to [Rail Engine Support](/reference/rail-engine-support). ### Server and Deployment diff --git a/docs/reference/rail-engine-support.mdx b/docs/reference/rail-engine-support.mdx new file mode 100644 index 0000000000..d4fa9f969e --- /dev/null +++ b/docs/reference/rail-engine-support.mdx @@ -0,0 +1,307 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +title: "Rail Engine Support" +sidebar-title: "Rail Engine Support" +description: "Per-rail support matrix showing which built-in guardrail flows run on the LLMRails and IORails engines in the NVIDIA NeMo Guardrails library." +keywords: ["rail support matrix", "llmrails", "iorails", "guardrail catalog", "rail surfaces", "supported rails"] +content: + type: "reference" +--- + +The NVIDIA NeMo Guardrails library ships 33 built-in rails under `nemoguardrails/library`. +Each rail declares its behavior in a [rail manifest](/reference/rail-manifests), and each manifest exposes one or more *surfaces*. +A surface is the flow name you list under `rails.input.flows`, `rails.output.flows`, or `rails.retrieval.flows` in `config.yml`. + +The 33 manifests declare 78 surfaces in total. +`LLMRails` runs all 78 through its Colang runtime, in both Colang 1.0 and Colang 2.x. +`IORails` runs 59 of them by compiling the manifest directly, without Colang. + +This page lists every surface and the engine that can run it. +For capability areas that are not tied to a specific rail, such as streaming, observability, and the generation API, refer to [Engine Feature Support](/reference/engine-feature-support). + +## How to Read This Page + +Legend: ✓ supported · ✗ not supported. + +The **Flow** column is the exact string you put in `config.yml`. +The **Notes** column records either the conversation value the rail rewrites or the reason `IORails` cannot run it. + +A ✓ in the `IORails` column means the surface is *servable*: nothing in its manifest contract puts it out of reach of the engine. +Whether a particular configuration actually runs it also depends on that configuration, because `IORails` validates models and optional dependencies at startup. +Refer to [Configuration-Dependent Refusals](#configuration-dependent-refusals). + +## Support Summary + +The following table summarizes surface support by direction: + +| Direction | Surfaces | LLMRails | IORails | +|---|---:|---:|---:| +| Input | 32 | 32 | 31 | +| Output | 35 | 35 | 28 | +| Retrieval | 11 | 11 | 0 | +| **Total** | **78** | **78** | **59** | + +`IORails` does not accept a `rails.retrieval` section at all, so every retrieval surface is an `LLMRails` capability. +A configuration that declares one routes to `LLMRails` by default. +Refer to [What Happens to an Unsupported Configuration](#what-happens-to-an-unsupported-configuration). + +## Support Matrix + +### Input Surfaces + +The following table lists the input surfaces: + +| Flow | Rail | LLMRails | IORails | Notes | +|---|---|:---:|:---:|---| +| `activefence moderation on input` | ActiveFence | ✓ | ✓ | | +| `activefence moderation on input detailed` | ActiveFence | ✓ | ✓ | | +| `autoalign check input` | AutoAlign | ✓ | ✓ | Rewrites `user_message` | +| `ai defense inspect prompt` | Cisco AI Defense | ✓ | ✓ | | +| `clavata check input` | Clavata | ✓ | ✓ | | +| `content safety check input` | Content Safety | ✓ | ✓ | | +| `context bloat detection on input` | Context Bloat Detection | ✓ | ✓ | Rewrites `user_message` | +| `crowdstrike aidr guard input` | CrowdStrike AIDR | ✓ | ✓ | Rewrites `user_message` | +| `f5 guardrails scan input` | F5 AI Guardrails | ✓ | ✓ | | +| `fiddler user safety` | Fiddler Guardrails | ✓ | ✓ | | +| `gcpnlp moderation` | GCP Text Moderation | ✓ | ✓ | | +| `gcpnlp moderation detailed` | GCP Text Moderation | ✓ | ✓ | | +| `gliner detect pii on input` | GLiNER | ✓ | ✓ | | +| `gliner mask pii on input` | GLiNER | ✓ | ✓ | Rewrites `user_message` | +| `guardrailsai check input` | Guardrails AI | ✓ | ✓ | | +| `hf classifier check input` | Hugging Face Classifier | ✓ | ✓ | | +| `jailbreak detection heuristics` | Jailbreak Detection | ✓ | ✗ | Backend ambiguity, [issue #2285](https://github.com/NVIDIA-NeMo/Guardrails/issues/2285) | +| `jailbreak detection model` | Jailbreak Detection | ✓ | ✓ | | +| `llama guard check input` | Llama Guard | ✓ | ✓ | | +| `pangea ai guard input` | Pangea AI Guard | ✓ | ✓ | Rewrites `user_message`. Deprecated rail | +| `policyai moderation on input` | PolicyAI | ✓ | ✓ | | +| `polygraf detect pii on input` | Polygraf PII Detection | ✓ | ✓ | | +| `polygraf mask pii on input` | Polygraf PII Detection | ✓ | ✓ | Rewrites `user_message` | +| `detect pii on input` | Private AI | ✓ | ✓ | | +| `mask pii on input` | Private AI | ✓ | ✓ | Rewrites `user_message` | +| `protect prompt` | Prompt Security | ✓ | ✓ | Rewrites `user_message` | +| `regex check input` | Regex Detection | ✓ | ✓ | | +| `self check input` | Self-Check Input | ✓ | ✓ | | +| `detect sensitive data on input` | Sensitive Data Detection | ✓ | ✓ | | +| `mask sensitive data on input` | Sensitive Data Detection | ✓ | ✓ | Rewrites `user_message` | +| `topic safety check input` | Topic Safety | ✓ | ✓ | | +| `trend ai guard input` | Trend Micro Vision One AI Guard | ✓ | ✓ | | + +### Output Surfaces + +The following table lists the output surfaces: + +| Flow | Rail | LLMRails | IORails | Notes | +|---|---|:---:|:---:|---| +| `activefence moderation on output` | ActiveFence | ✓ | ✓ | | +| `alignscore check facts` | AlignScore Fact Checking | ✓ | ✗ | Reads `relevant_chunks` | +| `autoalign check output` | AutoAlign | ✓ | ✓ | Rewrites `bot_message` | +| `autoalign factcheck output` | AutoAlign | ✓ | ✓ | | +| `autoalign groundedness output` | AutoAlign | ✓ | ✗ | Reads `relevant_chunks_sep` | +| `ai defense inspect response` | Cisco AI Defense | ✓ | ✓ | | +| `clavata check output` | Clavata | ✓ | ✓ | | +| `cleanlab trustworthiness` | Cleanlab | ✓ | ✓ | | +| `content safety check output` | Content Safety | ✓ | ✓ | | +| `crowdstrike aidr guard output` | CrowdStrike AIDR | ✓ | ✓ | Rewrites `bot_message` | +| `f5 guardrails scan output` | F5 AI Guardrails | ✓ | ✓ | | +| `fiddler bot faithfulness` | Fiddler Guardrails | ✓ | ✗ | Reads `relevant_chunks` | +| `fiddler bot safety` | Fiddler Guardrails | ✓ | ✓ | | +| `gliner detect pii on output` | GLiNER | ✓ | ✓ | | +| `gliner mask pii on output` | GLiNER | ✓ | ✓ | Rewrites `bot_message` | +| `guardrailsai check output` | Guardrails AI | ✓ | ✓ | | +| `self check hallucination` | Hallucination Detection | ✓ | ✗ | Reads `_last_bot_prompt` | +| `hf classifier check output` | Hugging Face Classifier | ✓ | ✓ | | +| `injection detection` | Injection Detection | ✓ | ✓ | Rewrites `bot_message` | +| `llama guard check output` | Llama Guard | ✓ | ✓ | | +| `pangea ai guard output` | Pangea AI Guard | ✓ | ✓ | Rewrites `bot_message`. Deprecated rail | +| `patronus api check output` | Patronus AI | ✓ | ✗ | Reads `relevant_chunks` | +| `patronus lynx check output hallucination` | Patronus AI | ✓ | ✗ | Reads `relevant_chunks` | +| `policyai moderation on output` | PolicyAI | ✓ | ✓ | | +| `polygraf detect pii on output` | Polygraf PII Detection | ✓ | ✓ | | +| `polygraf mask pii on output` | Polygraf PII Detection | ✓ | ✓ | Rewrites `bot_message` | +| `detect pii on output` | Private AI | ✓ | ✓ | | +| `mask pii on output` | Private AI | ✓ | ✓ | Rewrites `bot_message` | +| `protect response` | Prompt Security | ✓ | ✓ | Rewrites `bot_message` | +| `regex check output` | Regex Detection | ✓ | ✓ | | +| `self check facts` | Self-Check Facts | ✓ | ✗ | Reads `relevant_chunks` | +| `self check output` | Self-Check Output | ✓ | ✓ | | +| `detect sensitive data on output` | Sensitive Data Detection | ✓ | ✓ | | +| `mask sensitive data on output` | Sensitive Data Detection | ✓ | ✓ | Rewrites `bot_message` | +| `trend ai guard output` | Trend Micro Vision One AI Guard | ✓ | ✓ | | + +### Retrieval Surfaces + +Retrieval surfaces run against the knowledge-base chunks retrieved for a turn. +`IORails` has no retrieval pipeline, so all of them are `LLMRails` only: + +| Flow | Rail | LLMRails | IORails | Notes | +|---|---|:---:|:---:|---| +| `context bloat detection on retrieval` | Context Bloat Detection | ✓ | ✗ | Rewrites `relevant_chunks` | +| `gliner detect pii on retrieval` | GLiNER | ✓ | ✗ | Reads `relevant_chunks` | +| `gliner mask pii on retrieval` | GLiNER | ✓ | ✗ | Rewrites `relevant_chunks` | +| `hf classifier check retrieval` | Hugging Face Classifier | ✓ | ✗ | Rewrites `relevant_chunks` | +| `polygraf detect pii on retrieval` | Polygraf PII Detection | ✓ | ✗ | Reads `relevant_chunks` | +| `polygraf mask pii on retrieval` | Polygraf PII Detection | ✓ | ✗ | Rewrites `relevant_chunks` | +| `detect pii on retrieval` | Private AI | ✓ | ✗ | Reads `relevant_chunks` | +| `mask pii on retrieval` | Private AI | ✓ | ✗ | Rewrites `relevant_chunks` | +| `regex check retrieval` | Regex Detection | ✓ | ✗ | Rewrites `relevant_chunks` | +| `detect sensitive data on retrieval` | Sensitive Data Detection | ✓ | ✗ | Reads `relevant_chunks` | +| `mask sensitive data on retrieval` | Sensitive Data Detection | ✓ | ✗ | Rewrites `relevant_chunks` | + +### Tool Surfaces + +Tool rails are not manifest surfaces. +`IORails` registers them separately as local structural and schema validators that reach no model: + +| Flow | Section | LLMRails | IORails | Notes | +|---|---|:---:|:---:|---| +| `tool call validation` | `rails.tool_output.flows` | ✓ | ✓ | Validates model-emitted tool calls | +| `tool result validation` | `rails.tool_input.flows` | ✓ | ✓ | Validates application-returned tool results | + +For configuration and behavior, refer to [Tool Calling](/configure-guardrails/guardrail-catalog/tool-calling). + +## Why IORails Refuses a Surface + +`IORails` derives its scope from the manifest rather than from a hardcoded list of rail names. +A surface is unsupported when the engine cannot satisfy its declared contract. +Adding a new rail to the catalog therefore needs no change in the engine. + +Two of the per-surface reasons are structural and one is temporary: + +| Reason | Surfaces | Explanation | +|---|---:|---| +| Rewrites a value the direction cannot apply | 7 | `IORails` can apply a rewrite to `user_message` on input and `bot_message` on output. A retrieval surface that rewrites `relevant_chunks` has nowhere to put the result. | +| Reads a conversation value `IORails` does not supply | 11 | `IORails` supplies only `user_message` and `bot_message` to a manifest `Binding.context` entry. A surface that binds `relevant_chunks`, `relevant_chunks_sep`, or `_last_bot_prompt` needs Colang runtime state that the engine does not build. | +| Explicit blocklist | 1 | `jailbreak detection heuristics` shares a manifest with `jailbreak detection model`, so the engine cannot tell whether the configuration needs `torch` and `transformers` installed. Tracked in [issue #2285](https://github.com/NVIDIA-NeMo/Guardrails/issues/2285). | + +The checks run in the order the table lists them and report the first reason that applies. +As a result, a retrieval surface that both rewrites and reads `relevant_chunks` is reported under the rewrite reason. + +A third structural constraint applies to the configuration as a whole rather than to any one surface. +`IORails` accepts only the `input`, `output`, `config`, `tool_input`, and `tool_output` rail sections, so a configuration that declares `rails.retrieval`, `rails.dialog`, or any other section routes to `LLMRails` by default, regardless of which flows it names. + +The engine checks that constraint first, so it does not report a per-surface reason for a retrieval flow. +`IORails.unsupported_reason()` reports the unsupported rail section and stops, so a configuration with a `rails.retrieval` section reports `config has rails outside the IORails-supported set: ['retrieval']` rather than the rewrite or read reason its surfaces would give. +The reasons in the table above describe what each surface's manifest declares, which is what the retrieval rows of the matrix record. + +## What Happens to an Unsupported Configuration + +The `Guardrails` facade decides what an unsupported rail means for your application: + +| Constructor | Unsupported-Configuration Behavior | +|---|---| +| `Guardrails(config)` | Routes the configuration to `LLMRails` and logs the reason. This is the default | +| `Guardrails(config, require_iorails=True)` | Raises `ValueError` naming the reason, instead of falling back | +| `Guardrails(config, use_iorails=False)` | Uses `LLMRails` without consulting the rail catalog at all | + +Every "routes to `LLMRails`" statement on this page and in the guardrail catalog describes the default. +Use `require_iorails=True` when you depend on an `IORails`-only capability such as OpenTelemetry metrics and would rather fail at startup than silently lose it. + +## Configuration-Dependent Refusals + +Beyond the fixed matrix above, `IORails` validates two things per configuration when it compiles a rail at startup. +Where `LLMRails` does not, it reports the same problem at request time instead: + +| Check | IORails | LLMRails | +|---|---|---| +| Optional dependency installed | Refuses to compile the rail, naming the missing distribution and the extra to install | Imports lazily inside the action, so the failure surfaces on the first request that runs the rail | +| Model type named by `$model=` is declared in `models` | Rejected before either engine is constructed | Rejected before either engine is constructed | +| Model type fixed in the manifest is declared in `models` | Refuses to compile the rail, naming the missing model type and the types the configuration does declare | Fails when the flow executes and the action asks for the model | + +A flow that carries a `$model=` parameter is validated by `RailsConfig` rather than by an engine, so loading the configuration raises `InvalidRailsConfigurationError` on both engines. +The other two refusals come from the `IORails` compiler and raise `RailCompilationError`. +Through the `Guardrails` facade those two are fallback conditions: by default the configuration routes to `LLMRails` and the reason is logged. + +The rails that declare an optional dependency are the following: + +| Rail | Distributions | Extra | +|---|---|---| +| Cleanlab | `cleanlab-studio` | | +| GCP Text Moderation | `google-cloud-language` | | +| Guardrails AI | `guardrails-ai` | | +| Hugging Face Classifier | `transformers` | | +| Injection Detection | `yara-python` | | +| Jailbreak Detection | `torch`, `transformers` | | +| Sensitive Data Detection | `presidio-analyzer`, `presidio-anonymizer`, `spacy` | `sdd` | + +Hugging Face Classifier and Jailbreak Detection each offer an in-process backend and a remote endpoint. +`IORails` enforces the dependency only when the configuration selects the in-process backend. + +The surfaces that bind a model type are the following: + +| Flow | Model Type | Source | Validated by | +|---|---|---|---| +| `content safety check input` | Selected by `$model=` | Surface parameter | `RailsConfig`, on both engines | +| `content safety check output` | Selected by `$model=` | Surface parameter | `RailsConfig`, on both engines | +| `topic safety check input` | Selected by `$model=` | Surface parameter | `RailsConfig`, on both engines | +| `llama guard check input` | `llama_guard` | Fixed in the manifest | The `IORails` compiler only | +| `llama guard check output` | `llama_guard` | Fixed in the manifest | The `IORails` compiler only | +| `patronus lynx check output hallucination` | `patronus_lynx` | Fixed in the manifest | The `IORails` compiler only | + +`RailsConfig` reads the `$model=` parameter off the flow name, so it catches only the first three. +A manifest-fixed model type is invisible to that check, which is why the last three are an `IORails`-only refusal. + +`IORails` never reaches the model check for `patronus lynx check output hallucination`, because the surface check refuses it first. + +A manifest can also declare a model under `requirements.models`, as Self-Check and Hallucination Detection do for `llm`. +That declaration is descriptive metadata for the catalog. +Neither engine enforces it at startup, so a missing model surfaces when the rail runs. + +## Rail Behavior Differences + +The manifest and the action are shared, so both engines reach the same allow, block, or transform decision. +The following table shows how each engine acts on that decision: + +| Behavior | LLMRails | IORails | +|---|---|---| +| Block presentation | The Colang flow decides: emit a refusal intent, or raise a configured exception when `enable_rails_exceptions` is set | Returns a fixed refusal message for a non-streaming request, or a `guardrails_violation` payload for a streaming request | +| Rail action raises an exception | The Colang runtime converts ordinary action failures to an internal-error response. An `LLMCallException` propagates instead | The engine's fail-closed envelope converts it to a block, marks the outcome as failed, and returns an internal-error message. An error carrying an upstream HTTP status propagates instead | +| Transform (mask or rewrite) rails | Applied through the Colang context | Applied by rerunning the remaining rails against the rewritten text. Rewriting rails are ordered ahead of judging rails so a mask reaches the checks behind it | +| Transform rails with `parallel: true` | Honored independently | Not honored. Configuring any rewriting rail forces both directions sequential and emits a warning, because concurrent rails read the arriving text and a rewrite cannot compose | +| Transform rails with `speculative_generation: true` | Not applicable, `LLMRails` has no speculative generation | Not honored. A rewriting input rail disables speculative generation and emits a warning, because the model would read the text before the rewrite lands | +| Retrieval-direction rails | Run against retrieved chunks | Not available | + +One transform rule is shared rather than per-engine. +`RailsConfig` rejects a configuration that combines a rewriting output rail with output-rail streaming unless `rails.output.streaming.stream_first` is `false` and `context_size` is `0`, because a rewrite cannot be applied to chunks that have already been sent. +That validation runs for both engines. + +For the shared decision object that both engines read, refer to [Rail Outcomes](/configure-guardrails/actions/rail-outcomes). + +## Checking Support Programmatically + +Ask the engine whether it can serve a whole configuration: + +```python +from nemoguardrails import RailsConfig +from nemoguardrails.guardrails.iorails import IORails + +config = RailsConfig.from_path("path/to/config") + +reason = IORails.unsupported_reason(config) +if reason is None: + print("IORails can serve this configuration") +else: + print(f"Falls back to LLMRails: {reason}") +``` + +Enumerate the surfaces the installed version declares: + +```python +from nemoguardrails.manifests import RailDirection, default_rail_catalog + +catalog = default_rail_catalog() + +for (direction, name), surface in sorted(catalog.surfaces().items()): + print(f"{direction.value:10} {name}") + +input_surfaces = catalog.surfaces(direction=RailDirection.INPUT) +``` + +## Related Information + +Use the following references for related concepts and configuration: + +- [Engine Feature Support](/reference/engine-feature-support) compares the two engines on capabilities that are not rail-specific. +- [Rail Manifests](/reference/rail-manifests) documents the manifest schema that this matrix is derived from. +- [Guardrail Catalog](/configure-guardrails/guardrail-catalog) documents each rail's configuration and behavior. +- [Rail Outcomes](/configure-guardrails/actions/rail-outcomes) documents the allow, block, and transform decision that rail actions return. diff --git a/docs/run-rails/using-python-apis/check-messages.mdx b/docs/run-rails/using-python-apis/check-messages.mdx index de29b0358e..1384621e6b 100644 --- a/docs/run-rails/using-python-apis/check-messages.mdx +++ b/docs/run-rails/using-python-apis/check-messages.mdx @@ -11,7 +11,8 @@ content: The `check_async()` and `check()` methods validate messages against input and output rails without triggering full LLM generation. Use these methods instead of [generation options](/run-guardrailed-inference/using-python-apis/generation-options) when you only need to run rails without generating a response. If you need to run generation while selectively disabling rail categories, see [Generation Options: Disabling Rails](/run-guardrailed-inference/using-python-apis/generation-options#disabling-rails). -Both the `LLMRails` and `IORails` engines implement these methods, and the `Guardrails` facade forwards the call to whichever engine it selected. +Both the `LLMRails` and `IORails` engines implement these methods. +The `Guardrails` facade forwards each call to the selected engine. The method signatures and returned `RailsResult` type are the same on both engines. For engine-specific rail selection and execution behavior, refer to [Engine Differences](#engine-differences). @@ -248,22 +249,24 @@ On `LLMRails`, the blocked content is the bot message the Colang flow defines, s On `IORails`, a blocked check returns the engine's fixed refusal message, `I'm sorry, I can't respond to that.`, which the configuration cannot change. That is the same string the built-in rails use by default, so an unmodified configuration produces the same text on both engines. -`IORails` also distinguishes a rail that fired from a rail that broke. +`IORails` also distinguishes a policy block from a rail execution failure. When a rail fails to execute, the check returns `BLOCKED` with `I'm sorry, an internal error has occurred.` instead of the refusal message. This distinction prevents the application from reporting an operational failure as a content refusal. That wording matches the sentence the Colang runtime produces on `LLMRails` when an action fails. -In every case, treat `status` as the decision and `content` as the message to show a user. Do not match on the message text. +In every case, treat `status` as the decision and `content` as the message to display. Do not match on the message text. ### Missing Content for a Requested Direction `IORails` skips a direction when the messages carry no content for it, and reports `PASSED`. -An explicit `rail_types=[RailType.OUTPUT]` on a message list with no assistant text runs no output rails, and an explicit `rail_types=[RailType.INPUT]` with no user text runs no input rails. +An explicit `rail_types=[RailType.OUTPUT]` on a message list with no assistant text runs no output rails. +Similarly, `rail_types=[RailType.INPUT]` with no user text runs no input rails. This keeps a rail that requires the missing text from raising and surfacing as a false block. -A `PASSED` result from `IORails` means no rail blocked the messages, which includes the case where a requested direction had nothing to check. +A `PASSED` result from `IORails` means no rail blocked the messages. +This result includes the case where a requested direction had nothing to check. When your application requires that a rail actually ran, assert that the message you are validating is present and non-empty before calling `check_async()`. @@ -276,7 +279,8 @@ When your application requires that a rail actually ran, assert that the message The check makes no main-model call, so the tree has no top-level `chat {model}` CLIENT span. LLM calls from rail actions still appear beneath those actions. For more information, refer to [Span Reference](/observability/tracing/span-reference#iorails). -- Metrics record the check in the request-level instruments: `guardrails.requests`, `guardrails.requests.active`, `guardrails.request.duration`, `guardrails.requests.errors`, and, on a block, `guardrails.requests.blocked` with the `rail.type` label. +- Metrics record the check in `guardrails.requests`, `guardrails.requests.active`, `guardrails.request.duration`, and `guardrails.requests.errors`. + A block also updates `guardrails.requests.blocked` with the `rail.type` label. A check rejected by a full admission queue increments `guardrails.nonstream.rejections`. For more information, refer to [Metric Reference](/observability/metrics/reference). - Content capture records the checked messages in `guardrails.request.input` and the returned content in `guardrails.request.output` on the check's request span. @@ -287,7 +291,8 @@ The synchronous `check()` builds a short-lived engine with tracing and metrics d ### Guardrails Server The bundled server exposes checks through the `/v1/checks` endpoint, which calls `check_async()` on the engine serving the requested configuration. -The server resolves that engine through the top-level `LLMRails` import, so setting `NEMO_GUARDRAILS_IORAILS_ENGINE=1` makes the endpoint run on `IORails` for configurations IORails can serve, and on `LLMRails` for the rest. +The server resolves that engine through the top-level `LLMRails` import. +Setting `NEMO_GUARDRAILS_IORAILS_ENGINE=1` runs compatible configurations on `IORails` and the rest on `LLMRails`. Requesting a rail type with no configured flows returns HTTP 422. diff --git a/docs/run-rails/using-python-apis/generation-options.mdx b/docs/run-rails/using-python-apis/generation-options.mdx index b4b180ea76..bf1628fdcb 100644 --- a/docs/run-rails/using-python-apis/generation-options.mdx +++ b/docs/run-rails/using-python-apis/generation-options.mdx @@ -36,7 +36,8 @@ For the complete field-by-field comparison, refer to [IORails and LLMRails Diffe -Passing `options` to `generate()` or `generate_async()` on `IORails`, or on a `Guardrails` facade that selected `IORails`, returns a `GenerationResponse` object instead of an OpenAI-style message dictionary. +Passing `options` to `generate()` or `generate_async()` on `IORails` returns a `GenerationResponse` object instead of an OpenAI-style message dictionary. +This behavior also applies to a `Guardrails` facade that selected `IORails`. Earlier versions honored the `llm_params` and rail toggles inside `options` but always returned a message dictionary. Update code that reads `result["content"]` to read `result.response[0]["content"]`. @@ -83,7 +84,7 @@ res = rails.generate(messages=messages, options={ }) ``` -The two calls return different types, because passing `options` selects the structured `GenerationResponse` return. +The two calls return different types because passing `options` selects the structured `GenerationResponse` return. The list form is exhaustive: every category you leave out of the list is disabled. To disable one category and keep the rest, use the dictionary form instead. @@ -417,7 +418,7 @@ Use [`stream_async()`](/run-guardrailed-inference/using-python-apis/streaming) t `stream_async()` accepts `options` too, but reads only a subset of it and never yields a `GenerationResponse`. Refer to [Generation Options and Streaming](/run-guardrailed-inference/using-python-apis/streaming#generation-options-and-streaming). -Legend: ✓ supported · ✗ not supported (raises) · ◐ accepted but not populated · n/a not applicable to the engine. +Legend: ✓ supported · ✗ not supported (raises) · ◐ accepted but not populated · N/A not applicable to the engine. The following table compares generation option support for `generate()` and `generate_async()`: @@ -427,9 +428,9 @@ The following table compares generation option support for `generate()` and `gen | `rails.output` | `bool \| list[str]` | ✓ | ✓ | Same list-selection difference as `rails.input`. | | `rails.tool_input` | `bool \| list[str]` | ✓ | ✓ | Gates the tool-result rails. Same list-selection difference. | | `rails.tool_output` | `bool \| list[str]` | ✓ | ✓ | Gates the tool-call rails. Same list-selection difference. | -| `rails.dialog` | `bool` | ✓ | n/a | `IORails` runs no dialog rails and ignores the toggle. On `LLMRails`, `dialog: False` suppresses the main LLM call and echoes the supplied message back. | -| `rails.retrieval` | `bool \| list[str]` | ✓ | n/a | `IORails` runs no retrieval rails and ignores the toggle. | -| `llm_params` | `dict \| None` | ✓ | ✓ | Applied to the main model call only, never to the rail models. On `IORails` this is also how you pass tool definitions, and it is accepted by `stream_async()`. | +| `rails.dialog` | `bool` | ✓ | N/A | `IORails` runs no dialog rails and ignores the toggle. On `LLMRails`, `dialog: False` suppresses the main LLM call and echoes the supplied message back. | +| `rails.retrieval` | `bool \| list[str]` | ✓ | N/A | `IORails` runs no retrieval rails and ignores the toggle. | +| `llm_params` | `dict \| None` | ✓ | ✓ | Applied to the main model call only, never to the rail models. On `IORails`, this is also how you pass tool definitions, and it is accepted by `stream_async()`. | | `llm_output` | `bool` | ◐ | ◐ | Accepted by both. The resulting `llm_output` field is `None` on both. | | `output_vars` | `bool \| list[str] \| None` | ✓ | ✗ | `IORails` raises `ValueError`: it has no Colang context to read. | | `log` | `GenerationLogOptions` | ✓ | ◐ | Refer to [GenerationLogOptions Support](#generationlogoptions-support). | @@ -458,13 +459,13 @@ For the `IORails` log structure and how it differs from the Colang-derived log, ### GenerationResponse Field Support -The following table compares the fields returned when you pass `options` to `generate()` or `generate_async()`: `stream_async()` never yields or returns a `GenerationResponse` on either engine. +The following table compares the fields returned when you pass `options` to `generate()` or `generate_async()`: | Field | Type | LLMRails | IORails | Notes | |---|---|:---:|:---:|---| | `response` | `str \| list[dict]` | ✓ | ✓ | `IORails` always returns a one-element `[assistant_msg]` list. `LLMRails` returns a plain string when called with `prompt=` instead of `messages=`. | -| `tool_calls` | `list \| None` | ✓ | ✓ | Both use the canonical `ToolCall.to_dict()` shape with dictionary arguments. On IORails, the bare-message and streaming paths use the OpenAI wire shape with `function.arguments` as a JSON string. | +| `tool_calls` | `list \| None` | ✓ | ✓ | `IORails` uses the canonical `ToolCall.to_dict()` shape with dictionary arguments. `LLMRails` can preserve a supplied assistant tool call in the OpenAI wire shape, including JSON-string arguments, when dialog rails are disabled. The bare-message and streaming paths on `IORails` also use the OpenAI wire shape. | | `reasoning_content` | `str \| None` | ✓ | ✓ | Provider `reasoning` field, or content extracted from `` tags. On this path `IORails` keeps the message content clean. The bare dictionary returned without `options` inlines reasoning as a `` prefix. No streamed equivalent exists on either engine. | | `log` | `GenerationLog \| None` | ✓ | ◐ | `IORails` populates `activated_rails`, `llm_calls`, and `stats` only. Refer to [Log Options on IORails](#log-options-on-iorails). | | `llm_metadata` | `dict \| None` | ✓ | ◐ | Provider metadata blob verbatim, such as `response_headers`. `IORails` reports the main model call. `LLMRails` reports the most recent call, usually the last output rail. The library does not add normalized usage here. Read token counts from `log`. | @@ -476,10 +477,11 @@ Reasoning bypasses the output rails on both engines, so output rails check the f ### Requests Stopped by Rails -When a rail blocks content, both engines return the configured refusal message in `response`. -On `IORails`, a rail execution failure returns the internal-error message instead of the refusal message. -In either `IORails` case, `tool_calls`, `reasoning_content`, and `llm_metadata` are `None`, and `log` holds the records through the rail that stopped processing. -In `log.activated_rails`, that rail has `stop` set to `True`, and its action's `return_value["failed"]` value distinguishes a failure from a policy block. +When a rail blocks content, `LLMRails` returns the configured refusal message in `response`. +`IORails` returns its fixed refusal message for a policy block and an internal-error message for a rail execution failure. +In either `IORails` case, `tool_calls`, `reasoning_content`, and `llm_metadata` are `None`. +To inspect records through the rail that stopped processing, request `log.activated_rails`. +That rail has `stop` set to `True`, and its action's `return_value["failed"]` value distinguishes a failure from a policy block. ## Limitations diff --git a/docs/run-rails/using-python-apis/overview.mdx b/docs/run-rails/using-python-apis/overview.mdx index abf5fdcd75..97e1345ae3 100644 --- a/docs/run-rails/using-python-apis/overview.mdx +++ b/docs/run-rails/using-python-apis/overview.mdx @@ -89,7 +89,7 @@ def handler(event, context): | ----------------------------------------------- | ----------------------------------------------------- | | `generate()` / `generate_async()` | Standard chat interactions with messages | | `stream_async()` | Real-time token streaming | -| `check()` / `check_async()` | Validating messages against rails, without generation | +| `check()` / `check_async()` | Validate messages against rails without generation | | `generate_events()` / `generate_events_async()` | Low-level event control for custom integrations | ## Synchronous vs Asynchronous @@ -105,6 +105,6 @@ The NeMo Guardrails library provides both synchronous and asynchronous methods: -Use asynchronous methods (`generate_async`, `stream_async`, `check_async`) in async contexts for better performance. The synchronous `generate()` and `check()` methods cannot be called from within an async context. +Use asynchronous methods (`generate_async()`, `stream_async()`, `check_async()`) in async contexts for better performance. The synchronous `generate()` and `check()` methods cannot be called from within an async context. diff --git a/docs/run-rails/using-python-apis/streaming.mdx b/docs/run-rails/using-python-apis/streaming.mdx index 02350ec31c..5e5bf8d3b7 100644 --- a/docs/run-rails/using-python-apis/streaming.mdx +++ b/docs/run-rails/using-python-apis/streaming.mdx @@ -157,7 +157,7 @@ async for chunk in rails.stream_async(messages=messages, include_metadata=True): With `include_metadata=True`, each chunk is a `dict` with a mandatory `"text"` key. A chunk that carries metadata also has a `"metadata"` key, which can hold the following: -- `provider_metadata`: non-standard fields the provider returned with the chunk, such as `response_headers` and vendor extensions. +- `provider_metadata`: nonstandard fields the provider returned with the chunk, such as `response_headers` and vendor extensions. - `usage`: normalized token counts as `input_tokens`, `output_tokens`, and `total_tokens`. For text responses, the stream ends with a single terminal frame that has empty text. @@ -180,7 +180,7 @@ For example, an `IORails` stream with a final usage-only provider chunk has this The empty terminal frame always includes the `response_metadata` and `usage_metadata` keys. They are retained for backward compatibility, and the internal `stream_async()` paths leave both as `None`. Read token counts from `usage` instead. -A caller driving `StreamingHandler` directly can still push these keys, and the handler accumulates them onto the terminal frame. +If you use `StreamingHandler` directly, the handler can still push these keys and accumulate them onto the terminal frame. An `IORails` tool-call response adds an assembled tool-call JSON frame after this empty frame, so the tool-call frame is the final frame in that stream. @@ -197,7 +197,8 @@ The `include_generation_metadata` parameter is deprecated. Use `include_metadata `IORails` uses the same `provider_metadata` and `usage` keys as `LLMRails`. The following `IORails`-specific behaviors apply: -- `IORails` sets `stream_options` to `{"include_usage": True}` on the main model request so the provider returns usage on the terminal chunk. Override it through `llm_params` if your provider needs different behavior. +- `IORails` sets `stream_options` to `{"include_usage": True}` on the main model request so the provider returns usage on the terminal chunk. + Override it through `llm_params` if your provider needs different behavior. ```python async for chunk in rails.stream_async( @@ -213,8 +214,9 @@ The following `IORails`-specific behaviors apply: On `LLMRails`, `include_metadata=True` preserves metadata only when output-rail streaming is disabled. When output-rail streaming is enabled, the buffer yields plain strings and drops `provider_metadata` and `usage`. -A chunk that carries only provider metadata and no text is not emitted as a standalone empty frame. -The metadata remains visible when the provider also includes it on a content or usage chunk, which keeps repeated response headers from flooding the stream. +The library does not emit a standalone empty frame for a chunk that carries only provider metadata and no text. +The metadata remains visible when the provider also includes it on a content or usage chunk. +This behavior prevents repeated response headers from flooding the stream. ## Generation Options and Streaming @@ -223,6 +225,10 @@ Unlike `generate()` and `generate_async()`, passing it does not change what the The chunk shape is governed solely by `include_metadata`: plain strings when it is `False`, and `{"text": ..., "metadata": ...}` dictionaries when it is `True`. Neither engine yields or returns a `GenerationResponse` from a stream. +This behavior applies when the engine generates the stream. +On `LLMRails`, passing an external `generator` bypasses internal generation, so `options` and `include_metadata` do not change the supplied iterator. +The generator determines the chunk shape, and only configured streaming output rails can wrap it. + ```python # Identical chunk shapes: `options` does not select a structured return here. async for chunk in rails.stream_async(messages=messages): @@ -235,7 +241,7 @@ async for chunk in rails.stream_async( ... # still str ``` -What each engine reads from `options` while streaming: +The following table shows what each engine reads from `options` while streaming: | Option | LLMRails | IORails | |---|---|---| @@ -243,7 +249,8 @@ What each engine reads from `options` while streaming: | `rails.input`, `rails.output`, `rails.tool_input`, `rails.tool_output` | Applied | Applied | | `output_vars`, `log.*` | Computed, then discarded with the `GenerationResponse` | Not read | -`LLMRails` runs `generate_async()` in a background task and forwards `options` to it, then discards the `GenerationResponse` that call returns. +`LLMRails` runs `generate_async()` in a background task and forwards `options` to it. +It then discards the `GenerationResponse` that the call returns. Anything you request through `log` or `output_vars` is therefore computed but unreachable from the stream. `IORails` reads only `llm_params` and the four rail toggles in its streaming path. @@ -258,7 +265,7 @@ Do not rely on a stream rejecting an option that the non-streaming call refuses. Two `GenerationResponse` fields have no streamed equivalent: - `reasoning_content`: Neither engine emits provider reasoning deltas as chunk text. - A model that embeds `` tags in its content streams them as ordinary text, which means output rails see the reasoning. + A model that embeds `` tags in its content streams them as ordinary text, so output rails process the reasoning. `IORails` logs one warning per request when it detects this. - `tool_calls`: `IORails` emits assembled tool calls as a terminal chunk whose text is a `{"tool_calls": [...]}` JSON string. The string uses the OpenAI wire shape with `function.arguments` as a JSON string, while the non-streaming `GenerationResponse.tool_calls` uses dictionary arguments. diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index 021bc315f4..74f4d5d8a0 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -132,13 +132,14 @@ Test with `ConsoleMetricExporter` first to confirm IORails engine emission, then ### `LLMRails` Produces No Metrics Metrics are emitted only by the IORails engine. -Enable it either by setting `NEMO_GUARDRAILS_IORAILS_ENGINE=1` (which redirects `LLMRails(config)` to the IORails engine) or by constructing `Guardrails(config, use_iorails=True)` directly, and use `generate_async`, `stream_async`, or `check_async`. -Pass `require_iorails=True` to make the constructor raise (instead of silently falling back to LLMRails and emitting no metrics) when the config is incompatible with IORails. +Select IORails by setting `NEMO_GUARDRAILS_IORAILS_ENGINE=1`, which redirects `LLMRails(config)` to IORails, or by constructing `Guardrails(config, use_iorails=True)` directly. +Use `generate_async()`, `stream_async()`, or `check_async()` to emit metrics. +Pass `require_iorails=True` to raise when the configuration is incompatible with IORails instead of silently falling back to `LLMRails`, which emits no metrics. ### Synchronous `generate()` or `check()` Produces No Metrics Telemetry is disabled for the ephemeral engine constructed by the synchronous `generate()` and `check()` shims. -Use `generate_async`, `stream_async`, or `check_async` for production paths. +Use `generate_async()`, `stream_async()`, or `check_async()` for production paths. ### `gen_ai.client.token.usage` Is Missing for Streaming Requests diff --git a/tests/manifests/test_docs_engine_support_sync.py b/tests/manifests/test_docs_engine_support_sync.py new file mode 100644 index 0000000000..2e002ed35e --- /dev/null +++ b/tests/manifests/test_docs_engine_support_sync.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Keeps the published rail support matrix honest about the shipped catalog. + +The matrix in ``docs/reference/rail-engine-support.mdx`` is hand-maintained, so +adding a ``rail.py`` or lifting an IORails limitation would otherwise leave the +page quietly wrong. These tests read the page back and compare it against the +live catalog. +""" + +import re +from pathlib import Path + +import pytest + +from nemoguardrails.guardrails.compiled_rail import unsupported_surface_reason +from nemoguardrails.manifests import RailDirection, default_rail_catalog + +DOCS_PAGE = Path(__file__).resolve().parents[2] / "docs" / "reference" / "rail-engine-support.mdx" + +SUPPORTED = "✓" +UNSUPPORTED = "✗" + +_SECTION_HEADINGS = { + "### Input Surfaces": RailDirection.INPUT, + "### Output Surfaces": RailDirection.OUTPUT, + "### Retrieval Surfaces": RailDirection.RETRIEVAL, +} + +# | `flow name` | Rail | LLMRails | IORails | Notes | +_ROW = re.compile(r"^\|\s*`([^`]+)`\s*\|[^|]*\|\s*(\S+)\s*\|\s*(\S+)\s*\|") + + +def _documented_rows() -> dict[tuple[RailDirection, str], tuple[str, str]]: + """Parse the matrix tables into {(direction, flow): (llmrails mark, iorails mark)}.""" + rows: dict[tuple[RailDirection, str], tuple[str, str]] = {} + direction = None + for line in DOCS_PAGE.read_text(encoding="utf-8").splitlines(): + stripped = line.strip() + if stripped in _SECTION_HEADINGS: + direction = _SECTION_HEADINGS[stripped] + continue + if stripped.startswith("#"): + # Any other heading ends the table that the last surface heading opened. + direction = None + continue + if direction is None: + continue + match = _ROW.match(stripped) + if match is not None: + flow, llmrails, iorails = match.groups() + rows[(direction, flow)] = (llmrails, iorails) + return rows + + +def _catalog_rows() -> dict[tuple[RailDirection, str], tuple[str, str]]: + """Build the same mapping from the shipped manifests.""" + return { + key: (SUPPORTED, UNSUPPORTED if unsupported_surface_reason(surface) else SUPPORTED) + for key, surface in default_rail_catalog().surfaces().items() + } + + +@pytest.fixture(scope="module") +def documented(): + return _documented_rows() + + +@pytest.fixture(scope="module") +def catalog(): + return _catalog_rows() + + +def test_matrix_lists_every_catalog_surface(documented, catalog): + """Every surface the catalog declares appears in the documented matrix.""" + missing = sorted((direction.value, flow) for direction, flow in catalog.keys() - documented.keys()) + assert not missing, ( + f"{DOCS_PAGE.name} is missing {len(missing)} surface(s) the catalog declares: {missing}. " + "Add a row for each to the matching direction table." + ) + + +def test_matrix_lists_no_surface_the_catalog_dropped(documented, catalog): + """The documented matrix does not list a surface the catalog no longer declares.""" + stale = sorted((direction.value, flow) for direction, flow in documented.keys() - catalog.keys()) + assert not stale, ( + f"{DOCS_PAGE.name} documents {len(stale)} surface(s) the catalog does not declare: {stale}. " + "Remove the rows, or fix the flow name if the surface was renamed." + ) + + +def test_matrix_engine_marks_match_the_catalog(documented, catalog): + """Each documented row carries the support marks the catalog implies.""" + drifted = { + (key[0].value, key[1]): {"documented": documented[key], "expected": marks} + for key, marks in catalog.items() + if key in documented and documented[key] != marks + } + assert not drifted, ( + f"{DOCS_PAGE.name} disagrees with the catalog on {len(drifted)} row(s), as (LLMRails, IORails): {drifted}" + ) + + +def test_summary_counts_match_the_catalog(catalog): + """The support summary totals agree with the per-direction tables.""" + text = DOCS_PAGE.read_text(encoding="utf-8") + for direction in RailDirection: + total = sum(1 for key in catalog if key[0] is direction) + servable = sum(1 for key, marks in catalog.items() if key[0] is direction and marks[1] == SUPPORTED) + row = f"| {direction.value.capitalize()} | {total} | {total} | {servable} |" + assert row in text, f"{DOCS_PAGE.name} is missing the summary row {row!r}" + + total = len(catalog) + servable = sum(1 for marks in catalog.values() if marks[1] == SUPPORTED) + assert f"| **Total** | **{total}** | **{total}** | **{servable}** |" in text