fix(examples): repair guardrail e2e suite assertions and flaky triggers - #169
Merged
Conversation
ling-senpeng13
marked this pull request as ready for review
August 18, 2026 00:33
ling-senpeng13
marked this pull request as draft
August 18, 2026 16:02
ling-senpeng13
marked this pull request as ready for review
August 18, 2026 22:15
ling-senpeng13
force-pushed
the
fix/guardrail-e2e-assertions
branch
from
August 18, 2026 22:27
a0d9e58 to
0b6da07
Compare
kowser-orkes
approved these changes
Aug 19, 2026
The 27-test guardrail matrix had two suite-level defects: 1. check() stringified result.output with String(), yielding '[object Object]' — every expectContains/expectNotContains matched against that literal, so #9's REDACTED check could never pass and all leak checks passed vacuously. Use JSON.stringify for objects. 2. The raise/fix cells (#2, #8, #9) only prove anything when the model actually repeats the sensitive value, which the prompts left to chance (paraphrase -> guardrail correctly passes -> test 'fails'). Ask for the value directly as a natural support request so the leak the guardrail exists to catch happens reliably. Verbatim-quoting demands were tried and rejected: they trigger model refusals. Also accept retry exhaustion (FAILED) for #7 as long as the secret never escapes, matching the CI e2e suite's handling of the same case. Verified: 27/27 pass against conductor-oss 3.32.0-rc18.
ling-senpeng13
force-pushed
the
fix/guardrail-e2e-assertions
branch
from
August 19, 2026 00:34
0b6da07 to
04e6f28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 27-test guardrail matrix (
90-guardrail-e2e-tests.ts) reported failures on #8/#9, but the guardrails and server were behaving correctly — both defects were in the suite itself:1. Assertions inspected
"[object Object]".check()built its haystack withString(result.output), butoutputis an object. EveryexpectContains/expectNotContainsacross all 27 tests matched against that literal: #9'sREDACTEDcheck could never pass (the fix-guardrail's redacted output was verified present in the actual workflow result), and every leak assertion passed vacuously. Fixed withJSON.stringifyfor object outputs — the content assertions now audit real output for the first time.2. The raise/fix cells left their trigger to chance. #2/#8/#9 assert what happens when a guardrail fires, but firing requires the model to actually repeat the sensitive value — and the prompts sometimes got a paraphrase instead (guardrail correctly passes → test "fails"). Prompts now ask for the value directly as a natural support request ("tell me the SSN on file"), which reliably elicits it. Note: verbatim-quoting demands ("repeat EXACTLY word for word") were tried and made things worse — models refuse them outright.
Also: #7 (
retry) now accepts either terminal status as long as the secret never escapes — with a reliably-leaking prompt, retry exhaustion is legitimate (matches the CI e2e suite's handling of the same case).Verified: 27/27 pass against conductor-oss 3.32.0-rc18.