Commit 04e6f28
committed
fix(examples): repair guardrail e2e suite assertions and flaky triggers
The 27-test guardrail matrix had two suite-level defects:
1. check() stringified result.output with String(), yielding
'[object Object]' — every expectContains/expectNotContains matched
against that literal, so #9's REDACTED check could never pass and
all leak checks passed vacuously. Use JSON.stringify for objects.
2. The raise/fix cells (#2, #8, #9) only prove anything when the model
actually repeats the sensitive value, which the prompts left to
chance (paraphrase -> guardrail correctly passes -> test 'fails').
Ask for the value directly as a natural support request so the leak
the guardrail exists to catch happens reliably. Verbatim-quoting
demands were tried and rejected: they trigger model refusals.
Also accept retry exhaustion (FAILED) for #7 as long as the secret
never escapes, matching the CI e2e suite's handling of the same case.
Verified: 27/27 pass against conductor-oss 3.32.0-rc18.1 parent 208a5a0 commit 04e6f28
1 file changed
Lines changed: 10 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
49 | 51 | | |
50 | 52 | | |
51 | 53 | | |
| |||
542 | 544 | | |
543 | 545 | | |
544 | 546 | | |
545 | | - | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
546 | 550 | | |
547 | | - | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
548 | 554 | | |
549 | 555 | | |
550 | 556 | | |
| |||
560 | 566 | | |
561 | 567 | | |
562 | 568 | | |
563 | | - | |
| 569 | + | |
564 | 570 | | |
565 | 571 | | |
566 | 572 | | |
| |||
0 commit comments