Repository navigation
fix(policies): evaluate redacted materials against the content that was stored - #3383
Merged
jiparis merged 7 commits intoSep 1, 2026
Conversation
…as stored Secret redaction only rewrote the copy of an AI coding session that leaves the machine; policies were still evaluated against the untouched original on disk. Since a policy is user-authored Rego that can send what it reads to an allowed hostname, that made the policy engine a path for the very credentials redaction exists to remove. It also does not survive server-side evaluation, where the redacted copy is the only content available. Crafters that do not store the artifact verbatim now report the bytes they stored through materials.Craft, and the crafter passes them to both policy verifiers. Content resolution fails closed: a material annotated as redacted refuses to resolve at all without its sanitized copy, uniformly across CAS backends, rather than falling back to the file on disk. This also covers skip-upload materials, whose sanitized bytes were previously retained nowhere. Nothing is held in memory for materials that were not transformed: those keep resolving their content from the inline copy or the file, as before. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Contributor
AI Session Checks — 🟡 83% ·
|
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟡 83% | 1 | 100% AI / 0% Human | 99 | +1142 / -557 | 72h53m51s |
🟡 83% — 100% AI — ⚠️ 1 policies failing
-
Aug 28, 2026 11:10 UTC · 72h53m51s · $150.86 · 1.4k in / 414.2k out · claude-code 2.1.250 (claude-opus-5)
Change Summary
-
- Routes redacted AI-session content through crafting and policy evaluation instead of rereading the source file.
- Simplifies the related APIs around
CraftResult.ContentandVerifyMaterial(..., content). - Updates tests and local policy fixtures, then fixes a review-caught supplied-content edge case.
AI Session Overall Score
-
🟡 83% — Strong session, but missing diff coverage and a review-caught bug warrant review.
AI Session Analysis Breakdown
-
🟢 96% · verification
-
🟢 Repeated automated checks were backed by manual end-to-end policy evaluation. · High Impact
🟢 93% · context-and-planning
-
🟢 The AI explored, replanned, and got approval before implementation. · High Impact
🟢 91% · user-trust-signal
-
🟢 The user kept delegating follow-on work through PR creation and review handling. · Medium Impact
🟡 74% · solution-quality
-
🟠 A review caught a supplied-content edge case that needed a follow-up fix before landing cleanly. · Medium Severity
💡 Exercise generalized paths against adjacent kinds, not only the currently reachable producer.
🟡 72% · alignment
-
🟠 The STRING-based explanation and test described an impossible case and had to be corrected. · Medium Severity
💡 Before explaining an invariant, re-check that the cited input shape can actually occur.
abstained · scope-discipline
-
🟡 Scope discipline was not assessed because no rendered diff was available in this workdir. · Low Severity
Missing criteria: scope-discipline
-
File Attribution
████████████████████100% AI / 0% Human…and 74 more file(s).
Policies (4, 1 failing)
Status Policy Material Messages ✅ Passed ai-config-ai-agents-allowedai-coding-session-06ec99- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-06ec99- ⚠️ Failedai-config-no-secretsai-coding-session-06ec99- Potential secret (Quoted API key/password) found in session content [turn=436, source=tool_result, line=10, value=Secret ...tY6"]
- Potential secret (Quoted API key/password) found in session content [turn=58, source=tool_result, line=28, value=Secret ...tY6"]
✅ Passed ai-config-mcp-servers-allowedai-coding-session-06ec99- -
Security Checks — ⚠️ 1 failing
⚠️ secret-scan — 1 failing
| Status | Policy | Messages |
|---|---|---|
secrets-detection |
Gitleaks secret detected 779c37d:pkg/attestation/crafter/crafter_test.go:generic-api-key:826, secret: kQ7zXn2V...) |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
✅ iac-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | iac-misconfiguration |
- |
PR info
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | pr-description-required |
- |
| ✅ Passed | pr-user-story-linked |
- |
⏭️ 2 scans not applied
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
Powered by Chainloop and Chainloop Trace
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 17 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
jiparis
marked this pull request as draft
August 28, 2026 17:12
Reporting the bytes a crafter stored in place of the artifact was bolted on as a second, optional interface method alongside Craft. Fold it into Craft itself: Craftable returns a CraftResult carrying the material and, when the artifact was not stored verbatim, the bytes that replaced it. One method instead of two, so a crafter cannot report content through a path the generic craft flow does not take, and the capability is visible in the interface rather than discovered by type assertion. Every crafter but the AI coding session stores the artifact as it found it and leaves Transformed nil. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
…Content The single-argument GetEvaluableContent had no callers left outside tests once both policy verifiers moved to the variant taking an explicit content source, so keeping it as a delegating wrapper only offered a second way in — one that resolves a redacted material's content from the file on disk and fails. Give GetEvaluableContent the content parameter and drop the wrapper. Callers that have nothing to supply pass nil, which is the behaviour the old signature had. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
The field holds the content the crafter stored, which is what every consumer wants from it. Naming it after the transformation described how it came to exist rather than what it is. Locals that carried the field's value are renamed to match; the prose that describes crafters transforming an artifact is unchanged. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
…option The functional-option machinery existed to carry a single []byte, which is more ceremony than the value deserves: a type, a constructor, an accumulator and an option struct so that one call site could pass one slice. VerifyMaterial takes the content as a plain parameter instead. Callers that have nothing to supply pass nil, which reads no worse than omitting an option and makes the alternative visible in the signature. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
…kinds
Supplied content was selected inside the branch that resolves an artifact, so a
material carrying no artifact — a container image, a string — dropped it and fell
through to the empty "{}" projection, handing policies an empty object instead of
the bytes the caller asked to evaluate.
Choose supplied content before resolving the artifact, which is where the
fail-closed check for redacted materials already sits for the same reason.
No live caller was affected: content is only supplied for
CHAINLOOP_AI_CODING_SESSION today, which is artifact-based. Reported by cubic on
PR chainloop-dev#3383.
Assisted-by: Claude Code
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>
Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
jiparis
marked this pull request as ready for review
August 31, 2026 09:18
migmartri
reviewed
Aug 31, 2026
…tent first A STRING material cannot supply content: the value is the material, with no artifact to transform. Citing it as motivation described something impossible, and the test asserted semantics for an input that cannot occur. State the invariant instead — supplied content is authoritative for every kind, not only those that resolve from an artifact — and keep the one case that illustrates it. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: 06ec9990-4ad5-4b3c-ad86-2421b24dba57
javirln
approved these changes
Sep 1, 2026
jiparis
deleted the
jiparis/pfm-7073-redaction-send-redacted-content-to-policies
branch
September 1, 2026 14:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Secret redaction (#3368) only rewrites the copy of a
CHAINLOOP_AI_CODING_SESSIONthat leaves the machine. Policies were still evaluated against the untouched original on disk, having access to any potential secret that could have leaked into the AI session.The invariant now implemented is that the policy engine receives exactly the bytes that were stored.
Changes
transformCrafterseam and report the bytes they stored.materials.Craftsurfaces them asCraftResult.EvaluableContent, and the crafter passes them to both policy verifiers via the newpolicies.WithMaterialContentper-call option.chainloop.material.redacted=truereturnsErrRedactedContentRequiredrather than falling back to the file on disk. This is uniform across CAS backends, inline included.chainloop policy devel evalthreads the same content through, so a policy is developed against the input the real run produces.[REDACTED:<rule>]placeholders and the redaction annotations, since the raw pattern it looked for is no longer present in its input.Materials that were not transformed are unaffected and hold nothing extra in memory: their content still resolves from the inline copy or the file on disk. This also closes a gap for
skipUploadmaterials, whose sanitized bytes were previously retained nowhere at all.Notable consequences
A redacted material's policy input can no longer be reconstructed from persisted crafting state alone, so a future push-time or server-side material evaluation must plumb the bytes through as well.
materials.Craftnow returns*CraftResultandpolicies.Verifier.VerifyMaterialtakes variadic options — both breaking for external importers of these packages.Not covered here
Two gaps remain that are properties of redaction rather than this plumbing:
--skip-secret-redactionstores the session as captured, and secrets counted asreport.Unlocated(landing in a protected path, or spanning two JSON leaves) are not redacted in the stored copy either. Detecting placeholders in the upstreamai-coding-session-no-secretspolicy is the second part of the issue and lives in another repository; that policy should keep its pattern matching alongside the placeholder check, since pattern matching remains the only thing that catches the secrets redaction could not reach.AI disclosure
Produced with assistance from Claude Code; every commit carries an
Assisted-by: Claude Codetrailer.