Repository navigation
feat(trace): review feature table and add e2e checks for Claude Code gaps - #3651
Merged
Merged
Conversation
…gaps Add the missing feature rows to the Trace README (spec issue-3635, R-001): hook installation and deletion, trace run wrapper, more than one agent in one repository, missed session start, and spec source role, title, and description. Add e2e checks for subagents, spec source pointers, spec source headers, and shell commands in another checkout. Feature filters are now separated by semicolons, because feature names can hold commas. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: be5af72c-30e7-48f4-b46c-dc7a5a754c41
Contributor
PR validation — ✅ 3 passing
AI Session Checks — 🟢 88% ·
|
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟢 88% | 1 | 100% AI / 0% Human | 7 | +123 / -13 | 67h57m50s |
🟢 88% — 100% AI — ⚠️ 2 policies failing
-
Oct 16, 2026 16:40 UTC · 67h57m50s · $21.52 · 558 in / 249.1k out · claude-code 2.1.301 (claude-opus-5-5)
Change Summary
-
- Adds missing Trace README rows for setup/session/spec-capture coverage and marks the new Claude-verified spec-header row as
Yes. - Adds Claude e2e coverage for subagents, spec-source pointers and headers, and shell commands run from a cloned checkout.
- Updates runner/workspace handling so clone scenarios read stored sessions from the checkout that actually pushed, and changes feature filters to use
;.
- Adds missing Trace README rows for setup/session/spec-capture coverage and marks the new Claude-verified spec-header row as
AI Session Overall Score
-
🟢 88% — Focused phase delivery, verified with real Claude runs, with one version-mismatch caveat.
AI Session Analysis Breakdown
-
🟢 95% · user-trust-signal
-
No notes.
🟢 93% · scope-discipline
-
No notes.
🟢 92% · alignment
-
🟢 The shipped diff matches the user-approved “feature table plus 1-3” phase exactly. · High Impact
🟢 86% · context-and-planning
-
No notes.
🟢 85% · solution-quality
-
🟢 Clone-session support was fixed in runner/storage handling instead of weakening the scenario. · Medium Impact
🟢 82% · verification
-
🟢 A targeted real-Claude run passed for all three newly added checks. · High Impact
🟡 The real-agent validation used installed Claude 2.1.303 while
config.yamlstill pins 2.1.301. · Low Severity
-
File Attribution
████████████████████100% AI / 0% HumanStatus Attribution File Lines modified ai app/cli/internal/trace/e2e/checks.go+52 / -0 modified ai app/cli/internal/trace/e2e/scenarios.go+31 / -3 modified ai app/cli/internal/trace/README.md+19 / -1 modified ai app/cli/internal/trace/e2e/runner.go+12 / -2 modified ai app/cli/internal/trace/e2e/workspace.go+5 / -4 modified ai app/cli/internal/trace/e2e/e2e_test.go+3 / -2 modified ai app/cli/internal/trace/e2e/claude.go+1 / -1
Policies (5, 2 failing)
Status Policy Material Messages ⚠️ Failedai-session-no-auto-prai-coding-session-be5af7- Pull request opened without human approval (auto-approved: source=config, reason=subcommandResults): gh api -X PUT repos/chainloop-dev/chainloop/pulls/3648/merge -f merge_method=squash -f commit_title="feat(trace): end-to-end test matrix for Claude Code (feat(trace): end-to-end test matrix for Claude Code #3648)...
- Pull request opened without human approval (auto-approved: source=config, reason=subcommandResults): gh api repos/chainloop-dev/chainloop/pulls/3648/comments/4311478018/replies -f body="Fixed in b7504e5. The section now says that the suite runs the Claude Code ...
- Pull request opened without human approval (auto-approved: source=config, reason=subcommandResults): gh pr create --repo chainloop-dev/chainloop --base main --head jiparis:feat/trace-e2e-matrix --title "feat(trace): end-to-end test matrix for Claude Code" --bod...
✅ Passed ai-config-ai-agents-allowedai-coding-session-be5af7- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-be5af7- ⚠️ Failedai-config-no-secretsai-coding-session-be5af7- Secret (github-pat) detected in session content [turn=1343, source=tool_result, line=9]: |
⚠️ Failed |ai-config-no-secrets| [ai-coding-session-be5af7](https://app.chainloop.dev/u/chainloop/projects/chainloop/versi... - Secret (github-pat) detected in session content [turn=1386, source=assistant-text, line=10]: - AI session policy
ai-config-no-secrets: it flags the fake tokens in this session's transcript. Most of the matches are already the[CHAINLOOP_TRACE_REDACTED:github-pat]placeholder, so this ... - Secret (github-pat) detected in session content [turn=1412, source=tool_result, line=6]: 33 fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token that the suite expects redaction to remove
- Secret (github-pat) detected in session content [turn=1412, source=tool_result, line=9]: 36 fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token on a line with an allow marker
- Secret (github-pat) detected in session content [turn=1424, source=assistant-tool_use:Edit, line=1]: {"file_path":"/Users/jiparis/projects/chainloop/.claude/worktrees/cosmic-enchanting-flute/app/cli/internal/trace/e2e/scenarios.go","new_string":"// secret scanner, and they are not real tokens. The al...
- Secret (github-pat) detected in session content [turn=525, source=assistant-tool_use:Write, line=1]: {"content":"//\n// Copyright 2026 The Chainloop Authors.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\...
- Secret (github-pat) detected in session content [turn=672, source=tool_result, line=1]: deploy token [CHAINLOOP_TRACE_REDACTED:github-pat]
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=19]: fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=22]: fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=48]: fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=51]: fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=999, source=assistant-tool_use:Bash, line=42]: s=s.replace('fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"','fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token that the suite expects redaction to remove')
- Secret (github-pat) detected in session content [turn=999, source=assistant-tool_use:Bash, line=43]: s=s.replace('fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"','fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token on a line with an allow marker')
✅ Passed ai-config-mcp-servers-allowedai-coding-session-be5af7- -
Security Checks — ✅ 5 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
Scans not applied (3)
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
iac-scan |
no IaC files changed |
Security context
✅ Nothing this change touches has a recorded security-fix history.
View in Chainloop ↗ · How this works ↗
Powered by Chainloop and Chainloop Trace
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
migmartri
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues spec issue-3635 (#3635): the feature table review (milestone 1, R-001) and more checks for the Claude Code column (milestone 2).
Feature table review (R-001, D-012):
New end-to-end checks for Claude Code:
The feature filters of a manual run (
CHAINLOOP_E2E_FEATUREand the others) are now separated by;, because feature names can hold commas.AI disclosure: Claude Code helped to write this change.