Repository navigation
feat(trace): end-to-end test matrix for Claude Code - #3648
Merged
Merged
Conversation
Add a local end-to-end suite for Chainloop Trace (spec issue-3635, milestone 2). The suite reads the expected matrix from the feature table of the Trace README, runs Claude Code headless with real models in throwaway repositories in local storage mode, and stores the result matrix. It compares each full run with the previous one. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: be5af72c-30e7-48f4-b46c-dc7a5a754c41
Contributor
PR validation — ✅ 3 passing
AI Session Checks — 🟢 86% ·
|
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟢 86% | 1 | 100% AI / 0% Human | 19 | +2613 / -7 | 63h50m52s |
🟢 86% — 100% AI — ⚠️ 2 policies failing
-
Oct 16, 2026 16:40 UTC · 63h50m52s · $13.35 · 360 in / 179.7k out · claude-code 2.1.301 (claude-opus-5-5)
Change Summary
-
- Adds an
internal/trace/e2epackage that reads the Trace README tables, runs Claude Code scenarios, and stores a result matrix. - Adds real-agent checks for transcript, spec capture, skills, redaction, commit trailer, and local storage behavior.
- Adds unit tests, an
app/climake target, and README guidance for running the suite locally.
- Adds an
AI Session Overall Score
-
🟢 86% — Well-scoped milestone with strong testing; remaining no-check rows need follow-up.
AI Session Analysis Breakdown
-
🟢 92% · alignment
-
No notes.
🟢 92% · scope-discipline
-
No notes.
🟢 91% · user-trust-signal
-
No notes.
🟢 88% · context-and-planning
-
🟢 The session started from a captured spec plus explicit local-only milestone limits. · High Impact
🟢 85% · solution-quality
-
🟢 Spec-capture failures were fixed by correcting the runner flow, not by weakening checks. · High Impact
🟡 76% · verification
-
🟢 Real Claude end-to-end runs passed through the new
test-trace-e2etarget. · High Impact🟠 Five README rows still have no registered check, so parts of the matrix remain unvalidated. · Medium Severity
💡 When a new matrix row lands, add its check or defer that row explicitly in the same PR.
-
File Attribution
████████████████████100% AI / 0% Human
Policies (5, 2 failing)
Status Policy Material Messages ⚠️ Failedai-session-no-auto-prai-coding-session-be5af7Pull request opened without human approval (auto-approved: source=config, reason=subcommandResults): gh pr create --repo chainloop-dev/chainloop --base main --head jiparis:feat/trace-e2e-matrix --title "feat(trace): end-to-end test matrix for Claude Code" --bod... ✅ Passed ai-config-ai-agents-allowedai-coding-session-be5af7- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-be5af7- ⚠️ Failedai-config-no-secretsai-coding-session-be5af7- Secret (github-pat) detected in session content [turn=1343, source=tool_result, line=9]: |
⚠️ Failed |ai-config-no-secrets| [ai-coding-session-be5af7](https://app.chainloop.dev/u/chainloop/projects/chainloop/versi... - Secret (github-pat) detected in session content [turn=1386, source=assistant-text, line=10]: - AI session policy
ai-config-no-secrets: it flags the fake tokens in this session's transcript. Most of the matches are already the[CHAINLOOP_TRACE_REDACTED:github-pat]placeholder, so this ... - Secret (github-pat) detected in session content [turn=1412, source=tool_result, line=6]: 33 fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token that the suite expects redaction to remove
- Secret (github-pat) detected in session content [turn=1412, source=tool_result, line=9]: 36 fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token on a line with an allow marker
- Secret (github-pat) detected in session content [turn=1424, source=assistant-tool_use:Edit, line=1]: {"file_path":"/Users/jiparis/projects/chainloop/.claude/worktrees/cosmic-enchanting-flute/app/cli/internal/trace/e2e/scenarios.go","new_string":"// secret scanner, and they are not real tokens. The al...
- Secret (github-pat) detected in session content [turn=525, source=assistant-tool_use:Write, line=1]: {"content":"//\n// Copyright 2026 The Chainloop Authors.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\...
- Secret (github-pat) detected in session content [turn=672, source=tool_result, line=1]: deploy token [CHAINLOOP_TRACE_REDACTED:github-pat]
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=19]: fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=22]: fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=48]: fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=971, source=tool_result, line=51]: fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"
- Secret (github-pat) detected in session content [turn=999, source=assistant-tool_use:Bash, line=42]: s=s.replace('fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"','fakeSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token that the suite expects redaction to remove')
- Secret (github-pat) detected in session content [turn=999, source=assistant-tool_use:Bash, line=43]: s=s.replace('fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]"','fakeAllowedSecret = "[CHAINLOOP_TRACE_REDACTED:github-pat]" //nolint:gosec // a fake token on a line with an allow marker')
✅ Passed ai-config-mcp-servers-allowedai-coding-session-be5af7- -
Security Checks — ⚠️ 4 failing
⚠️ secret-scan — 1 failing
| Status | Policy | Messages |
|---|---|---|
secrets-detection |
⚠️ sast-scan — 3 failing
| Status | Policy | Messages |
|---|---|---|
cwe-top25 |
Weakness from CWE Top 25 found: CWE-94, "Improper Control of Generation of Code ('Code Injection')" | |
| ✅ Passed | cwe-top26-40-cusp |
- |
owasp-top10-2025 |
Weakness from OWASP Top 10 found: CWE-94, "Improper Control of Generation of Code ('Code Injection')" (A05:2025 – Injection) | |
sast |
|
✅ iac-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | iac-misconfiguration |
- |
Scans not applied (2)
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
Security context
✅ Nothing this change touches has a recorded security-fix history.
View in Chainloop ↗ · How this works ↗
Powered by Chainloop and Chainloop Trace
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 19 files
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
View guided diff | Re-trigger cubic
State in the README which cells the e2e suite tests and how it reports the others, and that it does not install the configured agent version. Mark the fake tokens of the scenarios for the repository secret scan. Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: be5af72c-30e7-48f4-b46c-dc7a5a754c41
migmartri
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements milestone 2 of spec issue-3635 (#3635): the framework of the end-to-end suite for Chainloop Trace, and the Claude Code column. This first version runs locally only. The nightly job and the Slack message come later.
The suite is in
app/cli/internal/trace/e2e/and runs withmake -C app/cli test-trace-e2e.claude -pand a real model, in a throwaway repository withtrace init --localand a local bare remote (R-003, D-001). The adapter reads the hook messages from the stream-json output, so the welcome message check reads what the user sees (R-004).config.yamlpins the harness version and lists the models of each column (D-009, D-013).Difference from the spec: in each scenario, the agent pushes inside the session. The spec does not say who pushes. A push after the session end stores no spec, because the session end removes the captured specs.
Not in this PR: the CI job, the Slack message, the installation of the pinned harness version, Cursor, OpenCode, Pi, and checks for five rows (Subagents, Spec source pointers, Pasted image pointers, Media skipped by redaction, Shell commands in another checkout).
AI disclosure: Claude Code helped to write this change.