Skip to content

feat(trace): e2e adapter for Codex - #3673

Merged
jiparis merged 1 commit into
chainloop-dev:mainfrom
jiparis:feat/trace-e2e-codex
Oct 20, 2026
Merged

jiparis merged 1 commit into
chainloop-dev:mainfrom
jiparis:feat/trace-e2e-codex

Conversation

@jiparis

@jiparis jiparis commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Part of #3635 (spec issue-3635). Adds a Codex column to the Chainloop Trace end-to-end suite, ahead of Codex support in Trace.

This change goes against a non-goal of spec issue-3635: "test Codex or other agents with no Trace provider". The Codex column is added first on purpose, the same way as the Pi column. Each cell is No until a Codex provider exists.

  • A Codex adapter runs codex exec --json with the scenario prompt, with no sandbox and no approvals, in the throwaway workspace. The project hooks run with --dangerously-bypass-hook-trust. The Codex configuration folder (CODEX_HOME) is next to the workspace, so the configuration, hooks, skills, and sessions of the user who runs the suite do not load. --ignore-user-config is not used, because it also turns off the project hooks in .codex/hooks.json. Pasted images go in with --image. Skills go in .agents/skills.
  • A model with the openrouter/ prefix runs on OpenRouter through a provider override, with OPENROUTER_API_KEY. Any other model runs on the default Codex provider, with CODEX_API_KEY.
  • Codex is pinned to 0.163.0 with openrouter/openai/gpt-5-mini.
  • The feature tables get a Codex column. A run with the cells set to Unknown gave an error for each testable cell: chainloop trace init --codex fails with "unknown flag: --codex". So each Codex cell is No, the README tells why, and the suite skips the column. When Codex support is added, set the cells to Unknown and run the suite to fill them.
  • The adapter reports no hook messages, and it cannot remove the session-start hook, until Trace has a Codex provider.

AI disclosure: Claude Code helped to write this change.

View guided diff

Add a Codex column to the Chainloop Trace end-to-end suite. Trace has no
Codex provider yet, so each Codex cell is No and the suite skips it.

Assisted-by: Claude Code
Signed-off-by: Jose I. Paris <jiparis@chainloop.dev>

Chainloop-Trace-Sessions: 552fa994-c621-4f9c-8150-21b0c9e51bb8
@chainloop-platform

chainloop-platform Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ✅ 3 passing

Status Policy Material Messages
✅ Passed pr-description-required pr-info -
✅ Passed pr-min-approvals pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — 🟢 84% · ✅ 0 failing

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟢 84% 1 ✅ 0 100% AI / 0% Human 5 +326 / -48 1h54m15s

🟢 84% — 100% AI — ✅ All policies passing

Oct 20, 2026 12:08 UTC · 1h54m15s · $8.15 · 252 in / 88.3k out · claude-code 2.1.304 (claude-opus-5-5)

View session details ↗

Change Summary

  • Adds a Codex harness adapter and unit tests under internal/trace/e2e/.
  • Registers Codex in the suite config and test harness list.
  • Adds a Codex column to the Trace README with unsupported cells kept at No.

AI Session Overall Score

🟢 84% — Mostly clean run; Codex verification stayed partial because the suite still stops early.

AI Session Analysis Breakdown

🟢 92% · scope-discipline

🟢 The final commit stayed on the README, config, registration, adapter, and adapter tests. · High Impact

🟢 90% · alignment

🟢 AI surfaced the spec's Codex non-goal before continuing. · High Impact

🟢 88% · user-trust-signal

No notes.

🟢 86% · context-and-planning

No notes.

🟢 84% · solution-quality

🟢 AI reproduced the hook issue in real Codex runs and removed the bad flag. · High Impact

🟡 72% · verification

🟠 A real Codex matrix run still stopped at trace init --codex, so end-to-end coverage stayed partial. · Medium Severity

💡 Keep one explicit harness-level smoke run for unsupported providers before calling the adapter validated.


File Attribution

████████████████████ 100% AI / 0% Human

Status Attribution File Lines
created ai app/cli/internal/trace/e2e/codex.go +195 / -0
modified ai app/cli/internal/trace/README.md +50 / -48
created ai app/cli/internal/trace/e2e/codex_test.go +74 / -0
modified ai app/cli/internal/trace/e2e/config.yaml +6 / -0
modified ai app/cli/internal/trace/e2e/e2e_test.go +1 / -0

Policies (5)

Status Policy Material Messages
✅ Passed ai-session-no-auto-pr ai-coding-session-552fa9 -
✅ Passed ai-config-ai-agents-allowed ai-coding-session-552fa9 -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-552fa9 -
✅ Passed ai-config-no-secrets ai-coding-session-552fa9 -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-552fa9 -

Security Checks — ✅ 6 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -

✅ iac-scan

Status Policy Messages
✅ Passed iac-misconfiguration -
Scans not applied (2)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

View guided diff | Re-trigger cubic

@jiparis
jiparis requested a review from a team October 20, 2026 14:10
@jiparis
jiparis merged commit ea78d58 into chainloop-dev:main Oct 20, 2026
17 checks passed
@jiparis
jiparis deleted the feat/trace-e2e-codex branch October 20, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants