feat(trace): review feature table and add e2e checks for Claude Code gaps

app/cli/internal/trace/README.md

@@ -8,6 +8,20 @@ In the tables, **Yes** means supported, **Partial** means supported with limits,
 
 Claude Code has the most complete support.
 
+## Setup
+
+How Trace is added to a repository and to an agent.
+
+| Feature | What it means | Claude Code | Cursor | OpenCode 1.x | OpenCode 2 |
+|---------|---------------|-------------|--------|--------------|------------|
+| Hook installation and deletion | `chainloop trace init` installs the hooks of the selected agent. A second init changes nothing. `chainloop trace uninstall` deletes only the Chainloop hooks and keeps the other hooks of the agent. | Unknown | Unknown | Unknown | Unknown |
+| Trace run wrapper | `chainloop trace run -- <agent command>` gives the agent temporary hooks for one session. When the command exits, the configuration of the agent is the same as before. | Unknown | Unknown | Unknown | Unknown |
+| More than one agent in one repository | Two agents initialized in the same repository record their sessions separately, and each commit names the sessions of the agent that made it. | Unknown | Unknown | Unknown | Unknown |
+
+Why not Yes:
+
+- **All rows, all agents:** not verified yet. The end-to-end suite finds the values (spec issue-3635).
+
 ## Session collection
 
 What the session evidence holds about the conversation.
@@ -18,6 +32,7 @@ What the session evidence holds about the conversation.
 | Token usage and cost | The evidence holds the tokens that the session used and their cost in USD. | Yes | No | Yes | Yes |
 | Model detection | The evidence names the primary model and the other models that the session used. | Yes | Partial | Yes | Yes |
 | Subagents | The evidence holds the conversations of the subagents, and their token usage, as part of the session. | Yes | No | No | No |
+| Missed session start | A session whose session-start hook did not fire is still recorded, from its first tool hook. | Unknown | Unknown | Unknown | Unknown |
 
 Why not Yes:
 
@@ -26,6 +41,7 @@ Why not Yes:
 - **Model detection, Cursor:** the primary model comes from the session-start hook. The transcript does not tell the models, so the evidence does not list the other models.
 - **Subagents, Cursor:** the provider does not read subagent conversations.
 - **Subagents, OpenCode:** a subagent runs in a child session with its own export. It is not part of the evidence of its parent session.
+- **Missed session start, all agents:** not verified yet. The end-to-end suite finds the values (spec issue-3635).
 
 ## Code attribution
 
@@ -70,13 +86,15 @@ What the session was asked to build, and which skills it used.
 | Spec capture instruction | At session start, the agent is told to capture the specs of the session (tickets, documents, plans, images) in the spec folder. A plan is captured only when it comes from an external source or from the plan mode of the agent: an agreement in the chat is not a plan. A pasted image is not captured, also when the agent has a file path for it. The agent tells the user in one line when it captures a file. | Yes | Yes | Yes | Yes |
 | Spec capture reminder | At each user prompt, the agent is reminded to capture new or changed specs. | Yes | No | Yes | Yes |
 | Local spec sources read at push | A capture whose source is a local file holds only a header and a placeholder. Each push reads the file and records its current content, so the evidence follows the edits of the file. Only a regular text file of up to 1 MiB is read. Otherwise the push keeps the body that the agent wrote, or drops the capture with a warning when there is no body (spec issue-3561). | Yes | Yes | Yes | Yes |
+| Spec source role, title, and description | Each captured spec source holds the role, the title, and the description that the agent wrote in its header (spec issue-3531). | Yes | Unknown | Unknown | Unknown |
 | Skills tracking | The evidence lists the skills that the session used, with a copy of each skill as it ran. | Yes | No | Partial | Partial |
 | Spec source pointers | Exact copies of spec sources in the transcript (full file reads and file writes) are replaced with a pointer to the spec material, so the evidence does not hold the same content two or three times (spec issue-3556). | Yes | No | No | No |
 | Pasted image pointers | Each distinct image that the user pasted into the session, including images in prompts sent while the agent was busy, is stored once as an image spec entry, from the data in the transcript, with the title "Pasted image N" and no role. A repeated paste of the same image adds no entry. The inline image in the transcript is replaced with a pointer to that material. An image whose data cannot be decoded, or that the push could not store, stays inline, and the push continues. When the agent captured the same image, its entry is kept and no second entry is added (spec issue-3569). | Yes | No | No | No |
 
 Why not Yes:
 
 - **Spec capture reminder, Cursor:** Cursor has no hook at each user prompt. The agent gets the session-start instruction only.
+- **Spec source role, title, and description, Cursor and OpenCode:** not verified yet. The end-to-end suite finds the values (spec issue-3635).
 - **Skills tracking, Cursor:** the provider does not track skills. The evidence has no skill entries, which means "not recorded", not "no skill used".
 - **Skills tracking, OpenCode:** only the skills that the model starts with the `skill` tool are counted. Skills that the user starts, and skills used in subagents, are not.
 - **Spec source pointers, Cursor and OpenCode:** the provider has no finder for the copies in its transcript yet. For OpenCode, the rebuilt transcript holds no tool outputs, so only the write copies would apply.
@@ -148,4 +166,4 @@ A feature row maps to its checks by the feature name. When you rename a feature
 make -C app/cli test-trace-e2e
 ```
 
-`CHAINLOOP_E2E_HARNESS`, `CHAINLOOP_E2E_MODEL`, and `CHAINLOOP_E2E_FEATURE` limit a run to some columns, models, or features (comma-separated). Set `CHAINLOOP_E2E_WORK_DIR` to keep the workspaces and the agent output. The results go to `e2e/results/`, or to `CHAINLOOP_E2E_RESULTS_DIR`. Only a run with no filter becomes the previous result of the next comparison.
+`CHAINLOOP_E2E_HARNESS`, `CHAINLOOP_E2E_MODEL`, and `CHAINLOOP_E2E_FEATURE` limit a run to some columns, models, or features. Separate the values with `;`, because feature names can hold commas. Set `CHAINLOOP_E2E_WORK_DIR` to keep the workspaces and the agent output. The results go to `e2e/results/`, or to `CHAINLOOP_E2E_RESULTS_DIR`. Only a run with no filter becomes the previous result of the next comparison.

app/cli/internal/trace/e2e/checks.go

@@ -25,6 +25,7 @@ import (
 	"strings"
 
 	"github.com/chainloop-dev/chainloop/app/cli/internal/trace"
+	"github.com/chainloop-dev/chainloop/app/cli/internal/trace/pointer"
 	"github.com/chainloop-dev/chainloop/app/cli/internal/trace/state"
 	"github.com/chainloop-dev/chainloop/pkg/attestation/crafter/materials/aicodingsession"
 )
@@ -72,6 +73,7 @@ var features = map[string]Feature{
 	"Transcript":           {Checks: []Check{{scenarioEditAndCommit, checkTranscript}}},
 	"Token usage and cost": {Checks: []Check{{scenarioEditAndCommit, checkTokenUsage}}},
 	"Model detection":      {Checks: []Check{{scenarioEditAndCommit, checkModel}}},
+	"Subagents":            {Checks: []Check{{scenarioSubagent, checkSubagents}}},
 
 	// Code attribution
 	"Line attribution":          {Checks: []Check{{scenarioEditAndCommit, checkFileAttributedToAI("greeting.txt")}}},
@@ -87,6 +89,10 @@ var features = map[string]Feature{
 	"Spec capture reminder":           {Checks: []Check{{scenarioEditAndCommit, checkPromptReminder}}},
 	"Local spec sources read at push": {ModelDriven: true, Checks: []Check{{scenarioTicket, checkTicketContentStored}}},
 	"Skills tracking":                 {ModelDriven: true, Checks: []Check{{scenarioSkill, checkSkillTracked}}},
+	"Spec source pointers":            {ModelDriven: true, Checks: []Check{{scenarioTicket, checkTicketPointer}}},
+	"Spec source role, title, and description": {
+		ModelDriven: true, Checks: []Check{{scenarioTicket, checkTicketHeader}},
+	},
 
 	// Security
 	"Secret redaction":           {Checks: []Check{{scenarioEditAndCommit, checkSecretRedacted}}},
@@ -96,6 +102,10 @@ var features = map[string]Feature{
 	"Commit trailer":        {Checks: []Check{{scenarioEditAndCommit, checkCommitTrailer}}},
 	"Attestation at push":   {ControlPlane: true},
 	"Local storage at push": {Checks: []Check{{scenarioEditAndCommit, checkLocalStorage}}},
+	"Shell commands in another checkout": {Checks: []Check{
+		{scenarioClone, checkCommitTrailer},
+		{scenarioClone, checkFileAttributedToAI(cloneFile)},
+	}},
 }
 
 func evidence(s *Session) *aicodingsession.Data {
@@ -224,6 +234,48 @@ func checkTicketContentStored(s *Session) error {
 	return nil
 }
 
+// checkTicketHeader checks that the capture of the ticket holds the role, the
+// title, and the description that the agent wrote in its header.
+func checkTicketHeader(s *Session) error {
+	e, err := ticketSpec(s)
+	if err != nil {
+		return err
+	}
+	if e.Role == "" || e.Title == "" || e.Description == "" {
+		return fmt.Errorf("the ticket entry has role %q, title %q, description %q: want all three", e.Role, e.Title, e.Description)
+	}
+	return nil
+}
+
+// checkTicketPointer checks that a copy of the ticket in the transcript, for
+// example the full read of the ticket file, is replaced with a pointer to the
+// spec material.
+func checkTicketPointer(s *Session) error {
+	e, err := ticketSpec(s)
+	if err != nil {
+		return err
+	}
+	for _, msg := range evidence(s).RawSession["main"] {
+		if bytes.Contains(msg, []byte(pointer.Marker)) && bytes.Contains(msg, []byte(e.Digest)) {
+			return nil
+		}
+	}
+	return errors.New("the transcript has no pointer to the ticket material")
+}
+
+func checkSubagents(s *Session) error {
+	subs := evidence(s).Subagents
+	if len(subs) == 0 {
+		return errors.New("the evidence has no subagent")
+	}
+	for _, sub := range subs {
+		if sub.Tokens.Input+sub.Tokens.Output > 0 {
+			return nil
+		}
+	}
+	return errors.New("no subagent has token usage")
+}
+
 func checkSkillTracked(s *Session) error {
 	for _, e := range evidence(s).Spec {
 		if e.Kind != aicodingsession.SpecKindSkill || !strings.Contains(e.Title+e.URI, skillName) {

app/cli/internal/trace/e2e/claude.go

@@ -28,7 +28,7 @@ import (
 // claudeTools are the tools that a scenario session may use with no
 // permission prompt. The session runs in a throwaway repository, and the list
 // keeps the session from asking, which a headless run cannot answer.
-var claudeTools = []string{"Read", "Write", "Edit", "Glob", "Grep", "Bash", "Skill"}
+var claudeTools = []string{"Read", "Write", "Edit", "Glob", "Grep", "Bash", "Skill", "Task", "Agent"}
 
 // Claude is the harness adapter of Claude Code.
 type Claude struct {

app/cli/internal/trace/e2e/e2e_test.go

@@ -31,7 +31,8 @@ import (
 // Environment of a run of the suite.
 const (
 	// envHarness, envModel and envFeature filter a manual run. Each is a
-	// comma-separated list. A filtered run does not become the previous result.
+	// semicolon-separated list, because feature names can hold commas. A
+	// filtered run does not become the previous result.
 	envHarness = "CHAINLOOP_E2E_HARNESS"
 	envModel   = "CHAINLOOP_E2E_MODEL"
 	envFeature = "CHAINLOOP_E2E_FEATURE"
@@ -143,7 +144,7 @@ func envList(name string) []string {
 	if v == "" {
 		return nil
 	}
-	parts := strings.Split(v, ",")
+	parts := strings.Split(v, ";")
 	for i := range parts {
 		parts[i] = strings.TrimSpace(parts[i])
 	}

app/cli/internal/trace/e2e/runner.go

@@ -234,11 +234,21 @@ func (r *Runner) runScenario(ctx context.Context, h Harness, model string, sc Sc
 	// the session: the session end removes the captured specs, so a later push
 	// stores the session with no spec. A session that did not push stores
 	// nothing, and its checks fail.
-	if s.HeadMessage, err = ws.Git(ctx, "log", "-1", "--format=%B"); err != nil {
+	checkout := ws.Dir
+	if sc.Checkout != "" {
+		checkout = filepath.Join(base, sc.Checkout)
+		if _, err := os.Stat(checkout); err != nil {
+			// The agent did not create the checkout, so it stored nothing.
+			r.Logf("%s / %s / %s #%d: no checkout %s", h.Column(), model, sc.Name, i+1, sc.Checkout)
+			return s, nil
+		}
+	}
+
+	if s.HeadMessage, err = ws.run(ctx, checkout, "git", "log", "-1", "--format=%B"); err != nil {
 		return nil, err
 	}
 
-	stored, err := ws.StoredSessions()
+	stored, err := StoredSessions(checkout)
 	if err != nil {
 		return nil, err
 	}

app/cli/internal/trace/e2e/scenarios.go

@@ -24,6 +24,9 @@ type Scenario struct {
 	Seed map[string]string
 	// Prompt is the only user prompt of the session.
 	Prompt string
+	// Checkout is the checkout where the agent commits and pushes, relative
+	// to the workspace base. Empty means the repository of the workspace.
+	Checkout string
 }
 
 // The fake secrets of the scenarios. They match the GitHub token rule of the
@@ -48,12 +51,19 @@ const (
 	scenarioEditAndCommit = "edit-and-commit"
 	scenarioTicket        = "ticket"
 	scenarioSkill         = "skill"
+	scenarioSubagent      = "subagent"
+	scenarioClone         = "clone"
 )
 
 const (
-	ticketPath  = "docs/ticket-e2e-42.md"
-	ticketTitle = "E2E-42: Add a farewell file"
-	skillName   = "e2e-farewell"
+	ticketPath   = "docs/ticket-e2e-42.md"
+	ticketTitle  = "E2E-42: Add a farewell file"
+	skillName    = "e2e-farewell"
+	subagentFile = "subagent.txt"
+	// cloneDir is the checkout that the clone scenario creates, relative to
+	// the workspace base. The bare remote is next to it.
+	cloneDir  = "clone"
+	cloneFile = "clone.txt"
 )
 
 // scenarios are the scenarios of the suite, by name.
@@ -95,4 +105,22 @@ Then run: git add farewell.txt && git commit -m "add farewell" && git push
 		},
 		Prompt: "This is an automated test. Use the " + skillName + " skill.",
 	},
+	scenarioSubagent: {
+		Name: scenarioSubagent,
+		Seed: map[string]string{readmePath: seedReadme},
+		Prompt: `This is an automated test. Do exactly these steps, in order, and nothing else:
+1. Start one subagent with your subagent tool. Tell it to create the file ` + subagentFile + ` with the single line: from the subagent.
+2. When the subagent is done, run this shell command: git add ` + subagentFile + ` && git commit -m "add subagent file" && git push`,
+	},
+	scenarioClone: {
+		Name:     scenarioClone,
+		Seed:     map[string]string{readmePath: seedReadme},
+		Checkout: cloneDir,
+		// Each step is a shell command of its own: a clone gets the git hooks
+		// only after the command that created it.
+		Prompt: `This is an automated test. Do exactly these steps, in order, each as a separate shell command, and nothing else:
+1. git clone ../remote.git ../` + cloneDir + `
+2. cd ../` + cloneDir + ` && printf 'from the clone\n' > ` + cloneFile + `
+3. cd ../` + cloneDir + ` && git add ` + cloneFile + ` && git commit -m "add clone file" && git push`,
+	},
 }

app/cli/internal/trace/e2e/workspace.go

@@ -159,10 +159,11 @@ func (s *StoredSession) HasStatement() bool {
 	return err == nil
 }
 
-// StoredSessions reads the sessions that local storage mode stored. Each
-// session directory holds the AI coding session evidence among its materials.
-func (w *Workspace) StoredSessions() ([]StoredSession, error) {
-	root := filepath.Join(w.Dir, localSessionsDir)
+// StoredSessions reads the sessions that local storage mode stored in a
+// checkout. Each session directory holds the AI coding session evidence among
+// its materials.
+func StoredSessions(checkout string) ([]StoredSession, error) {
+	root := filepath.Join(checkout, localSessionsDir)
 	dirs, err := os.ReadDir(root)
 	if errors.Is(err, fs.ErrNotExist) {
 		return nil, nil