feat: let the cli annotate why the test step failed by dfrankland · Pull Request #153 · trunk-io/analytics-uploader

dfrankland · 2026-10-23T22:52:02Z

What

Stacked on #152. The action side of ask 1 of TRUNK-19074: when the test step failed without running any tests, the annotation should say so, instead of hedging.

The CLI knows why a run failed; the action only knows that it did. So the action now has the CLI word the failure, rather than reading the CLI's summary format itself.

The action points the CLI's JSON summary at a file through TRUNK_SUMMARY_OUTPUT_FILE, the env var for --summary-output-file (Add --summary-output-file to upload and test analytics-cli#1246).
When the CLI exits non-zero, the action runs a hidden command: trunk-analytics-cli github-annotations --summary-file <path>. It reads the summary the run just wrote and prints ::error:: lines.
If the command printed something and exited 0, the action passes its output through to the log. It marks the step failed with process.exitCode, not core.setFailed, which would add a second annotation.
Otherwise the action falls back to fix: quiet setup logging, and stop blaming test failures for cli errors #152's messages. That covers a CLI without the command, a command with nothing to say, and a missing binary. The command's stderr is captured, so an older CLI's "unrecognized subcommand" never shows in the log.

A cancelled step skips the CLI and keeps the action's own message, because cancelled reaches the CLI as exit code 1, the same as failure. Exit 70 still goes to telemetry as Command failed with exit code 70.

The action never reads the summary, so it doesn't depend on the summary's format.

Keeping the summary

summary-output-file input: where the CLI writes the summary, e.g. for a PR comment.
summary-output-file output: the path, set whenever the CLI wrote a summary.
Default location: without the input, a TRUNK_SUMMARY_OUTPUT_FILE the workflow already set is used. Otherwise the file goes in $RUNNER_TEMP.
Cleanup: the action deletes the summary path before the CLI runs, because the CLI doesn't always write one (a clap error, a signal, a panic, or a version without the env var), and a file left by an earlier run at a reused path would otherwise be read as this run's. It never deletes the summary afterwards, so later steps can read it; the runner empties $RUNNER_TEMP after each job.

Depends on the CLI

github-annotations is added by trunk-io/analytics-cli#1265, which isn't merged or released yet. It implements this interface:

Invocation: trunk-analytics-cli github-annotations --summary-file <path>.
Output: workflow commands such as ::error::… on stdout, escaped per GitHub's rules.
Exit code: 0 when it printed an annotation, non-zero when it has nothing to say.

Until a CLI release has the command, every run takes the fallback, which is exactly #152's behaviour. Nothing checks the CLI version.

Testing

jest: 80 tests pass.
- errors.test.ts: a CLI-written annotation fails the step without setFailed; exit 70 still reaches telemetry; a cancelled step skips the CLI; the fallback when the CLI can't annotate.
- summary.test.ts: annotateFromCli against a stand-in CLI script. It checks the arguments passed, the output passed through, and the fallbacks for a missing command, empty output and a missing binary. It also checks where the summary path comes from: the input, then the env var, then RUNNER_TEMP.
- download.test.ts: the output is set when the summary exists and left unset when it doesn't, and TRUNK_SUMMARY_OUTPUT_FILE is passed to the CLI.
tsc --noEmit, eslint and prettier --check: clean.
dist/ is rebuilt with ncc build -m src/index.ts -o dist. That's the pre-commit hook's build step; I committed with --no-verify because trunk's hook couldn't find pnpm in my environment.

🤖 Generated with Claude Code

trunk-staging-io · 2026-10-23T23:01:08Z

_{View Full Report ↗︎ ⋅ Docs}

trunk-io · 2026-10-23T23:19:25Z

_{View Full Report ↗︎ ⋅ Docs}

TylerJang27

Will include more notes in the analytics-cli PRs

TylerJang27 · 2026-10-26T16:16:27Z

Should we an action to call this like https://github.com/trunk-io/trunk2/blob/main/.github/workflows/sync-dynamic-ci-filter-schema.yaml? Just worried about discoverability more than anything

TylerJang27 · 2026-10-26T16:17:24Z

+  ALL_FAILURES_QUARANTINED: undefined,
+  NO_FAILURES: undefined,


nit: Minor smell having these in the map as undefined

TylerJang27 · 2026-10-26T16:19:06Z

+  /**
+   * A run that fails outright exits with this code, or 0 when it's null. Upload reports exit with their own `exitCode`.
+   */
+  exitCode: number | null;


Is this a duplicate of the top level exit code?

TylerJang27 · 2026-10-26T16:20:02Z

+   * Failed tests that were quarantined, grouped like `failures`.
+   */
+  quarantined: FailureGroup[];
+  quarantining: Quarantining;


Nit: add one more wrapper/invocation type (was this a analytics-cli test vs. `analytics-cli upload)

The action now points the cli's summary at TRUNK_SUMMARY_OUTPUT_FILE and, when the cli exits non-zero, reads its outcome to say which case it is: - NO_TESTS_FOUND: the step failed without running any tests. - FAILURES_NOT_QUARANTINED: the results contain non quarantined failures. - NO_FAILURES_BUT_NON_ZERO_EXIT_CODE: the step failed, but none of the tests it reported did. Older cli versions ignore the env var and write nothing, so they keep the hedged message. A summary path the workflow set itself is read but left in place. Part of TRUNK-19074. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The summary's types and its runtime check now both come from src/summary-schema.json, a copy of `trunk-analytics-cli summary-output-schema`, instead of a hand-written guess at its shape: - `pnpm generate:summary-schema [cli]` refreshes the copy from a cli binary and generates src/summary-schema.ts with json-schema-to-typescript. CI checks the types match the copy. - readSummary validates the file with zod's `fromJSONSchema` against the same copy, so a summary whose shape changed under us reads as no summary rather than a wrong cast. - The outcome messages are keyed by the generated `Outcome`, so a new outcome in a refreshed schema fails the build until it's handled. Workflows can now keep the summary: a `summary-output-file` input picks where the cli writes it, and the `summary-output-file` output gives the path whenever the cli wrote one. The action no longer deletes it; the default lives in RUNNER_TEMP, which the runner empties after the job. Part of TRUNK-19074. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Reading the summary's outcome meant the action had to track the cli's summary format. Now, when the cli exits non-zero, the action asks it to annotate the failure itself with a hidden `trunk-analytics-cli github-annotations --summary-file <path>`, which reads the summary the run just wrote and prints `::error::` lines. The action passes those through and fails the step without an annotation of its own. Older cli versions don't have the command, so its error is swallowed and the action falls back to its own messages, as does a cancelled step, which the cli only sees as exit code 1. This drops zod, json-schema-to-typescript, the copy of the cli's schema and the generated types. The summary-output-file input and output stay; the action only passes that path through. Part of TRUNK-19074. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

The CLI doesn't write a summary when clap rejects its arguments, when it is stopped by a signal or panics, or when it predates TRUNK_SUMMARY_OUTPUT_FILE. A summary left at a reused path (the summary-output-file input, or a job-level TRUNK_SUMMARY_OUTPUT_FILE) was then read as this run's: the annotation gave the earlier run's reason, and the output pointed at the old file. The action removes the path before the run and never after, so later steps can still read the summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

dfrankland changed the title ~~feat: say why the test step failed, from the cli's summary~~ feat: let the cli annotate why the test step failed Oct 24, 2026

TylerJang27 approved these changes Oct 26, 2026

View reviewed changes

dfrankland mentioned this pull request Oct 27, 2026

Add a hidden github-annotations command for the uploader action trunk-io/analytics-cli#1265

Merged

dfrankland added this pull request to stack #154 November 4, 2026 21:14

dfrankland force-pushed the dylan/trunk-19074-read-cli-summary branch from 535ec9e to 24aa9a3 Compare November 4, 2026 21:30

dfrankland and others added 4 commits November 4, 2026 21:31

dfrankland force-pushed the dylan/trunk-19074-read-cli-summary branch from 24aa9a3 to 92e9b2c Compare November 4, 2026 21:31

dfrankland merged commit ea52c24 into main Nov 4, 2026
8 checks passed

dfrankland deleted the dylan/trunk-19074-read-cli-summary branch November 4, 2026 21:35

Navigation Menu

Repository navigation

feat: let the cli annotate why the test step failed - #153

feat: let the cli annotate why the test step failed#153
dfrankland merged 4 commits into
dylan/trunk-19074-uploadercli-copy-distinguish-test-step-failed-withoutfrom
dylan/trunk-19074-read-cli-summary

dfrankland commented Oct 23, 2026 •

edited

Loading

Uh oh!

trunk-staging-io Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

trunk-io Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

TylerJang27 left a comment

Uh oh!

TylerJang27 Oct 26, 2026

Uh oh!

TylerJang27 Oct 26, 2026

Uh oh!

TylerJang27 Oct 26, 2026

Uh oh!

TylerJang27 Oct 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Conversation

dfrankland commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Keeping the summary

Depends on the CLI

Testing

Uh oh!

trunk-staging-io Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TylerJang27 left a comment

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

TylerJang27 Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

dfrankland commented Oct 23, 2026 •

edited

Loading

trunk-staging-io Bot commented Oct 23, 2026 •

edited

Loading

trunk-io Bot commented Oct 23, 2026 •

edited

Loading