Skip to content

ci(brain): stop calling tri stress --record, and let the weekly stress step fail - #880

Open
gHashTag wants to merge 1 commit into
mainfrom
ci/brain-weekly-honest
Open

gHashTag wants to merge 1 commit into
mainfrom
ci/brain-weekly-honest

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Closes #879. Part of #851.

What was red, and what was green that should not have been

The weekly trend report runs only on Sunday's schedule, and it failed on both of its runs. On 2026-09-27 tri was not built yet (exit 127). On 2026-10-04 it was built, and tri stress --record returned NotImplemented. That is deliberate: runStressTestCommand implements --health and leaves every other mode an honest NotImplemented.

On 2026-09-27 the step before it, "Run Full Stress Test", ended success over 15 compile errors in src/brain/stress_test.zig. The cause: | tee under the default bash -e reports tee's exit code. The brain-stress job had the same defect and was fixed earlier; this copy was a known entry in tools/tee_pipefail_baseline.txt.

Changes

where before after
weekly, stress step default shell, no pipefail shell: bash (-eo pipefail)
weekly, snapshot tri stress --record, failed every run tri stress --health | tee, under shell: bash
weekly, report tri stress --history (NotImplemented) the health score, plus one line saying there is no history yet
weekly, artifact named literally weekly-brain-report-$(date +%Y%m%d) weekly-brain-report-${{ github.run_id }}
weekly, if: schedule only schedule or workflow_dispatch
brain-health --record || echo, then "Health snapshot recorded" in the summary on every run step removed
brain-stress --record || echo "Record skipped"; this job builds no tri call removed
tee-pipefail baseline 78 entries 77

Measured locally

Not in this PR

  • The weekly job has not run yet. workflow_dispatch on this branch is its test, and its result goes in a comment here. The pull-request run skips the weekly job.
  • A real history. src/brain/health_history.zig has record, recent and trend. Wiring --record and --history to it is the owner's call; see brain-ci weekly trend report: red on 2 of 2 scheduled runs, and its stress step went green over a compile error #879, step 2. On CI the jsonl would also have to survive between weekly runs.
  • failed command: .../test on passing stress runs is not a failure. Zig 0.16's build runner prints a step's stderr with its command line whatever the result, and the benchmarks write to stderr. A comment above the step says so, so the next reader does not chase it.

🤖 Generated with Claude Code

…s step fail

The weekly trend report has failed on both of its scheduled runs. On
2026-09-27 tri was not built yet (exit 127); on 2026-10-04 it was, and
`tri stress --record` returned NotImplemented, as runStressTestCommand
says every mode but --health does on purpose.

On 2026-09-27 the step before it, "Run Full Stress Test", ended success
over 15 compile errors in src/brain/stress_test.zig: `| tee` under the
default bash -e reports tee's exit code. The brain-stress job had the
same defect and was fixed; this copy sat in the tee-pipefail baseline.

- Weekly job: both piped steps under `shell: bash`; record the score
  from `tri stress --health` instead of --record; drop --history; fix
  the artifact name, which `with:` took literally as
  "weekly-brain-report-$(date +%Y%m%d)"; let workflow_dispatch run it,
  so a fix can be checked before Sunday.
- brain-health: drop "Record Health Snapshot", which wrote "Health
  snapshot recorded" to the summary on every run and recorded nothing.
- brain-stress: drop the --record call; that job builds no tri binary.
- tee_pipefail_baseline.txt: remove the weekly entry (77 known, 0 new).

Closes #879

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

@gHashTag

gHashTag commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

The weekly job's test, as promised in the description: workflow_dispatch on this branch at 1ffa2a182, run 37238268276, job "📈 Weekly Brain Trend Report" (111541654562).

step result from the log
Build TRI Binary success
Run Full Stress Test success shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}; the two OP/s lines, then failed command: ./.zig-cache/o/.../test ... on a passing step, as the comment above the step says
Weekly Health Score success same shell; probes 1/4 to 4/4 OK, Score: 100.0, Status: HEALTHY
Generate Trend Report success
Upload Weekly Report success there will be 3 files uploaded; artifact weekly-brain-report-37238268276 (11316378053), 1009 bytes

So the Done-when items of #879 that a run can show are met: the weekly job is green on workflow_dispatch with a Score: line, and the artifact name is the run id rather than the literal $(date +%Y%m%d).

Not shown by this run: that the stress step now goes red when the stress test fails. The run proves the step runs under pipefail; it does not exercise a failing build. That rests on tools/check_tee_pipefail.py (0 new, 0 stale) and on the brain-stress job, which has had pipefail since its own fix. The other brain-ci jobs in this run were still in progress when I read it.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

@tri-claw

tri-claw Bot commented Oct 10, 2026

Copy link
Copy Markdown

Summary by t27-bees

The PR fixes CI flakiness by replacing non-existent tri stress --record/--history calls with the implemented --health mode, enabling pipefail so tee no longer hides test failures, and making the weekly job runnable via workflow_dispatch for testing. It also corrects a literal $(date) in an artifact name.

Worth a look

  • .github/workflows/brain-ci.yml:583–590 — "Run Full Stress Test" now uses shell: bash (-eo pipefail). If zig build test-brain-stress fails, the step correctly fails. Previously the default shell reported tee's exit code (0) and the step passed despite 15 compile errors.
  • .github/workflows/brain-ci.yml:593–598 — "Weekly Health Score" runs tri stress --health | tee under shell: bash. --health exits 1 when score < 80; with pipefail this will fail the job. That matches the PR's stated goal ("let the weekly stress step fail"), but it means a low health score now red-flags the weekly run.
  • .github/workflows/brain-ci.yml:615–624 — "Generate Trend Report" cats /tmp/weekly-health.txt and adds a note that history is not implemented. The step uses the default shell (no explicit shell: bash); if the prior step failed, this step may not run unless if: always() is added (it isn't). The summary will be incomplete on health-score failures.
  • .github/workflows/brain-ci.yml:626–632 — Artifact upload includes .trinity/brain_health_history.jsonl. The PR notes this file doesn't exist yet and surviving it between weekly runs is out of scope. actions/upload-artifact@v4 fails the step if a listed path is missing, so the upload will fail on the first run.
  • tools/tee_pipefail_baseline.txt:69 — Removed the baseline entry for the weekly stress test. Correct, since the step now uses shell: bash with pipefail and is no longer a known tee pipefail exception.

Read: 2 files, +33 -16, the whole diff. Head 97323e8. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

brain-ci weekly trend report: red on 2 of 2 scheduled runs, and its stress step went green over a compile error

1 participant