Skip to content

test: key the golden extents and pipeline exports per entry, not by a shared total - #37679

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/merge-safe-budgets
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/merge-safe-budgets

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The memory golden and the channel-turn export tests stay green on main when two pull requests that were each green on the same base both merge.

Each stored one shared total beside or instead of a per-entry record. Two PRs that each change a different entry rewrite the total to the same value, git merges it cleanly, and main holds a different total than the number says. EXPECTED_TOTAL = 8003 in test/test_memory_v1_golden.py was re-pinned 4 times in 30 days. _EXPORT_BUDGET = 31 in test/test_channel_turn_structure.py sat exactly at its measured value, so any two PRs that each add an import collide.

Why it matters

A red main fails every open PR's CI until someone re-pins the number.

Not a goal

  • No product code changes; tests only.
  • No loosening. Each golden block length stays exact, and the export record now catches a swap (one name removed, another added) that the count missed.
  • Byte ceilings stay as they are: the SKILL.md byte caps in test/test_review_repair_routing_skill.py and PROMPT_BYTE_CEILINGS are real ceilings, where a red after a merge means the file really grew past its budget.

What changed (motivation → approach → change)

Test Why it went red after a merge Change
test/test_memory_v1_golden.py EXPECTED_TOTAL duplicates the sum of the exact per-block EXPECTED_EXTENTS Drop the total. Keep the coverage check that the blocks sum to the payload length, and every block length exact
test/test_channel_turn_structure.py _EXPORT_BUDGET = 31, one shared count of imported pipeline names _EXPORTED_NAMES, one module.name per line. A new import fails naming it; a retired one fails asking to delete its line

The rule is the one in docs/system-specs/common/testing-conventions.md § Merge-safe ratchets and counts (PR #37660).

Backwards compatibility

Compatible: test files only.

Removes nothing: the golden total is implied by the exact block lengths and the coverage check that stay; the export count is replaced by a stricter exact set.

Tests

Two-branch merge simulation (scratch clone; A and B cut from one base, each makes one change of the same kind and updates the record exactly as the test asks, then git merge):

Scenario origin/main this branch negative control (this branch, change made but record not updated)
golden blocks: A and B each grow a different block by 7 characters clean merge, FAILED assert 8017 == 8010 clean merge, 1 passed FAILED, per-block dict mismatch (242 vs 235)
export names: A and B each import a different new name clean merge, FAILED 33 names are imported ... over the budget of 32 clean merge, 9 passed FAILED new names imported from the pipeline's modules: [...]

On this branch rebased onto current main: both changed files plus test/test_flake_pattern_ratchet.py, 111 passed; black, isort and flake8 clean.

Proof: not provable -- test-only change

Manual verification

N/A — tests only; the simulation above is the behavioral check.

Related Issues

Part of #37656

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 19:41
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 02d6cf9add36372053e6dea15f772d5f19e6beea touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 02d6cf9add36372053e6dea15f772d5f19e6beea ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 02d6cf9add36372053e6dea15f772d5f19e6beea — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. Each per-entry record is at least as strict as the shared total it replaces.

[DESIGN-REVIEWED] 02d6cf9

Description read

[DESCRIPTION-READ] 3a10cc3fe83532ff6297096f50f69ed735bf7e48d8764fb293fdcccb415fe6b8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 02d6cf9add36372053e6dea15f772d5f19e6beea — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 02d6cf9

Verdict parsed from the review's SHA-scoped output markers for commit 02d6cf9add36372053e6dea15f772d5f19e6beea.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 02d6cf9add36372053e6dea15f772d5f19e6beea false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 02d6cf9add36372053e6dea15f772d5f19e6beea and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 02d6cf9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 02d6cf9add36372053e6dea15f772d5f19e6beea false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 4 commits (d470c46 -> 57eba16), one of which re-pinned test/test_memory_v1_golden.py's EXPECTED_TOTAL to 8776 and two block extents ([CRITICAL RULES 5057, [Learned corrections 374) while this PR removes the total. Resolved by keeping main's extents and the removal; the per-block extents and the sum == payload length check still decide. Re-ran test_memory_v1_golden.py and test_channel_turn_structure.py on the new base: 43 passed. New head: 02d6cf9.

@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/merge-safe-budgets branch from 80334b7 to 02d6cf9 Compare November 15, 2026 20:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only change. In the memory golden test, the removed EXPECTED_TOTAL (8776) equals the sum of the exact per-block EXPECTED_EXTENTS, and the payload coverage check stays, so nothing is loosened. In the export test, the shared count is replaced by an exact set of 31 names, one per line, checked in both directions, which is stricter than the count. CI is green on this head, so the set matches the current imports. No other code reads the removed constants.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 02d6cf9add · 70 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 15, 2026 22:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change that replaces a shared export count and a pinned golden total with per-entry sets, touching only test/ files.

@KiroCrewCaptain
KiroCrewCaptain merged commit 4507e9a into main Nov 15, 2026
98 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/merge-safe-budgets branch November 15, 2026 22:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants