Skip to content

fix(release): compare generated notes against the previous release on the same lineage - #24507

Merged
iamwhatever merged 1 commit into
mainfrom
fix/release-notes-previous-tag
Oct 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/release-notes-previous-tag

Conversation

@JLargent42

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: each GitHub release's generated "What's Changed" list shows only the pull requests that shipped since the previous release on the same line.

On every release page the generated list is roughly the same ~840 pull requests, starting at #1408 (2026-08-04), and the page has no "Full Changelog" compare line. v0.9.0 and v0.10.0 both show it.

Why it matters

The list is meant to say what a release contains. When it is nearly identical on every release, it hides the real changes, and anyone checking what shipped has to diff tags by hand.

Not a goal

  • Moving, re-cutting or re-creating any existing tag or release page. Past pages keep their bodies; they can be trimmed by hand as docs/build/release.md already describes.
  • Removing the generated list or GitHub's "New Contributors" block from the body.
  • Changing how the CHANGELOG section above the generated list is written or extracted.

What changed (motivation → approach → change)

  • Cause: the release step asks GitHub to generate notes but never names a base tag, so GitHub uses the newest release tag that is an ancestor of the new one. Every release is tagged on its own release branch, so none of the recent tags is an ancestor. For v0.10.0 GitHub fell back to v0.1.2, which gives thousands of entries, and the 125,000-character body limit then cut the list mid-page.
  • Approach: choose the base by version lineage instead of by git ancestry. A stable release compares against the previous stable release. A prerelease compares against the previous prerelease of the same version (rc.1 then insider.2 count as one line), and the first prerelease of a version compares against the last stable release.
  • Change: a new step in the github-release job lists the repository's tags from the API, picks the base with scripts/release_previous_tag.py, and passes it to the release action as previous_tag. When no tag fits, or the tag list cannot be read, the input is empty and GitHub's own choice applies, so the release page is never held back by it. docs/build/release.md now says how the base is chosen.

Measured against the real tags with GitHub's generate-notes API: v0.10.0 picks v0.9.0, v0.11.0-rc.1 picks v0.10.0, and v0.10.0-insider.5 picks v0.10.0-insider.4, a list of 2 pull requests instead of thousands. The stable list is still long, because v0.9.0 to v0.10.0 really is about 1,270 merged pull requests.

Backwards compatibility

Compatible: the release action already accepts previous_tag, and an empty value keeps today's behavior, so a tag the script cannot place still publishes.

Removes nothing: the step only adds an input to the existing release call; no tag, release, asset or body section is removed.

Tests

  • test/test_release_previous_tag.py pins the selection against a slice of the real tag history: stable to previous stable by version (v0.10.0 after v0.9.0, not by name), a patch on an older line, prereleases on one line with a changing label, numeric ordering (insider.10 after insider.9), the first prerelease of a line, no candidate, and a tag outside the release shapes. It also runs the script's command-line form and checks the workflow step passes the chosen base to the release action.
  • test_a_failed_tag_listing_does_not_hold_back_the_release_page pins that a failed tag listing falls back to GitHub's choice instead of failing the step.
  • The other release workflow contract tests still pass.

Proof: PROVEN test/test_release_previous_tag.py::test_the_base_follows_the_release_lineage[v0.10.0-v0.9.0], test/test_release_previous_tag.py::test_the_base_follows_the_release_lineage[v0.9.0-v0.8.0], test/test_release_previous_tag.py::test_the_base_follows_the_release_lineage[v0.8.0-v0.7.2], test/test_release_previous_tag.py::test_the_base_follows_the_release_lineage[v0.9.1-v0.9.0], test/test_release_previous_tag.py::test_the_base_follows_the_release_lineage[v0.7.2-v0.7.1] (head ee28ab7)

Manual verification

  • Ran the script against the repository's live tag list for v0.10.0, v0.9.0, v0.11.0-rc.1, v0.10.0-rc.1, v0.10.0-insider.2, v0.10.0-insider.5, v0.7.2, v0.2.0-rc.1 and v0.1.0; each picked the tag listed above or the previous one on its line.
  • Called the generate-notes API for v0.10.0 with and without the chosen base: without it the compare line names v0.1.2; with it, v0.9.0.
  • Not run: a real release. The step runs only when a release tag is pushed, and this PR creates no tag.

Related Issues

Closes #24461

Sibling sweep

Swept: git grep -nE "generate-notes|generate_release_notes|gh release create" -- .github scripts
Sibling: .github/workflows/build-ffmpeg-decoder.yml -- not affected -- it passes fixed --notes text and generates no list

Pattern harvest

Not generalizable: the only place this repository asks GitHub to generate release notes is the one release step changed here.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners October 26, 2026 17:51
@JLargent42 JLargent42 self-assigned this Oct 26, 2026
@JLargent42
JLargent42 requested a review from buluoray October 26, 2026 17:51
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b9d504dcbcabd26522348a0910a6009460752b4a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b9d504dcbcabd26522348a0910a6009460752b4a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The base tag is chosen where the cause sits, the one place notes get generated is covered, and the fallback keeps today's behaviour.

What this change ships

Inventory (5 items) — 5 justified

Intent: make each release page's "What's Changed" list show only what shipped since the previous release on the same line. FIX.

  1. A stable release's generated list now starts at the previous stable release instead of v0.1.2. — justified
  2. A prerelease's list starts at the previous prerelease on the same X.Y.Z line, and the first prerelease of a line starts at the last stable. — justified
  3. When the tags can't be listed or no base can be chosen, a warning or notice appears and GitHub picks the base as before. — justified
  4. A new scripts/release_previous_tag.py command reads tag names on stdin and prints the base. It reuses the existing check_feed_advance.parse_version, and the release workflow is its only consumer. — justified
  5. The release guide (docs/build/release.md) now says how the base is chosen. — justified

[FIRST-PRINCIPLES-REVIEWED] b9d504d

Description read

[DESCRIPTION-READ] 6098457a6c6ee5309391b35a57a0b8c2ca8e727f50b577a9c154def4e160c9d9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b9d504dcbcabd26522348a0910a6009460752b4a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only adds the previous_tag input, and any failure leaves it empty, so GitHub makes its own choice as before.

Notes

[DESIGN-REVIEWED] b9d504d

Description read

[DESCRIPTION-READ] 7d5b7787b7efcf26b7af435372e0e2d7ccc5f757698a801fad2f9f8b82aaef16

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b9d504dcbcabd26522348a0910a6009460752b4a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b9d504d

Verdict parsed from the review's SHA-scoped output markers for commit b9d504dcbcabd26522348a0910a6009460752b4a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b9d504dcbcabd26522348a0910a6009460752b4a false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings (all downgraded on adjudication)

GPT 6.1 flagged blocking issues on b9d504dcbcabd26522348a0910a6009460752b4a; Opus 5.5 adjudication downgraded every one of them. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

This comment is updated in place on each push.

Review details

BLOCKING -- scripts/release_previous_tag.py:84 -- Unpublished tags omit changes from the next release’s notes

known = [tag for tag in (parse_tag(name) for name in tags) if tag is not None]
Trigger: A required publishing lane fails for v0.11.0-rc.2, so github-release skips its page while the tag remains. When rc.3 succeeds, notes_base lists all tag refs and previous_tag() selects rc.2, not the last published rc.1.
Outcome: The rc.3 generated notes omit pull requests between rc.1 and rc.2, although no intervening release page announced them. Failed stable tags cause the same omission.
Anchor: defect
Fix: Supply candidate tags from published, non-draft GitHub releases, preserving lineage selection and the existing failure fallback.

[BLOCK-MERGE-DOWNGRADED] b9d504d
[GPT-REVIEWED] b9d504d

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — Harm rung: LOW. Release notes are text the maintainer can edit on the page afterwards, and the effect is that some changes are left out, not corrupted (the deferral is stated at scripts/release_previous_tag.py:26-29 and tracked as issue #25077). Confirmed conditions: a release run fails but its tag stays; every tag that parses becomes a candidate (scripts/release_previous_tag.py:84); the highest same-line number below the current one wins (:86-92). Recovery: if no base is chosen, GitHub falls back to its own choice (.github/workflows/release.yml:1388-1391), and the notes can be edited by hand.
Cost of the real fix: the workflow would have to fetch the release list from the GitHub API instead of reading tag refs. That adds a new API dependency and a new failure mode. It also has to handle draft pages correctly: stable pages are created as drafts and only become public at record-stable-promotion (release.yml:1399-1405), so a "non-draft only" filter would drop the newest stable base. Every future reader would carry that extra logic, and it only buys a better starting point for notes that are editable anyway.

[ADJUDICATION] b9d504dcbcabd26522348a0910a6009460752b4a total=1 uphold=0 downgrade=1
DOWNGRADE F1 scripts/release_previous_tag.py:84 reason=disproportionate-remedy
[GPT-ADJUDICATED] b9d504dcbcabd26522348a0910a6009460752b4a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b9d504dcbcabd26522348a0910a6009460752b4a false-positive: <reason, 20 to 500 characters>

@JLargent42

Copy link
Copy Markdown
Contributor Author

CI note for head ee28ab7: macOS Tests / Backend Tests (macOS) (2) is red on two test/test_telegram_dispatcher.py::TestABusyReplyCarriesItsContext cases (44,738 others passed). This PR does not touch that code. The same two cases fail on current main, and they start failing at #23730; its parent passes. This is tracked in #24514, and the fix is open in #24531. A re-run would fail the same way, so I have not re-run it. The PR head stays as is.

@JLargent42

Copy link
Copy Markdown
Contributor Author

CI note for head ee28ab7, follow-up: two more red shards, neither touched by this PR.

@JLargent42

Copy link
Copy Markdown
Contributor Author

CI note for head ee28ab7, correction to my last note: the test_crew_log_fullscan_guard.py reds are not order-dependent. Three strict-xfail guards now pass (XPASS(strict)) on this PR's base, 767556ac. On that base they fail locally on every run. They also fail on Backend Tests (3.12, 2), Backend Tests (Windows) (2) and macOS (3). #24485 (17:57Z) already fixes this on main by dropping the stale xfails, and current main 20c46e97 passes. The 119 teardown errors on macOS (3) follow the same file, which matches the lease leak in #24526.

So every red on this head comes from main:

A re-run reuses this head's merge ref, so it cannot clear either one. I am not re-running anything. The head stays as is until a rebase onto a main that carries both fixes is authorized.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@JLargent42
JLargent42 force-pushed the fix/release-notes-previous-tag branch from ee28ab7 to 01ccad7 Compare October 26, 2026 23:26
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once onto main to pick up the main-side test fixes (#24485, #24531, #24536); no code change.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026
@JLargent42
JLargent42 force-pushed the fix/release-notes-previous-tag branch from 01ccad7 to b65e8c3 Compare October 27, 2026 16:17
@JLargent42

Copy link
Copy Markdown
Contributor Author

Rebased once onto main (0a864a5) to pick up the #24609 Windows shard 6 fix; no code change.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (AI findings unanswered) at b65e8c344bd471b5246fd070a65a44db2e21f43a.

The fix addresses the real cause: the release step now passes a lineage-chosen previous_tag, which the pinned softprops/action-gh-release v3.0.3 supports, and a failed tag listing falls back to GitHub's own choice. Checks are green and the step uses the job's existing token with no new permissions.

Before approval, please post a disposition for these current-head review items:

  1. GPT 6.1 F1 (blocking, downgraded on adjudication): known = ... takes every v* tag ref, so a prerelease tag whose release run failed becomes the next prerelease's base. Either rebut it (for example, failed tags are deleted or re-cut) or accept it as a known limitation with a linked issue.
  2. First Principles note: the new parser duplicates scripts/check_feed_advance.py::parse_version and already disagrees with it on beta-preview.1. Reuse it, or say why a separate parser is wanted.
  3. Design note (minor): an error in the python3 call still fails the step under set -euo pipefail; || previous="" would match the listing fallback.

— crew-pr-reviewer (r130)

@JLargent42
JLargent42 force-pushed the fix/release-notes-previous-tag branch from b65e8c3 to b9d504d Compare October 28, 2026 00:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 28, 2026
@JLargent42

JLargent42 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition, GPT 6.1 F1 (span=bdeb326d319b): accepted as a known limitation, tracked in #25077

Finding: previous_tag() takes every v* tag ref, so a prerelease tag whose release run failed (no page published) becomes the next prerelease's notes base, and the pull requests between the last published prerelease and that tag appear on no page.

Why not fixed here: the remedy is reading published releases instead of tag refs. Stable pages are created as drafts and published later by record-stable-promotion, so a plain "published only" filter can skip the previous stable when two stable runs overlap; doing it right needs a draft/promotion ordering contract that this fix for #24461 does not. The harm is limited to the generated list: the release is unaffected, the page names its base in the "Full Changelog" link, and the notes can be edited. Adjudication reached the same call (downgraded as disproportionate).

Recorded: the limitation is stated in the scripts/release_previous_tag.py docstring and in docs/build/release.md, both linking #25077, on head b9d504dcbcabd26522348a0910a6009460752b4a.

@JLargent42

Copy link
Copy Markdown
Contributor Author

Disposition, First Principles note (duplicate tag parser): fixed on b9d504dcbcabd26522348a0910a6009460752b4a

  • _TAG and its regex are deleted. scripts/release_previous_tag.py::parse_tag now strips refs/tags/, requires the v prefix, and calls scripts/check_feed_advance.py::parse_version (loaded by path, since scripts/ is not a package). The base is (major, minor, patch) and the prerelease number is parse_version's pre_number, None for a stable tag.
  • The two now agree on beta-preview.1 and on label-less prereleases like v1.0.0-beta. New tests: test_tags_read_the_same_as_the_publish_guard compares both parsers on five spellings, and test_a_hyphenated_prerelease_label_follows_its_line pins v0.12.0-beta-preview.2 -> v0.12.0-beta-preview.1. Both fail on the previous head.

@JLargent42

Copy link
Copy Markdown
Contributor Author

Disposition, Design note (script error still fails the step): fixed on b9d504dcbcabd26522348a0910a6009460752b4a

  • The python3 scripts/release_previous_tag.py call in .github/workflows/release.yml is now wrapped the same way as the tag listing: if ! previous=$(...); then warns and sets previous="", so GitHub chooses the base and the release page is still created.
  • New test test_a_failed_choice_does_not_hold_back_the_release_page runs the step's own run: body under bash with stub gh and a failing python3, and asserts exit 0 with previous_tag= written to GITHUB_OUTPUT. It fails on the previous head.

@JLargent42

Copy link
Copy Markdown
Contributor Author

@iamwhatever thanks for the HOLD (r130 on b65e8c34). All three items are answered; the new head is b9d504dcbcabd26522348a0910a6009460752b4a.

  1. GPT 6.1 F1: accepted as a known limitation, tracked in Release notes base can be a prerelease tag whose release never published #25077 and noted in the script docstring and docs/build/release.md. Reason: the fix (read published releases instead of tags) has to fit the stable draft-then-promote flow, and the effect is limited to the editable generated list.
  2. First Principles: the duplicate parser is gone; parse_tag now calls scripts/check_feed_advance.py::parse_version, so both agree on beta-preview.1.
  3. Design: the python3 call now falls back to an empty base like the listing does, with a test that runs the step body under bash.

Locally before the push: pytest -n 2 over the PR's test file plus 29 release/workflow test files, 1,658 passed, 3 skipped, 0 failed; the 4 new tests fail on the previous head. black (py310), isort, flake8, mypy on the script, the black/subprocess-encoding/comment-history gates and docs-lint.sh are clean. Rebase not needed (not DIRTY). Dispositions are posted above.

@JLargent42

Copy link
Copy Markdown
Contributor Author

CI note for head b9d504dcbcabd26522348a0910a6009460752b4a: Backend Tests (3.12, 2) failed on 2 of 31,654 cases, test/test_session.py::TestWarmPool::test_cold_start_for_new_session and ::test_close_all_shuts_down_sessions (CapabilityStartupError: capability_harness_unsupported). This PR changes only the release workflow step, scripts/release_previous_tag.py, its test and docs/build/release.md; none of those runs in shard 2 or touches session capabilities. TestWarmPool passes 15/15 locally on this head, so the failure depends on test order within the shard. The failed job gets one re-run once its workflow run finishes.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 28, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at b9d504dcbcabd26522348a0910a6009460752b4a.

All three r130 items are answered on this head. The script now reuses check_feed_advance.parse_version. The new step falls back to GitHub's choice when either the tag listing or the choice fails. The failed-prerelease base limitation is documented and tracked in #25077, which covers it. The step adds no permissions or secrets and passes the tag through env. All AI lanes are answered, and checks are green.

— crew-pr-reviewer (r131)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 28, 2026 06:27
@iamwhatever
iamwhatever merged commit adb562f into main Oct 28, 2026
161 of 163 checks passed
@iamwhatever
iamwhatever deleted the fix/release-notes-previous-tag branch October 28, 2026 06:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release notes "What's Changed" lists hundreds of old PRs on every release

2 participants