From edd1ee15216d29f6b8a983bbf2430aec62650258 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 31 Aug 2026 12:12:49 -0700 Subject: [PATCH 1/5] fix(just): widen to the unscoped suite when a diff outgrows argv `_changed` prints the changed-file list and every hop of the dispatch takes it as one argument (`just _tools "$files"`, `just js check "$files"`, `just rs check-changed "$files"` -> `just rs _select "$FILES"`). just exports recipe parameters into the child's environment, so it travels as a single execve string, and Linux caps one string at MAX_ARG_STRLEN (32 pages) however large ARG_MAX is. Past that every hop dies with E2BIG, which just reports as exit code 126 and "Argument list too long", naming neither the diff nor the recipe that could not receive it. Budget the list and print the `ALL` sentinel the repo already uses when it does not fit, so `check`, `fix`, and `test` fall back to `check-all` / `fix-all` / `test all`, which pass no list at all. Batching cannot work here: each consumer needs the whole list for one scoping decision, so a chunked list would silently under-select. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 2 +- justfile | 91 ++++++++++++++++++++++++++++++++++++++++++++++++--- test/justfile | 7 ++++ 3 files changed, 95 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 73178ed4cf0..3fe77aa25bd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,7 +28,7 @@ revision first and print base-to-current changes. Timing changes are information benchmark crashes, zero delivery, and invalid samples still fail the command. Relay CPU and RSS are included on Linux, where `moq-bench-host` can read `/proc`. -`just check`, `just test`, and `just fix` all diff the branch against its base and touch only the crates that changed plus everything depending on them, which is what keeps them fast when several worktrees are building at once. They skip a language entirely when the diff doesn't touch it. Reach for `just check-all` / `just test all` / `just fix-all` when you want the unscoped suite. See [Workflow](#workflow) for how the base is resolved. +`just check`, `just test`, and `just fix` all diff the branch against its base and touch only the crates that changed plus everything depending on them, which is what keeps them fast when several worktrees are building at once. They skip a language entirely when the diff doesn't touch it. Reach for `just check-all` / `just test all` / `just fix-all` when you want the unscoped suite. They also switch to it themselves once the changed-file list outgrows what a single argv/env string can hold (64 KiB, roughly 1500 paths), because the list is passed to each per-language recipe as one argument. See [Workflow](#workflow) for how the base is resolved. To force a base, `just check origin/dev` and `just fix origin/dev` take it positionally. `just test` can't: it's a module, so `just test origin/dev` looks for a *recipe* named `origin/dev`. Name the recipe to get past that: `just test default origin/dev`. diff --git a/justfile b/justfile index 3609cda9ebd..23fd914af78 100644 --- a/justfile +++ b/justfile @@ -28,6 +28,10 @@ mod relay 'demo/relay' mod sub 'demo/sub' mod web 'demo/web' +# Byte budget for a changed-file list, sized so it survives being passed as a +# single argv/env string on every hop of the dispatch. See `_changed`. +changed_max := '65536' + # Run the demo by default. default: just demo @@ -47,10 +51,13 @@ install: cargo install --locked cargo-shear cargo-sort cargo-upgrades cargo-edit cargo-semver-checks release-plz # Reports the base it picked on stderr, so a surprising scope is traceable. +# +# LIMIT is the byte budget for the printed list, and exists as a parameter so +# `_changed-test` can force the oversized path without a synthetic 30k-file diff. -# Print the files this branch changed relative to BASE, one per line. +# Print the files this branch changed relative to BASE, one per line, or `ALL`. [private] -_changed $BASE: +_changed $BASE $LIMIT=changed_max: #!/usr/bin/env bash set -euo pipefail @@ -79,10 +86,71 @@ _changed $BASE: echo "base: $base" >&2 # Untracked files count too: a brand new crate or module is the whole change. - { + files=$({ git diff --name-only "$merge_base" git ls-files --others --exclude-standard - } | sort -u + } | sort -u) + + # Every hop of the dispatch takes this list as one argument, and just exports + # recipe parameters into the child's environment, so the whole list has to + # fit in a single execve string. Linux caps one string at MAX_ARG_STRLEN (32 + # pages, 131072 bytes) however large ARG_MAX is, so past that each hop dies + # with E2BIG, which just reports as exit code 126 and no mention of the diff. + # A diff that large selects most of the workspace anyway, so say `ALL` and + # let the caller widen to the unscoped suite, which passes no list at all. + [[ "$LIMIT" =~ ^[0-9]+$ ]] || { + echo "changed: LIMIT must be a byte count, got: $LIMIT" >&2 + exit 2 + } + if ((${#files} > LIMIT)); then + echo "changed: ${#files} bytes of paths exceeds the $LIMIT budget; selecting everything." >&2 + echo ALL + exit 0 + fi + + # Guarded because a bare printf of an empty list prints a newline, and the + # callers test the result for emptiness to decide whether anything changed. + if [[ -n "$files" ]]; then + printf '%s\n' "$files" + fi + +# Guards the thing that fails LOUDLY but unhelpfully: past the budget every +# `just` hop dies with "Argument list too long" and exit 126, naming neither the +# diff nor the recipe that could not receive it. Both halves matter -- a budget +# that never trips scopes nothing, and one above what execve accepts still dies. + +# Check that an oversized diff widens to `ALL`, and that the budget fits in argv. +[private] +_changed-test $LIMIT=changed_max: + #!/usr/bin/env bash + set -euo pipefail + + fail() { echo "changed: _changed-test: $1" >&2; exit 1; } + + # Any real diff is oversized against a one-byte budget, so this exercises the + # fallback without laying down a synthetic 30k-file corpus to trigger it. + [[ "$(just _changed "" 1)" == ALL ]] || fail "a diff over budget must print ALL" + + # ...and it must still be able to say no, or the scoping is gone. + [[ "$(just _changed "" 100000000)" != ALL ]] || fail "a diff under budget must print the list" + + # Linux caps a single argv/env string at MAX_ARG_STRLEN (32 pages), whatever + # ARG_MAX says, and the list travels as one string. Asserted rather than + # probed because this repo's CI is the Linux host and a dev box may be laxer. + ((LIMIT <= 131072)) || fail "budget $LIMIT exceeds Linux MAX_ARG_STRLEN" + + # The budget still has to survive the hop it was sized for, which the two + # tests above cannot show: they never pass a list that big to anything. + payload=$(head -c "$LIMIT" /dev/zero | tr '\0' x) + [[ "$(just _echo "$payload" | wc -c)" -eq $((LIMIT + 1)) ]] \ + || fail "a $LIMIT-byte list does not survive an argv hop" + + echo "changed: budget ok" + +# Print an argument back, to measure what survives a `just` invocation. +[private] +_echo $VALUE: + @printf '%s\n' "$VALUE" # Tools every scope guards with `command -v`, so an incomplete local toolchain # checks less instead of failing. That trade is wrong in CI, where a skip is @@ -149,6 +217,14 @@ check $BASE="": set -euo pipefail files=$(just _changed "$BASE") + + # `_changed` says ALL when the list outgrew what argv can carry. The unscoped + # suite is the path that passes no list at all, so it is the one that works. + if [[ "$files" == ALL ]]; then + just check-all + exit 0 + fi + just _tools "$files" # The dispatch below lives in these two files, and neither matches any @@ -267,6 +343,7 @@ _shell $ACTION: # Repository-wide lints, shared by `check` and `check-all`. [private] _check-common: + just _changed-test bun install --frozen-lockfile bun remark . --quiet --frail just _shell check @@ -285,6 +362,12 @@ fix $BASE="": files=$(just _changed "$BASE") + # Mirrors `check`: too long for argv means fix everything instead. + if [[ "$files" == ALL ]]; then + just fix-all + exit 0 + fi + if [[ -n "$files" ]]; then just js fix "$files" just rs fix-changed "$files" diff --git a/test/justfile b/test/justfile index 5af6a60a712..5b775991811 100644 --- a/test/justfile +++ b/test/justfile @@ -31,6 +31,13 @@ default $BASE="": set -euo pipefail files=$(just _changed "$BASE") + + # Mirrors the root `check`: too long for argv means test everything instead. + if [[ "$files" == ALL ]]; then + just all + exit 0 + fi + just _tools "$files" # Mirrors the root `check`: the dispatch below lives in these two files and From 593159080c5d952f73f9ec3b93069494c80d9ac6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 31 Aug 2026 12:21:22 -0700 Subject: [PATCH 2/5] fix(just): decouple the changed-list test from the working tree `_changed-test` asserted that `just _changed "" 1` prints ALL, which only holds when the working tree has a diff. cache.yml runs `just check-all` on a clean `main` checkout to warm the shared Rust cache, where the list is empty and the assertion aborts `_check-common` before the suite runs, failing the one job allowed to write that cache. Split the size decision into `_changed-cap`, a pure function of a byte count, so the test drives it with synthetic sizes instead of the ambient diff. The end-to-end assertion stays, guarded on there actually being a diff. Co-Authored-By: Claude Opus 5 --- justfile | 68 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/justfile b/justfile index 23fd914af78..0322751feb5 100644 --- a/justfile +++ b/justfile @@ -91,19 +91,7 @@ _changed $BASE $LIMIT=changed_max: git ls-files --others --exclude-standard } | sort -u) - # Every hop of the dispatch takes this list as one argument, and just exports - # recipe parameters into the child's environment, so the whole list has to - # fit in a single execve string. Linux caps one string at MAX_ARG_STRLEN (32 - # pages, 131072 bytes) however large ARG_MAX is, so past that each hop dies - # with E2BIG, which just reports as exit code 126 and no mention of the diff. - # A diff that large selects most of the workspace anyway, so say `ALL` and - # let the caller widen to the unscoped suite, which passes no list at all. - [[ "$LIMIT" =~ ^[0-9]+$ ]] || { - echo "changed: LIMIT must be a byte count, got: $LIMIT" >&2 - exit 2 - } - if ((${#files} > LIMIT)); then - echo "changed: ${#files} bytes of paths exceeds the $LIMIT budget; selecting everything." >&2 + if [[ "$(just _changed-cap "${#files}" "$LIMIT")" == ALL ]]; then echo ALL exit 0 fi @@ -114,6 +102,36 @@ _changed $BASE $LIMIT=changed_max: printf '%s\n' "$files" fi +# Every hop of the dispatch takes the changed-file list as one argument, and +# just exports recipe parameters into the child's environment, so the whole list +# has to fit in a single execve string. Linux caps one string at MAX_ARG_STRLEN +# (32 pages, 131072 bytes) however large ARG_MAX is, so past that each hop dies +# with E2BIG, which just reports as exit code 126 and no mention of the diff. A +# diff that large selects most of the workspace anyway, so the callers widen to +# the unscoped suite, which passes no list at all. +# +# Split out from `_changed` so the decision is a pure function of a byte count: +# `_changed-test` drives it with synthetic sizes, rather than needing the working +# tree to hold a diff of a particular size. + +# Print `ALL` when BYTES of paths is too much to pass through a single argument. +[private] +_changed-cap $BYTES $LIMIT=changed_max: + #!/usr/bin/env bash + set -euo pipefail + + for n in "$BYTES" "$LIMIT"; do + [[ "$n" =~ ^[0-9]+$ ]] || { + echo "changed: not a byte count: $n" >&2 + exit 2 + } + done + + if ((BYTES > LIMIT)); then + echo "changed: $BYTES bytes of paths exceeds the $LIMIT budget; selecting everything." >&2 + echo ALL + fi + # Guards the thing that fails LOUDLY but unhelpfully: past the budget every # `just` hop dies with "Argument list too long" and exit 126, naming neither the # diff nor the recipe that could not receive it. Both halves matter -- a budget @@ -127,24 +145,34 @@ _changed-test $LIMIT=changed_max: fail() { echo "changed: _changed-test: $1" >&2; exit 1; } - # Any real diff is oversized against a one-byte budget, so this exercises the - # fallback without laying down a synthetic 30k-file corpus to trigger it. - [[ "$(just _changed "" 1)" == ALL ]] || fail "a diff over budget must print ALL" + # Synthetic sizes rather than whatever the working tree happens to hold: a + # clean checkout has no diff at all, and `check-all` runs there (cache.yml + # warms the cache from `main`), so a test keyed on the real list would take + # down the one job allowed to write the shared Rust cache. + [[ "$(just _changed-cap 100 99)" == ALL ]] || fail "over budget must print ALL" + [[ -z "$(just _changed-cap 99 99)" ]] || fail "at budget must print nothing" + [[ -z "$(just _changed-cap 0 99)" ]] || fail "an empty diff must print nothing" - # ...and it must still be able to say no, or the scoping is gone. - [[ "$(just _changed "" 100000000)" != ALL ]] || fail "a diff under budget must print the list" + # A byte count is the whole input, so anything else is a caller bug, not a + # reason to silently scope to nothing. + ! just _changed-cap 1 not-a-number 2> /dev/null || fail "a bad budget must be rejected" # Linux caps a single argv/env string at MAX_ARG_STRLEN (32 pages), whatever # ARG_MAX says, and the list travels as one string. Asserted rather than # probed because this repo's CI is the Linux host and a dev box may be laxer. ((LIMIT <= 131072)) || fail "budget $LIMIT exceeds Linux MAX_ARG_STRLEN" - # The budget still has to survive the hop it was sized for, which the two - # tests above cannot show: they never pass a list that big to anything. + # The budget still has to survive the hop it was sized for, which the checks + # above cannot show: they never pass a list that big to anything. payload=$(head -c "$LIMIT" /dev/zero | tr '\0' x) [[ "$(just _echo "$payload" | wc -c)" -eq $((LIMIT + 1)) ]] \ || fail "a $LIMIT-byte list does not survive an argv hop" + # End to end, but only when there is a diff to be oversized: see above. + if [[ -n "$(just _changed "" 100000000)" ]]; then + [[ "$(just _changed "" 1)" == ALL ]] || fail "a diff over budget must print ALL" + fi + echo "changed: budget ok" # Print an argument back, to measure what survives a `just` invocation. From c9708979e39fd1b10dc2535f79d474a1668580dd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 31 Aug 2026 12:26:46 -0700 Subject: [PATCH 3/5] fix(just): budget the changed list in bytes, not characters `${#files}` counts characters under a UTF-8 locale while execve counts bytes, so a path set of 3-byte characters read as a third of its real size and could sail past a budget it actually blows, back into the E2BIG this guards against. Feed the list to `_changed-cap` on stdin and measure it with `wc -c`. Stdin is both the only channel that can carry an oversized list and the only honest way to measure one. Co-Authored-By: Claude Opus 5 --- justfile | 48 ++++++++++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/justfile b/justfile index 0322751feb5..4f0338812bd 100644 --- a/justfile +++ b/justfile @@ -91,7 +91,7 @@ _changed $BASE $LIMIT=changed_max: git ls-files --others --exclude-standard } | sort -u) - if [[ "$(just _changed-cap "${#files}" "$LIMIT")" == ALL ]]; then + if [[ "$(printf '%s' "$files" | just _changed-cap "$LIMIT")" == ALL ]]; then echo ALL exit 0 fi @@ -110,25 +110,30 @@ _changed $BASE $LIMIT=changed_max: # diff that large selects most of the workspace anyway, so the callers widen to # the unscoped suite, which passes no list at all. # -# Split out from `_changed` so the decision is a pure function of a byte count: -# `_changed-test` drives it with synthetic sizes, rather than needing the working -# tree to hold a diff of a particular size. +# Split out from `_changed` so `_changed-test` can drive the decision with a +# synthetic list, rather than needing the working tree to hold a diff of a +# particular size. +# +# The list arrives on stdin, which is both the only channel that can carry an +# oversized one and the only way to measure it honestly: `${#var}` counts +# CHARACTERS under a UTF-8 locale while execve counts BYTES, so a path set of +# 3-byte characters would read as a third of its real size and sail past a +# budget it actually blows. -# Print `ALL` when BYTES of paths is too much to pass through a single argument. +# Print `ALL` when the changed-file list on stdin is too long for one argument. [private] -_changed-cap $BYTES $LIMIT=changed_max: +_changed-cap $LIMIT=changed_max: #!/usr/bin/env bash set -euo pipefail - for n in "$BYTES" "$LIMIT"; do - [[ "$n" =~ ^[0-9]+$ ]] || { - echo "changed: not a byte count: $n" >&2 - exit 2 - } - done + [[ "$LIMIT" =~ ^[0-9]+$ ]] || { + echo "changed: not a byte count: $LIMIT" >&2 + exit 2 + } - if ((BYTES > LIMIT)); then - echo "changed: $BYTES bytes of paths exceeds the $LIMIT budget; selecting everything." >&2 + bytes=$(wc -c | tr -d '[:space:]') + if ((bytes > LIMIT)); then + echo "changed: $bytes bytes of paths exceeds the $LIMIT budget; selecting everything." >&2 echo ALL fi @@ -149,13 +154,20 @@ _changed-test $LIMIT=changed_max: # clean checkout has no diff at all, and `check-all` runs there (cache.yml # warms the cache from `main`), so a test keyed on the real list would take # down the one job allowed to write the shared Rust cache. - [[ "$(just _changed-cap 100 99)" == ALL ]] || fail "over budget must print ALL" - [[ -z "$(just _changed-cap 99 99)" ]] || fail "at budget must print nothing" - [[ -z "$(just _changed-cap 0 99)" ]] || fail "an empty diff must print nothing" + [[ "$(printf 'aaaa' | just _changed-cap 3)" == ALL ]] || fail "over budget must print ALL" + [[ -z "$(printf 'aaa' | just _changed-cap 3)" ]] || fail "at budget must print nothing" + [[ -z "$(printf '' | just _changed-cap 3)" ]] || fail "an empty diff must print nothing" + + # execve counts bytes, so the budget has to as well. Spelled as raw bytes + # rather than as characters: this is one CJK character, 3 bytes wide, which + # `${#var}` would count as 1 under a UTF-8 locale and let past a 2-byte + # budget it actually blows. + [[ "$(printf '\xe6\x97\xa5' | just _changed-cap 2)" == ALL ]] \ + || fail "the budget must count bytes, not characters" # A byte count is the whole input, so anything else is a caller bug, not a # reason to silently scope to nothing. - ! just _changed-cap 1 not-a-number 2> /dev/null || fail "a bad budget must be rejected" + ! printf '' | just _changed-cap not-a-number 2> /dev/null || fail "a bad budget must be rejected" # Linux caps a single argv/env string at MAX_ARG_STRLEN (32 pages), whatever # ARG_MAX says, and the list travels as one string. Asserted rather than From 31dfdeba13c2ad52f633a32d2e3ff834a40e72e2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 31 Aug 2026 12:30:09 -0700 Subject: [PATCH 4/5] fix(just): fail when the budget check itself fails Inside `[[ ]]` the exit status of a command substitution is discarded, so a rejected budget left `_changed` printing a list that never got budgeted, which on a large diff walks straight back into the E2BIG this recipe prevents. Assign first so `set -e` sees the failure, and assert it in `_changed-test`. Co-Authored-By: Claude Opus 5 --- justfile | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/justfile b/justfile index 4f0338812bd..38bffc370ab 100644 --- a/justfile +++ b/justfile @@ -91,7 +91,12 @@ _changed $BASE $LIMIT=changed_max: git ls-files --others --exclude-standard } | sort -u) - if [[ "$(printf '%s' "$files" | just _changed-cap "$LIMIT")" == ALL ]]; then + # Assigned rather than tested inline, so a failing `_changed-cap` aborts here + # under `set -e`. Inside `[[ ]]` its exit status is discarded, and the caller + # would then scope to a list that never got budgeted -- straight back into the + # E2BIG this exists to prevent. + cap=$(printf '%s' "$files" | just _changed-cap "$LIMIT") + if [[ "$cap" == ALL ]]; then echo ALL exit 0 fi @@ -169,6 +174,10 @@ _changed-test $LIMIT=changed_max: # reason to silently scope to nothing. ! printf '' | just _changed-cap not-a-number 2> /dev/null || fail "a bad budget must be rejected" + # ...and that rejection has to reach the caller. A swallowed one would hand + # back an unbudgeted list, which is the failure this whole recipe prevents. + ! just _changed "" not-a-number 2> /dev/null || fail "a rejected budget must fail _changed" + # Linux caps a single argv/env string at MAX_ARG_STRLEN (32 pages), whatever # ARG_MAX says, and the list travels as one string. Asserted rather than # probed because this repo's CI is the Linux host and a dev box may be laxer. From 8067b45f0343d6e962b031dae8cf3e25069ce971 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 31 Aug 2026 12:37:01 -0700 Subject: [PATCH 5/5] fix(just): budget the end-to-end cap test at zero, not one A checkout whose only change is a one-character root path yields a one-byte list, which a one-byte budget does not exceed, so `_changed` returned the list instead of ALL and the assertion failed every `just check`. Zero is below every nonempty list. Covered synthetically too, so the boundary no longer depends on what the checkout happens to hold. Co-Authored-By: Claude Opus 5 --- justfile | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/justfile b/justfile index 38bffc370ab..6815a8e327c 100644 --- a/justfile +++ b/justfile @@ -163,6 +163,9 @@ _changed-test $LIMIT=changed_max: [[ -z "$(printf 'aaa' | just _changed-cap 3)" ]] || fail "at budget must print nothing" [[ -z "$(printf '' | just _changed-cap 3)" ]] || fail "an empty diff must print nothing" + # The smallest nonempty list there is, against the only budget below it. + [[ "$(printf 'x' | just _changed-cap 0)" == ALL ]] || fail "a 1-byte list must exceed a 0 budget" + # execve counts bytes, so the budget has to as well. Spelled as raw bytes # rather than as characters: this is one CJK character, 3 bytes wide, which # `${#var}` would count as 1 under a UTF-8 locale and let past a 2-byte @@ -189,9 +192,12 @@ _changed-test $LIMIT=changed_max: [[ "$(just _echo "$payload" | wc -c)" -eq $((LIMIT + 1)) ]] \ || fail "a $LIMIT-byte list does not survive an argv hop" - # End to end, but only when there is a diff to be oversized: see above. + # End to end, but only when there is a diff to be oversized: see above. The + # budget is zero rather than one because the real list is whatever the + # checkout holds, and a single one-character root path is a one-byte list + # that a one-byte budget does not exceed. Zero is below every nonempty list. if [[ -n "$(just _changed "" 100000000)" ]]; then - [[ "$(just _changed "" 1)" == ALL ]] || fail "a diff over budget must print ALL" + [[ "$(just _changed "" 0)" == ALL ]] || fail "a diff over budget must print ALL" fi echo "changed: budget ok"