Skip to content

test: pin nested-payload linearity structurally, not by wall-clock (#8687) - #8702

Merged
iamwhatever merged 1 commit into
mainfrom
fix/nested-payload-linearity-structural-8687
Sep 5, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/nested-payload-linearity-structural-8687

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Summary

Converts the two remaining wall-clock-ratio pins in TestNestedPayloadExtractionIsLinear to deterministic structural call-count assertions, mirroring the two siblings already in the file (TestSelfModuleIndexIsLinear::test_the_scan_is_linear_not_quadratic, test_a_chain_of_glued_redirects_is_linear).

The ratio form measured the runner, not the code: on a starved shared Windows runner (PR #8672, Backend Tests (Windows) shard 2), scheduler noise alone produced a 3.1x ratio against the 3x bound and false-redded a PR whose diff never touched _nested_shell_payloads.

What the tests now pin

test_the_scan_is_linear_not_quadratic (padding shape ["bash", "-x"] * (n // 2), sizes 2000/4000/6000):

  1. _next_stop_indexes runs exactly TWICE per call (env stop table + past-the-dashes table), regardless of n
  2. _program_basename call count is an exact arithmetic progression (once per token; the pre-fix quadratic re-walked the tail per shell token)
  3. _is_shell_command_flag and _is_not_double_dash call counts hold exact arithmetic progressions — the predicates a per-shell-token forward re-walk must re-consult. The padding token is dash-prefixed so the flag predicate is genuinely consulted (n/2 calls) rather than holding vacuously.

test_a_long_double_dash_run_is_also_linear (shape ["$0"]*n + ["-c"] + ["--"]*n):

  1. _next_stop_indexes == 2 pinned
  2. _is_not_double_dash count is an exact arithmetic progression in n (the pre-fix third forward walk re-consulted it once per run token PER program token)

No absolute wall-clock cap, per the file's own #8630 precedent (coverage tracing prices line events). Both docstrings state the structural contract and the one residual (an inlined comparison is not observed by the counts) plainly. test_the_payload_set_is_unchanged untouched — it pins the payload set.

Testing

  • Standalone probe against the fixed code, 3 consecutive runs, byte-identical counts each run: shape 1 gives tables=2, basename=n, flag=n/2, dash=n; shape 2 gives tables=2, dash=2n+1; both shapes yield [] (the empty-payload worst case)
  • Local gates green: isort, flake8, mypy (1293 files), diff-scoped black gate (run post-commit)
  • Full pytest runs in CI (host policy: no local test suites)

Pre-push review

  • GPT lane (pinned gpt-5.6-sol): PASS, no findings
  • Opus lane (pinned claude-opus-5): PASS, 2 Medium + 1 Low advisory — all three fixed: vacuous flag-count series given real observation via dash-prefixed padding; docstrings now state the inlined-comparison residual instead of claiming exhaustiveness; assertion floors derived from the bound sizes tuple instead of duplicated literals

Closes #8687

@github-actions

github-actions Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 71ba3c547a684c12cf32c7b8832b9dc738063bb1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 71ba3c5

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 71ba3c547a684c12cf32c7b8832b9dc738063bb1: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 71ba3c547a684c12cf32c7b8832b9dc738063bb1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Helpers are module-level functions resolved through module globals at call time, so the monkeypatch instrumentation genuinely intercepts the path under test, and the test's own floor assertions guard against the instrument silently observing nothing. The change is exactly what the repo's testing conventions mandate (assert complexity by shape, not duration), mirrors the two sibling tests already converted in the same file, and states its one residual blind spot honestly. No design-level concerns.

Design-Verdict: PASS

Deterministic call-count pins replace a runner-dependent timing ratio, matching the file's established structural pattern and the repo's own flake-class doctrine; residuals stated honestly.

[DESIGN-REVIEWED] 71ba3c5

@github-actions

github-actions Bot commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 71ba3c547a684c12cf32c7b8832b9dc738063bb1 — this comment is updated in place on each push.

Review details

I've verified both replacement tests against the actual security.py implementation:

Test 1 (["bash","-x"]*(n//2), sizes 2000/4000/6000): _next_stop_indexes is called exactly twice (env_stop line 4131, past_dashes line 4184) → tables==2. _program_basename = n (main loop, one per token; sweep and array branches don't fire). _is_shell_command_flag = n/2 (only -x tokens hit line 4165). _is_not_double_dash = n (past_dashes build via monkeypatched global). All floors and arithmetic progressions hold exactly, and -x yields no payload (no c, not --command), so == [] holds.

Test 2 (["$0"]*n + ["-c"] + ["--"]*n): tables==2, dash count = 2n+1 (past_dashes build), progression exact. The -- run reaches list end so past_dashes[j+1]==limit → no payload; == [] holds regardless of whether $0 is recognized as a shell.

Both tests are deterministic, touch no host state (auto-reverting monkeypatch, no child processes, no files), and the instrumented globals resolve correctly since _nested_shell_payloads references them as module globals. The empty candidate list stands; nothing survives to Step 2.

No findings.

[OPUS-REVIEWED] 71ba3c5

Verdict parsed from the review's SHA-scoped output markers for commit 71ba3c547a684c12cf32c7b8832b9dc738063bb1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 71ba3c547a684c12cf32c7b8832b9dc738063bb1: <one-sentence reason>

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Status: functionally review-ready; the only red is main-inherited

All PR-owned signals are green:

  • 55/64 checks SUCCESS, 7 SKIPPED (frontend lanes — test-only backend change)
  • All 5 AI review lanes PASS (Opus 4.8 + GPT 5.6 blocking, First Principles / Design / UX advisory)
  • MERGEABLE

The two reds are Backend Tests (Windows) (3) and the PR Readiness aggregator counting it. That shard failure is inherited from main, not from this diff:

Holding this branch unchanged and re-checking readiness after main's Windows shard is fixed.

@CrysisDeu
CrysisDeu force-pushed the fix/nested-payload-linearity-structural-8687 branch from 95ed4c7 to afb69eb Compare September 5, 2026 13:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 5, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/nested-payload-linearity-structural-8687 branch from afb69eb to 71ba3c5 Compare September 5, 2026 14:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 5, 2026
@iamwhatever
iamwhatever merged commit a33d4cf into main Sep 5, 2026
65 checks passed
@iamwhatever
iamwhatever deleted the fix/nested-payload-linearity-structural-8687 branch September 5, 2026 22:43
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants