Skip to content

test(security): give real resolutions the shipped budget in the bounded resolver tests - #39141

Open
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/resolver-tests-shipped-budget
Open

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/resolver-tests-shipped-budget

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The bounded path-resolution tests that check a healthy path's answer stop going red on a loaded macOS runner.

Three tests in test/test_security_path_resolve_bounded.py failed together on a macOS shard (#39127): test_a_match_and_a_benign_path_keep_their_answers, test_a_matched_path_spelled_like_the_stall_wording_is_still_a_match and test_unc_paths_are_never_probed_on_windows. A benign tmp_path file got Blocked: the path could not be verified against the sensitive-path list within the resolver budget instead of None.

Why it matters

A red macOS shard blocks unrelated PRs (the run was on #38803, which touches nothing under security/). The failure also reads like a security regression, which sends the reader into the resolver logic.

Not a goal

  • No production change: the shipped budgets and the stall handling are unchanged.
  • No change to the stall tests, which keep the fixture's 0.2 s budget so they stay fast.

What changed (motivation → approach → change)

  • Cause: the autouse fixture _fresh_resolver_state scales the candidate and rebuild budgets to 0.2 s (0.5 s with grace) and stubs _child_blocked_in_filesystem to say every miss is a filesystem block. That is right for the stall doubles, which wait out any deadline. But some tests need a real resolution to finish. Each of them starts a cold resolver child, because the fixture shuts the pool down after every test. On a loaded runner that start can miss 0.5 s; the miss is charged as a stall (off Linux the child cannot be sampled at all), and the healthy path gets the stall refusal.
  • This is a different cause from the issue's guess: the cooldown cannot carry between tests, because the fixture resets _path_resolve_degraded for each one. Each failing test hit its own miss.
  • Change: a helper _shipped_budgets(monkeypatch) in security_path_resolve_bounded_helpers.py puts the shipped candidate and rebuild budgets back. The five tests whose assertion needs a real resolution to finish call it before that resolution: the three from the issue, test_symlink_alias_is_still_resolved_on_a_healthy_filesystem, and the healthy middle step of test_the_cooldown_is_scoped_to_the_stalled_prefix, which then puts the fixture's 0.2 s back for its last stall double.

Backwards compatibility

Compatible: test-only.

Removes nothing: no product code changes.

Tests

  • test/security_path_resolve_bounded_helpers.py: new _shipped_budgets.
  • test/test_security_path_resolve_bounded.py: the five tests above call it.

Forced condition: a local pytest plugin (not committed) made every real resolution in those five tests answer 0.6 s late, past the fixture's 0.5 s and well inside the shipped 5 s, standing in for a cold child on a loaded runner that cannot be sampled. On main that reproduces exactly the issue's three failures, with the same could not be verified ... within the resolver budget refusal. With this change all of them pass. The two symlink tests skip on this Windows host either way.

main this PR
normal run pass pass
0.6 s slow child the issue's 3 tests fail pass

Proof: not provable -- test-only change

Locally (Windows 11, Python 3.13): test_security_path_resolve_bounded.py 66 passed, 4 skipped, in about the same time as main, since a healthy resolution answers well inside either budget. Two tests in test_security_path_resolve_bounded_child.py fail identically on main on this host and are untouched. flake8, isort, black 26.10.1 and the flake-pattern ratchet are clean; the three budget lines carry # flake-ok because they restore budgets rather than add a short wait.

Manual verification

N/A — unit coverage sufficient: the forced-condition run above is the reproduction; there is no product behavior to check by hand.

Real-machine verification

N/A -- test-only: the diff changes test files and nothing that runs in the product. A macOS runner was not available; the forced condition above stands in for its load.

Related Issues

Fixes #39127

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

…ed resolver tests

The autouse fixture scales the resolver budget to 0.2 s (0.5 s with
grace) for the stall doubles and reads every miss as a filesystem block.
Five tests also need a REAL resolution to finish, and each starts a cold
resolver child because the fixture drops the pool after every test. On a
loaded macOS runner that start missed the deadline, the miss was charged
as a stall, and a healthy path got the stall refusal. Those tests now put
the shipped budgets back before the real resolution.

Fixes kirodotdev#39127

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_match_and_a_benign_path_keep_their_answers and two siblings red on macOS on a resolver-budget stall

1 participant