Skip to content

fix(work-ledger): make the path containment checks race-free - #9468

Closed
DeryFerd wants to merge 1 commit into
kirodotdev:mainfrom
DeryFerd:fix/work-ledger-containment-race
Closed

DeryFerd wants to merge 1 commit into
kirodotdev:mainfrom
DeryFerd:fix/work-ledger-containment-race

Conversation

@DeryFerd

@DeryFerd DeryFerd commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

test_two_conductors_binding_one_worker_at_once_yield_exactly_one_binding fails intermittently on the Windows CI shard with a refusal that names a perfectly innocent key:

chat-1-c=invalid_value (field=None msg=path traversal blocked for worker key: 'chat-2-worker')

chat-2-worker contains no separators and no dot segments; nothing about it traverses anything. The two path constructors behind it, binding_path() and conductor_dir() in src/kiro_crew/work_ledger.py, guarded themselves by resolving the child path and then the parent as two separate filesystem walks:

resolved = (base / f"{_store_name(worker_slot_key)}.json").resolve()
if not resolved.is_relative_to(base.resolve()):

On a host whose temp path carries a short (8.3) component — the GitHub Windows runners hand pytest a RUNNER~1-style prefix — those two walks can disagree. Path.resolve() finalizes everything that already exists on disk, so a resolve that runs after bindings/ was created returns the long form; a resolve whose final component does not exist yet falls back to the lexical form and keeps the short prefix. When the two sides of one containment check straddle the directory's creation, is_relative_to compares RUNNER~1 against runneradmin and refuses the key.

Why it matters

The failure is intermittent in exactly the way that wastes the most time. The binding race starts four threads at once; the first write creates bindings/, so whichever threads resolve after that moment see the expanded prefix and whichever resolve ran before it kept the short one. One thread gets refused with a "path traversal" error that accuses a benign key, the shard goes red, and a re-run burns half an hour of CI to pass or fail on thread timing. The same shape sits in conductor_dir() for any concurrent first access to a conductor's ledger.

It also misleads diagnosis. The refusal says "path traversal blocked", so the natural reaction is to audit the key for separators — and there are none. The real defect is that the guard's two inputs were taken from the filesystem at two different moments.

What changed (motivation → approach → change)

Symptom: a benign key refused as traversal. Root cause: containment decided by comparing two filesystem resolves that are not atomic with respect to each other. Fix: resolve the parent once and compose the child from it, then check containment lexically.

parent = bindings_dir().resolve()
resolved = parent / f"{_store_name(worker_slot_key)}.json"
if not resolved.is_relative_to(parent):

The guard keeps its full strength, including the part the composed form alone cannot see. The child's name comes from _store_name(), which sanitizes every unsafe character and appends a digest, so a composed child can never escape the parent — the lexical check still refuses anything that would, and the null-byte and shape checks upstream of it stay untouched. What is removed is the second walk, which was the only thing that could race. What the second walk could catch is restored separately: a symlink or junction planted at the composed child is rejected via os.lstat (a link or reparse point at the final component), which looks at one entry instead of walking the tree, so it carries none of the race back. Writes and reads through the returned path land in the same directory as before: Path.resolve() finalizes junctions and symlinks for existing prefixes either way, and every caller derives paths through these constructors, so no caller ever mixes the two forms.

One behavior note for reviewers: the returned path is now the composed resolved-parent / name rather than the fully-finalized path. On hosts without aliased prefixes the two are byte-identical, and on aliased hosts the composed form is actually the more stable choice — it does not flip between boots depending on whether the directory happened to exist at resolve time.

The change is two files:

  • src/kiro_crew/work_ledger.py — both constructors resolve the parent once and compose the child lexically
  • test/test_work_ledger.py — two regression tests plus a shared _racing_resolve() helper

Tests

I wrote the regression tests first and watched them fail with the exact CI error — WorkLedgerError: path traversal blocked for worker key: 'chat-2-worker' — reproduced deterministically on a machine where the real race cannot trigger, then watched them pass after the fix. That is the part that matters most: the old code's misfire no longer depends on runner luck to demonstrate.

The helper _racing_resolve() models the runner deterministically. It flips the second resolve under the data home onto a sibling prefix, which is precisely the divergence the 8.3 form produces when the directory appears between the two walks; on hosts without aliased prefixes the same flip is the only way to observe the two-resolve disagreement at all.

Two more tests pin the link guard: they plant a link at the composed child name, point it outside the ledger, and assert both constructors refuse it, watched to fail with "DID NOT RAISE" before the guard existed.

python -m pytest test/test_work_ledger.py
127 passed

The flaky binding-race test itself ran three times back to back, clean each time, and the full file covers the lock-ordering, cap, and torn-file criteria unchanged. python -m mypy src/kiro_crew is clean, and the repo's check_black_formatting.py / check_subprocess_encoding.py gates plus flake8, isort, and black pass on the touched files.

Manual verification

There is nothing to eyeball in the app: the observable behavior change is that a refusal stops happening. The honest reproduction of the original failure needs the runner's short-form temp path, which is why it only surfaced on the Windows shard; the deterministic flip test is the reviewer-accessible stand-in, and the binding-race test is the end-to-end check that the shard gate exercises on every run.

Pattern harvest

Not generalizable: the defect is a two-resolve ordering inside one path constructor, not a recurring API misuse — a static rule cannot tell a containment check that must survive directory creation from an ordinary resolve. The durable guard is the regression test, which flips the prefix between the two resolves deterministically.

Related Issues

No dedicated issue. The failing shard is visible directly on main's own CI runs, so this PR also unsticks the Windows lane for every open PR that inherited the flake.

Checklist

  • Regression tests written first, watched fail with the CI error, then pass
  • mypy and the repo's format/subprocess gates pass
  • Self-review done; the traversal guard still refuses anything that would escape the parent

@DeryFerd
DeryFerd requested review from a team and smeyffret September 8, 2026 13:55
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 3960622ea012d1e5a9079310dc26d8985e45c8a1 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All evidence gathered. The fix itself is cause-level within work_ledger.py and backed by a regression test I verified would fail on base by reading the code, but the identical two-resolve containment construction survives verbatim in sibling modules the description claims don't exist. Final review:

First-Principles-Verdict: CONCERNS

Point patch: the identical two-resolve containment shape survives in session_ledger.py and subagent_persistence.py, while "Pattern harvest" claims it is "not generalizable".

What this change ships

Intent: stop a Windows CI flake that refuses benign work-ledger keys as "path traversal" when two containment resolves straddle the directory's creation — a FIX.

  1. Benign slot/worker keys no longer intermittently refused when the ledger directory appears mid-check — justified
  2. A symlink/junction planted at a ledger child is now refused via a single-entry lstat check — justified
  3. Constructors return the composed parent-resolved path instead of the fully-finalized child — justified
  4. Four regression tests plus _racing_resolve/_plant_link helpers, watched to fail on base — justified

Watch

  • Grepped is_relative_to across src/: the same child-resolve-then-parent-resolve construction, byte-identical fold and error text, sits unfixed at session_ledger.py:163-165 (ledger_dir), subagent_persistence.py:315-317 (_agent_dir), and session_storage.py:1438-1442 (_staged_path) — 3 counted siblings, the first two with the same concurrent-first-access exposure. "Not generalizable: … not a recurring API misuse" is contradicted by that count. Accepted-and-deferred is fine, but the scope claim isn't honest as written.
    Clears when: the two live siblings get the same one-resolve fix, or a linked issue names them as deferred.

[FIRST-PRINCIPLES-REVIEWED] 3960622

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 3960622ea012d1e5a9079310dc26d8985e45c8a1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3960622

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 3960622ea012d1e5a9079310dc26d8985e45c8a1 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All key questions are answered. The fix is root-cause (removes the racing second resolve), keeps the guard's strength via a race-free lstat link check, is well-tested, and no spec pins the mechanics it changes. The one design-level signal: the byte-identical two-resolve pattern remains in session_ledger.ledger_dir() (session_ledger.py:163-165), and a similar two-resolve shape in members.py:297-301, while the PR's "Pattern harvest" section declares the defect not generalizable.

Design-Verdict: CONCERNS

Sound root-cause fix, but the identical two-resolve race survives in session_ledger.ledger_dir(), which the PR's own harvest section overlooks.

Watch

  • The defect class this PR fixes — "containment decided by comparing two filesystem resolves that are not atomic with respect to each other" — exists byte-for-byte in src/kiro_crew/session_ledger.py:163-165 (the module work_ledger imports _store_name from), and in two-resolve shape in src/kiro_crew/members.py:297-301. Concurrent first access to a session ledger on an 8.3-aliased Windows prefix produces the same false "Path traversal blocked" refusal on a core session path, and the PR's claim "the defect is a two-resolve ordering inside one path constructor, not a recurring API misuse" understates it: it recurs one import away.
    Clears when: a follow-up (issue or PR) applies the same resolved-parent/composed-child shape to session_ledger.ledger_dir() and members.py, or states why those call sites cannot race directory creation.

[DESIGN-REVIEWED] 3960622

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 3960622ea012d1e5a9079310dc26d8985e45c8a1 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/work_ledger.py:511 -- "st_file_attributes" & reparse rejects non-link Windows cloud placeholders, making ledger access fail -> Fix: use platform_compat.is_link_or_junction(child).
[GPT-REVIEWED] 3960622

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 8, 2026
@DeryFerd
DeryFerd force-pushed the fix/work-ledger-containment-race branch from 384e12e to d0934b6 Compare September 8, 2026 15:06
binding_path and conductor_dir resolved the child and then the
parent as two separate filesystem walks. On a host whose temp path
carries a short (8.3) component, the Windows CI runner's tmp_path does,
the two walks can disagree about the parent's prefix form once the
directory appears between them, and a benign key is refused as path
traversal -- the intermittent Windows-shard failure of the
binding-race test. The child is now composed from the resolved parent
and containment is checked lexically, so the guard keeps its
traversal guarantee without a second resolve to race.
@DeryFerd
DeryFerd force-pushed the fix/work-ledger-containment-race branch from d0934b6 to 3960622 Compare September 8, 2026 16:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@dwu96 dwu96 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 12, 2026
@NicholasRBowers NicholasRBowers added needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 12, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • main has since fixed the same Windows flake by a different mechanism. test: second five-run hygiene pass (real-home escapes, host reds, flakes) #9614 (merged 2026-09-09) and fix(work-ledger): retry a contended record read; make a test's ids salt-free #9673 replaced both containment checks in src/kiro_crew/work_ledger.py with the shared resolved_within() helper from session_ledger.py: it resolves the base once, builds the child from the resolved base, and strips Windows' extended-length (\\?\) prefix from both sides before is_relative_to. Its docstring diagnoses the same test_two_conductors_binding_one_worker_at_once_yield_exactly_one_binding misfire as a \\?\ prefix split rather than an 8.3 short-name split. This PR's lexical-containment + _refuse_planted_link() (lstat) design now conflicts with that helper on every hunk (conductor_dir, binding_path). Two roads, and only you can pick: (a) confirm main's fix already covers the race you observed and close this PR (or reduce it to the two _racing_resolve regression tests if they still reproduce against resolved_within), or (b) argue the 8.3 short-name divergence survives main's fix and rework the PR as a change to resolved_within() itself, so the session ledger and work ledger keep one containment rule.
  • Design Review and First Principles Review are both 🟡 CONCERNS on 3960622ea with no disposition posted; once the scope above is settled, each item needs its own target=design / target=first-principles disposition comment.
  • The only other red, Build Windows Installer (x64), is an artifact-upload 403 Forbidden at FinalizeArtifact (infra), not the diff. GPT and Opus are clean on the current head.

When you've addressed these, the pipeline will re-assess on its next cycle.

@DeryFerd

Copy link
Copy Markdown
Contributor Author

@NicholasRBowers Going with road (a). I read resolved_within() on current main: the base is resolved once and the child is composed from that resolved base, the \\?\ extended-length prefixes are stripped from both sides before the comparison, and the docstring names this exact binding-race test with a measured reproduction rate. That covers the failure my PR chased — my PR's single-resolve composition is the same core idea, and the \\?\ strip addresses the prefix-form split my write-up under-diagnosed (I suspected an 8.3 short-name split; the docstring's \\?\ mechanism is the measured one). Closing this as superseded by #9614 and #9673. One property worth noting: resolved_within still resolves the child, so a link planted at the composed child is still rejected — the guard my PR restored separately is already covered there.

@DeryFerd

Copy link
Copy Markdown
Contributor Author

target=design — The Watch item is fair: my "not generalizable" claim was contradicted by the sibling count, and the scope statement was written before the full extent was known. Disposition: main's #9614/#9673 fix (resolved_within) supersedes this PR — same single-resolve composition, plus the \\?\ prefix normalization — so the PR is closing as superseded. The session_ledger.ledger_dir sibling is covered by that helper; the members.py two-resolve shape is the remaining item your follow-up suggestion names, and that is the right home for it.

@DeryFerd

Copy link
Copy Markdown
Contributor Author

target=first-principles — All four items were marked justified at review time and stand as analysis; the disposition is supersession. #9614/#9673 landed the same containment shape upstream (resolved_within: base resolved once, child composed from it, \\?\ prefixes stripped before comparison), so this PR's changes are redundant and it is closing. The honest correction to the harvest section is that the class did recur — one import away, exactly as the review said — and the maintainers' fix is the broader one covering the session_ledger route.

@DeryFerd DeryFerd closed this Sep 12, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants