Skip to content

fix(instances): reclaim a leaked forwarder by its spawn-time argv fingerprint - #17819

Merged
bolichen97 merged 1 commit into
mainfrom
fix/forwarder-argv-fingerprint-5383
Oct 8, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/forwarder-argv-fingerprint-5383

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A forwarder left behind by a hard-killed gateway is still reclaimed when its command line drifted between spawn and reclaim.

Reclaim checks the leaked process's live argv against a command line rebuilt from the settings as they are at reclaim time. Three things make the two differ with no foul play: an aws CLI v1 entrypoint is a shebang script, so the kernel shows <interpreter> /path/aws ssm ...; instances.ssh_compression, the host or the port were edited since spawn; and the macOS basis was the flattened ps -ww string. Each one misses the reclaim, so the forwarder keeps its port and its remote session after every hard-kill and reconnect.

Why it matters

Each missed reclaim leaks one process, one loopback port and one live remote session. SSM users on a v1 aws never get a reclaim at all.

Not a goal

  • Persisting the identity before the token mint, to close the hard-kill-during-mint window. That is the separate second half of the issue.
  • Any change to the orphan test, the start-time pin, the signal sequence or the Windows rules.
  • Scanning the process table. Nothing here reads any pid but the recorded one.

What changed (motivation → approach → change)

The miss comes from comparing two different things: what the kernel shows for the child, and what we would build today. So we now record what the kernel shows, once, right after spawn, and compare like with like at reclaim.

Before After
At spawn pid, start time, port, MAC plus forwarder_argv_sig: sha256 of the child's argv as the kernel reports it; the MAC covers it and the spawn transport (ssh/ssm)
At reclaim live argv == rebuilt argv live fingerprint == recorded fingerprint
Record from an older gateway, or Windows rebuilt-argv check unchanged: rebuilt-argv check, four-field MAC

platform_compat.process_argv_fingerprint(pid) (in kiro_crew/platform_compat.py) reads /proc/<pid>/cmdline on Linux and the KERN_PROCARGS2 vector on macOS; on Windows it answers None, so Windows keeps today's exact command-line check. In instances/ssh_tunnel_manager.py, _forwarder_identity_hints records it and _forwarder_identity_sig signs it together with the spawn transport. The transport picks pid-vs-group signalling, so a record whose settings switched transport since spawn fails the MAC and is never signalled. _reclaim_orphan_forwarder skips the rebuild when the field is set, and _verify_and_reclaim_forwarder compares fingerprints in both identity checks (before SIGTERM and before SIGKILL). The Instance in instances/registry.py gains the field. The instances spec (docs/system-specs/modules/instances.md) describes both forms.

Backwards compatibility

Compatible: a record without forwarder_argv_sig signs and verifies the same four-field message as before and keeps the rebuilt-argv check, so forwarders in flight across an upgrade are reclaimed exactly as today and nothing is forced to rebuild. Erasing the field from a record signed with it fails the MAC (the field counts differ), so it cannot be pushed back onto the older check.

Removes nothing: the older check stays for records that predate the field; only a new optional registry field is added.

Tests

  • test_recorded_argv_fingerprint_survives_command_line_drift (with and without a compression flip, Linux and macOS): a real shebang child, whose kernel argv carries the interpreter the rebuild lacks, is reclaimed by its fingerprint. Red on the parent.
  • test_recorded_argv_fingerprint_mismatch_is_never_signalled: a validly signed fingerprint that is not the live argv leaves the process alive, even when the rebuild would match.
  • test_dropping_the_signed_fingerprint_does_not_fall_back: erasing the field fails the MAC before any process is read.
  • test_forwarder_sig_message_keeps_the_older_four_field_form: an empty fingerprint signs the exact older message; ssh and ssm sign differently.
  • test_switched_transport_is_refused_before_any_signal: an ssh-signed record read under SSM settings never reaches the signal path; the same record signed for SSM does. Red without the transport binding.
  • test_argv_fingerprint_is_stable_and_tells_processes_apart, test_windows_records_no_argv_fingerprint: the helper's basis and its None answers.
  • The connect / recover identity-write tests now pin the new field. All older reclaim tests, including the never-mis-kill ones, pass unchanged.

Determinism: the new and touched tests passed 10 repeated runs with -n 2, and once in a shuffled order with -n0.

Manual verification

The drift test drives a real connect() against a real shebang child holding a real port, and runs on Linux and macOS (it compares the live fingerprint with the one read from the same child, so the macOS framework-python re-exec does not matter). The macOS backend suite is requested for this head with the ci:macos label; its result is posted on the PR.

Related Issues

Closes #5383

Pattern harvest

Rule candidate: review-prompt
Pattern: an identity check that rebuilds the expected value from current settings instead of comparing against a value recorded at creation

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 7, 2026 19:01
@bolichen97
bolichen97 requested a review from buluoray October 7, 2026 19:01
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 1c97e17fbac39933e9aa03308beb8be04584fc76 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check the macOS ci:macos result for this head: the KERN_PROCARGS2 fingerprint path is exercised only there.

What this change ships

Inventory (6 items) — 6 justified

Intent: A forwarder left behind by a hard-killed gateway should still be reclaimed after its command line drifts. This is a FIX (#5383).

  1. A leaked forwarder whose rebuilt command line drifted (a shebang aws v1, a compression/host/port edit) is reclaimed by its spawn-time fingerprint — justified
  2. A new persisted registry field forwarder_argv_sig, written at connect/recover and read at reclaim — justified
  3. A new helper platform_compat.process_argv_fingerprint; its 2 consumers are both in ssh_tunnel_manager.py — justified
  4. The spawn transport (ssh/ssm) is now inside the MAC, so a record whose transport switched since spawn is never signalled — justified
  5. Records from older gateways and Windows keep the four-field MAC and the rebuilt-argv check — justified
  6. The instances spec describes both identity forms — justified

[FIRST-PRINCIPLES-REVIEWED] 1c97e17

Description read

[DESCRIPTION-READ] 2e6cced8cce7da9278b6c62e549c492818cf108f4c310c3f03a5eb3dd6ac496d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1c97e17fbac39933e9aa03308beb8be04584fc76 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merging, check that the ci:macos run passed for this head. The drift test's macOS KERN_PROCARGS2 fingerprint path has no other evidence.

[DESIGN-REVIEWED] 1c97e17

Description read

[DESCRIPTION-READ] ed26ce3629691b36514720edb1e98ff0509bc9adea470fe08341d04f36e8b320

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1c97e17fbac39933e9aa03308beb8be04584fc76 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1c97e17

Verdict parsed from the review's SHA-scoped output markers for commit 1c97e17fbac39933e9aa03308beb8be04584fc76.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1c97e17fbac39933e9aa03308beb8be04584fc76: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1c97e17fbac39933e9aa03308beb8be04584fc76 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1c97e17

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1c97e17fbac39933e9aa03308beb8be04584fc76: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/forwarder-argv-fingerprint-5383 branch from 1f8456a to 247267c Compare October 7, 2026 19:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@bolichen97
bolichen97 force-pushed the fix/forwarder-argv-fingerprint-5383 branch from 247267c to 4545a17 Compare October 7, 2026 20:05
@bolichen97

bolichen97 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: spawn transport signed into forwarder_sig alongside the argv fingerprint

  • span=b13845276a59 Unsigned transport edits escalate PID reclamation to a group kill

fixed in 4545a17. With a fingerprint, _forwarder_identity_sig now also signs the spawn transport (ssh/ssm, from the tunnel that was spawned).
At reclaim the expected MAC is computed with the transport the current settings would signal with, so a record whose settings switched ssh->ssm since spawn fails verification before any process is read or signalled.
test_switched_transport_is_refused_before_any_signal pins it (red without the binding, green with); the SSM-signed control still reaches the signal path.

@bolichen97

bolichen97 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=bcdd12041a33 Item 4: Windows branch of process_argv_fingerprint

fixed in 4545a17: the Windows branch is removed. process_argv_fingerprint answers None on Windows, so a Windows record carries no fingerprint and keeps the exact list2cmdline check, which never matches a .cmd/.bat shim, as the spec states.
test_windows_records_no_argv_fingerprint pins that nothing is read on Windows; the spec text now says Windows records none.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Oct 7, 2026
@bolichen97
bolichen97 force-pushed the fix/forwarder-argv-fingerprint-5383 branch 2 times, most recently from 4545a17 to 29931e5 Compare October 7, 2026 20:36
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=42863e1f43fa Fingerprint test always fails on Windows

fixed in 29931e5: test_argv_fingerprint_is_stable_and_tells_processes_apart asserts both fingerprints are None on Windows and keeps the non-empty, stable, distinct assertions on POSIX, so the verdict is the same on every platform.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=2c74f73a990b Cause: the macOS basis fix is a new darwin branch

fixed in 29931e5: the darwin branch is now exercised by tests that run on macOS. test_argv_fingerprint_is_stable_and_tells_processes_apart reads real KERN_PROCARGS2 fingerprints there, and the end-to-end drift test is no longer Linux-only.
The macOS backend suite is requested for this head with the ci:macos label.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=febc72305a5b Mechanism: the only real reclaim test is Linux-only

fixed in 29931e5: test_recorded_argv_fingerprint_survives_command_line_drift now skips only on Windows. It compares the live fingerprint with one read from the same child, so the macOS framework-python re-exec that forced the Linux-only skip on the exact-argv tests does not apply.
The ci:macos label runs the macOS suite, this test included, against this head; the result will be posted here.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=2d869b521f96 Consequence: a missed or mismatched macOS read fails quietly

fixed in 29931e5 by evidence rather than code: the same macOS run that covers the drift test drives the spawn-time read and the reclaim compare on a real macOS kernel, so a None or mismatch there fails a test instead of passing silently.
Behaviour on a miss stays fail-closed by design: None keeps today's check and a mismatch signals nothing, which never kills the wrong process.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@bolichen97
bolichen97 force-pushed the fix/forwarder-argv-fingerprint-5383 branch 2 times, most recently from b61206e to 7ff1a45 Compare October 7, 2026 21:56
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=42863e1f43fa Second-spawn failure leaks the first child

fixed in 7ff1a45: the first child is spawned, then a try/finally that reaps it wraps the second spawn, which has its own try/finally. A failed second spawn now still reaps the first child, and each child is reaped exactly once.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1c97e17fbac39933e9aa03308beb8be04584fc76 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests shard 6 (Linux and Windows) is red on test/test_session_control_doc.py::test_every_tabulated_refusal_code_is_one_the_source_raises. That test fails the same way on a clean origin/main (a2ca1b7), and this PR touches neither file. #17903 already fixes it; this PR will rebase once that lands.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests shard 2 (Linux and Windows) is also red on test/test_http_server_bind_ratchet.py::test_bare_http_server_constructions_match_the_recorded_count (16 vs recorded 15). That test fails the same way on a clean origin/main (a2ca1b7), and this diff adds no HTTPServer. Open #17906 and #17909 address it; this PR will rebase once main is fixed.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@bolichen97
bolichen97 force-pushed the fix/forwarder-argv-fingerprint-5383 branch from 7ff1a45 to 069de73 Compare October 8, 2026 00:18
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main now passes both inherited reds (test_http_server_bind_ratchet fixed by #17909; test_session_control_doc passes on main 972b782) (7ff1a45 -> 069de73); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026
…gerprint

Reclaim used to compare the orphan's live argv with a command line rebuilt
from today's settings, so a shebang aws v1 entrypoint or a compression/host
edit since spawn missed the reclaim. Record a sha256 of the argv the kernel
reports right after spawn (forwarder_argv_sig), sign it into forwarder_sig,
and compare it with the live process's fingerprint at reclaim. Records
without the field keep the rebuilt-argv check and their four-field MAC.

Closes #5383
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_runtime_reconcile.py (imports platform_compat) and docs/system-specs/modules since this head went green (972b782 -> 6c15ee7); CI re-running.

@bolichen97
bolichen97 force-pushed the fix/forwarder-argv-fingerprint-5383 branch from 069de73 to 1c97e17 Compare October 8, 2026 01:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 8, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 8, 2026 02:13
@iamwhatever

Copy link
Copy Markdown
Contributor

This PR fixes missed reclaims of forwarders left behind by a hard-killed gateway. It records a sha256 of the child's argv as the kernel reports it right after spawn (forwarder_argv_sig), covers that fingerprint and the spawn transport with the HMAC, and compares it with the live fingerprint at reclaim instead of an argv rebuilt from today's settings. That addresses the real cause: the reclaim was comparing two different things (the v1 aws shebang, edited settings, macOS ps -ww flattening). I read the production diff (registry.py, ssh_tunnel_manager.py, platform_compat.py). Older four-field records still verify, and deleting the fingerprint from a six-field record fails verification instead of falling back to the older check. All checks are green, including the four ci:macos backend shards the Design and First Principles lanes asked for, and every lane passes. I am holding only because this changes the identity check that gates a SIGTERM/SIGKILL (and group-kill) of a recorded pid, which is in the process-kill and identity class a maintainer must approve. I found no defect. — crew-pr-reviewer

@bolichen97
bolichen97 merged commit a6e8222 into main Oct 8, 2026
105 checks passed
@bolichen97
bolichen97 deleted the fix/forwarder-argv-fingerprint-5383 branch October 8, 2026 06:35
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Instances: record an observed-argv fingerprint at spawn so forwarder reclaim survives command-line drift

2 participants