Repository navigation
fix(instances): reclaim a leaked forwarder by its spawn-time argv fingerprint - #17819
Conversation
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check the macOS What this change shipsInventory (6 items) — 6 justifiedIntent: A forwarder left behind by a hard-killed gateway should still be reclaimed after its command line drifts. This is a FIX (#5383).
[FIRST-PRINCIPLES-REVIEWED] 1c97e17 Description read[DESCRIPTION-READ] 2e6cced8cce7da9278b6c62e549c492818cf108f4c310c3f03a5eb3dd6ac496d That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merging, check that the [DESIGN-REVIEWED] 1c97e17 Description read[DESCRIPTION-READ] ed26ce3629691b36514720edb1e98ff0509bc9adea470fe08341d04f36e8b320 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
1f8456a to
247267c
Compare
247267c to
4545a17
Compare
|
self-added: yes
|
|
4545a17 to
29931e5
Compare
|
|
|
|
b61206e to
7ff1a45
Compare
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
|
Backend Tests shard 6 (Linux and Windows) is red on |
|
Backend Tests shard 2 (Linux and Windows) is also red on |
7ff1a45 to
069de73
Compare
…gerprint Reclaim used to compare the orphan's live argv with a command line rebuilt from today's settings, so a shebang aws v1 entrypoint or a compression/host edit since spawn missed the reclaim. Record a sha256 of the argv the kernel reports right after spawn (forwarder_argv_sig), sign it into forwarder_sig, and compare it with the live process's fingerprint at reclaim. Records without the field keep the rebuilt-argv check and their four-field MAC. Closes #5383
069de73 to
1c97e17
Compare
|
This PR fixes missed reclaims of forwarders left behind by a hard-killed gateway. It records a sha256 of the child's argv as the kernel reports it right after spawn ( |
Problem / Motivation
Goal: A forwarder left behind by a hard-killed gateway is still reclaimed when its command line drifted between spawn and reclaim.
Reclaim checks the leaked process's live argv against a command line rebuilt from the settings as they are at reclaim time. Three things make the two differ with no foul play: an
awsCLI v1 entrypoint is a shebang script, so the kernel shows<interpreter> /path/aws ssm ...;instances.ssh_compression, the host or the port were edited since spawn; and the macOS basis was the flattenedps -wwstring. Each one misses the reclaim, so the forwarder keeps its port and its remote session after every hard-kill and reconnect.Why it matters
Each missed reclaim leaks one process, one loopback port and one live remote session. SSM users on a v1
awsnever get a reclaim at all.Not a goal
What changed (motivation → approach → change)
The miss comes from comparing two different things: what the kernel shows for the child, and what we would build today. So we now record what the kernel shows, once, right after spawn, and compare like with like at reclaim.
forwarder_argv_sig: sha256 of the child's argv as the kernel reports it; the MAC covers it and the spawn transport (ssh/ssm)platform_compat.process_argv_fingerprint(pid)(inkiro_crew/platform_compat.py) reads/proc/<pid>/cmdlineon Linux and theKERN_PROCARGS2vector on macOS; on Windows it answers None, so Windows keeps today's exact command-line check. Ininstances/ssh_tunnel_manager.py,_forwarder_identity_hintsrecords it and_forwarder_identity_sigsigns it together with the spawn transport. The transport picks pid-vs-group signalling, so a record whose settings switched transport since spawn fails the MAC and is never signalled._reclaim_orphan_forwarderskips the rebuild when the field is set, and_verify_and_reclaim_forwardercompares fingerprints in both identity checks (before SIGTERM and before SIGKILL). TheInstanceininstances/registry.pygains the field. The instances spec (docs/system-specs/modules/instances.md) describes both forms.Backwards compatibility
Compatible: a record without
forwarder_argv_sigsigns and verifies the same four-field message as before and keeps the rebuilt-argv check, so forwarders in flight across an upgrade are reclaimed exactly as today and nothing is forced to rebuild. Erasing the field from a record signed with it fails the MAC (the field counts differ), so it cannot be pushed back onto the older check.Removes nothing: the older check stays for records that predate the field; only a new optional registry field is added.
Tests
test_recorded_argv_fingerprint_survives_command_line_drift(with and without a compression flip, Linux and macOS): a real shebang child, whose kernel argv carries the interpreter the rebuild lacks, is reclaimed by its fingerprint. Red on the parent.test_recorded_argv_fingerprint_mismatch_is_never_signalled: a validly signed fingerprint that is not the live argv leaves the process alive, even when the rebuild would match.test_dropping_the_signed_fingerprint_does_not_fall_back: erasing the field fails the MAC before any process is read.test_forwarder_sig_message_keeps_the_older_four_field_form: an empty fingerprint signs the exact older message; ssh and ssm sign differently.test_switched_transport_is_refused_before_any_signal: an ssh-signed record read under SSM settings never reaches the signal path; the same record signed for SSM does. Red without the transport binding.test_argv_fingerprint_is_stable_and_tells_processes_apart,test_windows_records_no_argv_fingerprint: the helper's basis and its None answers.Determinism: the new and touched tests passed 10 repeated runs with
-n 2, and once in a shuffled order with-n0.Manual verification
The drift test drives a real
connect()against a real shebang child holding a real port, and runs on Linux and macOS (it compares the live fingerprint with the one read from the same child, so the macOS framework-python re-exec does not matter). The macOS backend suite is requested for this head with theci:macoslabel; its result is posted on the PR.Related Issues
Closes #5383
Pattern harvest
Rule candidate: review-prompt
Pattern: an identity check that rebuilds the expected value from current settings instead of comparing against a value recorded at creation
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)