Skip to content

fix: Windows pod restart adoption and macOS local-model test stall - #16122

Merged
bolichen97 merged 1 commit into
mainfrom
fix/nightly-windows-pod-restart-macos-local-runtime
Oct 2, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/nightly-windows-pod-restart-macos-local-runtime

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The nightly's Windows pod restart scenario and macOS local-model tests pass again, for the reasons they were red.

The Nightly Build has been red for three nights (09-30, 10-01, 10-02). Two legs stay red on every run:

  • Pod Scenario E2E (windows-latest): test_setting_survives_a_gateway_restart reports pod gateway did not come back after restart (last 0). The successor gateway boots, logs for about 14 seconds, then is never seen again. The next scenario fails with ConnectionRefusedError on the same pod.
  • Platform Tests / Backend Tests (macOS): the three tests in test_decisions_local_runtime_real_spawn.py (new on 10-01) fail with runtime stayed {'state': 'starting', ...}; log: pid=NNN. The stand-in server prints its pid and is alive, but never attests.

Why it matters

A red nightly publishes nothing: every Publish ... and Sign + Notarize job is skipped, so no nightly CLI, Docker image, Windows, Linux or macOS build has gone out since 09-29. On Windows the underlying defect is also a product one: an in-app restart of a pod gateway is killed by its own supervisor.

Not a goal

  • Any change to how a pod restart works on Linux (systemd) or macOS (launchd), where the gateway execs in place and keeps its pid.
  • Fixing test_context_composition_contract, the third nightly red on 10-02; test: fix two Windows-red tests and write down why #16083 already did.
  • Any change to the laya or strands-decider launchers, which serve through uvicorn and do not bind through http.server.

What changed (motivation → approach → change)

Where What
src/kiro_crew/instances/run_marker.py a gateway whose pid has exited is not "another live gateway"
src/kiro_crew/decisions/local_servers/plumb_cpu.py bind without the getfqdn lookup
test/test_decisions_local_runtime_real_spawn.py the stand-in server binds the same way; a stalled start now says whether the port listens and whether attest answers
test/e2e/scenarios/conftest.py a restart that does not come back prints pod status --json beside the logs
test/test_instance_credential_pairing.py pins liveness before identity in the boot-write gate

Windows pod restart. On Windows os.execv is CreateProcess plus an exit of the caller, so the restarted gateway is a child with a new pid. The pod supervisor waits up to 30 s for that child to publish run/gateway-<port>.pid, then drains the Job with everything in it. The successor never published: _record_names_another_live_gateway read the old record, asked for the old pid's creation time, got the same token back, and declined to overwrite "another live gateway". That token stays readable after the process exits for as long as any handle to it is open, and the supervisor holds one — its Popen to the gateway — for exactly the window the successor boots in. The gate now asks pid_exists first, the same liveness-before-identity order pod.windows.supervised_pid already documents for the same trap. An exited pid is a corpse, not a rival, so the successor publishes and is adopted.

One sibling compare is deliberately left as it is: pod/runtime_attestation.py::_pod_recorded_pid still answers from the token alone. Both of its readers ask liveness themselves before they act on it -- port_owner agrees it against main_pid, which is supervised_pid (pid_exists first), and pod/windows.py::_restart_successor checks pid_exists on the record before comparing -- and its tests pin synthetic pids on purpose, so a liveness check inside it would change what those tests prove rather than what this restart needs.

macOS local-model test. http.server.HTTPServer.server_bind calls socket.getfqdn(host) between bind() and listen(). On a macOS runner that resolver call can stall for longer than the test waits, so the stand-in server has a bound socket that refuses every connection. The stand-in and the one real launcher that binds through http.server (plumb_cpu.py) override server_bind to set the name from the address they already have. The gateway attests a server by its secret, never by a name, so nothing read that value.

Diagnostics. Both failures were hard to read from the nightly's output alone. The e2e restart failure now prints the pod's own pod status --json verdict next to the logs, and the local-runtime wait prints the port and attest probe state when it gives up.

Backwards compatibility

Compatible: the boot-write gate declines in one fewer case — a recorded pid that no longer exists — and that case could never have named a serving gateway. plumb_cpu.py serves the same routes on the same address; only server_name is no longer looked up.

Tests

  • test_instance_credential_pairing.py::TestABootWriteDoesNotOverwriteALiveSuccessor::test_declines_when_the_record_names_another_live_process — the live-rival case still declines, with liveness now mocked true.
  • ...::test_an_exited_predecessor_is_not_a_rival_even_while_its_token_still_reads — new: an exited pid whose token still matches is written over.
  • test_decisions_local_runtime_real_spawn.py — unchanged assertions; the stand-in binds without getfqdn, and the failure message names the stalled half.
  • test/e2e/scenarios/test_conftest.py — unchanged, still 49 passing.

Locally (Linux): the two gate modules plus every module that imports write_marker (1296 passed), the three test_decisions_local_* modules (166 passed), isort/flake8/mypy and the backend check scripts from the prepare-pr profile.

Manual verification

Windows and macOS cannot run here. This PR carries the ci:pod-scenarios and ci:macos labels so the hosted Windows pod scenario suite and the macOS backend suite run on this head; their verdict is the verification.

Related Issues

no linked issue: found from the nightly runs directly.

Pattern harvest

Rule candidate: review-prompt
Pattern: a Windows process identity (creation FILETIME) read as liveness — the token stays readable after exit while any handle is held, so every "is this pid still that process" check needs pid_exists before the token compare.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 09:18
@bolichen97 bolichen97 added ci:pod-scenarios Opt in to full Windows pod scenarios on the next PR push ci:macos Run the on-demand macOS suite against this PR head labels Oct 2, 2026
@bolichen97 bolichen97 changed the title fix: Windows pod restart publishes its successor; macOS local-model test binds without getfqdn fix: Windows pod restart adoption and macOS local-model test stall Oct 2, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@bolichen97
bolichen97 force-pushed the fix/nightly-windows-pod-restart-macos-local-runtime branch from 1f0036a to 6560759 Compare October 2, 2026 09:19
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 702a971b4be9f2eaa569bc8114c9bd098ef16293 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 702a971

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 702a971b4be9f2eaa569bc8114c9bd098ef16293: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 702a971b4be9f2eaa569bc8114c9bd098ef16293 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the ci:pod-scenarios and ci:macos runs on this head are green — they are the PR's only on-platform proof.

[DESIGN-REVIEWED] 702a971

Description read

[DESCRIPTION-READ] cdf977c5dc92da3db9c83f48946f66175b599e3e20e2cd8917ad4dad7a04b4c3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 702a971b4be9f2eaa569bc8114c9bd098ef16293 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Both fixes rest on platform behaviour the author could not run locally — confirm the ci:pod-scenarios and ci:macos hosted runs are green on this head before merge.

Not justified as shipped

  • Item 5 — rides along: declared diagnostics ("the local-runtime wait prints the port and attest probe state"), not the fix; harm-free, test-only.
  • Item 6 — rides along: declared diagnostics ("prints the pod's own pod status --json verdict next to the logs"), not the fix; harm-free, test-only.

What this change ships

Inventory (6 items) — 4 justified

Intent: make the nightly's Windows pod-restart scenario and macOS local-model tests pass for the reasons they were red — a FIX.

  1. A Windows in-app pod restart now publishes its successor and is adopted; an exited recorded pid no longer blocks the boot-time marker write — justified
  2. The plumb-cpu local-model server binds without the getfqdn lookup; server_name is the literal loopback address — justified
  3. The test stand-in server binds the same way, removing the macOS resolver stall — justified
  4. New test pins "exited pid with readable token is overwritten"; the live-rival test now mocks liveness true — justified
  5. A stalled local-runtime test failure now reports whether the port listens and whether attest answers — rides along: declared diagnostics, not the fix
  6. A failed e2e pod restart now prints pod status --json beside the logs — rides along: declared diagnostics, not the fix

[FIRST-PRINCIPLES-REVIEWED] 702a971

Description read

[DESCRIPTION-READ] a58f0fcaf2aedc6607b3fa53bec0b719ab4e0c7fd3cd5ab5e338f9bca66e50b9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 702a971b4be9f2eaa569bc8114c9bd098ef16293 — this comment is updated in place on each push.

Review details

The liveness short-circuit de-pins the fence's token-identity arm: the test that exists to prove it now returns before reaching it.

FINDING — test/test_instance_credential_pairing.py:513 — test_writes_when_the_recorded_pid_is_not_provably_live patches only pid_start_token, so write_marker(5476) now exits _record_names_another_live_gateway at the new if not platform_compat.pid_exists(recorded_pid) (pid 424242 does not exist on any runner — which is exactly why the sibling test_declines_when_the_record_names_another_live_process had to gain pid_exists=True in this diff), leaving return pid_start_token(recorded_pid) == recorded_token reached by no test in the class and mutable to return True with zero red → Fix: add mock.patch.object(run_marker.platform_compat, "pid_exists", return_value=True) to that test's with block so the write is proved to come from the token mismatch.

[OPUS-REVIEWED] 702a971

Verdict parsed from the review's SHA-scoped output markers for commit 702a971b4be9f2eaa569bc8114c9bd098ef16293.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 702a971b4be9f2eaa569bc8114c9bd098ef16293: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 1 — _pod_recorded_pid sibling compare (span=326f5439b4b2): rebutted

Of the four pid_start_token compare sites, this PR fixes the one that gates a write (write_marker), which is where a corpse read as a rival had a consequence. _pod_recorded_pid is a reader whose two consumers already ask liveness before acting on it: port_owner requires agreement with main_pid, which is pod/windows.py::supervised_pid and checks pid_exists first; _restart_successor checks pid_exists on the record before comparing tokens. A liveness check inside the reader would change what its tests (which pin synthetic pids on purpose) prove, not what the restart needs. The body's "What changed" now records this sibling as deliberately left.

@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 2 — plumb_cpu.py ride-along (span=a9f74e86ec58): rebutted

It is the same defect at its one production instance: http.server.HTTPServer.server_bind calls getfqdn between bind() and listen(), and the stall the test stub hit on macOS stalls a real plumb-4b start the same way, bounded only by the 600 s startup budget. The change is one eight-line server_bind override, declared in the body's table, and "Not a goal" already excludes the two uvicorn launchers that do not bind through http.server. Splitting it into a second PR would ship the test's fix without the user's.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 2, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 2, 2026 10:32
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 force-pushed the fix/nightly-windows-pod-restart-macos-local-runtime branch from 6560759 to 702a971 Compare October 2, 2026 16:26
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in files this PR imports or shares a test stem with since this head went green (3d465a6 -> 07e6f0b, +49 commits; no same-file overlap); CI re-running on 702a971. Diff unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 2, 2026
@bolichen97
bolichen97 merged commit a9cb741 into main Oct 2, 2026
96 of 97 checks passed
@bolichen97
bolichen97 deleted the fix/nightly-windows-pod-restart-macos-local-runtime branch October 2, 2026 16:32
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head ci:pod-scenarios Opt in to full Windows pod scenarios on the next PR push

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant