Skip to content

refactor(sel): route the restart path's SEL drain through flush_audit_queue_before_hard_exit - #22396

Merged
iamwhatever merged 1 commit into
mainfrom
kc-wt-19042
Oct 23, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
kc-wt-19042

Conversation

@dodgeblaster

@dodgeblaster dodgeblaster commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every gateway hard exit that drains the SEL audit queue does it by one spelling. The restart path drained SEL with its own inline block; route it through the shared helper the other hard exits use.

The gateway restart path (_handle_restart in src/kiro_crew/slack/events_slash.py) calls platform_compat.hard_exit(1), which is os._exit: it runs no atexit handler and does not join the SEL daemon writer. Before that exit it drained the SEL audit queue — so the approved-restart audit event is not dropped — but with its own inline run_in_executor(subprocess_executor(), sel().flush) block, a second spelling of the drain that #18089 had already centralised into flush_audit_queue_before_hard_exit() for the other hard-exit paths.

Why it matters

Two spellings of the same shutdown drain drift apart: a fix to the bound, the trust-root guard or the error handling made in the shared helper does not reach the inline copy. Routing the restart path through the one helper keeps the SEL drain contract in a single place for every gateway hard exit.

Not a goal

What changed (motivation → approach → change)

  • _handle_restart's inline run_in_executor(subprocess_executor(), sel().flush) + asyncio.wait_for block is replaced by await flush_audit_queue_before_hard_exit().
  • The helper offloads the blocking flush to the teardown pool under an outer deadline, so a wedged writer (disk full, unreachable sink) cannot block the loop indefinitely or prevent os._exit(1) from being reached, and it drains only an existing singleton so it never creates a trust root on the way out — the same guarantees the inline block carried, now shared.

Backwards compatibility

Removes nothing: one restart-path drain is swapped for an equivalent shared helper; no field, type, key, record or public function is changed, rejected or removed.

Tests

  • test/test_restart_command.py: the three restart tests assert the shared flush_audit_queue_before_hard_exit() helper is awaited (mock_flush_sel.assert_awaited_once_with()) instead of the removed inline sel().flush, across the owner-with-supervisor, hung-session-close, and slot-save-raises paths — so the restart path still drains SEL before os._exit(1) even when session close hangs or the slot save raises.

Manual verification

N/A — unit coverage is sufficient; the restart path is exercised deterministically with os._exit replaced, and the SEL drain helper is asserted awaited.

Related Issues

Fixes #18938

Pattern harvest

Not generalizable: this folds one remaining inline SEL drain into the shared flush_audit_queue_before_hard_exit() helper; the test_sel_hard_exit_flush.py ratchet already enforces the class (every os._exit/hard_exit in the scanned modules must drain), so no new rule is needed.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The restart exit's worst-case wait grows from 3s to 4s (3s inner drain plus 1s outer margin). Confirm that is acceptable before os._exit(1).

What this change ships

Inventory (4 items) — 3 justified

Intent: the restart command drains the SEL audit queue through the same single helper as every other gateway hard exit. FIX (#18938), a removed duplicate.

  1. The approved-restart audit drain now runs through flush_audit_queue_before_hard_exit() (src/kiro_crew/sel.py:1175) instead of an inline run_in_executor(…, sel().flush) copy — justified
  2. On a wedged writer, restart can now wait up to 4s before exiting instead of 3s; the inner flush(timeout=3.0) is now bounded too — undeclared: "No widening of … the drain's own deadline contract" doesn't mention the extra second
  3. The restart drain no longer builds a SEL singleton if none exists (flush_audit_queue returns early) — justified
  4. The three restart tests now assert the helper is awaited instead of sel().flush being called — justified

The duplicate is gone: a grep for sel\(\)\.flush across src/kiro_crew finds no remaining inline drain. The other three hard-exit sites (exit_path.py:558, gateway.py:413, platform_compat_runtime.py:308) already use the helper.

[FIRST-PRINCIPLES-REVIEWED] e1c49ba

Description read

[DESCRIPTION-READ] c0151b7c65dd8ab3e2e65846565403acc786b56688ab717947c865f13117567c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Worth confirming: the restart path's worst-case drain goes from 3s to about 4s (inner flush(timeout=3.0) plus a timeout + 1.0 outer deadline).

[DESIGN-REVIEWED] e1c49ba

Description read

[DESCRIPTION-READ] 072d48c029c2e7d7ea9c69d87d316a60b92a6c020a9fedaf74c8a8ce9365e4f2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e1c49ba

Verdict parsed from the review's SHA-scoped output markers for commit e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e1c49ba

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e1c49ba4b9b06cc3a378e16dc155f4d6ef057c7c: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 20, 2026
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

span=8ffa878c35ae — needs-a-decision — self-added: no — mechanism: GPT no-blocking-call-on-event-loop forced the move off-loop; Design then flagged a real loop/thread race, so the two lanes are in genuine tension a maintainer must settle.

The thread + guard are the fix, not undeclared surface: the inline flush_audit_queue(timeout=2.0) this review's subtraction asks me to restore was BLOCKED on the prior head by GPT 6.1 (Opus-adjudicated UPHOLD) under no-blocking-call-on-event-loop — SecurityEventLog.flush waits on a Condition up to the timeout, freezing the loop-registered _on_signal. That is the linked evidence the Clears-when asks for; the description now declares and justifies the thread.
But removing the thread is not available (re-blocks GPT), and Design has now shown the off-loop thread races the graceful _shutdown_and_exit already running on the loop (it can reach its own hard_exit mid-drain and truncate the very SEL tail this PR saves). Neither the inline shape nor this thread shape satisfies both lanes. Resolving it means a larger design change (make the force-exit thread the sole exiter / block the graceful path's hard_exit) that widens this PR and reverses more pinned decisions — a maintainer's call, escalated via work_report question.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author

span=67e2ef5946f5 — needs-a-decision — self-added: no — mechanism: verified-real race between the off-loop force-exit thread and the graceful _shutdown_and_exit on the loop.

Confirmed real against the code: first signal sets shutdown_event, the main run loop (service_boot.py:1404) then runs await _shutdown_and_exit(_watchdog) ON the loop to its own hard_exit; the second signal starts the kc-force-exit thread. They run concurrently — both can call cleanup_orphaned_sessions / _stop_log_queue_listener, and the graceful hard_exit can win mid-drain and truncate the SEL tail this PR exists to keep. The description now states+justifies the thread, but the race itself is not closed.
The fix Design/First-Principles point to (inline on-loop) is exactly what GPT no-blocking-call-on-event-loop BLOCKS, so the two lanes cannot both be satisfied by either current shape. Closing the race needs a design the frozen goal did not settle — make the force-exit thread the sole exiter (second signal blocks/cancels the graceful path's hard_exit) OR serialize the shared drains — which widens this PR. Escalated to the conductor/maintainer via work_report question rather than widening scope unilaterally.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

needs-a-decision — self-added: yes — mechanism: verified-real NEW concurrency angle on the force-exit span; it trips the conductor's circuit-breaker, so this is escalated rather than patched in a third design pass.

Confirmed real against the code: service_boot.py:1404 (await self._shutdown_and_exit(_watchdog)) is the last statement in run(), so when _shutdown_and_exit RETURNS on the force-claim the graceful coroutine unwinds out of run()/asyncio.run, interpreter shutdown begins, and the daemon kc-force-exit thread is killed mid-drain — dropping the SEL + gateway.log tail, with the atexit drain unable to compensate (listener already swapped to None). This is a NEW angle (interpreter-shutdown-vs-daemon-thread), distinct from last round's loop/thread race that Design/First-Principles raised.
GPT's suggested fix (await a pending loop future instead of returning, so the main coroutine stays alive until the force thread calls os._exit) is plausible, but this round's finding hits a conductor-set circuit-breaker: a new concurrency angle on the force-exit span means STOP and escalate rather than take a third design pass unilaterally. Reported status=question to the conductor. The #18938 restart-drain half (events_slash.py) is independent and may ship on its own; the #19042 second-signal half is a candidate to defer with this analysis.

…_queue_before_hard_exit

The gateway restart path (`_handle_restart` in src/kiro_crew/slack/events_slash.py)
drained the SEL audit queue before its `os._exit(1)` with its own inline
`run_in_executor(subprocess_executor(), sel().flush)` block. Route it through the
shared `flush_audit_queue_before_hard_exit()` helper instead, so every gateway
hard exit that drains SEL does it by one spelling: the helper offloads the
blocking drain to the teardown pool under an outer deadline (a wedged writer
cannot hang the loop or block `os._exit`), and drains only an existing singleton
so it never creates a trust root on the way out.

Fixes #18938
@dodgeblaster dodgeblaster changed the title fix(sel): drain the audit tail on the gateway's second-signal force exit refactor(sel): route the restart path's SEL drain through flush_audit_queue_before_hard_exit Oct 20, 2026
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Conductor narrowing after the circuit-breaker tripped. This PR originally carried both #18938 (restart-path SEL drain → shared helper) and #19042 (second-signal force-exit SEL drain). The #19042 force-exit work hit a conductor-set circuit-breaker when GPT 6.1 Review (Opus-adjudicated UPHOLD, data-loss class) raised a new concurrency angle on the force-exit span: with the Option-A sole-exiter guard, _shutdown_and_exit returning on the force-claim unwinds the graceful coroutine out of run()/asyncio.run() (service_boot.py:1404 is its last statement), so interpreter shutdown starts and kills the daemon kc-force-exit thread mid-drain — dropping the very SEL tail the change exists to save. Closing that needs its own design.

Per the conductor's ruling, the force-exit half (#19042) has been reverted out of this PR and deferred to a tracking issue; this PR now carries only the #18938 restart-drain refactor, and its Goal / Why / Not-a-goal sections were narrowed to match. #19042 stays open.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author

rebutted — self-added: no — mechanism: the +1s is the shared helper's standard outer margin; the bound is acceptable on a rare operator restart.

Confirmed and acceptable. The old inline block was wait_for(run_in_executor(... sel().flush), timeout=3.0); the shared flush_audit_queue_before_hard_exit() runs flush_audit_queue(timeout=3.0) offloaded and wraps the hop in wait_for(..., timeout=3.0 + 1.0) (sel.py:1175-1181, default _HARD_EXIT_FLUSH_TIMEOUT_SECS=3.0). So the worst case grows from 3s to ~4s.
The extra 1s is deliberate: the outer deadline sits just above the inner flush deadline so the inner bound is the one that actually fires and the executor hop can never outlast it — it is the helper's standard contract, already in effect on the gateway's other hard exits (_shutdown_and_exit, exit_after_failed_restart_exec). This path is an operator-initiated restart of the gateway, not a hot path; a <=1s longer bound before os._exit(1), only in the wedged-writer worst case, is proportional to keeping every hard exit on one drain contract.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author

rebutted — self-added: no — mechanism: same as the Design punchline — the ~4s worst-case bound is the shared helper's standard margin and is acceptable on an operator restart.

Confirmed acceptable. 3s inner flush + 1s outer wait_for margin = ~4s worst case (sel.py:1175-1181). The outer +1s guarantees the inner deadline is what fires and the executor hop cannot outlast it — the helper's existing contract on the other gateway hard exits. The restart path is operator-initiated and not hot; a <=1s longer bound before os._exit(1), only when the SEL writer is wedged, is proportional to folding this path onto the one shared drain spelling that #18938 asks for.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@dodgeblaster
dodgeblaster marked this pull request as ready for review October 20, 2026 21:19
@dodgeblaster
dodgeblaster requested review from a team and KiroCrewCaptain as code owners October 20, 2026 21:19
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (reserved for a human reviewer) at e1c49ba.

The change routes the approved-restart path's SEL audit drain, right before os._exit(1), through the shared flush_audit_queue_before_hard_exit() helper (events_slash.py L651-L674, sel.py L1164-L1186). It edits a process-exit path and the security audit drain with a small behaviour change (worst case 3s -> 4s), so it is not a mechanical move and the repository's review policy reserves it for a human.

I found no defect. The restart audit event is logged through sel() before the drain, so the helper's existing-singleton early return cannot skip it. Every current-head AI lane passed, the 3s -> 4s punchline has an author reply I agree with, checks are green, and the split-out force-exit work is tracked in #19042.

What clears it: a maintainer review of the restart-path drain.

— crew-pr-reviewer (r116)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 21, 2026 21:14
@iamwhatever
iamwhatever merged commit 2c7a34d into main Oct 23, 2026
107 checks passed
@iamwhatever
iamwhatever deleted the kc-wt-19042 branch October 23, 2026 00:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sel: route the restart path's SEL drain through flush_audit_queue_before_hard_exit

2 participants