Skip to content

PTYCAP-07: unified self-heal routing (routeHeal seam, flag OFF) [QC BLOCKED] - #512

Draft
finedesignz wants to merge 142 commits into
mainfrom
phase-ptycap-07-self-heal-routing
Draft

finedesignz wants to merge 142 commits into
mainfrom
phase-ptycap-07-self-heal-routing

Conversation

@finedesignz

Copy link
Copy Markdown
Owner

PTYCAP Phase 7 - Unified Self-Heal Routing

Stacked on #506 -> #505. Merge those first (base of this branch is the Phase 4 branch; this PR targets main so Woodpecker qc runs). Includes merged #346.

Status: DRAFT - /qc BLOCKED (1 open MAJOR after 2 fix rounds). Flag REMO_HEAL_UNIFIED default OFF = exact legacy behaviour.

What it builds

  • ONE routeHeal() seam (hub/src/heal/) fed by all four failure sources: Coolify deploy webhook, scheduled-task failure, error-capture, feedback intake. Guard test fails CI if a source hand-rolls routing/dispatch (heal-single-seam).
  • Routing: live repo session -> dispatch (no spawn); live-less + allowlisted -> governed autospawn (full gate list + atomic lifetime reservation BEFORE session.start, reservation reused on replay, released on failure/expiry, skip-permissions forced false); live-less + not allowlisted -> terminal skipped/no_routable_session within seconds, exactly one notification (claim-based owner on heal_events.notification_claimed_at).
  • heal_events ledger (additive idempotent DDL; advisory-lock claim; per-user/per-repo/per-task ceilings atomic), durable is_internal marker + recursion guard, counter key heal:<userId>:<canonical repo_ident>.
  • Boot sweep + deadline follow-up: no heal run can sit pending until the 6h reaper; incident fixtures (runs 8a6e0534, fa377e27; task 0e16bf38) replayed (unit + real-Postgres e2e in Woodpecker).
  • Docs: docs/self-heal-routing.md + updated subsystem docs + CLAUDE.md env section.

QC

Seats: sonnet correctness + adversary lenses, Codex read-only (agy MISSING). Record: .planning/phases/PTYCAP-07-self-heal-routing/QC/.
Open MAJOR (QC/VERDICT.md): a hub crash between the scheduled-task heal claim and the scheduler default email can yield zero notifications (needs a persisted notification obligation the sweep delivers).

Verification

  • bun run check-baseline: OK (pass 3271, skip 280, fail 0). skip_max raised 260 -> 280 only for new REMO_E2E_DB_URL-gated Postgres e2e files; pass_min/fail_max untouched.
  • tsc: phase files add no new errors (64 -> 19, the 19 pre-exist on base).
  • Postgres e2e (heal events, incident replay) first runs in Woodpecker.

🤖 Generated with Claude Code

finedesignz and others added 30 commits October 13, 2026 10:28
Goal-backward verification of Phase 1 (PTY token accounting, #395) and
Phase 2 (PTY pre-flight gate, #493) against ROADMAP success criteria, with
every proving test run per-file.

- Phase 1: 3/3 SC implemented; gaps_found (G-1 MAJOR: emitter dedupes on
  record uuid, not message.id, so the PTY ledger over-counts; G-2 MINOR:
  live-Postgres proofs not observed locally).
- Phase 2: passed 3/3; three MINOR gaps (CRLF-fragile guard test, stale
  docs mechanism claim, Telegram permission keystrokes un-preflighted).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-only recon of the gate chain, actor inference, PTY write paths,
counters, owner notification, token-cap defaults and kill-switch
precedents, with file:line evidence. Input to discuss/plan for the
governed-automation guard and the lifetime counter + kill switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ONTEXT

Unattended discuss for Phase 3 (governed-automation guard): the eight
design forks were answered by a claude+codex+agy design-fork panel (record
in GATES/design-fork-1.md, VERDICT PASS with amendments). CONTEXT captures
the binding decisions D-01..D-17: governance primitives only (no PTY
byte-producer), fail-closed lifetime/kill-switch slots, enabled-ceiling
rule, call-time flag default OFF, hard rename with eligible-actor rule,
automation-only governed chain (human chain untouched), door pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Planner (opus) output verified by gsd-plan-checker: 0 blockers, 2 warnings
(fails_when pass-count floors; plan 03-04 doc-heavy scope), accepted for
execution. Waves: 03-01 tracer (governed-pty module + rename + OFF==legacy
differential), 03-02 preflight automation branch + CRLF fix, 03-03 door pins
+ spoof tests + Telegram required source, 03-04 docs/OpenAPI/baseline gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…with OFF==legacy differential

- add hub/src/dispatch/governed-pty.ts: flag reader, eligible-actor rule, four governors (kill_switch/lifetime_counter fail closed), test-only DI, governedAutomationPtyGate
- export isDailyTokenCapEnforced from gates.ts
- telegram dispatch composes governedAutomationPtyGate
- add differential test against a frozen legacy oracle and flag-ON telegram cases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tionPtyGate

- ask and work compose governedAutomationPtyGate with unchanged server-inferred resolvers
- pin that an external-ask actor stays blocked with the flag ON and every governor passing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rnor semantics

- remove humanOnlyPtyGate from gates.ts (hard rename, no alias); keep AUTOMATION_ACTORS and humanOnlyRejectsActor
- human-only-guard test targets governedAutomationPtyGate with the flag pinned OFF
- pin governor order, enabled rule, fail-closed slots, throw, non-PTY, DI-only and no-alias

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…preflight

- automation actor refused before any DB gate unless the governed-automation flag is ON
- flag ON: spend chain then governors, prefixed reasons; throw/hang stay fail closed
- human chain and scanned gates literal untouched

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… humanInteractive guard CRLF-safe

- no governor in either chain constant; human turn passes with fail-closed governors and flag ON
- split source on optional CR plus LF (Phase-2 MINOR-1)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ed when missing

The webhook and the doctor replay assert source 'human' at the authenticated
boundary; a call without a source is refused on a pty-interactive session
instead of defaulting to human (D-09 iv).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and preflights as human

- term.input with source/actor extras -> preflight actor human, forwarded keys exactly type/session_id/bytes
- term.attach_file stays preflight-exempt (D-10) and forwards no source/actor keys

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…outing, privileged callers, flag/DI, exemptions

- emitter allowlist (ws/client.ts, telegram/transcript/pty-inject.ts) with non-vacuous self-check
- supervisor canary: user_message/permission_response never touch the PTY, term.input does
- caller pins for automation/human actor, preflight, injectPtyKeystroke, dispatchToSession
- flag/DI literals confined to dispatch/governed-pty.ts; actor-inference and Telegram source pins
- EXEMPTIONS record: injectPtyKeystroke and term.attach_file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-2 mechanism claims, Phase-4 hand-off

- usage-cost.md: PTYCAP Phase 3 section; structural reason automation cannot reach the PTY
- CLAUDE.md: REMO_GOVERNED_AUTOMATION_PTY env bullet, docs-map row, invariant, token-cap fail-closed fix
- ROADMAP: Phase 4 hand-off note (no SC change)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ate rename across docs and SESSION-ASK spec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onPtyGate; finish rename in hub/src comments

Comment-only source changes; docs:sync regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…59 fail 0) with phase-3 accounting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nity) ceiling config

maxInjectsPerHour() coerces non-finite env to the default 4, so the governor
passed an invalid value. Add isInjectRateConfigured() (raw validity, governor
only; legacy gate behavior unchanged) and align docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…am/doctor.ts)

doctor.ts takes dispatchToSession as an injected dispatch param, so the bare
call-site needle missed it. Pin importers instead (webhook + doctor), and
assert the doctor replay call passes source: 'human'. Docs name doctor.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pass)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eight executable plans (G-1 emitter fix first, tracer-led controls core, counting expansion, PTY door + governors, owner surfaces + 60% alarm, kill-switch coverage, spawn-site guards, e2e + docs + 20M cap last), VALIDATION map with source coverage audit, and 04-FORKS.md recording planner forks for panel confirm or overturn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Apply the locked fork-panel verdicts and the plan-check fixes: heal route classified machine (F-2), separate final halt check before send (F-3), release resumes every scope and once-task re-arm (F-4), 80 percent lifetime warning page (F-5a), triage key per application (F-5b), one atomic lifetime + de-duplicated rate-union reservation with routine_run_log.inject_token (F-6), zero new skipped tests with the DB proof moved into 04-02 via an existing e2e test body (F-8). 04-02 split into 4 tasks, 04-03 gains a run-log token task, 04-07 gains a heal task, 04-08 reduced to docs + cap-last-code-commit. Includes 04-FORKS-PANEL.md and 04-PLAN-CHECK.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ima and positive deltas

- G-1/D-15: records of one assistant message share message.id and carry cumulative usage; count once
- key = message.id, fallback uuid; negatives clamp to 0; maxima never decrease; replay emits nothing
- FIFO bound SEEN_UUID_CAP kept; six new fixtures in pty-usage-tail.test.ts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
finedesignz and others added 27 commits October 13, 2026 17:52
…plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…start and launch-caller guards

- heal-single-seam: pinned legacy token counts, seam isolation, positive flag-before-route wiring, SH-06 dead machinery
- token-cap-coverage: heal gate list discovered and order-asserted
- counter-key, session-start, launch-callers guards extended additively

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked when the heal flag is off

- OFF matrix (unset, empty, 0, false, no, off, maybe, garbage, 2, onn, 10) across all four entry points
- ON matrix (1, true, yes, on, mixed case, padded) calls routeHeal exactly once per eligible input
- flag read at call time; real finalizeRun failed path driven under a query-bypassed dispatcher

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…currency, autospawn ledger)

- V1: runs 8a6e0534/fa377e27 terminal skipped/no_routable_session under 5s, zero pending or open runs, reaper sees none, one notice per key, duplicate deduped
- V2: unrelated online session receives no frame and no message
- 8-way concurrency: one ledger row, one notice, 1 + 7 outcomes
- autospawn: one governed launch, ledger row, unreleased reservation; failed launch releases it
- skips locally without REMO_E2E_DB_URL; picked up by bun run orchestrator:e2e in Woodpecker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…play plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on and Coolify notes

- new docs/self-heal-routing.md: event union, routing table, outcomes, ledger, notification, bounds, knobs
- self-heal-integration: POST /heal unchanged, prefix-compatible only
- coolify-webhook-migration: flag-ON branch vs legacy flag-OFF path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…book

- error-capture, feedback-intake, scheduled-tasks: flag-ON routing through routeHeal, flag-OFF unchanged
- orchestrator-autospawn-runbook: heal arming order, verification queries, disarm
- auto-dev-orchestrator: governed launch extraction note

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…invariant

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in the Phase 4 coverage guards

- kill-switch-coverage: heal/route.ts is a COUNTED machine list (counter last); 12 -> 13 lists
- reserve-caller-guard: allow the heal pre-launch reservation binding
- additive classification only, no assertion loosened

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pass 2652 -> 3215, skip 259 -> 280 (all REMO_E2E_DB_URL-gated), fail 0, total 3495
- skip_max 260 -> 280 (exactly measured); pass_min and fail_max untouched

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmary

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ession exists (flag ON)

Active-session suppression skipped routeHeal and left the metadata run
pending. Suppression is now legacy (flag OFF) only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion

A heal pre-launch reservation at the cap boundary blocked its own
post-spawn dispatch. At the cap, an unreleased row for the same
(counter key, token) now passes, matching reserveInject idempotency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ownership is the notification claim, not delivery: every refused / failed /
no_routable_session outcome now reports notifyClaimed, so the scheduler
default failure email is suppressed whenever the heal path won the claim
(even when its one send failed). scheduled_task launched/parked takes the
claim for the scheduler default email so a later expiry or boot-sweep
notice cannot double-send. The unified error-capture sink runs the run
store with its legacy notifier disabled; routeHeal is the sole owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…task failures

only_if_active is a claim-then-finalize CAS used by real senders (work,
TEAB, agent), not a reaper marker. The heal source now ignores it; the
reaper stays excluded via its run_timeout error. A lost claim already
returns before the hook runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
launchSessionForUser gains skipPermissions:false; governedLaunch's heal policy
passes it, so untrusted-triggered machine spawns never inherit the session's
skip-permissions value. Orchestrator (legacy policy) unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
governedAutomationPtyGate (actor self-heal, never an eligible automation actor) now sits
after the token cap in the single heal gate list, evaluated against the session runner
type at dispatch time, closing the resolve-to-dispatch window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…po and per-task ceilings

- autospawn persists outcome autospawn_starting (not swallowed) before any reservation or
  spawn, and the stale-launch list selects it, so a crash in the reservation-to-launched
  window is recoverable by the sweep.
- the per-repo hourly and per-task daily ceilings move into the claim transaction, counted
  under the per-user advisory lock and inserted atomically (in-flight rows count); the
  router no longer pre-counts. heal_events gains a nullable recovery_ctx jsonb (additive).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, and has a deadline follow-up

- the sweep persists/uses recovery_ctx and the bound session id so the error_capture sink can
  be constructed; the reservation release and ledger outcome are reconciled even when the
  source run cannot be terminalized (no per-boot re-failing, no leaked slot).
- boot sweep arms ONE follow-up at 2x spawn timeout + 1s (event driven, no interval, true
  no-op when the heal flag is OFF) so a row stranded by a quick restart is swept as soon as it
  ages past the staleness guard instead of waiting for the 6h reaper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… atomic ceilings, sweep follow-up)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r sessions

launchSessionForUser did not forward skipPermissions:false to launchOrchestrator, which re-read
the persisted setting. Forward it and force false before session.start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ministic scheduled-task owner

- releaseInjectStrict (idempotent, throws on DB error) backs REAL_AUTOSPAWN.release; the route and
  the sweep no longer treat a swallowed release failure as success. A failed release leaves the
  event at autospawn_starting/launched/parked for the sweep instead of a terminal outcome.
- The sweep reconciles the source run and the reservation independently and closes the ledger row
  only when both are confirmed; otherwise it keeps the recoverable outcome (re-stamped) and notifies
  nobody.
- scheduled_task: every async heal finalizer (deadline, replay, sweep) is claim-only, so a failed
  launch-time claim can no longer yield a second email next to the scheduler default one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wner once

The unified sink disabled the legacy throttled notifier, so a queued turn that was promoted later
and then skipped or failed to send produced no notice (routeHeal had already returned
dispatched_live). The sink now claims the same per-event heal notification slot routeHeal uses, so
immediate failures still yield one notice and deferred ones yield exactly one too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@agentautofix-fixer

agentautofix-fixer Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

AgentAutofix AI Review

claude

PASSED

  • info hub/src/dispatch/gates.ts: Default daily token cap lowered 50M -> 20M only applies when env unset -- DEFAULT_DAILY_TOKEN_CAP drops from 50_000_000 to 20_000_000 tokens/day, affecting any deployment that has never explicitly set REMO_ORCHESTRATOR_DAILY_TOKEN_CAP. The docs already warn this can lock out human PTY usage for the rest of a day once the lower ceiling is hit; worth confirming the prod env var is set explicitly before deploy, as the PR itself recommends. Not a code defect — flagging for operational awareness only.

codex

BLOCKED

  • blocking hub/src/dispatch/automation-controls.ts: Reactivated reservations can bypass a lifetime reset -- reserveInject reuses a previously released row by clearing released_at. If that row's seq predates the latest reset marker, countLifetime permanently excludes it via seq > MAX(reset seq), so the retried injection is sent without consuming the new lifetime budget. Insert a new post-reset reservation row/token epoch instead of reactivating a pre-reset row, and serialize resets with reservations using the same counter-key advisory lock.

agy

advisory -- not part of the gate decision

PASSED

No findings.

Result: Blocking finding from: codex

Reviewer models (requested pin, not independently confirmed):

  • claude: claude-haiku-4-5-20251001 (requested pin, not independently confirmed)
  • codex: gpt-5.5 (requested pin, not independently confirmed)

finedesignz and others added 2 commits October 13, 2026 20:04
…l claim; sweep redelivers (outbox)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-4 findings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant