Skip to content

feat: add scheduled failure-family alerts and retained leases - #17

Merged
scbrown merged 6 commits into
mainfrom
feat/kelly-failure-family-alerts
Oct 11, 2026
Merged

scbrown merged 6 commits into
mainfrom
feat/kelly-failure-family-alerts

Conversation

@scbrown

@scbrown scbrown commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Repeated local command-family failures need a scheduled threshold collector that keeps UNKNOWN separate from recovery. Add a governed-policy adapter and opt-in generation-stable firing/recovery. Output contains hashed family identity and counts, without raw commands, arguments or errors.

A bounded one-shot entry retains adapter state, setup receipts and the sink write budget across processes. Graph setup reuse requires typed read-back. Alertmanager renewal requires a fresh positive reading and a received firing; it sends the frozen original identity only to the lease-aware receiver without adding another transition, graph or command event. UNKNOWN, missing evidence, recovery and old generations cannot renew.

Validation: 185 tests and 16 subtests on the corrected current-base tree pass under the HOME guard (153 watched paths and five resources untouched); ruff passes. The same controlled lease probe changes from an expired firing at time 100 to a received same-identity refresh, retaining one outbox event. Controls cover UNKNOWN, skipped measurements, pending firing receipts, recovery/re-crossing, restart backoff/budget, setup retraction/metadata changes and startup UNKNOWN.

Source only: a reviewed read-only desire-path release is a prerequisite. Schedule/pins, policy, routing, native scheduled delivery and downstream query acceptance remain separate. This PR does not install a listener, schedule or binary, alter routing, or send a test alert.

Current-base integration retains the existing received-firing recovery protocol; the prototype alternate recovery protocol was removed. Renewal reads the prior receiver label set and requires a receipt showing the extended expiry. An expired/absent prior identity is UNKNOWN rather than a new-policy recreation. Final counter/configuration adjustments additionally pass 21 focused controls and four subtests.

Independent review found a stale recovery recipe and inflated retry backoff after successful renewals. The recipe now loads with the existing recovery protocol. Successful lease receipts reset consecutive failures; the same three-success/first-failure probe changes 240 seconds to 30 seconds, matching the no-prior-success control. Restart, legacy success-only counters and recovery-after-failure controls pass.

@scbrown
scbrown merged commit 60964ee into main Oct 11, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant