Repository navigation
feat: add scheduled failure-family alerts and retained leases - #17
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated local command-family failures need a scheduled threshold collector that keeps UNKNOWN separate from recovery. Add a governed-policy adapter and opt-in generation-stable firing/recovery. Output contains hashed family identity and counts, without raw commands, arguments or errors.
A bounded one-shot entry retains adapter state, setup receipts and the sink write budget across processes. Graph setup reuse requires typed read-back. Alertmanager renewal requires a fresh positive reading and a received firing; it sends the frozen original identity only to the lease-aware receiver without adding another transition, graph or command event. UNKNOWN, missing evidence, recovery and old generations cannot renew.
Validation: 185 tests and 16 subtests on the corrected current-base tree pass under the HOME guard (153 watched paths and five resources untouched); ruff passes. The same controlled lease probe changes from an expired firing at time 100 to a received same-identity refresh, retaining one outbox event. Controls cover UNKNOWN, skipped measurements, pending firing receipts, recovery/re-crossing, restart backoff/budget, setup retraction/metadata changes and startup UNKNOWN.
Source only: a reviewed read-only desire-path release is a prerequisite. Schedule/pins, policy, routing, native scheduled delivery and downstream query acceptance remain separate. This PR does not install a listener, schedule or binary, alter routing, or send a test alert.
Current-base integration retains the existing received-firing recovery protocol; the prototype alternate recovery protocol was removed. Renewal reads the prior receiver label set and requires a receipt showing the extended expiry. An expired/absent prior identity is UNKNOWN rather than a new-policy recreation. Final counter/configuration adjustments additionally pass 21 focused controls and four subtests.
Independent review found a stale recovery recipe and inflated retry backoff after successful renewals. The recipe now loads with the existing recovery protocol. Successful lease receipts reset consecutive failures; the same three-success/first-failure probe changes 240 seconds to 30 seconds, matching the no-prior-success control. Restart, legacy success-only counters and recovery-after-failure controls pass.