Require High CPU to persist: port the built-in gauge alerts onto AlertPersistenceGate (#3282) - #3328
Conversation
High CPU fired on a single sample over the threshold and resolved on the next sample under it, on both engines. The shared AlertPersistenceGate now gates both: three consecutive breaching CPU samples to fire, two consecutive clearing samples to resolve, with the per-(server, metric) state persisted on both SKUs so a restart neither re-announces an open incident nor loses a partly-built streak. The gate counts SAMPLES, not sweeps. The alert sweep runs every 30 seconds while a CPU sample advances about once a minute, so counting sweeps would reach three breaches inside ninety seconds - shorter than every excursion measured. AlertServerSnapshot carries the sample instant for the SQL Server path; the PostgreSQL path reads the batch of new Performance Insights samples because that collector delivers five 60-second points at a time. A missing CPU reading now freezes the gate rather than resolving the incident. Store: PgMigrations V117 config.alert_persistence_state, DuckDB v58 config_alert_persistence_state, and the viewer probe's new top rung.
Test pins, not gate arithmetic - AlertPersistenceGateTests already covers the primitive. These cover the wiring: one sample over the bar does not fire, three consecutive samples do, a streak broken by one clear sample never fires, a re-read of the SAME sample instant does not advance the streak, and a replay of the worst measured excursion (two consecutive minutes at 93% against a 20% baseline) produces neither a fire nor a resolve. Also pinned: a missing CPU value freezes the gate instead of announcing a recovery; a restart resumes a partly-built streak and does not re-announce an already-open incident; the documented no-sample-instant fallback counts sweeps; and the PostgreSQL evaluator uses the same gate with the same constants over a batch of samples. Running these caught two things reading them did not. A restarted engine's in-memory cooldown clock is empty, so a standing condition delivered the reminder immediately after a restart - the seeding now stamps the clock when the persisted record says an incident is already open. And the mode-selection test restarted its sample sequence against a shared state store, which silently froze the gate. V116's migration test hands the top-rung claims to V117's, the way it took them from V115's.
The #1830 note cited cpuExceeded, which the gate replaced. The guarantee it describes still holds; it now comes from the null arm returning early.
Both are the same class of mistake this issue is about - one side of a shared mechanism treated differently from the other. A restart stamped the cooldown clock on the SQL Server side but not here, so a PostgreSQL restart over a standing High CPU still delivered the standing-condition reminder on its first pass. And the rising edge bypassed the cooldown here and not there, which would have made one threshold mean two things across the engines the metric names were deliberately shared for. The larger one: a Performance Insights batch can hold three breaching samples followed by two clearing ones, so an incident could open and close inside a single pass. The falling edge was then delivered on its own - a recovery notice for a message nobody received. Nothing is delivered for that pass now, the same call CustomAlertEvaluator makes on the same gate, and the state still records the whole cycle so the next excursion fires normally. Confirmed reachable by driving the real gate over a five-sample batch rather than assumed.
… the constant The README's High CPU row read "Fires when total CPU (SQL + other) exceeds the threshold", which is the rule this change replaced - and it is the sentence a user reads before tuning the threshold. The row now names the sample counts, and a test asserts they match CpuBreachSamples and CpuClearSamples. Writing that pin caught the row spelling one of them as a word, which the pin could not compare against the constant: a doc count that cannot be checked against the thing it describes is a count nobody can check.
#3013's counter is about reads the alert pass performs and swallows, measured against a denominator of alert passes, and an operator reads a non-zero value as the pass going blind on a condition. The gate's persist failure says that about a condition that was evaluated correctly - only the memory of it was lost. SaveOccurrencesAsync already makes exactly this call, in a comment, for exactly this reason; the gate's save did not. Warning rather than Error for the same reason. The seed LOAD is a read and stays counted. Pinned behaviourally rather than by text, because neither the compiler nor an Assert.Contains can carry the claim: a state store whose save always throws still fires on the third sample, still resolves, and leaves the counter at zero.
…raws The poison-wait field comment contrasts itself with CPU as the thing that needs no freshness guard. CPU has one now, for a different reason, and a reader working from that contrast could reasonably delete it. The note also records why poison wait is not a candidate for the gate at all: its breach arm observes once per collector cycle and its clear arm every sweep, so one gate over both would count breaches and clears in different units.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Review summaryReviewed the full diff (all 7 commits) against Overall: this is an unusually well-instrumented change. The core primitive ( Parity: V117 ( Style: T-SQL guidelines do not directly apply here (no SQL Server collector queries changed); the new PostgreSQL SQL in Correctness: left one inline comment on No security issues found (all queries parameterized, no dynamic SQL, no new file/network/process/secret handling), and no missing-index recommendations offered. |
…3282) Four CI failures, three causes, and all four were my verification scope being narrower than my change. Two repo guards had registration points I did not know existed. AlertReadFailureSurfaceTests ratchets that every swallowed alerting catch either counts a read or carries a stated exemption - so the decision to keep the gate's write off that counter has to be DECLARED, which is the right design and is now declared with its reason and its two counts. RepoFileAdoptionTests declares which pins anchor on LF; the new rung test does, so it is declared. Two Lite tests used CPU as the vehicle for suppression and muting without being named Cpu_*, so grepping for the name missed them. Both now drive the full streak. The gap that let all four through: the local harness compiled AlertEngineTests and AlertPersistenceGateTests only. It now compiles both guards too, and they were proved able to fail before their pass was trusted. Lite.Tests cannot run here at all, so the claim its suppression test depends on - that a suppressed streak still advances the gate - is now pinned on the shared engine where it can be executed.
…ened Review caught a second lost edge, and it is the same category as the first. Accumulating a batch's edges into flags compresses a SEQUENCE into a summary: a pre-existing incident's resolve, followed in the same batch by an unrelated fire and its resolve, ends with both flags set and the guard swallowing the whole pass - so the operator never hears that the incident they had open is over, and nothing re-sends it. Confirmed reachable at seven samples by driving the real gate, which is one restart or one collector gap. The first instance got a guard. Two instances of one category means the guard was the wrong shape, so the structure changed instead: the loop stops at the first edge and leaves the rest of the batch for the next sweep, 30 seconds later rather than the collector's five minutes. Both instances become unreachable, the special case and its pin are deleted, and the pass is now the same shape as the SQL Server twin - one observation, one edge, the two arms mutually exclusive by construction. Stated cost: after a data gap a whole excursion can arrive as a fire then a resolve one sweep apart. That is what happened, and reporting both is the only option that cannot drop an edge - the property that failed twice. In steady state the sweep sees each sample alone and this never arises.
#3315 landed V118's number first, so per the ladder rule the later merge renumbers: builtin-alert-persistence becomes V118, StorageVersion goes to 118, and the viewer sentinel takes ordinal 93 with its arm above V117's. The hand-over the ladder documents goes with it. MuteRuleReloadBeaconTests wrote "I am the top rung" when it merged - the all-true probe mapping, the sentinel-is-the-last-argument claim, and RungVersion == SchemaVersion - and all three are now false. Those move to BuiltinAlertPersistenceRungTests and V117's test keeps only its own identity and its own arm, asserted the same future-proof way: every sentinel above its ordinal off, so the next rung does not have to touch it either. CustomAlertCoreMigrationTests took dev's side wholesale; it had already de-topped itself the same way and slightly better. Its doc sentence named V117 as the current top, which this rung makes stale, so it now names the hand-over rather than one holder. All three levels verified by evaluating the real arm list: a fully-migrated store maps to 118, a store at exactly 117 to 117, at exactly 116 to 116, and each one-rung-behind case to its predecessor.
Review summaryWent through the CPU persistence-gate work in depth — Overall: this is unusually well-executed. It follows CONTRIBUTING.md's Darling migration checklist exactly (rung + version + symbolic pins + all four viewer-probe parts), keeps One correctness/parity issue found, posted inline on No SQL-injection, secrets, or missing-index concerns; all new SQL is parameterized and no missing-index recommendations are present. 🤖 Generated with Claude Code |
Splicing the two conflicting V117 doc blocks kept dev's opening <summary> and left mine without one, so the V118 block's run had one closing tag and no opening. The doc-hygiene guard is exactly the instrument for that - it names this as the artifact a keep-both-sides merge produces - and it says restore the opening rather than delete the closing, which would demote a documented block to a loose comment and lose nothing visibly. The run's first line is a whole sentence, so only the tag was lost. The local harness now compiles that guard too, with its output inside the gitignored test bin so its walk-up finds the solution and it scans the real tree instead of refusing to run. It was proved able to fail on this exact mutation before its pass was trusted.
ReviewWent through the full CPU-alert persistence-gate change (SQL Server path in No blocking issues found. Specifically checked and clean:
T-SQL style guide doesn't really apply here (all new SQL is DuckDB/PostgreSQL DDL/DML for Lite/Darling storage, not SQL Server collectors), but the new statements match the existing Solid change; test coverage (including the two-minute real-fleet excursion replay and the stale-sample/repeated-instant cases) is unusually thorough. |
… one Second review catch, and the pin I wrote made it look correct. The batch is empty on about nine sweeps in ten by design - 30-second sweep, five-minute collector - so deriving `breaching` from the batch alone skipped the standing-condition reminder on those sweeps and capped its cadence at the COLLECTOR interval instead of the configured cooldown. Masked at the default 15-minute cooldown, silent below five minutes, and a parity break with AlertEngine.CheckCpuAsync of exactly the kind the comment beside it claimed did not exist. The pre-batch code had the property for free by re-reading the latest reading every sweep. So the single-latest read comes back, for the REMINDER only: after the gate loop, reached only when the batch brought nothing and an incident is open. It never advances the gate and never moves the sample watermark, because it is not a new observation - it answers "is the condition still standing". At most one store read per sweep, which is what the pre-batch code cost, and it is the read that carries the 15-minute freshness bound so a stopped collector cannot leave the reminder firing on an hours-old reading. The pin asserted that GetLatestAsync was ABSENT, which encoded "the batch replaced it" - the wrong property, and a name's absence standing in for a behaviour is how it certified the break. It now pins the two ROLES and their order: the gate reads the batch, the fallback sits after the edge break so it cannot observe, and breaching is computed after it.
|
Reviewed the diff against Correctness. Traced the state machine in both Lite/Darling parity. Good — new Security. All new SQL (Postgres upsert/select in Performance. Persistence writes are skipped on unchanged state via record-struct equality, the PostgreSQL batch read is bounded by Overall this is an unusually well self-documented and self-tested change (mutation table, restart/seed behavior, and the three batch-read defects are all explicitly called out and have pinned regression tests). No high-confidence functional bugs found beyond the one low-severity comment nit above. |
… nice for it to do Review catch on the comment rather than the code. It claimed the skipped sample "is not marked as counted so a later re-read of it can still be used if the capacity column gets backfilled", and neither half holds: once any later sample in the same batch counts, the watermark advances past this one and sample_time > $2 excludes it for good - and there is no backfill to wait for, because RdsCpuIngestor COPYs new rows keyed off MAX(sample_time) and never updates an inserted one. The behaviour is right and unchanged. What was wrong was a rationale a future reader would have relied on, which is the worse kind of comment defect: it reads as a considered decision about a path that does not exist. It now states the real reasons - a capacity-less sample measures nothing the threshold is against, `continue` rather than `break` because it is not an edge and must not stall the batch behind it, and nothing is lost by never revisiting it.
| fingerprint's NEXT incident would decay its already-counted mark to the new window count and report | ||
| the recurrence as nothing new. */ | ||
| /* The BUILT-IN alert catalog's persistence-gate state (#3282), the twin of Darling's | ||
| config.alert_persistence_state (PgMigrations V117) — same columns, same key. "How long must this |
There was a problem hiding this comment.
Stale rung number: config.alert_persistence_state is created by PgMigrations V118 ("builtin-alert-persistence", StorageVersion.SchemaVersion = 118), not V117 — V117 is the mute-rule-reload-beacon trigger rung. BuiltinAlertPersistenceRungTests.RungVersion confirms 118.
Same stale "V117" reference appears in two more places and should be bumped to V118 for consistency:
Lite/Database/DuckDbInitializer.cs:1554("porting Darling's V117")Lite/Services/DuckDbAlertHistoryStore.cs:533("the Lite twin of Darling's V117 table")
This codebase leans heavily on these rung-number citations for cross-app parity verification (per CONTRIBUTING.md's two-store-parity guidance), so a wrong number here is exactly the kind of stale cross-reference that misleads the next person trying to verify Lite/Darling stay in sync.
|
Reviewed the CPU alert persistence-gate work (#3282 → shared Overall: this is exceptionally well-tested and carefully reasoned — the freshness-by-sample-instant mechanism, the "at most one edge per pass" batch-processing fix for the PostgreSQL side, the restart/seeding semantics, and the missing-value freeze behavior are all covered by targeted tests with clear rationale in the comments. No T-SQL was touched, so the collector-query conventions ( One nit (posted inline): three comments in the Lite codebase ( No correctness bugs, no Lite/Darling behavioral parity drift, and no security concerns found beyond that. |
…n that they do #3282's rung renumbered from V117 to V118 when #3315 merged first, and five comments across both SKUs still cite V117 - which is now the mute-rule reload-beacon rung. Review found three of them on #3328; the other two are in LiteAlertStateStore and PgAlertStateStore. The repo uses exactly these citations to check Lite/Darling parity, so a wrong one sends the next person to the wrong rung. This was the last commit on #3328 and the merge landed one commit short of it, so it comes back on its own. The pin derives the number from StorageVersion.SchemaVersion rather than comparing against a literal, so a future renumber reds it instead of leaving silent copies. Each phrase carries its own SUBJECT, which is what makes a citation attributable without parsing comments at all - and both earlier spellings are why. Scoped to the FILE it flagged nine correct citations for other rungs, which is a check someone turns off. Scoped to comment BLOCKS it needed a hand-rolled line-prefix comment filter, which CommentFilterAdoptionTests refuses without a stated bound - correctly, and the bound I measured for it came back unreliable on its own terms. A phrase naming both the table and the rung needs neither. Reddened three ways before its pass was trusted: one citation reverted to V117, a SchemaVersion bump that leaves all five behind, and a reworded phrase - which has to fail loudly rather than silently matching nothing, so the pattern is asserted to match at all before its captures are compared. Assert.NotEmpty is load-bearing there: Regex.Escape escapes '{' and not '}', which was verified by running it rather than assumed.
1 participant
Closes #3282 — except that PM merges to
dev, where closing keywords never fire, so it needs closing explicitly.What changed
AlertEngine.CheckCpuAsyncandDarlingWorker.EvaluatePgCpuAsyncboth fired on a single sample over the threshold and resolved on the next sample under it. Both now go through the sharedAlertPersistenceGatethat shipped with #3285: three consecutive breaching CPU samples to fire, two consecutive clearing samples to resolve.The gate counts SAMPLES, not sweeps, and that distinction is the fix
The alert sweep is 30 seconds (
DarlingWorker.s_alertSweepInterval, and Lite's 30-second overview timer). A SQL Server CPU sample advances about once a minute — the SCHEDULER_MONITOR ring buffer's own granularity, whichCpuUtilizationCollector's #2749 comment measures as "a real ~60s gap to the next", andcpu_utilizationis scheduled every minute.So counting sweeps would reach three breaches in ninety seconds. The worst excursion measured on the fleet held above the bar for two consecutive one-minute samples, which is longer than that — a sweep-counting gate would still have fired it, while looking like a fix.
AlertServerSnapshottherefore carriesCpuSampleTimeUtc, and an observation whose sample instant has not advanced does not count; the streak simply holds.The PostgreSQL path needed the opposite shape. Performance Insights is queried at a 60-second period (
RdsCpuIngestor) butpg_cpu_utilizationis a five-minute collector, so five points land together — counting only the newest would discard four in five and make three consecutive breaches take three batches, about fifteen minutes.DarlingPgCpuUtilizationReader.GetSamplesSinceAsyncreads the batch oldest-first, so three samples means three minutes on both engines, which is also what keeps one threshold meaning one thing across them.Where N=3 and M=2 come from
Measured independently of the issue, on a 42-server store over the 24 hours to 2026-09-11: seven High CPU / CPU Resolved pairs, elapsed 42, 55, 70, 87, 87, 147 and 147 seconds — median 87 s, none over 2.5 minutes. Pulling the 1-minute CPU series for the longest of them shows total CPU going 39 → 68 → 73 → 93 → 93 → 53 → 55 → 21 against the 80% default: two consecutive samples over the bar, on a server whose baseline is ~20%. #3282's own figures are the same shape (55 s, 87 s, a 30-second pair, and 99% back to 23% inside two minutes).
Three samples is the first bar above all of it, and the shortest excursion that could still be TRUE when a human opens the message. Stated plainly, so the trade is visible rather than implied: on that fleet, in that window, N=3 would have delivered zero High CPU alerts. That is the intended outcome — the issue's position is that brief excursions above a steady baseline are correctly detected and are not incidents — but it is a real behaviour change and the number to revisit if the fleet later proves otherwise.
M=2 is deliberately smaller than N. The costs are not symmetric: a late resolve is a stale open incident, an early one is a resolve/fire pair, which is the noise this issue is about. Two is the smallest value that survives one sample dipping under the bar during a genuine saturation event; three would hold incidents open a further minute and buy nothing.
Not configurable, on purpose
PostgresAlertEvaluatoralready states the position this follows: configuration gets added when someone wants a different number, not speculatively. A knob here means aconfig_alert_settingscolumn, a rung, a requiredIAlertEngineSettingsmember, Settings-window work and MCP plumbing — and half of that is worse than none. #3314 is open precisely because a delivery-governing number lives in the store and is unreachable fromget_alert_settings/update_alert_settings; adding a second such number is the one outcome to avoid.CpuBreachSamplesandCpuClearSamplesare public constants, so raising either is a one-line diff.Restart behaviour, which is answered rather than inherited
IAlertStateStoregainsLoadAlertPersistenceAsync/SaveAlertPersistenceAsync— required members, so the compiler found all three test fakes rather than leaving them silently no-op. Both real implementations land together:PgAlertStateStoreover V117config.alert_persistence_state,LiteAlertStateStoreover DuckDB v58config_alert_persistence_state. A separate table rather than columns onconfig_edge_trigger_watermarksfor the reason V61'sincident_occurrenceswas split out: that column is one monotonic int, and Lite'sINSERT OR REPLACEthere names a partial column list, so a streak living on it would zero itself on every fired blocking or deadlock alert.Firingbit stops the gate producing a second rising edge — and the seeding also stamps the cooldown clock, because those dictionaries are in-memory by design, so an empty clock plus a still-breaching condition delivered the standing-condition reminder seconds after a restart. That one was found by running the pins, not by reading them.The batch read cost three defects, and two of them changed the fix rather than getting a guard
The batch is what makes three samples mean three minutes on the PostgreSQL side. It also cost three defects, all three found rather than reasoned about, and the third changed a pin as well as the code.
CpuClearSamples + CpuBreachSamples + CpuClearSamples) by driving the real gate, which is one restart or one collector gap, well inside the 15-minute freshness window and the batch limit of 32.Two instances of one category means the guard was the wrong shape. The structure changed instead: the pass takes at most one edge and leaves the rest of the batch for the next sweep, 30 seconds later —
EvaluatePgCpuAsyncruns on the 30-second alert sweep, not the collector's five minutes, which is what makes that cheap. Both instances become unreachable, the special case and its pin are deleted, and the pass is now the same shape as the SQL Server twin: one observation, one edge, the fire and resolve arms mutually exclusive by construction. What is pinned now is thebreakand the absence of the accumulators, because a reintroduced flag is the category returning.Stated cost: after a data gap, a whole excursion can arrive as a fire and then a resolve one sweep apart. That is what happened — the condition did hold for three samples and did then clear — and reporting both is the only option that cannot drop an edge, which is the property that failed twice. In steady state the sweep sees each sample on its own and this never arises.
AlertEngine.CheckCpuAsyncof exactly the kind the comment beside it claimed did not exist. The single-latest read is back for the reminder only: after the gate loop, reached only when the batch brought nothing and an incident is open, never advancing the gate or the sample watermark. One store read per sweep at most, which is what the pre-batch code cost, and it is the read carrying the 15-minute freshness bound so a stopped collector cannot leave the reminder firing on an hours-old reading.The pin for that one had encoded the wrong property. It asserted
GetLatestAsyncwas absent, meaning "the batch replaced it" — a name's absence standing in for a behaviour, which is how it certified the break. It now pins the two roles and their order: the gate reads the batch, the fallback sits after the edgebreakso it cannot observe, andbreachingis computed after it.A restart also stamped the cooldown clock on the SQL Server side and not the PostgreSQL side, so a PostgreSQL restart over a standing High CPU still delivered the reminder on its first pass. And the rising edge bypassed the cooldown on the PostgreSQL side only, which would have made one threshold mean two things across the engines whose metric names #2719 deliberately shared. Both now match.
A missing CPU reading now freezes the gate
Both evaluators fell through to their resolve arm on an absent value. On SQL Server that sent "
<server>: Total CPU back to %" — a recovery message with no number in it, about a measurement nobody took. On PostgreSQL it resolved on an absent capacity sample, which #3281 had already established must never be treated as a reading. Resolving on absent evidence fabricates a recovery exactly as firing on it fabricates an alert, so the streak is frozen instead, the wayCustomAlertEvaluatorhandles a null scalar.The alert catalog now states the rule
README.md's High CPU row read "Fires when total CPU (SQL + other) exceeds the threshold" — the rule this replaces, and the sentence a user reads before tuning the threshold. It now names the sample counts, and a test asserts they matchCpuBreachSamples/CpuClearSamplesrather than being a prose copy of them. Writing that pin immediately caught the row spelling one count as a word, which the pin could not compare against the constant; a doc count that cannot be checked against the thing it describes is a count nobody can check.Explicitly out of scope
The rolling-count alerts. Deadlocks and blocking are edge-triggered over a rolling 1-hour count by
RollingCountAlertGate(#1091): a deadlock that happened did happen, and requiring it to persist is not a meaningful question. The gate is not applied there.The other gauges, and each for a reason rather than by omission:
wait_statsrow exists, the clear arm on every sweep — so one gate over both would count breaches in collector cycles and clears in sweeps. That asymmetry needs its own design, not this one.PvsAlertGateexists at all), and tempdb reserved space is a high-water mark that does not flap on a one-minute scale. Neither appears in No alert requires its condition to PERSIST: real fire/resolve pairs 55-87 seconds apart on both engines — and it is not the re-fire class closed seven times #3282's measurements, and neither produced a fire/resolve pair in the 24-hour window read for this PR. Applying a debounce to a signal that does not flap adds latency for nothing.Erik's ask was High CPU specifically. If tempdb or PVS later show the same fire/resolve shape in
config_alert_log, adopting the same gate is a small diff — the seam and the table are now there for it.Verification
Darling.Tests/Lite.Teststargetnet10.0-windowsand cannot run on macOS, so the actual test.csfiles were compiled into anet10.0console harness with an xunit shim and RUN: 90 passed, 0 failed across all ofAlertEngineTestsandAlertPersistenceGateTests. The harness was proved able to fail before any pass was trusted.AlertPersistenceGateTestsalready covers the primitive's arithmetic, so the new pins are about the wiring. Each was then checked by mutating the mechanism it guards:CpuBreachSamples3 → 1 (the pre-#3282 rule)Cpu_OneSampleOverTheBar_DoesNotFireandCpu_TheMeasuredTwoMinuteExcursion_IsNotAnIncidentCpu_RepeatedSampleInstant_DoesNotAdvanceTheStreakCpu_AStreakBrokenByOneClearSample_NeverFiresCpu_MissingValueFreezesTheGate_AndNeverAnnouncesARecoveryCpu_ARestartDoesNotReAnnounceAnAlreadyOpenIncidentCpu_APersistenceSaveFailure_StillFires_AndIsNotCountedAsASwallowedReadThree of the six redden a single pin each, so the guards discriminate rather than failing as a block.
That last one is a mistake this branch actually made and then fixed.
SaveOccurrencesAsyncalready carries a comment ruling on it — #3013's counter is about READS the alert pass swallows, against a denominator of alert passes, and an operator reads a non-zero value as the pass going blind on a condition; a write in its numerator says that about a condition that was evaluated correctly, where only the memory of it was lost. The gate's save had it wrong. It is pinned behaviourally rather than by text because neither the compiler nor anAssert.Containscan carry the claim: a store whose save always throws still fires on the third sample, still resolves, and leaves the counter at zero.The first CI round was red on four tests and every one was local verification scope being narrower than the change: two repo-wide guards with registration points this branch did not know about (
AlertReadFailureSurfaceTestsratchets that every swallowed alerting catch either counts a read or carries a stated exemption — which is the right design, and the decision to keep the gate's write off that counter is now declared with its reason;RepoFileAdoptionTestsdeclares which pins anchor on LF), and two Lite tests that use CPU as the vehicle for suppression and muting without being namedCpu_*. The harness now compiles both guards, and both were proved able to fail before their pass was trusted.Lite.Testscannot run here at all, so the claim its suppression test depends on — that a suppressed streak still advances the gate — is now pinned on the shared engine where it can be executed.Two checks that could not be expressed as a driven test were run as their own instruments and each proved able to fail: the same-pass open-and-close case was driven through the real
AlertPersistenceGateover a five-sample batch to confirm the guard is reachable (and, as a cross-check, the measured eight-sample excursion delivered as one batch still does not fire), and theEvaluatePgCpuAsyncsource assertions were executed directly against the shipped file.The V117 rung was verified against the shipped
PgMigrations.Scriptsin a secondnet10.0harness (top rung, dense ladder, schema-qualified DDL, every column, naive timestamps, no reload beacon) — also proved able to fail. The viewer probe cannot run locally (the Viewer isnet10.0-windows), so its wiring was checked by independent arithmetic over the real source instead: 93 top-level sentinel expressions inStoreSchemaProbeSql, 93reader.GetBooleanordinals that are dense and unique, 93 mapper parameters, and a strictly descending arm list whose top returns 117. That check was proved able to fail too. The xUnit form of it runs in thebuildjob.