diff --git a/.claude/probe-axes.md b/.claude/probe-axes.md index 866cbcb2..b19a06b0 100644 --- a/.claude/probe-axes.md +++ b/.claude/probe-axes.md @@ -46,13 +46,17 @@ and `GeneratedHarnessTests` in the conventions tests holds each file to its app' - How a row ends: a check returns null or the failure text, and runs through `Rec.Run`, which fails the test for a violation or a harness error and skips it, with its reasons, for any other outcome but a pass; a check that throws ends as a harness error, never a violation, and no seed counts it caught. A check - says what it saw with `r.Note(NoteKeys.Name, detail)`. The key, a member of `NoteKeys` in `harness/Row.cs`, - alone decides the outcome. Keys are constants the compiler checks: a new one goes into `NoteKeys` with its - outcome on one line, and a string, a misspelled member or a key built from text does not compile (decision - 103). A probe outside the namespace `Probes.Shapes` writes `using NoteKeys = Probes.Shapes.NoteKeys;`. + says how it compared with `r.Note(NoteKeys.Name, detail)`. The key, a member of `NoteKeys` in `harness/Row.cs`, + alone decides the note's outcome: every key is a pass key but `framework-refuses`, the three-sided + differential's expected refusal, and a check that did not apply, refused or could not tell says so through + its scope with a citation (below), never a note key. Keys are constants the compiler checks: a new one goes + into `NoteKeys` with its outcome on one line, and a string, a misspelled member or a key built from text does + not compile (decision 103). A probe outside the namespace `Probes.Shapes` writes `using NoteKeys = Probes.Shapes.NoteKeys;`. Never match a failure's text to excuse it. A difference the framework makes on its own is shown by a baseline in the same row (the document as ASP.NET writes it without `AddTypeKit()`, - `Hx.DocRefusedByAspNetAlone`), not by a substring. A row that would fail for an open finding calls + `Hx.DocRefusedByAspNetAlone`, a refusal compared by exception type and stage as `Sides.Alike` compares one), + not by a substring. A document both refuse alike is the framework's refusal for a check that compared nothing + else, and leaves a check that compared an answer on the wire first to be judged by that answer. A row that would fail for an open finding calls `r.Skip("F7", why)`, the finding as a string literal: `register` and `publish` refuse a skip whose finding is built from text or is not open in the ledger, naming the file and line. The gate refuses while any row skips, and the replay fails a skip whose finding the ledger has fixed or dismissed. diff --git a/.claude/skills/probe-triage/SKILL.md b/.claude/skills/probe-triage/SKILL.md index f4fc58ff..1373245c 100644 --- a/.claude/skills/probe-triage/SKILL.md +++ b/.claude/skills/probe-triage/SKILL.md @@ -122,8 +122,8 @@ dotnet tools/BuildVerdict/bin/Release/net10.0/build-verdict.dll test --project t - the citation, where the corpus holds one: a `not-promised` entry of the claim whose `source` is the fact's file and whose `sentence` is quoted from it (step 3's form), an `exclude` group whose `reason` - names `KB-NNN`, or a value in the axes file's `refused` object whose text names `KB-NNN`. Where the - check excuses the row by a note key, the key's line in `NoteKeys` names the fact in its comment; + names `KB-NNN`, or a value in the axes file's `refused` object whose text names `KB-NNN`, or the check's + own `Cite.Fact("KB-NNN")`; - with a finding id: ```bash diff --git a/design/adr/0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md b/design/adr/0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md new file mode 100644 index 00000000..24002634 --- /dev/null +++ b/design/adr/0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md @@ -0,0 +1,514 @@ +# 0108. A probe check earns credit against a baseline, applicability is declared and the harness compares three sides; WS2 closes, and WS3 to WS6 are dropped + +Date: 2026-10-17 +Status: accepted + +Context: 2026-10-16 and 2026-10-17, the second workstream (WS2, the harness) of the concept for finding issues with +agents, which set out to make the probe corpus (`tools/probe/corpus`) and the prober (`tools/Prober`) find defects +mechanically. The gate round of 2026-10-16 that preceded it had written 18 probes and found two product defects and about +ten harness traps, five traps for each defect, and left five failing rows of C07 untriaged (`.claude/skills/probe-triage`). +What the traps had in common was that the harness, not the product, decided a row: + +- **A row was credited all or nothing.** A row ended as one category, so one excuse anywhere in it voided the comparisons + that passed beside it, and one passing comparison credited every setting the row held. At 265309f, the commit before + the first per-check change, BaseProbe passed 135 of its 411 rows and C07's two probes 545 of 822 (66.3%); L01 credited + nothing for 160 rows whose write passed, because their read was refused for the wrapped type too. +- **An excuse was a string.** A check noted an outcome key of `NoteKeys` (63 keys at the start of WS2: 25 pass, 20 + inapplicable, 17 expected refusal, 1 inconclusive, [decision 103](0103-a-harness-error-is-never-a-catch-and-note-keys-are-constants.md)), + and the harness decided rows by an exception's words wherever a substring could tell (`stj-maxdepth`, + `mvc-400-unwritable-under-source-gen`, `json:read-refused-by-stj-alone`, the OpenAPI 2.0 and enum paths). +- **The baseline was the product beside itself.** The wrapped twin ran with TypeKit, not without it, and at a nullability + of its own (`Hx.PlainNull`), and a source-generated context fell back to reflection where it did not list a shape + (`+reflection-fallback`), so about 270 pairs could never be earned as the row named them. +- **Nothing tied a probe to what it read outside its footprint**, the generated harness files had lost their + generators, and an obligation could not say where it does not apply. + +The maintainer ended the programme on 2026-10-17 at WS2 and the 2.2.0 release. + +## Decision + +**A row runs checks, and the ledger credits checks.** `Row.Check(obligation, baseline, body)` (#41) is one check per +obligation the row's axes select, named by the claim's obligation id ([decision 104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md)). +A returned failure is the check's violation, an exception its harness error, and a pass without a `Baseline` (`Twin`, +`Framework`, `RoundTrip`, `Model`, `Refusal`, `Cost`, `Fact("KB-NNN")`, `Docs("file#anchor")`) a harness error. A row +passes when no check failed and one passed. The ledger credits each passing check with the settings it read and the +row's, never a pair put together from two checks, and `prober seeds` counts a seed naming an obligation caught only by a +clean violation of a check naming it. Evidence is schema version 2. + +**An excuse is a closed citation.** A check that did not apply, refused or could not tell says so through its scope: +`c.Inapplicable(Cite…)`, `c.Refused(Cite…)`, `c.Inconclusive(reason)`. A `Cite` names a not-promised entry of the claim, +a docs section, a verified fact or a record, with a string literal; `register`, `publish` and `ProbeCiteTests` refuse +anything else. A fact that prunes or excuses a row is executable, and its measured answer is a fixture beside its test +([decision 107](0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md)). `NoteKeys` keeps the pass keys, which +say how a check compared, and one outcome key, `framework-refuses`; the 41 outcome keys no source called any more are +gone (27 keys remain). + +**Applicability is declared.** An obligation's `inapplicable` entries in `claims.json` (axes and a cite, #50) take a pair +out of what the obligation wants; the universe, `rows`, the gate and `Row.Check` read the same declaration, and a pair no +obligation wants is counted as the gate counts an exclusion. + +**The harness compares three sides.** For the http, json and text families (#58, `harness/Sides.cs`): (1) the wrapped +twin without TypeKit, (2) the twin with it, (3) the value object. (1) and (3) refusing alike, by operation, stage, +exception type and status and never by message, is the framework's refusal (`framework-refuses`, an expected refusal +against side (1)'s baseline); one refusing where the other answers is a violation unless the check cites the sentence +that makes it intended; (2) differing from (1) is a violation. Side (1) comes from a fact's fixture where one covers the +row, else it is measured, on every row of an obligation that promises non-interference and lazily elsewhere. +`ProbeMessageTextTests` refuses a message read that excuses a row. **Decision 99's document rule joins it**: a document +ASP.NET alone refuses is compared with the one `AddTypeKit()` refused by type and stage (`Hx.DocAlike`), and leaves a +check that compared an answer on the wire first to be judged by that answer (`doc:refused-by-aspnet-alone`); a check +that compared nothing else is the framework's refusal and earns nothing. No message comparison is kept. + +**The twins and the context are generated with the shapes** (#59): the twin is the wrapped type at the value object's +nullability, `Sides.Twins` holds both sides to the same shape, and one `ProbeContext` lists every shape, so +`json.TypeInfoResolver=source-generated-context` is strict and the reflection fallback is the existing +`json.TypeInfoResolverChain=context-then-default`. + +**Rows are per probe, and `watched` stales a footprint** (#39): `rows/.json.gz`, deduplicated, and a registry +entry's `watched` paths stale the probe like its footprint. **The harness generators are C# file-based apps** +(`tools/probe/generators`, #35), held to their output by `GeneratedHarnessTests`. + +**The prober closes the loop it left open:** `prober fix F --commit SHA` records a finding fixed; `rows` never places +an excluded pair or one no obligation wants, and leaves out of a row an axis none of whose values can join it; +`prober gaps` lists, from the last replay's evidence, the wanted pairs no obligation wanting them has a passing check +that read both their axes of. + +**WS3 to WS6 are dropped**, by the maintainer's decision of 2026-10-17: WS3 (beyond pairs: three-way and deeper +combinations), WS4 (the mutation nightly), the second half of WS5 (the finder benchmark at equal budgets) and WS6 +(PostgreSQL in CI). Their work is not scheduled; the roadmap says so. + +## Consequences + +- **A probe writer** writes one `Row.Check` per obligation the row reaches, with its baseline; cites every excuse or + ends the check inconclusive; declares in `claims.json` an inapplicability the axes alone decide; builds two-sided + comparisons with `Sides.Twins` and refuses alike by `Sides.Alike`; never reads a message to excuse a row; names + `watched` paths the footprint cannot see; and gives each obligation a seed that a check of that obligation catches. +- **The gate counts** pairs earned by passing checks of probes current at their identity and footprint, less exclusions + and pairs no obligation wants. It now counts nothing for a document both hosts refuse in a check that compared nothing + else: 2,001 pairs that 401 such rows alone had credited are uncovered (C12 821, C13 48, C16 455, C21 677), nearly all + under `json.ReferenceHandler=Preserve` or `json.MaxDepth=3`, where System.Text.Json refuses the document for plain + ASP.NET too (decision 99). Each needs a declaration of its document obligations' inapplicability there, citing record + 0099 or a fact, and rows that hold the rest of those pairs without those settings. +- **Open, not done here:** + - The EF twins at the value object's nullability (#59): `EfEntry.Plain` is the type `From` takes, and an EF twin is the + entity's own wrapped-type property; giving EF twins the value object's nullability is its own step, with an EF + replay. + - The untriaged candidate rows #59 saw in regenerated arrays it did not commit, open leads for whoever regenerates + those files next: C21 under OpenAPI 2.0 with a class value object over a struct at the root, where ASP.NET writes the + twin's nullable body (`double?`, `int?`) without `pattern` and the value object keeps it (rows 6 and 143 of that + file); a validated struct over `string` as a nullable property, the twin `{"type":"string"}` against the value + object's `{}` (row 115); a validated class over `Uri`, the twin's `x-nullable` (row 181); a sealed record class over + `char` at the root, `format: char` alone against `type: string, minLength/maxLength 1` (row 282); a record class + bound from a form, the twin's `A`/`B` properties (rows 53, 208); `Instant` as a dictionary value (row 134); and C10 + row 266, an array of value objects over `byte` under `default-with-renaming-modifier` with + `input.json=metadata-property` that does not read back its own document, under `ignore-cycles-unaffected`. + - The narrowing of the pairs an obligation wants by its own `axes` (#54's option 2): an obligation that never reads an + axis still wants every pair over it, as C11's parse-shape obligations wanted `shape.wrapped`'s; `prober gaps` now + lists such pairs, and none is open today. + - `polymorphic-base-property` as a value of `shape.position` (#51): the axes file is in every probe's identity, so + adding it stales the corpus and widens every claim over `shape.position`. +- `gaps` reads the last replay's evidence beside the tracked corpus and the generated axes (decision 104 said it read + only those), and still decides nothing. +- A row that `rows` writes may leave an axis out where it cannot apply; a harness that reads it there ends the row as a + harness error, never a pass. + +## Evidence + +Measured 2026-10-17 on Linux x64 (32 threads, otherwise idle) with the pinned SDK. **Before**: release/2.2.0 at +265309f (#39 merged, before #41), replayed from an empty state in a detached worktree with its own built tools. +**Now**: `build/ws2-close-out` at 5c5affe, the same replays. Each replay ran `replay --all --family F` for http (one +probe process at a time, as a replay recording footprints runs), json, text, build and ef, with footprints. + +**Replays.** Before: 43 probes, 43 passed; 677 s wall (http 398 s, json 42 s, text 75 s, build 53 s, ef 102 s, each the +replay's own "in all"). Now: 45 probes, 45 passed; 818 s wall (http 502 s, json 50 s, text 83 s, build 68 s, ef 106 s). +The two new probes (C21, C22), side (1)'s hosts and the larger `ProbeContext` (#59: the Samples' rebuild after a harness +change went from 38 s to 74 s) are the difference. + +**Row for row, per probe.** A row that runs no `Row.Check` counts as one check. "Rows passing" is `Rec.Run`'s rule: no +violation or harness error in the row or a check, and one check that passed. Regained and lost rows are counted only +where the row is the same at the same index before and now; where rows were regenerated (#49, #59) or appended (#54) the +table says so, and those probes' pairs are in the next table. + +| Probe | rows before | checks before | rows now | checks now | rows passing, before → now | regained (category before) | lost (category now) | +|---|---|---|---|---|---|---|---| +| BaseProbe | expected-refusal 39, inapplicable 237, pass 135 | expected-refusal 39, inapplicable 237, pass 135 | expected-refusal 123, inapplicable 109, inconclusive 43, pass 136 | expected-refusal 180, inapplicable 388, inconclusive 63, pass 986 | 135 → 332 | 197 (expected-refusal 30, inapplicable 167) | 0 | +| C01DecimalSpellingProbe | expected-refusal 5, inapplicable 4, pass 108 | expected-refusal 5, inapplicable 4, pass 108 | expected-refusal 5, inconclusive 15, pass 97 | expected-refusal 25, inconclusive 32, pass 528 | 108 → 111 | 3 (inapplicable 3) | 0 | +| C02OptionalTextProbe | pass 46 | pass 46 | pass 46 | pass 52 | 46 → 46 | 0 | 0 | +| C03ComposedProbe | pass 648 | pass 648 | expected-refusal 397, pass 251 | expected-refusal 794, pass 2125 | 648 → 648 | 0 | 0 | +| C04WithoutFaultsProbe | expected-refusal 2, pass 394 | expected-refusal 2, pass 394 | expected-refusal 135, pass 162 | expected-refusal 135, pass 493 | 394 → 297 | 0; 284 of 297 regenerated | 0 | +| C05RuleProbe | pass 645 | pass 645 | pass 645 | pass 1666 | 645 → 645 | 0 | 0 | +| C06FaultKindsProbe | expected-refusal 4, pass 404 | expected-refusal 4, pass 404 | pass 306 | pass 1224 | 404 → 306 | 0; 295 of 306 regenerated | 0 | +| C07BlanketMappingProbe | expected-refusal 1, pass 410 | expected-refusal 1, pass 410 | expected-refusal 44, pass 367 | expected-refusal 44, pass 431 | 410 → 411 | 1 (expected-refusal 1) | 0 | +| C08SameNameWarningProbe | pass 318 | pass 318 | pass 302 | pass 540 | 318 → 302 | 0; 292 of 302 regenerated | 0 | +| C09BaseRecordEqualityProbe | pass 101 | pass 101 | pass 101 | pass 202 | 101 → 101 | 0 | 0 | +| C10PreserveGuardProbe | expected-refusal 14, pass 292 | expected-refusal 14, pass 292 | expected-refusal 100, inapplicable 1, inconclusive 2, pass 203 | expected-refusal 112, inapplicable 1, inconclusive 2, pass 497 | 292 → 294 | 2 (expected-refusal 2) | 0 | +| C11ParseShapesBuildProbe | pass 241 | pass 241 | inapplicable 4, pass 243 | inapplicable 4, pass 243 | 241 → 243 | 0; 6 rows appended | 4 (inapplicable 4) | +| C12DescribedConverterProbe | pass 332 | pass 332 | expected-refusal 156, pass 145 | expected-refusal 338, pass 363 | 332 → 145 | 0; 289 of 301 regenerated | 5 (expected-refusal 5) | +| C13PatchRequiredProbe | expected-refusal 55, pass 301 | expected-refusal 55, pass 301 | expected-refusal 82, inconclusive 51, pass 163 | expected-refusal 105, inconclusive 51, pass 436 | 301 → 273 | 0; 282 of 296 regenerated | 0 | +| C14DeclaredProblemProbe | expected-refusal 3, pass 418 | expected-refusal 3, pass 418 | expected-refusal 273, pass 148 | expected-refusal 633, pass 764 | 418 → 420 | 2 (expected-refusal 2) | 0 | +| C15MvcFaultStatusDocProbe | expected-refusal 7, pass 398 | expected-refusal 7, pass 398 | expected-refusal 104, pass 301 | expected-refusal 104, pass 447 | 398 → 404 | 6 (expected-refusal 6) | 0 | +| C16OptionsAgreementProbe | pass 310 | pass 310 | expected-refusal 93, pass 203 | expected-refusal 127, pass 287 | 310 → 203 | 0; 287 of 296 regenerated | 3 (expected-refusal 3) | +| C17Cmtk0014Probe | pass 310 | pass 310 | pass 283 | pass 512 | 310 → 283 | rows regenerated | rows regenerated | +| C18Cmtk1023Probe | pass 280 | pass 280 | pass 280 | pass 840 | 280 → 280 | 0 | 0 | +| C19Cmtk0010KindsProbe | pass 302 | pass 302 | pass 282 | pass 508 | 302 → 282 | rows regenerated | rows regenerated | +| C20ShouldHoldProbe | pass 261 | pass 261 | pass 261 | pass 727 | 261 → 261 | 0 | 0 | +| C21OpenapiSchemaParityProbe | - | - | expected-refusal 129, inapplicable 4, inconclusive 6, pass 155 | expected-refusal 230, inapplicable 7, inconclusive 12, pass 248 | 0 → 155 | new probe | new probe | +| C22GeneratorRefusalsProbe | - | - | pass 144 | pass 432 | 0 → 144 | new probe | new probe | +| E01ColumnParityProbe | pass 175 | pass 175 | pass 175 | pass 307 | 175 → 175 | 0 | 0 | +| E02ValueTranslatesProbe | expected-refusal 12, pass 203 | expected-refusal 12, pass 203 | inapplicable 12, pass 203 | inapplicable 24, pass 406 | 203 → 203 | 0 | 0 | +| E03ColumnRoundTripProbe | pass 177 | pass 177 | pass 177 | pass 177 | 177 → 177 | 0 | 0 | +| E04IntegerKeyProbe | pass 28 | pass 28 | pass 28 | pass 28 | 28 → 28 | 0 | 0 | +| E05OwnCompareToProbe | pass 90 | pass 90 | pass 90 | pass 90 | 90 → 90 | 0 | 0 | +| E06OwnConverterNoValueProbe | pass 12 | pass 12 | pass 12 | pass 12 | 12 → 12 | 0 | 0 | +| E07ClosedSetNameProbe | pass 9 | pass 9 | pass 9 | pass 11 | 9 → 9 | 0 | 0 | +| E08OptionResultNotColumnsProbe | pass 20 | pass 20 | pass 20 | pass 40 | 20 → 20 | 0 | 0 | +| E09OrNoneInDbProbe | pass 18 | pass 18 | pass 18 | pass 18 | 18 → 18 | 0 | 0 | +| E10ReadBackSkipsValidationProbe | pass 30 | pass 30 | pass 30 | pass 60 | 30 → 30 | 0 | 0 | +| E11UseTypeKitOrderProbe | pass 14 | pass 14 | pass 14 | pass 14 | 14 → 14 | 0 | 0 | +| E12ConventionByTypeProbe | pass 12 | pass 12 | pass 12 | pass 12 | 12 → 12 | 0 | 0 | +| E13PrimitiveCollectionProbe | pass 10 | pass 10 | pass 10 | pass 10 | 10 → 10 | 0 | 0 | +| E14InheritanceKeyProbe | pass 9 | pass 9 | pass 9 | pass 9 | 9 → 9 | 0 | 0 | +| E15CompositeKeyProbe | pass 8 | pass 8 | pass 8 | pass 8 | 8 → 8 | 0 | 0 | +| E16OwnConfigurationWinsProbe | pass 25 | pass 25 | pass 25 | pass 25 | 25 → 25 | 0 | 0 | +| L01JsonBytesProbe | expected-refusal 180, inapplicable 7, inconclusive 1, pass 393 | expected-refusal 180, inapplicable 7, inconclusive 1, pass 393 | expected-refusal 186, inapplicable 112, inconclusive 21, pass 262 | expected-refusal 202, inapplicable 128, inconclusive 22, pass 810 | 393 → 548 | 160 (expected-refusal 160) | 5 (inapplicable 1, inconclusive 4) | +| L02NoQuoteProbe | expected-refusal 29, inapplicable 11, pass 546 | expected-refusal 29, inapplicable 11, pass 546 | inapplicable 218, inconclusive 132, pass 236 | inapplicable 302, inconclusive 159, pass 843 | 546 → 434 | 16 (expected-refusal 15, inapplicable 1) | 128 (inapplicable 96, inconclusive 32) | +| L03OneRuleProbe | expected-refusal 5, inapplicable 13, pass 568 | expected-refusal 5, inapplicable 13, pass 568 | expected-refusal 56, inapplicable 79, inconclusive 21, pass 430 | expected-refusal 75, inapplicable 95, inconclusive 22, pass 1330 | 568 → 561 | 13 (expected-refusal 2, inapplicable 11) | 20 (expected-refusal 14, inconclusive 6) | +| L04RoundTripProbe | expected-refusal 78, inapplicable 127, pass 381 | expected-refusal 78, inapplicable 127, pass 381 | expected-refusal 18, inapplicable 74, inconclusive 59, pass 139 | expected-refusal 21, inapplicable 111, inconclusive 64, pass 994 | 381 → 290 | rows regenerated | rows regenerated | +| L05EqualityProbe | pass 309 | pass 309 | pass 309 | pass 793 | 309 → 309 | 0 | 0 | +| L06OffTheWireProbe | expected-refusal 9, inapplicable 12, pass 304 | expected-refusal 9, inapplicable 12, pass 304 | expected-refusal 21, inapplicable 3, inconclusive 29, pass 272 | expected-refusal 22, inapplicable 15, inconclusive 38, pass 575 | 304 → 303 | 0 | 1 (inconclusive 1) | +| **all** | 10850 rows | 10850 checks (9995 pass) | 10629 rows | 25810 checks (21123 pass) | 9995 → 9871 | 400 | 166 | + +The 400 rows regained all earned nothing before (expected refusal or inapplicable). In 391 one excuse had voided a row +whose other comparisons passed (BaseProbe 197, L01 160, L02 16, L03 13, C01 3, C10 2; #53, #57). C07's 1, C14's 2 and +C15's 6 are rows under `json.MaxDepth=3` that the `stj-maxdepth` message read had ended; both hosts now refuse their +documents alike by type, and their checks are judged by the wire they compared (this record). Every one of the 166 rows lost passed before on no comparison, each for its stated reason: + +| Probe | Rows | Reason, and where it was decided | +|---|---|---| +| C11ParseShapesBuildProbe | 4 | #174, #180, #183, #189: span parsing over `Uri`, an enum, a flags enum and `Instant`, declared inapplicable by `no-span-parsing-without-the-underlying-types` after #52 narrowed the Generators README (#54) | +| C12DescribedConverterProbe | 5 | this record: a document both hosts refuse alike, in checks that compared nothing else | +| C16OptionsAgreementProbe | 3 | the same | +| L01JsonBytesProbe | 5 | #57: four rows under an application's converter that cannot write the value, where the old branch passed with nothing read back, now inconclusive; #363 declared inapplicable (a closed-set key under a key policy, "Your own converter wins") | +| L02NoQuoteProbe | 128 | #57: 99 rows whose only check was `json-never-repeats` under an application's converter, now declared inapplicable (`not-promised:the-applications-own-converter`), and 29 rows that refuse nothing to quote (a string or `Uri` reads every input, or the options leave the member out), now inconclusive | +| L03OneRuleProbe | 20 | #58: `json-half:wrapped-alone-refuses` took any two refusals for the framework's; 14 refuse alike and end `framework-refuses`, 6 refuse differently and end inconclusive | +| L06OffTheWireProbe | 1 | #58: #240's pass compared two refusals' messages; it is `framework-refuses` by type and path | + +**The document rule (this record).** The http family replayed from empty states without footprints at 26e51e6, before +the change, and with it; 13 passed both times: + +| Probe | rows before | checks before | rows after | checks after | rows passing | regained | lost | +|---|---|---|---|---|---|---|---| +| BaseProbe | expected-refusal 123, inapplicable 109, inconclusive 43, pass 136 | expected-refusal 180, inapplicable 388, inconclusive 63, pass 986 | expected-refusal 123, inapplicable 109, inconclusive 43, pass 136 | expected-refusal 180, inapplicable 388, inconclusive 63, pass 986 | 332 → 332 | 0 | 0 | +| C03ComposedProbe | pass 648 | pass 2919 | expected-refusal 397, pass 251 | expected-refusal 794, pass 2125 | 648 → 648 | 0 | 0 | +| C04WithoutFaultsProbe | expected-refusal 4, pass 293 | expected-refusal 5, pass 623 | expected-refusal 135, pass 162 | expected-refusal 135, pass 493 | 297 → 297 | 0 | 0 | +| C05RuleProbe | pass 645 | pass 1666 | pass 645 | pass 1666 | 645 → 645 | 0 | 0 | +| C06FaultKindsProbe | expected-refusal 6, pass 300 | expected-refusal 6, pass 1218 | pass 306 | pass 1224 | 306 → 306 | 0 | 0 | +| C07BlanketMappingProbe | expected-refusal 1, pass 410 | expected-refusal 1, pass 474 | expected-refusal 44, pass 367 | expected-refusal 44, pass 431 | 410 → 411 | 1 (expected-refusal 1) | 0 | +| C08SameNameWarningProbe | pass 302 | pass 540 | pass 302 | pass 540 | 302 → 302 | 0 | 0 | +| C12DescribedConverterProbe | pass 301 | pass 701 | expected-refusal 156, pass 145 | expected-refusal 338, pass 363 | 301 → 145 | 0 | 156 (expected-refusal 156) | +| C13PatchRequiredProbe | inconclusive 51, pass 245 | inconclusive 51, pass 541 | expected-refusal 82, inconclusive 51, pass 163 | expected-refusal 105, inconclusive 51, pass 436 | 296 → 273 | 0 | 23 (inconclusive 23) | +| C14DeclaredProblemProbe | expected-refusal 4, pass 417 | expected-refusal 15, pass 1382 | expected-refusal 273, pass 148 | expected-refusal 633, pass 764 | 417 → 420 | 3 (expected-refusal 3) | 0 | +| C15MvcFaultStatusDocProbe | expected-refusal 9, pass 396 | expected-refusal 11, pass 540 | expected-refusal 104, pass 301 | expected-refusal 104, pass 447 | 396 → 404 | 8 (expected-refusal 8) | 0 | +| C16OptionsAgreementProbe | pass 296 | pass 414 | expected-refusal 93, pass 203 | expected-refusal 127, pass 287 | 296 → 203 | 0 | 93 (expected-refusal 93) | +| C21OpenapiSchemaParityProbe | inapplicable 4, inconclusive 6, pass 284 | inapplicable 7, inconclusive 12, pass 478 | expected-refusal 129, inapplicable 4, inconclusive 6, pass 155 | expected-refusal 230, inapplicable 7, inconclusive 12, pass 248 | 284 → 155 | 0 | 129 (expected-refusal 129) | +| **all** | 5033 rows | 13221 checks (12482 pass) | 5033 rows | 13221 checks (10010 pass) | 4930 → 4541 | 12 | 401 | + +In each of the 401 rows lost every check that had passed noted a document both hosts refused and compared nothing else +(checked over the evidence: 0 exceptions); the 12 regained are the `json.MaxDepth=3` rows above. Seeded: the +value-object schema transformer throwing an `InvalidOperationException` fails, in 8 sampled rows of C21, the 4 whose +document ASP.NET alone writes ("ASP.NET alone writes it") both before and after; of the 4 whose document ASP.NET alone +refuses, before the change 3 passed (the refusal came first, with the same words) and #156 failed on a message that +differed, and after it all 4 end `framework-refuses`. #156 is what comparing no message gives up. + +**`rows` on the whole corpus**, `rows --claim C --full` for each of the 44 claims with the tool before (9e23403, the same +`Covering` as ac4c561) and now: 7,746 rows of which 740 held an excluded pair, against 7,842 rows of which none holds an +excluded pair or one no obligation wants, with no pair left uncovered by either. A first version that gave up a pair no +full row could hold left 3,288 pairs uncovered (C09, C11 and C18 got no row at all) and was replaced by leaving the axis +out of the row. + +**Pairs per claim.** `prober status --json`; "wanted" is the claim's pairs less those no obligation wants: + +| Claim | 265309f: current / pairs | now: current / wanted | declared inapplicable | excluded or declared since | +|---|---|---|---|---| +| C01-decimal-spelling | 4344 / 4344 | 4344 / 4344 | - | | +| C02-optional-text | 119 / 126 | 119 / 126 | - | | +| C03-composed-faults | 15583 / 15607 | 15605 / 15607 | - | | +| C04-without-faults | 15409 / 15426 | 15427 / 15427 | - | +1 pairs in the universe | +| C05-with-rule | 15750 / 15783 | 15781 / 15783 | - | | +| C06-fault-kinds | 15231 / 15248 | 15249 / 15249 | - | +1 pairs in the universe | +| C07-fault-responses-blanket | 15945 / 15956 | 15957 / 15957 | - | +1 pairs in the universe | +| C08-same-name-warning | 15609 / 15609 | 15609 / 15609 | - | | +| C09-base-record-equality | 487 / 487 | 487 / 487 | - | | +| C10-preserve-guard | 6797 / 6797 | 6797 / 6797 | - | | +| C11-parse-shapes-build | 1048 / 1048 | 1044 / 1044 | - | -4 pairs in the universe | +| C12-described-converter | 14904 / 14904 | 14083 / 14904 | - | | +| C13-patch-required | 14731 / 14731 | 14683 / 14731 | - | | +| C14-declared-problem | 14886 / 14903 | 14904 / 14904 | - | +1 pairs in the universe | +| C15-mvc-fault-status-doc | 15238 / 15255 | 15256 / 15256 | - | +1 pairs in the universe | +| C16-options-agreement | 14904 / 14904 | 14449 / 14904 | - | | +| C17-cmtk0014 | 1652 / 1652 | 1791 / 1832 | - | +180 pairs in the universe | +| C18-cmtk1023 | 1067 / 1067 | 1067 / 1067 | - | | +| C19-cmtk0010-kinds | 2091 / 2091 | 2242 / 2292 | - | +201 pairs in the universe | +| C20-testing-shouldhold | 507 / 507 | 507 / 507 | - | | +| C21-openapi-schema-parity | (no claim) | 13185 / 13883 | 336 | | +| C22-generator-refusals | (no claim) | 0 / 621 | - | | +| E01-column-parity | 372 / 414 | 414 / 414 | - | | +| E02-value-translates-to-column | 400 / 412 | 400 / 406 | 6 | | +| E03-column-round-trip | 290 / 290 | 290 / 290 | - | | +| E04-integer-key-generated | 61 / 61 | 61 / 61 | - | | +| E05-own-compareto-rejected | 90 / 90 | 90 / 90 | - | | +| E06-own-converter-no-value | 26 / 26 | 26 / 26 | - | | +| E07-closed-set-stored-as-name | 9 / 9 | 9 / 9 | - | | +| E08-option-result-not-columns | 47 / 47 | 47 / 47 | - | | +| E09-or-none-in-database | 69 / 69 | 69 / 69 | - | | +| E10-read-back-skips-validation | 52 / 52 | 52 / 52 | - | | +| E11-usetypekit-order | 14 / 14 | 14 / 14 | - | | +| E12-convention-by-vo-type | 26 / 26 | 26 / 26 | - | | +| E13-primitive-collections | 10 / 10 | 10 / 10 | - | | +| E14-inheritance-key-generation | 9 / 9 | 9 / 9 | - | | +| E15-composite-key | 8 / 8 | 8 / 8 | - | | +| E16-own-configuration-wins | 25 / 25 | 25 / 25 | - | | +| L01-json-bytes | 6367 / 6478 | 6473 / 6478 | - | | +| L02-no-quote | 8280 / 8314 | 8240 / 8314 | - | | +| L03-one-rule | 7055 / 7084 | 7079 / 7084 | - | | +| L04-round-trip | 8025 / 8314 | 8312 / 8314 | - | | +| L05-equality | 1076 / 1076 | 1076 / 1076 | - | | +| L06-off-the-wire | 8022 / 8022 | 8021 / 8022 | - | | +| **all** | 226635 / 227295 | 239337 / 242175 | 342 | | + +**C07's pass share by check.** Before, each row was one check `claim`: BaseProbe 135 of 411 (32.8%), +C07BlanketMappingProbe 410 of 411; 545 of 822 (66.3%). Now: BaseProbe's checks 986 of 1,617 (61.0%; 332 of its 411 +rows pass); C07BlanketMappingProbe by obligation `by-kind` 57 of 76, `enum-needs-own-mapping` 66 of 66, +`mapping-refusal-stops-start` 138 of 138, `member-without-kind-stops-start` 68 of 68, `own-mapping` 38 of 63, +`own-set-keeps-mapping` 64 of 64; 1,417 of 2,092 checks in all (67.7%). + +**`NoteKeys`.** 63 keys at the start of WS2; 68 at ac4c561; +27 now: 26 pass keys and `framework-refuses`. The corpus makes 119 `Note` calls (`ProbeNoteKeyTests`' floor stays 100). + +**Product defects the conversions and the analyst found**, each fixed in its own pull request: + +| Found by | Defect | Fixed in | +|---|---|---| +| C21's first run (#40) | F3: a nullable value object over an enum written by name left out the `null` ASP.NET lists among a parameter's values | #45 (8ffad55, `Found-By: PR`), with a sibling found by the catalogue (25118b8, `VA`) | +| C11's conversion (#54) | a value object over `string` did not implement `ISpanParsable`, as `string` does | #52 (9247406, `GTS`, `PR`) | +| the code-analyst run of 2026-10-16 | A1: a non-abstract polymorphic base's own branch undescribed; A2: a query or header array of a described type lost its `items`; A3: a value object over a described type was not the parameter's schema outside the body | #51 (4917a03, ccfcb5d, 66b4d62, `CA`), with two siblings (5a8f048, 0293dcd, `VA`) | +| the code-analyst | `.Value` through a type parameter; Greatest, Least and row values over an own `CompareTo`; a Guid or string key generated on add | #44 (0764462), #43 (7c6c48f), #46 (d9ccfe0), `EFT`, `CA` | + +**Seeds.** `prober seeds --probe P` for every registered probe, from a copy of the final replay's state, in two lanes of +separate worktrees (10,941 s and 11,027 s wall; 21,855 s of seed runs). **170 seeds, 170 caught, each by a check of its own obligation**; BaseProbe's one seed names none, and its `claim` check catches it. Each line names the row and the obligation of the check that caught it: + +| Probe | Seed's obligation | Outcome | Rows violating; e.g. the row and the check that caught it | +|---|---|---|---| +| `BaseProbe` | none (`claim`) | caught | 55 of 411 rows; #10 `claim` | +| `C01DecimalSpellingProbe` | `smallest-scale` | caught | 57 of 117 rows; #0 `smallest-scale` | +| `C01DecimalSpellingProbe` | `number-is-a-number` | caught | 43 of 117 rows; #0 `number-is-a-number` | +| `C01DecimalSpellingProbe` | `string-is-a-string` | caught | 54 of 117 rows; #1 `string-is-a-string` | +| `C01DecimalSpellingProbe` | `reads-a-number-or-a-string` | caught | 101 of 117 rows; #0 `reads-a-number-or-a-string` | +| `C01DecimalSpellingProbe` | `keeps-the-scale-it-reads` | caught | 64 of 117 rows; #0 `keeps-the-scale-it-reads` | +| `C01DecimalSpellingProbe` | `covers-every-decimal` | caught | 4 of 117 rows; #32 `covers-every-decimal` | +| `C02OptionalTextProbe` | `hidden-is-an-error-in-both` | caught | 7 of 46 rows; #3 `hidden-is-an-error-in-both` | +| `C02OptionalTextProbe` | `required-blank-is-an-error` | caught | 11 of 46 rows; #2 `required-blank-is-an-error` | +| `C02OptionalTextProbe` | `optional-blank-is-no-value` | caught | 14 of 46 rows; #0 `optional-blank-is-no-value` | +| `C02OptionalTextProbe` | `null-is-blank` | caught | 6 of 46 rows; #6 `null-is-blank` | +| `C03ComposedProbe` | `each-error-own-set` | caught | 487 of 648 rows; #0 `each-error-own-set` | +| `C03ComposedProbe` | `detail-and-rule-intact` | caught | 163 of 648 rows; #2 `detail-and-rule-intact` | +| `C03ComposedProbe` | `statuses-ascending` | caught | 182 of 648 rows; #1 `statuses-ascending` | +| `C03ComposedProbe` | `shared-status-types` | caught | 182 of 648 rows; #1 `shared-status-types` | +| `C03ComposedProbe` | `unmapped-set-stops-start` | caught | 648 of 648 rows; #0 `unmapped-set-stops-start` | +| `C04WithoutFaultsProbe` | `doc-omits-left-out` | caught | 78 of 297 rows; #1 `doc-omits-left-out` | +| `C04WithoutFaultsProbe` | `reporting-left-out-throws` | caught | 238 of 297 rows; #0 `reporting-left-out-throws` | +| `C04WithoutFaultsProbe` | `narrows-wider-status` | caught | 93 of 297 rows; #0 `narrows-wider-status` | +| `C04WithoutFaultsProbe` | `left-out-everywhere-stops-start` | caught | 59 of 297 rows; #11 `left-out-everywhere-stops-start` | +| `C05RuleProbe` | `rule-qualified` | caught | 129 of 645 rows; #4 `rule-qualified` | +| `C05RuleProbe` | `nested-chain` | caught | 125 of 645 rows; #3 `nested-chain` | +| `C05RuleProbe` | `stays-the-errors` | caught | 376 of 645 rows; #2 `stays-the-errors` | +| `C05RuleProbe` | `unnamed-value-throws` | caught | 269 of 645 rows; #0 `unnamed-value-throws` | +| `C05RuleProbe` | `no-namespace` | caught | 247 of 645 rows; #2 `no-namespace` | +| `C06FaultKindsProbe` | `kind-gives-status` | caught | 131 of 306 rows; #3 `kind-gives-status` | +| `C06FaultKindsProbe` | `description-as-detail` | caught | 260 of 306 rows; #1 `description-as-detail` | +| `C06FaultKindsProbe` | `no-description-no-detail` | caught | 306 of 306 rows; #0 `no-description-no-detail` | +| `C06FaultKindsProbe` | `member-must-have-kind` | caught | 306 of 306 rows; #0 `member-must-have-kind` | +| `C07BlanketMappingProbe` | `by-kind` | caught | 40 of 411 rows; #22 `by-kind` | +| `C07BlanketMappingProbe` | `own-mapping` | caught | 27 of 411 rows; #2 `own-mapping` | +| `C07BlanketMappingProbe` | `own-set-keeps-mapping` | caught | 64 of 411 rows; #1 `own-set-keeps-mapping` | +| `C07BlanketMappingProbe` | `enum-needs-own-mapping` | caught | 66 of 411 rows; #0 `enum-needs-own-mapping` | +| `C07BlanketMappingProbe` | `member-without-kind-stops-start` | caught | 33 of 411 rows; #7 `member-without-kind-stops-start` | +| `C07BlanketMappingProbe` | `mapping-refusal-stops-start` | caught | 70 of 411 rows; #4 `mapping-refusal-stops-start` | +| `C08SameNameWarningProbe` | `bare-names-warn` | caught | 90 of 302 rows; #0 `bare-names-warn` | +| `C08SameNameWarningProbe` | `qualified-chain-warns` | caught | 74 of 302 rows; #8 `qualified-chain-warns` | +| `C08SameNameWarningProbe` | `warning-names-both` | caught | 164 of 302 rows; #0 `warning-names-both` | +| `C08SameNameWarningProbe` | `qualified-names-fix` | caught | 74 of 302 rows; #8 `qualified-names-fix` | +| `C08SameNameWarningProbe` | `leave-one-out-fixes` | caught | 138 of 302 rows; #1 `leave-one-out-fixes` | +| `C09BaseRecordEqualityProbe` | `in-a-collection-of-the-base` | caught | 12 of 101 rows; #12 `in-a-collection-of-the-base` | +| `C09BaseRecordEqualityProbe` | `equal-by-the-wrapped-value` | caught | 21 of 101 rows; #1 `equal-by-the-wrapped-value` | +| `C10PreserveGuardProbe` | `refused-where-references-could-be-written` | caught | 4 of 306 rows; #221 `refused-where-references-could-be-written` | +| `C10PreserveGuardProbe` | `value-converters-unaffected` | caught | 4 of 306 rows; #59 `value-converters-unaffected` | +| `C10PreserveGuardProbe` | `own-converters-unaffected` | caught | 15 of 306 rows; #1 `own-converters-unaffected` | +| `C10PreserveGuardProbe` | `ignore-cycles-unaffected` | caught | 4 of 306 rows; #72 `ignore-cycles-unaffected` | +| `C11ParseShapesBuildProbe` | `implicit-result` | caught | 28 of 247 rows; #2 `implicit-result` | +| `C11ParseShapesBuildProbe` | `ref-text` | caught | 22 of 247 rows; #4 `ref-text` | +| `C11ParseShapesBuildProbe` | `abstract-string-constructor` | caught | 22 of 247 rows; #3 `abstract-string-constructor` | +| `C11ParseShapesBuildProbe` | `parsing-where-it-can-be-parsed` | caught | 16 of 247 rows; #37 `parsing-where-it-can-be-parsed` | +| `C12DescribedConverterProbe` | `own-converter-describes-itself` | caught | 115 of 301 rows; #1 `own-converter-describes-itself` | +| `C12DescribedConverterProbe` | `options-describe` | caught | 30 of 301 rows; #26 `options-describe` | +| `C12DescribedConverterProbe` | `every-position` | caught | 145 of 301 rows; #1 `every-position` | +| `C12DescribedConverterProbe` | `parameter-keeps-aspnet` | caught | 73 of 301 rows; #2 `parameter-keeps-aspnet` | +| `C13PatchRequiredProbe` | `doc-required-where-rejected` | caught | 35 of 296 rows; #4 `doc-required-where-rejected` | +| `C13PatchRequiredProbe` | `doc-optional-where-absent` | caught | 120 of 296 rows; #0 `doc-optional-where-absent` | +| `C13PatchRequiredProbe` | `absent-is-default` | caught | 200 of 296 rows; #0 `absent-is-default` | +| `C14DeclaredProblemProbe` | `declared-describes-fault` | caught | 30 of 421 rows; #15 `declared-describes-fault` | +| `C14DeclaredProblemProbe` | `declared-names-types` | caught | 77 of 421 rows; #1 `declared-names-types` | +| `C14DeclaredProblemProbe` | `ascending-with-declared` | caught | 105 of 421 rows; #1 `ascending-with-declared` | +| `C14DeclaredProblemProbe` | `application-json-declared` | caught | 38 of 421 rows; #1 `application-json-declared` | +| `C15MvcFaultStatusDocProbe` | `builder-reaches-doc` | caught | 24 of 405 rows; #30 `builder-reaches-doc` | +| `C15MvcFaultStatusDocProbe` | `without-faults-leaves-doc` | caught | 45 of 405 rows; #20 `without-faults-leaves-doc` | +| `C15MvcFaultStatusDocProbe` | `status-per-scope` | caught | 76 of 405 rows; #0 `status-per-scope` | +| `C16OptionsAgreementProbe` | `differing-refused` | caught | 42 of 296 rows; #6 `differing-refused` | +| `C16OptionsAgreementProbe` | `error-names-type-and-descriptions` | caught | 42 of 296 rows; #6 `error-names-type-and-descriptions` | +| `C16OptionsAgreementProbe` | `patch-held-refused` | caught | 42 of 296 rows; #6 `patch-held-refused` | +| `C16OptionsAgreementProbe` | `same-converters-build` | caught | 86 of 296 rows; #1 `same-converters-build` | +| `C16OptionsAgreementProbe` | `minimal-host-not-compared` | caught | 25 of 296 rows; #0 `minimal-host-not-compared` | +| `C17Cmtk0014Probe` | `a-cast-naming-no-member` | caught | 75 of 283 rows; #4 `a-cast-naming-no-member` | +| `C17Cmtk0014Probe` | `default-and-zero` | caught | 122 of 283 rows; #0 `default-and-zero` | +| `C17Cmtk0014Probe` | `constructor-accessibility-too` | caught | 32 of 283 rows; #4 `constructor-accessibility-too` | +| `C17Cmtk0014Probe` | `reported-where-written` | caught | 139 of 283 rows; #0 `reported-where-written` | +| `C17Cmtk0014Probe` | `a-member-is-silent` | caught | 43 of 283 rows; #3 `a-member-is-silent` | +| `C18Cmtk1023Probe` | `names-each-base-member` | caught | 130 of 280 rows; #0 `names-each-base-member` | +| `C18Cmtk1023Probe` | `a-warning-and-generated` | caught | 280 of 280 rows; #0 `a-warning-and-generated` | +| `C18Cmtk1023Probe` | `generated-equality-never-calls-the-base` | caught | 83 of 280 rows; #4 `generated-equality-never-calls-the-base` | +| `C19Cmtk0010KindsProbe` | `methods-the-compiler-calls` | caught | 59 of 282 rows; #6 `methods-the-compiler-calls` | +| `C19Cmtk0010KindsProbe` | `interface-and-unsealed` | caught | 32 of 282 rows; #1 `interface-and-unsealed` | +| `C19Cmtk0010KindsProbe` | `value-object-and-closed-set` | caught | 28 of 282 rows; #0 `value-object-and-closed-set` | +| `C19Cmtk0010KindsProbe` | `flags-enum-and-read-only-struct` | caught | 33 of 282 rows; #2 `flags-enum-and-read-only-struct` | +| `C19Cmtk0010KindsProbe` | `a-pair-once-at-the-later-argument` | caught | 80 of 282 rows; #6 `a-pair-once-at-the-later-argument` | +| `C20ShouldHoldProbe` | `fails-unless-it-holds-that-reason` | caught | 111 of 261 rows; #0 `fails-unless-it-holds-that-reason` | +| `C20ShouldHoldProbe` | `whichever-of-its-types` | caught | 120 of 261 rows; #1 `whichever-of-its-types` | +| `C20ShouldHoldProbe` | `hands-back-detail-and-rule` | caught | 111 of 261 rows; #0 `hands-back-detail-and-rule` | +| `C20ShouldHoldProbe` | `uninitialized-fails` | caught | 28 of 261 rows; #6 `uninitialized-fails` | +| `C20ShouldHoldProbe` | `a-failure-says-the-outcome` | caught | 122 of 261 rows; #2 `a-failure-says-the-outcome` | +| `C21OpenapiSchemaParityProbe` | `the-underlying-types-schema` | caught | 10 of 294 rows; #15 `the-underlying-types-schema` | +| `C21OpenapiSchemaParityProbe` | `container-elements` | caught | 16 of 294 rows; #0 `container-elements` | +| `C21OpenapiSchemaParityProbe` | `dictionary-key` | caught | 1 of 294 rows; #137 `dictionary-key` | +| `C21OpenapiSchemaParityProbe` | `every-document-version` | caught | 52 of 294 rows; #5 `every-document-version` | +| `C21OpenapiSchemaParityProbe` | `nullable-admits-null-as-the-wrapped-does` | caught | 7 of 294 rows; #63 `nullable-admits-null-as-the-wrapped-does` | +| `C22GeneratorRefusalsProbe` | `every-error-links-the-table` | caught | 79 of 144 rows; #1 `every-error-links-the-table` | +| `C22GeneratorRefusalsProbe` | `unsealed-record-class` | caught | 21 of 144 rows; #5 `unsealed-record-class` | +| `C22GeneratorRefusalsProbe` | `over-value-object` | caught | 20 of 144 rows; #1 `over-value-object` | +| `C22GeneratorRefusalsProbe` | `generic-wrapped` | caught | 24 of 144 rows; #3 `generic-wrapped` | +| `C22GeneratorRefusalsProbe` | `abstract` | caught | 20 of 144 rows; #4 `abstract` | +| `C22GeneratorRefusalsProbe` | `over-system-enum` | caught | 7 of 144 rows; #8 `over-system-enum` | +| `C22GeneratorRefusalsProbe` | `named-as-generated-member` | caught | 20 of 144 rows; #2 `named-as-generated-member` | +| `C22GeneratorRefusalsProbe` | `closed-set-named-as-generated-member` | caught | 4 of 144 rows; #11 `closed-set-named-as-generated-member` | +| `C22GeneratorRefusalsProbe` | `equality-operator` | caught | 20 of 144 rows; #0 `equality-operator` | +| `C22GeneratorRefusalsProbe` | `no-type-half-generated` | caught | 79 of 144 rows; #1 `no-type-half-generated` | +| `C22GeneratorRefusalsProbe` | `over-object` | caught | 6 of 144 rows; #15 `over-object` | +| `E01ColumnParityProbe` | `column-type` | caught | 12 of 175 rows; #0 `column-type` | +| `E01ColumnParityProbe` | `nullable` | caught | 36 of 175 rows; #1 `nullable` | +| `E01ColumnParityProbe` | `facet-kept` | caught | 108 of 175 rows; #1 `facet-kept` | +| `E02ValueTranslatesProbe` | `value` | caught | 58 of 215 rows; #0 `value` | +| `E02ValueTranslatesProbe` | `get-value` | caught | 58 of 215 rows; #1 `get-value` | +| `E02ValueTranslatesProbe` | `value-or-null` | caught | 51 of 215 rows; #2 `value-or-null` | +| `E02ValueTranslatesProbe` | `column-not-cast` | caught | 203 of 215 rows; #0 `column-not-cast` | +| `E03ColumnRoundTripProbe` | `stored-as-underlying` | caught | 5 of 177 rows; #13 `stored-as-underlying` | +| `E03ColumnRoundTripProbe` | `keys` | caught | 35 of 177 rows; #0 `keys` | +| `E03ColumnRoundTripProbe` | `nullable-properties` | caught | 35 of 177 rows; #1 `nullable-properties` | +| `E04IntegerKeyProbe` | `generated-on-add` | caught | 5 of 28 rows; #8 `generated-on-add` | +| `E04IntegerKeyProbe` | `byte-not-generated` | caught | 1 of 28 rows; #9 `byte-not-generated` | +| `E04IntegerKeyProbe` | `char-key-yours` | caught | 1 of 28 rows; #16 `char-key-yours` | +| `E04IntegerKeyProbe` | `guid-key-yours` | caught | 1 of 28 rows; #12 `guid-key-yours` | +| `E04IntegerKeyProbe` | `not-on-efs-list` | caught | 1 of 28 rows; #7 `not-on-efs-list` | +| `E04IntegerKeyProbe` | `own-configuration-wins` | caught | 2 of 28 rows; #18 `own-configuration-wins` | +| `E05OwnCompareToProbe` | `comparison` | caught | 30 of 90 rows; #0 `comparison` | +| `E05OwnCompareToProbe` | `ordering` | caught | 24 of 90 rows; #7 `ordering` | +| `E05OwnCompareToProbe` | `min-max` | caught | 12 of 90 rows; #4 `min-max` | +| `E05OwnCompareToProbe` | `equality-allowed` | caught | 6 of 90 rows; #1 `equality-allowed` | +| `E05OwnCompareToProbe` | `value-order-allowed` | caught | 6 of 90 rows; #11 `value-order-allowed` | +| `E05OwnCompareToProbe` | `query-filter` | caught | 12 of 90 rows; #2 `query-filter` | +| `E06OwnConverterNoValueProbe` | `value-not-translated` | caught | 5 of 12 rows; #1 `value-not-translated` | +| `E06OwnConverterNoValueProbe` | `value-object-comparison` | caught | 7 of 12 rows; #0 `value-object-comparison` | +| `E07ClosedSetNameProbe` | `stored-as-name` | caught | 3 of 9 rows; #2 `stored-as-name` | +| `E07ClosedSetNameProbe` | `in-collections-and-json` | caught | 2 of 9 rows; #2 `in-collections-and-json` | +| `E07ClosedSetNameProbe` | `unknown-name-throws` | caught | 3 of 9 rows; #0 `unknown-name-throws` | +| `E07ClosedSetNameProbe` | `non-member-throws` | caught | 3 of 9 rows; #3 `non-member-throws` | +| `E08OptionResultNotColumnsProbe` | `model-fails` | caught | 10 of 20 rows; #4 `model-fails` | +| `E08OptionResultNotColumnsProbe` | `error-says-so` | caught | 20 of 20 rows; #0 `error-says-so` | +| `E09OrNoneInDbProbe` | `first-reads-one-row` | caught | 3 of 18 rows; #2 `first-reads-one-row` | +| `E09OrNoneInDbProbe` | `single-asks-for-two` | caught | 1 of 18 rows; #9 `single-asks-for-two` | +| `E09OrNoneInDbProbe` | `element-at-skips` | caught | 3 of 18 rows; #4 `element-at-skips` | +| `E10ReadBackSkipsValidationProbe` | `reads-without-create` | caught | 30 of 30 rows; #0 `reads-without-create` | +| `E10ReadBackSkipsValidationProbe` | `revalidate-finds` | caught | 30 of 30 rows; #0 `revalidate-finds` | +| `E11UseTypeKitOrderProbe` | `order-does-not-matter` | caught | 14 of 14 rows; #0 `order-does-not-matter` | +| `E12ConventionByTypeProbe` | `value-object-type-reaches` | caught | 4 of 12 rows; #1 `value-object-type-reaches` | +| `E12ConventionByTypeProbe` | `underlying-type-does-not-reach` | caught | 7 of 12 rows; #0 `underlying-type-does-not-reach` | +| `E13PrimitiveCollectionProbe` | `element-value` | caught | 5 of 10 rows; #0 `element-value` | +| `E13PrimitiveCollectionProbe` | `contains` | caught | 5 of 10 rows; #1 `contains` | +| `E14InheritanceKeyProbe` | `tph-tpt-root` | caught | 4 of 9 rows; #0 `tph-tpt-root` | +| `E14InheritanceKeyProbe` | `tpc-assign-on-sqlite` | caught | 2 of 9 rows; #4 `tpc-assign-on-sqlite` | +| `E15CompositeKeyProbe` | `as-for-primitives` | caught | 6 of 8 rows; #0 `as-for-primitives` | +| `E16OwnConfigurationWinsProbe` | `never` | caught | 2 of 25 rows; #17 `never` | +| `E16OwnConfigurationWinsProbe` | `on-add` | caught | 2 of 25 rows; #9 `on-add` | +| `E16OwnConfigurationWinsProbe` | `column-type` | caught | 5 of 25 rows; #0 `column-type` | +| `E16OwnConfigurationWinsProbe` | `conversion` | caught | 5 of 25 rows; #1 `conversion` | +| `E16OwnConfigurationWinsProbe` | `optional` | caught | 5 of 25 rows; #2 `optional` | +| `L01JsonBytesProbe` | `as-a-dictionary-key-too` | caught | 11 of 581 rows; #56 `as-a-dictionary-key-too` | +| `L01JsonBytesProbe` | `the-underlying-types-json` | caught | 12 of 581 rows; #11 `the-underlying-types-json` | +| `L01JsonBytesProbe` | `read-as-the-underlying-type-reads` | caught | 26 of 581 rows; #20 `read-as-the-underlying-type-reads` | +| `L02NoQuoteProbe` | `parsing-never-repeats` | caught | 183 of 586 rows; #0 `parsing-never-repeats` | +| `L02NoQuoteProbe` | `json-never-repeats` | caught | 199 of 586 rows; #0 `json-never-repeats` | +| `L02NoQuoteProbe` | `no-inner-that-quotes` | caught | 22 of 586 rows; #5 `no-inner-that-quotes` | +| `L03OneRuleProbe` | `parsing-runs-create` | caught | 109 of 586 rows; #3 `parsing-runs-create` | +| `L03OneRuleProbe` | `tryfrom-runs-create` | caught | 30 of 586 rows; #7 `tryfrom-runs-create` | +| `L03OneRuleProbe` | `json-runs-create` | caught | 157 of 586 rows; #0 `json-runs-create` | +| `L03OneRuleProbe` | `refusals-name-the-fault` | caught | 196 of 586 rows; #0 `refusals-name-the-fault` | +| `L04RoundTripProbe` | `invariant-whatever-the-culture` | caught | 21 of 290 rows; #15 `invariant-whatever-the-culture` | +| `L04RoundTripProbe` | `the-type-converter-writes-the-same` | caught | 5 of 290 rows; #14 `the-type-converter-writes-the-same` | +| `L04RoundTripProbe` | `text-reads-back` | caught | 1 of 290 rows; #64 `text-reads-back` | +| `L04RoundTripProbe` | `json-refusal-is-a-json-exception` | caught | 45 of 290 rows; #10 `json-refusal-is-a-json-exception` | +| `L05EqualityProbe` | `through-every-base` | caught | 41 of 309 rows; #0 `through-every-base` | +| `L05EqualityProbe` | `equality-is-the-underlying-values` | caught | 22 of 309 rows; #2 `equality-is-the-underlying-values` | +| `L05EqualityProbe` | `ordering-follows-the-underlying-type` | caught | 238 of 309 rows; #0 `ordering-follows-the-underlying-type` | +| `L06OffTheWireProbe` | `never-written` | caught | 268 of 325 rows; #0 `never-written` | +| `L06OffTheWireProbe` | `never-read` | caught | 267 of 325 rows; #0 `never-read` | + +**Tests**, through `build-verdict test --project P` at 5c5affe: Prober.Tests 268 passed (11 new: `FixTests`, +`CoveringTests`, one `GapsTests` case); Conventions.Tests 1,491 passed; Facts 87 passed, 9 skipped (the PostgreSQL facts, +without `TYPEKIT_PG`). `docs-query check`: 0 problems. Each new test fails against a seeded defect, one change at a +time: `fix` writing the commit as given, accepting any status, taking an unknown id, dropping the claim or the probe, +accepting any commit; `Covering` picking a blocked value, the seeded fill taking the first value or ignoring +exclusions; `gaps` counting a check that did not pass, an obligation's reads for a pair it does not want, a claim +without evidence, or no section. `ToolchainTests.A_dotnet_without_the_pinned_sdk_falls_back_to_the_installers` failed +with `PROBE_STATE_DIR` set and passes with it set now. A removed note key called again fails the probes' build with +CS0117. + +**The gate**, `prober gate --since v2.1.0` on the final replay's state: + +``` +probe gate: refused for build/ws2-close-out since v2.1.0 (44 claim(s) touched, 2838 pair(s) uncovered): + C22GeneratorRefusalsProbe: not replayed at its current identity and footprint (run `prober replay --all`) + C02-optional-text: 7 of 126 pairs uncovered (touched since v2.1.0) + C03-composed-faults: 2 of 15607 pairs uncovered (touched since v2.1.0) + C05-with-rule: 2 of 15783 pairs uncovered (touched since v2.1.0) + C12-described-converter: 821 of 14904 pairs uncovered (touched since v2.1.0) + C13-patch-required: 48 of 14731 pairs uncovered (touched since v2.1.0) + C16-options-agreement: 455 of 14904 pairs uncovered (touched since v2.1.0) + C17-cmtk0014: 41 of 1832 pairs uncovered (touched since v2.1.0) + C19-cmtk0010-kinds: 50 of 2292 pairs uncovered (touched since v2.1.0) + C21-openapi-schema-parity: 698 of 13883 pairs uncovered (touched since v2.1.0) + C22-generator-refusals: 621 of 621 pairs uncovered (touched since v2.1.0) + E02-value-translates-to-column: 6 of 406 pairs uncovered (touched since v2.1.0) + L01-json-bytes: 5 of 6478 pairs uncovered (touched since v2.1.0) + L02-no-quote: 74 of 8314 pairs uncovered (touched since v2.1.0) + L03-one-rule: 5 of 7084 pairs uncovered (touched since v2.1.0) + L04-round-trip: 2 of 8314 pairs uncovered (touched since v2.1.0) + L06-off-the-wire: 1 of 8022 pairs uncovered (touched since v2.1.0) +``` + +| Refusal | Since | What it needs | +|---|---|---| +| C22GeneratorRefusalsProbe not current at its footprint | #47 | The probe builds its consumer in a child process, so the in-process coverage shows no file under `src/` and `footprint` refuses an empty one (#54): `watched` paths for the generators, as C18 has, so its footprint records them. Its 621 pairs follow. | +| C12 821, C13 48, C16 455, C21 677 | this record | Checks that compared nothing but a document both hosts refuse no longer pass. A declaration of the document obligations' inapplicability under `json.ReferenceHandler=Preserve` and `json.MaxDepth=3`, citing record 0099 or a fact, and rows (`rows --probe`) for the pairs that remain. C21's other 21 were open at ac4c561. | +| C17 41, C19 50 | #49, #54 | The stored rows substitute excluded refused kinds and credit other values; rows written again with `rows --probe`, which now leaves such an axis out instead. | +| L02 74 | #57 | The own-converter pairs of its TryFrom and formatting rows, whose only check is declared inapplicable there: a declaration on the claim's other obligations or rows that reach them. | +| E02 6 | before WS2 (#56) | `where-value-compare` with a wrapped type that has no `>=` (the probe substitutes `where-equals`): an exclusion with that reason; and one `group-by` row for `nested-two-levels` × `DateTimeOffset`. | +| C02 7, C03 2, C05 2, L01 5, L03 5, L04 2, L06 1 | before or during WS2 | Scattered pairs no stored row holds: rows from `rows --claim`, or exclusions with their reasons. | + +None is a finding: 0 are open. Every refusal is answered by a probe's rows, a declaration, an exclusion, or a waiver naming the maintainer as reviewer (decision 97); this record answers none of them. diff --git a/design/adr/README.md b/design/adr/README.md index f1769041..27de2b5a 100644 --- a/design/adr/README.md +++ b/design/adr/README.md @@ -105,12 +105,13 @@ per area ([design](../plan.md)) link to the records they rest on. | [0096](0096-ci-replays-every-probe-and-the-release-runs-the-gate.md) | CI replays every probe, from an empty state, on every push and pull request, and a failing probe fails the run; the release replays them and runs the gate before it packs, the gate advisory; held by `ReleaseWiringTests` | 2026-10-15 | accepted; the gate binds from 2.2.0 by [0097](0097-2-2-0-is-a-consolidation-release-and-the-gate-binds.md); the build workflow's replay runs by family and footprint by [0101](0101-ci-replays-by-family-and-footprint.md) | | [0097](0097-2-2-0-is-a-consolidation-release-and-the-gate-binds.md) | 2.2.0 is a consolidation release: no public API, its additive items move to 2.3.0, and the probe gate binds from it | 2026-10-15 | accepted | | [0098](0098-a-null-passes-no-comparison-in-a-query.md) | A nullable record-class value object compared with `<`, `<=`, `>` or `>=` in a query answers as SQL does, leaving out its null rows; documented, not refused | 2026-10-15 | accepted | -| [0099](0099-a-probe-row-says-how-it-ended.md) | A probe row says how it ended: notes are a closed catalogue of keys and outcomes, a row that earned nothing is a skipped test labelled with its settings, a skip for a finding fails the replay once the finding is fixed and the gate while it stands, and a refused document is judged against ASP.NET alone | 2026-10-15 | accepted; note keys are a typed catalogue the compiler checks by [0103](0103-a-harness-error-is-never-a-catch-and-note-keys-are-constants.md) | +| [0099](0099-a-probe-row-says-how-it-ended.md) | A probe row says how it ended: notes are a closed catalogue of keys and outcomes, a row that earned nothing is a skipped test labelled with its settings, a skip for a finding fails the replay once the finding is fixed and the gate while it stands, and a refused document is judged against ASP.NET alone | 2026-10-15 | accepted; note keys are a typed catalogue the compiler checks by [0103](0103-a-harness-error-is-never-a-catch-and-note-keys-are-constants.md); a refused document is compared by type and stage, and earns nothing in a check that compared nothing else, by [0108](0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md) | | [0100](0100-every-probe-is-shown-to-fail-against-a-seeded-defect.md) | Every registered probe names a seeded product defect, and `prober seeds` (local, a build per seed) requires a violation it causes; the prober tests hold that every seed exists and applies; pairwise coverage is a floor, three-way combinations belong to the variant catalogue | 2026-10-15 | accepted; a seed per obligation of a claim by [0104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md); seeds run by footprint, a release `--all`, by [0107](0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md) | | [0101](0101-ci-replays-by-family-and-footprint.md) | The build workflow replays the probes in legs by family, each running its probes in parallel, under a time limit: a push every probe with coverage and the merged state saved, a pull request only the probes it stales against its base branch's saved state and without a footprint; `prober merge` joins the legs and refuses a probe no leg saw | 2026-10-16 | accepted | | [0102](0102-an-exclusion-carries-its-reason.md) | Each `exclude` entry of claims.json is a required one-sentence reason and its pairs; the prober refuses a claims file its schema does not match, naming the claim and the field, so an invented reason field is refused; `prober universe` prints each exclusion's reason beside the claim's narrowings; the prober writes JSON with only JSON's own escapes, so an unchanged publish leaves a tracked file as it is | 2026-10-16 | accepted | | [0103](0103-a-harness-error-is-never-a-catch-and-note-keys-are-constants.md) | A harness error is never a catch: `Rec.Run` ends a row whose check throws as a harness error, not a violation, and `prober seeds` credits only a violation no harness error raised; note keys are members of a typed `NoteKeys` the compiler checks, and `register`, `publish` and a conventions test refuse a note or a skip whose key or finding is not written out | 2026-10-16 | accepted | -| [0104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md) | A claim lists its `obligations`, each a sentence quoted verbatim from its source, and what it does not promise in `not-promised`, each quoted from the document or record that makes it intended; a claim without the field has one obligation, `claim`; a probe carries a seed per obligation it checks, and the check without a build refuses a seed naming an unlisted obligation and a listed obligation no seed names; `status` shows each claim's obligations, and `prober gaps` reports what the corpus leaves out, deciding nothing | 2026-10-16 | accepted | +| [0104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md) | A claim lists its `obligations`, each a sentence quoted verbatim from its source, and what it does not promise in `not-promised`, each quoted from the document or record that makes it intended; a claim without the field has one obligation, `claim`; a probe carries a seed per obligation it checks, and the check without a build refuses a seed naming an unlisted obligation and a listed obligation no seed names; `status` shows each claim's obligations, and `prober gaps` reports what the corpus leaves out, deciding nothing | 2026-10-16 | accepted; `gaps` reads the last replay's evidence too by [0108](0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md) | | [0105](0105-every-tool-is-built-in-the-session-s-checkout.md) | SessionStart builds the guard in `$CLAUDE_PROJECT_DIR` and every other tool, through `tools/AgentTools.slnf`, in the session's own checkout; a hook, the prober, the smoke test and the verdict fall back to the SDK installer's `dotnet` when the one on PATH lacks the pinned SDK, and a session start that builds nothing tells the agent; the guard lets `dotnet msbuild -getProperty/-getItem` without a target through | 2026-10-17 | accepted | | [0106](0106-a-parameter-of-a-value-object-over-a-described-type-is-inlined.md) | A value object over a type the JSON options describe is written inline as a route, query, header or form parameter and as an element of a query or header array, where value objects are components: the component is its JSON positions'; every other value object keeps its reference; the schema below a parameter is asked for with the parameter's source and type alone | 2026-10-17 | accepted | | [0107](0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md) | A fact that prunes or excuses a row is executable: `docs-query check` fails `fact-not-executable-cited` where a refused entry, a not-promised entry, an obligation's declaration or the harness cites a fact without its test; a fact's measured answer is `KBNNN.fixture.json` beside its test, named by `fixture:`, read by a BCL-only reader the harness will use; `prober seeds` runs only the probes whose seeds identity or product footprint changed since every seed was caught, and a release runs `--all` | 2026-10-17 | accepted | +| [0108](0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md) | A probe row runs a check per obligation, each against a baseline, and the ledger credits checks; an excuse is a closed citation, applicability is declared on the obligation, the http, json and text harness compares three sides by exception type and stage, decision 99's refused document among them, and the twins and the context are generated with the shapes; `NoteKeys` keeps its pass keys and `framework-refuses`; `prober fix`, a covering array that places no blocked pair, and `gaps` listing the pairs no passing check read both axes of; WS2 closes, and WS3 to WS6 are dropped | 2026-10-17 | accepted | diff --git a/design/roadmap.md b/design/roadmap.md index 42142f1f..1729c070 100644 --- a/design/roadmap.md +++ b/design/roadmap.md @@ -13,6 +13,12 @@ defect, a cost or a missing guard may land here; one that adds surface waits for - Grow the probe corpus until `prober gate` passes on probes and reviewed waivers alone. Every claim touched since v2.1.0 is covered pairwise, and no finding is open. +- The probing harness (WS2 of the agentic issue-finding concept) is done + ([decision 108](adr/0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md)). What it leaves the gate: the pairs of C12, C13, C16 and C21 that only a document + ASP.NET alone refuses had credited (a declaration of the document obligations' inapplicability under + `json.ReferenceHandler=Preserve` and `json.MaxDepth=3`, and rows for the rest), and C22's empty footprint. + WS3 (beyond pairs), WS4 (the mutation nightly), the second half of WS5 (the finder benchmark at equal budgets) and + WS6 (PostgreSQL in CI) were dropped on 2026-10-17 by the maintainer's decision, and are not scheduled. - The generated-surface snapshot under "Generators" below. ## 2.3.0, additive diff --git a/test/CodoMetis.TypeKit.Conventions.Tests/ProbeNoteKeyTests.cs b/test/CodoMetis.TypeKit.Conventions.Tests/ProbeNoteKeyTests.cs index 6fb968f4..8876ae13 100644 --- a/test/CodoMetis.TypeKit.Conventions.Tests/ProbeNoteKeyTests.cs +++ b/test/CodoMetis.TypeKit.Conventions.Tests/ProbeNoteKeyTests.cs @@ -24,7 +24,8 @@ public sealed class ProbeNoteKeyTests [Fact] public void The_corpus_notes_are_found() => Calls(Sources()).Count(c => c.Method == "Note").ShouldBeGreaterThanOrEqualTo(100, - "the corpus made 136 notes when this rule was written, and 127 once the http family's outcome notes became checks' citations (WS2 step 4)"); + "the corpus made 136 notes when this rule was written, 127 once the http family's outcome notes became checks' citations (WS2 step 4), " + + "and 119 at WS2's close, where every note is a pass key or framework-refuses"); [Fact] public void Every_note_and_skip_in_the_corpus_names_its_key_as_written() => diff --git a/test/CodoMetis.TypeKit.Prober.Tests/CoveringTests.cs b/test/CodoMetis.TypeKit.Prober.Tests/CoveringTests.cs new file mode 100644 index 00000000..0a824ae1 --- /dev/null +++ b/test/CodoMetis.TypeKit.Prober.Tests/CoveringTests.cs @@ -0,0 +1,80 @@ +using Prober; + +namespace CodoMetis.TypeKit.Prober.Tests; + +/// +/// The covering array never puts an excluded pair or one no obligation wants in a row, not even to fill an axis whose every +/// value would bring one (#54 and #59 saw `rows` place them): such an axis is left out of that row, and an axis with no wanted +/// pair at all is left out of every row. +/// +public sealed class CoveringTests +{ + private static Inapplicability Declared(string cite, params (string Axis, string[] Values)[] axes) => + new(axes.ToDictionary(a => a.Axis, a => a.Values.ToList(), StringComparer.Ordinal), cite); + + private static Claim Over(string[] axes, IEnumerable? excluded = null, params Inapplicability[] declared) => new() + { + Id = "C1", Source = "README.md#a", Surfaces = ["s"], Watched = [], Excluded = [.. excluded ?? []], + Obligations = declared.Length == 0 ? [] : [new("keys", "The claim.", new SortedDictionary>(StringComparer.Ordinal)) { Inapplicable = declared }], + Axes = new(axes.ToDictionary(a => a, _ => new List { "1", "2" }), StringComparer.Ordinal), + }; + + private static bool Holds(IReadOnlyDictionary row, Pair pair) => + new[] { pair.A, pair.B }.All(item => row.TryGetValue(item[..item.IndexOf('=', StringComparison.Ordinal)], out var v) && v == item[(item.IndexOf('=', StringComparison.Ordinal) + 1)..]); + + private static IEnumerable Shown(IEnumerable pairs) => pairs.Select(p => p.ToString()).Order(StringComparer.Ordinal); + + [Fact] + public void An_axis_whose_every_value_brings_an_excluded_pair_is_left_out_of_the_row() + { + // x=1 goes with no y (as shape.containing goes with no shape.position=root): a row holding x=1 holds no y. + Pair[] excluded = [new("x=1", "y=1"), new("x=1", "y=2")]; + var claim = Over(["x", "y", "z"], excluded); + for (var seed = 1; seed <= 20; seed++) + { + var rows = Covering.Rows(claim, claim.Pairs(), 2000, seed); + rows.ShouldNotContain(r => excluded.Any(p => Holds(r.Row, p)), $"seed {seed}"); + rows.Where(r => r.Row["x"] == "1").ShouldAllBe(r => !r.Row.ContainsKey("y"), $"seed {seed}"); + Shown(rows.SelectMany(r => r.Pairs)).ShouldBe(Shown(claim.Pairs()), $"seed {seed}: every pair once, x=1 with z among them"); + } + } + + [Fact] + public void An_axis_whose_every_value_brings_a_pair_no_obligation_wants_is_left_out_of_the_row() + { + var claim = Over(["x", "y", "z"], null, Declared("record:0001", ("x", ["1"]), ("y", ["1", "2"]))); + var dead = claim.DeclaredInapplicable().Keys.ToList(); + dead.Count.ShouldBe(2); + for (var seed = 1; seed <= 20; seed++) + { + var rows = Covering.Rows(claim, claim.PairsWanted(), 2000, seed); + rows.ShouldNotContain(r => dead.Any(p => Holds(r.Row, p)), $"seed {seed}"); + Shown(rows.SelectMany(r => r.Pairs)).ShouldBe(Shown(claim.PairsWanted()), $"seed {seed}"); + } + } + + [Fact] + public void An_axis_with_no_wanted_pair_is_in_no_row() + { + // every pair of z is excluded (as shape.refused-kind is where the refusal is not the claim): rows over x and y alone. + var claim = Over(["x", "y", "z"], [.. from x in new[] { "x=1", "x=2", "y=1", "y=2" } from z in new[] { "z=1", "z=2" } select new Pair(x, z)]); + for (var seed = 1; seed <= 5; seed++) + { + var rows = Covering.Rows(claim, claim.Pairs(), 2000, seed); + rows.ShouldAllBe(r => !r.Row.ContainsKey("z"), $"seed {seed}"); + Shown(rows.SelectMany(r => r.Pairs)).ShouldBe(["x=1 y=1", "x=1 y=2", "x=2 y=1", "x=2 y=2"], $"seed {seed}"); + } + } + + [Fact] + public void A_seeded_row_leaves_out_an_axis_no_value_of_which_can_join_it() + { + // c=1 goes with no d: the row seeded from a=1 c=1 is a, b and c. + var claim = Over(["a", "b", "c", "d"], [new("c=1", "d=1"), new("c=1", "d=2")]); + Covering.Seeded(claim, new Pair("a=1", "c=1"), claim.DeclaredInapplicable()) + .ShouldBe(new SortedDictionary(StringComparer.Ordinal) { ["a"] = "1", ["b"] = "1", ["c"] = "1" }); + Covering.Seeded(claim, new Pair("a=1", "b=2"), claim.DeclaredInapplicable()) + .ShouldBe(new SortedDictionary(StringComparer.Ordinal) { ["a"] = "1", ["b"] = "2", ["c"] = "1" }, + "c=1 is c's first value, and d then has none"); + } +} diff --git a/test/CodoMetis.TypeKit.Prober.Tests/FixTests.cs b/test/CodoMetis.TypeKit.Prober.Tests/FixTests.cs new file mode 100644 index 00000000..38cf460c --- /dev/null +++ b/test/CodoMetis.TypeKit.Prober.Tests/FixTests.cs @@ -0,0 +1,90 @@ +using System.Text.Json.Nodes; +using Prober; + +namespace CodoMetis.TypeKit.Prober.Tests; + +/// +/// prober fix F --commit SHA writes the fixed line of findings.jsonl as finding-line.schema.json wants it, where a +/// fixed finding was a line written by hand (F3, #45): the open line's claim and probe, the commit's full id. +/// +public sealed class FixTests : IDisposable +{ + private readonly Fixture fx = new(); + public void Dispose() => fx.Dispose(); + + private string Findings => Path.Combine(fx.BranchDir, "findings.jsonl"); + + private void Open(string id) => + File.AppendAllText(Findings, new JsonObject + { + ["schemaVersion"] = 1, ["id"] = id, ["claim"] = "C1", ["status"] = "open", ["probe"] = "P", ["reason"] = "row 3 differs", + }.ToJsonString() + "\n"); + + private JsonObject Last() => JsonNode.Parse(File.ReadAllLines(Findings)[^1])!.AsObject(); + + private Ledger Ledger() => new(new Context(fx.Root, new Dictionary(), TextWriter.Null, TextWriter.Null), null); + + [Fact] + public void It_writes_the_fixed_line_with_the_open_lines_claim_and_probe_and_the_full_commit() + { + Open("F9"); + var head = fx.Sh("git", "rev-parse", "HEAD").Trim(); + var (exit, output, error) = fx.Ledger("fix", "F9", "--commit", head[..7], "--reason", "fixed by the converter's null"); + exit.ShouldBe(0, error); + output.ShouldBe($"F9 fixed at {head[..7]}{Environment.NewLine}"); + var line = Last(); + line.Select(kv => kv.Key).ShouldBe(["schemaVersion", "id", "claim", "status", "commit", "probe", "reason"]); + line["schemaVersion"]!.GetValue().ShouldBe(1); + line["id"]!.GetValue().ShouldBe("F9"); + line["claim"]!.GetValue().ShouldBe("C1"); + line["status"]!.GetValue().ShouldBe("fixed"); + line["commit"]!.GetValue().ShouldBe(head); + line["probe"]!.GetValue().ShouldBe("P"); + line["reason"]!.GetValue().ShouldBe("fixed by the converter's null"); + Ledger().OpenFindings().ShouldBeEmpty(); + Ledger().Findings()["F9"]["status"]!.GetValue().ShouldBe("fixed"); + } + + [Fact] + public void A_commit_outside_heads_history_is_recorded_and_counts_once_it_is_in_it() + { + Open("F9"); + fx.Sh("git", "checkout", "-q", "-b", "fix"); + fx.Write("src/a.cs", "class A { }\n"); + fx.Sh("git", "-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qam", "the fix"); // the state stays untracked + var fix = fx.Sh("git", "rev-parse", "HEAD").Trim(); + fx.Sh("git", "checkout", "-q", "main"); + var (exit, output, error) = fx.Ledger("fix", "F9", "--commit", fix); + exit.ShouldBe(0, error); + output.ShouldContain($"F9 fixed at {fix[..7]}; it counts once that commit is in HEAD's history"); + Ledger().OpenFindings().ShouldBe(["F9"]); + fx.Sh("git", "merge", "-q", "--ff-only", "fix"); + Ledger().OpenFindings().ShouldBeEmpty(); + } + + [Theory] + [InlineData("F8", "HEAD", 1, "F8 is no finding of this ledger")] + [InlineData("F9", "0123456789abcdef0123456789abcdef01234567", 1, "--commit 0123456789abcdef0123456789abcdef01234567 is no commit of this repository")] + [InlineData("F9", "", 2, "--commit is required")] + public void It_refuses_a_finding_the_ledger_does_not_hold_and_a_commit_the_repository_lacks(string id, string commit, int code, string message) + { + Open("F9"); + var before = File.ReadAllText(Findings); + var (exit, _, error) = commit.Length > 0 ? fx.Ledger("fix", id, "--commit", commit) : fx.Ledger("fix", id); + exit.ShouldBe(code); + error.ShouldContain(message); + File.ReadAllText(Findings).ShouldBe(before); + } + + [Fact] + public void It_refuses_a_finding_already_fixed_and_a_call_without_one_id() + { + Open("F9"); + fx.Ledger("fix", "F9", "--commit", "HEAD").Exit.ShouldBe(0); + var (exit, _, error) = fx.Ledger("fix", "F9", "--commit", "HEAD"); + exit.ShouldBe(1); + error.ShouldContain("F9 is fixed, not open"); + fx.Ledger("fix", "--commit", "HEAD").Exit.ShouldBe(2); + fx.Ledger("fix", "F9", "F10", "--commit", "HEAD").Exit.ShouldBe(2); + } +} diff --git a/test/CodoMetis.TypeKit.Prober.Tests/GapsTests.cs b/test/CodoMetis.TypeKit.Prober.Tests/GapsTests.cs index 567d8051..d9db71e3 100644 --- a/test/CodoMetis.TypeKit.Prober.Tests/GapsTests.cs +++ b/test/CodoMetis.TypeKit.Prober.Tests/GapsTests.cs @@ -72,6 +72,7 @@ declarations the generators refuse (shape.refused-kind): 1 x=1|2: C1/order "An ordering of keys is not promised." (design/adr/0001-limits.md#1-limits) y: C1 narrows it, "The claim." (README.md#a) pairs no obligation wants (declared inapplicable): none + pairs no passing check read both axes of (the last replay's evidence): none out of reach: 1 postgresql: needs a server @@ -89,7 +90,71 @@ declarations the generators refuse (shape.refused-kind): none values the frameworks refuse on their own: none not promised, by what it concerns: none pairs no obligation wants (declared inapplicable): none + pairs no passing check read both axes of (the last replay's evidence): none out of reach: none """.ReplaceLineEndings("\n")); + + /// + /// C1 over x, y and z: shape wants every pair and its passing checks read x and y; parse does not apply where x=1 and + /// z=1 and its passing checks read x and z. x=1 z=1 is wanted by shape alone, which never read z, and no passing check + /// read y with z: those five pairs are listed. A check that did not pass reads nothing for the listing. + /// + [Fact] + public void The_pairs_no_obligation_wanting_them_read_both_axes_of_are_listed() + { + fx.Write(".claude/probe-axes.md", "```json probe-axes\n" + new JsonObject + { + ["axes"] = new JsonObject { ["x"] = new JsonArray("1", "2"), ["y"] = new JsonArray("1", "2"), ["z"] = new JsonArray("1", "2") }, + ["surfaces"] = new JsonObject { ["s"] = new JsonObject { ["axes"] = new JsonArray("x", "y", "z"), ["watched"] = new JsonArray("src") } }, + }.ToJsonString() + "\n```\n"); + fx.Write("tools/probe/corpus/claims.json", new JsonArray(new JsonObject + { + ["id"] = "C1", ["source"] = "README.md#a", ["surfaces"] = new JsonArray("s"), + ["obligations"] = new JsonArray( + new JsonObject { ["id"] = "shape", ["sentence"] = "The claim." }, + new JsonObject + { + ["id"] = "parse", ["sentence"] = "The claim.", + ["inapplicable"] = new JsonArray(new JsonObject { ["axes"] = new JsonObject { ["x"] = new JsonArray("1"), ["z"] = new JsonArray("1") }, ["cite"] = "docs:README.md#a" }), + }), + }).ToJsonString()); + fx.Commit("three axes"); + fx.Ledger("universe", "--base", "HEAD").Exit.ShouldBe(0); + Gaps("\n").ShouldContain("pairs no passing check read both axes of (the last replay's evidence): none\n", Case.Sensitive, "no evidence yet: nothing to say"); + + static JsonObject Check(string obligation, string category, params (string Axis, string Value)[] read) => new() + { + ["obligation"] = obligation, ["category"] = category, ["baseline"] = "twin", + ["applied"] = new JsonObject(read.Select(r => KeyValuePair.Create(r.Axis, (JsonNode?)r.Value))), + }; + var evidence = Path.Combine(fx.Root, "evidence.json"); + File.WriteAllText(evidence, new JsonObject + { + ["variants"] = new JsonArray(new JsonObject { ["env"] = new JsonObject(), ["expected"] = new JsonArray(0), ["executed"] = new JsonArray(0) }), + ["rows"] = new JsonObject + { + ["0"] = new JsonObject + { + ["category"] = "pass", ["applied"] = new JsonObject(), ["findings"] = new JsonArray(), + ["checks"] = new JsonArray( + Check("shape", "pass", ("x", "1"), ("y", "1")), + Check("parse", "pass", ("x", "2"), ("z", "1")), + Check("parse", "inapplicable", ("y", "2"), ("z", "2"))), + }, + }, + }.ToJsonString()); + fx.Footprint(); + fx.Ledger("record-run", "--probe", "P", "--result", "pass", "--evidence", evidence).Exit.ShouldBe(0); + + Gaps("\n").ShouldContain(""" + pairs no passing check read both axes of (the last replay's evidence): 5 + C1: x=1 z=1 + C1: y=1 z=1 + C1: y=1 z=2 + C1: y=2 z=1 + C1: y=2 z=2 + + """.ReplaceLineEndings("\n")); + } } diff --git a/test/CodoMetis.TypeKit.Prober.Tests/ToolchainTests.cs b/test/CodoMetis.TypeKit.Prober.Tests/ToolchainTests.cs index d2e4cb54..3c22bc0d 100644 --- a/test/CodoMetis.TypeKit.Prober.Tests/ToolchainTests.cs +++ b/test/CodoMetis.TypeKit.Prober.Tests/ToolchainTests.cs @@ -112,7 +112,12 @@ public void A_dotnet_without_the_pinned_sdk_falls_back_to_the_installers() File.WriteAllText(stub, "#!/bin/sh\necho 0.0.1\n"); if (!OperatingSystem.IsWindows()) File.SetUnixFileMode(stub, UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute); using var sandbox = new ToolSandbox("Prober", "prober"); - var environment = new Dictionary { ["TOOLS_DOTNET_FALLBACK_DIR"] = installer }; + // The prober runs as its own process, so the caller's state settings would reach it: under a PROBE_STATE_DIR it + // recorded the run there, not in this repository's state, and status found no claim (#58). None reaches it. + var environment = new Dictionary + { + ["TOOLS_DOTNET_FALLBACK_DIR"] = installer, ["PROBE_STATE_DIR"] = null, ["PROBE_CACHE_DIR"] = null, ["PROBE_HOLDER"] = null, + }; (int, string, string) Prober(params string[] arguments) => ToolSandbox.RunProcess("dotnet", [sandbox.Dll, .. arguments], fx.Root, environment); var (recorded, _, why) = Prober("record-run", "--probe", "P", "--result", "pass", "--evidence", evidence); diff --git a/tools/Prober/Commands.cs b/tools/Prober/Commands.cs index a4bcfcb3..abfd7418 100644 --- a/tools/Prober/Commands.cs +++ b/tools/Prober/Commands.cs @@ -269,6 +269,7 @@ public static int Gaps(Ledger l, Args a) .OrderBy(x => x.Key, StringComparer.Ordinal).ThenBy(x => x.Line, StringComparer.Ordinal).Select(x => x.Line)); Section("pairs no obligation wants (declared inapplicable)", universe.Claims.Select(c => Inapplicable(c) is { } line ? $"{c.Id}: {line}" : null).OfType()); + Section("pairs no passing check read both axes of (the last replay's evidence)", Unread(l, universe)); Section("out of reach", (block["out-of-reach"] as JsonObject ?? []).Select(kv => $"{kv.Key}: {Files.Str(kv.Value)}")); return 0; @@ -283,6 +284,48 @@ static string Concerns(IReadOnlyDictionary> axes) => axes.Count == 0 ? "the whole claim" : string.Join(" ", axes.Select(kv => $"{kv.Key}={string.Join("|", kv.Value)}")); } + /// + /// The wanted pairs of each claim with evidence that no passing check could earn: no obligation that wants the pair + /// has a passing check, in its probes' last replay, that read both of the pair's axes. More rows cannot cover such a + /// pair; a check that reads both axes, an exclusion or a declaration can (#54's four C11 pairs, wanted by obligations + /// whose checks never read shape.wrapped). A claim whose probes have no evidence is left out. + /// + private static IEnumerable Unread(Ledger l, Universe universe) + { + var registry = l.Registry(); + foreach (var claim in universe.Claims.OrderBy(c => c.Id, StringComparer.Ordinal)) + { + // The axes each obligation's passing checks read together, as "a\nb" with a before b. + var read = new Dictionary>(StringComparer.Ordinal); + var any = false; + foreach (var (probe, entry) in registry) + { + if (Files.Str(entry["claim"]) != claim.Id || Files.ReadJson(l.EvidencePath(probe)) is not JsonObject evidence) continue; + any = true; + foreach (var (_, row) in evidence["rows"] as JsonObject ?? []) + if (row is JsonObject r) + foreach (var check in Ledger.ChecksOf(r).Where(c => c.Category == "pass")) + { + if (!read.TryGetValue(check.Obligation, out var axes)) read[check.Obligation] = axes = new(StringComparer.Ordinal); + var names = check.Applied.Keys.Order(StringComparer.Ordinal).ToList(); + for (var i = 0; i < names.Count; i++) + for (var j = i + 1; j < names.Count; j++) axes.Add(names[i] + "\n" + names[j]); + } + } + if (!any) continue; + var wantedBy = claim.ObligationIdsBuilt.Concat(read.Keys).Distinct(StringComparer.Ordinal).ToDictionary(o => o, claim.PairsWantedBy, StringComparer.Ordinal); + foreach (var pair in claim.PairsWanted().OrderBy(p => p.A, StringComparer.Ordinal).ThenBy(p => p.B, StringComparer.Ordinal)) + { + var (a, b) = (Axis(pair.A), Axis(pair.B)); + var both = string.CompareOrdinal(a, b) <= 0 ? a + "\n" + b : b + "\n" + a; + if (!wantedBy.Any(o => o.Value.Contains(pair) && read.TryGetValue(o.Key, out var axes) && axes.Contains(both))) + yield return $"{claim.Id}: {pair.A} {pair.B}"; + } + } + + static string Axis(string setting) => setting[..setting.IndexOf('=', StringComparison.Ordinal)]; + } + public static int Gate(Ledger l, Args a) => ReleaseGate.Run(l, a.Get("since"), a.Flag("json")); public static int Footprint(Ledger l, Args a) @@ -507,6 +550,32 @@ public static int Dismiss(Ledger l, Args a) return 0; } + /// + /// Records an open finding fixed at a commit: the fixed line of findings.jsonl (finding-line.schema.json), with the + /// open line's claim and probe and the commit's full id. It refuses an id this ledger does not hold open and a + /// commit the repository does not have; a commit outside HEAD's history is recorded and counts once it is in it. + /// + public static int Fix(Ledger l, Args a) + { + var id = a.Positional is [var only] ? only : throw new Refusal("fix takes one finding id: fix F --commit ", 2); + var given = a.Need("commit"); + var resolved = Git.Run(l.Top, "rev-parse", "-q", "--verify", given + "^{commit}"); + if (resolved.ExitCode != 0) throw new Refusal($"--commit {given} is no commit of this repository"); + var commit = resolved.Stdout.Trim(); + if (!l.Findings().TryGetValue(id, out var open)) throw new Refusal($"{id} is no finding of this ledger"); + if (Files.Str(open["status"]) is var status and not "open") throw new Refusal($"{id} is {status}, not open"); + var line = new JsonObject + { + ["schemaVersion"] = Files.SchemaVersion, ["id"] = id, ["claim"] = open["claim"]?.DeepClone(), ["status"] = "fixed", ["commit"] = commit, + }; + if (Files.Str(open["probe"]) is { Length: > 0 } probe) line["probe"] = probe; + if (a.Get("reason") is { Length: > 0 } reason) line["reason"] = reason; + Files.AppendLine(l.P("findings.jsonl"), line); + var inHistory = Git.Run(l.Top, "merge-base", "--is-ancestor", commit, "HEAD").ExitCode == 0; + l.Ctx.Out.WriteLine($"{id} fixed at {commit[..7]}" + (inHistory ? "" : "; it counts once that commit is in HEAD's history")); + return 0; + } + public static int Lock(Ledger l, Args a) { var holder = a.Need("holder"); var mode = a.Need("mode"); diff --git a/tools/Prober/Covering.cs b/tools/Prober/Covering.cs index f5261625..d4cfd13c 100644 --- a/tools/Prober/Covering.cs +++ b/tools/Prober/Covering.cs @@ -3,8 +3,11 @@ namespace Prober; /// /// A greedy pairwise covering array: each row is the best of a few randomized greedy candidates, and the /// rows together cover every pair asked for that at least one of the claim's obligations wants. A pair no -/// obligation wants (declared inapplicable) is never asked for and never put in a row, as an excluded pair is -/// not: every check of such a row would be inapplicable. Deterministic for a seed. +/// obligation wants (declared inapplicable) is never asked for, and neither it nor an excluded pair is ever put +/// in a row, not even to fill an axis: every check of such a row would be inapplicable, and an excluded pair is +/// one the claim says no row can run. An axis none of whose values can join a row without one is left out of +/// that row: shape.containing beside shape.position=root, and shape.refused-kind wherever the claim excludes each of +/// its pairs. Deterministic for a seed. /// internal static class Covering { @@ -14,7 +17,7 @@ internal static class Covering bool Blocked(Pair pair) => claim.Excluded.Contains(pair) || dead.ContainsKey(pair); // The pairs asked for, each once however many obligations want it, so a row's gain counts it once. var todo = new HashSet(wanted); - todo.ExceptWith(dead.Keys); + todo.RemoveWhere(Blocked); var names = claim.Axes.Keys.ToList(); var random = new Random(seed); var rows = new List<(SortedDictionary, List)>(); @@ -28,7 +31,7 @@ internal static class Covering foreach (var name in order) { var options = claim.Axes[name].OrderBy(_ => random.Next()).ToList(); - string? pick = null; var pickGain = int.MinValue; + string? pick = null; var pickGain = -1; foreach (var v in options) { var gain = 0; @@ -40,7 +43,8 @@ internal static class Covering } if (gain > pickGain) { pick = v; pickGain = gain; } } - row[name] = pick!; + // Every value of this axis puts a blocked pair in the row: the axis is left out of it. + if (pick is not null) row[name] = pick; } var g = Gained(row, todo).Count; if (g > bestGain) { best = row; bestGain = g; } @@ -57,16 +61,20 @@ internal static class Covering return rows; } + static string Axis(string setting) => setting[..setting.IndexOf('=', StringComparison.Ordinal)]; + /// - /// A row seeded from one pair: the pair's two settings, and each other axis at its first value that puts no pair - /// no obligation wants () in the row, else at its first value. + /// A row seeded from one pair: the pair's two settings, and each other axis at its first value that puts neither an excluded pair + /// nor a pair no obligation wants () in the row, or left out where none does. /// internal static SortedDictionary Seeded(Claim claim, Pair pair, IReadOnlyDictionary> dead) { + bool Blocked(Pair p) => claim.Excluded.Contains(p) || dead.ContainsKey(p); var row = new SortedDictionary(StringComparer.Ordinal); - foreach (var item in new[] { pair.A, pair.B }) { var at = item.IndexOf('=', StringComparison.Ordinal); row[item[..at]] = item[(at + 1)..]; } + foreach (var item in new[] { pair.A, pair.B }) row[Axis(item)] = item[(item.IndexOf('=', StringComparison.Ordinal) + 1)..]; foreach (var name in claim.Axes.Keys.Where(n => !row.ContainsKey(n)).ToList()) - row[name] = claim.Axes[name].FirstOrDefault(v => !row.Any(kv => dead.ContainsKey(new Pair($"{name}={v}", $"{kv.Key}={kv.Value}")))) ?? claim.Axes[name][0]; + if (claim.Axes[name].FirstOrDefault(v => !row.Any(kv => Blocked(new Pair($"{name}={v}", $"{kv.Key}={kv.Value}")))) is { } value) + row[name] = value; return row; } diff --git a/tools/Prober/Program.cs b/tools/Prober/Program.cs index 1d931437..7b62b3ae 100644 --- a/tools/Prober/Program.cs +++ b/tools/Prober/Program.cs @@ -15,7 +15,8 @@ inside a capped round it refuses past the deadline or the claim cap status [--json] coverage from current evidence, open findings, done or not; per claim its obligations and which have a seed gaps [--base REF] what the corpus leaves out, each with its reason: unassigned settings, - refused shapes and values, what claims do not promise, out of reach + refused shapes and values, what claims do not promise, out of reach, and + the wanted pairs no passing check of the last replay read both axes of gate [--since REF] [--json] the release gate (needs only the tracked corpus and a replay) stale [--json] [--include-retired] the probes a replay would run identity --probe P what a result of P is bound to @@ -26,6 +27,8 @@ stale [--json] [--include-retired] the probes a replay would run --obligation into `seeds`, in place of a seed naming O; refuses an O that claim C does not list publish | dismiss --id F --reason TEXT | mark retired --probe P + fix F --commit SHA [--reason TEXT] records open finding F fixed at the commit (findings.jsonl); refuses + an id this ledger does not hold open and a commit the repository lacks lock --holder H --mode replay|explore|full [--pid N] | unlock --holder H | heartbeat --holder H round-start --holder H --minutes M [--claims N] | round-end --holder H | repair | abandon merge --from DIR... joins the states of a replay split by family (one DIR per leg) into @@ -101,6 +104,7 @@ public static int Run(Context ctx, string[] args) "register" => Commands.Register(ledger, a), "publish" => Commands.Publish(ledger, a), "dismiss" => Commands.Dismiss(ledger, a), + "fix" => Commands.Fix(ledger, a), "mark" => Commands.Mark(ledger, a), "lock" => Commands.Lock(ledger, a), "unlock" => Commands.Unlock(ledger, a), diff --git a/tools/Prober/README.md b/tools/Prober/README.md index 86e913f5..c957ffab 100644 --- a/tools/Prober/README.md +++ b/tools/Prober/README.md @@ -16,7 +16,8 @@ dotnet tools/Prober/bin/Release/net10.0/prober.dll help | `merge --from DIR...` | none | anywhere, CI included: joins the states of a replay split by family, and refuses while a registered probe was in no leg | | `gate [--since REF]` | none | anywhere, CI included: needs only the tracked corpus and a replay | | `status`, `stale`, `universe`, `rows [--claim C \| --probe P]`, … | none | anywhere | -| `gaps [--base REF]` | none | anywhere: what the corpus leaves out, each with its recorded reason; a report that decides nothing (decision 104) | +| `gaps [--base REF]` | none | anywhere: what the corpus leaves out, each with its recorded reason, and, from the last replay's evidence, each wanted pair no obligation wanting it has a passing check that read both its axes of (more rows cannot cover it); a report that decides nothing (decision 104) | +| `fix F --commit SHA [--reason TEXT]` | none | anywhere: records an open finding fixed at a commit, the fixed line of the state's `findings.jsonl` as `finding-line.schema.json` wants it; it counts once the commit is in HEAD's history | | `new --claim C [--probe P] [--family F]` | none | anywhere: stages a probe's skeleton, its own rows and its registry entry with no seed yet, and prints what the writer still owes | | `try --probe P [--rows i,j,...] [--no-build]` | none | locally: builds once in the replay's worktree and runs that one probe, every row or the rows given; records nothing, and holds the worktree from a replay while it runs | | `round explore\|full` | the `prober` agent | locally only, under caps it cannot overrun | diff --git a/tools/probe/corpus/claims.json b/tools/probe/corpus/claims.json index b1bed7c8..05eb1ed8 100644 --- a/tools/probe/corpus/claims.json +++ b/tools/probe/corpus/claims.json @@ -11703,7 +11703,7 @@ ] }, { - "reason": "Not promised (a-closed-set-is-a-string-enum): a closed set wraps nothing and is described as the string enum of its names, so it has no underlying type's schema to equal; the probe notes closed-set-differs.", + "reason": "Not promised (a-closed-set-is-a-string-enum): a closed set wraps nothing and is described as the string enum of its names, so it has no underlying type's schema to equal.", "pairs": [ [ "http.binding=route", diff --git a/tools/probe/corpus/harness/ComposedProbes.cs b/tools/probe/corpus/harness/ComposedProbes.cs index d7597769..56532242 100644 --- a/tools/probe/corpus/harness/ComposedProbes.cs +++ b/tools/probe/corpus/harness/ComposedProbes.cs @@ -94,7 +94,7 @@ public static class ComposedClaims if (!Aspect.Doc.Any(want)) return null; // the document: one ASP.NET alone refuses the same way has nothing to check (noted there) if (docFailed is not null) - return await Hx.DocRefusedByAspNetAlone(r, en, Cfg(r), docFailed) is { } alone + return await Hx.DocRefusedByAspNetAlone(r, en, Cfg(r), docFailed, (Aspect.Answer.Any(want) || want(Rule)) && sides.Count > 0) is { } alone ? alone + " @ " + string.Join(" < ", (docFailed.StackTrace ?? "").Split('\n').Take(4).Select(x => x.Trim().Split('(')[0])) : null; foreach (var path in sides) diff --git a/tools/probe/corpus/harness/FaultClaim.cs b/tools/probe/corpus/harness/FaultClaim.cs index fba5965e..03018dd1 100644 --- a/tools/probe/corpus/harness/FaultClaim.cs +++ b/tools/probe/corpus/harness/FaultClaim.cs @@ -280,13 +280,19 @@ static void Walk(IEndpointRouteBuilder? b, MGrp g, Decl[] above, string prefix, { var f = await r.Check(o, Against(o), async c => { - bool want(string aspect, Mem? m) => s.Of(aspect, m).Contains(o); + var wire = false; // whether the check compared an answer on the wire before the document + bool want(string aspect, Mem? m) + { + var wanted = s.Of(aspect, m).Contains(o); + if (wanted && !Aspect.Doc.Contains(aspect)) wire = true; + return wanted; + } foreach (var ep in eps) foreach (var e in s.Errors) if (Wire(r, ep, FxErr.H[e - 1], wires[(ep.Url, e)], display, want) is { } fail) return fail; if (!Aspect.Doc.Any(a => want(a, null))) return null; // A document ASP.NET alone refuses too has nothing to check (noted there); any other failure is returned. - if (docFailed is not null) return await DocRefusedByAspNetAlone(c, r, en, cfg, docFailed); + if (docFailed is not null) return await DocRefusedByAspNetAlone(c, r, en, cfg, docFailed, wire); foreach (var ep in eps) if (DocOf(r, doc!, ep, want) is { } fail) return fail; return null; @@ -314,7 +320,7 @@ static void Walk(IEndpointRouteBuilder? b, MGrp g, Decl[] above, string prefix, /// (README "A Result as a response"), so ASP.NET alone writing a document of the same host without any problem response says nothing about /// them: under json.MaxDepth=3 System.Text.Json refuses the problem schema of ASP.NET's own ProducesProblem too (decision 99). /// - static async Task DocRefusedByAspNetAlone(Check c, Row r, Entry en, HostCfg cfg, Exception withTypeKit) + static async Task DocRefusedByAspNetAlone(Check c, Row r, Entry en, HostCfg cfg, Exception withTypeKit, bool comparedBefore) { static string Say(Exception x) => x.GetType().Name + ": " + Cut(x.Message); var alone = cfg.Copy(); alone.TypeKitOpenApi = false; @@ -325,11 +331,10 @@ static void Walk(IEndpointRouteBuilder? b, MGrp g, Decl[] above, string prefix, await using var app = await Hx.Start(r, en, alone); await Hx.Doc(app); } - // the same refusal, at another byte of each document (Hx.Unplaced: the two differ in length by what AddTypeKit() wrote; rows 73 and - // 157 of C06's regenerated rows, a char under WriteAsString) - catch (Exception ex) when (ex.GetType() == withTypeKit.GetType() && Hx.Unplaced(ex.Message) == Hx.Unplaced(withTypeKit.Message)) + // the same refusal by type and stage, never by its words (Hx.DocAlike) + catch (Exception ex) when (Hx.DocAlike(ex, withTypeKit)) { - r.Note(NoteKeys.DocRefusedByAspnetAlone, ex.GetType().Name); + Hx.DocRefused(r, ex, comparedBefore); return null; } catch (Exception ex) when (r.Is("json.MaxDepth", "3") && Depth(ex) && Depth(withTypeKit)) diff --git a/tools/probe/corpus/harness/HttpHost.cs b/tools/probe/corpus/harness/HttpHost.cs index 490aa712..d37238fc 100644 --- a/tools/probe/corpus/harness/HttpHost.cs +++ b/tools/probe/corpus/harness/HttpHost.cs @@ -307,12 +307,12 @@ public static async Task Send(WebApplication app, HttpMethod m, string pat public static async Task Doc(WebApplication app) => JsonNode.Parse(await app.GetTestClient().GetStringAsync("/openapi/v1.json")); /// - /// The document failed with AddTypeKit(). The same host without it is the baseline: when ASP.NET alone - /// refuses the document with the same exception, the document part has nothing to check and the note says - /// so; when it writes the document, or fails otherwise, the failure is returned (F2 hid behind a baseline - /// that kept AddTypeKit() on both sides). + /// The document failed with AddTypeKit(). The same host without it is side (1) of the three (Sides): where ASP.NET alone refuses the + /// document alike, by exception type and stage and never by message (), the document part has nothing to check + /// (); where it writes the document, or refuses it otherwise, the failure is returned (F2 hid behind a baseline + /// that kept AddTypeKit() on both sides). says whether the check compared an answer before the document. /// - public static async Task DocRefusedByAspNetAlone(Row r, Entry e, HostCfg c, Exception withTypeKit) + public static async Task DocRefusedByAspNetAlone(Row r, Entry e, HostCfg c, Exception withTypeKit, bool comparedBefore = false) { static string Say(Exception x) => x.GetType().Name + ": " + (x.Message.Length > 200 ? x.Message[..200] + "..." : x.Message); var alone = c.Copy(); alone.TypeKitOpenApi = false; @@ -321,9 +321,9 @@ public static async Task Send(WebApplication app, HttpMethod m, string pat await using var app = await Start(r, e, alone); await Doc(app); } - catch (Exception ex) when (ex.GetType() == withTypeKit.GetType() && Unplaced(ex.Message) == Unplaced(withTypeKit.Message)) + catch (Exception ex) when (DocAlike(ex, withTypeKit)) { - r.Note(NoteKeys.DocRefusedByAspnetAlone, ex.GetType().Name); + DocRefused(r, ex, comparedBefore); return null; } catch (Exception ex) { return $"document: AddTypeKit() fails ({Say(withTypeKit)}), ASP.NET alone fails otherwise ({Say(ex)})"; } @@ -331,11 +331,24 @@ public static async Task Send(WebApplication app, HttpMethod m, string pat } /// - /// A JsonException's message without the line and byte it names: the two documents differ in length by what - /// AddTypeKit() wrote, so one refusal is at another byte of each (rows 31 and 140 of C21: "Path: $ | LineNumber: 0 | - /// BytePositionInLine: 32." against 41, the same refusal of a char under WriteAsString). + /// ASP.NET alone (side (1)) and AddTypeKit() (side (3)) refuse the document alike: the same exception type at the same stage (a + /// JsonException's Path), as compares two refusals. No message is compared: the two documents differ in length + /// by what AddTypeKit() wrote, so one refusal is at another line and byte of each (rows 31 and 140 of C21), and a refusal's words decide + /// no row (decision 99, WS2's three-sided differential). /// - public static string Unplaced(string message) => System.Text.RegularExpressions.Regex.Replace(message, @" \| LineNumber: \d+ \| BytePositionInLine: \d+\.$", "."); + public static bool DocAlike(Exception one, Exception three) => + Sides.Alike(new Sides.Answer("document", null, one), new Sides.Answer("document", null, three)); + + /// + /// The document refused alike by ASP.NET alone: its part has nothing to check. A check that compared nothing before it is the framework's + /// refusal, an expected refusal that earns nothing (framework-refuses); one that compared an answer on the wire first is judged by that + /// answer (doc:refused-by-aspnet-alone, a pass: decision 99's "judged by its other checks"). + /// + public static void DocRefused(Row r, Exception one, bool comparedBefore) + { + if (comparedBefore) r.Note(NoteKeys.DocRefusedByAspnetAlone, one.GetType().Name); + else r.Note(NoteKeys.FrameworkRefuses, $"document refused by ASP.NET alone too ({one.GetType().Name}), side (1) measured on the same host without AddTypeKit()"); + } public static string BindText(Entry e, object plain) => e.Wrapped switch { diff --git a/tools/probe/corpus/harness/Row.cs b/tools/probe/corpus/harness/Row.cs index e46beed1..79703c65 100644 --- a/tools/probe/corpus/harness/Row.cs +++ b/tools/probe/corpus/harness/Row.cs @@ -338,15 +338,16 @@ static string Form(string value, Regex form, string what, string example) => /// /// Every note a check may make, by exact key, and the outcome it stands for: the catalogue, and each of its /// members one key. A check writes r.Note(NoteKeys.Compared, detail); the constructor is private, so a -/// key outside the catalogue, a misspelled one or one built from text does not compile (decision 103). A row -/// whose checks returned no failure is a pass only if none of its notes says a check was skipped -/// (inapplicable), ended in a refusal (expected refusal) or could not tell (inconclusive): a refusal is -/// evidence about refusing, not about serializing or binding successfully. "pass" keys only explain how a -/// check that ran was compared. +/// key outside the catalogue, a misspelled one or one built from text does not compile (decision 103). "pass" +/// keys only explain how a check that ran was compared. A check that did not apply, ended in a refusal or could +/// not tell says so through its scope, citing what makes it so (c.Inapplicable(Cite…), c.Refused(Cite…), +/// c.Inconclusive(reason)); the one other key is , the three-sided +/// differential's expected refusal, against side (1)'s baseline. The outcome keys of the notes those replaced +/// were removed once no harness or probe source called them (WS2's close). /// public sealed class NoteKeys { - const string Pass = "pass", Inapplicable = "inapplicable", ExpectedRefusal = "expected-refusal", Inconclusive = "inconclusive"; + const string Pass = "pass", ExpectedRefusal = "expected-refusal"; /// The key as a row's notes and reasons spell it. public string Key { get; } /// The outcome a note with this key stands for; a note never makes a violation or a harness error. @@ -384,65 +385,17 @@ public sealed class NoteKeys // L03's JSON half has nothing to compare: the wrapped type alone refuses under the row's options, the value object alike, in the same row public static readonly NoteKeys JsonHalfWrappedAloneRefuses = new("json-half:wrapped-alone-refuses", Pass); // not called since the three-sided differential (Sides): framework-refuses // JSON family (C01, C10, L06): the options leave the member out of one direction for the plain type without TypeKit too, shown by a - // baseline in the same row; that direction has nothing to check, and the row is judged by the other one (both left out: inapplicable) + // baseline in the same row; that direction has nothing to check, and the row is judged by the other one (both left out: each direction inconclusive) public static readonly NoteKeys JsonMemberLeftOutByOptions = new("json:member-left-out-by-options", Pass); - public static readonly NoteKeys JsonBothDirectionsLeftOutByOptions = new("json:both-directions-left-out-by-options", Inapplicable); // a read the wrapped type's shape refuses with the very same exception and message: the serializer refused the document before any // value object ran (the baseline in the same row), so the read has nothing to check; the row is judged by its write and other reads public static readonly NoteKeys JsonReadRefusedByStjAlone = new("json:read-refused-by-stj-alone", Pass); // not called since the three-sided differential (Sides): framework-refuses // ASP.NET's OpenAPI 2.0 writer drops the reference of a nullable struct component; a plain struct beside it in the same // schema, without a converter, is written the same, so that position has nothing to check (C12) public static readonly NoteKeys SchemaNullableReferenceLostForPlainToo = new("schema:nullable-reference-lost-for-plain-too", Pass); - // a check that did not apply - public static readonly NoteKeys OwnVoConverter = new("own-vo-converter", Inapplicable); - public static readonly NoteKeys ClosedSetVsOwnWrappedConverter = new("closed-set-vs-own-wrapped-converter", Inapplicable); - public static readonly NoteKeys ClosedSetDiffers = new("closed-set-differs", Inapplicable); - public static readonly NoteKeys ByteArrayIsBase64 = new("byte-array-is-base64", Inapplicable); - public static readonly NoteKeys OwnSpelling = new("own-spelling", Inapplicable); - public static readonly NoteKeys NotBindableFromText = new("not-bindable-from-text", Inapplicable); - public static readonly NoteKeys RouteTextUnsafe = new("route-text-unsafe", Inapplicable); - public static readonly NoteKeys OwnVoBodyBindingSkipped = new("own-vo:body-binding-skipped", Inapplicable); - public static readonly NoteKeys MvcMaxvalidationdepth1IgnoresBindingErrors = new("mvc-maxvalidationdepth1-ignores-binding-errors", Inapplicable); - public static readonly NoteKeys StoredFactorySkipsValidation = new("stored-factory-skips-validation", Inapplicable); - public static readonly NoteKeys SchemaConverterNotDescribed = new("schema:converter-not-described", Inapplicable); - public static readonly NoteKeys SchemaNullabilityNotComparable = new("schema:nullability-not-comparable", Inapplicable); - // C21's not-promised entries, made by ASP.NET's OpenAPI 2.0 writer: a nullable property of a component loses its reference (KB-026), - // a nullable body of a value type keeps only its null (KB-001) - public static readonly NoteKeys SchemaOpenApi2NullableComponentLosesType = new("schema:openapi2-nullable-component-loses-type", Inapplicable); - public static readonly NoteKeys SchemaOpenApi2NullableBodyKeepsOnlyNull = new("schema:openapi2-nullable-body-keeps-only-null", Inapplicable); - // C21's not-promised enum-component-is-its-first-use, made by ASP.NET: an enum's component is the schema of its first use (KB-027) - public static readonly NoteKeys SchemaEnumComponentIsItsFirstUse = new("schema:enum-component-is-its-first-use", Inapplicable); - public static readonly NoteKeys ClosedSetKeyUnpoliced = new("closed-set-key-unpoliced", Inapplicable); - public static readonly NoteKeys PlainRefused = new("plain-refused", Inapplicable); - public static readonly NoteKeys EntryNotApplicable = new("entry-not-applicable", Inapplicable); - public static readonly NoteKeys NoText = new("no-text", Inapplicable); - public static readonly NoteKeys TextWrappedRoundTripLossyToo = new("text:wrapped-round-trip-lossy-too", Inapplicable); - public static readonly NoteKeys NoOperator = new("no-operator", Inapplicable); - public static readonly NoteKeys RefusedInputNotPlaceable = new("refused-input-not-placeable", Inapplicable); - // a refusal, intended or the framework's own - public static readonly NoteKeys DocumentedDiff = new("documented-diff", ExpectedRefusal); - public static readonly NoteKeys StjMaxdepth = new("stj-maxdepth", ExpectedRefusal); // not called since the three-sided differential (Sides): KB-040's side (1) - public static readonly NoteKeys ValidatedRefusesTypeLimit = new("validated-refuses-type-limit", ExpectedRefusal); - public static readonly NoteKeys ClassVoTakesNull = new("class-vo-takes-null", ExpectedRefusal); - public static readonly NoteKeys BindPlainUnbindable = new("bind:plain-unbindable", ExpectedRefusal); - public static readonly NoteKeys PreserveDocumented = new("preserve-documented", ExpectedRefusal); - public static readonly NoteKeys Mvc400UnwritableUnderSourceGen = new("mvc-400-unwritable-under-source-gen", ExpectedRefusal); // not called since the three-sided differential (Sides): KB-041's side (1) - public static readonly NoteKeys OwnVoReadBackRefused = new("own-vo:read-back-refused", ExpectedRefusal); - public static readonly NoteKeys BothThrew = new("both-threw", ExpectedRefusal); - public static readonly NoteKeys ReadBothThrew = new("read-both-threw", ExpectedRefusal); - public static readonly NoteKeys SerBothThrew = new("ser-both-threw", ExpectedRefusal); - public static readonly NoteKeys SerRefused = new("ser-refused", ExpectedRefusal); - public static readonly NoteKeys ReadBackRefused = new("read-back-refused", ExpectedRefusal); - public static readonly NoteKeys PlainReadBackFailsToo = new("plain-read-back-fails-too", ExpectedRefusal); - public static readonly NoteKeys OwnVoConverterThrew = new("own-vo-converter-threw", ExpectedRefusal); - public static readonly NoteKeys OwnConverterRefusedForWrappedToo = new("own-converter-refused-for-wrapped-too", ExpectedRefusal); - // MVC's validator throws for a nested struct of a body (MaxValidationDepth) for a plain twin on the same side too (C13) - public static readonly NoteKeys MvcPlainNestedBodyThrowsToo = new("mvc:plain-nested-body-throws-too", ExpectedRefusal); // the three-sided differential (Sides): the wrapped twin without TypeKit and the value object refuse alike, at the same operation and // stage with the same exception type and status, so the framework refuses; no message is compared public static readonly NoteKeys FrameworkRefuses = new("framework-refuses", ExpectedRefusal); - // both sides refused, but not alike - public static readonly NoteKeys BothThrewDifferently = new("both-threw-differently", Inconclusive); } public static class Rows diff --git a/tools/probe/corpus/harness/SchemaClaim.cs b/tools/probe/corpus/harness/SchemaClaim.cs index da3b92de..65b49f19 100644 --- a/tools/probe/corpus/harness/SchemaClaim.cs +++ b/tools/probe/corpus/harness/SchemaClaim.cs @@ -556,6 +556,7 @@ void Register(JsonSerializerOptions o) { var f = await r.Check(o, Baseline.Docs("docs/merge-patches.md#required-constructor-parameters"), async c => { + var compared = false; // whether the check judged an answer before the document foreach (var (path, kind, body, w, twinThrew) in answers) { // a body without the patch is the listing's premise where the options reject it, else what absent-is-default reads @@ -566,6 +567,7 @@ void Register(JsonSerializerOptions o) c.Inconclusive($"{path} {body}: MVC's validator throws {twinThrew} for the plain twin's body too"); continue; } + compared = true; if (w.Thrown is not null) return $"{path} {body}: request threw {w.Thrown}"; switch (kind) { @@ -581,7 +583,7 @@ void Register(JsonSerializerOptions o) } if (o != listing) return null; // a document ASP.NET alone refuses the same way has nothing to check (noted there) - if (docFailed is not null) return await Hx.DocRefusedByAspNetAlone(r, en, cfg, docFailed); + if (docFailed is not null) return await Hx.DocRefusedByAspNetAlone(r, en, cfg, docFailed, compared); foreach (var path in Sides(r, family)) { var op = Op(doc!, path, "post");