diff --git a/.claude/probe-axes.md b/.claude/probe-axes.md index 634e6ecf..6b86a28b 100644 --- a/.claude/probe-axes.md +++ b/.claude/probe-axes.md @@ -152,7 +152,10 @@ axes from its surfaces. The prober assigns surfaces to a claim; it does not choo declarations and records such a check inapplicable before its body runs; a pair no obligation wants is never put in a generated row, and the gate leaves it out as it leaves out an exclusion. A condition only the shape instance decides at run time stays in the check's body as `c.Inapplicable(cite)`. -- `refused` drops a value the framework itself refuses on its own, so no row wastes its pairs on it. +- `refused` drops a value the framework itself refuses on its own, so no row wastes its pairs on it. A reason that + rests on a verified fact names it (`fact:KB-NNN`), and that fact has a test: `docs-query check` fails + `fact-not-executable-cited` for a refused entry, a not-promised entry, an applicability condition or a harness + citation of a fact without one ([decision 107](../design/adr/0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md)). `shape.refused-kind` holds the declarations the generators refuse; only the generation surface takes it, where the refusal is the claim. Each value is a declaration and the id the Generators README's "Build errors" table gives it, declared as C22's probe does (`harness/BuildRefusals.cs`): an unsealed diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a8fabbb1..2d9df4a8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -143,7 +143,10 @@ jobs: echo "PROBE_CACHE_DIR=$RUNNER_TEMP/probe-cache" >> "$GITHUB_ENV" # Every probe of the corpus, with no model, as CI replays it on every push (design/adr/0096). A probe - # that fails fails the release, like a test. The Build above built the prober. + # that fails fails the release, like a test. The Build above built the prober. Every probe's seeds are + # shown caught before the tag, locally, by `prober seeds --all` (AGENTS.md, Releasing): a build per seed, + # which this job does not pay (design/adr/0100), and --all because a run without it skips the probes + # unchanged since their seeds were caught (design/adr/0107). - name: Replay every probe run: dotnet tools/Prober/bin/Release/net10.0/prober.dll replay --all diff --git a/AGENTS.md b/AGENTS.md index 811f0caa..06300ff6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,7 +78,9 @@ dotnet tools/Prober/bin/Release/net10.0/prober.dll gate # the release gate, ``` Proving that each probe can fail needs no model either, but builds once per seed, so it runs locally -only ([decision 100](design/adr/0100-every-probe-is-shown-to-fail-against-a-seeded-defect.md)): +only ([decision 100](design/adr/0100-every-probe-is-shown-to-fail-against-a-seeded-defect.md)), and only for the +probes changed since their seeds were last caught; a release runs `--all` +([decision 107](design/adr/0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md)): ```bash dotnet tools/Prober/bin/Release/net10.0/prober.dll seeds # each probe's seeded defect, applied in a scratch worktree, must fail it @@ -268,11 +270,13 @@ the whole page. A ranked miss proves nothing. ## Releasing -Before `git tag`, replay the whole probe corpus and run the gate: +Before `git tag`, replay the whole probe corpus and run the gate, and show every probe's seeds caught (locally, a +build per seed; without `--all` it runs only the probes changed since their seeds were last caught, decision 107): ```bash dotnet tools/Prober/bin/Release/net10.0/prober.dll replay --all dotnet tools/Prober/bin/Release/net10.0/prober.dll gate +dotnet tools/Prober/bin/Release/net10.0/prober.dll seeds --all ``` The gate never starts a model and needs nothing but the tracked corpus and a replay, so the release diff --git a/design/adr/0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md b/design/adr/0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md new file mode 100644 index 00000000..f11db031 --- /dev/null +++ b/design/adr/0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md @@ -0,0 +1,69 @@ +# 0107. A fact that prunes or excuses is executable, its answer is a fixture beside its test, and seeds run by footprint + +Date: 2026-10-17 +Status: accepted + +Context: 2026-10-17, WS2b of the probing concept (facts at a pinned version shorten every loop), after the probe +triage of 2026-10-16 and 2026-10-17 pinned KB-026 to KB-038 and the harness began to cite them +(`Cite.Fact("KB-037")`, PR #53). [Decision 93](0093-verified-facts-are-pinned-dated-and-executable-where-they-can-be.md) +made a fact executable where it can be; nothing held the probe corpus to citing only those. A prose fact cited by a +`refused` entry, a `not-promised` entry or an applicability condition would prune or excuse rows on the strength of a +sentence no test re-reads when a dependency moves, which is the substring excuse in a nicer form. The facts whose +tests compare a document or a status held the measured answer inside the test's code, where the harness, which will +read a fact as the framework's side of a row (`Baseline.Fact`), cannot reach it. And `prober seeds` built once per +seed of every probe on every run, a release step's cost paid after every probe change. + +## Decision + +**A fact that prunes or excuses is executable.** `docs-query check` gains `fact-not-executable-cited`. It fails, at +the citing place and naming the fact, when a fact without a test is cited by a `refused` entry of +`.claude/probe-axes.md` (its reason names `KB-NNN`), a `not-promised` entry of `tools/probe/corpus/claims.json` (its +`source` is the fact's file, or a field says `fact:KB-NNN`), an obligation's `inapplicable` declaration (its `cite` is +`fact:KB-NNN`), or a `Cite.Fact("KB-NNN")` or `Baseline.Fact("KB-NNN")` literal of the harness and probe sources, read +as text with comments blanked out as `SourceLint` reads them. A fact without a test is one whose `test:` is `none, …`, +names no file, or names a file that does not exist or declares no class of the fact's id; a cited id that is no fact +fails too. `Baseline.Fact` takes a fixture as well. The check's summary counts the citations of each kind, and +`RepositoryTests` holds floors on the places read and the citations found, so a scan that finds nothing cannot pass. +A fact without a test keeps its pins and its date and may be named in a reason's prose; it never decides a row. + +**A fact's measured answer is a fixture beside its test.** Where the answer is data (a document, a status, an +exception type at an operation), it is `test/CodoMetis.TypeKit.Facts/KBNNN.fixture.json`, named by the fact's +`fixture:` line: the fact's id and its answers, each with an `id`, a `kind` (`document` with `at`, a JSON pointer; +`status`; `exception`), the settings it was measured under (`when`, axis to value, as a row names them), the `value` +and a `note`. The fact's test measures every answer of the fixture, a theory over its ids, so an answer the test does +not measure fails it. `docs-query check` fails a `fixture:` whose file is missing (`fact-fixture-missing`), one that is +not `KBNNN.fixture.json` beside the test or belongs to a fact without a test (`fact-fixture`), and a fixture no fact +names (`fact-orphan-fixture`). The reader, `FactFixture`, is BCL only, so the harness can compile it as it is, and +states the rule the harness will follow: only a fixture a fact names, only while the check is clean; the one answer +whose every `when` setting is the row's, none meaning the harness measures the framework itself and several a harness +error; compared by kind; never written by the harness. KB-001, KB-026, KB-027 and KB-037 are in this form. + +**Seeds run by footprint, the release runs them all.** Without `--all`, `prober seeds` runs only the probes whose seeds +identity (the probe's identity, its seeds and their patches' content) changed, or whose product footprint or seeded +file changed, since the run that caught every one of its seeds, and prints each probe it skips and why. +`seeds-last.json` records, per probe, the identity and the commit of that run; a probe with a seed missed loses its +record, so it runs next time whatever changed, and a probe without a recorded footprint runs. `--all` runs every probe +whatever the record says, and `--probe P` runs P. A release runs `seeds --all` locally before the tag, beside +`replay --all` and the gate; CI still never builds a seed ([decision 100](0100-every-probe-is-shown-to-fail-against-a-seeded-defect.md)). + +## Evidence + +On 2026-10-17 the citation rule read 3 refused entries, 41 not-promised entries, 11 obligation declarations and 87 +harness and probe sources, and found 4 citations: KB-001 and KB-027 by not-promised entries, KB-037 twice by the +harness; every one is executable. Each kind was seeded in a scratch copy of the repository and failed the check: a +cited fact's `test:` line removed, one pointed at a missing file, and a prose fact (KB-005) cited from a refused entry, +an obligation declaration and the harness. A wrong answer seeded into KB001's and KB037's fixtures failed +`KB001` and `KB037`. Each rule of the seeds selection was seeded in the prober (identity, footprint, seeded file, +`--all`, a missed seed's record, a missing footprint, never skipping) and failed its `SeedsTests`. + +Rejected: a fixture kept in `design/knowledge/` beside the entry, which no build copies and the facts project would +reach by path. + +## Consequences + +- A triage that settles a row on a framework fact writes the test first, then cites it; a prose fact that would + explain a row is research to do, not a citation. +- `prober seeds` after a probe or product change runs seconds to minutes instead of one build per seed of every probe. +- Wiring `Baseline.Fact` to the fixtures is the harness's next step; this record only fixes the rule it follows. A + probe's identity does not yet include the facts it cites, so a changed fixture does not stale the probes reading it + until that step adds it. diff --git a/design/adr/README.md b/design/adr/README.md index 3fd67f44..f1769041 100644 --- a/design/adr/README.md +++ b/design/adr/README.md @@ -99,17 +99,18 @@ per area ([design](../plan.md)) link to the records they rest on. | [0090](0090-faults-alike-are-a-warning-until-3-0.md) | Two fault types writing one name are a startup warning until 3.0.0 | 2026-10-14 | accepted | | [0091](0091-an-exclusion-narrows-a-wider-status.md) | An endpoint's exclusion narrows a wider scope's status; a status every endpoint leaves out is refused | 2026-10-14 | accepted | | [0092](0092-repository-tooling-is-dotnet-and-ci-needs-no-ai.md) | Repository tooling is .NET, a file-based app or a tool project by how it is used; git and the pinned SDK are the only prerequisites; the tests run on Linux, Windows and macOS; CI never depends on an AI | 2026-10-15 | accepted; session-start builds by [0105](0105-every-tool-is-built-in-the-session-s-checkout.md) | -| [0093](0093-verified-facts-are-pinned-dated-and-executable-where-they-can-be.md) | Verified facts are pinned, dated and executable where they can be: one fact per file in `design/knowledge/`, a characterization test per observable fact, and `docs-query check` holding the pins to what is installed | 2026-10-15 | accepted | +| [0093](0093-verified-facts-are-pinned-dated-and-executable-where-they-can-be.md) | Verified facts are pinned, dated and executable where they can be: one fact per file in `design/knowledge/`, a characterization test per observable fact, and `docs-query check` holding the pins to what is installed | 2026-10-15 | accepted; a cited fact is executable and an answer is a fixture beside the test by [0107](0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md) | | [0094](0094-topic-branches-into-a-release-branch-and-fix-trailers.md) | From 2.2.0, topic branches merge into a release branch with merge commits only, and every `fix:` commit carries `Defect-Class:` and `Found-By:` trailers, held by a test | 2026-10-15 | accepted | | [0095](0095-a-fix-is-tested-across-its-input-class.md) | A fix is tested across its input class: the fixer names the class, runs the fix's test over that class's variant catalogue (`TypeArgumentSites` for a type argument, the `shape.*` axes, the settings axes) and fixes the siblings it finds; `Found-By: VA` | 2026-10-15 | accepted | | [0096](0096-ci-replays-every-probe-and-the-release-runs-the-gate.md) | CI replays every probe, from an empty state, on every push and pull request, and a failing probe fails the run; the release replays them and runs the gate before it packs, the gate advisory; held by `ReleaseWiringTests` | 2026-10-15 | accepted; the gate binds from 2.2.0 by [0097](0097-2-2-0-is-a-consolidation-release-and-the-gate-binds.md); the build workflow's replay runs by family and footprint by [0101](0101-ci-replays-by-family-and-footprint.md) | | [0097](0097-2-2-0-is-a-consolidation-release-and-the-gate-binds.md) | 2.2.0 is a consolidation release: no public API, its additive items move to 2.3.0, and the probe gate binds from it | 2026-10-15 | accepted | | [0098](0098-a-null-passes-no-comparison-in-a-query.md) | A nullable record-class value object compared with `<`, `<=`, `>` or `>=` in a query answers as SQL does, leaving out its null rows; documented, not refused | 2026-10-15 | accepted | | [0099](0099-a-probe-row-says-how-it-ended.md) | A probe row says how it ended: notes are a closed catalogue of keys and outcomes, a row that earned nothing is a skipped test labelled with its settings, a skip for a finding fails the replay once the finding is fixed and the gate while it stands, and a refused document is judged against ASP.NET alone | 2026-10-15 | accepted; note keys are a typed catalogue the compiler checks by [0103](0103-a-harness-error-is-never-a-catch-and-note-keys-are-constants.md) | -| [0100](0100-every-probe-is-shown-to-fail-against-a-seeded-defect.md) | Every registered probe names a seeded product defect, and `prober seeds` (local, a build per seed) requires a violation it causes; the prober tests hold that every seed exists and applies; pairwise coverage is a floor, three-way combinations belong to the variant catalogue | 2026-10-15 | accepted; a seed per obligation of a claim by [0104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md) | +| [0100](0100-every-probe-is-shown-to-fail-against-a-seeded-defect.md) | Every registered probe names a seeded product defect, and `prober seeds` (local, a build per seed) requires a violation it causes; the prober tests hold that every seed exists and applies; pairwise coverage is a floor, three-way combinations belong to the variant catalogue | 2026-10-15 | accepted; a seed per obligation of a claim by [0104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md); seeds run by footprint, a release `--all`, by [0107](0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md) | | [0101](0101-ci-replays-by-family-and-footprint.md) | The build workflow replays the probes in legs by family, each running its probes in parallel, under a time limit: a push every probe with coverage and the merged state saved, a pull request only the probes it stales against its base branch's saved state and without a footprint; `prober merge` joins the legs and refuses a probe no leg saw | 2026-10-16 | accepted | | [0102](0102-an-exclusion-carries-its-reason.md) | Each `exclude` entry of claims.json is a required one-sentence reason and its pairs; the prober refuses a claims file its schema does not match, naming the claim and the field, so an invented reason field is refused; `prober universe` prints each exclusion's reason beside the claim's narrowings; the prober writes JSON with only JSON's own escapes, so an unchanged publish leaves a tracked file as it is | 2026-10-16 | accepted | | [0103](0103-a-harness-error-is-never-a-catch-and-note-keys-are-constants.md) | A harness error is never a catch: `Rec.Run` ends a row whose check throws as a harness error, not a violation, and `prober seeds` credits only a violation no harness error raised; note keys are members of a typed `NoteKeys` the compiler checks, and `register`, `publish` and a conventions test refuse a note or a skip whose key or finding is not written out | 2026-10-16 | accepted | | [0104](0104-a-claim-states-its-obligations-and-each-has-a-seed.md) | A claim lists its `obligations`, each a sentence quoted verbatim from its source, and what it does not promise in `not-promised`, each quoted from the document or record that makes it intended; a claim without the field has one obligation, `claim`; a probe carries a seed per obligation it checks, and the check without a build refuses a seed naming an unlisted obligation and a listed obligation no seed names; `status` shows each claim's obligations, and `prober gaps` reports what the corpus leaves out, deciding nothing | 2026-10-16 | accepted | | [0105](0105-every-tool-is-built-in-the-session-s-checkout.md) | SessionStart builds the guard in `$CLAUDE_PROJECT_DIR` and every other tool, through `tools/AgentTools.slnf`, in the session's own checkout; a hook, the prober, the smoke test and the verdict fall back to the SDK installer's `dotnet` when the one on PATH lacks the pinned SDK, and a session start that builds nothing tells the agent; the guard lets `dotnet msbuild -getProperty/-getItem` without a target through | 2026-10-17 | accepted | | [0106](0106-a-parameter-of-a-value-object-over-a-described-type-is-inlined.md) | A value object over a type the JSON options describe is written inline as a route, query, header or form parameter and as an element of a query or header array, where value objects are components: the component is its JSON positions'; every other value object keeps its reference; the schema below a parameter is asked for with the parameter's source and type alone | 2026-10-17 | accepted | +| [0107](0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md) | A fact that prunes or excuses a row is executable: `docs-query check` fails `fact-not-executable-cited` where a refused entry, a not-promised entry, an obligation's declaration or the harness cites a fact without its test; a fact's measured answer is `KBNNN.fixture.json` beside its test, named by `fixture:`, read by a BCL-only reader the harness will use; `prober seeds` runs only the probes whose seeds identity or product footprint changed since every seed was caught, and a release runs `--all` | 2026-10-17 | accepted | diff --git a/design/knowledge/KB-001-openapi-2-nullable-body-drops-pattern.md b/design/knowledge/KB-001-openapi-2-nullable-body-drops-pattern.md index 1adb50f4..347d7641 100644 --- a/design/knowledge/KB-001-openapi-2-nullable-body-drops-pattern.md +++ b/design/knowledge/KB-001-openapi-2-nullable-body-drops-pattern.md @@ -6,6 +6,7 @@ pins: source: the document Microsoft.AspNetCore.OpenApi writes for an MVC [FromBody] int and int?, and a minimal-API [FromBody] enum?, under OpenApiSpecVersion.OpenApi2_0 and OpenApi3_1, measured in a host without TypeKit turned-on: the probe finding G1, dismissed as no TypeKit bug (tools/probe/corpus/dismissed.jsonl), and the probe harness comparing a nullable value object with the nullable wrapped type test: test/CodoMetis.TypeKit.Facts/KB001.cs +fixture: test/CodoMetis.TypeKit.Facts/KB001.fixture.json --- # KB-001. Under OpenAPI 2.0, ASP.NET writes a nullable request body as `allOf: [{x-nullable: true}]` and drops its `pattern` diff --git a/design/knowledge/KB-026-openapi-2-nullable-component-property-loses-its-reference.md b/design/knowledge/KB-026-openapi-2-nullable-component-property-loses-its-reference.md index 97a8bb15..c7319fbe 100644 --- a/design/knowledge/KB-026-openapi-2-nullable-component-property-loses-its-reference.md +++ b/design/knowledge/KB-026-openapi-2-nullable-component-property-loses-its-reference.md @@ -6,6 +6,7 @@ pins: source: the document Microsoft.AspNetCore.OpenApi writes for an MVC [FromBody] record with a property of a record struct and one of its Nullable, under OpenApiSpecVersion.OpenApi2_0 and OpenApi3_1, measured in a host without TypeKit turned-on: row 38 of C21OpenapiSchemaParityProbe (claim C21-openapi-schema-parity), where a nullable value-object property that is a component had no type under OpenAPI 2.0 test: test/CodoMetis.TypeKit.Facts/KB026.cs +fixture: test/CodoMetis.TypeKit.Facts/KB026.fixture.json --- # KB-026. Under OpenAPI 2.0, ASP.NET writes a nullable property of a component as `allOf: [{x-nullable: true}]`, without the reference diff --git a/design/knowledge/KB-027-an-enum-component-is-its-first-use.md b/design/knowledge/KB-027-an-enum-component-is-its-first-use.md index 0a4e2e85..ecc69392 100644 --- a/design/knowledge/KB-027-an-enum-component-is-its-first-use.md +++ b/design/knowledge/KB-027-an-enum-component-is-its-first-use.md @@ -6,6 +6,7 @@ pins: source: the document Microsoft.AspNetCore.OpenApi writes for minimal API header parameters of two enums written by name (JsonStringEnumConverter), each used once nullable and once not, in opposite orders, under OpenApiSpecVersion.OpenApi3_1 and OpenApi3_0, measured in a host without TypeKit turned-on: row 285 of C21OpenapiSchemaParityProbe (claim C21-openapi-schema-parity, finding F3), where the wrapped E? header's enum listed null through E's component and the value object's did not test: test/CodoMetis.TypeKit.Facts/KB027.cs +fixture: test/CodoMetis.TypeKit.Facts/KB027.fixture.json --- # KB-027. The component ASP.NET makes for an enum written by name is the schema of the first use it meets diff --git a/design/knowledge/KB-037-mvc-under-max-validation-depth-1-runs-an-unbound-action.md b/design/knowledge/KB-037-mvc-under-max-validation-depth-1-runs-an-unbound-action.md index 2c6f62e6..e21341d9 100644 --- a/design/knowledge/KB-037-mvc-under-max-validation-depth-1-runs-an-unbound-action.md +++ b/design/knowledge/KB-037-mvc-under-max-validation-depth-1-runs-an-unbound-action.md @@ -5,6 +5,7 @@ pins: source: measured in a host without TypeKit (Microsoft.AspNetCore.App 10.0 of the pinned SDK, an [ApiController] with an int parameter per binding source); the API reference of MvcOptions.MaxValidationDepth says only that "ValidationVisitor will throw if traversing an object exceeds the maximum allowed validation depth" and that "The default value is 32", nothing of binding errors turned-on: 28 rows of BaseProbe (claim C07-fault-responses-blanket) under mvc.MaxValidationDepth=1, where a validated value object answered a value its Create refuses with 200 holding its default (WS2 step 4 triage) test: test/CodoMetis.TypeKit.Facts/KB037.cs +fixture: test/CodoMetis.TypeKit.Facts/KB037.fixture.json --- # KB-037. Under `MaxValidationDepth = 1`, MVC runs an action whose parameter it could not bind, with the parameter at its default diff --git a/design/knowledge/README.md b/design/knowledge/README.md index 7698b2df..ee4c6bd9 100644 --- a/design/knowledge/README.md +++ b/design/knowledge/README.md @@ -15,6 +15,18 @@ falsifiable, checked against a primary source. A competitor's feature is a fact `external {name} {version}`: no check compares it, so it is as fresh as its date. Take the next number, copy an entry's frontmatter, write the test first where a test can observe it, and add a row below. +**A fact that prunes or excuses is executable** ([decision 107](../adr/0107-a-cited-fact-is-executable-and-seeds-run-by-footprint.md)). +A `refused` entry of the axes file, a `not-promised` entry, an obligation's declaration and the probe harness may +cite only a fact with a test; `docs-query check` fails `fact-not-executable-cited` otherwise. A fact without one +explains, and decides no row. + +**A measured answer that is data is a fixture.** Where the fact's answer is a document, a status or an exception at an +operation, it is `test/CodoMetis.TypeKit.Facts/KBNNN.fixture.json` beside the test, named by a `fixture:` line after +`test:`: the fact's id and its `answers`, each an `id`, a `kind` (`document` with `at`, a JSON pointer; `status`; +`exception`), the settings it was measured under (`when`, axis to value as a row names them), the `value` and a +`note`. The test is a theory over the fixture's answers, measuring each (`KB001`), and `FactFixture` reads it, for the +test and later for the harness's `Baseline.Fact`. + | Id | Fact | Pins | Test | |---|---|---|---| | [KB-001](KB-001-openapi-2-nullable-body-drops-pattern.md) | Under OpenAPI 2.0, ASP.NET writes a nullable request body as `allOf: [{x-nullable: true}]` and drops its `pattern` | OpenApi 10.0.12, SDK 10.0.4xx | `KB001` | diff --git a/test/CodoMetis.TypeKit.DocsQuery.Tests/FactCitationTests.cs b/test/CodoMetis.TypeKit.DocsQuery.Tests/FactCitationTests.cs new file mode 100644 index 00000000..6ca113d4 --- /dev/null +++ b/test/CodoMetis.TypeKit.DocsQuery.Tests/FactCitationTests.cs @@ -0,0 +1,216 @@ +using DocsQuery; + +namespace CodoMetis.TypeKit.DocsQuery.Tests; + +/// +/// A fact is executable or it explains only (design/adr/0107): check fails a fact the probe corpus cites to +/// prune or excuse a row (a refused entry of the axes file, a not-promised entry or an obligation's declaration in +/// claims.json, a Cite.Fact or Baseline.Fact of the harness) when it has no test, and a fixture: +/// whose file is missing or not beside the test. +/// +public sealed class FactCitationTests : IDisposable +{ + private readonly TempRepo _repo = Store(new TempRepo()); + + public void Dispose() => _repo.Dispose(); + + private static string Fact(int number, string test, string fixture = "") => + $""" + --- + verified: 2026-10-15 + pins: + - sdk 10.0.4xx + source: the framework's own output, measured + turned-on: decision 1 + test: {test} + {fixture} + --- + + # KB-{number:D3}. A fact + """; + + /// KB-001 tested, with a fixture; KB-002 prose only. + private static TempRepo Store(TempRepo repo) => repo + .With("global.json", """{ "sdk": { "version": "10.0.402", "rollForward": "latestPatch" } }""") + .With("design/adr/0001-first.md", "# 0001. First\n\nDate: 2026-10-01\nStatus: accepted\n") + .With("design/knowledge/KB-001-tested.md", Fact(1, "test/CodoMetis.TypeKit.Facts/KB001.cs", "fixture: test/CodoMetis.TypeKit.Facts/KB001.fixture.json")) + .With("test/CodoMetis.TypeKit.Facts/KB001.cs", "public sealed class KB001 { }\n") + .With("test/CodoMetis.TypeKit.Facts/KB001.fixture.json", """{ "fact": "KB-001", "answers": [] }""") + .With("design/knowledge/KB-002-prose.md", Fact(2, "none, a sentence of the framework's documentation")); + + /// The axes file, its refused entries citing in their reason. + private TempRepo Axes(string fact) => _repo.With(".claude/probe-axes.md", + $$""" + # Axes + + ```json probe-axes + { + "axes": {}, + "refused": { + "json.ReadCommentHandling": { + "Allow": "System.Text.Json refuses it on the options object" + }, + "mvc.MaxValidationDepth": { + "0": "MVC refuses it, fact:{{fact}}" + } + } + } + ``` + """); + + /// claims.json with a not-promised entry whose source is and an obligation declaration citing . + private TempRepo Claims(string source, string cite) => _repo.With("tools/probe/corpus/claims.json", + $$""" + [ + { + "id": "C01", + "source": "README.md#a", + "obligations": [ + { + "id": "binds", + "sentence": "It binds.", + "inapplicable": [ + { "axes": { "x": ["1"] }, "cite": "not-promised:np" }, + { "axes": { "x": ["2"] }, "cite": "{{cite}}" } + ] + } + ], + "not-promised": [ + { "id": "np", "source": "{{source}}", "sentence": "So.", "axes": { "x": ["1"] } } + ] + } + ] + """); + + private List Problems() => + [.. _repo.Queries().Check().Rows.Select(row => $"{row.Fields[0].Value} {row.Fields[1].Value} {row.Fields[2].Value}")]; + + [Fact] + public void Citations_of_a_tested_fact_check_clean_and_are_counted_by_kind() + { + Axes("KB-001"); + Claims("design/knowledge/KB-001-tested.md#kb-001-a-fact", "fact:KB-001"); + _repo.With("tools/probe/corpus/harness/Http.cs", "class Http { void M() { c.Inapplicable(Cite.Fact(\"KB-001\"), \"why\"); } }\n") + .With("tools/probe/corpus/probes/P.cs", "class P { Baseline b = Baseline.Fact( \"KB-001\" ); }\n"); + + var answer = _repo.Queries().Check(); + + answer.Rows.ShouldBeEmpty(answer.Text()); + answer.Summary.ShouldContain("5 fact citations (1 refused, 1 not-promised, 1 obligation, 2 harness)"); + var scan = FactCitations.Find(_repo.Root); + scan.Read.ShouldBe(new Dictionary { ["refused"] = 2, ["not-promised"] = 1, ["obligation"] = 2, ["harness"] = 2 }); + scan.Citations.Select(c => $"{c.Kind} {c.At} {c.Fact} {c.Baseline}").ShouldBe( + [ + "refused .claude/probe-axes.md:11 KB-001 False", + "obligation tools/probe/corpus/claims.json:11 KB-001 False", + "not-promised tools/probe/corpus/claims.json:16 KB-001 False", + "harness tools/probe/corpus/harness/Http.cs:1 KB-001 False", + "harness tools/probe/corpus/probes/P.cs:1 KB-001 True", + ]); + } + + [Fact] + public void A_refused_entry_citing_a_fact_without_a_test_fails_where_it_is_written() + { + Axes("KB-002"); + + Problems().ShouldBe([".claude/probe-axes.md:11 fact-not-executable-cited KB-002"]); + _repo.Queries().Check().Rows.Single().Fields.Single(f => f.Name == "detail").Value + .ShouldBe("the refused entry mvc.MaxValidationDepth=0 cites KB-002, and its test: is 'none, a sentence of the framework's documentation': " + + "a fact without its test explains only, never prunes or excuses a row; write its test in test/CodoMetis.TypeKit.Facts/, or cite a docs sentence"); + } + + [Fact] + public void A_not_promised_entry_whose_source_is_a_fact_without_a_test_fails() + { + Claims("design/knowledge/KB-002-prose.md#kb-002-a-fact", "fact:KB-001"); + + Problems().ShouldBe(["tools/probe/corpus/claims.json:16 fact-not-executable-cited KB-002"]); + } + + [Fact] + public void An_obligation_s_declaration_citing_a_fact_without_a_test_fails() + { + Claims("README.md#a", "fact:KB-002"); + + Problems().ShouldBe(["tools/probe/corpus/claims.json:11 fact-not-executable-cited KB-002"]); + } + + [Fact] + public void A_harness_citation_of_a_fact_without_a_test_fails_and_one_in_a_comment_does_not_count() + { + _repo.With("tools/probe/corpus/harness/Http.cs", + """ + class Http + { + // Cite.Fact("KB-002") is only mentioned here + /* and Cite.Fact("KB-002") here */ + string url = "http://example"; object o = Cite.Fact("KB-002"); + void M() => c.Inapplicable(Cite.Fact("KB-002"), "why"); + } + """); + + Problems().ShouldBe(["tools/probe/corpus/harness/Http.cs:5 fact-not-executable-cited KB-002", "tools/probe/corpus/harness/Http.cs:6 fact-not-executable-cited KB-002"]); + } + + [Fact] + public void A_cited_fact_whose_test_file_is_missing_fails_twice() + { + _repo.With("design/knowledge/KB-003-test-gone.md", Fact(3, "test/CodoMetis.TypeKit.Facts/KB003.cs")) + .With("tools/probe/corpus/probes/P.cs", "class P { object o = Cite.Fact(\"KB-003\"); }\n"); + + Problems().ShouldBe( + [ + "design/knowledge/KB-003-test-gone.md:7 fact-untested test/CodoMetis.TypeKit.Facts/KB003.cs", + "tools/probe/corpus/probes/P.cs:1 fact-not-executable-cited KB-003", + ]); + } + + [Fact] + public void A_cited_fact_that_does_not_exist_fails() + { + _repo.With("tools/probe/corpus/probes/P.cs", "class P { object o = Cite.Fact(\"KB-099\"); }\n"); + + Problems().ShouldBe(["tools/probe/corpus/probes/P.cs:1 fact-not-executable-cited KB-099"]); + } + + /// A baseline is the fact's measured answer, so it takes the fixture as well as the test. + [Fact] + public void A_baseline_of_a_tested_fact_without_a_fixture_fails() + { + _repo.With("design/knowledge/KB-004-no-fixture.md", Fact(4, "test/CodoMetis.TypeKit.Facts/KB004.cs")) + .With("test/CodoMetis.TypeKit.Facts/KB004.cs", "public sealed class KB004 { }\n") + .With("tools/probe/corpus/harness/Http.cs", "class Http { Baseline a = Baseline.Fact(\"KB-001\"), b = Baseline.Fact(\"KB-004\"); object c = Cite.Fact(\"KB-004\"); }\n"); + + Problems().ShouldBe(["tools/probe/corpus/harness/Http.cs:1 fact-not-executable-cited KB-004"]); + } + + [Fact] + public void A_fixture_must_exist_beside_the_test_of_a_tested_fact() + { + _repo.With("design/knowledge/KB-005-fixture-gone.md", Fact(5, "test/CodoMetis.TypeKit.Facts/KB005.cs", "fixture: test/CodoMetis.TypeKit.Facts/KB005.fixture.json")) + .With("test/CodoMetis.TypeKit.Facts/KB005.cs", "public sealed class KB005 { }\n") + .With("design/knowledge/KB-006-fixture-elsewhere.md", Fact(6, "test/CodoMetis.TypeKit.Facts/KB006.cs", "fixture: test/fixtures/KB006.json")) + .With("test/CodoMetis.TypeKit.Facts/KB006.cs", "public sealed class KB006 { }\n") + .With("test/fixtures/KB006.json", "{}") + .With("design/knowledge/KB-007-fixture-untested.md", Fact(7, "none, prose", "fixture: test/CodoMetis.TypeKit.Facts/KB007.fixture.json")) + .With("test/CodoMetis.TypeKit.Facts/KB007.fixture.json", "{}") + .With("test/CodoMetis.TypeKit.Facts/KB008.fixture.json", "{}"); + + Problems().ShouldBe( + [ + "design/knowledge/KB-005-fixture-gone.md:8 fact-fixture-missing test/CodoMetis.TypeKit.Facts/KB005.fixture.json", + "design/knowledge/KB-006-fixture-elsewhere.md:8 fact-fixture test/fixtures/KB006.json", + "design/knowledge/KB-007-fixture-untested.md:8 fact-fixture test/CodoMetis.TypeKit.Facts/KB007.fixture.json", + "test/CodoMetis.TypeKit.Facts/KB008.fixture.json:1 fact-orphan-fixture test/CodoMetis.TypeKit.Facts/KB008.fixture.json", + ]); + } + + [Fact] + public void Comments_are_blanked_and_strings_kept() + { + var source = "a // b\nc /* d\ne */ f \"// g\" @\"h \"\" // i\" \"\"\"raw // j\"\"\" '\"' k"; + + FactCitations.WithoutComments(source).ShouldBe("a \nc \n f \"// g\" @\"h \"\" // i\" \"\"\"raw // j\"\"\" '\"' k"); + } +} diff --git a/test/CodoMetis.TypeKit.DocsQuery.Tests/RepositoryTests.cs b/test/CodoMetis.TypeKit.DocsQuery.Tests/RepositoryTests.cs index 87616598..fd61d078 100644 --- a/test/CodoMetis.TypeKit.DocsQuery.Tests/RepositoryTests.cs +++ b/test/CodoMetis.TypeKit.DocsQuery.Tests/RepositoryTests.cs @@ -58,6 +58,25 @@ public void The_repository_checks_clean() answer.Rows.ShouldBeEmpty(answer.Text()); } + /// + /// Floors on the places the citation rule reads and on the citations it finds there (design/adr/0107), so that a + /// scan that found nothing cannot let the check above pass. On 2026-10-17: 3 refused entries, 41 not-promised + /// entries, 11 obligation declarations and 87 harness and probe sources were read, and KB-001 and KB-027 were + /// cited by not-promised entries, KB-037 twice by the harness. + /// + [Theory] + [InlineData(FactCitations.Refused, 3, 0)] + [InlineData(FactCitations.NotPromised, 30, 2)] + [InlineData(FactCitations.Obligation, 10, 0)] + [InlineData(FactCitations.Harness, 40, 2)] + public void The_corpus_s_citations_of_facts_are_found(string kind, int read, int cited) + { + var scan = FactCitations.Find(RepositoryRoot.Path); + + scan.Read[kind].ShouldBeGreaterThanOrEqualTo(read); + scan.Of(kind).ShouldBeGreaterThanOrEqualTo(cited); + } + [Fact] public void Every_record_is_listed_with_its_status_and_date() => Queries.List(["kind=adr"]).Rows.ShouldAllBe(row => row.Fields.Single(f => f.Name == "status").Value.Length > 0 && row.Fields.Single(f => f.Name == "date").Value.Length == 10); diff --git a/test/CodoMetis.TypeKit.Facts/CodoMetis.TypeKit.Facts.csproj b/test/CodoMetis.TypeKit.Facts/CodoMetis.TypeKit.Facts.csproj index 84bdc895..52527faf 100644 --- a/test/CodoMetis.TypeKit.Facts/CodoMetis.TypeKit.Facts.csproj +++ b/test/CodoMetis.TypeKit.Facts/CodoMetis.TypeKit.Facts.csproj @@ -26,6 +26,11 @@ + + + + + diff --git a/test/CodoMetis.TypeKit.Facts/FactFixture.cs b/test/CodoMetis.TypeKit.Facts/FactFixture.cs new file mode 100644 index 00000000..e3bbe587 --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/FactFixture.cs @@ -0,0 +1,191 @@ +using System.Text.Json; +using System.Text.Json.Nodes; + +namespace CodoMetis.TypeKit.Facts; + +/// +/// A fact's measured answers as data (design/adr/0107): KBNNN.fixture.json beside the fact's test KBNNN.cs, +/// named by the fact's fixture: line. The test measures what the framework does and asserts it is still each +/// answer of the fixture; the probe harness reads the same answers as a baseline, Baseline.Fact("KB-NNN"), in +/// place of starting a host for the framework's side. +/// +/// +/// The rule the harness follows when it reads a fixture as Baseline.Fact: +/// +/// +/// It reads only a fixture that a fact's fixture: line names, which docs-query check allows only for +/// a fact with a test (fact-fixture, fact-not-executable-cited), and only while the check is clean, so a +/// fact whose pins fell behind what is installed (fact-behind) is no baseline until its test passed again. +/// It takes the one answer that the row: every setting the answer's when +/// names is the row's, axis for axis (openapi.OpenApiVersion, http.binding, ...). A setting the row does +/// not have is not covered. No answer covers it: the fact says nothing about this row, and the harness measures the +/// framework's side itself. More than one: the fixture is ambiguous, and the row ends as a harness error. +/// It compares by the answer's kind: a document at its at (a JSON pointer) byte for byte in its +/// compact form, a status and, where the answer has one, its body, or an exception by its type and a part +/// of its message, at the operation and stage the answer names ( and its +/// overloads). A mismatch is the product's difference from the framework, as a twin's answer would be. +/// It never writes a fixture. An answer is changed by changing the fact: its test, its fixture and its entry, in +/// one commit. +/// +/// +/// BCL only and nothing of the test framework, so that the harness can compile this file as it is. +/// +public sealed class FactFixture +{ + public const string Suffix = ".fixture.json"; + + private FactFixture(string fact, IReadOnlyList answers) => (Fact, Answers) = (fact, answers); + + /// The fact's id, KB-001. + public string Fact { get; } + + public IReadOnlyList Answers { get; } + + /// The answer of that id; an id the fixture does not have throws. + public FactAnswer this[string id] => + Answers.FirstOrDefault(answer => answer.Id == id) ?? throw new KeyNotFoundException($"{Fact}'s fixture has no answer '{id}'"); + + /// The file of a fact's fixture: KB001.fixture.json for KB-001. + public static string FileName(string fact) => fact.Replace("-", "", StringComparison.Ordinal) + Suffix; + + /// The fixture as the facts project's build copies it beside its tests. + public static FactFixture Beside(string fact) => Load(Path.Combine(AppContext.BaseDirectory, FileName(fact))); + + /// The one answer that covers a row's settings, or null where none does; more than one throws. + public FactAnswer? For(IReadOnlyDictionary settings) + { + var covering = Answers.Where(answer => answer.Covers(settings)).ToList(); + return covering.Count <= 1 ? covering.SingleOrDefault() + : throw new InvalidOperationException($"{Fact}'s fixture is ambiguous for these settings: {string.Join(", ", covering.Select(answer => answer.Id))} each cover them"); + } + + /// Reads and checks a fixture: its fact is the one its file is named for, and each answer has an id of its own and the fields its kind needs. + public static FactFixture Load(string path) + { + var name = Path.GetFileName(path); + if (JsonNode.Parse(File.ReadAllText(path)) is not JsonObject root) throw Invalid(name, "is not a JSON object"); + + var fact = (string?)root["fact"] ?? throw Invalid(name, "names no \"fact\""); + if (FileName(fact) != name) throw Invalid(name, $"names the fact {fact}, whose fixture is {FileName(fact)}"); + + var answers = new List(); + foreach (var node in root["answers"] as JsonArray ?? throw Invalid(name, "has no \"answers\" array")) + { + if (node is not JsonObject entry) throw Invalid(name, "has an answer that is not an object"); + + var id = (string?)entry["id"] ?? throw Invalid(name, "has an answer without an \"id\""); + var kind = (string?)entry["kind"] ?? throw Invalid(name, $"answer {id} has no \"kind\""); + var when = (entry["when"] as JsonObject ?? []).ToDictionary(kv => kv.Key, kv => (string?)kv.Value ?? throw Invalid(name, $"answer {id}'s when {kv.Key} is no string"), StringComparer.Ordinal); + var at = (string?)entry["at"]; + var value = entry["value"] ?? throw Invalid(name, $"answer {id} has no \"value\""); + + if (answers.Any(answer => answer.Id == id)) throw Invalid(name, $"lists the answer {id} twice"); + switch (kind) + { + case FactAnswer.Document when at is null || (at.Length > 0 && at[0] != '/'): + throw Invalid(name, $"answer {id} is a document and needs \"at\", a JSON pointer such as /paths/~1a/get"); + case FactAnswer.Status when value["status"] is not JsonValue status || !status.TryGetValue(out _): + throw Invalid(name, $"answer {id} is a status and needs a \"status\" number"); + case FactAnswer.Exception when (string?)value["type"] is null: + throw Invalid(name, $"answer {id} is an exception and needs its \"type\""); + case FactAnswer.Document or FactAnswer.Status or FactAnswer.Exception: + break; + default: + throw Invalid(name, $"answer {id}'s kind '{kind}' is none of document, status and exception"); + } + + answers.Add(new FactAnswer(id, kind, when, at, value.DeepClone(), (string?)entry["note"])); + } + + return new FactFixture(fact, answers); + } + + private static InvalidDataException Invalid(string name, string why) => new($"{name} {why}"); +} + +/// +/// One measured answer of a fact: what the framework gave () for the settings in , +/// of one kind, document, status or exception, and a note saying what to read in it. +/// +public sealed class FactAnswer +{ + public const string Document = "document", Status = "status", Exception = "exception"; + + internal FactAnswer(string id, string kind, IReadOnlyDictionary when, string? at, JsonNode value, string? note) => + (Id, Kind, When, At, Value, Note) = (id, kind, when, at, value, note); + + public string Id { get; } + + public string Kind { get; } + + /// The settings the answer was measured under, axis to value, as a row names them. + public IReadOnlyDictionary When { get; } + + /// For a document, the JSON pointer of the part the answer is (RFC 6901: ~1 for /, ~0 for ~). + public string? At { get; } + + public JsonNode Value { get; } + + public string? Note { get; } + + /// Every setting the answer names is the row's. + public bool Covers(IReadOnlyDictionary settings) => + When.All(setting => settings.TryGetValue(setting.Key, out var value) && value == setting.Value); + + /// A document's difference from the answer at its pointer, or null where it is the same, byte for byte compact. + public string? Mismatch(JsonNode document) + { + Expect(Document); + var part = Resolve(document, At!); + var measured = part?.ToJsonString() ?? "(nothing)"; + var expected = Value.ToJsonString(); + return measured == expected ? null : $"{Id} at {At}: measured {measured}, the fixture says {expected}{Noted}"; + } + + /// A response's difference from the answer, or null: its status, and its body where the answer has one. + public string? Mismatch(int status, string body) + { + Expect(Status); + var expected = (int)Value["status"]!; + if (status != expected) return $"{Id}: measured status {status}, the fixture says {expected}{Noted}"; + return (string?)Value["body"] is { } text && text != body ? $"{Id}: measured body {body}, the fixture says {text}{Noted}" : null; + } + + /// A thrown exception's difference from the answer, or null: its type's full name, and a part of its message where the answer has one. + public string? Mismatch(System.Exception? thrown) + { + Expect(Exception); + var type = (string)Value["type"]!; + if (thrown is null) return $"{Id}: nothing was thrown, the fixture says {type} at {Value["operation"]}, {Value["stage"]}{Noted}"; + if (thrown.GetType().FullName != type) return $"{Id}: measured {thrown.GetType().FullName}, the fixture says {type}{Noted}"; + return (string?)Value["message"] is { } part && !thrown.Message.Contains(part, StringComparison.Ordinal) + ? $"{Id}: the message '{thrown.Message}' does not say '{part}'{Noted}" + : null; + } + + /// The node a JSON pointer names, or null where the document has none. + public static JsonNode? Resolve(JsonNode document, string pointer) + { + var node = (JsonNode?)document; + foreach (var token in pointer.Length == 0 ? [] : pointer[1..].Split('/')) + { + var key = token.Replace("~1", "/", StringComparison.Ordinal).Replace("~0", "~", StringComparison.Ordinal); + node = node switch + { + JsonObject o => o[key], + JsonArray a when int.TryParse(key, System.Globalization.NumberStyles.None, System.Globalization.CultureInfo.InvariantCulture, out var i) && i < a.Count => a[i], + _ => null, + }; + if (node is null) return null; + } + + return node; + } + + private string Noted => Note is null ? "" : $" ({Note})"; + + private void Expect(string kind) + { + if (Kind != kind) throw new InvalidOperationException($"{Id} is a {Kind}, not a {kind}"); + } +} diff --git a/test/CodoMetis.TypeKit.Facts/FixtureTests.cs b/test/CodoMetis.TypeKit.Facts/FixtureTests.cs new file mode 100644 index 00000000..c5b146aa --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/FixtureTests.cs @@ -0,0 +1,97 @@ +using System.Reflection; + +namespace CodoMetis.TypeKit.Facts; + +/// +/// The fixtures beside the facts' tests (design/adr/0107): each one the build copies loads, names the fact its file is +/// named for, and has that fact's test class here; the reader the harness will use covers, compares and refuses as +/// says. +/// +public sealed class FixtureTests +{ + public static TheoryData Fixtures() => + [.. Directory.EnumerateFiles(AppContext.BaseDirectory, "KB*" + FactFixture.Suffix).Select(path => Path.GetFileName(path)).Order(StringComparer.Ordinal)]; + + /// A floor, so the theory below cannot pass on a build that copied no fixture. + [Fact] + public void The_fixtures_are_copied_beside_the_tests() => Fixtures().Count.ShouldBeGreaterThanOrEqualTo(4); + + [Theory, MemberData(nameof(Fixtures))] + public void A_fixture_loads_and_its_fact_has_a_test_here(string file) + { + var fixture = FactFixture.Load(Path.Combine(AppContext.BaseDirectory, file)); + + fixture.Answers.ShouldNotBeEmpty(); + typeof(FixtureTests).Assembly.GetType($"{typeof(FixtureTests).Namespace}.{fixture.Fact.Replace("-", "", StringComparison.Ordinal)}").ShouldNotBeNull(); + } + + private static FactFixture Write(string json) + { + var path = Path.Combine(Path.GetTempPath(), $"fixture-{Guid.NewGuid():N}", "KB099.fixture.json"); + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + File.WriteAllText(path, json); + try { return FactFixture.Load(path); } + finally { Directory.Delete(Path.GetDirectoryName(path)!, recursive: true); } + } + + private const string Two = """ + { "fact": "KB-099", "answers": [ + { "id": "a", "kind": "document", "when": { "x": "1" }, "at": "/p/~1q/0", "value": { "v": [1, null] }, "note": "the note" }, + { "id": "b", "kind": "status", "when": { "x": "2", "y": "1" }, "value": { "status": 200, "body": "0" } }, + { "id": "c", "kind": "exception", "when": { "x": "2" }, "value": { "type": "System.InvalidOperationException", "message": "depth '1'" } } + ] } + """; + + [Fact] + public void An_answer_covers_a_row_that_holds_every_setting_it_names() + { + var fixture = Write(Two); + + fixture.For(new Dictionary { ["x"] = "1", ["z"] = "9" })!.Id.ShouldBe("a"); + fixture.For(new Dictionary { ["x"] = "3" }).ShouldBeNull("no answer covers it: the harness measures the framework itself"); + Should.Throw(() => fixture.For(new Dictionary { ["x"] = "2", ["y"] = "1" })) + .Message.ShouldBe("KB-099's fixture is ambiguous for these settings: b, c each cover them"); + } + + [Fact] + public void Each_kind_compares_what_it_holds() + { + var fixture = Write(Two); + + fixture["a"].Mismatch(JsonNode.Parse("""{ "p": { "/q": [{ "v": [1, null] }] } }""")!).ShouldBeNull(); + fixture["a"].Mismatch(JsonNode.Parse("""{ "p": { "/q": [{ "v": [1] }] } }""")!).ShouldBe("""a at /p/~1q/0: measured {"v":[1]}, the fixture says {"v":[1,null]} (the note)"""); + fixture["a"].Mismatch(JsonNode.Parse("{}")!).ShouldBe("""a at /p/~1q/0: measured (nothing), the fixture says {"v":[1,null]} (the note)"""); + fixture["b"].Mismatch(200, "0").ShouldBeNull(); + fixture["b"].Mismatch(400, "0").ShouldBe("b: measured status 400, the fixture says 200"); + fixture["b"].Mismatch(200, "5").ShouldBe("b: measured body 5, the fixture says 0"); + fixture["c"].Mismatch(new InvalidOperationException("exceeded depth '1' at V")).ShouldBeNull(); + fixture["c"].Mismatch(new ArgumentException("depth '1'")).ShouldBe("c: measured System.ArgumentException, the fixture says System.InvalidOperationException"); + fixture["c"].Mismatch((Exception?)null).ShouldStartWith("c: nothing was thrown"); + Should.Throw(() => fixture["b"].Mismatch(JsonNode.Parse("{}")!)); + } + + [Theory] + [InlineData("""{ "fact": "KB-098", "answers": [] }""", "KB099.fixture.json names the fact KB-098, whose fixture is KB098.fixture.json")] + [InlineData("""{ "fact": "KB-099", "answers": [{ "id": "a", "kind": "document", "value": 1 }] }""", "KB099.fixture.json answer a is a document and needs \"at\", a JSON pointer such as /paths/~1a/get")] + [InlineData("""{ "fact": "KB-099", "answers": [{ "id": "a", "kind": "status", "value": { "status": "200" } }] }""", "KB099.fixture.json answer a is a status and needs a \"status\" number")] + [InlineData("""{ "fact": "KB-099", "answers": [{ "id": "a", "kind": "exception", "value": {} }] }""", "KB099.fixture.json answer a is an exception and needs its \"type\"")] + [InlineData("""{ "fact": "KB-099", "answers": [{ "id": "a", "kind": "guess", "value": 1 }] }""", "KB099.fixture.json answer a's kind 'guess' is none of document, status and exception")] + [InlineData("""{ "fact": "KB-099", "answers": [{ "id": "a", "kind": "status", "value": { "status": 1 } }, { "id": "a", "kind": "status", "value": { "status": 1 } }] }""", "KB099.fixture.json lists the answer a twice")] + public void A_fixture_that_says_less_than_its_kind_needs_is_refused(string json, string message) => + Should.Throw(() => Write(json)).Message.ShouldBe(message); + + /// The reader is meant to be compiled into the probe harness as it is: nothing of the test framework in it. + [Fact] + public void The_reader_needs_only_the_base_class_library() + { + var root = AppContext.BaseDirectory; + while (!File.Exists(Path.Combine(root, "CodoMetis.TypeKit.slnx"))) root = Path.GetDirectoryName(root) ?? throw new InvalidOperationException("no CodoMetis.TypeKit.slnx above the test output"); + + File.ReadAllLines(Path.Combine(root, "test", "CodoMetis.TypeKit.Facts", "FactFixture.cs")) + .Where(line => line.StartsWith("using ", StringComparison.Ordinal)) + .ShouldAllBe(line => line.StartsWith("using System", StringComparison.Ordinal)); + typeof(FactFixture).GetMethods(BindingFlags.Public | BindingFlags.Instance | BindingFlags.Static | BindingFlags.DeclaredOnly) + .SelectMany(method => method.GetParameters().Select(p => p.ParameterType).Append(method.ReturnType)) + .ShouldAllBe(type => type.Namespace!.StartsWith("System", StringComparison.Ordinal) || type.Namespace == typeof(FactFixture).Namespace); + } +} diff --git a/test/CodoMetis.TypeKit.Facts/KB001.cs b/test/CodoMetis.TypeKit.Facts/KB001.cs index 545b43cf..b0085cdf 100644 --- a/test/CodoMetis.TypeKit.Facts/KB001.cs +++ b/test/CodoMetis.TypeKit.Facts/KB001.cs @@ -12,7 +12,9 @@ namespace CodoMetis.TypeKit.Facts; /// public sealed class KB001 { - private const string Int32Pattern = @"""pattern"":""^-?(?:0|[1-9]\\d*)$"""; + private static readonly FactFixture Fixture = FactFixture.Beside("KB-001"); + + public static TheoryData Answers() => [.. Fixture.Answers.Select(answer => answer.Id)]; [ApiController] [Route("kb001")] @@ -25,43 +27,29 @@ public sealed class BodyController : ControllerBase public IActionResult Nullable([FromBody] int? value) => Ok(); } - private static async Task BodyAsync(OpenApiSpecVersion version, string path) - { - var document = await FactHost.DocumentAsync(version, [typeof(BodyController)]); - var operation = document["paths"]![$"/kb001/{path}"]!["post"]!; - - return version == OpenApiSpecVersion.OpenApi2_0 - ? operation["parameters"]!.AsArray().Single(parameter => (string?)parameter!["in"] == "body")! - : operation["requestBody"]!["content"]!["application/json"]!; - } - - [Fact] - public async Task Under_OpenApi_2_0_a_nullable_int_body_is_allOf_x_nullable_without_its_pattern() - { - var plain = await BodyAsync(OpenApiSpecVersion.OpenApi2_0, "plain"); - var nullable = await BodyAsync(OpenApiSpecVersion.OpenApi2_0, "nullable"); - - plain.ToJsonString().ShouldBe("""{"in":"body","name":"body","required":true,"schema":{"format":"int32",""" + Int32Pattern + "}}"); - nullable.ToJsonString().ShouldBe("""{"in":"body","name":"body","schema":{"format":"int32","allOf":[{"x-nullable":true}]}}""", "optional, and no pattern"); - } - public enum Shade { Light, Dark } - [Fact] - public async Task Under_OpenApi_2_0_a_minimal_API_nullable_enum_body_is_allOf_x_nullable_alone() + /// + /// The document an answer was measured in: the MVC host's, with a plain and a nullable int body, or the minimal + /// API's, with a nullable enum body, under the answer's OpenAPI version. + /// + private static Task DocumentAsync(FactAnswer answer) { - var document = await FactHost.DocumentAsync(OpenApiSpecVersion.OpenApi2_0, [], - minimal: app => app.MapPost("/kb001/minimal", ([FromBody] Shade? value) => Results.Ok())); - var body = document["paths"]!["/kb001/minimal"]!["post"]!["parameters"]!.AsArray().Single(parameter => (string?)parameter!["in"] == "body")!; - - body.ToJsonString().ShouldBe("""{"in":"body","name":"body","schema":{"allOf":[{"x-nullable":true}]}}""", "no type, nothing but the null"); + var version = Enum.Parse(answer.When["openapi.OpenApiVersion"]); + return answer.When["http.host"] == "minimal" + ? FactHost.DocumentAsync(version, [], minimal: app => app.MapPost("/kb001/minimal", ([FromBody] Shade? value) => Results.Ok())) + : FactHost.DocumentAsync(version, [typeof(BodyController)]); } - [Fact] - public async Task Under_OpenApi_3_1_the_nullable_body_keeps_its_pattern_inside_oneOf() + /// + /// Each answer of KB001.fixture.json: under OpenAPI 2.0 a nullable body is allOf: [{x-nullable: true}], without + /// the pattern and type its non-nullable form has, in MVC and in minimal APIs alike; under 3.1 it keeps its pattern. + /// + [Theory, MemberData(nameof(Answers))] + public async Task The_framework_writes_the_body_the_fixture_holds(string id) { - var nullable = await BodyAsync(OpenApiSpecVersion.OpenApi3_1, "nullable"); + var answer = Fixture[id]; - nullable["schema"]!.ToJsonString().ShouldBe("""{"oneOf":[{"type":"null"},{""" + Int32Pattern + ""","type":["null","integer","string"],"format":"int32"}]}"""); + answer.Mismatch(await DocumentAsync(answer)).ShouldBeNull(); } } diff --git a/test/CodoMetis.TypeKit.Facts/KB001.fixture.json b/test/CodoMetis.TypeKit.Facts/KB001.fixture.json new file mode 100644 index 00000000..e8bc42b3 --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/KB001.fixture.json @@ -0,0 +1,105 @@ +{ + "fact": "KB-001", + "answers": [ + { + "id": "openapi-2-0-mvc-int-body", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi2_0", + "http.host": "mvc", + "http.binding": "body", + "http.nullability": "as-declared", + "shape.wrapped": "int" + }, + "at": "/paths/~1kb001~1plain/post/parameters/0", + "value": { + "in": "body", + "name": "body", + "required": true, + "schema": { + "format": "int32", + "pattern": "^-?(?:0|[1-9]\\d*)$" + } + }, + "note": "required, with its pattern" + }, + { + "id": "openapi-2-0-mvc-nullable-int-body", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi2_0", + "http.host": "mvc", + "http.binding": "body", + "http.nullability": "nullable", + "shape.wrapped": "int" + }, + "at": "/paths/~1kb001~1nullable/post/parameters/0", + "value": { + "in": "body", + "name": "body", + "schema": { + "format": "int32", + "allOf": [ + { + "x-nullable": true + } + ] + } + }, + "note": "optional, and no pattern" + }, + { + "id": "openapi-2-0-minimal-nullable-enum-body", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi2_0", + "http.host": "minimal", + "http.binding": "body", + "http.nullability": "nullable", + "shape.wrapped": "enum" + }, + "at": "/paths/~1kb001~1minimal/post/parameters/0", + "value": { + "in": "body", + "name": "body", + "schema": { + "allOf": [ + { + "x-nullable": true + } + ] + } + }, + "note": "no type, nothing but the null" + }, + { + "id": "openapi-3-1-mvc-nullable-int-body", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi3_1", + "http.host": "mvc", + "http.binding": "body", + "http.nullability": "nullable", + "shape.wrapped": "int" + }, + "at": "/paths/~1kb001~1nullable/post/requestBody/content/application~1json/schema", + "value": { + "oneOf": [ + { + "type": "null" + }, + { + "pattern": "^-?(?:0|[1-9]\\d*)$", + "type": [ + "null", + "integer", + "string" + ], + "format": "int32" + } + ] + }, + "note": "3.1 keeps the pattern inside oneOf" + } + ] +} diff --git a/test/CodoMetis.TypeKit.Facts/KB026.cs b/test/CodoMetis.TypeKit.Facts/KB026.cs index fa3fccbe..391f36c9 100644 --- a/test/CodoMetis.TypeKit.Facts/KB026.cs +++ b/test/CodoMetis.TypeKit.Facts/KB026.cs @@ -21,28 +21,20 @@ public sealed class HolderController : ControllerBase public IActionResult Post([FromBody] Holder holder) => Ok(); } - private static async Task PropertiesAsync(OpenApiSpecVersion version) - { - var document = await FactHost.DocumentAsync(version, [typeof(HolderController)]); - var schemas = version == OpenApiSpecVersion.OpenApi2_0 ? document["definitions"]! : document["components"]!["schemas"]!; - - return schemas["Holder"]!["properties"]!; - } + private static readonly FactFixture Fixture = FactFixture.Beside("KB-026"); - [Fact] - public async Task Under_OpenApi_2_0_a_nullable_property_of_a_component_loses_its_reference() - { - var properties = await PropertiesAsync(OpenApiSpecVersion.OpenApi2_0); - - properties["always"]!.ToJsonString().ShouldBe("""{"$ref":"#/definitions/Spot"}"""); - properties["maybe"]!.ToJsonString().ShouldBe("""{"allOf":[{"x-nullable":true}]}""", "no reference, no type"); - } + public static TheoryData Answers() => [.. Fixture.Answers.Select(answer => answer.Id)]; - [Fact] - public async Task Under_OpenApi_3_1_the_nullable_property_is_null_or_the_reference() + /// + /// Each answer of KB026.fixture.json: under OpenAPI 2.0 the nullable property is allOf: [{x-nullable: true}] + /// where its non-nullable twin is the reference; under 3.1 it is null or the reference. + /// + [Theory, MemberData(nameof(Answers))] + public async Task The_framework_writes_the_property_the_fixture_holds(string id) { - var properties = await PropertiesAsync(OpenApiSpecVersion.OpenApi3_1); + var answer = Fixture[id]; + var document = await FactHost.DocumentAsync(Enum.Parse(answer.When["openapi.OpenApiVersion"]), [typeof(HolderController)]); - properties["maybe"]!.ToJsonString().ShouldBe("""{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/Spot"}]}"""); + answer.Mismatch(document).ShouldBeNull(); } } diff --git a/test/CodoMetis.TypeKit.Facts/KB026.fixture.json b/test/CodoMetis.TypeKit.Facts/KB026.fixture.json new file mode 100644 index 00000000..4bea1214 --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/KB026.fixture.json @@ -0,0 +1,61 @@ +{ + "fact": "KB-026", + "answers": [ + { + "id": "openapi-2-0-component-property", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi2_0", + "http.host": "mvc", + "http.binding": "body", + "http.nullability": "as-declared" + }, + "at": "/definitions/Holder/properties/always", + "value": { + "$ref": "#/definitions/Spot" + }, + "note": "the reference" + }, + { + "id": "openapi-2-0-nullable-component-property", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi2_0", + "http.host": "mvc", + "http.binding": "body", + "http.nullability": "nullable" + }, + "at": "/definitions/Holder/properties/maybe", + "value": { + "allOf": [ + { + "x-nullable": true + } + ] + }, + "note": "no reference, no type" + }, + { + "id": "openapi-3-1-nullable-component-property", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi3_1", + "http.host": "mvc", + "http.binding": "body", + "http.nullability": "nullable" + }, + "at": "/components/schemas/Holder/properties/maybe", + "value": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/Spot" + } + ] + }, + "note": "null or the reference" + } + ] +} diff --git a/test/CodoMetis.TypeKit.Facts/KB027.cs b/test/CodoMetis.TypeKit.Facts/KB027.cs index 64996d02..c14be0e8 100644 --- a/test/CodoMetis.TypeKit.Facts/KB027.cs +++ b/test/CodoMetis.TypeKit.Facts/KB027.cs @@ -28,18 +28,27 @@ await FactHost.DocumentAsync(version, [], minimal: app => app.MapGet("/plain-first/b", ([FromHeader(Name = "v")] MetPlainFirst? v) => 0); }); + private static readonly FactFixture Fixture = FactFixture.Beside("KB-027"); + + public static TheoryData Answers() => [.. Fixture.Answers.Select(answer => answer.Id)]; + + /// Each answer of KB027.fixture.json: the enum's values in its component, null among them only where the nullable use came first. + [Theory, MemberData(nameof(Answers))] + public async Task An_enum_component_is_the_schema_of_its_first_use(string id) + { + var answer = Fixture[id]; + + answer.Mismatch(await DocumentAsync(Enum.Parse(answer.When["openapi.OpenApiVersion"]))).ShouldBeNull(); + } + + /// Both uses of each enum refer to the one component, so the null goes with the component, not with the use. [Theory] [InlineData(OpenApiSpecVersion.OpenApi3_1)] [InlineData(OpenApiSpecVersion.OpenApi3_0)] - public async Task An_enum_component_is_the_schema_of_its_first_use(OpenApiSpecVersion version) + public async Task Both_uses_refer_to_the_one_component(OpenApiSpecVersion version) { var document = await DocumentAsync(version); - var schemas = document["components"]!["schemas"]!; - - schemas[nameof(MetNullableFirst)]!["enum"]!.ToJsonString().ShouldBe("""["X","Y",null]""", "met first as nullable: null among its values"); - schemas[nameof(MetPlainFirst)]!["enum"]!.ToJsonString().ShouldBe("""["X","Y"]""", "met first as itself: no null"); - // Both uses of each refer to the one component, so the null goes with the component, not with the use. foreach (var path in new[] { "/nullable-first/a", "/nullable-first/b" }) document["paths"]![path]!["get"]!["parameters"]![0]!["schema"]!["$ref"]!.GetValue().ShouldEndWith(nameof(MetNullableFirst)); foreach (var path in new[] { "/plain-first/a", "/plain-first/b" }) diff --git a/test/CodoMetis.TypeKit.Facts/KB027.fixture.json b/test/CodoMetis.TypeKit.Facts/KB027.fixture.json new file mode 100644 index 00000000..6494e737 --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/KB027.fixture.json @@ -0,0 +1,79 @@ +{ + "fact": "KB-027", + "answers": [ + { + "id": "openapi-3-1-met-nullable-first", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi3_1", + "http.host": "minimal", + "http.binding": "header", + "json.Converters": "JsonStringEnumConverter", + "shape.wrapped": "enum", + "first-use": "nullable" + }, + "at": "/components/schemas/MetNullableFirst/enum", + "value": [ + "X", + "Y", + null + ], + "note": "met first as nullable: null among its values, for the plain use too" + }, + { + "id": "openapi-3-1-met-plain-first", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi3_1", + "http.host": "minimal", + "http.binding": "header", + "json.Converters": "JsonStringEnumConverter", + "shape.wrapped": "enum", + "first-use": "as-declared" + }, + "at": "/components/schemas/MetPlainFirst/enum", + "value": [ + "X", + "Y" + ], + "note": "met first as itself: no null, for the nullable use too" + }, + { + "id": "openapi-3-0-met-nullable-first", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi3_0", + "http.host": "minimal", + "http.binding": "header", + "json.Converters": "JsonStringEnumConverter", + "shape.wrapped": "enum", + "first-use": "nullable" + }, + "at": "/components/schemas/MetNullableFirst/enum", + "value": [ + "X", + "Y", + null + ], + "note": "met first as nullable: null among its values, for the plain use too" + }, + { + "id": "openapi-3-0-met-plain-first", + "kind": "document", + "when": { + "openapi.OpenApiVersion": "OpenApi3_0", + "http.host": "minimal", + "http.binding": "header", + "json.Converters": "JsonStringEnumConverter", + "shape.wrapped": "enum", + "first-use": "as-declared" + }, + "at": "/components/schemas/MetPlainFirst/enum", + "value": [ + "X", + "Y" + ], + "note": "met first as itself: no null, for the nullable use too" + } + ] +} diff --git a/test/CodoMetis.TypeKit.Facts/KB037.cs b/test/CodoMetis.TypeKit.Facts/KB037.cs index 4b451e2d..597b4164 100644 --- a/test/CodoMetis.TypeKit.Facts/KB037.cs +++ b/test/CodoMetis.TypeKit.Facts/KB037.cs @@ -32,11 +32,9 @@ public sealed class BindController : ControllerBase [HttpGet("kb037/params")] public IActionResult Params([FromQuery] Q q) => Ok(q.V); } - /// Each binding source of a simple parameter, sent a text no int reads. - public static TheoryData Simple() => ["route", "query", "header", "form", "body"]; + private static readonly FactFixture Fixture = FactFixture.Beside("KB-037"); - /// The same, and the property of a class bound from the query. - public static TheoryData Sources() => ["route", "query", "header", "form", "body", "params"]; + public static TheoryData Answers() => [.. Fixture.Answers.Select(answer => answer.Id)]; private static async Task<(HttpStatusCode Status, string Body)> SendAsync(int? maxValidationDepth, string source, string text = "abc") { @@ -67,38 +65,27 @@ public sealed class BindController : ControllerBase return (response.StatusCode, await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken)); } - [Theory, MemberData(nameof(Simple))] - public async Task Under_MaxValidationDepth_1_an_unbindable_parameter_is_its_default_and_the_action_answers_200(string source) + /// + /// Each answer of KB037.fixture.json, sent as its settings say: the binding source (as-parameters is a class + /// bound from the query), the text, and MaxValidationDepth 1 or the default. + /// + [Theory, MemberData(nameof(Answers))] + public async Task MVC_answers_as_the_fixture_holds(string id) { - var (status, body) = await SendAsync(1, source); + var answer = Fixture[id]; + var depth = answer.When["mvc.MaxValidationDepth"] == "default" ? (int?)null : int.Parse(answer.When["mvc.MaxValidationDepth"], System.Globalization.CultureInfo.InvariantCulture); + var source = answer.When["http.binding"] == "as-parameters" ? "params" : answer.When["http.binding"]; - status.ShouldBe(HttpStatusCode.OK, body); - body.ShouldBe("0", "the int is left at its default"); - } - - [Theory, InlineData("abc"), InlineData("5")] - public async Task Under_MaxValidationDepth_1_a_class_from_the_query_makes_the_validator_throw_at_its_property(string text) - { - var thrown = await Should.ThrowAsync(() => SendAsync(1, "params", text)); - - thrown.Message.ShouldContain("ValidationVisitor exceeded the maximum configured validation depth '1' when validating property 'V'"); - } - - [Theory, MemberData(nameof(Simple))] - public async Task Under_MaxValidationDepth_1_a_text_the_parameter_reads_binds_as_usual(string source) - { - var (status, body) = await SendAsync(1, source, "5"); - - status.ShouldBe(HttpStatusCode.OK, body); - body.ShouldBe("5"); - } - - [Theory, MemberData(nameof(Sources))] - public async Task With_the_default_depth_the_same_request_is_a_400(string source) - { - var (status, body) = await SendAsync(null, source); - - status.ShouldBe(HttpStatusCode.BadRequest, body); + if (answer.Kind == FactAnswer.Exception) + { + var thrown = await Should.ThrowAsync(() => SendAsync(depth, source, answer.When["text"])); + answer.Mismatch(thrown).ShouldBeNull(); + } + else + { + var (status, body) = await SendAsync(depth, source, answer.When["text"]); + answer.Mismatch((int)status, body).ShouldBeNull(); + } } private sealed class Only : IApplicationFeatureProvider diff --git a/test/CodoMetis.TypeKit.Facts/KB037.fixture.json b/test/CodoMetis.TypeKit.Facts/KB037.fixture.json new file mode 100644 index 00000000..c916bd68 --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/KB037.fixture.json @@ -0,0 +1,291 @@ +{ + "fact": "KB-037", + "answers": [ + { + "id": "depth-1-route-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "route", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 200, + "body": "0" + }, + "note": "the action runs, the int left at its default" + }, + { + "id": "depth-1-query-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "query", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 200, + "body": "0" + }, + "note": "the action runs, the int left at its default" + }, + { + "id": "depth-1-header-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "header", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 200, + "body": "0" + }, + "note": "the action runs, the int left at its default" + }, + { + "id": "depth-1-form-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "form", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 200, + "body": "0" + }, + "note": "the action runs, the int left at its default" + }, + { + "id": "depth-1-body-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "body", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 200, + "body": "0" + }, + "note": "the action runs, the int left at its default" + }, + { + "id": "depth-1-route-readable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "route", + "shape.wrapped": "int", + "text": "5" + }, + "value": { + "status": 200, + "body": "5" + }, + "note": "a text the int reads binds as usual" + }, + { + "id": "depth-1-query-readable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "query", + "shape.wrapped": "int", + "text": "5" + }, + "value": { + "status": 200, + "body": "5" + }, + "note": "a text the int reads binds as usual" + }, + { + "id": "depth-1-header-readable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "header", + "shape.wrapped": "int", + "text": "5" + }, + "value": { + "status": 200, + "body": "5" + }, + "note": "a text the int reads binds as usual" + }, + { + "id": "depth-1-form-readable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "form", + "shape.wrapped": "int", + "text": "5" + }, + "value": { + "status": 200, + "body": "5" + }, + "note": "a text the int reads binds as usual" + }, + { + "id": "depth-1-body-readable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "body", + "shape.wrapped": "int", + "text": "5" + }, + "value": { + "status": 200, + "body": "5" + }, + "note": "a text the int reads binds as usual" + }, + { + "id": "depth-1-as-parameters-unreadable", + "kind": "exception", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "as-parameters", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "type": "System.InvalidOperationException", + "operation": "GET /kb037/params?V=abc", + "stage": "model validation, at the class's property V", + "message": "ValidationVisitor exceeded the maximum configured validation depth '1' when validating property 'V'" + }, + "note": "a class bound from the query: the validator throws at its property, bound or not" + }, + { + "id": "depth-1-as-parameters-readable", + "kind": "exception", + "when": { + "mvc.MaxValidationDepth": "1", + "http.host": "mvc", + "http.binding": "as-parameters", + "shape.wrapped": "int", + "text": "5" + }, + "value": { + "type": "System.InvalidOperationException", + "operation": "GET /kb037/params?V=5", + "stage": "model validation, at the class's property V", + "message": "ValidationVisitor exceeded the maximum configured validation depth '1' when validating property 'V'" + }, + "note": "a class bound from the query: the validator throws at its property, bound or not" + }, + { + "id": "default-depth-route-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "default", + "http.host": "mvc", + "http.binding": "route", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 400 + }, + "note": "with the default depth every unbindable request is a 400" + }, + { + "id": "default-depth-query-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "default", + "http.host": "mvc", + "http.binding": "query", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 400 + }, + "note": "with the default depth every unbindable request is a 400" + }, + { + "id": "default-depth-header-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "default", + "http.host": "mvc", + "http.binding": "header", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 400 + }, + "note": "with the default depth every unbindable request is a 400" + }, + { + "id": "default-depth-form-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "default", + "http.host": "mvc", + "http.binding": "form", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 400 + }, + "note": "with the default depth every unbindable request is a 400" + }, + { + "id": "default-depth-body-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "default", + "http.host": "mvc", + "http.binding": "body", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 400 + }, + "note": "with the default depth every unbindable request is a 400" + }, + { + "id": "default-depth-as-parameters-unreadable", + "kind": "status", + "when": { + "mvc.MaxValidationDepth": "default", + "http.host": "mvc", + "http.binding": "as-parameters", + "shape.wrapped": "int", + "text": "abc" + }, + "value": { + "status": 400 + }, + "note": "with the default depth every unbindable request is a 400" + } + ] +} diff --git a/test/CodoMetis.TypeKit.Prober.Tests/SeedsTests.cs b/test/CodoMetis.TypeKit.Prober.Tests/SeedsTests.cs index 5cb25cf2..2fce1dab 100644 --- a/test/CodoMetis.TypeKit.Prober.Tests/SeedsTests.cs +++ b/test/CodoMetis.TypeKit.Prober.Tests/SeedsTests.cs @@ -58,7 +58,7 @@ private void Seed(string? file = "src/a.cs", string? edit = "class A { /* DEFECT var output = new StringWriter(); var error = new StringWriter(); var context = new Context(fx.Root, new Dictionary { ["PROBE_CACHE_DIR"] = cache }, output, error); int exit; - try { exit = global::Prober.Seeds.Run(context, new Args(args, new HashSet(StringComparer.Ordinal) { "dry-run" }), adapter); } + try { exit = global::Prober.Seeds.Run(context, new Args(args, new HashSet(StringComparer.Ordinal) { "dry-run", "all" }), adapter); } catch (Refusal refusal) { error.WriteLine(refusal.Message); exit = refusal.ExitCode; } return (exit, output.ToString(), error.ToString()); } @@ -170,6 +170,175 @@ public void The_seed_is_not_part_of_the_probe_s_identity() fx.Ledger("identity", "--probe", "P").Out.ShouldBe(before, "a seed proves the probe; adding or changing one must not stale its results"); } + /// What the replay records for P: the product files it ran, at HEAD. + private void Footprint(params string[] files) => + File.AppendAllText(Path.Combine(fx.BranchDir, "footprints.jsonl"), new JsonObject + { + ["probe"] = "P", + ["footprint"] = new JsonObject(files.Select(f => KeyValuePair.Create(f, (JsonNode?)fx.Sh("git", "rev-parse", "HEAD:" + f).Trim()))), + ["compileTime"] = new JsonObject(), ["toolchain"] = new JsonObject(), + }.ToJsonString() + "\n"); + + /// P seeded and caught once, with a footprint over src/a.cs and src/b.cs or the files given, so the next run has a record to skip by. + private void CaughtOnce(params string[] footprint) + { + fx.Write("src/b.cs", "class B {}\n"); + fx.Commit("a second product file"); + Seed(); + Footprint(footprint.Length > 0 ? footprint : ["src/a.cs", "src/b.cs"]); + var (exit, output, error) = Seeds(); + exit.ShouldBe(0, output + error); + output.ShouldContain(" runs P: no run caught every seed of it yet"); + output.ShouldContain("P (C1): caught"); + } + + private JsonObject Last() => (JsonObject)JsonNode.Parse(File.ReadAllText(Path.Combine(fx.BranchDir, "seeds-last.json")))!; + + [Fact] + public void An_unchanged_probe_is_skipped_and_says_why() + { + CaughtOnce(); + var builds = adapter.Worktrees.Count; + + var (exit, output, error) = Seeds(); + + exit.ShouldBe(0, output + error); + output.ShouldContain(" skips P: every seed caught at " + fx.Sh("git", "rev-parse", "--short=7", "HEAD").Trim() + ", and neither its identity nor its product footprint changed since"); + output.ShouldContain("0 seeds of 0 probes"); + output.ShouldContain("1 probes skipped"); + adapter.Worktrees.Count.ShouldBe(builds, "nothing built or ran"); + Last()["skipped"]!.AsArray().Single()!["probe"]!.GetValue().ShouldBe("P"); + Last()["probes"]!["P"]!["commit"]!.GetValue().ShouldBe(fx.Sh("git", "rev-parse", "HEAD").Trim(), "a skipped probe's record is carried forward"); + } + + [Fact] + public void A_changed_identity_reruns() + { + CaughtOnce(); + fx.Write("tools/probe/corpus/harness/P.cs", "// probe P, changed\n"); + fx.Commit("the harness changed"); + + var (_, output, _) = Seeds(); + + output.ShouldContain(" runs P: its identity or its seeds changed since they were caught at "); + output.ShouldContain("P (C1): caught"); + } + + [Fact] + public void A_changed_seed_reruns() + { + CaughtOnce(); + Seed(edit: "class A { /* DEFECT, another */ }\n"); + + Seeds().Out.ShouldContain(" runs P: its identity or its seeds changed since they were caught at "); + } + + [Fact] + public void A_changed_product_footprint_reruns() + { + CaughtOnce(); + fx.Write("src/b.cs", "class B { int changed; }\n"); + fx.Commit("the product changed where P runs"); + + var (_, output, _) = Seeds(); + + output.ShouldContain(" runs P: its product footprint changed since "); + output.ShouldContain("P (C1): caught"); + } + + /// The seed's own file may be outside what the probe ran; a change there may leave the seed behind, so it reruns. + [Fact] + public void A_change_to_the_file_the_seed_patches_reruns() + { + CaughtOnce("src/b.cs"); + fx.Write("src/a.cs", "class A {}\n// moved on\n"); + fx.Commit("the seeded file changed"); + + Seeds().Out.ShouldContain(" runs P: a file its seeds patch changed since "); + } + + [Fact] + public void A_change_outside_the_footprint_skips() + { + CaughtOnce(); + fx.Write("src/c.cs", "class C {}\n"); + fx.Commit("the product changed where P does not run"); + + Seeds().Out.ShouldContain(" skips P: "); + } + + [Fact] + public void A_probe_without_a_recorded_footprint_runs() + { + Seed(); + Seeds().Exit.ShouldBe(0); + + Seeds().Out.ShouldContain(" runs P: no product footprint recorded for it (a replay records one)"); + } + + [Fact] + public void All_ignores_the_record() + { + CaughtOnce(); + + var (exit, output, _) = Seeds("--all"); + + exit.ShouldBe(0); + output.ShouldNotContain("skips P"); + output.ShouldContain("P (C1): caught"); + Last()["all"]!.GetValue().ShouldBeTrue(); + } + + [Fact] + public void A_probe_named_runs_whatever_its_record_says() + { + CaughtOnce(); + + Seeds("--probe", "P").Out.ShouldContain("P (C1): caught"); + } + + [Fact] + public void A_missed_seed_is_never_skipped_next_time() + { + fx.Write("src/b.cs", "class B {}\n"); + fx.Commit("a second product file"); + Seed(edit: "class A { /* harmless */ }\n"); + Footprint("src/a.cs", "src/b.cs"); + Seeds().Exit.ShouldBe(1); + Last()["probes"]!.AsObject().ShouldBeEmpty(); + + var (exit, output, _) = Seeds(); + + exit.ShouldBe(1); + output.ShouldContain(" runs P: no run caught every seed of it yet"); + output.ShouldContain("P (C1): missed"); + } + + [Fact] + public void A_seed_missed_after_a_catch_drops_the_record() + { + CaughtOnce(); + Seed(edit: "class A { /* harmless */ }\n"); + Seeds().Exit.ShouldBe(1); + + Last()["probes"]!.AsObject().ContainsKey("P").ShouldBeFalse(); + Seeds().Out.ShouldContain(" runs P: no run caught every seed of it yet"); + } + + [Fact] + public void The_record_is_what_its_schema_says() + { + CaughtOnce(); + Seeds(); + var root = AppContext.BaseDirectory; + while (!File.Exists(Path.Combine(root, "CodoMetis.TypeKit.slnx"))) root = Path.GetDirectoryName(root) ?? throw new InvalidOperationException("no CodoMetis.TypeKit.slnx above the test output"); + var schema = (JsonObject)JsonNode.Parse(File.ReadAllText(Path.Combine(root, "tools", "Prober", "Schemas", "seeds-run.schema.json")))!["properties"]!; + + foreach (var field in new[] { "probes", "skipped", "all" }) + SchemaCheck.Problems(Last()[field], (JsonObject)schema[field]!, field).ShouldBeEmpty(field); + Last()["probes"]!["P"]!["identity"]!.GetValue().ShouldBe(global::Prober.Seeds.SeedsIdentity(new Ledger(new Context(fx.Root, new Dictionary(), TextWriter.Null, TextWriter.Null), null), "P")); + } + private List Problems() => global::Prober.Seeds.Problems(new Ledger(new Context(fx.Root, new Dictionary(), TextWriter.Null, TextWriter.Null), null)); [Fact] diff --git a/tools/DocsQuery/Citations.cs b/tools/DocsQuery/Citations.cs new file mode 100644 index 00000000..ef97fba4 --- /dev/null +++ b/tools/DocsQuery/Citations.cs @@ -0,0 +1,260 @@ +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Text.RegularExpressions; + +namespace DocsQuery; + +/// +/// Where the probe corpus leans on a verified fact to prune a row or excuse one (design/adr/0107): a refused +/// entry of the axes file whose reason names KB-NNN, a not-promised entry of claims.json whose +/// source is a fact's file or which says fact:KB-NNN, an obligation's declaration of where it does not +/// apply whose cite is fact:KB-NNN, and a Cite.Fact("KB-NNN") or Baseline.Fact("KB-NNN") +/// literal in the harness and probe sources, read as text with comments blanked out. Each is read from the checkout +/// at every check, never from the index. +/// +internal static partial class FactCitations +{ + public const string AxesFile = ".claude/probe-axes.md", ClaimsFile = "tools/probe/corpus/claims.json"; + + public static readonly string[] SourceDirectories = ["tools/probe/corpus/harness", "tools/probe/corpus/probes"]; + + /// The four places, in the order check counts them. + public static readonly string[] Kinds = [Refused, NotPromised, Obligation, Harness]; + + public const string Refused = "refused", NotPromised = "not-promised", Obligation = "obligation", Harness = "harness"; + + /// + /// One citation: the kind of place, where it is written (path:line), the fact's id, the place in words, and + /// whether it reads the fact's fixture as a baseline (Baseline.Fact), which takes a fixture as well as a test. + /// + internal sealed record Citation(string Kind, string At, string Fact, string Place, bool Baseline = false); + + /// What a scan read: every citation, and how many places of each kind it looked at, cited or not. + internal sealed record Scan(IReadOnlyList Citations, IReadOnlyDictionary Read) + { + public int Of(string kind) => Citations.Count(citation => citation.Kind == kind); + } + + public static Scan Find(string root) + { + var citations = new List(); + var read = Kinds.ToDictionary(kind => kind, _ => 0, StringComparer.Ordinal); + + ReadAxes(root, citations, read); + ReadClaims(root, citations, read); + ReadSources(root, citations, read); + + return new Scan(citations, read); + } + + /// The axes file's refused entries: axis to value to the reason the framework refuses it. + private static void ReadAxes(string root, List citations, Dictionary read) + { + var path = Path.Combine(root, AxesFile); + if (!File.Exists(path)) return; + + const string marker = "```json probe-axes"; + var text = File.ReadAllText(path).ReplaceLineEndings("\n"); + var start = text.IndexOf(marker, StringComparison.Ordinal); + if (start < 0) return; + start += marker.Length; + var end = text.IndexOf("```", start, StringComparison.Ordinal); + if (end < 0) return; + + using var block = JsonDocument.Parse(text[start..end]); + if (!block.RootElement.TryGetProperty("refused", out var refused) || refused.ValueKind != JsonValueKind.Object) return; + + var cursor = new Cursor(AxesFile, text, text.IndexOf("\"refused\"", start, StringComparison.Ordinal)); + foreach (var axis in refused.EnumerateObject().Where(axis => axis.Value.ValueKind == JsonValueKind.Object)) + { + foreach (var value in axis.Value.EnumerateObject()) + { + read[Refused]++; + var place = $"the refused entry {axis.Name}={value.Name}"; + cursor.Seek($"\"{value.Name}\""); + foreach (var fact in FactIds(value.Value.ValueKind == JsonValueKind.String ? value.Value.GetString()! : value.Value.GetRawText())) + citations.Add(new Citation(Refused, cursor.At(fact), fact, place)); + } + } + } + + /// Each claim's not-promised entries and its obligations' inapplicable declarations. + private static void ReadClaims(string root, List citations, Dictionary read) + { + var path = Path.Combine(root, ClaimsFile); + if (!File.Exists(path)) return; + + var text = File.ReadAllText(path).ReplaceLineEndings("\n"); + using var claims = JsonDocument.Parse(text); + if (claims.RootElement.ValueKind != JsonValueKind.Array) return; + + // Walked in the file's order, so each citation's line is the next occurrence of its id from the last one. + var cursor = new Cursor(ClaimsFile, text, 0); + foreach (var claim in claims.RootElement.EnumerateArray()) + { + var id = Str(claim, "id"); + cursor.Seek($"\"{id}\""); + + foreach (var property in claim.EnumerateObject()) + { + if (property.NameEquals("obligations") && property.Value.ValueKind == JsonValueKind.Array) + { + foreach (var obligation in property.Value.EnumerateArray()) + { + cursor.Seek($"\"{Str(obligation, "id")}\""); + var number = 0; + foreach (var declaration in Array(obligation, "inapplicable")) + { + read[Obligation]++; + number++; + if (FactOf(Str(declaration, "cite")) is not { } fact) continue; + citations.Add(new Citation(Obligation, cursor.At("fact:" + fact), fact, + $"claim {id}, obligation {Str(obligation, "id")}, inapplicable declaration {number.ToString(CultureInfo.InvariantCulture)}")); + } + } + } + else if (property.NameEquals("not-promised") && property.Value.ValueKind == JsonValueKind.Array) + { + foreach (var entry in property.Value.EnumerateArray()) + { + read[NotPromised]++; + cursor.Seek($"\"{Str(entry, "id")}\""); + var place = $"claim {id}, not-promised {Str(entry, "id")}"; + var facts = new List(); + if (KnowledgeSource().Match(Str(entry, "source")) is { Success: true } source) facts.Add(source.Groups["id"].Value); + foreach (var field in entry.EnumerateObject().Where(field => field.Value.ValueKind == JsonValueKind.String)) + facts.AddRange(CiteFact().Matches(field.Value.GetString()!).Select(match => match.Groups["id"].Value)); + + foreach (var fact in facts.Distinct(StringComparer.Ordinal)) + citations.Add(new Citation(NotPromised, cursor.At(fact), fact, place)); + } + } + } + } + } + + /// The harness's and the probes' Cite.Fact and Baseline.Fact literals, comments blanked out. + private static void ReadSources(string root, List citations, Dictionary read) + { + foreach (var directory in SourceDirectories.Select(directory => Path.Combine(root, directory)).Where(Directory.Exists)) + { + foreach (var file in Directory.EnumerateFiles(directory, "*.cs").Order(StringComparer.Ordinal)) + { + read[Harness]++; + var path = Corpus.Relative(root, file); + var source = File.ReadAllText(file).ReplaceLineEndings("\n"); + foreach (Match match in FactLiteral().Matches(WithoutComments(source))) + { + var line = 1 + source.AsSpan(0, match.Index).Count('\n'); + citations.Add(new Citation(Harness, $"{path}:{line.ToString(CultureInfo.InvariantCulture)}", match.Groups["id"].Value, + $"{match.Groups["factory"].Value}.Fact(\"{match.Groups["id"].Value}\")", match.Groups["factory"].Value == "Baseline")); + } + } + } + } + + /// + /// The source with each comment's content replaced by spaces, newlines kept, and string literals as they are, so a + /// citation written in a comment does not count and a line number still holds. + /// + internal static string WithoutComments(string source) + { + var result = new StringBuilder(source); + for (var i = 0; i < source.Length; i++) + { + if (source[i] == '/' && i + 1 < source.Length && source[i + 1] == '/') + { + for (; i < source.Length && source[i] != '\n'; i++) result[i] = ' '; + } + else if (source[i] == '/' && i + 1 < source.Length && source[i + 1] == '*') + { + var close = source.IndexOf("*/", i + 2, StringComparison.Ordinal); + var stop = close < 0 ? source.Length : close + 2; + for (; i < stop; i++) if (source[i] != '\n') result[i] = ' '; + i--; + } + else if (source[i] == '"') + { + i = AfterString(source, i); + } + else if (source[i] == '\'') + { + // A character literal: '"' must not open a string. + var j = i + 1; + if (j < source.Length && source[j] == '\\') j++; + if (j + 1 < source.Length && source[j + 1] == '\'') i = j + 1; + } + } + + return result.ToString(); + } + + /// The index of the closing quote of the string literal opened at : raw, verbatim or regular. + private static int AfterString(string source, int i) + { + var quotes = 0; + while (i + quotes < source.Length && source[i + quotes] == '"') quotes++; + + if (quotes >= 3) + { + var close = source.IndexOf(new string('"', quotes), i + quotes, StringComparison.Ordinal); + return close < 0 ? source.Length : close + quotes - 1; + } + + if (quotes == 2) return i + 1; // "" + + var verbatim = i > 0 && source[i - 1] == '@' || i > 1 && source[i - 1] == '$' && source[i - 2] == '@'; + for (var j = i + 1; j < source.Length; j++) + { + if (verbatim && source[j] == '"' && j + 1 < source.Length && source[j + 1] == '"') { j++; continue; } + if (!verbatim && source[j] == '\\') { j++; continue; } + if (source[j] == '"' || !verbatim && source[j] == '\n') return j; + } + + return source.Length; + } + + /// The ids a refused entry's reason names, as fact:KB-NNN or plain KB-NNN. + private static IEnumerable FactIds(string text) => AnyFact().Matches(text).Select(match => match.Value).Distinct(StringComparer.Ordinal); + + private static string? FactOf(string cite) => cite.StartsWith("fact:", StringComparison.Ordinal) ? cite["fact:".Length..] : null; + + private static string Str(JsonElement element, string name) => + element.ValueKind == JsonValueKind.Object && element.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.String ? value.GetString()! : ""; + + private static IEnumerable Array(JsonElement element, string name) => + element.ValueKind == JsonValueKind.Object && element.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.Array ? value.EnumerateArray() : []; + + /// Line numbers for places read in a file's order: each needle is looked for from the last place found. + private sealed class Cursor(string path, string text, int offset) + { + private int at = Math.Max(0, offset); + + public void Seek(string needle) + { + var found = text.IndexOf(needle, at, StringComparison.Ordinal); + if (found >= 0) at = found; + } + + public string At(string needle) + { + var found = text.IndexOf(needle, at, StringComparison.Ordinal); + var line = 1 + text.AsSpan(0, found >= 0 ? found : at).Count('\n'); + if (found >= 0) at = found + needle.Length; + return $"{path}:{line.ToString(CultureInfo.InvariantCulture)}"; + } + } + + [GeneratedRegex(@"\bKB-\d{3}\b")] + private static partial Regex AnyFact(); + + [GeneratedRegex(@"\bfact:\s?(?KB-\d{3})\b")] + private static partial Regex CiteFact(); + + [GeneratedRegex(@"^design/knowledge/(?KB-\d{3})-[^/#]*\.md(#.*)?$")] + private static partial Regex KnowledgeSource(); + + [GeneratedRegex(@"\b(?Cite|Baseline)\s*\.\s*Fact\s*\(\s*""(?KB-\d{3})""\s*\)")] + private static partial Regex FactLiteral(); +} diff --git a/tools/DocsQuery/Facts.cs b/tools/DocsQuery/Facts.cs index f222da25..9f5d81ec 100644 --- a/tools/DocsQuery/Facts.cs +++ b/tools/DocsQuery/Facts.cs @@ -124,7 +124,10 @@ public static int CompareVersions(string left, string right) /// /// The rules a verified fact is held to (design/adr/0093): its frontmatter is complete and plain, each pin /// matches what is installed, and its test: names the characterization test that observes it, or says -/// none and why. A test of the facts project that no fact names is a problem too. +/// none and why. A test of the facts project that no fact names is a problem too. A fixture:, the +/// measured answer of the fact's test as data, sits beside that test and exists (design/adr/0107). A fact the probe +/// corpus cites to prune or excuse a row () is executable: its test exists, and where the +/// harness reads it as a baseline, its fixture too. /// internal static partial class FactRules { @@ -134,11 +137,19 @@ internal static partial class FactRules internal sealed record Problem(string At, string Kind, string Target, string Detail); - public static IReadOnlyList Check(DocIndex index, string root, out int facts) + public const string FixtureSuffix = ".fixture.json"; + + public static IReadOnlyList Check(DocIndex index, string root, out int facts) => Check(index, root, out facts, out _); + + public static IReadOnlyList Check(DocIndex index, string root, out int facts, out FactCitations.Scan cited) { var problems = new List(); var installed = Installed.Read(root); var named = new HashSet(StringComparer.Ordinal); + var fixtures = new HashSet(StringComparer.Ordinal); + // Why each fact cannot settle a pruning or an excuse, or null where its test exists; and its fixture, where one exists. + var notExecutable = new Dictionary(StringComparer.Ordinal); + var fixtureOf = new Dictionary(StringComparer.Ordinal); facts = 0; for (var d = 0; d < index.Docs.Count; d++) @@ -174,29 +185,73 @@ public static IReadOnlyList Check(DocIndex index, string root, out int }, pin.Value, detail)); } + var className = doc.Id!.Replace("-", "", StringComparison.Ordinal); + notExecutable[doc.Id!] = "it has no test: line"; + if (doc.Meta.FirstOrDefault(field => field.Key == "test") is { } test) { - var className = doc.Id!.Replace("-", "", StringComparison.Ordinal); - - if (test.Value.StartsWith("none, ", StringComparison.Ordinal) && test.Value.Length > "none, ".Length) continue; - - var path = TestPath(test.Value); - if (!path.StartsWith(FactsProject + "/", StringComparison.Ordinal)) + if (test.Value.StartsWith("none, ", StringComparison.Ordinal) && test.Value.Length > "none, ".Length) + { + notExecutable[doc.Id!] = $"its test: is '{test.Value}'"; + } + else if (TestPath(test.Value) is var path && !path.StartsWith(FactsProject + "/", StringComparison.Ordinal)) { problems.Add(new Problem(At(test.Line), "fact-untested", test.Value, $"write the path of its test in {FactsProject}/, or 'none, ' and why no test can observe it")); - continue; + notExecutable[doc.Id!] = $"its test: '{test.Value}' is no test of {FactsProject}/"; } + else + { + named.Add(path); + var file = Path.Combine(root, path); + + if (!File.Exists(file)) + { + problems.Add(new Problem(At(test.Line), "fact-untested", path, "the file does not exist")); + notExecutable[doc.Id!] = $"its test {path} does not exist"; + } + else if (!Regex.IsMatch(File.ReadAllText(file), $@"\bclass\s+{className}\b")) + { + problems.Add(new Problem(At(test.Line), "fact-untested", path, $"declares no class {className}, the test named by the fact's id")); + notExecutable[doc.Id!] = $"its test {path} declares no class {className}"; + } + else + { + notExecutable[doc.Id!] = null; + } + } + } - named.Add(path); - var file = Path.Combine(root, path); - - if (!File.Exists(file)) - problems.Add(new Problem(At(test.Line), "fact-untested", path, "the file does not exist")); - else if (!Regex.IsMatch(File.ReadAllText(file), $@"\bclass\s+{className}\b")) - problems.Add(new Problem(At(test.Line), "fact-untested", path, $"declares no class {className}, the test named by the fact's id")); + fixtureOf[doc.Id!] = null; + if (doc.Meta.FirstOrDefault(field => field.Key == "fixture") is { } fixture) + { + var expected = $"{FactsProject}/{className}{FixtureSuffix}"; + fixtures.Add(fixture.Value); + + if (NotPlain(fixture.Value) is { } why) + problems.Add(new Problem(At(fixture.Line), "fact-frontmatter", fixture.Key, why)); + else if (notExecutable[doc.Id!] is { } untested) + problems.Add(new Problem(At(fixture.Line), "fact-fixture", fixture.Value, $"a fixture is the measured answer of the fact's test, and {untested}")); + else if (fixture.Value != expected) + problems.Add(new Problem(At(fixture.Line), "fact-fixture", fixture.Value, $"write it beside the test, as {expected}")); + else if (!File.Exists(Path.Combine(root, fixture.Value))) + problems.Add(new Problem(At(fixture.Line), "fact-fixture-missing", fixture.Value, "the file does not exist")); + else + fixtureOf[doc.Id!] = fixture.Value; } } + cited = FactCitations.Find(root); + foreach (var citation in cited.Citations) + { + var why = !notExecutable.TryGetValue(citation.Fact, out var reason) ? "there is no such fact under design/knowledge/" + : reason is not null ? reason + : citation.Baseline && fixtureOf[citation.Fact] is null ? "the harness reads its fixture as a baseline, and it names no fixture: that exists" + : null; + if (why is not null) + problems.Add(new Problem(citation.At, "fact-not-executable-cited", citation.Fact, + $"{citation.Place} cites {citation.Fact}, and {why}: a fact without its test explains only, never prunes or excuses a row; write its test in {FactsProject}/, or cite a docs sentence")); + } + var project = Path.Combine(root, FactsProject); if (Directory.Exists(project)) { @@ -205,6 +260,12 @@ public static IReadOnlyList Check(DocIndex index, string root, out int if (TestFile().IsMatch(Path.GetFileName(path)) && !named.Contains(path)) problems.Add(new Problem($"{path}:1", "fact-orphan-test", path, "no fact in design/knowledge/ names this test")); } + + foreach (var path in Directory.EnumerateFiles(project, "KB*" + FixtureSuffix, SearchOption.TopDirectoryOnly).Select(path => Corpus.Relative(root, path)).Order(StringComparer.Ordinal)) + { + if (!fixtures.Contains(path)) + problems.Add(new Problem($"{path}:1", "fact-orphan-fixture", path, "no fact in design/knowledge/ names this fixture")); + } } return problems; diff --git a/tools/DocsQuery/Queries.cs b/tools/DocsQuery/Queries.cs index cd7ba897..ec174ae0 100644 --- a/tools/DocsQuery/Queries.cs +++ b/tools/DocsQuery/Queries.cs @@ -206,7 +206,8 @@ public Answer Claims(string? page, string? kind) /// /// Links whose file or anchor does not exist, ids that name no record, diagnostic or fact, and the facts' /// rules (): a pin behind or ahead of what is installed, or naming nothing installed, - /// a fact without its test, a test without its fact. + /// a fact without its test, a test without its fact, a fixture missing, and a fact the probe corpus cites to prune + /// or excuse a row that has no test. /// public Answer Check() { @@ -238,9 +239,12 @@ public Answer Check() } } - rows.AddRange(FactRules.Check(index, root, out var facts).Select(problem => Row.Of(("at", problem.At), ("kind", problem.Kind), ("target", problem.Target), ("detail", problem.Detail)))); + rows.AddRange(FactRules.Check(index, root, out var facts, out var cited).Select(problem => Row.Of(("at", problem.At), ("kind", problem.Kind), ("target", problem.Target), ("detail", problem.Detail)))); - return new Answer(Answer.Exact, "check", "", $"{Count(rows.Count, "problem")} in {links} links, {ids} ids and {Count(facts, "fact")}", rows, rows.Count == 0 ? 0 : 1); + // Each kind of citation counted, so a scan that found none of a kind it should have is seen in the summary. + var citations = string.Join(", ", FactCitations.Kinds.Select(kind => $"{cited.Of(kind).ToString(CultureInfo.InvariantCulture)} {kind}")); + + return new Answer(Answer.Exact, "check", "", $"{Count(rows.Count, "problem")} in {links} links, {ids} ids, {Count(cited.Citations.Count, "fact citation")} ({citations}) and {Count(facts, "fact")}", rows, rows.Count == 0 ? 0 : 1); Row Problem(Edge edge, string kind, string detail) => Row.Of(("at", At(edge.Doc, edge.Line)), ("kind", kind), ("target", edge.Written), ("detail", detail)); diff --git a/tools/DocsQuery/README.md b/tools/DocsQuery/README.md index 454951ba..6e8e666e 100644 --- a/tools/DocsQuery/README.md +++ b/tools/DocsQuery/README.md @@ -41,8 +41,9 @@ line moves. Its `hash` changes with its content. ## Verified facts A fact (`design/knowledge/KB-NNN-slug.md`, design/adr/0093) has YAML frontmatter, read as fields: `verified:`, -`pins:` (one field per list item), `source:`, `turned-on:` and `test:`. `list kind=fact` shows each with the date it -was verified, and `check` holds every fact to these rules: +`pins:` (one field per list item), `source:`, `turned-on:`, `test:` and, where its measured answer is data, `fixture:` +(design/adr/0107). `list kind=fact` shows each with the date it was verified, and `check` holds every fact to these +rules: | Problem | When | |---|---| @@ -51,8 +52,13 @@ was verified, and `check` holds every fact to these rules: | `fact-incomplete`, `fact-frontmatter` | A field is missing, `verified:` is no `YYYY-MM-DD`, or a value is not a plain YAML string (a `: ` or ` #` inside, a quote or backtick first) | | `fact-untested` | `test:` is neither `none, ` and a reason nor a file in `test/CodoMetis.TypeKit.Facts/` that declares the class named by the id (`KB007`); a note in parentheses after the path, such as where the test runs, is not part of it | | `fact-orphan-test` | A `KBNNN.cs` in the facts project that no fact names | +| `fact-fixture-missing`, `fact-fixture` | A `fixture:` whose file does not exist; one that is not `KBNNN.fixture.json` beside the test, or belongs to a fact without a test | +| `fact-orphan-fixture` | A `KBNNN.fixture.json` in the facts project that no fact names | +| `fact-not-executable-cited` | The probe corpus cites a fact that has no test: a `refused` entry of `.claude/probe-axes.md` whose reason names `KB-NNN`, a `not-promised` entry of `tools/probe/corpus/claims.json` whose `source` is the fact's file or which says `fact:KB-NNN`, an obligation's `inapplicable` declaration citing `fact:KB-NNN`, or a `Cite.Fact("KB-NNN")` or `Baseline.Fact("KB-NNN")` literal in the harness and probe sources (comments blanked out). No test means `none, …`, no file, a file that does not exist or declares no class of the id; a `Baseline.Fact` takes a fixture as well. Reported at the citing place | -The pins are read from the checkout at each `check`, never from the index. +The pins and the corpus's citations are read from the checkout at each `check`, never from the index. The summary +counts the citations of each kind (`4 fact citations (0 refused, 2 not-promised, 0 obligation, 2 harness)`), and +`RepositoryTests` holds floors on them, so a scan that finds nothing cannot pass for a clean one. ## Freshness diff --git a/tools/Prober/Program.cs b/tools/Prober/Program.cs index b7a430e7..1d931437 100644 --- a/tools/Prober/Program.cs +++ b/tools/Prober/Program.cs @@ -47,8 +47,10 @@ replay [--all] [--family F] [--parallel N] [--no-footprint] [--holder H] [--dry- of the replay configuration; N probe processes at a time (default: the cores); without coverage or footprint, for a state nobody keeps round explore|full [--minutes M] [--claims N] [--usd X] a capped exploration round (local only) - seeds [--probe P] [--dry-run] applies each probe's seeded defect in a scratch worktree, replays the probe - there and requires a violation it causes (local only, a build per seed) + seeds [--all] [--probe P] [--dry-run] applies each probe's seeded defect in a scratch worktree, replays the + probe there and requires a violation it causes (local only, a build per + seed): the probes whose identity or footprint changed since every seed of + theirs was caught, or all of them, as a release runs it State goes under PROBE_STATE_DIR when set (CI), else the main checkout's prober memory. Exit codes: 0 done, 1 refused or failed, 2 usage, 3 this dll is stale; `try` exits with the probe's own. diff --git a/tools/Prober/README.md b/tools/Prober/README.md index c35a50b6..86e913f5 100644 --- a/tools/Prober/README.md +++ b/tools/Prober/README.md @@ -20,7 +20,7 @@ dotnet tools/Prober/bin/Release/net10.0/prober.dll help | `new --claim C [--probe P] [--family F]` | none | anywhere: stages a probe's skeleton, its own rows and its registry entry with no seed yet, and prints what the writer still owes | | `try --probe P [--rows i,j,...] [--no-build]` | none | locally: builds once in the replay's worktree and runs that one probe, every row or the rows given; records nothing, and holds the worktree from a replay while it runs | | `round explore\|full` | the `prober` agent | locally only, under caps it cannot overrun | -| `seeds [--probe P]` | none | locally only: one build per seed, in a scratch worktree (decision 100) | +| `seeds [--all] [--probe P]` | none | locally only: one build per seed, in a scratch worktree (decision 100); the probes whose seeds identity or product footprint changed since every seed was caught, or all of them, as a release runs it (decision 107) | No command in the gate's path needs a login, an API key or a Claude executable. A gate obligation is met by a probe in the corpus, written by hand or by a round, or by a reviewed waiver in @@ -119,6 +119,14 @@ writes `seeds-last.json` (`Schemas/seeds-run.schema.json`), one outcome per seed `--seed-env NAME=VALUE...` for a toggle) writes a seed into `seeds`, in place of any seed of the entry that names O, `seed` included, and refuses an O that the claim's entry in `claims.json` does not list; without `--obligation`, `--seed` and `--seed-env` set the one `seed`. No seed is written into the registry by hand. +Without `--all`, `seeds` runs only the probes that changed since the run that caught every one of their seeds +(design/adr/0107), as `replay` runs only the stale ones: a probe runs when its seeds identity (the probe's identity, +its seeds and the content of their patches) differs from the one recorded, when the product footprint its last +replay recorded or a file its seeds patch changed since the recorded commit, when no footprint is recorded, and when +no run caught all its seeds. It prints each probe it runs or skips with the reason. `seeds-last.json` keeps, per probe, +the identity and the commit of the run that caught them all, carried forward for the probes a run skips; a missed seed +drops the probe's record, so it is never skipped next time. `--all` runs every probe whatever the record says, and a +release runs it before the tag; `--probe P` runs P. A seed is not part of a probe's identity, neither `seed` nor `seeds`. Without a build, `Seeds.Problems`, which the prober tests run over the repository, refuses a probe without a seed, a seed that does not apply, a seed naming an obligation its claim does not list, and a listed obligation no seed names. diff --git a/tools/Prober/Schemas/seeds-run.schema.json b/tools/Prober/Schemas/seeds-run.schema.json index 8d6a007a..824b9919 100644 --- a/tools/Prober/Schemas/seeds-run.schema.json +++ b/tools/Prober/Schemas/seeds-run.schema.json @@ -2,7 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://codometis.typekit/probe/schemas/seeds-run.schema.json", "title": "Seeds run", - "description": "State (seeds-last.json): the last `prober seeds` run on the branch, one outcome per seed (a probe without one has a single outcome, `no seed`). Never evidence: a seeded run proves the probe, not the product.", + "description": "State (seeds-last.json): the last `prober seeds` run on the branch, one outcome per seed it ran (a probe without one has a single outcome, `no seed`), the probes it skipped and why, and per probe the run that last caught every one of its seeds, which decides what the next run without --all skips (design/adr/0107). Never evidence: a seeded run proves the probe, not the product.", "type": "object", "required": [ "branch", @@ -24,6 +24,56 @@ "seconds": { "type": "number" }, + "all": { + "type": "boolean", + "description": "Whether the run was `--all`, every probe whatever its record says, as a release runs it. Absent in a file written before footprint selection." + }, + "probes": { + "type": "object", + "description": "Per probe, the run that last caught every seed it owes; a probe with a seed missed, or that never ran, has no entry, so it runs next time. Carried forward for the probes a run skipped.", + "additionalProperties": { + "type": "object", + "required": [ + "identity", + "commit" + ], + "properties": { + "identity": { + "type": "string", + "description": "The seeds identity it was caught under: the probe's identity, its seeds and the content of their patches." + }, + "commit": { + "type": "string", + "description": "HEAD of the run that caught them; the product footprint and the patched files are compared with it." + }, + "seeds": { + "type": "integer", + "description": "How many seeds were caught." + } + }, + "additionalProperties": false + } + }, + "skipped": { + "type": "array", + "description": "The probes this run skipped, unchanged since every seed of theirs was caught, each with the reason printed.", + "items": { + "type": "object", + "required": [ + "probe", + "reason" + ], + "properties": { + "probe": { + "type": "string" + }, + "reason": { + "type": "string" + } + }, + "additionalProperties": false + } + }, "seeds": { "type": "array", "items": { @@ -73,7 +123,8 @@ } }, "additionalProperties": false - } + }, + "description": "One outcome per seed this run ran; the probes it skipped have none." } }, "additionalProperties": false diff --git a/tools/Prober/Seeds.cs b/tools/Prober/Seeds.cs index 09f48509..92e5fe86 100644 --- a/tools/Prober/Seeds.cs +++ b/tools/Prober/Seeds.cs @@ -14,7 +14,9 @@ namespace Prober; /// Nothing it runs is recorded as evidence: a seeded run proves the probe, /// not the product. Never in the checkout and never in the replay's worktree; one build per seed, so it runs /// locally, not in CI. A missed seed, a probe without one, a seed that no longer applies and a seeded build -/// that fails all fail the command. +/// that fails all fail the command. Without --all it runs only the probes whose seeds identity or product +/// footprint changed since the run that last caught every seed of the probe (design/adr/0107), and says why it skips +/// each of the others; the release runs --all. /// internal static class Seeds { @@ -51,6 +53,26 @@ public static int Run(Context ctx, Args a, IProbeAdapter? adapter = null) var head = ledger.Head; var home = Replay.Home(ctx, ledger); var worktree = Path.Combine(home, "seeds"); + var include = Files.Strings(config["include"]).ToArray() is { Length: > 0 } inc ? inc : ["src"]; + // The probes whose every seed the last runs caught, each at the commit and the identity it was caught at; a probe + // named by --probe, and every probe under --all, runs whatever its record says. + var last = Files.ReadJson(ledger.P(LastFile)) as JsonObject; + var record = (last?["probes"] as JsonObject ?? []).Where(kv => kv.Value is JsonObject).ToDictionary(kv => kv.Key, kv => (JsonObject)kv.Value!.DeepClone(), StringComparer.Ordinal); + var identities = states.ToDictionary(s => s.Probe, s => SeedsIdentity(ledger, s.Probe), StringComparer.Ordinal); + var skipped = new List<(string Probe, string Why)>(); + if (!a.Flag("all") && a.Get("probe") is null) + { + var footprints = ledger.Footprints(); + var runs = new List(); + foreach (var s in states) + { + var why = Stale(ledger, s.Probe, registry[s.Probe], record.GetValueOrDefault(s.Probe), identities[s.Probe], footprints); + if (why.Run) runs.Add(s); + else skipped.Add((s.Probe, why.Reason)); + ctx.Out.WriteLine($" {(why.Run ? "runs" : "skips")} {s.Probe}: {why.Reason}"); + } + states = runs; + } var seedsOf = states.ToDictionary(s => s.Probe, s => SeedsOf(registry[s.Probe]), StringComparer.Ordinal); // A probe without a seed owes one per obligation its claim lists (decision 104), each an outcome of its own; // under a claim without obligations, one. @@ -59,7 +81,8 @@ public static int Run(Context ctx, Args a, IProbeAdapter? adapter = null) seedsOf[s.Probe].Count > 0 ? [.. seedsOf[s.Probe].Select(x => (x.Obligation, (JsonNode?)x.Seed))] : listed.GetValueOrDefault(s.Claim) is { } ids && ids is not [Claim.ImplicitObligation] ? [.. ids.Select(id => ((string?)id, (JsonNode?)null))] : [(null, null)]; - ctx.Out.WriteLine($"prober seeds: {ledger.Branch} at {head[..7]}: {seedsOf.Values.Sum(x => x.Count)} seeds of {states.Count} probes, in {worktree}"); + ctx.Out.WriteLine($"prober seeds: {ledger.Branch} at {head[..7]}: {seedsOf.Values.Sum(x => x.Count)} seeds of {states.Count} probes, in {worktree}" + + (skipped.Count > 0 ? $"; {skipped.Count} probes skipped, unchanged since every seed of theirs was caught (--all runs them)" : "")); if (a.Flag("dry-run")) { foreach (var s in states) @@ -70,69 +93,85 @@ public static int Run(Context ctx, Args a, IProbeAdapter? adapter = null) } var clock = Stopwatch.StartNew(); - Directory.CreateDirectory(home); - if (!Directory.Exists(worktree)) - { - var added = Git.Run(ledger.Root, "worktree", "add", "--detach", worktree, head); - if (added.ExitCode != 0) throw new Refusal("could not add the scratch worktree: " + added.Stderr.Trim()); - } - var include = Files.Strings(config["include"]).ToArray() is { Length: > 0 } inc ? inc : ["src"]; - var environment = ProbeRun.Environment(ctx, adapter, home); - var results = Directory.CreateTempSubdirectory("prober-seeds-").FullName; var outcomes = new List(); - try + // Nothing to run: no worktree and no build, only the record carried forward. + if (states.Count > 0) { - // The baseline: the checks each probe already fails without a seed, by row and obligation. From the ledger - // where its evidence is current, else from an unseeded run here, so a seed is credited only with violations - // of its own. - var baseline = new Dictionary>(StringComparer.Ordinal); - var unseeded = new List(); - foreach (var s in states) + Directory.CreateDirectory(home); + if (!Directory.Exists(worktree)) { - if (!s.Stale && Files.ReadJson(ledger.EvidencePath(s.Probe)) is JsonObject evidence) - baseline[s.Probe] = [.. (evidence["rows"] as JsonObject ?? []).Where(kv => kv.Value is JsonObject) - .SelectMany(kv => Ledger.ChecksOf((JsonObject)kv.Value!).Where(c => c.Category == "violation") - .Select(c => (int.Parse(kv.Key, CultureInfo.InvariantCulture), c.Obligation)))]; - else unseeded.Add(s); + var added = Git.Run(ledger.Root, "worktree", "add", "--detach", worktree, head); + if (added.ExitCode != 0) throw new Refusal("could not add the scratch worktree: " + added.Stderr.Trim()); } - ctx.Out.WriteLine($" baseline: {states.Count - unseeded.Count} from the ledger, {unseeded.Count} to run unseeded"); - if (unseeded.Count > 0) + var environment = ProbeRun.Environment(ctx, adapter, home); + var results = Directory.CreateTempSubdirectory("prober-seeds-").FullName; + try { - Reset(worktree, head); - Replay.Overlay(ledger, config, worktree); - var build = Timed(() => adapter.Build(worktree, environment), out var seconds); - ctx.Out.WriteLine($" unseeded build: {Seconds(seconds)}, {FirstLine(build)}"); - if (build.ExitCode != 0) { ctx.Out.WriteLine(build.Stdout); throw new Refusal("the unseeded build failed; no seed can be judged"); } - foreach (var s in unseeded) + // The baseline: the checks each probe already fails without a seed, by row and obligation. From the ledger + // where its evidence is current, else from an unseeded run here, so a seed is credited only with violations + // of its own. + var baseline = new Dictionary>(StringComparer.Ordinal); + var unseeded = new List(); + foreach (var s in states) { - var rows = RunProbe(adapter, worktree, s, environment, null, results, "base"); - baseline[s.Probe] = Violating(rows); - ctx.Out.WriteLine($" {s.Probe}: {rows.Count} rows, {baseline[s.Probe].Select(x => x.Row).Distinct().Count()} violations unseeded"); + if (!s.Stale && Files.ReadJson(ledger.EvidencePath(s.Probe)) is JsonObject evidence) + baseline[s.Probe] = [.. (evidence["rows"] as JsonObject ?? []).Where(kv => kv.Value is JsonObject) + .SelectMany(kv => Ledger.ChecksOf((JsonObject)kv.Value!).Where(c => c.Category == "violation") + .Select(c => (int.Parse(kv.Key, CultureInfo.InvariantCulture), c.Obligation)))]; + else unseeded.Add(s); + } + ctx.Out.WriteLine($" baseline: {states.Count - unseeded.Count} from the ledger, {unseeded.Count} to run unseeded"); + if (unseeded.Count > 0) + { + Reset(worktree, head); + Replay.Overlay(ledger, config, worktree); + var build = Timed(() => adapter.Build(worktree, environment), out var seconds); + ctx.Out.WriteLine($" unseeded build: {Seconds(seconds)}, {FirstLine(build)}"); + if (build.ExitCode != 0) { ctx.Out.WriteLine(build.Stdout); throw new Refusal("the unseeded build failed; no seed can be judged"); } + foreach (var s in unseeded) + { + var rows = RunProbe(adapter, worktree, s, environment, null, results, "base"); + baseline[s.Probe] = Violating(rows); + ctx.Out.WriteLine($" {s.Probe}: {rows.Count} rows, {baseline[s.Probe].Select(x => x.Row).Distinct().Count()} violations unseeded"); + } } - } - foreach (var s in states) - { - // A seed missing is an outcome "no seed"; a probe with several runs each under its obligation. - foreach (var (obligation, seed) in Owed(s)) + foreach (var s in states) { - var outcome = One(ledger, config, adapter, worktree, head, include, environment, results, s, obligation, seed, baseline[s.Probe]); - outcomes.Add(outcome); - ctx.Out.WriteLine($" {Named(s.Probe, obligation)} ({s.Claim}): {outcome.Result}{(outcome.Detail.Length > 0 ? ", " + outcome.Detail : "")}"); + // A seed missing is an outcome "no seed"; a probe with several runs each under its obligation. + foreach (var (obligation, seed) in Owed(s)) + { + var outcome = One(ledger, config, adapter, worktree, head, include, environment, results, s, obligation, seed, baseline[s.Probe]); + outcomes.Add(outcome); + ctx.Out.WriteLine($" {Named(s.Probe, obligation)} ({s.Claim}): {outcome.Result}{(outcome.Detail.Length > 0 ? ", " + outcome.Detail : "")}"); + } } } + finally + { + // The worktree is left at HEAD, every seed undone, for the next run to build incrementally. + Reset(worktree, head); + try { Directory.Delete(results, recursive: true); } catch (IOException) { } + } } - finally + + // A probe is recorded only where every seed it owes was caught, so a missed one runs again next time, whatever changed. + foreach (var s in states) { - // The worktree is left at HEAD, every seed undone, for the next run to build incrementally. - Reset(worktree, head); - try { Directory.Delete(results, recursive: true); } catch (IOException) { } + var own = outcomes.Where(o => o.Probe == s.Probe).ToList(); + if (own.Count > 0 && own.All(o => o.Result == "caught")) + record[s.Probe] = new JsonObject { ["identity"] = identities[s.Probe], ["commit"] = head, ["seeds"] = own.Count }; + else record.Remove(s.Probe); } + foreach (var gone in record.Keys.Where(probe => !registry.ContainsKey(probe)).ToList()) record.Remove(gone); - Files.WriteJson(ledger.P("seeds-last.json"), new JsonObject + Files.WriteJson(ledger.P(LastFile), new JsonObject { ["schemaVersion"] = Files.SchemaVersion, ["branch"] = ledger.Branch, ["commit"] = head, ["seconds"] = Math.Round(clock.Elapsed.TotalSeconds, 1), + ["all"] = a.Flag("all"), + ["probes"] = new JsonObject(record.OrderBy(kv => kv.Key, StringComparer.Ordinal).Select(kv => KeyValuePair.Create(kv.Key, (JsonNode?)kv.Value))), + ["skipped"] = new JsonArray([.. skipped.Select(x => (JsonNode)new JsonObject { ["probe"] = x.Probe, ["reason"] = x.Why })]), ["seeds"] = new JsonArray([.. outcomes.Select(o => (JsonNode)new JsonObject { ["probe"] = o.Probe, ["claim"] = o.Claim, ["obligation"] = o.Obligation, ["seed"] = o.Seed, ["result"] = o.Result, @@ -145,6 +184,52 @@ public static int Run(Context ctx, Args a, IProbeAdapter? adapter = null) return caught == outcomes.Count ? 0 : 1; } + public const string LastFile = "seeds-last.json"; + + /// + /// What a probe's seeded runs depend on besides the product: the probe's identity (: + /// its registry entry, the harness and probe sources, its rows, the axes file, the build environment, the tool), + /// its seeds, which that identity leaves out, and the content of each patch they apply. + /// + internal static string SeedsIdentity(Ledger ledger, string probe) + { + using var sha = System.Security.Cryptography.IncrementalHash.CreateHash(System.Security.Cryptography.HashAlgorithmName.SHA256); + void Add(string text) => sha.AppendData(System.Text.Encoding.UTF8.GetBytes(text + "\0")); + Add(ledger.Identity(probe)); + foreach (var (obligation, seed) in SeedsOf(ledger.Registry().GetValueOrDefault(probe) ?? [])) + { + Add(obligation ?? "-"); + Add(Files.Canonical(seed)); + var patch = Files.Str(seed["patch"]); + if (patch.Length > 0 && File.Exists(ledger.CorpusPath(patch))) sha.AppendData(File.ReadAllBytes(ledger.CorpusPath(patch))); + } + return Convert.ToHexStringLower(sha.GetHashAndReset())[..16]; + } + + /// + /// Whether a probe's seeds run without --all, and why: they run unless every seed was caught at a recorded + /// commit, under the seeds identity the probe has now, and neither the product footprint the last replay recorded for + /// it nor a file its seeds patch changed since that commit. A probe without a recorded footprint runs. + /// + internal static (bool Run, string Reason) Stale(Ledger ledger, string probe, JsonObject entry, JsonObject? caught, string identity, Dictionary footprints) + { + if (caught is null) return (true, "no run caught every seed of it yet"); + var commit = Files.Str(caught["commit"]); + var at = commit.Length >= 7 ? commit[..7] : commit; + if (Files.Str(caught["identity"]) != identity) return (true, $"its identity or its seeds changed since they were caught at {at}"); + if (!footprints.ContainsKey(probe)) return (true, "no product footprint recorded for it (a replay records one)"); + if (ledger.FootprintChangedSince(probe, commit, footprints)) return (true, $"its product footprint changed since {at}"); + var patched = SeedsOf(entry).Select(x => Files.Str(x.Seed["patch"])).Where(p => p.Length > 0 && File.Exists(ledger.CorpusPath(p))) + .SelectMany(p => Patched(File.ReadAllText(ledger.CorpusPath(p)))).Distinct(StringComparer.Ordinal).ToArray(); + if (patched.Length > 0 && Git.Run(ledger.Top, ["diff", "--quiet", commit, "HEAD", "--", .. patched]).ExitCode != 0) + return (true, $"a file its seeds patch changed since {at}"); + return (false, $"every seed caught at {at}, and neither its identity nor its product footprint changed since"); + } + + /// The files a patch changes, as its +++ b/ lines name them. + private static IEnumerable Patched(string patch) => + patch.Split('\n').Where(line => line.StartsWith("+++ b/", StringComparison.Ordinal)).Select(line => line["+++ b/".Length..].TrimEnd('\r')); + /// Applies one probe's seed, builds, runs the probe, and undoes the seed. private static Outcome One(Ledger ledger, JsonObject config, IProbeAdapter adapter, string worktree, string head, string[] include, Dictionary environment, string results, Ledger.ProbeState s, string? obligation, JsonNode? seed,