From e0b776b4af5876647eae2099149cabced9cd9d5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ricardo=20Gro=C3=9F?= Date: Sun, 18 Oct 2026 00:39:14 +0200 Subject: [PATCH] docs: how BLite applied the probing concept by hand, as a design note The document the BLite port wrote on 2026-10-18, copied unchanged into design/ so the method can be tried on this repository later without the prober, the covering arrays and the ledger. It maps each tool function to what BLite kept, replaced or dropped, reconstructs the process from the sessions that ran it, lists what it produced, and ends with an adoption guide. The design map gets its row. Co-Authored-By: Claude Fable 5.1 --- design/plan.md | 1 + design/probing-without-tooling.md | 958 ++++++++++++++++++++++++++++++ 2 files changed, 959 insertions(+) create mode 100644 design/probing-without-tooling.md diff --git a/design/plan.md b/design/plan.md index 19829ebb..9000757b 100644 --- a/design/plan.md +++ b/design/plan.md @@ -23,6 +23,7 @@ you touch; the rest is reference. | [Fix history](fix-history.md) | Every `fix:` to 2026-10-15 by defect class, finder, sibling chain and release escape: the baseline for reviewing | when the classification is run again | | [Mutation survivors, 2026-10-16](mutation-2026-10-16.md) | The first Stryker.NET run, on the Serilog satellite: counts per package, each survivor read as a missing assertion or an equivalent mutant, and the valid mutants the compile lost | one dated page per run | | [Finder benchmark, 2026-10-16](finder-benchmark-2026-10-16.md) | The 24 release escapes put back as seeds: which test, probe row or nothing detects each, totals per finder as applied, reached and detected, and the accounting every probe round and analyst run reports | one dated page per run | +| [Probing without the tooling](probing-without-tooling.md) | How BLite applied the probing concept by hand, with no prober, covering array or ledger: what each tool function became, the process as it ran, the issues and PRs it produced, and an adoption guide for trying the same here without the tooling | copied 2026-10-18; as the method is tried here | The spikes that decided something are in [spikes/](../spikes/), each with a README stating the question, the answer and the evidence. A spike's README names the decision it led to. diff --git a/design/probing-without-tooling.md b/design/probing-without-tooling.md new file mode 100644 index 00000000..5921c441 --- /dev/null +++ b/design/probing-without-tooling.md @@ -0,0 +1,958 @@ +# Probing without the tooling + +How BLite applied CodoMetis.TypeKit's probing concept by hand, with no prober, no covering arrays and no +ledger, in two days of October 2026; what it produced; and how to do the same in any repository. + +Written 2026-10-18 from the Claude Code session transcripts of the BLite work, the branches they left on +EntglDb/BLite, the issues they filed, and TypeKit's own files for contrast. Every statement carries a +source tag in square brackets; the appendix resolves each tag to a transcript position, a file path or an +issue number. Nothing here is reconstructed from memory. + +A note on names. TypeKit calls the thing a *probe*; BLite called the same thing an *oracle test*. The +document uses BLite's word where it describes BLite and TypeKit's where it describes TypeKit, and says +"catalogue test" when it means the generic idea: a test whose rows are the product of the settings a +consumer varies, each judged by an exact reference. + +--- + +## 1. The concept that was ported + +This section defines the vocabulary of TypeKit's probing in plain terms and says what each part is for. +The definitions come from TypeKit's axes file, the prober agent's instructions and the Prober README +[K1] [K3] [K2]. Knowing what each part exists *for* is what made it possible to drop the part and keep +the purpose. + +**Claim.** A sentence the project's own documents state about its behaviour, in a form a program can +check: "under these conditions, this observable result". TypeKit keeps them in `claims.json`, each with +an `id`, the `source` as `file#heading`, the surfaces it belongs to, and later `obligations` (sentences +quoted verbatim from the source) and `not-promised` entries (the sentence that makes a difference +intended) [K1 "Claim sources", "Axes"] [K2 "Files"]. A claim exists so that a test checks a promise the +project made, not an expectation the tester invented. A vague sentence ("works well with EF Core") is not +a claim and is skipped [K3 §4]. + +**Surface and axis.** An axis is a dimension a consumer can vary; its values are what they can choose. A +surface groups axes: a claim about JSON takes every JSON setting, every type shape and every position, +"because that is what a consumer can combine with it" [K3 §3]. Axes exist because, in TypeKit's own +words, "defects that survive review are combinations nobody listed: an option of a serializer, a hosting +model, a document version, a type shape. So you do not imagine the dimensions to vary. You derive them" +[K3 preamble]. TypeKit derives them two ways: *generated* axes, enumerated by reflection over every public +settable property of each options type the feature reads (`enumerate.cs`, 56 lines), and *hand-listed* +axes and a *type-shape catalogue* in a fenced `json probe-axes` block of `.claude/probe-axes.md` [K1 +"Axes"] [K8]. + +**Covering array and row.** A row sets every axis of a claim to one value. A pairwise covering array is +the smallest set of rows such that every pair of values of any two axes appears in at least one row; "a +claim with ten thousand pairs takes about three hundred rows" [K3 §6.3]. It exists to make a large +product affordable: the full product of ten axes with five values each is nearly ten million rows, the +pairwise array a few hundred. `prober rows` computes it, and "do not choose pairs yourself" [K3 §6.3]. + +**Probe.** One parameterised test per claim, fed by its rows, written in the consumer's own position (in +TypeKit, the Samples project, "the closest thing to a consumer") [K1 "Harness"]. A probe applies a row to +the consumer's configuration (options, host, declared type) and runs one *check per obligation*. A check +returns null or the failure text; it never decides what a refusal means [K1 "Harness"] [K3 preamble]. + +**Verdict, outcome, evidence.** A row ends in a category the harness assigns from the check's note key, +not from the text: pass, violation, harness error, inconclusive, expected refusal, inapplicable. "Never +match a failure's text to excuse it" [K1 "Harness"]. The replay writes one evidence line per row with its +checks, the settings that applied and the reasons; a pair counts as covered only by a pass "in a row that +was expected and executed, for the settings that actually applied, while the result's identity holds" +[K2 "Files"]. This exists so that coverage is counted by a script from facts, never by the model's +impression of what it ran. + +**Triage.** A failing row "is a question, not a finding" [K5 preamble]. TypeKit's triage skill asks four +questions in a fixed order and stops at the first that applies: a harness error; a fact about the +framework at the pinned version explains it; the claim's `not-promised` says it is intended; else a +finding, handed to the code-analyst agent for confirmation [K5 headings]. The order exists because "a +harness that compares unlike things produces the most convincing product bugs there are" [K5 preamble], +and because a wrong finding costs more than a missed one: the caller stops trusting the rounds [K3 +preamble]. + +**Seed.** A patch against `src/` that breaks what the claim promises, kept beside the probe, which the +probe was shown to catch. "A probe that passes proves something only if it can fail" [K1 "Seeds"]. Since +decision 104 a seed per *obligation*, so that a probe is shown to check each promise, not merely to be +able to fail somewhere [K1 "Seeds"]. + +**Corpus and replay.** Everything a round wrote (probes, rows, claims, registry, seeds) is tracked under +`tools/probe/corpus/`, and `prober replay` re-runs it with no model after each commit, "only for the +probes whose coverage footprint the commit changed", opening a finding when a probe that passed starts +failing [K3 preamble] [K2 "What runs where"]. The footprint is the set of `src/` files a probe executed in +its own process, measured by code coverage, plus declared `watched` paths [K2 "Files"]. The corpus exists +so that exploration is paid for once and the check is free afterwards. + +**Round, budget, lock, gate.** A round is a bounded exploration: "two claims or ten minutes of work after +the first build, whichever comes first", enforced by the script, not the model [K3 §6.7] [K4]. A lock +makes two rounds on one branch impossible [K2 "Files"]. The gate is the release check: it refuses while a +claim has uncovered pairs, unless a reviewed waiver says so [K2 "What runs where"]. These exist to keep +exploration from running away and to make "done" a word a script says [K3 §8 last paragraph]. + +**Fix trailers.** Every fix commit carries `Found-By:` and `Defect-Class:` trailers, so that six months +later the question "which finder found what" is answered by `git log` alone [K7 header] [K1]. + +--- + +## 2. What TypeKit's tooling does, and what BLite did with each function + +TypeKit's probing is about 6,600 lines of tool and harness [K6 Appendix A: "Harness size: `wc -l`, 6,617 +lines"]: `tools/Prober` (about 3,900 lines plus schemas), the `BuildVerdict` tool, `tools/probe/corpus/` +with its harness and generators, `enumerate.cs`, the agent, two skills and a hook [V105 inventory] [K8]. +BLite's whole framework is 188 lines: `Candidates.cs` (52), `OracleSeedsTests.cs` (58) and +`tools/oracle-seeds.cs` (78) [F2] [F3] [F4]. The table says, function by function, what happened to +each. + +| TypeKit function | What it does mechanically | BLite | How, and evidence | +|---|---|---|---| +| `.claude/probe-axes.md` | Prose for the writer plus a `json probe-axes` block: hand axes, ignored settings, refused values, surfaces, replay configuration, harness rules [K1] | Dropped; replaced by the test's own source | Each test's `` and `` state the contract and the ways it runs; the axes are the nested loops that build its `TheoryData` [F5 lines 1–30] [F6 lines 9–50]. A fork-only `.claude/oracle-catalogue.md` (27 lines) lists written and not-yet-written oracles with what each compares and catches [F13] | +| `tools/probe/enumerate.cs`, generated axes | Reflection over every public settable property of each options type at the pinned versions; unassigned settings reported [K1 "Axes"] | Dropped | BLite's "options" are the engine's modes (indexed or plain, sync or async, crash mode, backend, publish variant), which the writer listed by hand from the API and the README section [T2 brief] [T5 brief] [T7 report] | +| Type-shape generators (`tools/probe/generators/*.cs`) | File-based apps that emit the declarations of the shape catalogue as `*.g.cs`, held to their output by a conventions test [K1 "Harness"] | Dropped; replaced by one hand-written document type per test | `DiffDoc` with string, int, long, double, decimal, DateTime, bool, nullable and enum properties; the value lists carry the boundary cases (`-0.0`, `5m` beside `5.00m`, `"p7"` beside `"p70"`, int and long extremes, nulls) [F5 `Generate`] | +| `claims.json` with `surfaces`, `narrow`, `exclude`, `obligations`, `not-promised`, `inapplicable` | The claim catalogue the universe is built from; quotes checked verbatim against the source file; a schema refuses unknown fields [K1 "Axes"] [K2 "Files"] | Dropped | The README section *is* the claim. The brief names the sections to read and the recent fix commits whose shapes the test must cover [T2 brief]; the test's `` quotes the contract ("README 'Transactions & ACID', the write methods' docs") [F6 lines 18–27]; the issue body quotes the README line it contradicts [I260] | +| `prober universe` | Builds each claim's axes and pairs from the surfaces; prints narrowings and exclusions for a reviewer [K2] [K3 §6.2] | Dropped | Not needed without a claims file | +| `prober rows` (pairwise covering array) | The smallest row set covering every pair not yet current [K3 §6.3] | Replaced by the full product, kept small by hand | 331 shapes × 2 collections × 2 modes [F5]; 9 key generators × up to 33 checks [T5 report]; 6 seeds × 60 steps × 3 crash modes × every write [T7 report]; 49 scenarios × 3 backends × 4 publish variants [T13 report] | +| `prober new`, `prober try` | Scaffold a probe with one throwing check per obligation; run one probe's rows in the replay's worktree [K2] | Dropped | The writer copies the previous oracle's shape; the brief says "following the pattern of the two oracle tests in PR #250" and "read docs/ORACLE_TESTS.md and tests/BLite.Tests/Oracles/*.cs on that branch before writing anything" [T5 brief] | +| Harness (`Row`, `Rec.Run`, `NoteKeys`, `Cite`, `Rows.Data`, 639-line `Row.cs`) | Outcome categories from compile-checked note keys; skips tied to open findings; citations to facts and records [K1 "Harness"] [K8] | Replaced by a 52-line `Candidates` helper | Three outcomes: unlisted disagreement fails; listed with an empty issue fails labelled; listed with an issue passes and logs `SKIPPED`; a listed row that agrees fails with `candidate fixed` [F2] | +| `BuildVerdict` tool | One-line build and test verdict, pinned English output, failing tests by name [K1 "Harness"] | Dropped | `dotnet test --filter FullyQualifiedName~BLite.Tests.Oracles` and a TRX log read with a short script when the skip lines were needed [T8 07:41] [F4] | +| `prober register`, `publish`, staging | A probe is staged in the agent's memory and published into the corpus by a command; a registry entry per probe [K3 §6.5] | Dropped | The test file is committed like any test; the "registry" is the `oracles.json`-free folder `tests/BLite.Tests/Oracles/`, discovered by `OracleSeedsTests` as `*Tests.cs` [F3] | +| Replay by coverage footprint, `watched` paths | Re-run only the probes a commit touched; a footprint per probe measured by coverage [K2 "Files"] | Dropped | Every oracle test runs on every `dotnet test` and in CI; each class is kept to about a minute [T5 report "about 55 s"] [T7 report "in about a minute"] | +| Ledger (`universe.json`, `ledger.jsonl`, `findings.jsonl`, `footprints.jsonl`, `runs.jsonl`, `dismissed.jsonl`) | Per-branch state; pairs covered, findings opened and closed, dismissals as decisions [K3 §8] | Dropped; replaced by the registry, the drafts file and the issues | `Candidates` in the test, `Seeds/candidate-issues.md` with the drafts and a table mapping each family to its draft, and the GitHub issue [F9] [T4 22:26] | +| `prober round`, lock, budget | A round under caps the script enforces; one round per branch [K4] | Replaced by one agent session per oracle | Each test was one session with a written brief and a worktree; the parent held the sequencing ("hold the PRs until #250 has had its first review") [T1 06:19] | +| `prober gate`, waivers | Refuses a release while a claim has uncovered pairs [K2] | Replaced by a rule in the epic | "A section counts as stable in the README's stability matrix once its oracle is merged and its registry is empty" [I288 "Oracle coverage by library section"] | +| `prober seeds` (decisions 100, 104) | Applies each seed in a scratch worktree, builds, replays, requires a violation the unseeded probe lacks; one per obligation [K1 "Seeds"] | Kept, reduced | `OracleSeedsTests` (CI, no build: a seed exists per test, touches only `src/`, still applies) and `tools/oracle-seeds.cs` (local: scratch worktree, build, run the class, require a failure not labelled `[known candidate:`) [F3] [F4]. One seed per test, not per obligation; a second seed where the first proves only part (two in the dynamic tests, two in the AOT tool) [T9 report] [T11 report] | +| `prober status`, `gaps` | Worklist and what the corpus leaves out, with reasons [K2] | Replaced by a table in the epic | The epic's "Oracle coverage by library section": written and in review, not pushed, next, and "not covered by any oracle, stays with review" [I288] | +| `prober dismiss` | A dismissal is a decision recorded in the corpus [K3 §8] | Dropped | A row registered as intended cites the README line in the registry comment; none was needed in the end ("No README-intended rows") [T9 report] | +| `probe-triage` skill, four steps with pinned facts | Harness error; fact at the pinned version; not-promised; finding for the analyst [K5] | Kept, rewritten for the domain, fork-only | `oracle-triage`: the oracle or model is wrong; both paths agree but the reference differs (a semantics question for the README); only the indexed path diverges (an `IDX` bug); hand off to the code-analyst [F13 SKILL.md] | +| `prober` agent, `/probe` skill | A headless explorer under caps; modes replay, explore, full, status, gate, triage [K3] [K4] | Dropped | The parent session wrote a brief per oracle and started an ordinary session [T1 throughout] | +| `code-analyst` agent | Confirms a finding and lists siblings [K5 §4] | Kept as is | "The code-analyst agent is already global in your home directory and runs here today" [T1 21:11]; confirmation in practice was throwaway repro `Fact`s read against the code [T2 22:03] [T9 08:17] | +| Fix trailers | `Found-By:`, `Defect-Class:` [K7] | Kept, with BLite's classes | `RV OT IS US CA`; `IDX WAL TXN BSON LINQ MP PLT`; a `git log` one-liner counts them [F12 "Commit trailers"] | +| Checkout-guard hook on the agent | Refuses edits to the caller's checkout [K3 frontmatter] | Dropped | Worktrees per session instead [T2 header] | + +The one thing BLite added that TypeKit does not have in this form: the registry row carries the issue +number and the rule is self-cleaning. TypeKit's `r.Skip("F7", why)` ties a skip to an open finding in the +ledger [K1 "Harness"]; BLite ties it to the GitHub issue and makes the fix PR delete the row [F2] [F1 +"Candidate registry"]. + +--- + +## 3. The lightweight process, step by step, as BLite ran it + +### 3.1 The decision, and the order of entry + +The question was put on 2026-10-16 at 21:09: "I want to try out the bug-detection machinery that is +already implemented in CodoMetis.TypeKit and adopt it to an up-to-date fork of the upstream repo" [T1 +21:09]. Two minutes later the answer was: "your instinct is right about the approach, but wrong about +which part to port" [T1 21:11]. The reasoning was TypeKit's own record: + +| Finder | Fixes of 133 | Cost recorded | +|---|---|---| +| Review sessions and outside reviews | 89 | hours per session, 44 sessions | +| code-analyst agent | 10 | three runs on one day | +| prober | 3 real, 1 false | about a week, 6,617 lines of tool and harness | + +[T1 21:11, from K7 "Class by found-by": RS 69 + OR 20, CA 10, PR 3]. "Harness traps outnumbered product +defects found by probes roughly ten to one" [T1 21:11]. And TypeKit's concept note had already written +the transfer rule for exactly this case: "Two ordinary tests in the existing test project first, no +tooling: the model-based or differential test for the product's central obligation, and a round-trip or +refusal test for its second, each with a seed that shows it can fail. If they find nothing in a week, +stop; the rest is not worth installing there. If they find what the profile predicts, they have earned +the obligations file, the seeds and the corpus" [K6 §12]. The session quoted this and added why BLite +would do better than TypeKit: "BLite has exact ones where TypeKit had to construct a twin": LINQ-to-Objects +for the query provider, a dictionary for the store, round trips for BSON and encryption [T1 21:11]. + +What to keep was decided the same evening, after the first results: "exact oracles with a seed each, the +rule that every guard is shown to fail, fix trailers that make finders countable from git log, a fixed +triage order, and the code-analyst agent. What did not pay off at its cost: pairwise covering arrays, +axes enumerated by reflection, generated type-shape harnesses, the ledger with locks and rounds, the +obligations schema and the gate. The first list is a few hundred lines. The second is six thousand, and it +is TypeKit-shaped" [T1 21:55]. The split by location followed from the repository being someone else's: +"only tests and plain conventions go there. Agents and skills stay in your fork and your home directory" +[T1 21:55]. + +### 3.2 Who derives the axes, from what, and where they are written + +Nobody enumerated settings by reflection. The axes of each test were chosen by the session that wrote +it, from three inputs the brief named: the README sections that make the promise, the public API's modes, +and the recent fix commits whose defect shapes the test must reach [T2 brief: "Read README.md 'Advanced +Indexing', 'Powerful Query Engine (LINQ)' and the BLQL sections for what is promised; look at the recent +upstream fix commits (7dbe21d, f8f437b, 33c6bbf, ffdaaba, 7e37f33, 70eb3c3) for the defect shapes this +must cover"]. The axes are written nowhere but in the test file, as the loops that build the theory data, +and in the row's name, which carries its coordinates. The documentation then describes them in prose +[F1 "What it is"]. + +What the axes were, per oracle: + +| Oracle | Axis 1: the case | Axis 2: the ways | Axis 3 | Reference | Source | +|---|---|---|---|---|---| +| LINQ differential | 331 hand-listed query shapes, grouped by property type and operator (`count == 5`, `count < 9.5 (double bound)`, `count two lower bounds`, `orderby maybe take 20 (nulls)`) | indexed collection, plain collection | sync, async | LINQ-to-Objects over the seeded list | [F5 `BuildCatalogue`] [T2 report] | +| Store model | 8 seeds of 300 random steps over 13 operation kinds | single-process, multi-process | plaintext, encrypted (added later) | dictionary with committed and pending views | [F6] [T2 report] | +| B-tree model | 9 key generators (int, long and double extremes, prefix strings, control bytes, keys up to the size limit, unique, multi-process) | up to 33 checks (each query primitive, cursor walks, structure, page accounting) | | sorted dictionary in the index's own key order, plus "reachable pages plus free list plus queued frees equals pages allocated" | [T5 report] [F1] | +| WAL crash | the store model's sequences, 6 seeds of 60 steps | 3 crash modes (process crash, torn write, power loss) | 8 checks; every write a crash point | the store model's committed view after reopen, plus the page invariant | [T7 report] [F8] | +| DynamicCollection | the same 331 shapes | dynamic indexed, dynamic plain | API, BLQL | LINQ-to-Objects **and** the typed collection over the same documents | [T9 brief and report] [F14] | +| Native AOT | the consumer's sections (store sequence, encryption, DynamicCollection with CDC and BLQL, key-value store, BLite.Caching) | JIT, Native AOT | the publish log | the JIT dump, byte for byte; the publish log against the document's promise | [T11 report] [F10] | +| Browser | 49 scenarios (later 54) | OPFS, IndexedDB, in-memory | 4 publish variants; later 3 browsers and 2 context modes | the scenario's own contract (a committed write survives reload; two tabs keep both writes) | [T13 report] [T15 report] | + +Two details show how the axes were kept honest without a script. First, the generators were guarded: the +B-tree test fails if a key generator did not force what it was meant to force ("Every generator gave pages +back at least 4 times, and the longest duplicate runs held 133–388 entries per key"; "generator guards" +counted among the seed's failures) [T5 report]. Second, where a way cannot express a case, the row says so +rather than passing: "Six query shapes BLQL can't express are listed with their reason and print +`INAPPLICABLE`", and a new one fails a test until it is listed [T9 report] [F14 lines 43–44]. + +The value lists were chosen for boundaries, not breadth: counts `[-7, -1, 0, 0, 1, 2, 3, 5, 5, 5, …, +100]`, ratios including `-0.0`, `0.0`, `1e300`, `double.Epsilon`, prices with `5m` and `5.00m`, nicks with +two nulls and an empty string, and exactly one document at each integer extreme "for Single and for bound +tests" [F5 `Generate`]. Documents are inserted in a shuffled order "so that no index order is the scan +order" [F5 `Fixture`]. + +### 3.3 How claims were collected and tagged + +There is no claims file. The unit of a claim is a README section, and the unit of coverage is "one +oracle for every library section the documentation makes a promise about" [I288]. The tagging that +TypeKit does with `surfaces` happened in three places: + +1. **In the brief.** The parent session named the sections and the API the oracle must read before + writing ("Read README.md 'Transactions & ACID', 'Hot Backup', 'Multi-Process WAL' and docs on + checkpointing for what is promised") [T7 brief]. +2. **In the test's remarks.** The store model's `` begins "The contract the model encodes (README + 'Transactions & ACID', the write methods' docs):" and then states it as rules: a write without a + transaction commits on its own; a `BLiteValidationException` leaves the transaction usable while any + other failure rolls it back; a checkpoint, a backup and a reopen change nothing [F6 lines 18–27]. +3. **In the issue.** The body quotes the sentence the behaviour contradicts, with its line: `README + "Transactions & ACID" (README.md:394): "Durable: WAL ensures data safety even in power loss."` [I260]. + +The parent session named this as one of the three reasons it worked: "Claims as the test list. Every +test started from a sentence the repository makes: durable in power loss, the index is used +transparently, this API is AOT-safe, BLQL supports these operators. The oracle checks the sentence, which +is why the findings read as broken promises rather than opinions" [T1 08:48]. + +### 3.4 How a probe was written + +**File layout.** `tests/BLite.Tests/Oracles/Tests.cs` in namespace `BLite.Tests.Oracles`; entity +contexts in `tests/BLite.Shared/`; seeds in `Oracles/Seeds/Tests..patch`; drafts in +`Oracles/Seeds/candidate-issues.md`; a two-line `Oracles/README.md` pointing at `docs/ORACLE_TESTS.md` +[F1 "How to use it"] [T3 report step 1]. Tools that cannot be unit tests (a Native AOT publish, a +browser) live under `tools/aot-smoke/` and `tests/BLite.Wasm.BrowserTests/` with the same conventions: +a `candidates.md` table the runner reads, a `seeds/` directory, a `--prove-seeds` mode [F10] [F11]. + +**Test framework.** xunit 2.9.3, as the repository already had; a `[Theory]` over +`[MemberData(nameof(Cases))]` where `Cases` is a `TheoryData` of row names, and the test looks the +case up by name [F5 lines 561–580]. The model tests use `TheoryData` built by nested +loops over seeds, process mode and encryption [F6 lines 40–50]. Nothing was added to the test project's +references. + +**Shape of the theory body.** For each way the case runs, run it, capture the value *or the exception* +as an `Outcome` ("both are compared"), compare against the reference with the case's own rule (sets by id +for unordered queries, key sequences for ordered ones, values or the exception type for terminals), +collect one line per disagreement naming the way (`indexed/async: …`), and hand the lines to the registry: + +```csharp +s_candidates.Verify(name, errors.Count == 0 ? null + : $"seed {_f.Seed}: query '{name}' disagrees with LINQ-to-Objects:\n " + string.Join("\n ", errors), _output); +``` + +[F5 lines 561–586]. The failure text carries the seed and the row name, so a run can be repeated +(`BLITE_DIFF_SEED=`) [F1 "How to use it"]. + +**Shared pieces.** The store model's generator and model were moved into `StoreModel.cs` when the WAL +test needed the same sequences; the dynamic tests reused the differential catalogue by extracting it; the +AOT consumer copied the generator [T7 report] [T9 08:00] [T11 08:17]. The page-accounting walk of the WAL +test was made `internal` for the dynamic model test "rather than copied" [T9 08:07]. + +**Local tools follow the same shape.** The AOT tool's `candidates.md` has a machine-readable table at the +top (`Id | Kind | Match | Issue | Summary`) and the drafts below it; `run.sh` prints `KNOWN … [known +candidate: A]` for a matching warning or dump line and fails on anything else [F10]. The browser +harness's `Candidates.cs` reads the same kind of table, with a `Kind` of `always` or `sometimes`, because +"a pass proves nothing" for a row that fails only in some runs [F11 Candidates.cs]. + +**Naming and endings.** Row names are human sentences with the operand in them (`count < 9.5 (double +bound)`), family labels state the hypothesis (`null equality on an indexed property finds nothing`) [F5 +lines 136–155]. Upstream uses CRLF, the user's global git has `autocrlf=input`, so every new file was +converted and staged with `git -c core.autocrlf=false add`; this cost time in every session and is in +every brief [T4 report] [T5 brief]. + +### 3.5 How the covering array, or its substitute, was chosen + +BLite ran the full product every time and kept it small by choosing values, not by choosing pairs. +The sizes: + +| Oracle | Rows per run | Run time | Source | +|---|---|---|---| +| LINQ differential | 331 cases, each 4 ways | not recorded on its own; the whole oracle filter of the first branch (both tests) ran inside the full suite's few minutes | [T2 report] [T4 report] | +| Store model | 16 rows (8 seeds × 2 modes), 300 steps each | about 3 s plaintext; longer encrypted | [T1 21:45] | +| B-tree model | 270 tests (9 generators × up to 33 checks) | about 55 s | [T5 report] | +| WAL crash | 24 rows (3 modes × 8 checks), 1,886 crash states | about a minute | [T7 report] | +| DynamicCollection | 3,335 tests | 1 min 16 s | [T16 11:02] | +| Encrypted, whole oracle set | 680 tests | 3 min 2 s | [T16 11:07] | +| AOT tool | 871 dump lines, two publishes | about 20 s; 37 s with seeds | [T11 report] | +| Browser | 53–54 rows | about 5 minutes per browser | [T15 brief] | + +Where the product would have been too large, it was sampled by a rule stated in the code, not by a +pairwise algorithm: the WAL test crashes recovery only at "sampled crash states" ("every third such step, +or every one when shrinking") [F8 line 252] [T7 report]; the browser harness publishes four variants but +runs the same 49 scenarios in each [T13 report]. The parent session's position on pairwise arrays was +explicit: "If the catalogue of query shapes grows past what a hand-written list holds, a pairwise +generator is a fifty-line addition to the test, decided then" [T1 21:55]. It was not needed in the two +days. + +### 3.6 How a run was budgeted + +There was no round budget in the TypeKit sense. The units of budget were: + +- **One session per oracle**, with a written brief that fixed the scope, the files to read first, the + base branch, the seed recipe and "done means" [T2 brief] [T5 brief] [T7 brief]. Sessions took 30 to 55 + minutes each (3.4 below). The brief forbade pushing until the maintainer had reviewed the first PR, so + the sequencing was the parent's, not the tool's [T5 brief last sentence]. +- **Per-class run time**, held to about a minute so the class can stay in the normal suite [T5 report] + [T7 report]. +- **Shrink budgets** inside a failing row: 45 s for the store model, 20 s then 10 s for the WAL test, and a + 60 s watchdog around the B-tree walk after a seeded run hung "for over 10 minutes inside an engine + traversal" [F6 line 88] [F8 line 100] [T5 report]. +- **Drafts before the session ends.** Every writing session ended with drafts for what it found, so that a + filing session could run without re-deriving anything [T5 report] [T7 report] [T9 report]. + +### 3.7 How a failing row was judged and shrunk + +**The oracle judges.** A row fails only when its reference disagrees; the writer writes no expectation. +The three outcomes are the registry's, quoted in full from the helper: "Passes a row that agrees and is +not listed, or that disagrees and is listed with an issue; fails every other row" [F2 `Verify` doc +comment]. A listed row that agrees fails with `candidate fixed: remove the entry for ''`, "so the +list never goes stale" [F2 class comment]. + +**Shrinking.** The model tests shrink a failing sequence by removing chunks "half the sequence, then +quarters, … down to single steps", replaying after each removal, within the budget, and print the +shortest sequence that still fails, one operation per line, with the seed and the step [F6 lines 164–192] +[F1 "Read a failure"]. The WAL test shrinks to "the shortest sequence that still fails at a crash in its +last step" and prints the crash point as "crash after FreePage(24): page 24 written during step 3" [F9 +WalCrashTests seed header]. A shrunk sequence is the repro that goes into the draft and the issue [I260 +"Repro"]. + +**Confirmation before filing.** Every family was reproduced outside the oracle, as "throwaway repro Facts +for D1–D9 (not to be committed) so every draft's repro is actually run, not just reasoned" [T9 08:17], or +"standalone repros for the drafts, including what each defect does at the collection level" [T5 23:13], +and the mechanism was read in the code with `file:line` [I249 "Mechanism (verified on 0450b97)"]. + +**The triage order**, as BLite's skill states it, in the order the differential test makes natural [F13 +SKILL.md]: + +1. Is the oracle or the model wrong? Cases that stopped here: the store model expected + `InvalidOperationException` for a duplicate key where BLite throws `BLiteValidationException` and + leaves the transaction active [T2 report]; the AOT consumer's CDC count "was my arithmetic (50 inserts, + not 51)" [T11 08:22]; the dynamic model's rule that "an explicit null counts as a value for the unique + index" [T9 report]. +2. Do both engine paths agree with each other but differ from the reference? Then it is a semantics + question to settle against the README before anything is filed. The `!=` family was this: both paths + drop null-valued documents, LINQ keeps them; the issue "should ask rather than assert" [T1 22:09]. +3. Does only the indexed path diverge? "That is a bug with no further question: the index serves a + different answer than a scan of the same documents" [F1 "Triage"]. +4. Hand the family to the code-analyst for confirmation and siblings; file only when confirmed. + +**A correction that shows why the order matters.** The parent session first reported that all fourteen +rows diverged on the indexed path only; a rerun "with a complete filter" showed its own filter had +dropped the plain collection's lines, and two rows were in fact wrong on both paths [T1 22:09]. The +lesson it drew is TypeKit's first triage rule in different words: read every way's line before deciding +which step applies. + +### 3.8 How a finding became an issue + +The route, in the order it was run every time: + +1. **Family label, issue empty.** The rows go under one label in the test's registry with an empty issue, + so they keep failing, "so nothing is quarantined before it is filed" [F1 "Candidate registry"]. +2. **Draft.** `Seeds/candidate-issues.md` gets a section per family: rows, repro (three or four documents, + the plain collection as control), expected, actual, hypothesis with code references, and at the top a + table mapping family label to draft number [F9 candidate-issues.md head]. +3. **Cross-check.** Open issues first; after the WAL round, closed issues and PRs too, with `git log -S` + for the commit that introduced the behaviour: "Extend step 1 before filing: cross-check W1–W3 against + CLOSED issues and PRs too, and reference them in the bodies" [T8 cross-session message]. That check + rewrote W1's history (the fsync went missing in three steps between v1.0.0 and v4.0.0, not in the commit + the draft had named) and reframed half of W2 as a trade-off two merged PRs had already accepted [T8 + 07:51]. +4. **Group by root cause.** D6 and D7 became one issue "since both live in `BsonValueToIndexKey` / + `CreateIndexKeyFromObject`"; A3, A4, A6 and A7 became one issue, the pragma pattern; BW2, BW3, BW4 and + BW7 became one, the blocking wait [T10 08:45] [T12 report] [F11 candidates.md]. Families 1 and 4 of the + differential shared a cause and a decision, so one issue [T1 22:12]. +5. **File**, with the labels the repository has (`bug`, `package/core`, `area/indexing` or `area/linq`, + `severity/*` where it is clear; `encryption`, `package/generator`, `package/client` where they apply) + [T6 report] [T12 report]. +6. **Number into the registry**, commit, run the oracle filter and confirm every family's rows print + `SKIPPED` at the draft's row counts and nothing says `candidate fixed` [T10 08:50]. +7. **Comment on siblings** (#237, #234, #254 got one line each saying the test reproduces them and carries + them as known) and link the test PR from each issue [T6 report] [T10 report]. +8. **Fix PR removes the entry** and carries `Found-By: OT` and `Defect-Class:` [F12]. + +**The issue body**, as the filed issues have it [I249] [I260] [I273]: + +- First paragraph: what fails in one sentence, then "Found by (, ), which carries the + affected rows as a known candidate until this is fixed." +- The README or document sentence contradicted, quoted with its line. +- `## Repro`: the shrunk sequence or three documents and the call, with expected and actual as comments; + the `dotnet test` command and how to make the row fail again ("With the issue number removed from the + family in `s_candidates`, the row fails and prints the seed, the crash point and the shortest sequence"). +- `**Expected:**` / `**Actual:**`. +- `## Mechanism (verified on )`: the code path with `file:line` for every step. +- `## History` where a decision is behind the behaviour: commits, dates, versions, the design note that + intended something else. +- `**Affected versions:**` from `git describe --contains` of the introducing commit; "Not reproduced on + hardware; the test models a power loss as …" where that is true. +- `## Fix` or `## Fix considerations`: options, framed as the maintainer's choice where it is a trade-off + ("This is a durability/latency trade-off for the maintainers to decide"), with the per-commit cost + stated; "Happy to send the fix if you want it." +- `## Workaround` where one exists. +- `Related:` the sibling issues and PRs, and the other findings of the same test. + +### 3.9 How candidates that are not fixed yet are kept + +The registry lives in the test that owns the rows, not in one central file, and it is the only +quarantine. Its rules, verbatim from the documentation [F1 "Candidate registry"]: + +- "it still disagrees and its issue is **filed**: the row passes and writes `SKIPPED [known candidate: + , ] ` to the test output (xunit 2 has no dynamic skip, so this line is the + trace a run shows);" +- "it still disagrees and its issue is **empty**: the row keeps failing with `[known candidate: + ]`, so nothing is quarantined before it is filed;" +- "it **agrees**: the row fails with `candidate fixed: remove the entry for ''`, so the fix's PR + removes the entry and the registry cannot outlive the defect." + +A `Fact` per test fails on an entry that names no row, "for example after a row is renamed" [F1] [F2 +`AssertAllIn`]. For tools outside xunit the same table lives in `candidates.md`, with a `Kind` column so a +row that fails only sometimes is never reported as fixed [F11 Candidates.cs]. Already-known issues go in +with their number from the start, even in a session told to leave issues empty: "That departs from your +'empty Issue' instruction, because the registry's own rule is that a filed issue goes in the entry" [T5 +report]. The registry is the thing the maintainer sees: "a quarantine that is visible, classified, and +self-cleaning, which is the one piece of the replay machinery worth keeping" [T1 21:55]. + +### 3.10 How the corpus is kept alive without per-commit replay + +- **It is the test suite.** The oracle tests are ordinary tests in `tests/BLite.Tests`; CI runs them on + every push with no extra job. The cost of "replay everything" is the per-class minute of 3.5 [F1 "Run + the oracle tests"]. +- **Seeds stay applicable by a CI test without a build**, and are re-proven locally before a release by + the tool [F3] [F4] [F1 "Seeds"]. "A refactor that strands a seed fails CI instead of leaving the oracle + unproven" [F1]. +- **The registry cleans itself** through the `candidate fixed` rule, and the fix PR is the one that removes + the row [F1]. +- **Growth is by catalogue, not by test.** The documentation states the rule per oracle: "A new LINQ + operator or terminal is one more `QueryCase` in the `BuildCatalogue` method"; "A new index key type or + key shape is one more key generator"; "A new collection operation or storage feature is one more + `StoreOpKind`" and "once it is a `StoreOpKind`, one more step the injector of `WalCrashTests.cs` + interrupts"; "A new public API is one more call in `tools/aot-smoke/Consumer`" [F1 "How detection grows + with the feature set"]. +- **What is written and what is next** is a table in the epic, with the rule that makes it a gate by + convention: stable in the stability matrix only when the oracle is merged and its registry is empty + [I288]. The fork keeps the same list as `.claude/oracle-catalogue.md`, with the instruction "Add a row + to 'Written' in the same commit that adds the test and its seed" [F13]. +- **Stacked branches until merged.** Because every oracle depends on the machinery of the first PR, the + test PRs were stacked (#250 ← #259 ← #263 ← {#272, #277} ← #294) and the parent held them until the + maintainer's first review [T1 06:19] [T1 11:22]. A late commit on the bottom branch then had to be + merged up through every branch by hand [T16 report]; see 4.3. + +--- + +## 4. What it produced in BLite + +### 4.1 The issues + +28 issues, 27 of them straight from failing rows and one found while fixing another; plus one draft not +filed. Labels as filed; every one also carries the repository's default `triage` label [G1]. + +| Issue | One line | Found by | Labels beyond `bug`, `package/core` | +|---|---|---|---| +| #247 | `== null` and `OrderBy(nullable).Take(n)` served from a typed index return no null-keyed documents | LINQ differential, families 1 and 4 | area/indexing, area/linq | +| #248 | `x.Prop != constant` does not match documents whose property is null; the evaluator has no branch for a stored null | LINQ differential, family 2 | area/linq | +| #249 | Indexed double equality misses `-0.0`: the key encoding gives the two zeros different keys | LINQ differential, family 3 | area/indexing | +| #254 | DynamicCollection range filters and sorts on an indexed double are wrong: keys are little-endian `BitConverter` bytes | noticed in #249's body, filed while fixing it | area/indexing | +| #255 | `BTreeIndex.Equal/In/GreaterThan/LessThan/Between/StartsWith` miss duplicate keys and fail past the last key | B-tree model, B1 | area/indexing | +| #256 | Equal keys come back in reverse insertion order on a forward scan | B-tree model, B2 | area/indexing | +| #257 | A NaN with the sign bit clear sorts above `+Infinity` in an index on a double | B-tree model, B3 | area/indexing | +| #258 | Every reopen strands the unwritten rest of the last page growth block | B-tree model, B4 (page invariant) | area/indexing | +| #260 | WAL commits are never flushed to disk: a power loss drops acknowledged commits and can leave documents pointing at freed pages | WAL crash, W1 | severity/high | +| #261 | A crash leaks pages freed or allocated outside the WAL | WAL crash, W2 | area/indexing | +| #262 | A crash between the two page writes of `FreePage`/`AllocatePage` leaves a Free page off the free list | WAL crash, W3 | area/indexing | +| #264 | BLQL filters and sorts cannot reach a nested field, though an index can be built on one | Dynamic differential, D1 | area/linq | +| #265 | BLQL orders and compares ObjectIds by their little-endian bytes | Dynamic differential, D2 | area/linq | +| #266 | BLQL `First`/`FirstOrDefault` ignore Sort and Skip | Dynamic differential, D3 | area/linq | +| #267 | BLQL range operators match fields of another BSON type: null under `$lt`, strings under `$gt` | Dynamic differential, D4 | area/linq | +| #268 | BLQL compares two Int64 values through double | Dynamic differential, D5 | area/linq | +| #269 | A dynamic index misses keys for a numeric bound of another BSON type and keys no Decimal128 or Boolean value | Dynamic differential, D6 + D7 | area/indexing | +| #270 | `QueryIndexAsync` rejects a DateTime bound | Dynamic differential, D8 | area/indexing | +| #271 | A failed unique `CreateIndexAsync` leaves the half-filled index registered | Dynamic store model, D9 | area/indexing | +| #273 | Typed collections cannot write under Native AOT: the generated mapper registers its BSON keys by reflection that trimming removes | AOT twin, A1 | severity/critical, package/generator | +| #274 | Encrypted database: a page allocated by a transaction that never commits cannot be decrypted afterwards | AOT twin, A2 (JIT too) | severity/high, encryption | +| #275 | `#pragma warning disable IL…` hides reflection from BLite's build but not from the consumer's publish | AOT twin, A3 + A4 + A6 + A7 | | +| #276 | BLite.Caching uses reflection-based System.Text.Json: throws in any trimmed or AOT app | AOT twin, A5 | package/client | +| #289 | BLite.Wasm: a committed write is lost on a page reload unless a checkpoint ran | Browser, BW1 | severity/critical | +| #290 | BLite.Wasm: synchronous waits block or kill the browser runtime | Browser, BW2 + BW3 + BW4 + BW7 | severity/high | +| #291 | BLite.Wasm: two tabs on one database lose a committed write (IndexedDB) or cannot open it (OPFS) | Browser, BW5 | area/multi-process | +| #292 | BLite.Wasm: a 50 MB database cannot be reopened; `CheckpointAsync` returns at once while a background checkpoint runs | Browser, BW6 | severity/medium | +| #293 | BLite.Wasm: the package lacks its JS files and offers no typed-context factory | Browser, BW8 + BW9 | | +| BW10 | WebKit without persistent storage has no OPFS and `Auto` picks it anyway; drafted, not filed | Browser on WebKit | | + +[G1] [T1 22:37, 06:08, 08:07, 08:55, 09:05, 11:22] [T15 report]. Two things the tally should not hide: the +store model found nothing on upstream HEAD in its first run, "which is also information: the +collection-level contract holds in one process without crashes" [T1 06:18]; and three of the issues are +design questions as much as bugs, filed as questions (#247, the `!=` half of #248's framing, #260's sync +mode) [T1 06:18] [T1 22:12]. + +Alongside the issues: 7 test PRs (#250, #259, #263, #272, #277, #294, #295), 12 fix PRs (#251–#253, +#278–#286), Discussion #287 and epic #288 with the coverage table [G2] [T1 11:22]. + +### 4.2 The effort + +Wall clock from the first question to the last PR of the stack: 2026-10-16 21:09 to 2026-10-17 11:22, +about 14 hours including a night with no activity between 23:48 and 06:00 [T1 timestamps]. The sessions, +each a brief from the parent and a worktree: + +| Session | Start | End | Result | +|---|---|---|---| +| Decision and orchestration (parent) | 16th 21:09 | 17th 12:00 | briefs, filing of the first three issues and PR #250, relays | +| First two tests [T2] | 21:14 | 21:44 | differential (14 rows in four families) and store model, two seeds | +| Triage of the four families | 21:54 | 22:03 | standalone repros, drafts | +| Machinery [T3] | 21:58 | 22:08 | Oracles folder, seeds test, registry, CONTRIBUTING, seeds tool, fork-only skill and catalogue | +| Assemble and document [T4] | 22:15 | 22:26 | skip-while-open rule, CRLF, `docs/ORACLE_TESTS.md` | +| Fix PRs #251–#253 | 22:40 | 23:16 | three PRs, #254 filed | +| B-tree model [T5] | 22:53 | 23:47 | 73 rows in four new families, two known reproduced | +| B-tree filing [T6] | 06:00 | 06:08 | #255–#258, PR #259 | +| WAL crash [T7] | 06:22 | 07:16 | three families, 1,886 crash states | +| WAL filing [T8] | 07:37 | 07:51 | #260–#262, PR #263, history cross-check | +| DynamicCollection [T9] | 07:47 | 08:32 | nine drafts, 252 candidate rows | +| Native AOT [T11] | 08:12 | 08:51 | seven candidates, two seeds | +| Dynamic filing [T10] | 08:43 | 08:54 | #264–#271, PR #272 | +| AOT filing [T12] | 08:58 | 09:05 | #273–#276, PR #277 | +| Fix round 2 | 09:08 | 09:59 | PRs #278–#286 | +| Browser harness [T13], in parallel | about 08:57 | 10:23 | 49 scenarios, 20 failing, nine findings | +| Restack [T16] | 10:58 | 11:19 | five merge commits, PR #294 | +| Browser into the frame [T14] | about 11:00 | 11:22 | registry, seed, #289–#293, PR #295 | +| Firefox and WebKit [T15] | 11:22 | 12:17 | BW10 drafted, persistent-context mode | + +Code written on the top of the stack, excluding the dynamic, AOT and browser branches: `BTreeModelTests` +1,419 lines, `WalCrashTests` 1,325, `DifferentialQueryTests` 839, `StoreModel.cs` 428, `StoreModelTests` +193, `candidate-issues.md` 744, `docs/ORACLE_TESTS.md` 243, the four seed patches 231, the three framework +files 188 [F-wc]. The dynamic tests add 1,684 [F14 wc]. The browser harness is two projects [T13 report]. + +### 4.3 What went wrong, what it missed, and the lessons the transcripts record + +**Mistakes made and caught.** + +- The parent misreported the `!=` family as indexed-only because its own output filter dropped the plain + collection's lines; corrected after a rerun with a complete filter [T1 22:09]. Lesson: read every way's + line before classifying. +- A seeded B-tree run hung for over ten minutes inside an engine traversal of a corrupted tree. The test + gained its own cycle-safe walk before any engine traversal and a 60 s watchdog, "so a broken tree is + reported as a `Structure` failure instead of a hang" [T5 report]. +- Two tests backed up into the same temp folder because the backup manifest has a fixed name; the fix + landed as a late commit on the bottom branch and broke the mergeability of the four stacked PRs above + it, which then needed a merge commit each, one with a conflict [T9 08:25] [T1 10:56] [T16 report]. + Lesson: a stack is cheap to build and expensive to move; land the base before stacking, or expect one + restack. +- A `sed` whose delimiter clashed with `#277` emptied four issue bodies for a minute; restored, visible in + the edit history [T12 report]. +- The bottom branch had been rewritten upstream with new commit ids and the same tree; the B-tree branch + was replayed onto it and force-pushed with lease before its PR existed [T6 report]. +- zsh did not word-split a label variable, so an issue was created without labels on the first try [T6 + 06:01]; zsh also mangled a browser configuration string [T15 11:36]. +- The registry's "skip" is a logged pass in xunit 2.9.3, "so the quarantine is invisible in CI counts" + [T1 22:27] [T1 06:18]. +- CRLF: every file needed converting and staging with `core.autocrlf=false`; the seed patches had to stay + LF because a CRLF copy no longer applied [T4 report]. +- Pre-existing concurrency tests flake under the load of the full suite; every full run was followed by + reruns in isolation [T16 11:12] [T5 brief]. +- The AOT publish needed `LIBRARY_PATH` for Homebrew's OpenSSL and the `xcrun` clang shim; both are handled + in `run.sh` [T11 08:26, 08:37]. + +**What the method cannot see**, as its own documentation states: targets other than `net10.0`, real +multi-process races, crashes in multi-process mode and inside `Backup`, torn page writes, performance, +"and anything whose correct answer has no exact reference, such as API design and documentation drift. +Those stay with review" [F1 last paragraph]. Two concrete misses: #212 (keys not prefix-free) "does not +show at this level: a raw `BTreeIndex` key has no document id appended", so the oracle sat below the +defect [T5 report]; and the first-run claim that the store contract held was limited to one process with +no crashes, which the WAL test then contradicted under power loss [T1 06:18] [T7 report]. + +**Lessons the transcripts record**, in their words where possible: + +1. "Exact oracles. LINQ-to-Objects, a sorted dictionary, the committed view of a model, the JIT build of + the same program. None of them is an approximation, so no finding needed an argument about what the + right answer was" [T1 08:48]. "That is what the principle 'model budget goes where the oracle is + weakest' predicts: an exact oracle plus deep state is where reading finds least and programs find + most" [T1 21:11]. +2. "Claims as the test list" [T1 08:48] (3.3 above). +3. "Seeds. Each test was shown to fail before it was trusted, so none of the twenty-six is a test that + happens to pass or fail for reasons unrelated to the product" [T1 08:48]. The seeds tool had to count + only unlabelled failures, because a test with registered candidates "would always pass" the naive + check [T3 report step 5]. +4. "Findings make the case for the machinery" and "agents and workflows never go into someone else's + repository on the strength of an invitation" [T1 21:11] [T1 22:12 "What not to send yet"]. +5. Hold the test PRs until the maintainer has seen the pattern once, "so the maintainer sees the pattern + once before it multiplies" [T1 22:50]; but file a durability finding at once, because "the issue stands + on its own without the test PR" [T1 07:18]. +6. Cross-check closed issues and PRs and the commit that introduced the behaviour; "an issue that cites + that history and frames the fix as a trade-off lands better with the maintainer than one that reads as + a discovery" [T1 07:40]. +7. One issue per root cause, and ask rather than assert where the README is silent [T1 22:12] [T10 + 08:45]. +8. The registry must be self-cleaning and visible; a deleted row loses the guard [T1 22:12 step 2]. +9. "The method finds what has an exact reference. API misuse that compiles, documentation drift and + unclear error messages are invisible to it and stay with review. The TypeKit fix history showed exactly + that split, and it holds here too" [T1 08:48]. +10. A side effect worth naming: several of the later issues were found by review or by running the + README samples for the v6 documentation work, not by oracles (#299 "while compiling and running the + README samples for #298"; #301 "while checking the secure-erase behaviour for #299"; #304, #305 and + #307 "while reviewing … for #296"), and two of them (#304, #305) propose the oracle row that would + hold them [G1 bodies]. The catalogue tests created a habit of asking "which oracle would hold this". + +--- + +## 5. Adoption guide for any repository + +### 5.1 Prerequisites + +- A test project the repository already runs in CI, in whatever framework it uses. Nothing here needs a + particular one; the templates below are given for xunit (BLite) and MSTest (ValueRanges). +- At least one **exact reference** for a central promise: a simpler implementation of the same + computation (LINQ-to-Objects, a dictionary, set theory over a small universe, PostgreSQL itself), an + invariant (pages reachable plus free equals allocated), a twin build (JIT against AOT), or an identity + (round trip). If the only reference you can think of is "what the code did last week", stop: that is a + snapshot test, and this method will produce noise [K6 §12 "Which oracle a domain gets"]. +- Documents that make promises: a README, a docs folder, a limitations file. The promise is the claim. +- Issues with labels, so a finding has somewhere to go that is not a list in a file. +- `git apply` and a scratch worktree, for seeds. + +### 5.2 The artefacts, with templates + +Create them in this order. Each one is small; none depends on a tool. + +**1. The first catalogue test**, with its seed, before anything else. Pick the section with the densest +recent bug history and an exact reference ("the last 35 upstream commits are almost all index/LINQ/WAL +correctness fixes found one at a time" [T2 brief]). Write it in the consumer's position. + +xunit skeleton, the shape of `DifferentialQueryTests` [F5]: + +```csharp +public class DifferentialTests : IClassFixture<DifferentialTests.Fixture> +{ + // Fixture: seeded data (override with an environment variable), one instance per "way". + public sealed class Fixture : IDisposable { /* seed, data, the ways */ } + + // Known candidates: family label (the hypothesis), issue (empty until filed), the row names. + private static readonly Candidates s_candidates = new() + { + { "", "", "", "" }, + }; + + // The catalogue: one named case per shape. The name carries the operand. + private static readonly Dictionary s_cases = BuildCatalogue(); + public static TheoryData Cases => new(s_cases.Keys); + + [Fact] public void Candidates_NameExistingRows() => s_candidates.AssertAllIn(s_cases.Keys); + [Fact] public void Fixture_ReachesWhatTheAxesPromise() { /* guards: indexes exist, splits happened, … */ } + + [Theory, MemberData(nameof(Cases))] + public async Task Case_AgreesWithTheReference(string name) + { + var c = s_cases[name]; + var errors = new List(); + foreach (var (way, run) in Ways()) // the product of the hand-listed axes + { + var outcome = await Guard(() => run(c)); // value or exception, both compared + var error = c.Compare(Reference(c), outcome); // the case's own rule: set, sequence, scalar + if (error != null) errors.Add($"{way}: {error}"); + } + s_candidates.Verify(name, errors.Count == 0 ? null + : $"seed {Seed}: '{name}' disagrees with the reference:\n " + string.Join("\n ", errors), _output); + } +} +``` + +MSTest skeleton, for a repository that already has a `Candidates.Verdict` over a list of failure lines +(as ValueRanges does [V-cand]): + +```csharp +[TestClass] +public class ProbeTests +{ + public static IEnumerable Rows() + { + foreach (var type in DiscoveredTypes()) + foreach (var declaredAs in new[] { "union", "variant", "interface", "object" }) + foreach (var options in OptionsCatalogue()) + yield return [type, declaredAs, options.Name]; + } + + [DataTestMethod, DynamicData(nameof(Rows), DynamicDataSourceType.Method)] + public void Row_AgreesWithTheReference(Type type, string declaredAs, string optionsName) { /* one row */ } + + // Or one sweep method that collects every disagreement and ends in the verdict, + // which is how the existing small-model sweeps are written: + [TestMethod] + public void Sweep() + { + var failures = new List(); + foreach (var row in Rows()) { var line = Check(row); if (line != null) failures.Add(line); } + Candidates.Verdict(oracle: nameof(ProbeTests), failures, header: "", method: nameof(Sweep)); + } +} +``` + +The row's failure line must start with a stable prefix (the operands and the operation, never the actual +value), because the registry matches by that prefix [V-cand remarks]. + +**2. The seed**, in the same commit. Revert a known fix restricted to `src/`, or plant a defect by hand; +run the test; confirm it fails *for the defect*; save the diff with a header [F1 "Add a new oracle test"] +[F9 seed header]: + +```text +Seeded defect for tests//Oracles/Tests.cs + + (#)>: . + +With it applied, rows that pass on fail, e.g. , ( of ). shrinks to: + 1. + 2. + + +Reproduce: git apply && dotnet test --filter +Restore: git checkout -- src + +Never commit the seeded state. + +diff --git a/src/... b/src/... +``` + +**3. The registry helper** (52 lines in BLite [F2]); if the repository already has one, reuse it. The +three outcomes are the contract; the implementation is incidental. For a framework with a real dynamic +skip, use it; otherwise log the skip and say so in CONTRIBUTING. + +**4. The seeds test** (58 lines [F3]): every `*Tests.cs` under the oracle folder has a seed; every seed +touches only `src/` (`git apply --numstat`); every seed still applies (`git apply --check`). No build. + +**5. The seeds prover** (78 lines [F4]): for each seed, a detached scratch worktree of HEAD, apply, build, +run the class, read the results file, require at least one failure whose message does not contain the +known-candidate label; print `CAUGHT`, `MISSED` or `ERROR`; exit 1 unless all caught; remove the +worktree. Local only. + +**6. The drafts file**, `Seeds/candidate-issues.md`: a status line at the top, a table from family label +to draft number and row count, then one section per draft [F9]: + +````markdown +## . + +**Rows:** `<row>`, `<row>`, … + +**Repro:** +```<language> +<three or four inputs; the control path; expected and actual as comments> +``` + +**Expected:** <what the reference and the control path say>. +**Actual:** <what the path under test says>. + +**Hypothesis** (checked against the code): <mechanism with file:line>. +**Affected versions:** <from git describe --contains of the introducing commit>. +**Existing issues checked:** <numbers, and why none covers it>. +```` + +**7. The issue body template**, the structure of 3.8 reduced to headings: + +```markdown +<One sentence: what fails.> Found by <test> (`<path>`, <PR>), which carries the affected rows as a known +candidate until this is fixed. + +<Document> "<section>" (`<file>:<line>`): "<the promise, quoted>." + +## Repro +<shrunk sequence or inputs and call; expected and actual as comments; the test command; how to make the +row fail again> + +**Expected:** … +**Actual:** … + +## Mechanism (verified on <sha>) +- <step, file:line> + +## History <!-- only where a decision is behind the behaviour --> +- <commit, date, version: what it changed and why, quoting the design note> + +**Affected versions:** <range>. <What was not reproduced on real hardware, if anything.> + +## Fix considerations +<options; the trade-off named as the maintainer's call; cost stated; "happy to send the fix"> + +## Workaround <!-- where one exists --> + +Related: <siblings, closed fixes it neighbours, the other findings of the same test> +``` + +**8. The registry row**, as a C# entry or a table row: + +```csharp +{ "<family label stating the hypothesis>", "#<issue or empty>", "<row>", "<row>" }, +``` + +```markdown +| <Id> | always or sometimes | `<test method or row pattern>` | `<substring of the failure>` | #<issue or empty> | <summary> | +``` + +**9. CONTRIBUTING section and trailers** (about 22 lines in BLite [T3 report step 4]): what an oracle test +is, that every one ships a seed, how the registry behaves, and the two trailers with the repository's own +defect classes. + +**10. The method document**, `docs/ORACLE_TESTS.md` or its equivalent, in four parts: why it exists (the +bug history it answers), what it is (each mechanism in one paragraph), how to use it (run, read a +failure, triage order, register, retire, add a test), how detection grows with the feature set (one +sentence per catalogue), and what it does not see [F1 headings] [T1 22:21]. + +**11. The catalogue of oracles**, written and not yet written, each with what it compares, what it would +catch, and a seed idea (27 lines in BLite's fork; a table in the epic upstream) [F13] [I288]. This is the +axes file reduced to what a human needs: where the next oracle goes. + +**12. The triage skill or checklist**, four questions in the order the oracle makes natural, each with +"applies when" and "then", and a table of what each step leaves behind in the registry, the issue and the +commit [F13 SKILL.md]. In a repository without agents, this is a section of the method document. + +### 5.3 Fitting it to an existing suite and an existing claim catalogue + +- **If the repository already has claims** (a `claims.json` that quotes document sentences and names the + test holding each, as ValueRanges has [V-claims]), do not write a second list. A catalogue test names + the claim ids it holds in their `heldBy`; the axes it varies go in a short hand-written file beside the + claims (`test/oracles/axes.md`), one section per surface family, each axis with its values, the test + that already varies it, and the pairs no test varies [V105 comment]. The unheld claims and the uncovered + pairs are the worklist that TypeKit's `status` and `gaps` would print. +- **If it already has a registry and a seed convention** (`candidates.json`, `seeds/*.patch`, a prover), + register the new class where the others are registered and give it a seed; do not add a second + mechanism [V-oracles]. +- **If it has sweeps that already vary one axis each** (every culture, every type, every shape pair), the + catalogue test's job is the *combinations* those sweeps do not reach: declared type × serializer + option, entry point × culture × type, one literal across every parser against the external system's + own answer [V105 comment]. List the existing sweeps as the first row of each axis so the new test does + not repeat them. +- **Put each class in the project that already has its harness**: the conventions project for type + discovery, the web test project for `TestServer`, the integration project for a live server [V105 + comment]; BLite put the AOT and browser tools beside the tests because a publish takes minutes and needs + a toolchain [F1 "Native AOT smoke test"]. + +### 5.4 Keeping it cheap + +- A class runs in about a minute; above that, sample by a rule written in the code (every third crash + state), or split by mode so one slow axis does not multiply the rest [F8 line 252]. +- Shrink under a budget and with a watchdog; print the shortest sequence, the seed and the step, so the + repro is the failure message [F6 lines 164–192] [T5 report]. +- Shared generators and models: one `StoreModel.cs` fed four oracles and a tool [T7 report] [T9 08:07] + [T11 08:17]. +- One session, one oracle, one brief; drafts before the session ends; a separate short session files [4.2]. +- No tooling until the hand-written list hurts; then "a pairwise generator is a fifty-line addition to the + test" [T1 21:55]. Replay by footprint is worth building only when the suite's minute becomes an hour. +- Do not send agents, hooks or skills into a repository you do not own; send tests, a document and a + CONTRIBUTING section [T1 21:55] [T1 22:12]. + +### 5.5 Failure modes to avoid + +1. **An inexact reference.** "A harness that compares unlike things produces the most convincing bugs + there are" [K5] [F13]. Step 1 of triage exists for this; three of BLite's first disagreements were the + model's [3.7]. +2. **Reading one way's output and deciding.** The parent's filter error [T1 22:09]. +3. **A seed that proves the wrong thing.** A test with registered candidates always "fails", so the prover + must ignore labelled failures [T3 report]; a seed that breaks some assertion shows the test can fail, + not that the claim is checked [K1 "Seeds", decision 104]; a seed whose build fails proves nothing [K1]. +4. **A registry that cannot go stale, going stale.** Without the `candidate fixed` rule the quarantine + outlives the defect; without the `AssertAllIn` check a renamed row leaves a ghost entry [F2]. +5. **Quarantine by deletion.** "Quarantining by deleting rows loses the guard" [T1 22:12]. +6. **Filing before confirming, or before checking closed issues.** Both were caught in BLite by an explicit + step [T8 cross-session message] [T9 08:17]; TypeKit's record of a false prober finding is why [K7]. +7. **Assertions in the issue where the maintainer owns the decision.** Frame semantics and trade-offs as + questions with options and costs [I260 "Fix considerations"]. +8. **A catalogue that grows as new tests instead of new rows.** The growth rules in the method document + are what keep one new case checking against everything old [F1 "How detection grows"]. +9. **Oracles at the wrong level.** A defect in the layer above the oracle's seam does not show (#212 at the + raw B-tree) [T5 report]; a tool that steps aside (the WAL recorder during `Backup`) leaves a documented + hole [T7 report]. Say what each oracle does not see. +10. **Mechanical traps that cost hours:** line endings, shell word splitting, a `sed` delimiter, a stack + that moves, flaky neighbours under load [4.3]. Put each in the brief of the next session. + +--- + +## 6. Appendix: sources + +### 6.1 Source key + +Transcripts are JSON Lines files under `~/.claude/projects/`; a position is the assistant message's +timestamp on 2026-10-16 or 2026-10-17 as printed in the file. The `local_…` id is the desktop app's +session id, for `search_session_transcripts`. + +| Tag | What | Where | +|---|---|---| +| T1 | The adoption and orchestration session ("TypeKit bug-detection adoption", `local_5f13179c-c4e6-4a17-a5a5-76adc4a039ee`) | `-Users-ricardo-gross-RiderProjects-BLite/2fff4cfa-ef17-42b6-9be3-0c572e0b2aa1.jsonl` | +| T2 | First two oracle tests (`local_0ff03db5-0549-47d3-8b13-d3468efee6ba`) | `-Users-ricardo-gross-RiderProjects-BLite--claude-worktrees-awesome-nobel-c18118/da25b03b-e4b5-4e8a-a3ef-cd38f4ac2917.jsonl` | +| T3 | Minimal machinery (`local_7e91d830-d86b-45b6-b7e1-410856125ace`) | `…--claude-worktrees-amazing-shirley-d6a17b/29e567f2-7168-4b21-9853-2daf71f9c86a.jsonl` | +| T4 | Assemble the upstream branch and write the doc (`local_f348875c-7895-417b-b6a7-f41aebb94b03`) | `…--claude-worktrees-bold-jemison-ab4112/d11cfd8c-ecac-4e34-b91d-72826be4677b.jsonl` | +| T5 | B-tree model test (`local_59590bd2-73d2-4b18-aadc-c29124f62c46`) | `…--claude-worktrees-gallant-dhawan-099ff2/d7e134a2-0062-4103-bb54-b0fc3aefb69d.jsonl` | +| T6 | B-tree issues and PR (`local_0bcf26c6-c767-4fea-ad0d-c9dc8f7a3d96`) | `-Users-ricardo-gross-RiderProjects-BLite/aaa634d7-5790-4cab-8f8c-6d0e8df33e80.jsonl` | +| T7 | WAL crash test (`local_8c7a6120-02cd-4ae6-8d90-6c70111de988`) | `…--claude-worktrees-nice-kepler-e43635/237a51b1-e924-4cae-be95-87c7d8571796.jsonl` | +| T8 | WAL issues and PR (`local_299441f5-a412-408c-b780-e74a8d2da9d2`) | `-Users-ricardo-gross-RiderProjects-BLite/d93cd37d-94a9-4a59-a781-9b8b97ba6700.jsonl` | +| T9 | DynamicCollection tests (`local_1662d46c-7300-4b1b-957d-92b109d821ba`) | `-Users-ricardo-gross-RiderProjects-BLite/f8230ab4-f7f2-4dff-983a-638cddf35031.jsonl` | +| T10 | DynamicCollection issues and PR (`local_cd76b880-e49d-4abe-9c07-6f01b45b3b77`) | `-Users-ricardo-gross-RiderProjects-BLite/e548a07a-2716-4d2f-96ca-a86923239b18.jsonl` | +| T11 | Native AOT smoke test (`local_86a39a6c-aa47-4858-85a8-7fd9077afc1d`) | `…--claude-worktrees-compassionate-swirles-586c00/7c9352d3-753f-430b-91c1-0f9e3551203e.jsonl` | +| T12 | AOT issues and PR (`local_01a36223-675a-40ae-bacc-518cd341b617`) | `-Users-ricardo-gross-RiderProjects-BLite/e4803599-52d8-4f4c-9944-c64b4537f29e.jsonl` | +| T13 | Browser harness built (`local_43b916c1-3f2a-4db3-abd1-9c0ac45ca64a`) | `-Users-ricardo-gross-RiderProjects-BLite/1dbc59f3-97f4-4d6b-b2a9-ff8956ed7b19.jsonl` | +| T14 | Browser harness into the oracle frame (`local_9f981451-221d-4943-9140-ec0280ac0083`) | `-Users-ricardo-gross-RiderProjects-BLite/be98704b-fd11-4efa-8eed-54621ef1d1d0.jsonl` | +| T15 | Firefox and WebKit runs (`local_a2d52883-225a-4309-b283-95dc2dad45b8`) | `…--claude-worktrees-festive-villani-1d94c1/d956d69f-3ff8-4422-a5d0-7a557bd9c862.jsonl` | +| T16 | Restack of the PR stack (`local_0edf32a2-08d7-488b-bad6-eb13fd916412`) | `-Users-ricardo-gross-RiderProjects-BLite/c02d43e2-c1b2-4108-8c2a-9cae41be9363.jsonl` | +| F1 | `docs/ORACLE_TESTS.md`, 243 lines | EntglDb/BLite branch `test/oracle-encrypted` at `0de4379` (local worktree `BLite/.claude/worktrees/eloquent-mcclintock-d7bc82`) | +| F2 | `tests/BLite.Tests/Oracles/Candidates.cs`, 52 lines | same | +| F3 | `tests/BLite.Tests/Oracles/OracleSeedsTests.cs`, 58 lines | same | +| F4 | `tools/oracle-seeds.cs`, 78 lines | same | +| F5 | `tests/BLite.Tests/Oracles/DifferentialQueryTests.cs`, 839 lines | same | +| F6 | `tests/BLite.Tests/Oracles/StoreModelTests.cs` (193) and `StoreModel.cs` (428) | same | +| F7 | `tests/BLite.Tests/Oracles/BTreeModelTests.cs`, 1,419 lines | same | +| F8 | `tests/BLite.Tests/Oracles/WalCrashTests.cs`, 1,325 lines | same | +| F9 | `tests/BLite.Tests/Oracles/Seeds/*.patch` (four) and `candidate-issues.md` (744 lines) | same | +| F10 | `tools/aot-smoke/candidates.md`, `run.sh`, `README.md` | same branch (merged up from `test/oracle-aot`) | +| F11 | `tests/BLite.Wasm.BrowserTests/Candidates.cs`, `candidates.md`, `FINDINGS.md` | branch `test/wasm-browser-harness` at `f95a63e` (worktree `wasm-browser-harness`) | +| F12 | `CONTRIBUTING.md`, "Oracle tests" and "Commit trailers" | branch `test/oracle-encrypted` | +| F13 | `.claude/oracle-catalogue.md` and `.claude/skills/oracle-triage/SKILL.md` | fork-only commit `23f7182` on branch `claude/amazing-shirley-d6a17b` (worktree `amazing-shirley-d6a17b`) | +| F14 | `tests/BLite.Tests/Oracles/DynamicDifferentialTests.cs` (887), `DynamicStoreModelTests.cs` (797) | branch `test/oracle-dynamic` (worktree `oracle-dynamic`) | +| F-wc | Line counts | `wc -l` in the `eloquent-mcclintock-d7bc82` worktree on 2026-10-18 | +| K1 | TypeKit `.claude/probe-axes.md`, 751 lines | `/Users/ricardo.gross/RiderProjects/CodoMetis.TypeKit` | +| K2 | TypeKit `tools/Prober/README.md` | same | +| K3 | The `prober` agent definition | `~/.claude/agents/prober.md` | +| K4 | The `probe` skill | `~/.claude/skills/probe/SKILL.md` | +| K5 | TypeKit `.claude/skills/probe-triage/SKILL.md`, 261 lines | TypeKit checkout | +| K6 | TypeKit concept note, section 12 and Appendix A | `.claude/agent-memory-local/concept-2026-10-16-agentic-issue-finding.md` | +| K7 | TypeKit `design/fix-history.md`, "Class by found-by" | TypeKit checkout | +| K8 | Sizes of `tools/Prober/*.cs`, `tools/probe/enumerate.cs`, `tools/probe/corpus/harness/Row.cs` | `wc -l` in the TypeKit checkout on 2026-10-18 | +| I<n> | Issue n on EntglDb/BLite, read with `gh issue view` | GitHub | +| G1 | `gh issue list --repo EntglDb/BLite --state all` and the bodies of the user's issues since 2026-10-16 | run 2026-10-18 | +| G2 | `gh pr list --repo EntglDb/BLite --state all --author CaffeinatedCoder` | run 2026-10-18 | +| V105 | CodoMetis.ValueRanges issue #105, body and the 2026-10-18 comment | GitHub | +| V-claims, V-oracles, V-cand | `test/oracles/claims.json`, `oracles.json`, `Candidates.cs` in ValueRanges | this checkout | + +### 6.2 Statement index + +Section 1: every definition is from K1 ("Harness", "Seeds", "Claim sources", "Axes"), K2 ("What runs +where", "Files"), K3 (preamble, §3, §4, §6, §8) and K5 (preamble, headings), as tagged inline. The pairwise +size ("about three hundred rows") is K3 §6.3. The 56-line enumerator and 639-line `Row.cs` are K8. + +Section 2: the 6,617 figure is K6 Appendix A; the 188 figure is F2 + F3 + F4; every table row carries its +tags. The three-outcome rule is F2; the stability-matrix rule is I288; the BLQL `INAPPLICABLE` list is T9 +report and F14 lines 43–44. + +Section 3.1: T1 at 21:09, 21:11, 21:55; K6 §12; K7 table. +Section 3.2: T2 brief and report; T5 brief and report; T7 report; T9 brief and report; T11 report; T13 and +T15 reports; F5 `Generate` and `Fixture`; F14. +Section 3.3: T7 brief; F6 remarks; I260; I288; T1 08:48. +Section 3.4: F1 "How to use it"; T3 report; F5 lines 561–586; F6 lines 40–50; T7, T9, T11 as tagged; F10; +F11; T4 report; T5 brief. +Section 3.5: T2 report; T1 21:45; T5 report; T7 report; T16 11:02 and 11:07; T11 report; T15 brief; F8 +line 252; T13 report; T1 21:55. +Section 3.6: T2, T5, T7 briefs; F6 line 88; F8 line 100; T5 report. +Section 3.7: F2; F6 lines 164–192; F1 "Read a failure" and "Triage"; F9 WAL seed header; I260; T9 08:17; +T5 23:13; I249; F13; T2 report; T11 08:22; T9 report; T1 22:09. +Section 3.8: F1 "Candidate registry"; F9; T8 cross-session message and 07:51; T10 08:45; T12 report; F11; +T1 22:12; T6 report; T10 08:50 and report; F12; I249, I260, I273. +Section 3.9: F1 "Candidate registry"; F2; F11; T5 report; T1 21:55. +Section 3.10: F1 as tagged; F3; F4; I288; F13; T1 06:19 and 11:22; T16 report. +Section 4.1: G1 (titles and labels); T1 at 22:37, 06:08, 08:07, 08:55, 09:05, 11:22; T15 report; T1 06:18 +and 22:12; G2. +Section 4.2: the timestamps of the first and last user or assistant message of each transcript as listed +in the key; F-wc; F14; T13 report. +Section 4.3: T1 22:09; T5 report; T9 08:25; T1 10:56; T16 report; T12 report; T6 report and 06:01; T15 +11:36; T1 22:27 and 06:18; T4 report; T16 11:12; T11 08:26 and 08:37; F1 last paragraph; T7 report; T1 +08:48, 21:11, 22:12, 22:50, 07:18, 07:40; T10 08:45; T3 report; G1 bodies of #299, #301, #304, #305, #307. +Section 5: templates are reduced from F5, F2, F3, F4, F9, I249/I260/I273, F11, F12, F1, F13; the +ValueRanges fit is from V105's comment and V-claims, V-oracles, V-cand; the failure modes cite the +incidents of 4.3 and K1, K5, K7.