Consumer probing: port TypeKit's prober tooling (route A), or BLite's lightweight way with hand-written axes and MSTest probe sweeps (route B) #105

Activity

1. CaffeinatedCoder

   added

   testsAdds or changes tests only (oracles, harnesses); no library codeAdds or changes tests only (oracles, harnesses); no library code

   on Oct 17, 2026
2. CaffeinatedCoder

   added a parent issue on Oct 17, 2026

   • Epic: Backlog #83
3. CaffeinatedCoder commented on Oct 17, 2026

   

   CaffeinatedCoder

   on Oct 17, 2026

   OwnerAuthor

   More actions

   The lightweight alternative: how BLite did it without the prober

   Evidence: the BLite session transcripts of 2026-10-16/17 (the "TypeKit bug-detection adoption" session and its twelve side sessions), the upstream branches test/oracle-tests … test/oracle-encrypted and test/wasm-browser-harness, and the issues on EntglDb/BLite.

   What BLite decided, and why

   On 2026-10-16 the question was the same as this issue's: port TypeKit's bug-finding machinery. The session read TypeKit's own fix history (133 fixes: 89 from review sessions, 10 from the code-analyst agent, 3 real plus 1 false from the prober at about a week and 6,617 lines of tool and harness; harness traps outnumbered product defects roughly ten to one) and recommended against lifting the prober. It kept the parts of the method that had paid off and dropped the tooling:

   Kept (a few hundred lines, by hand)	Dropped (about 6,000 lines, TypeKit-shaped)
   Documented claims as the test list	.claude/probe-axes.md and axes enumerated by reflection (enumerate.cs)
   One catalogue test per documented section with an exact oracle	Pairwise covering arrays (prober rows)
   A seed patch per test, proven by local revert	Obligations schema, not-promised, inapplicable, exclude
   A self-cleaning candidate registry keyed to issue numbers	Ledger, round lock, budget, prober round
   A fixed triage order (a skill, fork-only)	Replay only where a commit changed a probe's coverage footprint
   Found-By: / Defect-Class: commit trailers	Verdict tool, release gate, prober agent, evidence per row
   The global code-analyst agent for confirmation

   The method, step by step

   1. Axes are chosen by hand, per test, by whoever writes it, from the README section's promise and the modes the API offers. They are written as nested enumeration inside the test file (a TheoryData of named cases × the "ways" a case runs), and the row's name carries its coordinates. The product is complete, not pairwise, because it stays small enough to run in minutes:
      • LINQ differential: 331 hand-listed query shapes (W("count < 9.5 (double bound)", x => x.Count < 9.5)) × {indexed, plain} × {sync, async}, against LINQ-to-Objects.
      • B-tree model: key generator (int, long, double extremes, prefix strings, control bytes, max-size keys, unique, multi-process) × each query primitive, against a sorted dictionary plus a page-accounting invariant; 620 rows.
      • WAL crash: the store model's sequences × three crash modes × every write as a crash point; about 1,900 crash states.
      • DynamicCollection: the same 331 shapes × {dynamic indexed, dynamic plain} × {API, BLQL}, against LINQ-to-Objects and the typed collection; 3,335 rows; shapes BLQL cannot express print INAPPLICABLE with a reason and must be listed.
      • Native AOT: consumer sections × {JIT, AOT}, the JIT dump and the publish log as oracles (a local tool, not a test).
      • Browser: 49 scenarios × {OPFS, IndexedDB, in-memory} × four publish variants × three browsers.
   2. A row is judged by its oracle, never by the writer. The three outcomes come from a 52-line Candidates helper: a row that disagrees and is unlisted fails; listed with an empty issue it fails with [known candidate: <family>]; listed with an issue it passes and logs SKIPPED (xunit 2 has no dynamic skip); a listed row that agrees fails with candidate fixed: remove the entry.
   3. A failing row becomes an issue through a fixed route: triage in order (oracle or model wrong → both paths agree but differ from the reference, a semantics question for the README → only the indexed path diverges, a bug) → family label in the registry with an empty issue → a draft in Seeds/candidate-issues.md (rows, repro, expected and actual, mechanism with file:line, affected versions, fix options) → repro run as a throwaway Fact and confirmed from the code → cross-check of open and closed issues and PRs and the README → filed with labels → number into the registry → the fix PR removes the entry and carries Found-By: OT.
   4. Each test ships a seed (Seeds/<Test>.<defect>.patch, a revert of a known fix or a hand-made defect, header says what it breaks). OracleSeedsTests (58 lines) holds in CI without a build that every test has one, it touches only src/, and it still applies; tools/oracle-seeds.cs (78 lines) proves locally that each is caught.
   5. Artefacts: tests/BLite.Tests/Oracles/*Tests.cs, Seeds/*.patch, Candidates.cs, OracleSeedsTests.cs, tools/oracle-seeds.cs, docs/ORACLE_TESTS.md (243 lines: why, what, how to use, how detection grows), CONTRIBUTING trailers. Fork-only and never sent upstream: .claude/oracle-catalogue.md (27 lines, written and not-yet-written oracles) and .claude/skills/oracle-triage/SKILL.md.

   What it produced

   28 issues in about 14 wall-clock hours (2026-10-16 21:09 to 2026-10-17 11:22, including a night), 27 of them straight from failing rows:

   Test	Issues
   LINQ differential (first run, about one hour after start)	#247, #248, #249 (#254 found while fixing #249)
   B-tree model with page accounting	#255, #256, #257, #258
   WAL crash injection	#260, #261, #262 (the WAL was never fsynced)
   DynamicCollection differential and model	#264 to #271
   Native AOT twin	#273 (typed collections cannot write under AOT), #274, #275, #276
   Browser harness	#289 to #293

   Plus 7 test PRs (#250, #259, #263, #272, #277, #294, #295), 12 fix PRs (#251–#253, #278–#286), epic #288 with an "oracle coverage by library section" table and a planned README stability matrix. Cost in code: about 5,600 lines on the stacked branch (two of the test files are over 1,300 lines each), plus the dynamic tests, the AOT tool and the browser harness; framework code is 188 lines in total.

   What it gave up compared with the tooling

   • No replay by coverage footprint: every run runs everything, so each catalogue is kept to minutes.
   • No pairwise arrays: the full product, pruned by hand when it grows (the dynamic differential is already 3,335 rows).
   • No verdict by lookup: triage is a judgement in a fixed order, recorded in the registry and the issue, not in a ledger.
   • No gate and no obligations per sentence: the test's <remarks> states the contract; the release rule is "registry empty and oracle merged" per section.
   • The skip is a logged pass in xunit 2, invisible in CI counts.

   Where ValueRanges stands, and the lightweight route here

   ValueRanges already has BLite's machinery, adopted from it in #54: test/oracles/claims.json (126 claims, 2 unheld), oracles.json (28 sweeps), 24 seeds, candidates.json (8 rows, all with an issue), Candidates.Verdict with the same three outcomes, tools/oracles.cs seeds|status, trailers, docs/oracles.md. What this repository does not have is the layer BLite's catalogue tests supply: consumer-shaped tests whose rows are the product of the settings a consumer varies. The 10.0.0 bug hunt found #114 ("every entry point"), #116 (variant type × minimal API) and #117 (infinity spelling × another writer) by exactly such combinations, as one-off probes that were not kept.

   So the lightweight route is not a port; it is three additions to what exists:

   1. A hand-written axes list, test/oracles/axes.md (next to claims.json; about 60 lines; not .claude/probe-axes.md, which the /probe skill would read and then stop on for lack of tools/Prober). One section per surface family of claims.json (literals and JSON, TypeConverter and binding, EF Core, hosting and AOT), each axis with its values and the test that already varies it, and the pairs no test varies. Values come from the existing sweeps where they exist: five cultures (LiteralCultureContractTests), shape pairs (ShapeMatrixParityTests), every type and shape (JsonDefaultContractTests), Npgsql.DisableDateTimeInfinityConversions (the InfinityDisabled project).
   2. Probe classes as ordinary MSTest sweeps, *ProbeTests, [DataTestMethod] + [DynamicData] over the hand-written product, ending in Candidates.Verdict, registered in oracles.json with a seed like every other oracle, and named in the heldBy of the claims they cover. Each class lives in the project that already has the harness: the conventions project for type discovery, AspNetCore.Tests for TestServer, the integration project for a live server. Three first candidates, in order of what the bug hunt showed:
      • declared type × JSON options: every range and set type × {union, sealed variant, interface, object, record property, dictionary value} × {default, camelCase, snake_case, ReferenceHandler.Preserve, source-generated context, NumberHandling}, oracle: the wire form equals the literal (claims J*);
      • entry point × culture × type: {Parse, TypeConverter, configuration binding, MVC [FromQuery], minimal API, JSON body} × {invariant, de-DE, th-TH} × every type including a variant-typed parameter, oracle: one value on every path or a loud refusal (Minimal API binds a variant-typed parameter as null for a valid literal and answers 200 #116 is one missing pair of this product);
      • literal corpus × entry point against PostgreSQL's own answer: {inverted, quoted empty, whitespace, infinity spellings, 24:00:00, >28 digits} × every parse entry point, oracle: psql -At on the same text (the bug hunt's second technique; An inverted range literal parses as empty on every entry point; PostgreSQL rejects it #114 and Decimal text beyond 28 significant digits is rounded silently when a literal or JSON array is read #115 are rows of it).
   3. Findings and triage unchanged: candidates.json with the issue, found-by/oracle, the fix PR removes the row. Add BLite's fixed triage order to docs/oracles.md as one more step ("both paths agree with each other but not with the reference: a semantics question for the docs before an issue").

   Effort, from BLite's record: about one agent session per probe class including seed and drafts (400 to 900 lines each), and an hour for the axes list. Against this issue's inventory: 3 to 4 PRs and about 5,000 lines before the first round. What the lightweight route forgoes is listed above; the one that matters here is replay by footprint, so each probe class has to stay under about a minute, and the Docker-bound one runs with the integration suite.
4. CaffeinatedCoder

   changed the title [-]Port the probe tooling from CodoMetis.TypeKit (MSTest adapter) so /probe can run here[/-] [+]Consumer probing: port TypeKit's prober tooling (route A), or BLite's lightweight way with hand-written axes and MSTest probe sweeps (route B)[/+] on Oct 17, 2026
5. CaffeinatedCoder

   added a sub-issue on Oct 17, 2026

   • Route B, step 1: hand-written axes list and the JSON options probe sweep #121
6. CaffeinatedCoder commented on Oct 17, 2026

   

   CaffeinatedCoder

   on Oct 17, 2026

   OwnerAuthor

   More actions

   Route B, step 1 is implemented in #121 (PR open): test/oracles/axes.md and JsonOptionsProbeTests. Steps 2 and 3 (entry point × culture × type in AspNetCore.Tests; literal corpus × entry point against PostgreSQL in the integration project) remain; axes.md names them under "pairs no test varies".
7. CaffeinatedCoder

   added sub-issues on Oct 17, 2026

   • Route B, step 3: literal corpus × entry point against PostgreSQL #123
   • Route B, step 2: entry point × culture × type probe sweep #124
8. CaffeinatedCoder commented on Oct 17, 2026

   

   CaffeinatedCoder

   on Oct 17, 2026

   OwnerAuthor

   More actions

   Step 2 done in #125 (issue #124): entry point × culture × type.

   • CultureEntryPointProbeTests (conventions project): every discovered type, as union and variant. It reads and writes through Parse, TryParse, TypeDescriptor, configuration binding, JSON and interpolation, under four current cultures. That is 52 rows and ~10,300 checks in 0.3 s.
   • HostCultureProbeTests (AspNetCore project): 107 declared types, bound from a query, nullable query, route value, form field and JSON body. It covers MVC and a minimal API, under four request cultures set by RequestLocalization. That is 52 rows and ~12,400 requests in ~3 s.
   • Seed: parser-reads-the-current-culture. Both classes catch it. LiteralProviderContractTests catches it only on a machine whose own culture isn't invariant.
   • Pairs now varied (moved in axes.md): entry point × culture × type, current culture × JSON, host × request culture, configuration × every type, host × variant-typed parameter.
   • Findings: none new. The only disagreement is Minimal API binds a variant-typed parameter as null for a valid literal and answers 200 #116, measured wider: every variant of every range type, through all four TryParse sources of a minimal API, answers 200 with null for the right shape and for another shape alike. It's registered as 11 candidates against Minimal API binds a variant-typed parameter as null for a valid literal and answers 200 #116. The culture played no part in any row.

   Still open in axes.md:

   • the literal corpus × entry point (step 3, in progress)
   • a NodaTime element converter × declared type
   • options × dictionary key
   • a host × source-generated context
   • MVC × AllowInputFormatterExceptionMessages
9. CaffeinatedCoder commented on Oct 17, 2026

   

   CaffeinatedCoder

   on Oct 17, 2026

   Last edited by CaffeinatedCoder

   OwnerAuthor

   More actions

   Every bug issue against the oracles on main (3ecfa10)

   This checks all 16 bug issues, open and closed, against every class in oracles.json. Each issue now has a comment with the class, the row, and how I verified it. Run means:

   • for a closed issue: the fix's src/ diff was reverse-applied on 3ecfa10, then every registered oracle was run, live PostgreSQL included.
   • for an open issue: the oracles were run on 3ecfa10, where the defect is still present.

   For each reverted miss, the fix's own regression tests failed, which shows the revert put the defect back. Labels follow the decision of 2026-10-18. A catch gets found-by/oracle: "added" marks a retroactive catch (the issue was found some other way), and "already" marks an issue the machinery found at the time. A miss gets no label.

   Issue	State	Verdict	Verification	Label
   #45	closed	catch: RangeLiteralGrammarContractTests, LiteralGrammarParityTests	run (revert #73)	added
   #70	closed	catch: SetElementWhiteSpaceContractTests, LiteralGrammarParityTests	run (revert #81)	added
   #53	open	catch: DecorativeElementContractTests (5 candidate rows)	run	already
   #78	open	catch: SetElementWhiteSpaceContractTests, LiteralGrammarParityTests (162 candidate lines)	run	added
   #106	open	catch: DifferentialFuzzParityTests (8 lines, sampled)	run	added
   #109	open	catch: DifferentialFuzzParityTests (354 lines)	run	already
   #110	open	catch: DifferentialFuzzParityTests (2 lines, sampled)	run	already
   #116	open	catch: HostCultureProbeTests (#125, merged since), 11 candidate rows	run (by step 2)	added (by the parent session)
   #114	open	pending: literal corpus (inverted bounds) × entry point, expected from step 3	run	—
   #115	open	pending: literal corpus (> 28 digits) × entry point, expected from step 3	run	—
   #117	open	pending: another writer's spelling × read path (EF), not in steps 2–3	run	—
   #46	closed	catch: CapturedCollectionProbeTests (#138), seed set-collection-sent-as-one-parameter	run (seed CAUGHT)	added
   #71	closed	catch: CapturedCollectionProbeTests (#138), 1,540 lines	run (revert #77 on the PR branch)	added
   #76	open	catch: CapturedCollectionProbeTests (#138), candidate , alone: , 1,710 lines	run	added
   #57	closed	pending, expected miss: only the error wording was wrong, which is outside the machinery	run (revert #74)	—
   #79	closed	pending, expected miss: release wiring, and SECURITY.md is not a claim source	reasoned	—

   Update (after #125 merged and #138 opened): 11 of 16 are caught. #125 adds #116. CapturedCollectionProbeTests (#138, issue #136) closes the captured collection gap on the EF Core Operand position axis and adds #46, #71 and #76. Its first run also found #137: Any/Count over an empty captured collection throws. Still pending: #114 and #115 (expected from step 3), #117 (another writer's spelling × read path), and #57 and #79 (expected misses).

   One catalogue error turned up along the way: claim E02 names ContinuousMultirangeParityTests as a holder, but that sweep runs no predicate (#134).
10. CaffeinatedCoder commented on Oct 17, 2026

    

    CaffeinatedCoder

    on Oct 17, 2026

    OwnerAuthor

    More actions

    Route B, step 3 is in PR #135 (#123): LiteralCorpusProbeTests. It covers literal corpus × entry point × type, judged by PostgreSQL's own text input on the same text.

    • Scope. 48 types (10 ranges, 10 multiranges, 28 value sets) × 4,531 generated literals × 7 entry points (Parse/TryParse over a string and a span, JSON, TypeConverter, the EF JSON document reader), plus the EF write-back as a SQL constant and a JSON document. That is 29,495 cells. The authority is pg_input_is_valid and the canonical text, one batched statement per type. The sweep takes about 6 s.
    • Known cells, now kept. An inverted range literal parses as empty on every entry point; PostgreSQL rejects it #114 (inverted → empty) shows on all 20 range and multirange types. Decimal text beyond 28 significant digits is rounded silently when a literal or JSON array is read #115 (more than 28 digits) shows on the EF write-back of DecimalRange, its multirange and DecimalSet. GuidSet accepts whitespace inside an element that PostgreSQL's uuid_in rejects #78 shows as U+00A0 in a uuid.
    • New findings, each confirmed with psql on postgres:17.
      • Under Epic: 10.0.0 #58 (10.0.0): A temporal literal with sub-microsecond digits is a different value than PostgreSQL reads from the same text #126 (silent: sub-microsecond literal is a different value on each side), GuidSet reads Guid.Parse's grammar, not uuid_in's: accepts (…), refuses hyphen groups #129 (GuidSet follows Guid.Parse, not uuid_in) and StringSet accepts a NUL character, which PostgreSQL text cannot hold #133 (NUL accepted into text).
      • Under Epic: Backlog #83: Integer literals refuse the 0x/0o/0b and digit-separator forms PostgreSQL 16+ reads #127 (0x/0o/0b/_ integers refused), Value set literals refuse array_in's dimension decoration ([1:1]={…}) and {{}} #128 (array dimension decoration and {{}} refused), LocalDateTime bounds and elements refuse Z and an offset that tsrange and DateTimeRange read #130 (LocalDateTime refuses an offset), DateTimeOffset bounds and elements refuse offsets beyond ±14:00 that timestamptz reads #131 (DateTimeOffset refuses offsets beyond ±14:00) and Document (,max] at the discrete domain boundary: PostgreSQL raises integer out of range #132 ((,max] boundary row undocumented).
    • No cell for Document that a stored timerange bound of 24:00:00 fails to load #108. Every 24:00:00 cell agrees: the library refuses, and the server's value can't be a TimeOnly.
    • Pairs it leaves open (listed in axes.md): corpus × a value set's JSON array beyond [E], and corpus × dictionary key.
11. CaffeinatedCoder commented on Oct 17, 2026

    

    CaffeinatedCoder

    on Oct 17, 2026

    OwnerAuthor

    More actions

    Update to step 3 (#135): the corpus grew to 4,715 literals and 30,699 cells. The new rows put infinite temporal bounds next to finite ones (#117 also reproduces on the literal and JSON-document paths) and quoted or escaped whitespace bounds and elements. The retroactive check over every bug issue finds that this sweep catches #45 and #70 (proven with the reverts of their fixes) and #114, #115, #78 and #117 (open; candidates.json rows). Each of those carries found-by/oracle and a comment naming its row.