Skip to content

Benchmark record ID JSON scalars and quoted-label counterweights (generator v15) - #783

Merged
Naoray merged 14 commits into
mainfrom
bench/record-id-scalar-syntax
Oct 10, 2026
Merged

Naoray merged 14 commits into
mainfrom
bench/record-id-scalar-syntax

Conversation

@Naoray

@Naoray Naoray commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Append independent CC0 generator-v15 cells for numeric JSON person-linked IDs, bare JSON null/true/false, and prose placeholders following quoted labels. A quote around a key or label does not quote its value. This closes a benchmark blind spot before measuring the recognizers in #773.

Each partition adds 30 A positives, 115 D counterweights and 30 R repeat documents. Numeric values are JSON integers; each keyword document has a distinct public tool-turn field. Dev/test values and template IDs are disjoint. Cell definitions derive from JSON semantics and agent record traffic, with no recognizer-pattern imports. All v3–v14 corpora and frozen contracts remain byte-identical. The full partition is A/D/R 3,728 / 2,144 / 280 documents.

Benchmark evidence

Re-measured every displayed release on the current v15 harness (56b20a2575d543da898a5aa158fb1e7afa908176) against its own freshly built tagged binary. Each v0.15.x and v0.16.0 arm uses its own fresh setup policy. v0.14.0 retains the documented common v0.15.1 model settings, tokenize-only actions and composite-source splitting.

Release / arm Leaked A / D / R bytes FP A / D / R bytes Refused A / D / R
v0.16.0 / policy-file 17135 / 0 / 9894 1966 / 6898 / 479 0 / 0 / 0
v0.15.1 / policy-file 33228 / 0 / 10034 2823 / 8058 / 463 0 / 0 / 0
v0.15.0 / policy-file 33401 / 0 / 10034 2823 / 8058 / 463 0 / 0 / 0
v0.14.0 / full-stack-kiji-resolve 43733 / 0 / 10244 10161 / 7846 / 323 0 / 0 / 0
v0.14.0 / pass2-ner 44476 / 0 / 10270 2282 / 5977 / 200 0 / 0 / 0

All v0.16.0 and v0.14.0 NER restores are exact with valid manifests. The existing legacy limits remain: v0.15.x have four non-exact restores and four invalid manifests in A; v0.14.0 Kiji has 54 / 15 / 1 non-exact restores and no invalid manifests. No refusals occurred. These are historical measurements, not a gain claim for #773.

Evidence: docs/reference/benchmarks/agentic-adjacency-v15-history.json, four agentic-layers-v15-*.json.gz scorecards and four bound observations-agentic-v15-*.jsonl.gz records. The ledger pins binary revisions, binary/scorecard digests, corpus identity and legacy semantics. v14 evidence stays intact. Test corpus SHA-256: 0a5f6576f2d5f5c8143628e88ec9d2985197fa3bd74c3823de86f611979c09ba.

Commands: each tagged CLI's setup --non-interactive --force, release clean_for_bench builds (Nym for v0.15.x/v0.16.0; Kiji for v0.14.0), current agentic_layers.py measure, and render_agentic_adjacency_doc.py --generator-version 15 --record .... Primary corpus inputs/scoring and frozen competitor-comparison inputs did not change, so their committed measurements remain applicable; this remeasurement updates A/D/R only.

Machine: MacBook Pro, Apple M5 Max, 18 cores, 64 GB, macOS 26.5. Byte evidence only; concurrent slots supply no latency evidence.

Validation

  • CI-equivalent unittest discovery: 1,025 passed. Initial subprocess protocol/deadline timing failure preserved; isolated retry and final full run passed without a timeout change.
  • CI-equivalent pytest groups: 274 passed, 3 skipped, 10 subtests passed.
  • Focused syntax tests: 4 passed; agentic-layer tests: 248 passed.
  • All eight docs checks passed; 183 renderer tests passed; changed-Markdown scrub passed for three files.
  • Additional inherited Keep the MCP tier mutation probe from discarding uncommitted edits #780 tests: ten passed, one interrupt-matrix case timed out on macOS Bash 3.2/Python 3.13. Reported separately; no source or timeout changes.
  • Signed commits with DCO; synthetic fixtures only.

Axes and limits

No runtime detection or restore change is introduced. The new counterweights strengthen precision evidence; bound observations retain deterministic, auditable provenance. Every numeric ID in this slice has six digits. Shorter unlabelled repeats remain outside this slice. Generator v15 has no curly or typographic value-quote cases and no rejected-placeholder-then-valid-cue cases; these results do not establish coverage for either behavior. Independent positive, repeat and benign counterweight cells for both boundaries are deferred. Both limitations are disclosed in the scalar-syntax document and adjacent README coverage paragraph.

Naoray and others added 13 commits October 10, 2026 17:06
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Existing rulepack and model dependencies remain strictly identical; only newly declared path slots may be added.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>

# Conflicts:
#	docs/reference/benchmarks/competitors.md
#	docs/reference/benchmarks/scored-labels-agentic.json
#	scripts/bench/agentic_layers.py
#	scripts/bench/test_agentic_layers.py
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
State that generator v15 does not characterize typographic value quotes or a rejected placeholder followed by a valid labelled ID cue. Defer independent positives, repeats and benign counterweights for both boundaries.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
@Naoray
Naoray merged commit 1fdd945 into main Oct 10, 2026
20 checks passed
@Naoray
Naoray deleted the bench/record-id-scalar-syntax branch October 10, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant