Repository navigation
Benchmark record ID JSON scalars and quoted-label counterweights (generator v15) - #783
Merged
Merged
Conversation
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Existing rulepack and model dependencies remain strictly identical; only newly declared path slots may be added. Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com> # Conflicts: # docs/reference/benchmarks/competitors.md # docs/reference/benchmarks/scored-labels-agentic.json # scripts/bench/agentic_layers.py # scripts/bench/test_agentic_layers.py
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
State that generator v15 does not characterize typographic value quotes or a rejected placeholder followed by a valid labelled ID cue. Defer independent positives, repeats and benign counterweights for both boundaries. Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Krishan Koenig <krishan.koenig@googlemail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Append independent CC0 generator-v15 cells for numeric JSON person-linked IDs, bare JSON
null/true/false, and prose placeholders following quoted labels. A quote around a key or label does not quote its value. This closes a benchmark blind spot before measuring the recognizers in #773.Each partition adds 30 A positives, 115 D counterweights and 30 R repeat documents. Numeric values are JSON integers; each keyword document has a distinct public tool-turn field. Dev/test values and template IDs are disjoint. Cell definitions derive from JSON semantics and agent record traffic, with no recognizer-pattern imports. All v3–v14 corpora and frozen contracts remain byte-identical. The full partition is A/D/R 3,728 / 2,144 / 280 documents.
Benchmark evidence
Re-measured every displayed release on the current v15 harness (
56b20a2575d543da898a5aa158fb1e7afa908176) against its own freshly built tagged binary. Each v0.15.x and v0.16.0 arm uses its own fresh setup policy. v0.14.0 retains the documented common v0.15.1 model settings, tokenize-only actions and composite-source splitting.v0.16.0/policy-filev0.15.1/policy-filev0.15.0/policy-filev0.14.0/full-stack-kiji-resolvev0.14.0/pass2-nerAll v0.16.0 and v0.14.0 NER restores are exact with valid manifests. The existing legacy limits remain: v0.15.x have four non-exact restores and four invalid manifests in A; v0.14.0 Kiji has 54 / 15 / 1 non-exact restores and no invalid manifests. No refusals occurred. These are historical measurements, not a gain claim for #773.
Evidence:
docs/reference/benchmarks/agentic-adjacency-v15-history.json, fouragentic-layers-v15-*.json.gzscorecards and four boundobservations-agentic-v15-*.jsonl.gzrecords. The ledger pins binary revisions, binary/scorecard digests, corpus identity and legacy semantics. v14 evidence stays intact. Test corpus SHA-256:0a5f6576f2d5f5c8143628e88ec9d2985197fa3bd74c3823de86f611979c09ba.Commands: each tagged CLI's
setup --non-interactive --force, releaseclean_for_benchbuilds (Nym for v0.15.x/v0.16.0; Kiji for v0.14.0), currentagentic_layers.py measure, andrender_agentic_adjacency_doc.py --generator-version 15 --record .... Primary corpus inputs/scoring and frozen competitor-comparison inputs did not change, so their committed measurements remain applicable; this remeasurement updates A/D/R only.Machine: MacBook Pro, Apple M5 Max, 18 cores, 64 GB, macOS 26.5. Byte evidence only; concurrent slots supply no latency evidence.
Validation
Axes and limits
No runtime detection or restore change is introduced. The new counterweights strengthen precision evidence; bound observations retain deterministic, auditable provenance. Every numeric ID in this slice has six digits. Shorter unlabelled repeats remain outside this slice. Generator v15 has no curly or typographic value-quote cases and no rejected-placeholder-then-valid-cue cases; these results do not establish coverage for either behavior. Independent positive, repeat and benign counterweight cells for both boundaries are deferred. Both limitations are disclosed in the scalar-syntax document and adjacent README coverage paragraph.