diff --git a/docs/reference/benchmarks/README.md b/docs/reference/benchmarks/README.md index 9305f9874..0f80c268b 100644 --- a/docs/reference/benchmarks/README.md +++ b/docs/reference/benchmarks/README.md @@ -1926,16 +1926,17 @@ The preserved phone and cued-cell prose above describes the historical [v8 measurements](agentic-adjacency-v8-history.json), and the release statement of the CRLF and plate section the historical [v9 measurements](agentic-adjacency-v9-history.json). Their links lead to this -current comparison, which has since been re-measured on v14 below. +current comparison, which has since been re-measured on v15 below. -The release rows below use generator v14's test partition and each release's -setup policy, so they include the v5 to v14 cells (labelled lookalikes, address +The release rows below use generator v15's test partition and each release's +setup policy, so they include the v5 to v15 cells (labelled lookalikes, address blocks, phone shapes, cued ages, birth dates, short cards, postcodes, CRLF blocks and German plates, URLs in structured text, tax and government IDs, ZIP compression lookalikes, personal age wording and the shipped-class -identifier, credential, name and phone cells, plus person-linked CRM/HR IDs). +identifier, credential, name and phone cells, plus person-linked CRM/HR IDs, numeric JSON ID scalars and benign JSON keywords). The table is rendered from -the [v14 ledger](agentic-adjacency-v14-history.json). The earlier +the [v15 ledger](agentic-adjacency-v15-history.json). The earlier +[v14 ledger](agentic-adjacency-v14-history.json) on corpus `1e07150b…`, [v13 ledger](agentic-adjacency-v13-history.json) on corpus `db3c3f36…`, [v12 ledger](agentic-adjacency-v12-history.json) on corpus `0748ced8…`, [v11 ledger](agentic-adjacency-v11-history.json) on corpus `bab908df…`, @@ -1950,7 +1951,7 @@ retained as historical data and do not feed this table. Record the past-release `agentic_layers.py measure` outputs with `render_agentic_adjacency_doc.py --record`, then render this table from its committed ledger. Do not edit the rows by hand. The renderer pins the -generator version, so a later generator leaves these rows bound to v14 until +generator version, so a later generator leaves these rows bound to v15 until they are re-measured. Generator v12 adds 80 postcode and age positives and 80 benign counterweights @@ -1988,7 +1989,7 @@ cases per partition for explicitly labelled customer, employee and person/contac record identifiers. [Cell definitions](person-linked-record-ids.md) describe numeric, UUID, grouped and long opaque values, semantic JSON fields, same-line English/German cues, public resource IDs and prose placeholders. All previous -v3–v13 corpora remain byte-identical. The full test partition now has 3,698 A, +v3–v13 corpora remain byte-identical. The v14 test partition has 3,698 A, 2,029 D and 250 R documents. The primary corpus inputs and scoring are unchanged, so layer C release rows retain their committed measurements. @@ -1998,25 +1999,40 @@ sweep's six-digit floor, without deterministic sweep protection. Independent short-ID positives, repeats and benign counterweights are required before changing that floor. +Generator v15 adds 30 A numeric JSON scalar IDs, 115 D benign counterweights +and 30 R repeat cases per partition. [Cell definitions](record-id-scalar-syntax.md) +cover unquoted numeric customer/employee/person/contact IDs, bare JSON +`null`/`true`/`false`, and quoted-label prose placeholders. A quote around a key +or label does not quote its value. These CC0 cells derive from JSON scalar +semantics and agent record layouts independently of detector expressions. +All v3–v14 corpora and their contracts remain frozen. The full test partition +now has 3,728 A, 2,144 D and 280 R documents; layer C inputs and scoring stay +unchanged. The added numeric IDs also have six digits, so the short-repeat +limitation above remains. +Generator v15 has no curly or typographic value-quote cases and no cases where +a rejected unquoted placeholder precedes a valid labelled-ID cue. These results +do not establish coverage for either behavior. Independent positive, repeat and +benign counterweight cells for both boundaries are deferred. + | Release and arm | A leaked / gold B | A FP B | D FP B | R leaked / gold B | R FP B | | --- | ---: | ---: | ---: | ---: | ---: | -| `v0.16.0` `policy-file` | 16,955 / 76,311 | 1,963 | 6,762 | 9,354 / 15,912 | 479 | -| `v0.15.1` `policy-file` | 33,048 / 76,311 | 2,820 | 7,922 | 9,494 / 15,912 | 463 | -| `v0.15.0` `policy-file` | 33,221 / 76,311 | 2,820 | 7,922 | 9,494 / 15,912 | 463 | -| `v0.14.0` `full-stack-kiji-resolve` | 43,553 / 76,311 | 10,161 | 7,678 | 9,704 / 15,912 | 323 | -| `v0.14.0` `pass2-ner` | 44,296 / 76,311 | 2,282 | 5,977 | 9,730 / 15,912 | 200 | +| `v0.16.0` `policy-file` | 17,135 / 76,491 | 1,966 | 6,898 | 9,894 / 16,452 | 479 | +| `v0.15.1` `policy-file` | 33,228 / 76,491 | 2,823 | 8,058 | 10,034 / 16,452 | 463 | +| `v0.15.0` `policy-file` | 33,401 / 76,491 | 2,823 | 8,058 | 10,034 / 16,452 | 463 | +| `v0.14.0` `full-stack-kiji-resolve` | 43,733 / 76,491 | 10,161 | 7,846 | 10,244 / 16,452 | 323 | +| `v0.14.0` `pass2-ner` | 44,476 / 76,491 | 2,282 | 5,977 | 10,270 / 16,452 | 200 | | Release and arm | Refused A / D / R | Leaked on all processed A / D / R B | | --- | ---: | ---: | -| `v0.16.0` `policy-file` | 0 / 0 / 0 | 16,955 / 0 / 9,354 | -| `v0.15.1` `policy-file` | 0 / 0 / 0 | 33,048 / 0 / 9,494 | -| `v0.15.0` `policy-file` | 0 / 0 / 0 | 33,221 / 0 / 9,494 | -| `v0.14.0` `full-stack-kiji-resolve` | 0 / 0 / 0 | 43,553 / 0 / 9,704 | -| `v0.14.0` `pass2-ner` | 0 / 0 / 0 | 44,296 / 0 / 9,730 | +| `v0.16.0` `policy-file` | 0 / 0 / 0 | 17,135 / 0 / 9,894 | +| `v0.15.1` `policy-file` | 0 / 0 / 0 | 33,228 / 0 / 10,034 | +| `v0.15.0` `policy-file` | 0 / 0 / 0 | 33,401 / 0 / 10,034 | +| `v0.14.0` `full-stack-kiji-resolve` | 0 / 0 / 0 | 43,733 / 0 / 10,244 | +| `v0.14.0` `pass2-ner` | 0 / 0 / 0 | 44,476 / 0 / 10,270 | -These are layers A, D and R only, measured by the current harness against each release's own binary. Layer C release headlines above are unchanged. The [committed measurement ledger](agentic-adjacency-v14-history.json) records binary and scorecard SHA-256 digests, arm and manifest semantics. Generator v14, test corpus `1e07150bb9ba…`. Each release uses its own setup policy; v0.15.x `f909a23aecac…`, v0.16.0 `65cd6aa588ce…`; v0.14.0 retains the v0.15.x model settings. +These are layers A, D and R only, measured by the current harness against each release's own binary. Layer C release headlines above are unchanged. The [committed measurement ledger](agentic-adjacency-v15-history.json) records binary and scorecard SHA-256 digests, arm and manifest semantics. Generator v15, test corpus `0a5f6576f2d5…`. Each release uses its own setup policy; v0.15.x `f909a23aecac…`, v0.16.0 `65cd6aa588ce…`; v0.14.0 retains the v0.15.x model settings. diff --git a/docs/reference/benchmarks/agentic-adjacency-v15-history.json b/docs/reference/benchmarks/agentic-adjacency-v15-history.json new file mode 100644 index 000000000..8b9914dc8 --- /dev/null +++ b/docs/reference/benchmarks/agentic-adjacency-v15-history.json @@ -0,0 +1,187 @@ +{ + "schema_version": 1, + "generator_version": 15, + "corpus_sha256": "0a5f6576f2d5f5c8143628e88ec9d2985197fa3bd74c3823de86f611979c09ba", + "contract_sha256": "c8542632aad9373bc251c8ab10f2eaabd5df01f9f66ba217e914f93a4ffecabb", + "policy_sha256": "f909a23aecacc5695388223be5e71bc1e303c845563396d6658448396a0a9ebe", + "rows": [ + { + "version": "v0.16.0", + "arm": "policy-file", + "binary_revision": "5a7dac0fa198f20a628ed6b7fa2b5c9d41a91eb0", + "binary_sha256": "afe7a2000945126cd08363c3dcc5203c1a7a1e89d121bb6c743d98ac0356e2b4", + "scorecard_sha256": "a9253231d33286513d514e3e4251cc59bcbdc1c18f55f5aa7198f2e6243ae971", + "manifest_actions": [ + "redact", + "tokenize" + ], + "split_composite_source_ids": false, + "layers": { + "A": { + "gold": 76491, + "leaked": 17135, + "false_positive": 1966 + }, + "D": { + "gold": 0, + "leaked": 0, + "false_positive": 6898 + }, + "R": { + "gold": 16452, + "leaked": 9894, + "false_positive": 479 + } + }, + "scorecard_file": "agentic-layers-v15-v0.16.0.json.gz", + "policy_sha256": "65cd6aa588ce6923f9e2156699893fcffb9a870967d9a0db5018deaa0c22c8cc", + "refused": { + "A": 0, + "D": 0, + "R": 0 + } + }, + { + "version": "v0.15.1", + "arm": "policy-file", + "binary_revision": "4927f7c0a862ece2cdcd8c00df34538a510d008e", + "binary_sha256": "76faf0ab4088f8c1b321522ed651ce05fd8b6191022de586ef3234d99fb49a20", + "scorecard_sha256": "a80b1f0947c03a4dab008f3e1ac1da19e2023f42ddf4ec668c9dea60953a97fb", + "manifest_actions": [ + "redact", + "tokenize" + ], + "split_composite_source_ids": false, + "layers": { + "A": { + "gold": 76491, + "leaked": 33228, + "false_positive": 2823 + }, + "D": { + "gold": 0, + "leaked": 0, + "false_positive": 8058 + }, + "R": { + "gold": 16452, + "leaked": 10034, + "false_positive": 463 + } + }, + "scorecard_file": "agentic-layers-v15-v0.15.1.json.gz", + "policy_sha256": "f909a23aecacc5695388223be5e71bc1e303c845563396d6658448396a0a9ebe", + "refused": { + "A": 0, + "D": 0, + "R": 0 + } + }, + { + "version": "v0.15.0", + "arm": "policy-file", + "binary_revision": "5127788b089f0f6f4e3b8bcf176503953744be01", + "binary_sha256": "d3696c1483cdb7fa8066afd7930a06067c124470e5d43fff806d72de34e9ea50", + "scorecard_sha256": "73b6201041c7573520d61802e5de5bb91ec3cc05075a0ad291f688e30e8e6aa1", + "manifest_actions": [ + "redact", + "tokenize" + ], + "split_composite_source_ids": false, + "layers": { + "A": { + "gold": 76491, + "leaked": 33401, + "false_positive": 2823 + }, + "D": { + "gold": 0, + "leaked": 0, + "false_positive": 8058 + }, + "R": { + "gold": 16452, + "leaked": 10034, + "false_positive": 463 + } + }, + "scorecard_file": "agentic-layers-v15-v0.15.0.json.gz", + "policy_sha256": "f909a23aecacc5695388223be5e71bc1e303c845563396d6658448396a0a9ebe", + "refused": { + "A": 0, + "D": 0, + "R": 0 + } + }, + { + "version": "v0.14.0", + "arm": "full-stack-kiji-resolve", + "binary_revision": "616e8a10b957961f9f7dcb745610afa978783c3a", + "binary_sha256": "c2de4dd80b59d3df3c12913af7d2e4b037d62194fc54554aeac542a3f6b6473f", + "scorecard_sha256": "940c60fa5d447da7b833ff073060747a2e6991cb23f6318f13d2e438df6355d7", + "manifest_actions": [ + "tokenize" + ], + "split_composite_source_ids": true, + "layers": { + "A": { + "gold": 76491, + "leaked": 43733, + "false_positive": 10161 + }, + "D": { + "gold": 0, + "leaked": 0, + "false_positive": 7846 + }, + "R": { + "gold": 16452, + "leaked": 10244, + "false_positive": 323 + } + }, + "scorecard_file": "agentic-layers-v15-v0.14.0.json.gz", + "policy_sha256": "f909a23aecacc5695388223be5e71bc1e303c845563396d6658448396a0a9ebe", + "refused": { + "A": 0, + "D": 0, + "R": 0 + } + }, + { + "version": "v0.14.0", + "arm": "pass2-ner", + "binary_revision": "616e8a10b957961f9f7dcb745610afa978783c3a", + "binary_sha256": "c2de4dd80b59d3df3c12913af7d2e4b037d62194fc54554aeac542a3f6b6473f", + "scorecard_sha256": "940c60fa5d447da7b833ff073060747a2e6991cb23f6318f13d2e438df6355d7", + "manifest_actions": [ + "tokenize" + ], + "split_composite_source_ids": true, + "layers": { + "A": { + "gold": 76491, + "leaked": 44476, + "false_positive": 2282 + }, + "D": { + "gold": 0, + "leaked": 0, + "false_positive": 5977 + }, + "R": { + "gold": 16452, + "leaked": 10270, + "false_positive": 200 + } + }, + "scorecard_file": "agentic-layers-v15-v0.14.0.json.gz", + "policy_sha256": "f909a23aecacc5695388223be5e71bc1e303c845563396d6658448396a0a9ebe", + "refused": { + "A": 0, + "D": 0, + "R": 0 + } + } + ] +} diff --git a/docs/reference/benchmarks/agentic-layers-v15-v0.14.0.json.gz b/docs/reference/benchmarks/agentic-layers-v15-v0.14.0.json.gz new file mode 100644 index 000000000..aeb7e033e Binary files /dev/null and b/docs/reference/benchmarks/agentic-layers-v15-v0.14.0.json.gz differ diff --git a/docs/reference/benchmarks/agentic-layers-v15-v0.15.0.json.gz b/docs/reference/benchmarks/agentic-layers-v15-v0.15.0.json.gz new file mode 100644 index 000000000..eb7fd7de0 Binary files /dev/null and b/docs/reference/benchmarks/agentic-layers-v15-v0.15.0.json.gz differ diff --git a/docs/reference/benchmarks/agentic-layers-v15-v0.15.1.json.gz b/docs/reference/benchmarks/agentic-layers-v15-v0.15.1.json.gz new file mode 100644 index 000000000..e84c01765 Binary files /dev/null and b/docs/reference/benchmarks/agentic-layers-v15-v0.15.1.json.gz differ diff --git a/docs/reference/benchmarks/agentic-layers-v15-v0.16.0.json.gz b/docs/reference/benchmarks/agentic-layers-v15-v0.16.0.json.gz new file mode 100644 index 000000000..248329073 Binary files /dev/null and b/docs/reference/benchmarks/agentic-layers-v15-v0.16.0.json.gz differ diff --git a/docs/reference/benchmarks/competitors.md b/docs/reference/benchmarks/competitors.md index 429392ed2..9d0158676 100644 --- a/docs/reference/benchmarks/competitors.md +++ b/docs/reference/benchmarks/competitors.md @@ -1,6 +1,6 @@ # Competitor comparison -Same corpus and scorer; tools run with documented configurations. UTF-8 byte counts use the Gaze scorer. For v4 and v3, FP is the scorer's false-positive count after its audited gold-gap credit. Presidio all runs English, German, Dutch, French, and Portuguese spaCy models with the documented German recognizers. Presidio English default is a secondary row. Latency was not measured under a quiet machine; timing comparisons are withheld. This measures detection; competitor restore and manifest behavior is not scored. The agentic-layer rows were measured on generator v4 (test corpus `387a35ac1551…`, 3,250 documents). The current generator is v14; its added documents are not measured here. +Same corpus and scorer; tools run with documented configurations. UTF-8 byte counts use the Gaze scorer. For v4 and v3, FP is the scorer's false-positive count after its audited gold-gap credit. Presidio all runs English, German, Dutch, French, and Portuguese spaCy models with the documented German recognizers. Presidio English default is a secondary row. Latency was not measured under a quiet machine; timing comparisons are withheld. This measures detection; competitor restore and manifest behavior is not scored. The agentic-layer rows were measured on generator v4 (test corpus `387a35ac1551…`, 3,250 documents). The current generator is v15; its added documents are not measured here. Leaked and false-positive byte counts are class-agnostic. A skipped document's scored gold counts in full as leaked. Subtract Skipped gold B from Leaked B to get leakage on processed documents. For example, Presidio English-only v4 Synthetic identifiers in agentic formats leaks 34,733 B, including 28,151 B of scored gold from 1,320 skipped non-English documents. The reviewed label map controls repeated-gold credit and the exact typed-span metrics below. diff --git a/docs/reference/benchmarks/observations-agentic-v15-v0.14.0.jsonl.gz b/docs/reference/benchmarks/observations-agentic-v15-v0.14.0.jsonl.gz new file mode 100644 index 000000000..2e5df80b7 Binary files /dev/null and b/docs/reference/benchmarks/observations-agentic-v15-v0.14.0.jsonl.gz differ diff --git a/docs/reference/benchmarks/observations-agentic-v15-v0.15.0.jsonl.gz b/docs/reference/benchmarks/observations-agentic-v15-v0.15.0.jsonl.gz new file mode 100644 index 000000000..280f9b75d Binary files /dev/null and b/docs/reference/benchmarks/observations-agentic-v15-v0.15.0.jsonl.gz differ diff --git a/docs/reference/benchmarks/observations-agentic-v15-v0.15.1.jsonl.gz b/docs/reference/benchmarks/observations-agentic-v15-v0.15.1.jsonl.gz new file mode 100644 index 000000000..316d357f3 Binary files /dev/null and b/docs/reference/benchmarks/observations-agentic-v15-v0.15.1.jsonl.gz differ diff --git a/docs/reference/benchmarks/observations-agentic-v15-v0.16.0.jsonl.gz b/docs/reference/benchmarks/observations-agentic-v15-v0.16.0.jsonl.gz new file mode 100644 index 000000000..3b4146f38 Binary files /dev/null and b/docs/reference/benchmarks/observations-agentic-v15-v0.16.0.jsonl.gz differ diff --git a/docs/reference/benchmarks/record-id-scalar-syntax.md b/docs/reference/benchmarks/record-id-scalar-syntax.md new file mode 100644 index 000000000..f17de5088 --- /dev/null +++ b/docs/reference/benchmarks/record-id-scalar-syntax.md @@ -0,0 +1,33 @@ +# Record identifier scalar syntax + +Generator v15 adds authored CC0 synthetic scalar syntax cases for person-linked +customer, employee and person/contact record identifiers. The cells follow JSON +value semantics and schema-writing traffic, independently of recognizer patterns. +They add 30 A positives, 115 D counterweights and 30 R repeats per partition. + +A supplies explicitly labelled six-digit database keys as JSON numbers, alongside +the quoted-string values covered by v14. R repeats each numeric value three times, +including later references without a repeated label. Every occurrence is gold, +and each gold span selects the inserted number in UTF-8 bytes. + +D supplies valid JSON fields whose values are bare `null`, `true` or `false`. +These keywords supply no identifier value. A public tool-turn field makes each +scalar document distinct without changing its identifier semantics. Other counterweights quote the schema +label while leaving an ordinary placeholder word unquoted. Quoting the label +alone does not establish an opaque string value. All predicted bytes in these +documents are false positives. + +Numeric pools are independently seeded and disjoint between dev and test. Field +vocabulary and template IDs also differ between partitions. Values are invented +locally; no real customer or employee data is used. All earlier generator versions +3–14 remain byte-identical and retain their frozen agentic contracts. The new +syntax cells use the same three scored labels under primary contracts v1–v4; +the primary corpus, contracts and gate-credit exceptions are unchanged. + +Known limitations: numeric identifiers still have six digits. Shorter unlabelled +repeats remain unmeasured and below the existing deterministic sweep floor. +Separate positive, repeat and benign coverage is required before changing it. +Generator v15 has no curly or typographic value-quote cases and no cases where +a rejected unquoted placeholder precedes a valid labelled-ID cue. These results +do not establish coverage for either behavior. Independent positive, repeat and +benign counterweight cells for both boundaries are deferred. diff --git a/docs/reference/benchmarks/scored-labels-agentic-generator-v14.json b/docs/reference/benchmarks/scored-labels-agentic-generator-v14.json new file mode 100644 index 000000000..29d854403 --- /dev/null +++ b/docs/reference/benchmarks/scored-labels-agentic-generator-v14.json @@ -0,0 +1,251 @@ +{ + "schema_version": 1, + "contract": "scored-labels-agentic", + "contract_version": 2, + "corpus": { + "generator": "scripts/bench/agentic_layers.py", + "generator_version": 14, + "published_partition": "test" + }, + "semantics": "Every label the agentic-layer generator emits is listed here and scored. Gold that fails its own checksum (the invalid twins) stays scored, exactly as in the Kiji validator gold census; the per-label validity split is reported beside it and never replaces it. Layer A adjacent-address gold uses synthetic private-network host endpoints associated with a user's device. Documentation, loopback and link-local ranges are benign and scored as Layer D lookalikes instead. Layer D documents carry no gold, so every byte predicted there is a false positive. A label names a value even when it stands lines away, past a blank line, below the value, in a JSON key, sibling or nested field, or a CSV header: the labelled benign-lookalike cells of layer A are gold, and their layer D counterweights carry the same structures with no cue anywhere in the document. The address cells of layer A score every part of a whole address (house number, street, secondary unit or box, city, state, postcode) under its own label; the separators between parts are not gold, and a benign designator written beside an address (`test Suite 4`) is a recorded decoy, so covering it is a false positive. Their layer D counterweights carry the same designator words with no address anywhere. The phone-shape cells of layer A score a whole phone number, international prefix and parenthesised trunk zero included, in French dotted groups, national digit groups behind a phone label, `+CC (0)` and `00` / `001-` forms; their layer D counterweights carry dotted versions, dates and OIDs, firmware and part numbers written in the dotted phone shape, digit groups behind an order, invoice or ticket label, a signed score with a parenthesised zero, and `00`- or `001`-prefixed product, ticket, item and document codes (behind a spare country code or an unassignable NANP exchange when written in the phone's exact shape), with no phone label anywhere. The cued cells of layer A score a person's age written after `turned` (or before `geworden`), after `at the age of` (`im Alter von`), before `y/o` or before `year old female` / `year old male`, the number alone; a date of birth given one sentence after the question (`date of birth? It's ...`), the date alone; a 12- to 15-digit card number with a Maestro issuer prefix, compact or grouped 4-4-4, directly after a card label or in a card column; and a postcode in a short or foreign shape (`NNN NN`, `NN-NNN`, six digits, `NNNNN-NNN`, three digits) directly after a postal label or in a postal column. Their layer D counterweights carry the same wording and digit shapes where the text says they are not personal: the age of an object or an animal, a person turning some degrees, a date after an unrelated sentence that follows the date-of-birth question, order, tracking, transaction and timestamp numbers, and codes behind batch, seat, room, gate, error, part, invoice or build labels, some of them one clause away from a card or postal word. The URL cells of layer A score a whole URL that points at a person, scheme included: a profile page with the person's name in its path or a personal site on their surname, written with an `https` scheme or a `www` host prefix, where a structural delimiter ends it (the closing quote of a compact JSON value with sibling keys after it, JSON whose slashes are escaped as `\\/`, a JSON-escaped quote, a double- or single-quoted HTML attribute, HTML element text, a Markdown link) or inside prose with an apostrophe in its path. Their layer D counterweights carry the same structures with no scheme and no `www` prefix anywhere: JSON-escaped routes and MIME types, service hosts with a path, plain and escaped, and relative links in HTML and Markdown. Layer R decoys (words spelled like or containing a name part, digit runs shared with a repeated identifier) are not gold either, so a byte predicted over one is a false positive; every repeat of a real value is gold. An unlisted label fails closed, and so does a ruling on a label the generator no longer emits. CRLF address cells score every address part and treat CRLF as one line break. Bare CR and blank-line designator twins must remain benign. German plates are whole-value gold after an immediate label; same-shape build and batch references, including near but unrelated plate words, are benign. Explicitly labelled manufacturing plate references remain benign; their whole values are false-positive counterweights even when the plate cue is immediate. Government identifier cells score only whole synthetic values directly owned by a tax, social-insurance, identity-card or national-ID cue, with natural copulas, parentheticals, line breaks and structured fields. SSN tails require an explicit ending relation. Shape-matched references, vehicle IDs, technical licences, rates and amounts remain benign even with a nearby personal-ID cue. Zero-led schemes are deliberately unassignable; Steuer-ID-shaped failures are recorded as invalid gold. ZIP/age cells score the postcode alone in shipping instructions and address suffixes, and the age alone for collective birthdays and children. Archive filenames followed by sizes, compression commands followed by counts, decimal distances or laps after turned, and explicitly stated company anniversaries are benign; their entire inserted numeric values are decoys. Generator v13 adds independently seeded class-coverage cells. Checksum-invalid identifiers in layer D and values explicitly owned by inventory or batch references are benign. Password field values and fake access credentials are scored gold; password-policy examples and incomplete access-key fragments are benign. Name gold covers only inserted given names and surnames; automated queues and roles are benign. German postal cells require a de-DE-first locale chain when measuring postal.de source attribution; language/region metadata alone does not change the setup policy. Generator v14 adds explicitly labelled person-linked CRM/HR identifiers in semantic JSON fields and same-line English/German cues. Public order, invoice, ticket, issue, build, commit and tracking identifiers, generic id/record_id/unique_id keys and schema-writing placeholders carry no gold. This agentic contract scores these new labels under v1/v2/v3/v4 gate use; the primary corpus contracts remain unchanged because their corpus emits none of these labels.", + "labels": [ + { + "label": "AGE", + "scored": true, + "category": "quasi_identifier", + "ruling": "settled", + "reason": "A person's age in years, the number alone, written after `turned` or before `geworden`, after `at the age of` or `im Alter von`, before `y/o`, or before `year old female` / `year old male`, in a sentence about a person. An object's or an animal's age, and a person turning some degrees, are benign." + }, + { + "label": "BSN", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Dutch citizen service number; generated checksum-valid with a checksum-invalid twin." + }, + { + "label": "BUILDINGNUM", + "scored": true, + "category": "address_constituent", + "ruling": "settled", + "reason": "House number, or a secondary unit or box written with its designator (`Suite 312`, `Apt. 771`, `Wohnung 4`, `3. Etage`, `PO Box 417`, `Postfach 505`, the `Box` of a US military address), inside a whole synthetic address. Scored per label until a composite ADDRESS entity exists; same label as layer C. A designator and number with no address in the document is benign." + }, + { + "label": "CITY", + "scored": true, + "category": "quasi_identifier", + "ruling": "settled", + "reason": "Invented city name inside a whole synthetic address, and the APO, FPO or DPO post office of a US military address, which stands where a city stands." + }, + { + "label": "CPF", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Brazilian taxpayer number; generated checksum-valid with a checksum-invalid twin." + }, + { + "label": "CREDITCARDNUMBER", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Payment card number; generated Luhn-valid with a Luhn-invalid twin. Includes a 12- to 15-digit number with a Maestro issuer prefix directly after a card label or under a card column header, Luhn-valid and Luhn-failing in turn; the same digits after an order, tracking, transaction, reference or terminal label are benign." + }, + { + "label": "DATEOFBIRTH", + "scored": true, + "category": "quasi_identifier", + "ruling": "settled", + "reason": "Date of birth, with and without a cue; a date inserted as a birth date is personal data. Includes a date given as the answer one sentence after a date-of-birth question (`It's`, `It is`, `Es ist der`, `Das ist der`); a date after any other sentence that follows the question is benign." + }, + { + "label": "EMAIL", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Personal email address derived from a synthetic name." + }, + { + "label": "GIVENNAME", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Given name of the sender in an email header or agent payload." + }, + { + "label": "IBAN", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Personal bank account (DE, AT, NL, FR, GB); generated mod-97-valid with an invalid twin." + }, + { + "label": "IDCARDNUM", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Independently invented zero-led test identifiers under an explicit IDCARDNUM label; partial SSN tails require an explicit ending relation." + }, + { + "label": "IPADDRESS", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Synthetic private-network host endpoint associated with a user's device; documentation, loopback and link-local addresses are benign." + }, + { + "label": "NATIONALID", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Independently invented zero-led test identifiers under an explicit NATIONALID label; partial SSN tails require an explicit ending relation." + }, + { + "label": "NHSNUMBER", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "UK NHS number; generated modulus-11-valid with an invalid twin." + }, + { + "label": "SSN", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Independently invented zero-led test identifiers under an explicit SSN label; partial SSN tails require an explicit ending relation." + }, + { + "label": "STATE", + "scored": true, + "category": "quasi_identifier", + "ruling": "pending", + "reason": "US state abbreviation between city and ZIP, and the AA, AE or AP code of a US military address. Kept in as in layer C; a state alone rarely identifies a person, so its place in the safety score needs an explicit ruling." + }, + { + "label": "STREET", + "scored": true, + "category": "address_constituent", + "ruling": "settled", + "reason": "Invented street name inside a whole synthetic address, and the PSC, CMR or Unit line of a US military address. Scored per label until a composite ADDRESS entity exists; same label as layer C." + }, + { + "label": "SURNAME", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Surname, including hyphenated double surnames." + }, + { + "label": "TAXNUM", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "German Steuer-ID; generated MOD 11,10-valid with an invalid twin." + }, + { + "label": "TELEPHONENUM", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Personal phone number (German mobile, US). Includes a reserved-range phone (NANPA 555-01xx, German 01555) written as the tail of an ORDER- style reference when a label anywhere in the same document or record names it as a phone, in any language or script; the phone is gold, the reference word is not. A 16-digit run is never phone gold: no phone number has 16 digits. Includes a whole number in a documented fictional range (ARCEP fiction blocks, BNetzA media-production numbers, Ofcom drama numbers, NANPA 555-01xx) written as French dotted groups, with a parenthesised trunk zero after the country code, or behind a `00` or `001` international prefix, the prefix included; and national digit groups with no `+` behind a phone label, synthesized non-reachable (a leading 0 that the Spanish nine-digit and Danish eight-digit plans never use). The same digit groups with no phone label are benign." + }, + { + "label": "URL", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "A URL that points at a person: a profile page with the person's name in its path or a personal site on their surname, with an explicit `https` scheme or `www` host prefix, plain or with JSON-escaped `\\/` slashes. The whole URL is gold, scheme included, up to the delimiter that ends it. A route, a MIME type, a service host with a path or a relative link carries no anchor and is benign." + }, + { + "label": "ZIPCODE", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Synthetic postal code in an adjacent address field, including intentionally unassigned CA and GB shapes. Includes a five-digit US or German postcode written inside a benign-looking structure (the tail of an ORDER- style reference, or after a currency code) when a label anywhere in the same document or record names it as a postcode. Includes the postcode of a whole synthetic address: US ZIPs in the unassigned 000xx range, German PLZ in the unassigned 00xxx range and GB postcodes in the unused ZZ area. Includes a synthesized postcode in a short or foreign shape (Swedish `NNN NN`, Polish `NN-NNN`, six digits, Brazilian `NNNNN-NNN`, three digits) directly after a postal label or under a postal column header; the same shapes after a batch, seat, room, gate, error, part, invoice or build label are benign, also when a postal word stands one clause away." + }, + { + "label": "LICENSEPLATE", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Synthetic German registration plate, unassignable (leading-zero serial), directly after a plate label or key. The whole district, letter series and serial number are gold. Identically shaped build and batch references without an immediate plate label are benign." + }, + { + "label": "AADHAAR", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "CNPJ", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "ETHADDRESS", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "NIR", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "PAN", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "VATID", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "PASSWORD", + "scored": true, + "category": "credential", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "SECURITYTOKEN", + "scored": true, + "category": "credential", + "ruling": "settled", + "reason": "Project-authored synthetic shipped-class coverage; exact inserted value is gold. See class-coverage.md for format sources and counterweight semantics." + }, + { + "label": "CUSTOMER_ID", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Explicitly labelled identifier of a customer in CRM. Numeric keys, UUIDs, grouped and long opaque values are whole-value gold. Public resource IDs, generic keys without person linkage and prose placeholders are benign. All repeats are gold." + }, + { + "label": "EMPLOYEE_ID", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Explicitly labelled identifier of a employee in HR. Numeric keys, UUIDs, grouped and long opaque values are whole-value gold. Public resource IDs, generic keys without person linkage and prose placeholders are benign. All repeats are gold." + }, + { + "label": "RECORD_ID", + "scored": true, + "category": "direct_identifier", + "ruling": "settled", + "reason": "Explicitly labelled identifier of a person or contact record. Numeric keys, UUIDs, grouped and long opaque values are whole-value gold. Public resource IDs, generic keys without person linkage and prose placeholders are benign. All repeats are gold." + } + ] +} diff --git a/docs/reference/benchmarks/scored-labels-agentic.json b/docs/reference/benchmarks/scored-labels-agentic.json index 29d854403..853622e66 100644 --- a/docs/reference/benchmarks/scored-labels-agentic.json +++ b/docs/reference/benchmarks/scored-labels-agentic.json @@ -4,10 +4,10 @@ "contract_version": 2, "corpus": { "generator": "scripts/bench/agentic_layers.py", - "generator_version": 14, + "generator_version": 15, "published_partition": "test" }, - "semantics": "Every label the agentic-layer generator emits is listed here and scored. Gold that fails its own checksum (the invalid twins) stays scored, exactly as in the Kiji validator gold census; the per-label validity split is reported beside it and never replaces it. Layer A adjacent-address gold uses synthetic private-network host endpoints associated with a user's device. Documentation, loopback and link-local ranges are benign and scored as Layer D lookalikes instead. Layer D documents carry no gold, so every byte predicted there is a false positive. A label names a value even when it stands lines away, past a blank line, below the value, in a JSON key, sibling or nested field, or a CSV header: the labelled benign-lookalike cells of layer A are gold, and their layer D counterweights carry the same structures with no cue anywhere in the document. The address cells of layer A score every part of a whole address (house number, street, secondary unit or box, city, state, postcode) under its own label; the separators between parts are not gold, and a benign designator written beside an address (`test Suite 4`) is a recorded decoy, so covering it is a false positive. Their layer D counterweights carry the same designator words with no address anywhere. The phone-shape cells of layer A score a whole phone number, international prefix and parenthesised trunk zero included, in French dotted groups, national digit groups behind a phone label, `+CC (0)` and `00` / `001-` forms; their layer D counterweights carry dotted versions, dates and OIDs, firmware and part numbers written in the dotted phone shape, digit groups behind an order, invoice or ticket label, a signed score with a parenthesised zero, and `00`- or `001`-prefixed product, ticket, item and document codes (behind a spare country code or an unassignable NANP exchange when written in the phone's exact shape), with no phone label anywhere. The cued cells of layer A score a person's age written after `turned` (or before `geworden`), after `at the age of` (`im Alter von`), before `y/o` or before `year old female` / `year old male`, the number alone; a date of birth given one sentence after the question (`date of birth? It's ...`), the date alone; a 12- to 15-digit card number with a Maestro issuer prefix, compact or grouped 4-4-4, directly after a card label or in a card column; and a postcode in a short or foreign shape (`NNN NN`, `NN-NNN`, six digits, `NNNNN-NNN`, three digits) directly after a postal label or in a postal column. Their layer D counterweights carry the same wording and digit shapes where the text says they are not personal: the age of an object or an animal, a person turning some degrees, a date after an unrelated sentence that follows the date-of-birth question, order, tracking, transaction and timestamp numbers, and codes behind batch, seat, room, gate, error, part, invoice or build labels, some of them one clause away from a card or postal word. The URL cells of layer A score a whole URL that points at a person, scheme included: a profile page with the person's name in its path or a personal site on their surname, written with an `https` scheme or a `www` host prefix, where a structural delimiter ends it (the closing quote of a compact JSON value with sibling keys after it, JSON whose slashes are escaped as `\\/`, a JSON-escaped quote, a double- or single-quoted HTML attribute, HTML element text, a Markdown link) or inside prose with an apostrophe in its path. Their layer D counterweights carry the same structures with no scheme and no `www` prefix anywhere: JSON-escaped routes and MIME types, service hosts with a path, plain and escaped, and relative links in HTML and Markdown. Layer R decoys (words spelled like or containing a name part, digit runs shared with a repeated identifier) are not gold either, so a byte predicted over one is a false positive; every repeat of a real value is gold. An unlisted label fails closed, and so does a ruling on a label the generator no longer emits. CRLF address cells score every address part and treat CRLF as one line break. Bare CR and blank-line designator twins must remain benign. German plates are whole-value gold after an immediate label; same-shape build and batch references, including near but unrelated plate words, are benign. Explicitly labelled manufacturing plate references remain benign; their whole values are false-positive counterweights even when the plate cue is immediate. Government identifier cells score only whole synthetic values directly owned by a tax, social-insurance, identity-card or national-ID cue, with natural copulas, parentheticals, line breaks and structured fields. SSN tails require an explicit ending relation. Shape-matched references, vehicle IDs, technical licences, rates and amounts remain benign even with a nearby personal-ID cue. Zero-led schemes are deliberately unassignable; Steuer-ID-shaped failures are recorded as invalid gold. ZIP/age cells score the postcode alone in shipping instructions and address suffixes, and the age alone for collective birthdays and children. Archive filenames followed by sizes, compression commands followed by counts, decimal distances or laps after turned, and explicitly stated company anniversaries are benign; their entire inserted numeric values are decoys. Generator v13 adds independently seeded class-coverage cells. Checksum-invalid identifiers in layer D and values explicitly owned by inventory or batch references are benign. Password field values and fake access credentials are scored gold; password-policy examples and incomplete access-key fragments are benign. Name gold covers only inserted given names and surnames; automated queues and roles are benign. German postal cells require a de-DE-first locale chain when measuring postal.de source attribution; language/region metadata alone does not change the setup policy. Generator v14 adds explicitly labelled person-linked CRM/HR identifiers in semantic JSON fields and same-line English/German cues. Public order, invoice, ticket, issue, build, commit and tracking identifiers, generic id/record_id/unique_id keys and schema-writing placeholders carry no gold. This agentic contract scores these new labels under v1/v2/v3/v4 gate use; the primary corpus contracts remain unchanged because their corpus emits none of these labels.", + "semantics": "Every label the agentic-layer generator emits is listed here and scored. Gold that fails its own checksum (the invalid twins) stays scored, exactly as in the Kiji validator gold census; the per-label validity split is reported beside it and never replaces it. Layer A adjacent-address gold uses synthetic private-network host endpoints associated with a user's device. Documentation, loopback and link-local ranges are benign and scored as Layer D lookalikes instead. Layer D documents carry no gold, so every byte predicted there is a false positive. A label names a value even when it stands lines away, past a blank line, below the value, in a JSON key, sibling or nested field, or a CSV header: the labelled benign-lookalike cells of layer A are gold, and their layer D counterweights carry the same structures with no cue anywhere in the document. The address cells of layer A score every part of a whole address (house number, street, secondary unit or box, city, state, postcode) under its own label; the separators between parts are not gold, and a benign designator written beside an address (`test Suite 4`) is a recorded decoy, so covering it is a false positive. Their layer D counterweights carry the same designator words with no address anywhere. The phone-shape cells of layer A score a whole phone number, international prefix and parenthesised trunk zero included, in French dotted groups, national digit groups behind a phone label, `+CC (0)` and `00` / `001-` forms; their layer D counterweights carry dotted versions, dates and OIDs, firmware and part numbers written in the dotted phone shape, digit groups behind an order, invoice or ticket label, a signed score with a parenthesised zero, and `00`- or `001`-prefixed product, ticket, item and document codes (behind a spare country code or an unassignable NANP exchange when written in the phone's exact shape), with no phone label anywhere. The cued cells of layer A score a person's age written after `turned` (or before `geworden`), after `at the age of` (`im Alter von`), before `y/o` or before `year old female` / `year old male`, the number alone; a date of birth given one sentence after the question (`date of birth? It's ...`), the date alone; a 12- to 15-digit card number with a Maestro issuer prefix, compact or grouped 4-4-4, directly after a card label or in a card column; and a postcode in a short or foreign shape (`NNN NN`, `NN-NNN`, six digits, `NNNNN-NNN`, three digits) directly after a postal label or in a postal column. Their layer D counterweights carry the same wording and digit shapes where the text says they are not personal: the age of an object or an animal, a person turning some degrees, a date after an unrelated sentence that follows the date-of-birth question, order, tracking, transaction and timestamp numbers, and codes behind batch, seat, room, gate, error, part, invoice or build labels, some of them one clause away from a card or postal word. The URL cells of layer A score a whole URL that points at a person, scheme included: a profile page with the person's name in its path or a personal site on their surname, written with an `https` scheme or a `www` host prefix, where a structural delimiter ends it (the closing quote of a compact JSON value with sibling keys after it, JSON whose slashes are escaped as `\\/`, a JSON-escaped quote, a double- or single-quoted HTML attribute, HTML element text, a Markdown link) or inside prose with an apostrophe in its path. Their layer D counterweights carry the same structures with no scheme and no `www` prefix anywhere: JSON-escaped routes and MIME types, service hosts with a path, plain and escaped, and relative links in HTML and Markdown. Layer R decoys (words spelled like or containing a name part, digit runs shared with a repeated identifier) are not gold either, so a byte predicted over one is a false positive; every repeat of a real value is gold. An unlisted label fails closed, and so does a ruling on a label the generator no longer emits. CRLF address cells score every address part and treat CRLF as one line break. Bare CR and blank-line designator twins must remain benign. German plates are whole-value gold after an immediate label; same-shape build and batch references, including near but unrelated plate words, are benign. Explicitly labelled manufacturing plate references remain benign; their whole values are false-positive counterweights even when the plate cue is immediate. Government identifier cells score only whole synthetic values directly owned by a tax, social-insurance, identity-card or national-ID cue, with natural copulas, parentheticals, line breaks and structured fields. SSN tails require an explicit ending relation. Shape-matched references, vehicle IDs, technical licences, rates and amounts remain benign even with a nearby personal-ID cue. Zero-led schemes are deliberately unassignable; Steuer-ID-shaped failures are recorded as invalid gold. ZIP/age cells score the postcode alone in shipping instructions and address suffixes, and the age alone for collective birthdays and children. Archive filenames followed by sizes, compression commands followed by counts, decimal distances or laps after turned, and explicitly stated company anniversaries are benign; their entire inserted numeric values are decoys. Generator v13 adds independently seeded class-coverage cells. Checksum-invalid identifiers in layer D and values explicitly owned by inventory or batch references are benign. Password field values and fake access credentials are scored gold; password-policy examples and incomplete access-key fragments are benign. Name gold covers only inserted given names and surnames; automated queues and roles are benign. German postal cells require a de-DE-first locale chain when measuring postal.de source attribution; language/region metadata alone does not change the setup policy. Generator v14 adds explicitly labelled person-linked CRM/HR identifiers in semantic JSON fields and same-line English/German cues. Public order, invoice, ticket, issue, build, commit and tracking identifiers, generic id/record_id/unique_id keys and schema-writing placeholders carry no gold. This agentic contract scores these new labels under v1/v2/v3/v4 gate use; the primary corpus contracts remain unchanged because their corpus emits none of these labels. Generator v15 adds numeric JSON scalar identifiers and repeats. Bare JSON null and boolean keywords do not supply identifier values; quoted schema labels followed by unquoted prose placeholders remain benign. Quoting a label alone does not establish an opaque string value.", "labels": [ { "label": "AGE", diff --git a/scripts/bench/agentic_layers.py b/scripts/bench/agentic_layers.py index ccdcaac20..15815dfab 100644 --- a/scripts/bench/agentic_layers.py +++ b/scripts/bench/agentic_layers.py @@ -41,9 +41,10 @@ import gaze_bench_score as score import government_id_cells as government_ids import record_id_cells as record_ids +import record_id_syntax_cells as record_id_syntax -GENERATOR_VERSION = 14 +GENERATOR_VERSION = 15 PARTITIONS = ("dev", "test") PUBLISHED_PARTITION = "test" PARTITION_SEEDS = {"dev": 2026092601, "test": 2026092602} @@ -4447,6 +4448,7 @@ def segment(data: bytes) -> str: 12: "zipage_", 13: "coverage_", 14: "recordids_", + 15: "recordids_syntax_", } @@ -4460,6 +4462,7 @@ def records_as_of(version: int, records: Iterable[Record]) -> list[Record]: # The committed contract each older generator version was scored under. HISTORICAL_CONTRACTS = { + 14: Path("docs/reference/benchmarks/scored-labels-agentic-generator-v14.json"), 13: Path("docs/reference/benchmarks/scored-labels-agentic-generator-v13.json"), 12: Path("docs/reference/benchmarks/scored-labels-agentic-generator-v12.json"), 11: Path("docs/reference/benchmarks/scored-labels-agentic-generator-v11.json"), @@ -4718,6 +4721,7 @@ def generate(partition: str) -> list[Record]: + _zip_age_records(partition) + _coverage_records(partition) + record_ids.records(sys.modules[__name__], partition) + + record_id_syntax.records(sys.modules[__name__], partition) ) check_lookalike_pairs(records) check_address_cells(records) diff --git a/scripts/bench/record_id_syntax_cells.py b/scripts/bench/record_id_syntax_cells.py new file mode 100644 index 000000000..79780907b --- /dev/null +++ b/scripts/bench/record_id_syntax_cells.py @@ -0,0 +1,59 @@ +"""Authored CC0 record-ID scalar syntax, independent of runtime rules. + +Numeric JSON scalars can be database identifiers. JSON null and booleans do +not supply an identifier value. Quoting a schema label does not turn the +following prose placeholder into an application key. +""" +from types import ModuleType + +LABELS = {"customer_id": "CUSTOMER_ID", "employee_id": "EMPLOYEE_ID", "record_id": "RECORD_ID"} +FIELDS = { + "dev": { + "customer_id": ("customer_id", "customer ID", "Kundennummer"), + "employee_id": ("employee_id", "employee ID", "Personalnummer"), + "record_id": ("person_record_id", "person record ID", "Personendatensatz-ID"), + }, + "test": { + "customer_id": ("customerId", "customer number", "Kunden-ID"), + "employee_id": ("employeeId", "employee number", "Mitarbeiter-ID"), + "record_id": ("contact_record_id", "contact record ID", "Kontaktdatensatz-ID"), + }, +} + + +def records(api: ModuleType, partition: str) -> list: + result = [] + + def emit(layer, family, surface, index, template, fields, language="en"): + text, gold, decoys = api._fill_with_decoys(template, fields) + result.append(api.Record( + uid=f"agentic-{partition}-{layer}-{family}-{index:03d}-{surface}", + partition=partition, layer=layer, family=family, surface=surface, + validity=api.BENIGN if layer == "D" else api.UNCHECKED, + group=f"{partition}-recordids-syntax-{family}-{index:03d}", + template=f"recordids-syntax/{family}/{surface}/{partition}", + language=language, region="DE" if language == "de" else "US", + text=text, gold=gold, decoys=decoys, + )) + + for family, label in LABELS.items(): + rng = api.Rng(api.PARTITION_SEEDS[partition], "recordids-syntax/" + family) + key, en, de = FIELDS[partition][family] + for index in range(api.DOCS_PER_FAMILY): + identifier = ("6" if partition == "dev" else "8") + rng.digits(5) + fields = {"K": (key, None), "V": (identifier, label), "T": (str(index + 1), None)} + emit("A", family, "recordids_syntax_json_numeric", index, + '{"tool":"lookup","arguments":{"{K}":{V}},"turn":{T}}', fields) + emit("R", family, "recordids_syntax_json_numeric_repeat", index, + '{"{K}":{V}}\nUse {V} next; confirmed {V}.', fields) + for keyword in ("null", "true", "false"): + emit("D", family, "recordids_syntax_json_" + keyword, index, + '{"tool":"lookup","arguments":{"{K}":{V}},"turn":{T}}', + {"K": (key, None), "V": (keyword, "decoy:json_keyword"), "T": (str(index + 1), None)}) + for index, word in enumerate(("token", "value", "placeholder", "example")): + for language, cue in (("en", en), ("de", de)): + emit("D", family, "recordids_syntax_quoted_label_" + language, index, + '[schema "{C}" {V}]', {"C": (cue, None), "V": (word, "decoy:placeholder")}, language) + emit("D", "customer_id", "recordids_syntax_bridge_quoted_label", 0, + '[session A "customer ID" {V}]', {"V": ("token", "decoy:placeholder")}) + return result diff --git a/scripts/bench/render_agentic_adjacency_doc.py b/scripts/bench/render_agentic_adjacency_doc.py index 5fd37206a..efde8176c 100644 --- a/scripts/bench/render_agentic_adjacency_doc.py +++ b/scripts/bench/render_agentic_adjacency_doc.py @@ -18,7 +18,7 @@ DOC = ROOT / "docs/reference/benchmarks/README.md" # Pin the measured version so a later generator cannot relabel these rows. # Earlier v4, v8, v9, v10 and v11 ledgers remain committed as historical measurements. -GENERATOR_VERSION = 14 +GENERATOR_VERSION = 15 HISTORY = ROOT / f"docs/reference/benchmarks/agentic-adjacency-v{GENERATOR_VERSION}-history.json" CORPUS_SHA256, CONTRACT_SHA256 = agentic_layers.corpus_identity(ROOT, GENERATOR_VERSION) @@ -225,7 +225,7 @@ def main() -> int: parser.add_argument("--check", action="store_true") parser.add_argument("--record", type=Path, action="append", default=[]) parser.add_argument('--generator-version', type=int, default=GENERATOR_VERSION, - choices=sorted({12, 13, GENERATOR_VERSION, agentic_layers.GENERATOR_VERSION}), + choices=sorted({12, 13, 14, GENERATOR_VERSION, agentic_layers.GENERATOR_VERSION}), help='keep measured history frozen by default; select the current corpus for new measurements') args = parser.parse_args() try: diff --git a/scripts/bench/test_agentic_layers.py b/scripts/bench/test_agentic_layers.py index 555a9d839..75b52e69b 100644 --- a/scripts/bench/test_agentic_layers.py +++ b/scripts/bench/test_agentic_layers.py @@ -29,8 +29,8 @@ # generator_version and these hashes together: a silent corpus change would # make base and candidate scorecards measure different documents. PINNED_CORPUS_SHA256 = { - "dev": "fa9df2412b489d2c94981f145e09666a0ff6009130597bd8022cba80b2415380", - "test": "1e07150bb9ba391488355b5212f0797bfd511b8095c8e3e2ebd6936627e26bec", + "dev": "3aa6c164957056dbad916fae6959a7ed75701cc656d2ade35f5492e044f669d1", + "test": "0a5f6576f2d5f5c8143628e88ec9d2985197fa3bd74c3823de86f611979c09ba", } # v9: everything before the URL cells. V9_CORPUS_SHA256 = { diff --git a/scripts/bench/test_record_id_syntax_cells.py b/scripts/bench/test_record_id_syntax_cells.py new file mode 100644 index 000000000..9db484222 --- /dev/null +++ b/scripts/bench/test_record_id_syntax_cells.py @@ -0,0 +1,75 @@ +import hashlib +import json +import unittest +from pathlib import Path + +import agentic_layers as agentic + + +ROOT = Path(__file__).resolve().parents[2] +V14_HASHES = { + "dev": "fa9df2412b489d2c94981f145e09666a0ff6009130597bd8022cba80b2415380", + "test": "1e07150bb9ba391488355b5212f0797bfd511b8095c8e3e2ebd6936627e26bec", +} + + +class RecordIdScalarSyntaxTests(unittest.TestCase): + def test_v14_corpus_remains_byte_identical(self): + for partition, expected in V14_HASHES.items(): + records = agentic.records_as_of(14, agentic.generate(partition)) + self.assertEqual(hashlib.sha256(agentic.corpus_bytes(records)).hexdigest(), expected) + contract = agentic.load_contract(ROOT, version=14) + frozen = ROOT / agentic.HISTORICAL_CONTRACTS[14] + self.assertEqual(hashlib.sha256(frozen.read_bytes()).hexdigest(), "ba4ade507783cd521d11aba13ff4992f0250d4df3fe1fd34a759f4e1ab2bdf19") + self.assertEqual(json.loads(frozen.read_text())["corpus"]["generator_version"], 14) + self.assertLessEqual({"CUSTOMER_ID", "EMPLOYEE_ID", "RECORD_ID"}, contract.scored_labels) + + def test_numeric_json_values_are_numbers_with_exact_gold_and_repeats(self): + for partition in agentic.PARTITIONS: + rows = agentic.record_id_syntax.records(agentic, partition) + self.assertEqual(len(rows), 175) + self.assertEqual({layer: sum(r.layer == layer for r in rows) for layer in ("A", "D", "R")}, + {"A": 30, "D": 115, "R": 30}) + self.assertEqual(len({r.uid for r in rows}), 175) + for record in rows: + self.assertEqual(bool(record.gold), record.layer != "D") + self.assertEqual(bool(record.decoys), record.layer == "D") + for span in (*record.gold, *record.decoys): + self.assertEqual(record.text.encode()[span.start:span.end].decode(), span.value) + if record.layer == "A": + value, = json.loads(record.text)["arguments"].values() + self.assertIs(type(value), int) + self.assertEqual(record.gold[0].value, str(value)) + self.assertEqual(len(str(value)), 6) + elif record.layer == "R": + self.assertEqual(len(record.gold), 3) + self.assertEqual(len({span.value for span in record.gold}), 1) + + def test_json_keywords_and_quoted_schema_labels_are_benign(self): + for partition in agentic.PARTITIONS: + rows = agentic.record_id_syntax.records(agentic, partition) + for family in agentic.record_id_syntax.LABELS: + for keyword in ("null", "true", "false"): + cells = [r for r in rows if r.family == family and r.surface == "recordids_syntax_json_" + keyword] + self.assertEqual(len(cells), 10) + self.assertEqual(len({r.text for r in cells}), 10) + for record in cells: + value, = json.loads(record.text)["arguments"].values() + self.assertTrue(value is None or type(value) is bool) + self.assertFalse(record.gold) + self.assertEqual(record.decoys[0].value, keyword) + quoted = [r for r in rows if "quoted_label" in r.surface] + self.assertEqual(len(quoted), 25) + self.assertTrue(all(not r.gold and r.decoys for r in quoted)) + self.assertTrue(any(r.text == '[session A "customer ID" token]' for r in quoted)) + + def test_positive_values_and_template_ids_are_partition_disjoint(self): + rows = [agentic.record_id_syntax.records(agentic, part) for part in agentic.PARTITIONS] + pools = [{span.value for r in partition for span in r.gold} for partition in rows] + templates = [{r.template for r in partition} for partition in rows] + self.assertFalse(pools[0] & pools[1]) + self.assertFalse(templates[0] & templates[1]) + + +if __name__ == "__main__": + unittest.main()