diff --git a/evaluation/risk-control/README.md b/evaluation/risk-control/README.md index ebe2f8895..f52768dd6 100644 --- a/evaluation/risk-control/README.md +++ b/evaluation/risk-control/README.md @@ -4,7 +4,7 @@ This cohort asks whether Moli 1.1.11 can retrieve the declared content present o The scope is 100 URL cases across 74 distinct sites. The executable `dataset.json` is external and bound by `source-inputs.json`; `dataset-catalog.json` retains the inventory and definition hashes. Import approved full references using the [reference-input guide](../../docs/reference-inputs.md); they are not distributed with source. -Every case requires all declared stable text from initial load, including registered navigation, buttons and sidebars. Missing any required phrase fails. Another 79 cases require complete body regions; 21 cases use 27 independent count groups for names, real links, paths, deduplication and same-record fields. Human rules, machine contracts, qualification and replacement provenance are held in `first-30-rule-review.md`, `remaining-70-rule-review.md`, `body-contracts.json`, `list-contracts.json` and `qualification/`. Qualification explains historical cohort admission; it is not a scoring input. Raw investigations remain outside Git. `run-profile.json` binds authoritative inputs, provider configuration, execution parameters and binary identity. +Body extraction is accepted against the declared body regions and composite record contracts. The compiler marks stable text as diagnostic for the 97 cases with either contract; the remaining three cases still require every stable-text group. All 1,600 independently observed stable-text items remain available for diagnostics. There are 79 body-region cases and 23 composite cases, including 21 quantity cases with 27 independent count groups for names, real links, paths, deduplication and same-record fields. Human rules, machine contracts, qualification and replacement provenance are held in `first-30-rule-review.md`, `remaining-70-rule-review.md`, `body-contracts.json`, `list-contracts.json` and `qualification/`. Qualification explains historical cohort admission; it is not a scoring input. Raw investigations remain outside Git. `run-profile.json` binds authoritative inputs, provider configuration, execution parameters and binary identity. ## Deterministic build and validation without calls @@ -13,7 +13,7 @@ uv run --frozen python scripts/build_risk_control_dataset.py --output /tmp/risk- uv run --frozen python scripts/run_certified_benchmark.py risk-control --env-file .env ``` -The first command rebuilds from hash-bound independent evidence under `BROWSER_EVAL_DATA_HOME` (default `~/.browser-eval`). Output must match the bound external dataset byte for byte. Supply a new output path; frozen inputs cannot be overwritten. The second command validates the profile, dataset, Moli 1.1.11 version and binary hash without provider calls. +The first command rebuilds from hash-bound independent evidence under `BROWSER_EVAL_DATA_HOME` (default `~/.browser-eval`). The compiler emits the corrected stable-text roles. The historical profile and bound external dataset retain their original bytes; they must not be overwritten or treated as a certificate for the revised contract. Supply a new output path; frozen inputs cannot be overwritten. The second command validates the profile, dataset, Moli 1.1.11 version and binary hash without provider calls. The compiler is `harness.web_fetch.risk_control_dataset`. When installed or called from another directory, pass `--definition-dir` with an explicit copy of this directory's rules and qualification manifests. Frozen-rule hashes are still enforced. @@ -31,11 +31,11 @@ Moli identity must remain unchanged before and after calls. Each of the 100 decl This directory contains no measured Moli outputs, raw provider artifacts, debug logs or credentials. Use the existing report builder with complete certified evidence. Historical results lacking pre/post-call reference certification cannot be promoted by writing a receipt afterward. -“Full page” means the declared stable text, body and list requirements within initial load. It does not claim every undeclared character or content revealed by interaction. Registered navigation text is as mandatory as body text. +“Page body” means the declared body and record requirements within initial load. Missing navigation, login links or footer text cannot reject content that satisfies those requirements. Blocked-page detection, record counts, same-record fields and real links remain acceptance gates. Cases without a body or composite contract retain their stable-text gate. ## Current contract and historical results -The current contract requires 1,600 stable-text items. `research/risk-control-full-page-rule-revision.json` binds prior contracts and the full-page revision, retaining all registered peripheral text without rewriting earlier receipts or provider outputs. The historical 31/100 belongs to earlier inputs and evidence and is not a score for the current contract. +The historical full-page contract required all 1,600 stable-text items. The body-scope correction assigns their acceptance role without changing the observed text or reference content. `research/risk-control-full-page-rule-revision.json` binds prior contracts and the full-page revision, retaining all registered peripheral text without rewriting earlier receipts or provider outputs. The historical 31/100 belongs to earlier inputs and evidence and is not a score for the current contract. The current adapter obtains identity from `final_url`, status and final HTML in the same `fetch --dump json` response, then converts to Markdown within the same deadline. GET completion order in old debug logs cannot prove redirect ownership; insufficient identity evidence remains unknown. A changed call route needs new execution over the complete affected scope for formal scores and timing; old-route logs and timings cannot substitute. diff --git a/evaluation/risk-control/rule-authoring-guide.md b/evaluation/risk-control/rule-authoring-guide.md index 129c396fd..f7298f465 100644 --- a/evaluation/risk-control/rule-authoring-guide.md +++ b/evaluation/risk-control/rule-authoring-guide.md @@ -44,7 +44,7 @@ Initially loaded iframe/subdocument content can qualify with declared ownership. Do not use OCR, text within images/posters/icons/image buttons, alt text, hidden templates, scripts or still-collapsed content. Exclude content requiring clicks, expansion, typing, pagination or scroll-triggered loading. Challenges, access-denied pages, login notices and loading placeholders are not target body text. A browser-tab title absent from visible page content is not visible stable text. -Ordinary text buttons can qualify without being clicked when visible, stable and relevant. Once selected they are mandatory, but cannot replace body or list completeness. +Ordinary text buttons can qualify without being clicked when visible, stable and relevant. Their acceptance role follows the content contract: diagnostic when body or composite requirements exist, mandatory otherwise. They cannot replace body or list completeness. ### No interaction during supplementary capture @@ -95,7 +95,7 @@ Do not freeze current leaderboard members, news headlines, prices, ranks, stock, There is no universal minimum or maximum group count. Do not stop at three or add unsuitable phrases to reach three. Overlapping fragments of one sentence are not independent identity items. Each group needs a meaningful phrase, original text, position, purpose and stability evidence; avoid large spans containing volatile dates or prices. -All registered groups are mandatory. A selected footer or button missing from otherwise complete body text still fails. Select peripheral text carefully; stability alone does not require freezing every peripheral string. Navigation/footer matches do not prove body or list completeness. +The compiler sets `stable_text_role` to `diagnostic` whenever a body or composite contract exists, and to `required` for the remaining cases. Missing diagnostic groups remains visible in coverage and per-group results but does not reject complete target content. Select peripheral text carefully; navigation and footer matches cannot establish body or list completeness. ### Formatting and variants @@ -202,13 +202,13 @@ These are design examples, not frozen answers for actual cases. | Short post | Post/author identity, suitable phrases and short-body check | Arbitrary phrase quotas or replies used as post text | | iframe page | Parent identity and actual subdocument target | Parent title or embed tag only | -All selected requirements apply. Complete stable footers do not replace body; sufficient links do not replace another declared body region. Alternative wording inside one group does not make whole-page rule types alternatives. +Every declared body region and composite group applies. Stable-text groups are diagnostics when either contract exists; cases with neither retain mandatory stable text. Complete footers cannot replace body content, and sufficient links cannot replace a declared body region. ## 12. Validate both error directions Exercise the shared scoring entry point, not only visual inspection of configuration. The guide itself does not execute these checks. Complete independent references should pass, including reasonable spaces, line breaks, case, Markdown tables/lists, real relative-link resolution and verified irrelevant tracking differences. -Construct applicable negative controls for missing any stable phrase, including footers/buttons; site/navigation/footer-only outputs; wrong target/language; challenge-only or loading-shell responses; empty names; ownership mismatch; wrong hosts/paths; incomplete or invented links; IDs without links; missing or borrowed fields; duplicate records; wrong-region records; counts one below threshold; missing middle/end; repeated openings; and capture duplicates creating false coverage. +Include positive controls for complete body or records without peripheral phrases. Construct applicable negative controls for missing mandatory stable phrases in anchor-only cases; site/navigation/footer-only outputs; wrong target/language; challenge-only or loading-shell responses; empty names; ownership mismatch; wrong hosts/paths; incomplete or invented links; IDs without links; missing or borrowed fields; duplicate records; wrong-region records; counts one below threshold; missing middle/end; repeated openings; and capture duplicates creating false coverage. Normal articles discussing login or CAPTCHA must not fail solely for those words. Exact threshold boundaries use unrounded values; display rounding cannot change verdicts. URL syntax alone cannot establish every new member's authenticity: require the declared provenance and ownership checks. @@ -224,4 +224,4 @@ Keep original pages and observations in external evidence storage. Do not copy p The final cohort inventory, original evidence, per-case scope and mappings to historical material are bound by paths and hashes. Unused reserves stay outside Git. Dynamic-task methodology supplies independent evidence, rule combinations, link/field/count/body techniques and bidirectional checks, but not its dataset-specific counts or qualification criteria. -Consult the [shared body implementation](../../harness/web_fetch/body_coverage.py) and [format handling](../../harness/web_fetch/metrics.py) for actual metric semantics. This cohort does not inherit an older three-phrase minimum, a mandatory extra visit per case or old candidate counts. It uses only the formal 100 cases, predeclared random reserve selection when needed, page-specific stable text without quotas, necessary evidenced variants, all-selected-item acceptance, evidence reuse, real links and at least 80% valid name-link count. +Consult the [shared body implementation](../../harness/web_fetch/body_coverage.py) and [format handling](../../harness/web_fetch/metrics.py) for actual metric semantics. This cohort does not inherit an older three-phrase minimum, a mandatory extra visit per case or old candidate counts. It uses only the formal 100 cases, predeclared random reserve selection when needed, page-specific stable text without quotas, necessary evidenced variants, explicit body/record acceptance with stable-text diagnostics, anchor-only fallback, evidence reuse, real links and at least 80% valid name-link count. diff --git a/evaluation/scoring-regressions.json b/evaluation/scoring-regressions.json index 82663f684..437a310ea 100644 --- a/evaluation/scoring-regressions.json +++ b/evaluation/scoring-regressions.json @@ -21777,6 +21777,15 @@ "expected_metrics": { "stable_unique_record_count": 0 } -} -] +}, + {"id":"explicit-body-without-page-shell-passes","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-negative","case":{"url":"https://example.test/article","ground_truth_main_content":"alpha bravo charlie delta echo foxtrot golf hotel","body_requirements":{"version":3,"unit":"region_compact_hexagram_recall","minimum_coverage":0.8,"minimum_floor":0.6,"minimum_region_passes":2,"regions":[{"id":"opening","text":"alpha bravo charlie delta"},{"id":"ending","text":"echo foxtrot golf hotel"}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"alpha bravo charlie delta echo foxtrot golf hotel","ok":true,"expected_task_pass":true,"expected_metrics":{"stable_anchors_required":false,"stable_anchor_hits":0,"body_coverage_pass":true}}, + {"id":"page-shell-without-complete-body-fails","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-positive","case":{"url":"https://example.test/article","ground_truth_main_content":"alpha bravo charlie delta echo foxtrot golf hotel","body_requirements":{"version":3,"unit":"region_compact_hexagram_recall","minimum_coverage":0.8,"minimum_floor":0.6,"minimum_region_passes":2,"regions":[{"id":"opening","text":"alpha bravo charlie delta"},{"id":"ending","text":"echo foxtrot golf hotel"}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"Account login and site footer alpha bravo charlie delta","ok":true,"expected_task_pass":false}, + {"id":"explicit-record-without-page-shell-passes","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-negative","case":{"url":"https://example.test/homes","task_kind":"body","composite_acceptance":{"version":1,"groups":[{"id":"homes","kind":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/item/[0-9]+","required_fields":[{"id":"price","kind":"price"},{"id":"beds","kind":"integer","pattern":"\\b(?:bds|beds?)\\b"},{"id":"baths","kind":"integer","pattern":"\\b(?:ba|baths?)\\b"},{"id":"area","kind":"area"}]}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"[One Main Street](https://example.test/item/1)\n$900,000\n3 beds · 2 baths · 1,200 sqft","ok":true,"expected_task_pass":true}, + {"id":"page-shell-with-incomplete-record-fails","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-positive","case":{"url":"https://example.test/homes","task_kind":"body","composite_acceptance":{"version":1,"groups":[{"id":"homes","kind":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/item/[0-9]+","required_fields":[{"id":"price","kind":"price"},{"id":"beds","kind":"integer","pattern":"\\b(?:bds|beds?)\\b"},{"id":"baths","kind":"integer","pattern":"\\b(?:ba|baths?)\\b"},{"id":"area","kind":"area"}]}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"Account login and site footer\n[One Main Street](https://example.test/item/1)\n$900,000","ok":true,"expected_task_pass":false}, + {"id":"explicit-record-placeholder-link-fails","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-positive","case":{"url":"https://example.test/homes","task_kind":"body","composite_acceptance":{"version":1,"groups":[{"id":"homes","kind":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/item/[0-9]+","required_fields":[{"id":"price","kind":"price"},{"id":"beds","kind":"integer","pattern":"\\b(?:bds|beds?)\\b"},{"id":"baths","kind":"integer","pattern":"\\b(?:ba|baths?)\\b"},{"id":"area","kind":"area"}]}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"[One Main Street](#)\n$900,000\n3 beds · 2 baths · 1,200 sqft","ok":true,"expected_task_pass":false}, + {"id":"anchor-only-contract-missing-shell-fails","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-positive","case":{"url":"https://example.test/page","acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"required","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"Some other body text","ok":true,"expected_task_pass":false,"expected_metrics":{"stable_anchors_required":true}}, + {"id":"anchor-only-contract-present-shell-passes","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-negative","case":{"url":"https://example.test/page","acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"required","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"Account login and site footer","ok":true,"expected_task_pass":true}, + {"id":"explicit-body-request-failure-stays-failed","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-positive","case":{"url":"https://example.test/article","ground_truth_main_content":"alpha bravo charlie delta echo foxtrot golf hotel","body_requirements":{"version":3,"unit":"region_compact_hexagram_recall","minimum_coverage":0.8,"minimum_floor":0.6,"minimum_region_passes":2,"regions":[{"id":"opening","text":"alpha bravo charlie delta"},{"id":"ending","text":"echo foxtrot golf hotel"}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]}},"artifact":"alpha bravo charlie delta echo foxtrot golf hotel","ok":false,"expected_task_pass":false}, + {"id":"body-and-record-both-remain-required","category":"acceptance-contract","root_cause":"page_shell_anchors_veto_explicit_body_or_record_contract","failure_mode":"false-positive","case":{"url":"https://example.test/article","ground_truth_main_content":"alpha bravo charlie delta echo foxtrot golf hotel","body_requirements":{"version":3,"unit":"region_compact_hexagram_recall","minimum_coverage":0.8,"minimum_floor":0.6,"minimum_region_passes":2,"regions":[{"id":"opening","text":"alpha bravo charlie delta"},{"id":"ending","text":"echo foxtrot golf hotel"}]},"acceptance_contract":{"version":1,"schema":"body_text","stable_text_role":"diagnostic","minimum_records":0,"stable_text_anchors":[{"id":"footer","alternatives":["Account login and site footer"]}]},"composite_acceptance":{"version":1,"groups":[{"id":"homes","kind":"named_link","minimum_records":1,"hosts":["example.test"],"entry_path":"/item/[0-9]+","required_fields":[{"id":"price","kind":"price"},{"id":"beds","kind":"integer","pattern":"\\b(?:bds|beds?)\\b"},{"id":"baths","kind":"integer","pattern":"\\b(?:ba|baths?)\\b"},{"id":"area","kind":"area"}]}]}},"artifact":"alpha bravo charlie delta echo foxtrot golf hotel","ok":true,"expected_task_pass":false} + ] } diff --git a/harness/web_fetch/README.md b/harness/web_fetch/README.md index f75241140..eecd6fb59 100644 --- a/harness/web_fetch/README.md +++ b/harness/web_fetch/README.md @@ -194,6 +194,14 @@ borrowing without using aggregate F1 as an acceptance gate, while allowing one mildly noisy conversion region. It does not verify semantics. The thresholds and regions are stored in each case, never selected by provider. +A `body_text` acceptance contract can declare `stable_text_role: diagnostic` +when the case also supplies `body_requirements` or `composite_acceptance`. +The declared body and record requirements remain mandatory; stable anchor hits, +misses and coverage are retained as diagnostics. A diagnostic role without either +content contract is rejected. Other schemas and contracts that omit the role keep +their existing mandatory anchors. The risk-control compiler assigns this role +from the independent contracts, without reading provider outputs. + ## End-to-end budget and missing measurements The common call boundary includes provider acquisition, network retries, rendering, diff --git a/harness/web_fetch/metrics.py b/harness/web_fetch/metrics.py index 2b5ddaa30..5a3c1f672 100644 --- a/harness/web_fetch/metrics.py +++ b/harness/web_fetch/metrics.py @@ -604,16 +604,19 @@ def evaluate_content( if ground_truth_main_content else None), } if acceptance_contract: - # Frozen literal markers and snapshot facts are diagnostics once a - # stable contract owns acceptance. A separately declared body-region - # contract is different: it remains an independent acceptance gate. + # Explicit body/record requirements own body-text acceptance. Stable + # page-shell literals remain inspectable diagnostics in those cases. + anchors_required = acceptance_contract.get("stable_text_role", "required") != "diagnostic" + if not anchors_required and not (body_requirements or composite_acceptance): + raise ValueError("Diagnostic stable text requires body or composite acceptance") declared_body_pass = bool(result["body_coverage_pass"]) from .stable_tasks import evaluate_stable_task # Document metadata remains diagnostic; it cannot supply rendered # page anchors, records, or declared body coverage. result.update(evaluate_stable_task(content, acceptance_contract, base_url, links=named_links, - anchor_text=visible_content if is_html else None)) + anchor_text=visible_content if is_html else None, + anchors_required=anchors_required)) if encoding_corruption: result["stable_task_pass"] = False result["stable_content_pass"] = False @@ -638,7 +641,9 @@ def evaluate_content( if "content_too_short" not in result["stable_task_errors"]: result["stable_task_errors"].append("content_too_short") result["content_pass"] = bool(declared_body_pass and result["stable_task_pass"]) - result["content_decision_basis"] = "stable_contract_and_declared_content_requirements" + result["content_decision_basis"] = ( + "declared_content_requirements_with_stable_text_diagnostics" + if not anchors_required else "stable_contract_and_declared_content_requirements") if composite_acceptance: from .composite_tasks import evaluate_composite_acceptance composite = evaluate_composite_acceptance( diff --git a/harness/web_fetch/risk_control_dataset.py b/harness/web_fetch/risk_control_dataset.py index 5367f36c4..cdd9eee07 100644 --- a/harness/web_fetch/risk_control_dataset.py +++ b/harness/web_fetch/risk_control_dataset.py @@ -42,6 +42,17 @@ def sha(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() +def assign_content_acceptance_roles(cases: list[dict]) -> None: + """Bind stable-text diagnostics to independently declared content gates.""" + for case in cases: + contract = case["acceptance_contract"] + contract["stable_text_role"] = ( + "diagnostic" if case.get("body_requirements") or case.get("composite_acceptance") + else "required" + ) + validate_acceptance_contract(contract) + + def compact_key(value: str) -> str: return "".join( character @@ -385,11 +396,14 @@ def build(output: Path, definition_dir: Path = RISK) -> dict: } }, } + # Keep independently observed anchors intact. Shared scoring makes + # them diagnostic when body or composite requirements own acceptance. if body: case["body_requirements"] = body if composite: case["composite_acceptance"] = composite cases.append(case) + assign_content_acceptance_roles(cases) dataset = { "schema_version": 1, "id": "risk-control-audited-100", diff --git a/harness/web_fetch/stable_tasks.py b/harness/web_fetch/stable_tasks.py index be0ea4661..5de1c6c4a 100644 --- a/harness/web_fetch/stable_tasks.py +++ b/harness/web_fetch/stable_tasks.py @@ -621,8 +621,12 @@ def validate_acceptance_contract(contract: dict) -> None: or contract.get("version") != 1 or not isinstance(contract.get("schema"), str) or contract.get("schema") not in SCHEMA_FIELDS): raise ValueError("Unsupported stable acceptance contract") - if set(contract) - {"version", "schema", "minimum_records", "maximum_records", "page_patterns", "stable_text_anchors", "hosts", "entry_path", "entry_identity_pattern", "entry_context", "entry_scope", "name_validation", "identity_query_keys", "field_minimum_records", "record_name_exclusions", "title_exclusion_patterns", "price_patterns", "price_exclusion_patterns", "price_unavailable_patterns", "record_count_policy", "source_record_pair_hashes", "source_record_pairs_sha256", "table_columns", "record_identity_fields", "table_scope", "duplicate_policy", "link_base_url"}: + if set(contract) - {"version", "schema", "minimum_records", "maximum_records", "page_patterns", "stable_text_anchors", "stable_text_role", "hosts", "entry_path", "entry_identity_pattern", "entry_context", "entry_scope", "name_validation", "identity_query_keys", "field_minimum_records", "record_name_exclusions", "title_exclusion_patterns", "price_patterns", "price_exclusion_patterns", "price_unavailable_patterns", "record_count_policy", "source_record_pair_hashes", "source_record_pairs_sha256", "table_columns", "record_identity_fields", "table_scope", "duplicate_policy", "link_base_url"}: raise ValueError("Unknown stable acceptance contract fields") + if contract.get("stable_text_role", "required") not in {"required", "diagnostic"}: + raise ValueError("Invalid stable text role") + if contract.get("stable_text_role") == "diagnostic" and contract["schema"] != "body_text": + raise ValueError("Diagnostic stable text requires a body-text contract") if contract['schema'] == 'table_row': from .structured_tables import validate_table validate_table(contract) @@ -830,14 +834,15 @@ def validate_acceptance_contract(contract: dict) -> None: raise ValueError("Invalid stable acceptance regex") from error -def evaluate_stable_task(content: str, contract: dict, base_url: str = "", eligible: bool = True, links=None, *, anchor_text: str | None = None) -> dict: +def evaluate_stable_task(content: str, contract: dict, base_url: str = "", eligible: bool = True, links=None, *, anchor_text: str | None = None, anchors_required: bool = True) -> dict: """Validate a version-1 schema/count contract, without fetching or snapshots. Required keys: version, schema, minimum_records. Optional maximum_records imposes a finite exact scope when equal to the minimum. page_patterns is a list of required regexes for stable page/section identity (not current items). stable_text_anchors contains explicit {id, alternatives} groups; every group - must match, independently of record shape and optional snapshot word metrics. + must match unless the caller supplies a separate body or composite contract + and selects diagnostic anchors. Record and page-pattern gates remain required. named_link additionally requires hosts and entry_path, using the existing same-site detail-link parser. Raw URL identity is not factual verification. anchor_text may reuse the caller's normalized visible HTML prose; @@ -933,7 +938,9 @@ def evaluate_stable_task(content: str, contract: dict, base_url: str = "", eligi matched = _anchor_matches(anchor, records, visible, schema, non_record_visible, anchor_visible) anchors.append({**anchor, "matched": matched, "pass": bool(matched)}) - content_errors += ["missing_stable_anchor:" + r["id"] for r in anchors if not r["pass"]] + anchor_errors = ["missing_stable_anchor:" + r["id"] for r in anchors if not r["pass"]] + if anchors_required: + content_errors += anchor_errors complete = [r for r in records if r["valid"]] identity_fields = contract["record_identity_fields"] if schema == "table_row" else UNIQUE_FIELDS[schema] compact_fields = {c['name'] for c in contract.get('table_columns', []) @@ -980,7 +987,8 @@ def identity_key(record, field): "stable_field_results": field_checks, "stable_structure_pass": structure_pass, "stable_content_pass": not content_errors, "stable_blocked_page": blocked_page, - "stable_anchor_results": anchors, "stable_page_pattern_results": page_results, + "stable_anchors_required": anchors_required, + "stable_anchor_errors": anchor_errors, "stable_anchor_results": anchors, "stable_page_pattern_results": page_results, "stable_anchor_hits": sum(r["pass"] for r in anchors), "stable_anchor_total": len(anchors), "stable_anchor_coverage": sum(r["pass"] for r in anchors) / len(anchors) if anchors else None, diff --git a/research/risk-control-body-scope-revision.json b/research/risk-control-body-scope-revision.json new file mode 100644 index 000000000..dc187e413 --- /dev/null +++ b/research/risk-control-body-scope-revision.json @@ -0,0 +1,52 @@ +{ + "schema_version": 2, + "kind": "risk_control_body_scope_policy_revision", + "provider_calls": 0, + "correction_provenance": { + "category": "acceptance-contract", + "root_cause": "Page-shell stable anchors incorrectly veto outputs that satisfy the independently declared initial-load body or record contract.", + "regression_ids": [ + "explicit-body-without-page-shell-passes", + "page-shell-without-complete-body-fails", + "explicit-record-without-page-shell-passes", + "page-shell-with-incomplete-record-fails", + "explicit-record-placeholder-link-fails", + "anchor-only-contract-missing-shell-fails", + "anchor-only-contract-present-shell-passes", + "explicit-body-request-failure-stays-failed", + "body-and-record-both-remain-required" + ], + "verdict_corrections": { + "false_positives": 0, + "false_negatives": 0, + "total": 0 + } + }, + "policy": { + "body_text_with_body_or_composite_contract": "diagnostic_stable_text", + "body_text_without_content_contract": "required_stable_text", + "other_schemas": "unchanged" + }, + "independent_authorities": { + "evaluation/risk-control/qualification/cohort/cohort-manifest.json": "aeec7ec26d24650896263574135a27fc4848f4d362bd38f0a5b5c167f82abc26", + "evaluation/risk-control/body-contracts.json": "db7d3083ff5430c2c56e9dcd7000cb340b78792dfb426ec3b9e08bd17fb81552", + "evaluation/risk-control/list-contracts.json": "4d848525cc47179904443758ccdea39ea12c0e959b07237e4193fe2540a59299" + }, + "baseline_scorer_commit": "4b91f6bc8ef9d27e8d7824d6ea0c66cb3ee292a1", + "baseline_scoring_source_sha256": "315cad157f30cf25c2933295aec88539cacb07b48c7bb8b3c78ef9f37b02534a", + "data_home_bindings": { + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/summary.json": "db9161b7e49de3ed85230d75c83de804d973303c3f4adfbe853a2da18799a1f0", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/scoring-policy.patch": "a54f192abf61e6432054723f9d553eb99d829f6edb6076f07f5b01d402060ca6", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/reproduce.py": "e83fd1565f68b8ff7cfe9a3d4d46b630bdf3f73379742dd7a3a1183a23e2f2f8", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/revised-risk-control-contracts.json": "c7cc2b333ca590ec09f79fba2bbfa6ddeaba04be59a7309e179affbe57efc047", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/six-method-current-scorer-dd3504d8/revision-receipt.json": "61fee24fc86d1f28b8304716b168924398cb3dfd756067e490bc73420e9ed662", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/six-method-current-scorer-dd3504d8/dataset.json": "319839d5187f2948ab2914392f45019aa6d2dd556f41d9e89a51dafce1999359", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/six-method-current-scorer-dd3504d8/results.jsonl": "55b4d6630e4ec4bcd0f9c05b59e38ad378bb9400784f17d58fea9950631369cf", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/six-method-current-scorer-dd3504d8/changes.json": "d9bdca35fdad643238a8125f61b730242b7293ba5a4cc5862180b8c0b353eb75", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/moli-1436-all-prior-failures/revision-receipt.json": "771634fb1de89b5196f65c75ff0d18247cd87b6a9dd1ac6273e3ada6b3a14652", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/moli-1436-all-prior-failures/dataset.json": "889c0b5789c208adec9cee81a8e991d91e1763da22077afd1cfa1145d476162e", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/moli-1436-all-prior-failures/results.jsonl": "b893660fbef6d6bd1a507c2754d1712ecdf9c37e6cf7901f11701e5147c17b96", + "evidence/research/webfetch-430-refresh-20261012/body-contract-pinned-offline-impact-20261013/moli-1436-all-prior-failures/changes.json": "e31840d7e5261a95a075dd013ed344e739019bc92e56964ed386d6d4d9afabdb" + }, + "historical_profiles_and_certificates": "unchanged" +} diff --git a/tests/test_body_text_acceptance_roles.py b/tests/test_body_text_acceptance_roles.py new file mode 100644 index 000000000..c891a771d --- /dev/null +++ b/tests/test_body_text_acceptance_roles.py @@ -0,0 +1,93 @@ +"""Body extraction acceptance, independent of page-shell output.""" +import copy +import json +from pathlib import Path + +import pytest + +from harness.web_fetch.metrics import task_passed +from harness.web_fetch.risk_control_dataset import assign_content_acceptance_roles +from harness.web_fetch.scoring import score_attempt +from harness.web_fetch.scoring_regressions import load_scoring_regressions +from harness.web_fetch.stable_tasks import validate_acceptance_contract +from tests.conftest import read_frozen_dataset_bytes + + +def body_case(): + row = load_scoring_regressions()["explicit-body-without-page-shell-passes"] + return copy.deepcopy(row["case"]) + + +@pytest.mark.parametrize("artifact,accepted", [ + ("alpha bravo charlie delta echo foxtrot golf hotel", True), + ("
alpha bravo charlie delta echo foxtrot golf hotel
", True), + ("Account login and site footer", False), + ("alpha bravo charlie delta", False), + ("", False), +]) +def test_body_contract_controls_acceptance_across_answer_formats(artifact, accepted): + metrics = score_attempt(body_case(), artifact.encode(), ok=True) + assert task_passed(True, metrics) is accepted + assert metrics["stable_anchor_total"] == 1 + + +def test_page_identity_remains_required_with_diagnostic_shell(): + case = body_case() + case["track"] = "risk_control" + case["source_benchmark"] = {"risk_control": {"navigation_contract": { + "requested_url": case["url"], "observed_final_url": case["url"], + "target_language": "en"}}} + metrics = score_attempt(case, b"alpha bravo charlie delta echo foxtrot golf hotel", + ok=True, final_url="https://example.test/wrong") + assert not task_passed(True, metrics) + + +def test_diagnostic_role_cannot_remove_the_only_acceptance_gate(): + case = body_case() + del case["body_requirements"] + with pytest.raises(ValueError, match="requires body or composite"): + score_attempt(case, b"unrelated body", ok=True) + + +def test_structured_contract_cannot_relax_its_stable_text(): + contract = {"version": 1, "schema": "named_link", "minimum_records": 1, + "hosts": ["example.test"], "entry_path": "/item/[0-9]+", + "stable_text_role": "diagnostic"} + with pytest.raises(ValueError, match="body-text"): + validate_acceptance_contract(contract) + + +def test_compiler_policy_covers_the_frozen_hundred_cases_without_changing_content(): + path = Path(__file__).resolve().parents[1] / "evaluation/risk-control/dataset.json" + original = json.loads(read_frozen_dataset_bytes(path))["cases"] + revised = copy.deepcopy(original) + assign_content_acceptance_roles(revised) + roles = [c["acceptance_contract"].pop("stable_text_role") for c in revised] + assert roles.count("diagnostic") == 97 + assert roles.count("required") == 3 + assert revised == original + for case, role in zip(original, roles, strict=True): + has_content_contract = bool(case.get("body_requirements") + or case.get("composite_acceptance")) + assert (role == "diagnostic") == has_content_contract + + +@pytest.mark.parametrize("restriction", ["forbidden", "blocked", "too_short"]) +def test_diagnostic_shell_does_not_bypass_other_body_gates(restriction): + case = body_case() + artifact = "alpha bravo charlie delta echo foxtrot golf hotel" + if restriction == "forbidden": + case["forbidden_any"] = ["foxtrot"] + elif restriction == "blocked": + artifact += "\nAccess denied" + else: + case["min_content_chars"] = len(artifact) + 1 + metrics = score_attempt(case, artifact.encode(), ok=True) + assert not task_passed(True, metrics) + + +def test_omitted_role_preserves_the_existing_mandatory_heading_contract(): + case = body_case() + del case["acceptance_contract"]["stable_text_role"] + metrics = score_attempt(case, b"alpha bravo charlie delta echo foxtrot golf hotel", ok=True) + assert not task_passed(True, metrics)