diff --git a/.agents/skills/benchmark-report/SKILL.md b/.agents/skills/benchmark-report/SKILL.md index 5e63bc9fc..9486bd268 100644 --- a/.agents/skills/benchmark-report/SKILL.md +++ b/.agents/skills/benchmark-report/SKILL.md @@ -19,12 +19,13 @@ Public report CLIs default to English where supported and expose `--language en| 1. Establish the decision being supported, the measured unit, cohort, versions, execution location, freshness policy, and intended deliverable. Inspect the actual dataset, runner, outputs, scorer, prior review evidence, and current report. Do not infer them from screenshots or prose. 2. Freeze a machine-readable evaluation contract before expensive calls. It must identify the exact cases and methods, run count, timeout, concurrency policy, input and output contract, scoring provenance, reference state, and publication target when one exists. Existing compiled reference answers are frozen by default. Source metadata, capture dates, and absent newer acceptance-contract fields do not imply a live-reference task. Only tasks explicitly declaring `source_verification.reference_mode: live_exact` require live checks. For those tasks, fail closed: verify and freeze references immediately before calls, verify the same declared scope afterward, and certify the cohort only when both checks pass. An unchanged compiled acceptance-contract dataset is already the benchmark input: freeze its task and rule bytes for execution, without requiring historical source-qualification raw files or re-auditing cohort admission. Qualify sources when adding tasks or changing references/acceptance rules; do not turn an ordinary version rerun into dataset reconstruction. Partial or uncertified cohorts remain diagnostic evidence and cannot enter a release. -3. Let each method use its strongest documented route to produce the required output. Equalize the task, inputs, output contract, call budget, and scoring. Do not require identical internals. Record material asymmetries such as local versus remote execution, cold versus warm start, bundled conversion, network transit, account tier, and billable time. -4. Preserve evidence as an append-only chain: task definition, reference, provider call, raw artifact, normalized artifact, score, review, revision, and release. Keep provider output and call identity immutable. A presentation or scoring change must not masquerade as a new provider result. +3. Let each method use its strongest documented route to produce the required output. Equalize the task, inputs, output contract, call budget, and scoring. Do not require identical internals. For fixed-native cohorts, declare the input media type independently of URL patterns and expected answers, freeze the class-level route and transport parameters before calls, and verify the retained request receipts at aggregation and release. Record material asymmetries such as local versus remote execution, cold versus warm start, bundled conversion, network transit, account tier, and billable time. +4. Preserve evidence as an append-only chain: task definition, reference, provider call, raw artifact, normalized artifact, score, review, revision, and release. Keep provider output and call identity immutable. A presentation or scoring change must not masquerade as a new provider result. Declare the native answer fields and validate their structural projection before accepting a scored artifact. Adapters must not remove returned candidates by URL meaning or consult expected answers when selecting candidate fields. Apply privacy redaction only to separately identified public views after scoring, preserving the private originals and distinguishing original, scored and public-view hashes. 5. Separate retrieval success, task acceptance, content quality, latency, cost, and freshness. A missing or partial reference makes the affected metric unavailable or scoped; it does not prove success or failure. Do not derive binary acceptance from an unrelated aggregate similarity score. 6. For heterogeneous same-window latency evidence, preserve the full predeclared task-by-method matrix. Show all-call latency on that common denominator separately from successful-call latency on each method's conditional subset. Treat the all-method shared-pass intersection as a diagnostic, never as a post-outcome selector or ranking sample. 7. At aggregate and publication boundaries, copy only explicit hash-bound evidence and public audit files. Reject symlinks and exclude worker trees, host paths, credentials, and internal run logs. Verify primary, auxiliary, delivered-file, and retained retry artifacts before release. -8. Independent result review is opt-in: do not start it unless the user explicitly requests review. When requested without a broader scope, review machine failures only; review machine passes or every attempt only when the user explicitly asks for full review. Keep the initial verdict blind, but do not equate agreement with correctness: run the anomaly investigation in [audit and revision](references/audit-and-revision.md) for triggered results before calling the review complete. For a large independent review that is safely batchable, prefer resumable Codex CLI jobs with an exact model ID, read-only sandbox, strict output schema, non-overlapping packets, and bounded concurrency; persist each completed packet so a retry never repeats substantive review. Record the exact reviewed/total coverage, accepted unreviewed passes, reviewer, disagreements, adjudications, anomalies, and unresolved items. A review summary never replaces per-attempt evidence. + When a saved-input handoff needs host-metadata redaction, use the shared publication privacy owner. Preserve private originals and distinguish original from public file hashes; rebind dependent metadata references and disclose auxiliary debug-log projections. Never rewrite original scored answers, references or response payloads as a privacy repair. When a scored answer or native envelope itself contains private values, create a separately identified post-score public view through the shared privacy owner, verify its native-field projection and unchanged task verdict, and disclose that omitted original bytes cannot be reconstructed from the public copy. A public frozen presentation must explicitly bind its complete view inventory, verify displayed bytes against those views, keep acquisition gates strict, and distinguish preserved original diagnostics from what redacted text can reproduce. Verify unchanged verdicts, timings and review decisions, and rebuild from the public inputs. Keep regenerated package/report wrappers outside the original-input projection receipt so normal rebuilding does not invalidate it or create circular hashes. +8. Independent result review is opt-in: do not start it unless the user explicitly requests review. When requested without a broader scope, review machine failures only; review machine passes or every attempt only when the user explicitly asks for full review. Label a review performed with known machine outcomes as an evidence audit, separately from independent blind-review coverage; a fresh review-origin tag alone does not establish blindness. Keep the initial verdict blind, but do not equate agreement with correctness: run the anomaly investigation in [audit and revision](references/audit-and-revision.md) for triggered results before calling the review complete. For a large independent review that is safely batchable, prefer resumable Codex CLI jobs with an exact model ID, read-only sandbox, strict output schema, non-overlapping packets, and bounded concurrency; persist each completed packet so a retry never repeats substantive review. Record the exact reviewed/total coverage, accepted unreviewed passes, reviewer, disagreements, adjudications, anomalies, and unresolved items. A review summary never replaces per-attempt evidence. For an explicitly requested full review, set `independent_result_review_scope: all` in the report profile and carry it into the assembled dataset comparison. The release boundary must reject failure-only review, missing attempts and accepted unreviewed passes. Reuse an existing content judgment only when the raw answer and acceptance contract are unchanged; verify each new call identity separately. For failed calls, inspect the current runtime evidence even when the answer is unchanged or empty: the same empty output can hide different failures. Changed evidence needs supplemental review, not another review of the unchanged population. 9. Rerun only calls whose inputs, provider route/version, runtime contract, or source validity changed. Recompute downstream scores and reports without provider calls when only scoring, annotations, or presentation changed. Never retry merely to obtain a better result. A reference or verdict correction must identify the shared root cause and add executable positive/negative controls through the production scorer; bind those regression IDs in the revision receipt instead of patching stored verdicts. diff --git a/evaluation/lexbench-webfetch.json b/evaluation/lexbench-webfetch.json index 8e8d8b6b7..7a8484849 100644 --- a/evaluation/lexbench-webfetch.json +++ b/evaluation/lexbench-webfetch.json @@ -2,7 +2,7 @@ "schema_version": 1, "kind": "external_benchmark_binding", "repository": "https://github.com/lexmount/lexbench-webfetch", - "revision": "6d8d72637bb96f4938b9a2f7c46d0c7dffd74945", + "revision": "c75efb365c7133d9f82f70fb1044f2ba88a36445", "ownership": { "benchmark": "Task populations, task model, acceptance scoring, file checks, compiler and frozen definitions", "harness": "Execution, reference lifecycle, evidence and reports" @@ -78,7 +78,7 @@ "evaluation/risk-control/remaining-70-rule-review.md": "301574dddcfddf6d30ed4d882cdd40b1ae859fdc95a0a02cf9868f55e3fd1079", "evaluation/risk-control/rule-authoring-guide.md": "74363ab502978179e5438a0b992bac7563090e41c91784e492636832dc1cd088", "evaluation/risk-control/source-inputs.json": "77c5a4b62c07d59ed57154a833e85f9e7a97bc739730f51cd4ca3a698edb9258", - "evaluation/scoring-regressions.json": "d519f86e327d9e2fd1c884670642eb31714435864e834592e2c7b65ef21ee655", + "evaluation/scoring-regressions.json": "159adaa11e5272adfe51555fddeba33283c5e5809a29a1d30bfc5ad911bcbe70", "evaluation/source-inputs.json": "c202abc4624811b938a837402bbaf0ecbe716e375fbca70567346d2edb3b1708" } } diff --git a/harness/reporting/links.py b/harness/reporting/links.py index a769f8fe0..971bbd02c 100644 --- a/harness/reporting/links.py +++ b/harness/reporting/links.py @@ -3,6 +3,7 @@ import argparse import posixpath +import re from html.parser import HTMLParser from pathlib import Path from urllib.parse import parse_qs, unquote, urlsplit @@ -94,6 +95,12 @@ def parse_page(path: Path) -> Links: return parsed_pages[path] for page in pages: + # Frozen source HTML retains the original site's relative navigation. + # It is evidence, while report links pointing to it still need to exist. + relative = page.relative_to(root) + if "local-evidence" in relative.parts and not re.search( + r']*name=[\'"]browser-eval-report[\'"]', page.read_text(encoding="utf-8", errors="replace")): + continue parser = parse_page(page) base_dir = root if (root / 'sample-manifest.json').is_file() else page.parent if parser.base_href: diff --git a/harness/web_fetch/README.md b/harness/web_fetch/README.md index e016f54de..e9392622f 100644 --- a/harness/web_fetch/README.md +++ b/harness/web_fetch/README.md @@ -655,6 +655,45 @@ protocol remain dated observations and cannot be certified retroactively. ### Native caller limits and DNS +With `uniform_native_parameters`, webpage body/structure calls use the same +native request regardless of the web, dynamic, popular (`challenge`), or +risk-control family. Case-specific retrieval hints remain prohibited. Scoring +keeps the original case and frozen acceptance contract; family is not a reason +to inject scripts, convert HTML to Markdown, or fetch another URL. + +New fixed-native cohorts declare `source_benchmark.input_media_kind` for every +case (`html_page`, `direct_pdf`, `direct_docx`, `direct_text`, `direct_image`, or +`direct_media`). JSONL workers receive the same declaration as +`input_media_kind`. Native document routes use this frozen type, not URL suffixes +or expected answers. Historical calls without this declaration are not silently +upgraded to the new protocol. + +Before any case call, the runner writes `native-route-contract.json`, binding +every method/track/task-kind/input-type cell to its interface, answer fields, +configuration and normalized transport parameters. Actual arguments are checked +before transport; changing the task URL, endpoint or parameters fails the run. +Each physical call permits one adapter invocation and retains its own request +receipt, including every caller retry. Each receipt binds to the pre-call contract. +Aggregation retains the exact contract bytes; report and release checks verify +its hash, complete declared scope and per-result receipts. A retrospective audit +of old logs cannot substitute for this pre-call proof. + +Resource tasks may select a declared native output: Firecrawl image/link/video +fields, Cloudflare's generic media-element scrape, and Crawl4AI media fields. +Moli uses native HTML for resource discovery and native Markdown for webpage +body/structure, without caller-authored extraction scripts. +File tasks may collect Moli's raw output, Firecrawl rawBase64, or Crawl4AI's +native downloaded files. No caller follows returned links to manufacture files. +Document text uses the product's own URL handling; Crawl4AI direct PDFs select +its native PDF strategies. Moli direct PDF/DOCX responses remain original +file bytes, not extracted text; a body contract cannot pass on bytes alone. +The fixed mode never uses the legacy Moli external +document parser or Cloudflare download-then-upload converter. Unsupported or +incomplete native output remains incomplete under the unchanged scoring rules. +Cloudflare's fixed media selectors are independent of reference answers, and +native resource/PDF routes retain product-default internal timeouts. + + Native workers receive the caller deadline and fixed native capacity options, and record effective limits. In fixed-native mode, the outer deadline does not override product timers. CowAgent uses its native `DEFAULT_TIMEOUT` unchanged. @@ -669,6 +708,25 @@ product retrieval failure. Full native tool/worker envelopes are retained privat with hashes, separately from scored answer text. Historical outputs without those envelopes remain insufficient for an exact raw-response audit. +When those retained bytes contain private host paths or credentials, the shared +`publication_privacy.project_evidence_view` creates a separate post-score copy +under `public-evidence/`. Keep the originals private. The bound receipt records +original and public hashes and byte sizes; `require_public_evidence_view` checks +the delivered copy without claiming it can reconstruct omitted values. Native +JSON envelopes are projected structurally to preserve field types. Before using +such views in a handoff, verify that the public native-field projection still +matches the public answer and that rescoring it preserves the task verdict. +Do not replace an original scored binding with a public-view hash or describe a +redacted reproduction as byte-for-byte recovery of the private call. Public +frozen presentations declare `publication_evidence_views` in their presentation +binding. The shared gate requires complete current answer/native coverage, +checks the actual displayed aliases, rejects private artifact trees, and keeps +ordinary acquisition/aggregation checks strict. Redacted excerpts can verify +visible text, not original private coordinates. Report rebuilding uses saved +original scores; changed public text does not reproduce every original text +quality diagnostic. Disk-backed metadata projection and independent file clones +keep large histories practical without hard-linking writable outputs to inputs. + For native caller comparisons, `dns_mode` is explicitly `system` (default) or `public_doh`. All six local methods (Moli, Crawl4AI, OpenClaw, DeepSeek, CowAgent and @@ -774,22 +832,41 @@ The policy is declared before calls and applies to every task in that family; case IDs, hostname rules, answer-dependent delays, and extraction scripts are not accepted by this setting. Call metadata records the selected family policy. -### Fixed native webpage calls - -`uniform_native_parameters: true` selects the fixed-native body/structure policy for web and dynamic tasks. Choose documented native extraction options once per method (for example Tavily advanced, Cloudflare networkidle2, Crawl4AI iframe/overlay options, native freshness and capacity options). The caller deadline is 45 seconds; product connection, navigation and extraction timers retain their native defaults. HTTP-client deadlines and worker cancellation enforce collection time, not product-side budgets. CowAgent globals are never patched. - -Moli uses its native `done` completion for ordinary tasks and native -`domstable` completion for the complete dynamic-body family. The same policy -applies to the release and candidate builds. It is fixed before calls and may -not vary by case or change after a failure. - -Per-case retrieval hints and auxiliary document/media/challenge routes are rejected before invocation in this mode. Cross-provider fallback, injected extraction algorithms and answer synthesis are prohibited. Caller failure retries follow the shared [retry policy](#retries); historical cohorts retain their frozen retry budgets. Keep the native response envelope and its declared answer; the product may still impose its own limits or internal retries. Moli webpage, document-body and webpage-media-information tasks all use its native Markdown; only direct-media-file tasks use raw bytes. Prepare a new frozen invocation profile after a policy change; preserve previous measurements and do not infer revised scores without calls. -### 原生内容接口与评测边界 - -给定网址的正文任务验收返回的目标内容，不要求接口返回最终网址或 HTML 语言标签。 -最终网址、HTTP 状态和语言标签保留为辅助诊断；缺失这些字段本身不扣分。 -登录页、验证页、错误页、空正文、缺少题目要求的文字或字段仍然失败。 -验收规则对所有选手一致，禁止从标准答案补正文或推断缺失输出。 +### Fixed native calls + +`uniform_native_parameters: true` freezes class-level interfaces, answer fields, +and request parameters before calls. Each task declares `input_media_kind` +independently of its URL spelling or reference answer. The 45-second caller +budget leaves product timers at their native defaults. Per-case extraction hints, +cross-provider fallback, injected extraction algorithms, and answer synthesis are +prohibited. The exact dynamic marker `{"version": 3, "mode": "native"}` declares +no per-case assistance and is allowed with a stable acceptance contract. + +Moli's fixed-native request builder uses Markdown for webpage text, HTML for +resource discovery, and original bytes for declared direct PDF, DOCX, and file +inputs. Its ordinary native task-family interface supports native Markdown with +layout and child documents. The invocation profile declares which interface was +measured; a newer implementation never relabels historical calls. Output capture +limits do not cap underlying downloads. Caller retries follow the shared +[retry policy](#retries), while historical cohorts retain their frozen budgets. +Keep the complete native response, projection declaration, and request receipts. + +### Content acceptance and page identity + +Final URLs and document-language metadata are diagnostic by default. A task that +requires these observations declares `page_identity_required: true`; its existing +navigation/language fields define the accepted identity. Missing identity stays +unavailable unless another independently failed requirement establishes failure. +A proven mismatch fails that task. Native response provenance is always checked, +including the title and final URL supplied to the scorer, regardless of whether +identity itself is an acceptance requirement. + +When migrating a historical task that already required page identity, make that +requirement explicit through a bound offline revision. Preserve its original +outputs, timings, review observations, and prior task definition. Adding this +annotation does not authorize dropping any other task requirement or creating a +new provider call. Content requirements continue to reject missing target text, +records, fields, empty answers, or error-page shells independently of metadata. ### Main-document identity on declared navigation tasks @@ -904,6 +981,98 @@ physical-call output, uses the final declared response, verifies hashes and exposes the directory only after the entire matrix passes. Unbound review receipts cannot authorize a new assembly. Existing published reports retain their frozen evidence and are restored from their pinned release. + +### Current local 430-task report + +`research/native-430-local-report.json` binds the reviewed ten-method snapshot +by dataset, result and scorer hashes. Its `evidence_path` is relative to +`BROWSER_EVAL_DATA_HOME` (default `~/.browser-eval`), so an installed snapshot +can be located without inferring it from earlier work logs. The 4,300 measured responses, execution +segments, review payloads, archived interpretation article and its translations +remain in the external evidence directory; they are not distributed in Git. +The shared report owner reads those saved inputs and does not call providers +during rendering. A frozen view can be checked without claiming that a newer +scorer produced its historical scores: + +```sh +uv run --frozen python scripts/check_report.py /absolute/path/to/local-report \ + --presentation-binding research/native-430-local-report.json +``` + +For a fresh build, use `scripts/build_report.py --local-audit` with the matching +external evidence, an explicit output directory, and `--language zh --include-english`. +When rebuilding the bound historical report with a newer scorer, also pass +`--presentation-binding research/native-430-local-report.json`; this verifies +the existing report snapshot and preserves its saved scores and evidence bytes. +The matching binding is required for `check_report.py` as shown above. +Any scoring change must first follow the declared correction protocol; a new +render alone does not update scores. The public release profile and its +publication target are separate and are not changed by this local snapshot. + +Historical audit tests read their frozen scorer bytes from +`/scoring-snapshots//`; they verify the complete scorer, +registry and scenario-test hashes rather than substituting current code. +Ordinary tests skip absent historical snapshots; strict reference-input tests +require them. Changed installed bytes fail either mode. + +### Current acceptance and retained corrections + +The shared scorer uses one current acceptance predicate. Verification/error hints +are diagnostics; declared `forbidden_any` requirements remain contract gates. +Treating previously forbidden words as hints requires an explicit case revision +to `diagnostic_any` and a complete rescore of the affected method/task scope. +Content, records, page identity and completion deadlines still determine acceptance; +reference precision and observed challenge components remain separate diagnostics. + +Historical results retain their hash-bound frozen scorer rather than selecting +old behavior in the current runtime. The current predicate rejects legacy metrics +that need that frozen scorer or an explicit offline rescore. Current regression +controls verify current behavior; superseded controls retain their IDs, definition +hashes and retirement reasons, with original definitions preserved in the frozen +history. Do not mix old metrics with a new predicate. + +Stable text contracts declare `content_scope` (`target_content` or +`full_page_stable_coverage`) and may declare `text_exclusions`. A target-content +contract rejects positive anchors observed in excluded navigation/footer regions. +A whole-page contract can require their stable text, alongside body coverage or +complete record requirements. `source_location` records the independent observation; +it is not an implicit DOM selector. HTML exclusions apply before text matching; +plain-text outputs need explicit textual exclusions where roles otherwise disappear. + +`local_revision.revise_local_snapshot` only rescores a separate saved snapshot. +Changed contracts require `rule_revision: true`. Its receipt binds the predecessor, +selected complete method/task scope, regression IDs, source hashes, explicit contract +changes, actual scoring time and verdict changes. Physical output bytes, call identities, +timestamps and latencies remain unchanged. Proven environment defects exclude a +case symmetrically across every method. The result ledger derives all transitions +from per-attempt states; overridden review opinions require task/answer hashes, +original answer excerpts, concrete rules and a reason. This is supplemental +adjudication, not a new blind review of the entire matrix. + +Local uncertified reports retain a disclosure on every page and omit ordinal +rankings. The CLI cannot hide the disclosure. Historical article HTML is limited +to inert allowlisted markup and safe links/raster images; its translations apply +only inside the article. A diagnostic snapshot cannot certify a formal release. + + +### Native scored result context + +Native projection schema 2 includes `result_fields.title` and `result_fields.final_url` +alongside text, resource metadata and delivered files. Validate these fields before +persisting a call, before native-response scoring, and when aggregating saved calls. +JSON fields use declared structural pointers. A final address may also be replayed +from a retained same-call main-document log or a source HTTP response receipt; +request-URL fallback applies only when the selected native address is absent. +Resolve relative resource attributes against that verified address. A title absent +from the native response cannot supply required facts. + +Schema-1 declarations must not contain `result_fields`, even an empty object. +Current persistence rejects mixed declarations, including discarded timeout answers. +Immutable schema-1 acquisitions retain their original text/resource/file scope; +they do not retroactively prove title/address provenance. Bind a separate offline +context audit to historical results, replay the complete affected route when a +context difference is found, and retain the original records and exact score effects. +These checks add no provider calls. 按任务类型预选官方支持的正文、资源或原始文件接口。JSON 包装中的原生 Markdown 可以解包；不能把 JSON DOM、HTML 或下载的 PDF/DOCX 另行转换成选手的正文成绩。 Chromium 没有原生 Markdown，网页 HTML 使用固定版本的 node-html-markdown；此例外 diff --git a/harness/web_fetch/adjudication_evidence.py b/harness/web_fetch/adjudication_evidence.py new file mode 100644 index 000000000..48ae7abfd --- /dev/null +++ b/harness/web_fetch/adjudication_evidence.py @@ -0,0 +1,318 @@ +"""Require inspectable task and answer evidence when a verdict overrides review.""" +import hashlib +import json + +from .publication_privacy import read_evidence_binding, evidence_excerpt_matches + + +TEMPLATE = '核对已有原始返回、冻结要求和相应修正证据；当前结论见评分明细。' + + +def case_hash(case): + return hashlib.sha256(json.dumps(case, sort_keys=True, ensure_ascii=False, + separators=(',', ':')).encode()).hexdigest() + + +def requires_adjudication(row): + final = row.get('final_verdict') + return final is not None and any( + row.get(key) not in {None, final} for key in ('agent_verdict', 'effective_agent_verdict')) + + +def current_rule_outcomes(case, record): + """Expose only declared, computed rules as structured adjudication facts.""" + from lexbench_webfetch.metrics import task_passed + m = record['metrics'] + outcomes = {'task_result': {'passed': task_passed(record['ok'], m)}, + 'request_status': {'passed': bool(m.get('request_ok', record['ok']))}} + if case.get('required_facts'): + outcomes['required_facts'] = { + 'passed': m['fact_coverage'] >= case.get('fact_coverage_threshold', 1.0), + 'matched': m['fact_hits'], 'total': m['fact_total']} + if case.get('expected_any'): + outcomes['expected_any'] = {'passed': bool(m.get('matched_markers'))} + if case.get('min_content_chars') is not None: + outcomes['min_content_chars'] = {'passed': m['content_chars'] >= case['min_content_chars'], + 'actual': m['content_chars'], 'minimum': case['min_content_chars']} + if case.get('body_requirements'): + outcomes['body_requirements'] = {'passed': m['body_coverage_pass'], + 'coverage': [{'id': r['id'], 'coverage': r['coverage'], + 'target_pass': r['target_pass'], 'floor_pass': r['floor_pass']} + for r in m['body_region_results']]} + if case.get('required_patterns'): + outcomes['required_patterns'] = {'passed': all(r['pass'] for r in m['pattern_results']), + 'results': m['pattern_results']} + if case.get('acceptance_contract'): + outcomes['acceptance_contract'] = { + 'passed': m['stable_task_pass'], + 'anchor_hits': m['stable_anchor_hits'], 'anchor_total': m['stable_anchor_total'], + 'missing_anchors': [r['id'] for r in m['stable_anchor_results'] if not r['pass']], + 'records': m['stable_unique_record_count'], 'minimum_records': m['stable_minimum_records']} + if case.get('composite_acceptance'): + outcomes['composite_acceptance'] = {'passed': m['composite_acceptance_pass']} + if case.get('expected_resources'): + outcomes['expected_resources'] = {'passed': m['resource_pass']} + if case.get('forbidden_any'): + outcomes['forbidden_any'] = {'passed': not m['forbidden_requirement_hits'], + 'hits': m['forbidden_requirement_hits']} + if case.get('diagnostic_any'): + hits = m['diagnostic_hits'] + if any(hit not in case['diagnostic_any'] for hit in hits): + raise ValueError('Diagnostic hits do not belong to the declared case') + outcomes['diagnostic_any'] = {'hits': list(hits), 'affects_verdict': False} + return outcomes + + +def adjudication_reason(assertions): + """Render claims from checked facts, never unchecked free-form hit text.""" + labels = {'task_result': '任务判定', 'request_status': '请求状态', + 'required_facts': '必需文字', 'expected_any': '目标标记', + 'min_content_chars': '最低长度', 'body_requirements': '正文覆盖', + 'required_patterns': '格式与代码要求', 'acceptance_contract': '内容与条目要求', + 'composite_acceptance': '组合要求', 'expected_resources': '目标资源', + 'forbidden_any': '明确禁止要求'} + parts = [] + for key, value in assertions.items(): + if key == 'diagnostic_any': + parts.append('诊断词命中：' + ('、'.join(value['hits']) or '无') + '（不单独决定通过或失败）') + else: + detail = '' + if key == 'required_facts': + detail = f"（{value['matched']}/{value['total']}）" + elif key == 'min_content_chars': + detail = f"（{value['actual']}，要求至少 {value['minimum']}）" + elif key == 'acceptance_contract': + detail = (f"（稳定文字 {value['anchor_hits']}/{value['anchor_total']}；" + f"独立条目 {value['records']}，要求至少 {value['minimum_records']}" + + ('；缺少 ' + '、'.join(value['missing_anchors']) + if value['missing_anchors'] else '') + '）') + elif key == 'required_patterns': + missing = [r['id'] for r in value['results'] if not r['pass']] + detail = ('（未满足 ' + '、'.join(missing) + '）') if missing else '' + parts.append(labels[key] + '：' + ('通过' if value['passed'] else '未通过') + detail) + return '；'.join(parts) + '。' + + +def require_rule_assertions(case, record, row, proof): + available = current_rule_outcomes(case, record) + identifiers = proof.get('rule_ids', []) + if (not identifiers or len(identifiers) != len(set(identifiers)) + or any(key not in available for key in identifiers) + or 'task_result' not in identifiers): + raise ValueError('Adjudication cites an unknown or missing task rule') + expected = {key: available[key] for key in identifiers} + if proof.get('rule_assertions') != expected: + raise ValueError('Adjudication rule claims differ from current metrics') + final = 'pass' if available['task_result']['passed'] else 'fail' + if row.get('final_verdict') != final: + raise ValueError('Adjudication verdict contradicts computed task rules') + reason = adjudication_reason(expected) + if proof.get('reason') != reason or row.get('adjudication') != reason: + raise ValueError('Adjudication reason differs from verified rule claims') + + +def require_adjudication_evidence(source, dataset, records, review): + require_retained_reviews(source, dataset, records, review) + _require_supplemental_reviews(source, dataset, records, review) + if review.get('rule_binding_revision'): + raise ValueError('Historical review migrations are not current-contract review') + cases = {case['id']: case for case in dataset.get('cases', [])} + indexed = {record['attempt_id']: record for record in records} + checked = 0 + for row in review.get('attempts', []): + if not requires_adjudication(row): + continue + record = indexed[row['attempt_id']] + proof = row.get('adjudication_evidence', {}) + reason = proof.get('reason', '').strip() + if (not reason or reason == TEMPLATE or row.get('adjudication') == TEMPLATE + or proof.get('case_sha256') != case_hash(cases[record['case_id']]) + or proof.get('content_sha256') != record['metrics']['content_sha256'] + or not proof.get('rule_ids')): + raise ValueError('Overridden review lacks specific rule/evidence: ' + row['attempt_id']) + require_rule_assertions(cases[record['case_id']], record, row, proof) + answer_binding = {'path': record['artifact_path'], 'sha256': proof['content_sha256']} + raw = read_evidence_binding(source, answer_binding).read_bytes() + text = raw.decode('utf-8', errors='replace') + samples = proof.get('samples', []) + if not text: + if proof.get('empty_output') is not True: + raise ValueError('Empty adjudication answer must be explicit') + elif not samples: + raise ValueError('Adjudication lacks an answer excerpt') + for sample in samples: + if not evidence_excerpt_matches(source, answer_binding, sample, text): + raise ValueError('Adjudication excerpt differs from original answer') + checked += 1 + return checked + + +def require_current_review_state(records, review): + from lexbench_webfetch.metrics import task_measured, task_passed + indexed={record['attempt_id']:record for record in records} + rows=review.get('attempts',[]) + unreviewed=review.get('unreviewed_attempt_ids',[]) + reviewed_ids={row['attempt_id'] for row in rows} + if (len(rows)!=len(reviewed_ids) or len(unreviewed)!=len(set(unreviewed)) + or reviewed_ids & set(unreviewed) or reviewed_ids | set(unreviewed)!=set(indexed)): + raise ValueError('Review state inventory differs from current results') + if unreviewed and (review.get('reviewed_attempts')!=len(rows) + or review.get('accepted_unreviewed_passes')!=sum( + task_passed(indexed[k]['ok'],indexed[k]['metrics']) for k in unreviewed)): + raise ValueError('Unreviewed result coverage differs from current scores') + for row in rows: + record=indexed[row['attempt_id']] + expected='unavailable' if not task_measured(record['metrics']) else 'pass' if task_passed(record['ok'],record['metrics']) else 'fail' + if row.get('current_verdict')!=expected or row.get('final_verdict') not in {expected,None}: + raise ValueError('Review state differs from current scored result') + if requires_adjudication(row) and row.get('status')!='adjudicated': + raise ValueError('Review override has no adjudicated state') + unresolved=sum(row.get('final_verdict') is None for row in rows) + if review.get('unresolved')!=unresolved: + raise ValueError('Review unresolved total differs from result states') + effective=sum(row.get('effective_agent_verdict',row.get('agent_verdict'))!=row.get('final_verdict') for row in rows) + if review.get('effective_disagreements',effective)!=effective: + raise ValueError('Review disagreement total differs from result states') + + +def _require_supplemental_reviews(source, dataset, records, review): + """Bind newly reviewed answers without rewriting inherited opinions.""" + from .local_revision import read_bound + fresh = {r['attempt_id'] for r in review.get('attempts', []) + if r.get('review_origin') == 'fresh'} + if not review.get('supplemental_reviews'): + if fresh: + raise ValueError('Fresh reviews require their supplemental evidence') + return set() + indexed = {row['attempt_id']: row for row in records} + cases = {case['id']: case for case in dataset['cases']} + current = {row['attempt_id']: row for row in review.get('attempts', [])} + seen = set() + for binding in review.get('supplemental_reviews', []): + supplement = json.loads(read_bound(source, binding).read_text()) + rows = supplement.get('attempts', []) + identifiers = [row['attempt_id'] for row in rows] + if (supplement.get('kind') != 'supplemental_saved_output_review' + or supplement.get('results_sha256') != review.get('results_sha256') + or supplement.get('provider_calls_added') != 0 + or not identifiers or len(set(identifiers)) != len(identifiers) + or seen.intersection(identifiers) + or supplement.get('reviewed_attempts') != len(rows) + or not supplement.get('reviewer') + or 'blind_initial_verdict' not in supplement.get('review_protocol', {})): + raise ValueError('Supplemental review scope or provenance differs') + initial = {} + if supplement.get('review_protocol', {}).get('blind_initial_verdict') is True: + for output_binding in supplement.get('blind_initial_outputs', []): + output = json.loads(read_bound(source, output_binding).read_text()) + if output.get('machine_judgments_withheld') is not True: + raise ValueError('Fresh initial review was not blind') + for opinion in output.get('attempts', []): + identifier = opinion['attempt_id'] + if identifier in initial: + raise ValueError('Fresh initial review repeats an attempt') + initial[identifier] = opinion + if not set(identifiers).issubset(initial): + raise ValueError('Fresh review lacks its immutable initial opinions') + for row in rows: + if initial: + opinion = initial[row['attempt_id']] + if (row.get('agent_verdict') != opinion.get('verdict') + or row.get('agent_evidence') != opinion.get('reason') + or row.get('reviewer_anomalies') != opinion.get('anomalies')): + raise ValueError('Fresh initial review opinion changed') + record = indexed.get(row['attempt_id']) + if record is None or current.get(row['attempt_id']) != row: + raise ValueError('Supplemental review opinions changed') + proof = row.get('review_evidence', {}) + if (proof.get('case_sha256') != case_hash(cases[record['case_id']]) + or row.get('content_sha256') != record['metrics']['content_sha256'] + or not row.get('agent_evidence', '').strip() + or row.get('agent_verdict') not in {'pass', 'fail', 'insufficient'}): + raise ValueError('Supplemental review lacks bound task and answer evidence') + answer_binding = {'path': record['artifact_path'], 'sha256': row['content_sha256']} + raw = read_evidence_binding(source, answer_binding).read_bytes() + text = raw.decode('utf-8', errors='replace') + if text and not proof.get('samples'): + raise ValueError('Supplemental review lacks an answer excerpt') + for sample in proof.get('samples', []): + if not evidence_excerpt_matches(source, answer_binding, sample, text): + raise ValueError('Supplemental review excerpt differs') + seen.update(identifiers) + if fresh and seen != fresh: + raise ValueError('Fresh review coverage differs from supplemental evidence') + return seen + + + + +def require_retained_reviews(source, dataset, records, review): + """Retain actual old opinions only when their decision inputs still suffice. + + A removed, unmatched ban does not invalidate an otherwise identical review. + A hit, a dependent old explanation, changed positive rule, changed answer or + changed verdict requires new review. Retention never becomes a fresh review. + """ + binding = review.get('retained_reviews') + retained = {r['attempt_id'] for r in review.get('attempts', []) + if r.get('review_origin') == 'retained'} + if not binding: + if retained: + raise ValueError('Retained reviews require their original evidence') + return set() + from .local_revision import read_bound + from .result_review import review_input_identity + from harness.record_io import load_jsonl + receipt = json.loads(read_bound(source, binding).read_text()) + prior = json.loads(read_bound(source, receipt['prior_review']).read_text()) + before = json.loads(read_bound(source, receipt['prior_dataset']).read_text()) + previous = load_jsonl(read_bound(source, receipt['prior_results'])) + ids = receipt.get('attempt_ids', []) + if (receipt.get('kind') != 'unchanged_review_dependency_retention' + or receipt.get('provider_calls_added') != 0 + or len(ids) != len(set(ids)) or set(ids) != retained + or review.get('retained_reviewed_attempts') != len(ids) + or prior.get('rule_binding_revision') + or prior.get('results_sha256') != receipt['prior_results']['sha256']): + raise ValueError('Retained review scope or original provenance differs') + old_cases = {c['id']: c for c in before['cases']} + cases = {c['id']: c for c in dataset['cases']} + old_records = {r['attempt_id']: r for r in previous} + indexed = {r['attempt_id']: r for r in records} + opinions = {r['attempt_id']: r for r in prior['attempts']} + current = {r['attempt_id']: r for r in review['attempts']} + immutable = ('agent_verdict', 'agent_evidence', 'effective_agent_verdict', + 'effective_agent_evidence', 'initial_review_sha256', 'content_sha256') + for identifier in ids: + if identifier not in indexed or identifier not in old_records or identifier not in opinions: + raise ValueError('Retained review has no original attempt') + record, old_record = indexed[identifier], old_records[identifier] + case, old_case = cases[record['case_id']], old_cases[old_record['case_id']] + # Empty removed fields are immaterial; every positive requirement and + # every other task field must remain byte-equivalent as JSON values. + old_bans, bans = old_case.get('forbidden_any', []), case.get('forbidden_any', []) + old_hints, hints = old_case.get('diagnostic_any', []), case.get('diagnostic_any', []) + ignored = {'forbidden_any', 'diagnostic_any'} + if ({k: v for k, v in old_case.items() if k not in ignored} + != {k: v for k, v in case.items() if k not in ignored} + or not ((old_bans == bans and old_hints == hints) + or (not bans and hints == old_bans and not old_hints))): + raise ValueError('Retained review has changed positive requirements') + opinion, row = opinions[identifier], current[identifier] + cited = set(opinion.get('adjudication_evidence', {}).get('rule_ids', [])) + if ((old_bans != bans or old_hints != hints) + and (record['metrics'].get('diagnostic_hits') + or cited & ignored or '命中词：。' in opinion.get('adjudication', ''))): + raise ValueError('Retained review depends on the changed text policy') + if (review_input_identity(record) != review_input_identity(old_record) + or any(row.get(k) != opinion.get(k) for k in immutable) + or opinion.get('final_verdict') not in {'pass', 'fail'} + or row.get('final_verdict') != opinion['final_verdict']): + raise ValueError('Retained review answer, opinion or verdict changed') + read_evidence_binding(source, {'path': record['artifact_path'], + 'sha256': row.get('content_sha256')}) + fresh = sum(r.get('review_origin') == 'fresh' for r in review['attempts']) + if (review.get('fresh_reviewed_attempts') != fresh + or fresh + len(ids) != len(review['attempts'])): + raise ValueError('Fresh and retained review counts differ') + return retained diff --git a/harness/web_fetch/admission.py b/harness/web_fetch/admission.py new file mode 100644 index 000000000..f3ca803d8 --- /dev/null +++ b/harness/web_fetch/admission.py @@ -0,0 +1,122 @@ +"""Cross-process scheduling outside provider timing and deadline budgets.""" +from __future__ import annotations + +import fcntl +import json +import os +import threading +import time +from contextlib import contextmanager +from pathlib import Path +from urllib.parse import urlsplit + + +class CaseAdmission: + """Bound local/cloud calls and serialize a shared destination site. + + Only scheduling is controlled. No request, provider result, or score changes. + The state directory belongs to a single declared execution, outside Git. + """ + def __init__(self, root: Path, *, group: str, max_parallel: int = 2, + site_cooldown: float = 10.1): + if group not in {'local', 'cloud'} or max_parallel not in {1, 2}: + raise ValueError('Unsupported admission policy') + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + self.group = group + self.max_parallel = max_parallel + self.site_cooldown = site_cooldown + + def describe(self): + return {'max_parallel': self.max_parallel, 'max_local': 1, 'max_cloud': 1, + 'same_site_parallel': False, 'cross_method_site_cooldown_seconds': self.site_cooldown, + 'same_method_interval': 'unchanged_provider_configuration', + 'queue_wait_in_answer_latency': False, + 'parallel_timeout_or_429': 'reduce_to_serial; do_not_retry_recorded_result'} + + @contextmanager + def _state(self): + with (self.root/'state.lock').open('a') as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + path = self.root/'state.json' + state = json.loads(path.read_text()) if path.exists() else { + 'active': {}, 'finished_sites': {}, 'parallel_enabled': True} + now = time.time() + for key, item in list(state['active'].items()): + try: + os.kill(item['pid'], 0) + except ProcessLookupError: + state['finished_sites'][item['site']] = {'at': now, 'provider': item['provider']} + del state['active'][key] + try: + yield state + finally: + temporary = self.root/'state.json.tmp' + temporary.write_text(json.dumps(state, ensure_ascii=False, indent=2)+'\n') + temporary.replace(path) + fcntl.flock(lock, fcntl.LOCK_UN) + + def _event(self, value): + with (self.root/'events.jsonl').open('a') as handle: + handle.write(json.dumps({'at': time.time(), **value})+'\n') + + def acquire(self, provider, case): + host = (urlsplit(case.url).hostname or '').lower().rstrip('.') + # Deliberately conservative: e.g. *.co.uk shares one scheduling group. + site = '.'.join(host.split('.')[-2:]) + key = str(os.getpid())+':'+provider.name + queued = time.monotonic() + while True: + with self._state() as state: + active = list(state['active'].values()) + limit = self.max_parallel if state['parallel_enabled'] else 1 + previous = state['finished_sites'].get(site, {}) + cross_method_cooldown = (previous.get('provider') not in {None, provider.name} + and time.time()-previous.get('at', 0) < self.site_cooldown) + blocked = (len(active) >= limit + or any(item['group'] == self.group or item['site'] == site + for item in active) + or cross_method_cooldown) + if not blocked: + if key in state['active']: + raise ValueError('A method has an unreleased call') + token = {'key': key, 'pid': os.getpid(), 'provider': provider.name, + 'case_id': case.id, 'site': site, 'group': self.group} + state['active'][key] = token + self._event({'event': 'admitted', **token, + 'queue_wait_ms': round((time.monotonic()-queued)*1000, 2), + 'active_calls': len(active)+1}) + return token + time.sleep(0.1) + + def _release_when_finished(self, token, pending): + pending.join() + self.release(token) + + def release(self, token, *, pending=None, record=None): + with self._state() as state: + if token['key'] not in state['active']: + raise ValueError('Admission token was lost') + record = record or {} + message = str(record.get('error_message') or '').lower() + limit_error = (record.get('status_code') == 429 + or any(value in message for value in + ('too many requests', 'rate limit', 'rate_limit', 'http 429'))) + if len(state['active']) > 1 and (record.get('error_type') == 'timeout' or limit_error): + state['parallel_enabled'] = False + self._event({'event': 'reduce_to_serial', 'attempt_id': record.get('attempt_id'), + 'reason': 'timeout_or_429_observed_during_overlap; causality_unproven'}) + if pending is not None and pending.is_alive(): + state['active'][token['key']]['quarantined'] = True + self._event({'event': 'quarantined_until_call_exit', **token}) + threading.Thread(target=self._release_when_finished, args=(token, pending), + daemon=True, name='webfetch-admission-cleanup').start() + return + del state['active'][token['key']] + state['finished_sites'][token['site']] = {'at': time.time(), 'provider': token['provider']} + self._event({'event': 'released', **token}) + + def set_serial(self, reason): + with self._state() as state: + state['parallel_enabled'] = False + self._event({'event': 'reduce_to_serial', 'reason': reason}) diff --git a/harness/web_fetch/anomaly_review.py b/harness/web_fetch/anomaly_review.py index 2e2ebb4b0..da96f4bfe 100644 --- a/harness/web_fetch/anomaly_review.py +++ b/harness/web_fetch/anomaly_review.py @@ -105,7 +105,7 @@ def _normalized(text): def _only_forbidden_failed(metrics): stable_errors = metrics.get('stable_task_errors', []) if (metrics.get('stable_structure_pass') is True - and metrics.get('forbidden_hits') + and metrics.get('forbidden_requirement_hits') and stable_errors and all('forbidden' in str(error).lower() for error in stable_errors)): return True diff --git a/harness/web_fetch/artifacts.py b/harness/web_fetch/artifacts.py index 80f9eb792..908075641 100644 --- a/harness/web_fetch/artifacts.py +++ b/harness/web_fetch/artifacts.py @@ -60,7 +60,7 @@ def record_public_call_log_bindings(record: dict[str, Any]) -> list[dict[str, An for path, (digest, size) in seen.items()] -def verify_public_call_log(root: Path, binding: dict[str, Any]) -> Path: +def verify_public_call_log(root: Path, binding: dict[str, Any], *, allow_public_views=False) -> Path: """Verify one public call log's containment, bytes, digest and closed schema.""" normalized = public_call_log_binding(binding) if normalized is None: # pragma: no cover - caller supplied an object @@ -131,21 +131,29 @@ def verify_public_call_log(root: Path, binding: dict[str, Any]) -> Path: output_path = Path(str(payload["output"]["path"])) if not output_path.parts or output_path.parts[0] != "artifacts": raise ValueError("Public call log output must stay under artifacts") - output = contained_path(root, output_path) - if not output.is_file() or output.is_symlink(): - raise ValueError("Public call log output is missing or unsafe") - output_bytes = output.read_bytes() claimed = payload["output"] - if (claimed["bytes"] != len(output_bytes) - or claimed["sha256"] != hashlib.sha256(output_bytes).hexdigest() - or claimed["empty"] is not (len(output_bytes) == 0)): - raise ValueError("Public call log output binding changed") + if allow_public_views: + from .publication_privacy import read_evidence_binding, original_evidence_size + answer = {'path': output_path.as_posix(), 'sha256': claimed['sha256']} + read_evidence_binding(root, answer) + size = original_evidence_size(root, answer) + if claimed['bytes'] != size or claimed['empty'] is not (size == 0): + raise ValueError("Public call log original output binding changed") + else: + output = contained_path(root, output_path) + if not output.is_file() or output.is_symlink(): + raise ValueError("Public call log output is missing or unsafe") + output_bytes = output.read_bytes() + if (claimed["bytes"] != len(output_bytes) + or claimed["sha256"] != hashlib.sha256(output_bytes).hexdigest() + or claimed["empty"] is not (len(output_bytes) == 0)): + raise ValueError("Public call log output binding changed") return path -def verify_record_public_call_logs(root: Path, record: dict[str, Any]) -> None: +def verify_record_public_call_logs(root: Path, record: dict[str, Any], *, allow_public_views=False) -> None: for binding in record_public_call_log_bindings(record): - verify_public_call_log(root, binding) + verify_public_call_log(root, binding, allow_public_views=allow_public_views) def raw_response_binding(metadata: dict[str, Any]) -> dict[str, Any] | None: @@ -331,6 +339,8 @@ def verify_record_artifacts(root: Path, record: dict[str, Any]) -> None: if not source.is_file() or hashlib.sha256(source.read_bytes()).hexdigest() != digest: raise ValueError('Artifact differs from its result binding: ' + relative) verify_record_public_call_logs(root, record) + from harness.web_fetch.native_routes import verify_saved_native_answer + verify_saved_native_answer(root, record) def materialize_bound_artifacts(output: Path, source_root: Path, records: list[dict]) -> None: @@ -358,7 +368,9 @@ def materialize_bound_artifacts(output: Path, source_root: Path, records: list[d def materialize_reference_checks(output: Path, source: Path, batches: list[dict]) -> None: - """Retain only hash-bound observations named by the execution certificates.""" + """Retain hash-bound source observations and native execution contracts.""" + from .native_routes import materialize_native_route_contracts + materialize_native_route_contracts(output, source, batches) for batch in batches: post = batch.get('reference_checks', {}).get('postcheck', {}) if not post: diff --git a/harness/web_fetch/call_boundary.py b/harness/web_fetch/call_boundary.py index 01a41bc8d..c6cf48348 100644 --- a/harness/web_fetch/call_boundary.py +++ b/harness/web_fetch/call_boundary.py @@ -70,6 +70,10 @@ def work(): late=None if isinstance(late,ProviderResult): metadata.update(late.metadata) + for key in ('links', 'images', 'videos', 'media', 'extras', 'file_evaluation'): + metadata.pop(key, None) + if case.task_kind == 'file': + metadata['file_evaluation'] = evaluate_files(vars(case), []) metadata['_answer_elapsed_ms']=answer_elapsed_ms return ProviderResult( False, diff --git a/harness/web_fetch/cohort_exclusions.py b/harness/web_fetch/cohort_exclusions.py new file mode 100644 index 000000000..5d118b7a5 --- /dev/null +++ b/harness/web_fetch/cohort_exclusions.py @@ -0,0 +1,38 @@ +"""Exclude a proven evaluation defect symmetrically without discarding calls.""" +import hashlib + +from lexbench_webfetch.paths import contained_path + + +def require_cohort_exclusions(source, dataset, records): + declared = dataset.get('comparison', {}).get('excluded_cases', []) + cases = {case['id'] for case in dataset['cases']} + indexed = {} + for item in declared: + case_id = item.get('case_id') + if case_id not in cases or case_id in indexed or not item.get('reason', '').strip(): + raise ValueError('Invalid or duplicate cohort exclusion') + proof = contained_path(source, item['evidence_path'], reject_symlinks=True) + if hashlib.sha256(proof.read_bytes()).hexdigest() != item.get('evidence_sha256'): + raise ValueError('Cohort exclusion evidence changed') + indexed[case_id] = item + for record in records: + metrics = record['metrics'] + declaration = indexed.get(record['case_id']) + if bool(metrics.get('evaluation_excluded')) != bool(declaration): + raise ValueError('Cohort exclusion must apply to every method for the same case') + if declaration and metrics.get('evaluation_exclusion_reason') != declaration['reason']: + raise ValueError('Cohort exclusion reason differs from its evidence binding') + return indexed + + +def apply_cohort_exclusions(source, dataset, records): + """Annotate derived metrics only; acceptance and physical call facts remain saved.""" + declared = {item['case_id']: item for item in dataset.get('comparison', {}).get('excluded_cases', [])} + for record in records: + metrics = record['metrics'] + metrics.pop('evaluation_excluded', None) + metrics.pop('evaluation_exclusion_reason', None) + if item := declared.get(record['case_id']): + metrics.update(evaluation_excluded=True, evaluation_exclusion_reason=item['reason']) + require_cohort_exclusions(source, dataset, records) diff --git a/harness/web_fetch/failure_diagnosis.py b/harness/web_fetch/failure_diagnosis.py index a4006ec4e..b58431f69 100644 --- a/harness/web_fetch/failure_diagnosis.py +++ b/harness/web_fetch/failure_diagnosis.py @@ -134,15 +134,6 @@ def set_result(label,where,evidence):return label,where,evidence category,stage,proof=set_result( '正文范围不完整','内容核验', '供应商明确声明返回内容已截断：'+', '.join(m['declared_truncation_hits'])) - elif m.get('forbidden_hits'): - category,stage,proof=set_result('返回非目标内容','内容核验','命中错误页 / 排除标记：'+', '.join(m['forbidden_hits'])) - elif (m.get('blocked_page') or m.get('stable_blocked_page') - or 'blocked_page' in m.get('stable_task_errors', [])): - hits = m.get('blocked_page_hits') or [] - detail = ('：' + ', '.join(hits)) if hits else '' - category,stage,proof=set_result('返回非目标内容','内容核验','内容检查命中 blocked_page'+detail+'；记录、链接或资源数量足够也不代表取得目标内容。') - elif 'forbidden_content' in m.get('stable_task_errors', []): - category,stage,proof=set_result('返回非目标内容','内容核验','稳定内容检查命中 forbidden_content：返回正文含登记的排除标记。') elif 'stable_task_pass' in m: if not m.get('stable_task_pass'): category,stage,proof=_stable_failure(m) diff --git a/harness/web_fetch/local_audit.py b/harness/web_fetch/local_audit.py index e25c98c67..550f6f027 100644 --- a/harness/web_fetch/local_audit.py +++ b/harness/web_fetch/local_audit.py @@ -418,16 +418,199 @@ def _require_local_scoring_revision(source, contract, records): provenance.get("regression_ids", []), receipt.get("task_verdict_changes", [])) if provenance != expected_provenance: raise ValueError("Invalid local scoring correction provenance") + rule_revision = receipt.get('rule_revision', False) + if type(rule_revision) is not bool: + raise ValueError('Local scoring rule revision must be explicit') for record in records: scoring = record.get("scoring", {}) if (record.get("metrics", {}).get("scoring_source_sha256") != SCORING_SOURCE_SHA256 or scoring.get("mode") != "offline_rescore" or scoring.get("input_results_sha256") != receipt.get("original_results_sha256") - or scoring.get("rule_revision") is not False + or scoring.get("rule_revision") is not rule_revision or scoring.get("reference_revision") is not False): raise ValueError("Local scoring correction changed rules or lost its predecessor") +def _require_local_record_correction(source, original, corrected, receipt=None): + """Permit derived corrections while binding every physical call fact.""" + if receipt is None: + path = contained_path(source, "local-evidence/scoring-correction.json") + if not path.is_file(): + raise ValueError("Changed local record lacks a correction receipt") + receipt = json.loads(path.read_text()) + identifier = original["attempt_id"] + if (receipt.get("original_call_identities", {}).get(identifier) + != local_measurement_call_sha256(original) + or receipt.get("call_identities", {}).get(identifier) + != local_measurement_call_sha256(corrected)): + raise ValueError("Local correction lost its physical call binding") + derived = {"metrics", "scoring", "diagnosis", "failure_category"} + changed = {key for key in original.keys() | corrected.keys() + if original.get(key) != corrected.get(key)} - derived + if changed - {"final_url"}: + raise ValueError("Local correction changed measured call facts") + if "final_url" in changed: + proof = receipt.get("field_projections", {}).get(identifier, {}).get("final_url", {}) + path = contained_path(source, proof.get("path", ""), reject_symlinks=True) + if not path.is_file() or _sha256(path) != proof.get("sha256"): + raise ValueError("Local URL correction lost its native response") + value = json.loads(path.read_text()) + for key in proof.get("json_pointer", []): + value = value.get(key) if isinstance(value, dict) else None + if (not proof.get("json_pointer") or value != corrected.get("final_url") + or proof.get("original_value") != original.get("final_url")): + raise ValueError("Local URL correction differs from native response") + + +def score_reviewed_page_identity(source, case, record, annotation, captures, *, historical_scoring_source=None): + """Apply a bound retrospective content correspondence, never a native URL.""" + from lexbench_webfetch.metrics import normalize_for_match, task_measured + from lexbench_webfetch.scoring import score_attempt, _same_page_url, SCORING_SOURCE_SHA256 + + if (annotation.get("attempt_id") != record["attempt_id"] + or annotation.get("case_sha256") != _canonical_sha256(case) + or annotation.get("call_sha256") != local_measurement_call_sha256(record) + or annotation.get("content_sha256") != record["metrics"]["content_sha256"] + or record.get("final_url") is not None + or annotation.get("basis") != "retrospective_content_correspondence" + or not annotation.get("reviewed_at")): + raise ValueError("Page review lost its case, answer or physical call binding") + historical = historical_scoring_source is not None and historical_scoring_source != SCORING_SOURCE_SHA256 + original_metrics = None if historical else score_attempt( + case, contained_path(source, record["artifact_path"]).read_bytes(), + ok=record["ok"], metadata=record.get("provider_metadata"), + previous_metrics=record["metrics"], status_code=record.get("status_code"), + title=record.get("title"), final_url=record.get("final_url"), + ) + if not historical and (task_measured(original_metrics) + or original_metrics.get("page_identity_status") != "unavailable"): + raise ValueError("Page review may only fill an unavailable page identity") + capture = captures[annotation["capture_index"]] + corresponding_url = annotation["corresponding_url"] + if (capture.get("url") != corresponding_url + or _canonical_sha256(capture) != annotation.get("capture_sha256")): + raise ValueError("Page review capture binding differs") + answer = contained_path(source, record["artifact_path"]).read_bytes() + if hashlib.sha256(answer).hexdigest() != annotation["content_sha256"]: + raise ValueError("Page review answer changed") + samples = annotation.get("matching_samples", []) + saved = normalize_for_match(answer.decode("utf-8")) + observed = normalize_for_match(capture["text"]) + normalized = [normalize_for_match(value).strip() for value in samples if isinstance(value, str)] + strong_samples = {value for value in normalized if sum(char.isalnum() for char in value) >= 20} + if (len(normalized) != len(samples) or len(strong_samples) < 2 + or any(not any(char.isalnum() for char in value) + or value not in saved or value not in observed for value in normalized)): + raise ValueError("Page review needs matching substantive text in both bodies") + navigation = (case.get("source_benchmark", {}).get("risk_control", {}) + .get("navigation_contract", {})) + acceptable = [navigation.get("requested_url"), navigation.get("observed_final_url")] + if not any(value and _same_page_url(corresponding_url, value) for value in acceptable): + raise ValueError("Reviewed page does not satisfy the frozen URL requirement") + if historical: + # The revision chain already hash-verifies these historical results. + # Validate their review evidence without replaying a different scorer + # and pretending its metrics were produced by the historical version. + metrics = record['metrics'] + if (metrics.get('scoring_source_sha256') != historical_scoring_source + or metrics.get('page_identity_required') is not True + or metrics.get('page_identity_status') != 'verified' + or metrics.get('final_url_consistent') is not True + or metrics.get('content_chars') != len(answer.decode('utf-8')) + or metrics.get('page_identity_review_sha256') != _canonical_sha256(annotation)): + raise ValueError('Reviewed result differs from its bound historical shared scorer') + return dict(metrics) + metrics = score_attempt( + case, answer, ok=record["ok"], metadata=record.get("provider_metadata"), + previous_metrics=record["metrics"], status_code=record.get("status_code"), + title=record.get("title"), final_url=corresponding_url, + ) + metrics["page_identity_basis"] = "retrospective_content_correspondence" + metrics["page_identity_review_sha256"] = _canonical_sha256(annotation) + return metrics + + +def _require_page_identity_review(source, dataset, records, *, scoring_sources=None): + binding = dataset.get("comparison", {}).get("page_identity_review") + marked = {r["attempt_id"] for r in records + if r["metrics"].get("page_identity_basis") == "retrospective_content_correspondence"} + if binding is None: + if marked: + raise ValueError("Reviewed page identities lack a receipt") + return + path = contained_path(source, binding["path"], reject_symlinks=True) + if _sha256(path) != binding["sha256"]: + raise ValueError("Page identity review receipt changed") + receipt = json.loads(path.read_text()) + capture_path = contained_path(source, receipt["captures_path"], reject_symlinks=True) + if _sha256(capture_path) != receipt["captures_sha256"]: + raise ValueError("Page review captures changed") + captures = json.loads(capture_path.read_text()) + _require_identity_binding_revision(source, dataset, receipt) + annotations = receipt["attempts"] + if (receipt.get("kind") != "retrospective_page_identity_review" + or receipt.get("provider_calls_added") != 0 + or len(annotations) != len(marked) + or {a["attempt_id"] for a in annotations} != marked): + raise ValueError("Page review scope differs from annotated results") + cases = {case["id"]: case for case in dataset["cases"]} + indexed = {r["attempt_id"]: r for r in records} + for annotation in annotations: + record = indexed[annotation["attempt_id"]] + declared = None + if scoring_sources is not None: + declared = scoring_sources.get(record["attempt_id"]) + if not declared or record["metrics"].get("scoring_source_sha256") != declared: + raise ValueError("Reviewed result lost its revision scoring source") + expected = score_reviewed_page_identity( + source, cases[record["case_id"]], record, annotation, captures, + historical_scoring_source=declared) + if expected != record["metrics"]: + raise ValueError("Reviewed result differs from the shared scorer") + + +def _require_local_review(source, dataset, records): + """Bind optional full independent review without certifying local evidence.""" + _require_page_identity_review(source, dataset, records) + binding = dataset.get("comparison", {}).get("independent_review") + if binding is None: + return None + path = contained_path(source, binding["path"], reject_symlinks=True) + if _sha256(path) != binding.get("sha256"): + raise ValueError("Local independent review binding changed") + review = json.loads(path.read_text()) + rows = review.get("attempts", []) + indexed = {row["attempt_id"]: row for row in rows} + expected = {row["attempt_id"]: row for row in records} + if (review.get("kind") != "local_independent_review" + or review.get("review_protocol", {}).get("scope") != "all" + or review.get("reviewed_attempts") != len(records) + or review.get("expected_attempts") != len(records) + or review.get("results_sha256") != _sha256(source / "results.jsonl") + or len(indexed) != len(rows) or set(indexed) != set(expected)): + raise ValueError("Local independent review scope differs") + unresolved = 0 + from lexbench_webfetch.metrics import task_measured + for identifier, row in indexed.items(): + record = expected[identifier] + verdict = ("unavailable" if not task_measured(record["metrics"]) else + "pass" if task_passed(record["ok"], record["metrics"]) else "fail") + if (row.get("content_sha256") != record["metrics"]["content_sha256"] + or row.get("current_verdict") != verdict + or row.get("agent_verdict") not in {"pass", "fail", "insufficient"} + or row.get("final_verdict") not in {verdict, None}): + raise ValueError("Local review differs from scored evidence") + if row.get("final_verdict") is None: + unresolved += 1 + if row.get("status") != "deferred_unadjudicated": + raise ValueError("Local unresolved review lacks explicit deferral") + if review.get("unresolved") != unresolved: + raise ValueError("Local review unresolved count differs") + from .adjudication_evidence import require_adjudication_evidence + require_adjudication_evidence(source, dataset, records, review) + return review + + def _require_local_method_bindings(contract, batches, field): """Join each method's execution to its declared runtime or configuration.""" variants = contract.get(field + "_by_provider") @@ -455,37 +638,249 @@ def _require_local_method_bindings(contract, batches, field): raise ValueError(f"Local {label} batches repeat or omit a method") -def require_local_measurement(source: Path, dataset: dict, records: list[dict]) -> None: - """Validate a new complete local matrix without claiming independent review.""" - from .reliability import require_reliable +def local_measurement_cases(contract: dict, source=None) -> list[dict]: + """Read an ordered selection from the original hash-bound input trees.""" + bindings = contract.get("dataset_sources") + if bindings is None: + bindings = [{"path": contract["dataset_path"], + "sha256": contract["dataset_sha256"]}] + if not isinstance(bindings, list) or not bindings: + raise ValueError("Local measurement dataset sources are missing") + cases, seen_paths, seen_ids = [], set(), set() + for binding in bindings: + relative = binding["path"] + root = source if contract.get("dataset_storage") == "evidence" else ROOT + if root is None: + raise ValueError("Evidence-relative inputs require their evidence root") + path = contained_path(root, relative, reject_symlinks=True) + raw = read_dataset_bytes(path) + if relative in seen_paths or hashlib.sha256(raw).hexdigest() != binding["sha256"]: + raise ValueError("Local measurement dataset source changed") + seen_paths.add(relative) + available = json.loads(raw)["cases"] + selected = binding.get("case_ids", [case["id"] for case in available]) + if not selected or len(selected) != len(set(selected)): + raise ValueError("Local measurement source selection differs") + rows = [case for case in available if case["id"] in set(selected)] + if [case["id"] for case in rows] != selected or seen_ids.intersection(selected): + raise ValueError("Local measurement source selection differs") + seen_ids.update(selected) + cases.extend(rows) + return cases + + +def _require_local_execution_segments(source, contract, batches, records): + """Verify resumed measurements against retained pre-call plans and physical files.""" + correction_path = contained_path(source, "local-evidence/scoring-correction.json") + correction = json.loads(correction_path.read_text()) if correction_path.is_file() else None + def read(binding, *, json_lines=False): + path = contained_path(source, binding["path"], reject_symlinks=True) + if _sha256(path) != binding["sha256"]: + raise ValueError("Local execution evidence binding changed") + if json_lines: + return [json.loads(line) for line in path.read_text().splitlines() if line.strip()] + return json.loads(path.read_text()) + + plans = contract["execution_plans"] + if set(plans) not in ({"original", "resumed"}, {"original", "resumed", "admitted"}): + raise ValueError("Local execution plans are incomplete") + original, resumed = (read(plans[key]) for key in ("original", "resumed")) + if (original.get("kind") != "uniform_native_local_measurement" + or resumed.get("kind") != "post_merge_resume_and_runtime_correction" + or original.get("providers") != contract["providers"] + or original.get("dataset_sources") != contract.get("dataset_sources") + or original.get("scoring_source_sha256") != contract["scoring_source_sha256"] + or original.get("runs") != 1 or original.get("timeout_seconds") != 45 + or original.get("max_failure_retries") != 0): + raise ValueError("Local execution changed the frozen protocol") + expected_policy = {"runs": 1, "timeout": 45, "delay": original["delay"], + "provider_concurrency": 1, "max_failure_retries": 0} + if resumed.get("parameters") != expected_policy: + raise ValueError("Local resumed execution parameters differ") + configs = {key: read(plans[key]["configuration"]) for key in plans} + admitted = read(plans["admitted"]) if "admitted" in plans else None + if admitted and (admitted.get("kind") != "bounded_admission_continuation" + or admitted.get("predecessor_plan_sha256") != plans["resumed"]["sha256"] + or admitted.get("parameters") != expected_policy + or admitted.get("providers") != contract["providers"] + or admitted.get("dataset_sha256") != resumed.get("dataset_sha256") + or configs["admitted"] != configs["resumed"]): + raise ValueError("Local admission changed the frozen protocol or configuration") + plan_values = {"original": original, "resumed": resumed} + if admitted: + plan_values["admitted"] = admitted + for key, plan in plan_values.items(): + if plans[key]["configuration"]["sha256"] != plan["provider_config_sha256"]: + raise ValueError("Local execution configuration differs from its pre-call plan") + from copy import deepcopy + normalized = deepcopy(configs["resumed"]) + for provider in normalized.get("providers", {}).values(): + for field in ("command", "required_paths"): + if field in provider: + provider[field] = [value.replace( + "{harness_root}/harness/web_fetch/workers/", "{harness_root}/scripts/oss/") + if isinstance(value, str) else value for value in provider[field]] + if normalized != configs["original"]: + raise ValueError("Local resumed configuration changed beyond worker paths") + names = set(contract["providers"]) + scopes = resumed.get("case_ids_by_provider", {}) + cases = [case["id"] for case in local_measurement_cases(contract)] + if set(scopes) != names or any( + len(scope) != len(set(scope)) or not set(scope) <= set(cases) + for scope in scopes.values() + ): + raise ValueError("Local resumed execution has an invalid task scope") + admitted_scopes = admitted.get("case_ids_by_provider", {}) if admitted else {} + if admitted and (set(admitted_scopes) != names or any( + len(scope) != len(set(scope)) or not set(scope) <= set(scopes[name]) + for name, scope in admitted_scopes.items())): + raise ValueError("Local admission has an invalid remaining task scope") + by_attempt = {record["attempt_id"]: record for record in records} + seen_names, seen_attempts = set(), set() + for batch in batches: + providers = batch.get("providers", []) + if len(providers) != 1: + raise ValueError("Local execution batch must identify one method") + name = providers[0]["name"] + if name not in names or name in seen_names: + raise ValueError("Local execution repeats or omits a method") + seen_names.add(name) + segments = batch.get("segments", []) + if not segments: + raise ValueError("Local resumed execution has no physical segments") + expected_scopes = {"resumed": scopes[name], + "original": [case for case in cases if case not in set(scopes[name])]} + if admitted: + expected_scopes["resumed"] = [case for case in scopes[name] + if case not in set(admitted_scopes[name])] + expected_scopes["admitted"] = admitted_scopes[name] + declared_scopes = {"original": expected_scopes["original"], + "resumed": scopes[name]} + if admitted: + declared_scopes["admitted"] = admitted_scopes[name] + seen_plans, ids, manifest_paths = set(), [], set() + offsets = {key: 0 for key in plans} + for segment in segments: + key = segment.get("plan_id") + if key not in plans or (key == "original" and key in seen_plans): + raise ValueError("Local execution repeats or substitutes a physical segment") + seen_plans.add(key) + plan = plan_values[key] + manifest_path = segment["manifest"]["path"] + if manifest_path in manifest_paths: + raise ValueError("Local execution repeats a physical manifest") + manifest_paths.add(manifest_path) + manifest = read(segment["manifest"]) + physical = read(segment["records"], json_lines=True) + full_ids = [f"{name}__{case}__r001" for case in expected_scopes[key]] + expected_ids = full_ids[offsets[key]:offsets[key] + len(physical)] + if not physical: + raise ValueError("Local execution has an empty physical segment") + offsets[key] += len(physical) + if ([row.get("attempt_id") for row in physical] != expected_ids + or segment.get("attempt_ids") != expected_ids + or segment.get("attempt_count") != len(expected_ids)): + raise ValueError("Local execution segment task selection differs") + if (manifest.get("runtime_source_sha256") != plan["runtime_source_sha256"] + or manifest.get("provider_config_sha256") != plan["provider_config_sha256"] + or manifest.get("providers") != segment.get("providers") + or manifest.get("parameters") != segment.get("parameters")): + raise ValueError("Local execution segment differs from its physical manifest") + if (len(manifest["providers"]) != 1 or manifest["providers"][0]["name"] != name + or any(manifest["parameters"].get(field) != value + for field, value in expected_policy.items())): + raise ValueError("Local execution segment parameters differ") + if key != "original" and ( + not manifest.get("completed_at") + or manifest.get("attempt_ids") != expected_ids + or manifest.get("attempt_count") != len(expected_ids) + ): + raise ValueError("Local resumed execution is incomplete") + if key != "original" and manifest.get("unexecuted"): + if (physical[-1].get("error_type") != "timeout" + or any(item.get("reason") != + "non_cancellable_call_still_running; remaining provider cases not executed" + or item.get("after_attempt_id") != physical[-1]["attempt_id"] + for item in manifest["unexecuted"])): + raise ValueError("Local execution stopped for an unaccounted reason") + if key == "admitted" and manifest.get("scheduling") != admitted.get("scheduling"): + raise ValueError("Local admission manifest changed the scheduling contract") + continuation = segment.get("continuation_plan") + if continuation: + continuation = read(continuation) + expected_kind = {"resumed": "resume_unexecuted_after_process_exit", + "admitted": "resume_unexecuted_with_admission"}.get(key) + parent_field = "admission_plan_sha256" if key == "admitted" else "resumed_plan_sha256" + if (key == "original" or continuation.get("kind") != expected_kind + or continuation.get("method") != name + or continuation.get(parent_field) != plans[key]["sha256"] + or continuation.get("parameters") != expected_policy + or continuation.get("provider_config_sha256") != plan["provider_config_sha256"] + or continuation.get("runtime_source_sha256") != plan["runtime_source_sha256"] + or continuation.get("dataset_sha256") != plan.get("dataset_sha256") + or continuation.get("case_ids") != declared_scopes[key][offsets[key] - len(physical):]): + raise ValueError("Local continuation differs from its pre-call scope") + for row in physical: + identifier = row["attempt_id"] + if identifier in seen_attempts or identifier not in by_attempt: + raise ValueError("Local execution altered or repeated a measured result") + if row != by_attempt[identifier]: + _require_local_record_correction(source, row, by_attempt[identifier], correction) + seen_attempts.add(identifier) + ids.extend(expected_ids) + if (any(offsets[key] != len(scope) for key, scope in expected_scopes.items()) + or seen_plans != {key for key, scope in expected_scopes.items() if scope} + or len(ids) != batch.get("attempt_count") + or set(ids) != set(batch.get("attempt_ids", []))): + raise ValueError("Local execution segments repeat or omit a call") + if seen_names != names or seen_attempts != set(by_attempt): + raise ValueError("Local execution lineage is incomplete") + + +def require_complete_local_inventory(dataset: dict, records: list[dict]) -> dict: + """Allow explicit unknown verdicts locally, never missing physical results.""" + from .reliability import assess_reliability + + comparison = dataset.get("comparison", {}) + if (comparison.get("kind") != "uniform_native_local_measurement" + or comparison.get("publication_state") != "local_audit_only_not_for_release"): + raise ValueError("Incomplete adjudication is only allowed in a local measurement") + reliability = assess_reliability(dataset, records) + if (not reliability["expected_calls"] or reliability["missing"] + or reliability["extra"] or reliability["duplicates"]): + raise ValueError("Local measurement physical inventory is incomplete") + return reliability + + +def require_local_execution_contract(source, dataset, records, contract, execution): + """Shared physical-call boundary for complete cohorts and task replacements.""" from .artifacts import verify_record_public_call_logs, record_public_call_log_bindings - contract = json.loads(contained_path(source, "local-run-contract.json").read_text()) + from datetime import datetime if (contract.get("schema_version") != 1 or contract.get("kind") != "uniform_native_local_measurement" or contract.get("runs") != 1 or contract.get("timeout_seconds") != 45 or contract.get("max_failure_retries") != 0): raise ValueError("Invalid uniform native local measurement contract") - _require_local_scoring_revision(source, contract, records) - raw = read_dataset_bytes(contained_path(ROOT, contract["dataset_path"], reject_symlinks=True)) - canonical = json.loads(raw) - if (hashlib.sha256(raw).hexdigest() != contract["dataset_sha256"] - or dataset["cases"] != canonical["cases"]): + canonical = {"cases": (local_measurement_cases(contract, source) + if contract.get("dataset_storage") == "evidence" + else local_measurement_cases(contract))} + if dataset["cases"] != canonical["cases"]: raise ValueError("Local measurement changed the committed cases") expected = {(p, c["id"]) for p in contract["providers"] for c in canonical["cases"]} if (len(records) != len(expected) or {(r["provider"], r["case_id"]) for r in records} != expected or any(r["run"] != 1 or r.get("provider_metadata", {}).get("retry") for r in records)): raise ValueError("Local measurement matrix is incomplete or has retries") - require_reliable(dataset, records) - execution = json.loads(contained_path(source, "execution.json").read_text()) - if dataset.get("comparison", {}).get("local_run_contract_sha256") != _sha256(source / "local-run-contract.json"): - raise ValueError("Local measurement pre-call contract changed") if len(execution["batches"]) != len(contract["providers"]): raise ValueError("Local measurement execution scope differs") - for field in ("runtime_source_sha256", "provider_config_sha256"): - _require_local_method_bindings(contract, execution["batches"], field) + if contract.get("execution_plans"): + _require_local_execution_segments(source, contract, execution["batches"], records) + else: + for field in ("runtime_source_sha256", "provider_config_sha256"): + _require_local_method_bindings(contract, execution["batches"], field) for batch in execution["batches"]: - _require_local_batch_segments(batch) + if not contract.get("execution_plans"): + _require_local_batch_segments(batch) ids = batch.get("attempt_ids", []) measured_ids = {r["attempt_id"] for r in records if r["provider"] == batch["providers"][0]["name"]} @@ -497,7 +892,12 @@ def require_local_measurement(source: Path, dataset: dict, records: list[dict]) or batch.get("parameters", {}).get("provider_concurrency") != 1 or batch.get("parameters", {}).get("max_failure_retries") != 0): raise ValueError("Local measurement execution parameters differ") + if contract.get("runner_inputs"): + _require_local_runner_inputs(source, contract, execution, records) + canonical_by_id = {case['id']: case for case in canonical['cases']} for record in records: + if record.get('url') != canonical_by_id[record['case_id']]['url']: + raise ValueError('Local measurement request URL differs from frozen task') if len(record_public_call_log_bindings(record)) != 1: raise ValueError("Local measurement requires one complete call log per result") lexical = source @@ -514,9 +914,27 @@ def require_local_measurement(source: Path, dataset: dict, records: list[dict]) if (any(log["identity"][key] != record[key] for key in ("attempt_id", "provider", "case_id", "run")) or log["output"]["path"] != record["artifact_path"] - or log["output"]["sha256"] != record["metrics"]["content_sha256"]): + or log["output"]["sha256"] != record["metrics"]["content_sha256"] + or log['request']['url'] != record['url'] + or log['timing'].get('elapsed_ms') != record.get('elapsed_ms') + or not (datetime.fromisoformat(record['started_at']) + <= datetime.fromisoformat(log['timing']['started_at']) + <= datetime.fromisoformat(log['timing']['completed_at']) + <= datetime.fromisoformat(record['completed_at'])) + or log['timing'].get('deadline_budget_ms') != 45000): raise ValueError("Local measurement call log belongs to a different result") - return None + + +def require_local_measurement(source: Path, dataset: dict, records: list[dict]) -> None: + """Validate a new complete local matrix without claiming independent review.""" + contract = json.loads(contained_path(source, "local-run-contract.json").read_text()) + execution = json.loads(contained_path(source, "execution.json").read_text()) + if dataset.get("comparison", {}).get("local_run_contract_sha256") != _sha256(source / "local-run-contract.json"): + raise ValueError("Local measurement pre-call contract changed") + _require_local_scoring_revision(source, contract, records) + require_local_execution_contract(source, dataset, records, contract, execution) + require_complete_local_inventory(dataset, records) + return _require_local_review(source, dataset, records) def require_local_audit(source: Path, dataset: dict, records: list[dict]) -> dict: @@ -547,3 +965,169 @@ def require_local_audit(source: Path, dataset: dict, records: list[dict]) -> dic raise ValueError("Local audit manifest binding mismatch: " + name) _require_lineage(source, receipt) return _require_review(source, dataset, records, receipt, manifest) + + +def _require_local_runner_inputs(source, contract, execution, records): + """Verify retained runner inputs and source observations, without certifying them.""" + from datetime import datetime + from .local_revision import read_bound + inputs = contract['runner_inputs'] + def read(key): + return json.loads(read_bound(source, inputs[key]).read_text()) + frozen, configuration, network = (read(key) for key in ('dataset', 'providers', 'network')) + canonical = local_measurement_cases(contract, source) + if frozen['cases'] != canonical or inputs['dataset'] != contract['dataset_sources'][0]: + raise ValueError('Local runner dataset differs from frozen task inputs') + if inputs['providers']['sha256'] != contract['provider_config_sha256']: + raise ValueError('Local runner configuration binding differs') + if not set(contract['providers']) <= set(configuration['providers']): + raise ValueError('Local runner configuration omits a method') + if network.get('dns_mode') != 'system' or network.get('usable_for_guarded_local_comparison') is not True: + raise ValueError('Local runner lacks usable system DNS evidence') + from .network_preflight import GUARDED_NATIVE_TYPES, require_usable_public_dns + require_usable_public_dns(network) + from urllib.parse import urlsplit + method_types = {name: configuration['providers'][name].get('method_type', + configuration['providers'][name]['type']) for name in contract['providers']} + if any(value in GUARDED_NATIVE_TYPES for value in method_types.values()): + hosts = network.get('hosts', []) + expected_hosts = {urlsplit(case['url']).hostname for case in canonical} + if (len(hosts) != len(expected_hosts) or {host.get('host') for host in hosts} != expected_hosts + or any(not host.get('addresses') or host.get('dns_error') for host in hosts)): + raise ValueError('Local runner system DNS scope is incomplete') + for batch in execution['batches']: + name = batch['providers'][0]['name'] + guarded = method_types[name] in GUARDED_NATIVE_TYPES + if (batch['providers'][0].get('type') != method_types[name] + or (guarded and not batch['providers'][0].get('implementation_identity')) + or any(row.get('provider_type') != method_types[name] + for row in records if row['provider'] == name)): + raise ValueError('Local runner method identity differs from frozen configuration') + expected_network = ({'dns_mode': 'system', 'sha256': inputs['network']['sha256']} + if guarded else None) + if (batch.get('dataset_sha256') != inputs['dataset']['sha256'] + or batch.get('environment') != contract['environment'] + or batch.get('parameters') != contract['parameters'] + or batch.get('network_preflight') != expected_network + or not batch.get('completed_at') or batch.get('unexecuted')): + raise ValueError('Local runner execution differs from bound inputs or environment') + if any(not batch.get('environment', {}).get(key) for batch in execution['batches'] + for key in ('python', 'platform')): + raise ValueError('Local runner execution lacks environment identity') + observations = inputs['source_observations'] + if set(observations) != {case['id'] for case in canonical}: + raise ValueError('Local runner source observation scope differs') + for case in canonical: + times = [datetime.fromisoformat(row['started_at']) for row in records if row['case_id'] == case['id']] + ends = [datetime.fromisoformat(row['completed_at']) for row in records if row['case_id'] == case['id']] + for stage in ('pre', 'post'): + capture = observations[case['id']][stage] + evidence = json.loads(read_bound(source, capture['evidence']).read_text()) + for key, field in [('source', 'sha256'), + ('reference', 'reference_sha256'), ('extracted', 'extracted_sha256')]: + read_bound(source, capture[key]) + if capture[key]['sha256'] != evidence.get(field): + raise ValueError('Local runner source bytes differ from observation') + observed = datetime.fromisoformat(evidence['checked_at']) + if (evidence.get('case_id') != case['id'] or evidence.get('url') != case['url'] + or evidence.get('independent') is not True or evidence.get('status') != 200 + or (stage == 'pre' and observed > min(times)) + or (stage == 'post' and observed < max(ends))): + raise ValueError('Local runner source observation does not bracket the calls') + + +def _hint_migrated_case(case): + import copy + result = copy.deepcopy(case) + hints = result.pop('forbidden_any', []) + if hints: + result['diagnostic_any'] = list(dict.fromkeys(result.get('diagnostic_any', []) + hints)) + result.pop('acceptance_policy_version', None) + return result + + +def _require_identity_binding_revision(source, dataset, receipt): + from .local_revision import read_bound + migration = receipt.get('binding_revision') + if not migration: + return + previous = json.loads(read_bound(source, migration['prior_receipt']).read_text()) + old_cases = json.loads(read_bound(source, migration['prior_cases']).read_text()) + cases = {case['id']: case for case in dataset['cases']} + if (migration.get('kind') != 'diagnostic_field_migration' + or migration.get('new_independent_review') is not False + or receipt.get('captures_path') != previous.get('captures_path') + or receipt.get('captures_sha256') != previous.get('captures_sha256')): + raise ValueError('Page identity evidence migration changed source observations') + annotations = [] + for annotation in previous['attempts']: + identifier = annotation['attempt_id'].split('__')[1] + old = old_cases[identifier] + migrated = _hint_migrated_case(old) + # Older compiled navigation tasks implied identity acceptance. Keep + # that requirement explicit when adopting the diagnostic default. + if (cases[identifier].get('page_identity_required') is True + and 'page_identity_required' not in old): + navigation = old.get('source_benchmark', {}).get('risk_control', {}).get('navigation_contract', {}) + dynamic_identity = (old.get('track') == 'dynamic' and old.get('evaluation_url') + and old.get('expected_final_url') and old.get('target_language') + and old.get('target_region')) + if ((old.get('track') == 'risk_control' and navigation.get('requested_url') + and navigation.get('observed_final_url')) or dynamic_identity): + migrated['page_identity_required'] = True + if (annotation['case_sha256'] != _canonical_sha256(old) + or migrated != cases[identifier]): + raise ValueError('Page identity migration changed its target contract') + annotations.append({**annotation, 'prior_case_sha256': annotation['case_sha256'], + 'case_sha256': _canonical_sha256(cases[identifier])}) + if receipt['attempts'] != annotations: + raise ValueError('Page identity migration changed an existing reviewed annotation') + + +def carry_reviewed_page_identities(source, before_dataset, dataset, records): + """Reapply retained human evidence after offline scoring; never invent a URL.""" + from .local_revision import read_bound + binding = before_dataset.get('comparison', {}).get('page_identity_review') + if not binding: + return None + original = json.loads(read_bound(source, binding).read_text()) + captures_path = contained_path(source, original['captures_path'], reject_symlinks=True) + if _sha256(captures_path) != original['captures_sha256']: + raise ValueError('Page identity capture changed before rescoring') + captures = json.loads(captures_path.read_text()) + old_cases = {case['id']: case for case in before_dataset['cases']} + cases = {case['id']: case for case in dataset['cases']} + indexed = {row['attempt_id']: row for row in records} + changed = any(annotation['case_sha256'] != _canonical_sha256(cases[indexed[annotation['attempt_id']]['case_id']]) + for annotation in original['attempts']) + receipt = original + if changed: + prior_cases = {indexed[a['attempt_id']]['case_id']: old_cases[indexed[a['attempt_id']]['case_id']] + for a in original['attempts']} + key = _canonical_sha256({'prior_receipt': binding, 'cases': {key: cases[key] for key in prior_cases}})[:16] + directory = source / 'local-evidence/page-identity-review/rebindings' / key + directory.mkdir(parents=True, exist_ok=True) + old_path = directory / 'prior-cases.json' + prior_bytes = (json.dumps(prior_cases, ensure_ascii=False, indent=2)+'\n').encode() + if old_path.exists() and old_path.read_bytes() != prior_bytes: + raise ValueError('Page identity migration would overwrite prior case evidence') + old_path.write_bytes(prior_bytes) + receipt = {key: original[key] for key in ('kind', 'reviewed_at', 'basis', 'provider_calls_added', 'captures_path', 'captures_sha256')} + receipt['attempts'] = [{**a, 'prior_case_sha256': a['case_sha256'], + 'case_sha256': _canonical_sha256(cases[indexed[a['attempt_id']]['case_id']])} for a in original['attempts']] + receipt['binding_revision'] = {'kind': 'diagnostic_field_migration', 'prior_receipt': binding, + 'prior_cases': {'path': str(old_path.relative_to(source)), 'sha256': _sha256(old_path)}, + 'new_independent_review': False} + _require_identity_binding_revision(source, dataset, receipt) + path = directory / 'receipt.json' + raw = (json.dumps(receipt, ensure_ascii=False, indent=2)+'\n').encode() + if path.exists() and path.read_bytes() != raw: + raise ValueError('Page identity migration would overwrite a prior receipt') + path.write_bytes(raw) + binding = {'path': str(path.relative_to(source)), 'sha256': _sha256(path)} + dataset['comparison']['page_identity_review'] = binding + for annotation in receipt['attempts']: + row = indexed[annotation['attempt_id']] + row['metrics'] = score_reviewed_page_identity(source, cases[row['case_id']], row, annotation, captures) + _require_page_identity_review(source, dataset, records) + return binding diff --git a/harness/web_fetch/local_revision.py b/harness/web_fetch/local_revision.py new file mode 100644 index 000000000..9da19227a --- /dev/null +++ b/harness/web_fetch/local_revision.py @@ -0,0 +1,383 @@ +"""Validate task replacements and offline revisions of a bound local snapshot.""" +import hashlib +import json +from pathlib import Path + +from harness.record_io import load_jsonl + +from .cohort_exclusions import require_cohort_exclusions +from lexbench_webfetch.paths import contained_path +from lexbench_webfetch.scoring_regressions import require_scoring_regressions +from .scoring_revision import ALLOWED_FIELDS + + +def read_bound(source, binding): + from .publication_privacy import read_evidence_binding + return read_evidence_binding(source, binding) + + +def require_local_task_replacement(source, binding, dataset, records): + """Verify complete new task calls while preserving every unrelated saved row.""" + receipt = json.loads(read_bound(source, binding).read_text()) + if receipt.get('kind') != 'local_task_replacement' or not receipt.get('reason'): + raise ValueError('Task replacement needs a declared reason') + before = json.loads(read_bound(source, receipt['baseline_dataset']).read_text()) + previous = load_jsonl(read_bound(source, receipt['baseline_results'])) + cohort = json.loads(read_bound(source, receipt['cohort_dataset']).read_text()) + calls = load_jsonl(read_bound(source, receipt['cohort_results'])) + removed, added = receipt['removed_case_id'], receipt['added_case_id'] + old_cases = {c['id']: c for c in before['cases']} + new_cases = {c['id']: c for c in dataset['cases']} + if (removed == added or removed not in old_cases or added in old_cases + or len(old_cases) != len(before['cases']) or len(new_cases) != len(dataset['cases']) + or set(new_cases) != (set(old_cases) - {removed}) | {added} + or before.get('coverage_plan') != dataset.get('coverage_plan') + or any(new_cases[k] != v for k, v in old_cases.items() if k != removed) + or cohort['cases'] != [new_cases[added]] + or old_cases[removed].get('track') != new_cases[added].get('track')): + raise ValueError('Task replacement changed unrelated cases or task family') + providers = set(before['coverage_plan'][old_cases[removed].get('track', 'web')]) + expected = {(p, added) for p in providers} + if (len(calls) != len(expected) or receipt.get('provider_calls_added') != len(calls) + or {(r['provider'], r['case_id']) for r in calls} != expected + or any(r.get('run') != 1 or r.get('provider_metadata', {}).get('retry') for r in calls)): + raise ValueError('Task replacement needs one new call from every method') + inherited = [r for r in previous if r['case_id'] != removed] + rows = {r['attempt_id']: r for r in records} + wanted = {r['attempt_id']: r for r in inherited + calls} + if (len(rows) != len(records) or len(wanted) != len(inherited) + len(calls) + or len(records) != len(previous) or rows != wanted): + raise ValueError('Task replacement altered an inherited or newly measured result') + if any(r['metrics'].get('evaluation_excluded') for r in calls): + raise ValueError('Replacement task cannot be excluded after seeing its results') + from .local_audit import require_local_execution_contract + if not receipt.get('execution_contract') or not receipt.get('cohort_execution'): + raise ValueError('Task replacement requires its complete execution contract') + contract = json.loads(read_bound(source, receipt['execution_contract']).read_text()) + execution = json.loads(read_bound(source, receipt['cohort_execution']).read_text()) + if not contract.get('runner_inputs'): + raise ValueError('Task replacement requires retained runner inputs and source observations') + require_local_execution_contract(source, cohort, calls, contract, execution) + return {identifier: row['metrics']['scoring_source_sha256'] + for identifier, row in rows.items()} + + +def validate_revision_chain(source, binding, dataset=None, records=None): + """Validate each transition, then expose the earliest replacement inputs.""" + if dataset is None: + dataset = json.loads((source / 'dataset.json').read_text()) + if records is None: + records = load_jsonl(source / 'results.jsonl') + outputs = {name: hashlib.sha256((source / name).read_bytes()).hexdigest() + for name in ('dataset.json', 'results.jsonl')} + seen, latest_scoring = set(), None + while True: + identity = (binding['path'], binding['sha256']) + if identity in seen: + raise ValueError('Local score revision lineage repeats a receipt') + seen.add(identity) + receipt = json.loads(read_bound(source, binding).read_text()) + expected, before, previous = _require_revision_step(source, receipt, dataset, records, outputs) + if latest_scoring is None: + latest_scoring = expected + parent = receipt.get('predecessor_revision') + if parent is None: + return latest_scoring, receipt['input_dataset'], receipt['input_results'] + outputs = {'dataset.json': receipt['input_dataset']['sha256'], + 'results.jsonl': receipt['input_results']['sha256']} + dataset, records, binding = before, previous, parent + + +def require_local_revision(source, binding, dataset, records): + return validate_revision_chain(source, binding, dataset, records)[0] + + +def _require_native_answer_correction(source, original, corrected, proof): + """Verify a derived answer against the same immutable native response.""" + from lexbench_webfetch.answer_projection import project_native_answer, project_native_result_fields + from .publication_privacy import original_evidence_size + if proof.get('attempt_id') != original['attempt_id']: + raise ValueError('Native answer correction lost its attempt identity') + native = read_bound(source, proof['native_response']) + metadata = original.get('provider_metadata', {}) + if (proof['native_response']['sha256'] != metadata.get('raw_response_sha256') + or original_evidence_size(source, proof['native_response']) != metadata.get('raw_response_bytes')): + raise ValueError('Native answer correction changed its original response') + before = proof['original_artifact'] + after = proof['corrected_artifact'] + if (before['path'] != original['artifact_path'] + or before['sha256'] != original['metrics']['content_sha256'] + or after['path'] != corrected['artifact_path'] + or after['sha256'] != corrected['metrics']['content_sha256'] + or before['path'] == after['path']): + raise ValueError('Native answer correction must preserve its old artifact') + read_bound(source, before) + payload = json.loads(native.read_bytes()) + text, fields = project_native_answer(payload, proof['projection']) + result_fields = (project_native_result_fields(payload, proof['projection'], request_url=original.get('url')) + if 'result_fields' in proof['projection'] else {}) + if any(corrected.get(key) != value for key, value in result_fields.items()): + raise ValueError('Native result context correction differs from original fields') + if read_bound(source, after).read_bytes() != text.encode('utf-8', errors='replace'): + raise ValueError('Native answer correction differs from declared original fields') + for field in proof['projection'].get('answer_fields', {}).values(): + if field.get('format') == 'html_attribute_urls' and field.get('base_url') != (result_fields.get('final_url') or original['url']): + raise ValueError('Native URL projection must use the verified result or request URL') + expected_metadata = {**metadata, **fields, 'native_answer_projection': proof['projection']} + if corrected.get('provider_metadata') != expected_metadata: + raise ValueError('Native answer correction changed unrelated response metadata') + derived = {'metrics', 'scoring', 'diagnosis', 'failure_category', + 'artifact_path', 'provider_metadata'} | set(result_fields) + if ({k: v for k, v in original.items() if k not in derived} + != {k: v for k, v in corrected.items() if k not in derived}): + raise ValueError('Native answer correction changed physical call facts') + + +def _require_revision_step(source, receipt, dataset, records, outputs): + if receipt.get('kind') != 'local_offline_score_revision' or receipt.get('provider_calls_added') != 0: + raise ValueError('Local revision must preserve original provider calls') + if (not isinstance(receipt.get('authority', {}).get('description'), str) + or not receipt['authority']['description'].strip() + or not isinstance(receipt.get('changed_case_ids'), list) + or len(receipt['changed_case_ids']) != len(set(receipt['changed_case_ids'])) + or not isinstance(receipt.get('rescored_attempt_ids'), list) + or len(receipt['rescored_attempt_ids']) != len(set(receipt['rescored_attempt_ids'])) + or not isinstance(receipt.get('task_verdict_changes'), list)): + raise ValueError('Local revision needs authority, declared scope and verdict changes') + before_dataset = json.loads(read_bound(source, receipt['input_dataset']).read_text()) + previous = load_jsonl(read_bound(source, receipt['input_results'])) + for name, key in [('dataset.json', 'output_dataset_sha256'), ('results.jsonl', 'output_results_sha256')]: + if outputs[name] != receipt.get(key): + raise ValueError('Local revision output binding changed: ' + name) + regressions = receipt.get('regression_ids') + if not regressions: + raise ValueError('Local revision requires executable regression provenance') + require_scoring_regressions(regressions) + old_cases = {case['id']: case for case in before_dataset['cases']} + new_cases = {case['id']: case for case in dataset['cases']} + if (len(old_cases) != len(before_dataset['cases']) or len(new_cases) != len(dataset['cases']) + or set(old_cases) != set(new_cases) + or before_dataset.get('coverage_plan') != dataset.get('coverage_plan')): + raise ValueError('Local revision changed the original case/method matrix') + changed_cases = set() + for identifier, old in old_cases.items(): + new = new_cases[identifier] + fields = {key for key in old.keys() | new.keys() if old.get(key) != new.get(key)} + if fields - (ALLOWED_FIELDS | {'diagnostic_any', 'acceptance_policy_version'}): + raise ValueError('Local score revision changed source or acquisition inputs') + if fields: + changed_cases.add(identifier) + if changed_cases != set(receipt.get('changed_case_ids', [])): + raise ValueError('Local revision case scope differs') + if changed_cases and receipt.get('rule_revision') is not True: + raise ValueError('Changed acceptance contract requires an explicit rule revision') + old_rows = {row['attempt_id']: row for row in previous} + new_rows = {row['attempt_id']: row for row in records} + if len(old_rows) != len(previous) or len(new_rows) != len(records) or set(old_rows) != set(new_rows): + raise ValueError('Local revision changed the call inventory') + scored = set(receipt.get('rescored_attempt_ids', [])) + if not scored <= set(new_rows): + raise ValueError('Local revision rescoring scope differs') + native_proofs = receipt.get('native_answer_corrections', []) + corrected_answers = {item['attempt_id']: item for item in native_proofs} + if (len(corrected_answers) != len(native_proofs) + or not set(corrected_answers) <= scored): + raise ValueError('Native answer correction scope differs from rescoring scope') + expected_scoring = {} + for identifier, old in old_rows.items(): + new = new_rows[identifier] + if identifier in corrected_answers: + _require_native_answer_correction(source, old, new, corrected_answers[identifier]) + elif ({k: v for k, v in old.items() if k not in {'metrics', 'scoring', 'diagnosis', 'failure_category'}} + != {k: v for k, v in new.items() if k not in {'metrics', 'scoring', 'diagnosis', 'failure_category'}} + or old['metrics']['content_sha256'] != new['metrics']['content_sha256']): + raise ValueError('Local revision changed measured call facts: ' + identifier) + if identifier in scored: + expected_scoring[identifier] = receipt['scoring_source_sha256'] + if (receipt.get('rule_revision') and + new.get('scoring', {}).get('rule_revision') is not True): + raise ValueError('Scoring policy change cannot be recorded as a parser-only correction') + else: + expected_scoring[identifier] = old['metrics']['scoring_source_sha256'] + before = {k: v for k, v in old['metrics'].items() if not k.startswith('evaluation_exclu')} + after = {k: v for k, v in new['metrics'].items() if not k.startswith('evaluation_exclu')} + if before != after: + raise ValueError('Unselected score changed: ' + identifier) + if new['case_id'] in changed_cases and identifier not in scored: + raise ValueError('Changed case was not rescored across every method') + if any(new_rows[key]['metrics'].get('scoring_source_sha256') != value + for key, value in expected_scoring.items()): + raise ValueError('Local revision scoring source binding changed') + from lexbench_webfetch.metrics import task_passed + if receipt.get('input_verdicts'): + proof = json.loads(read_bound(source, receipt['input_verdicts']).read_text()) + verdicts = proof.get('verdicts', {}) + if (proof.get('input_results_sha256') != receipt['input_results']['sha256'] + or set(verdicts) != set(old_rows) + or any(type(value) is not bool for value in verdicts.values()) + or not proof.get('frozen_predicate') or not proof.get('snapshot_manifest')): + raise ValueError('Frozen input verdict proof is incomplete') + read_bound(source, proof['frozen_predicate']) + read_bound(source, proof['snapshot_manifest']) + else: + verdicts = {key: task_passed(row['ok'], row['metrics']) for key, row in old_rows.items()} + transitions = [{'attempt_id': row['attempt_id'], 'before': verdicts[row['attempt_id']], + 'after': task_passed(row['ok'], row['metrics'])} + for row in records if row['attempt_id'] in scored + and verdicts[row['attempt_id']] != task_passed(row['ok'], row['metrics'])] + if transitions != receipt['task_verdict_changes']: + raise ValueError('Local revision verdict transitions differ from input proof') + from .local_audit import _require_page_identity_review + _require_page_identity_review(source, dataset, records, scoring_sources=expected_scoring) + require_cohort_exclusions(source, dataset, records) + return expected_scoring, before_dataset, previous + + +def revise_local_snapshot(source, configuration): + """Apply declared score-only changes in a separate snapshot, retaining predecessors.""" + import copy + from datetime import datetime, timezone + from lexbench_webfetch.scoring import score_attempt, load_list_reference, SCORING_SOURCE_SHA256 + from .cohort_exclusions import apply_cohort_exclusions + from lexbench_webfetch.metrics import task_passed + + config = json.loads(configuration.read_text()) + regressions = config.get('regression_ids', []) + require_scoring_regressions(regressions) + if not regressions or not config.get('authority', {}).get('description'): + raise ValueError('Local revision requires authority and regression provenance') + dataset_bytes = (source / 'dataset.json').read_bytes() + results_bytes = (source / 'results.jsonl').read_bytes() + input_results_sha256 = hashlib.sha256(results_bytes).hexdigest() + original_dataset = json.loads(dataset_bytes) + dataset = copy.deepcopy(original_dataset) + records = load_jsonl(source / 'results.jsonl') + replacements = {case['id']: case for case in config['cases']} + if len(replacements) != len(config['cases']) or not replacements.keys() <= {c['id'] for c in dataset['cases']}: + raise ValueError('Local revision has an invalid case scope') + dataset['cases'] = [replacements.get(case['id'], case) for case in dataset['cases']] + if dataset['cases'] != original_dataset['cases'] and config.get('rule_revision') is not True: + raise ValueError('Changed acceptance contract requires an explicit rule revision') + dataset['comparison']['excluded_cases'] = config.get('excluded_cases', []) + if config.get('rule_revision'): + dataset['comparison'].pop('scoring_policy_version', None) + prior_review = dataset['comparison'].pop('independent_review', None) + if prior_review: + dataset['comparison']['historical_independent_review'] = prior_review + additional = set(config.get('rescore_attempt_ids', [])) + if not additional <= {row['attempt_id'] for row in records}: + raise ValueError('Local revision has an invalid additional rescoring scope') + revision_key = (input_results_sha256[:16] + '-' + + hashlib.sha256(configuration.read_bytes()).hexdigest()[:16]) + history = source / 'local-evidence/offline-revisions' / revision_key + if history.exists(): + raise ValueError('Local revision already exists; preserve the prior receipt') + native_inputs = config.get('native_answer_corrections', []) + native_requests = {item['attempt_id']: item for item in native_inputs} + if (len(native_requests) != len(native_inputs) + or not set(native_requests) <= {row['attempt_id'] for row in records}): + raise ValueError('Native answer correction has an invalid input scope') + additional.update(native_requests) + indexed = {case['id']: case for case in dataset['cases']} + input_verdicts = None + if config.get('input_verdicts'): + proof = json.loads(read_bound(source, config['input_verdicts']).read_text()) + input_verdicts = proof.get('verdicts', {}) + if (proof.get('input_results_sha256') != input_results_sha256 + or set(input_verdicts) != {row['attempt_id'] for row in records} + or any(type(value) is not bool for value in input_verdicts.values())): + raise ValueError('Frozen input verdicts do not match the complete input results') + selected = [] + native_proofs = [] + before_verdicts = {} + for row in records: + if row['case_id'] not in replacements and row['attempt_id'] not in additional: + continue + case = indexed[row['case_id']] + raw = contained_path(source, row['artifact_path'], reject_symlinks=True).read_bytes() + if hashlib.sha256(raw).hexdigest() != row['metrics']['content_sha256']: + raise ValueError('Local revision output integrity failure') + before = (input_verdicts[row['attempt_id']] if input_verdicts is not None + else task_passed(row['ok'], row['metrics'])) + before_verdicts[row['attempt_id']] = before + if row['attempt_id'] in native_requests: + from lexbench_webfetch.answer_projection import project_native_answer, project_native_result_fields + request = native_requests[row['attempt_id']] + native_path = read_bound(source, request['native_response']) + metadata = row.get('provider_metadata', {}) + if (request['native_response']['sha256'] != metadata.get('raw_response_sha256') + or native_path.stat().st_size != metadata.get('raw_response_bytes')): + raise ValueError('Native correction source differs from its original response') + payload = json.loads(native_path.read_bytes()) + answer, fields = project_native_answer(payload, request['projection']) + if 'result_fields' in request['projection']: + row.update(project_native_result_fields(payload, request['projection'], request_url=row.get('url'))) + raw = answer.encode('utf-8', errors='replace') + original_artifact = {'path': row['artifact_path'], 'sha256': row['metrics']['content_sha256']} + digest = hashlib.sha256(raw).hexdigest() + old_path = Path(row['artifact_path']) + new_relative = old_path.with_name(old_path.stem + '-native-' + digest[:12] + old_path.suffix) + corrected_path = contained_path(source, new_relative, reject_symlinks=True) + if corrected_path.exists() and corrected_path.read_bytes() != raw: + raise ValueError('Native correction would overwrite a different artifact') + corrected_path.parent.mkdir(parents=True, exist_ok=True) + corrected_path.write_bytes(raw) + row['artifact_path'] = new_relative.as_posix() + row['provider_metadata'] = {**metadata, **fields, 'native_answer_projection': request['projection']} + native_proofs.append({'attempt_id': row['attempt_id'], + 'native_response': request['native_response'], 'projection': request['projection'], + 'original_artifact': original_artifact, + 'corrected_artifact': {'path': row['artifact_path'], 'sha256': digest}}) + row['metrics'] = score_attempt(case, raw, ok=row['ok'], status_code=row.get('status_code'), + title=row.get('title'), final_url=row.get('final_url'), metadata=row.get('provider_metadata'), + previous_metrics=row['metrics'], list_reference=load_list_reference(case, source)) + # Diagnoses describe the former score and must be regenerated from this answer. + row.pop('diagnosis', None) + row.pop('failure_category', None) + row['scoring'] = {**row.get('scoring', {}), 'mode': 'offline_rescore', + 'scored_at': datetime.now(timezone.utc).isoformat(), + 'rule_revision': bool(config.get('rule_revision')), + 'input_results_sha256': input_results_sha256} + selected.append(row['attempt_id']) + from .local_audit import carry_reviewed_page_identities + identity_binding = carry_reviewed_page_identities(source, original_dataset, dataset, records) + changes = [{'attempt_id': row['attempt_id'], 'before': before_verdicts[row['attempt_id']], + 'after': task_passed(row['ok'], row['metrics'])} + for row in records if row['attempt_id'] in before_verdicts + and before_verdicts[row['attempt_id']] != task_passed(row['ok'], row['metrics'])] + apply_cohort_exclusions(source, dataset, records) + history.mkdir(parents=True, exist_ok=False) + (history / 'dataset-before.json').write_bytes(dataset_bytes) + (history / 'results-before.jsonl').write_bytes(results_bytes) + def binding(path): + return {'path': str(path.relative_to(source)), 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + new_dataset = (json.dumps(dataset, ensure_ascii=False, indent=2) + '\n').encode() + new_results = ''.join(json.dumps(row, ensure_ascii=True) + '\n' for row in records).encode() + receipt = {'kind': 'local_offline_score_revision', 'provider_calls_added': 0, + 'input_dataset': binding(history / 'dataset-before.json'), + 'input_results': binding(history / 'results-before.jsonl'), + 'output_dataset_sha256': hashlib.sha256(new_dataset).hexdigest(), + 'output_results_sha256': hashlib.sha256(new_results).hexdigest(), + 'scoring_source_sha256': SCORING_SOURCE_SHA256, + 'changed_case_ids': sorted(case['id'] for case in dataset['cases'] + if case != next(c for c in original_dataset['cases'] if c['id'] == case['id'])), + 'rescored_attempt_ids': selected, 'rule_revision': bool(config.get('rule_revision')), + **({'native_answer_corrections': native_proofs} if native_proofs else {}), + 'input_verdicts': config.get('input_verdicts'), + **({'predecessor_revision': config['predecessor_revision']} if config.get('predecessor_revision') else {}), + 'superseded_controls': config.get('superseded_controls', []), + 'page_identity_review': identity_binding, + 'task_verdict_changes': changes, 'regression_ids': regressions, 'authority': config['authority']} + receipt_path = history / 'receipt.json' + receipt_path.write_text(json.dumps(receipt, ensure_ascii=False, indent=2) + '\n') + if (source / 'dataset.json').read_bytes() != dataset_bytes or (source / 'results.jsonl').read_bytes() != results_bytes: + raise ValueError('Local revision source changed during scoring') + (source / 'dataset.json').write_bytes(new_dataset) + (source / 'results.jsonl').write_bytes(new_results) + try: + require_local_revision(source, binding(receipt_path), dataset, records) + except BaseException: + (source / 'dataset.json').write_bytes(dataset_bytes) + (source / 'results.jsonl').write_bytes(results_bytes) + raise + return receipt diff --git a/harness/web_fetch/native_routes.py b/harness/web_fetch/native_routes.py new file mode 100644 index 000000000..fa304b3ce --- /dev/null +++ b/harness/web_fetch/native_routes.py @@ -0,0 +1,458 @@ +"""Declared native input and route contracts, independent of URLs and answers.""" +import json + +from lexbench_webfetch.answer_projection import ( + SCORE_ANSWER_FIELDS, require_native_projection_schema, require_native_answer_projection, +) + +INPUT_MEDIA_KINDS = frozenset({ + 'html_page', 'direct_pdf', 'direct_docx', 'direct_text', + 'direct_image', 'direct_media', +}) + + +def declared_input_media_kind(case, *, required=True): + """Read a dataset declaration or its worker projection; never inspect a URL. + + Dataset declarations live in ``source_benchmark.input_media_kind``. Workers + receive the same value as ``input_media_kind`` in the JSONL request. + """ + get = case.get if isinstance(case, dict) else lambda key, default=None: getattr(case, key, default) + source = get('source_benchmark', {}) or {} + if not isinstance(source, dict): + raise ValueError('Native input media declaration must be an object') + nested, direct = source.get('input_media_kind'), get('input_media_kind') + if nested is not None and direct is not None and nested != direct: + raise ValueError('Conflicting native input media declarations') + value = direct if direct is not None else nested + if value is None and not required: + return None + if not isinstance(value, str) or value not in INPUT_MEDIA_KINDS: + raise ValueError('Native calls require a declared input_media_kind') + return value + + +# Native parameter values come from the immutable provider configuration and its +# pinned implementation, not a case URL, expected text, or provider outcome. +def _digest(value): + import hashlib + import json + return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True, + separators=(',', ':')).encode()).hexdigest() + + +def native_route_cell(provider, case): + media = declared_input_media_kind(case) + kind, track = case.task_kind, case.track + key = (provider.name, track, kind, media) + config, method = provider.config, provider.type + endpoint = None + if method == 'moli_webfetch': + transport = 'cli' + interface = ('raw_file' if kind == 'file' else 'native_resource_html' + if kind == 'resource' else 'native_document_bytes' + if track == 'document' and media in {'direct_pdf', 'direct_docx'} + else 'dynamic_native_markdown' if track == 'dynamic' else 'ordinary_markdown') + fields = (['original_bytes'] if interface in {'raw_file', 'native_document_bytes'} + else ['stdout', 'diagnostic.main_document_url']) + elif method in {'firecrawl', 'tavily_extract', 'exa_contents', 'cloudflare_markdown', 'cloudflare_content'}: + transport = 'http' + if method == 'firecrawl': + interface = '/v2/scrape' + endpoint = str(config.get('base_url', 'https://api.firecrawl.dev')).rstrip('/') + interface + fields = ['data.rawBase64'] if kind == 'file' else ['data.markdown', 'data.links', 'data.images', 'data.videos', 'data.video'] + fields += ['data.metadata.title', 'data.metadata.url'] + elif method == 'tavily_extract': + interface = '/extract' + endpoint = str(config.get('base_url', 'https://api.tavily.com')).rstrip('/') + interface + fields = ['results[].raw_content', 'results[].images', 'images', 'results[].url'] + elif method == 'exa_contents': + interface = '/contents' + endpoint = str(config.get('base_url', 'https://api.exa.ai')).rstrip('/') + interface + fields = ['results[].text', 'results[].extras', 'results[].title', 'results[].url'] + else: + interface = provider.native_http_interface(case) + endpoint = 'https://api.cloudflare.com/client/v4/accounts/' + str(config.get('account_id', '')) + '/browser-rendering' + interface + if method=='cloudflare_markdown' and config.get('browser'): + from urllib.parse import urlencode + endpoint += '?'+urlencode({'browser': config['browser']}) + fields = ['result', 'meta.title', 'meta.finalUrl'] + elif method in {'subprocess_jsonl', 'cowagent_builtin'}: + transport, interface = 'jsonl', 'native_worker_command' + fields = ['content', 'metadata.links', 'metadata.images', 'metadata.videos', 'metadata.media', 'metadata.extras', 'files', 'title', 'final_url'] + elif method == 'fixture': + transport, interface, fields = 'fixture', 'synthetic_fixture', ['content', 'title', 'final_url'] + else: + raise ValueError('Unsupported fixed-native interface: ' + method) + options = ('max_bytes', 'max_age', 'max_age_hours', 'only_main_content', 'include_images', + 'extract_depth', 'text_verbosity', 'include_resource_links', 'rich_links_limit', + 'wait_until', 'block_private_networks', 'dns_mode', 'log_level', 'formats') + cell = {'method': key[0], 'provider_type': method, 'track': track, 'task_kind': kind, 'input_media_kind': media, + 'transport': transport, 'interface': interface, 'answer_fields': fields, + 'configured_parameters': {k: config[k] for k in options if k in config}, + 'unspecified_parameters': 'native defaults from the bound implementation', + **({'command_sha256': _digest(config.get('command'))} if transport == 'jsonl' else {}), + 'provider_configuration_sha256': _digest(config), + 'endpoint_sha256': _digest(endpoint) if endpoint is not None else None} + return key, cell + + +def _request_parameters(provider, case, cell, timeout_seconds): + """Explicit pre-call transport contract, separately checked against adapters.""" + if cell['transport'] == 'http': + endpoint, arguments = provider.native_http_arguments(case, timeout_seconds) + if _digest(endpoint) != cell['endpoint_sha256']: + raise ValueError('Native request builder differs from its declared endpoint') + return normalized_http_parameters(case, arguments) + if cell['transport'] == 'cli': + command = provider.native_cli_command(case) + return {'argv': ['', *command[1:-1], ''], + 'binary_sha256': provider.config.get('expected_binary_sha256')} + if cell['transport'] == 'jsonl': + request = provider.native_worker_request(case, timeout_seconds, + '' if provider.config.get('log_level') == 'debug' else None) + request['url'] = '' + command = [provider._resolve_token(part) for part in provider.config.get('command', [])] + return {'request': request, 'command_sha256': _digest(command)} + return {} + + +def normalized_http_parameters(case, arguments): + from copy import deepcopy + body = deepcopy(arguments.get('json_body')) + if not isinstance(body, dict): + raise ValueError('Native HTTP request lacks its declared input field') + slots = [key for key in ('url', 'urls') if key in body] + if slots == ['url'] and body['url'] == case.url: + body['url'] = '' + elif slots == ['urls'] and body['urls'] == [case.url]: + body['urls'] = [''] + else: + raise ValueError('Native HTTP input differs from the frozen task URL') + return {'method': arguments.get('method', 'GET'), 'body': body, + 'headers_sha256': _digest(arguments.get('headers', {})), + 'timeout_seconds': arguments.get('timeout'), 'max_bytes': arguments.get('max_bytes')} + + + +class NativeRouteLedger: + """Freeze class-level intent before calls; reject observed per-case drift. + + Native defaults are bound by configuration and implementation identities. + Interfaces and normalized transport parameters are frozen for every cell + before the cohort. Actual arguments are checked before transport. No + response, URL pattern, or expected answer selects a route. + """ + def __init__(self, cases, providers, *, timeout_seconds=45): + from datetime import datetime, timezone + import threading + self.cells, self.observed, self.calls, self.errors = {}, {}, {}, [] + self.physical_attempts = {} + self.lock = threading.Lock() + self.providers = {p.name: p for p in providers if p.config.get('uniform_native_parameters')} + self.inventory = [] + for case in cases: + media = declared_input_media_kind(case) if self.providers else None + if self.providers: + self.inventory.append({'case_id': case.id, 'track': case.track, + 'task_kind': case.task_kind, 'input_media_kind': media, + 'input_url_sha256': _digest(case.url)}) + for provider in self.providers.values(): + key, cell = native_route_cell(provider, case) + if case.track == 'dynamic': + from .retrieval_hints import native_default_case + if not native_default_case(case): + raise ValueError('Native dynamic inputs require a stable acceptance contract') + cell['request_parameters'] = _request_parameters(provider, case, cell, timeout_seconds) + if key in self.cells and self.cells[key] != cell: + raise ValueError('Native route intent differs within its declared class') + self.cells[key] = cell + self.document = {'schema_version': 1, 'kind': 'declared_native_route_contract', + 'frozen_at': datetime.now(timezone.utc).isoformat(), + 'pre_call_frozen': True, + 'parameter_policy': 'Interfaces, adapter request parameters and configured options frozen before the cohort; actual transport arguments must match. Unspecified product internals retain pinned native defaults.', + 'cases': self.inventory, 'cells': list(self.cells.values())} + self.binding = None + + def write(self, source, *, dataset_sha256, runtime_source_sha256): + import hashlib + import json + self.document.update(dataset_sha256=dataset_sha256, runtime_source_sha256=runtime_source_sha256) + raw = (json.dumps(self.document, ensure_ascii=False, indent=2)+'\n').encode() + path = source/'native-route-contract.json' + if path.exists(): + raise ValueError('Native route contract already exists; do not overwrite a run') + path.write_bytes(raw) + self.binding = {'path': path.name, 'sha256': hashlib.sha256(raw).hexdigest()} + for provider in self.providers.values(): + provider._native_route_ledger = self + return self.binding + + def reject(self, provider, case, reason): + with self.lock: + self.errors.append({'method': provider.name, 'case_id': case.id, 'reason': reason}) + raise ValueError(reason) + + def begin_attempt(self, provider, case, physical_attempt): + """Give each caller-controlled physical call its own invocation receipt.""" + identity = (provider.name, case.id) + with self.lock: + previous = self.physical_attempts.get(identity, 0) + if type(physical_attempt) is not int or physical_attempt != previous + 1: + raise ValueError('Native physical attempts must be consecutive') + self.physical_attempts[identity] = physical_attempt + + def observe(self, provider, case, *, transport, parameters, endpoint=None, interface=None): + key, current = native_route_cell(provider, case) + cell = self.cells.get(key, {}) + with self.lock: + message = 'Native route protocol was already violated' if self.errors else None + if {k:v for k,v in cell.items() if k!='request_parameters'} != current: + message = 'Native route configuration changed after the pre-call freeze' + elif transport != cell['transport']: + message = 'Native transport differs from its frozen route' + elif endpoint is not None and _digest(endpoint) != cell['endpoint_sha256']: + message = 'Native endpoint differs from its frozen route' + elif interface is not None and interface != cell['interface']: + message = 'Native command differs from its frozen route' + signature = {'transport': transport, 'interface': cell['interface'], + 'endpoint_sha256': cell['endpoint_sha256'], 'parameters': parameters} + digest = _digest(signature) + if not message and parameters != cell.get('request_parameters'): + message = 'Native request parameters differ from the pre-call contract' + if not message and key in self.observed and self.observed[key] != digest: + message = 'Native request signature changed within the same declared class' + identity = (provider.name, case.id, self.physical_attempts.get((provider.name, case.id), 1)) + if not message and self.calls.get(identity): + message = 'Fixed-native calls permit only one adapter invocation per physical attempt' + if message: + self.errors.append({'method': provider.name, 'case_id': case.id, 'reason': message}) + raise ValueError(message) + self.observed[key] = digest + self.calls.setdefault(identity, []).append({'signature': signature, 'sha256': digest}) + + def receipt(self, provider, case, *, ok=None): + """Transport started means entry to the outbound call, not server receipt.""" + from copy import deepcopy + key, _ = native_route_cell(provider, case) + with self.lock: + ordinal = self.physical_attempts.get((provider.name, case.id), 1) + calls = self.calls.get((provider.name, case.id, ordinal), []) + return {'contract_sha256': self.binding['sha256'], 'cell': list(key), + 'physical_attempt': ordinal, + 'answer_fields': list(self.cells[key]['answer_fields']), + 'requests': deepcopy(calls), + 'transport_state': 'started' if calls else 'not_started', + **({'failure_stage': 'after_transport' if calls + else 'before_transport'} if ok is False else {}), + 'protocol_errors': [dict(e) for e in self.errors if e['method']==provider.name and e['case_id']==case.id]} + + + +def verify_native_route_contract(source, manifest, dataset, records): + """Shared execution/release check for newly declared native cohorts only.""" + import hashlib + import json + from datetime import datetime + from lexbench_webfetch.paths import contained_path + if manifest.get('native_route_errors'): + raise ValueError('Native route protocol was violated') + binding = manifest.get('native_route_contract') + if not binding: + if manifest.get('native_routes_required'): + raise ValueError('Fixed-native cohort lacks its pre-call route contract') + return None + path = contained_path(source, binding['path'], reject_symlinks=True) + raw = path.read_bytes() + if hashlib.sha256(raw).hexdigest() != binding.get('sha256'): + raise ValueError('Native route contract hash changed') + doc = json.loads(raw) + if (doc.get('kind') != 'declared_native_route_contract' or doc.get('pre_call_frozen') is not True + or doc.get('dataset_sha256') != manifest.get('dataset_sha256') + or doc.get('runtime_source_sha256') != manifest.get('runtime_source_sha256')): + raise ValueError('Native route contract differs from the frozen execution') + cells = {(c['method'], c['track'], c['task_kind'], c['input_media_kind']): c for c in doc['cells']} + cases = {c['id']: c for c in dataset['cases']} + inventory = {c['case_id']: c for c in doc['cases']} + if len(cells) != len(doc['cells']) or len(inventory) != len(doc['cases']): + raise ValueError('Native route contract has duplicate declarations') + names = {key[0] for key in cells} + required = {p['name'] for p in manifest.get('providers', []) if p.get('fixed_native_parameters')} + projection = manifest.get('projection', {}).get('kind') + allowed_projection = projection in {'complete-provider-case-subset-from-retained-run', + 'complete-provider-from-retained-incomplete-matrix', 'completed-case-from-live-preflight'} + if (not required or not required <= names or (required != names and not allowed_projection)): + raise ValueError('Native route method scope differs from the frozen execution') + names = required + selected_cases = ({row['case_id'] for row in records if row['provider'] in names} + if allowed_projection else set(inventory)) + if not selected_cases <= set(inventory): + raise ValueError('Native route projection changes its frozen input scope') + expected = {(name, identifier) for name in names for identifier in selected_cases} + actual, signatures = set(), {} + for row in records: + if row['provider'] not in names: + continue + identity = row['provider'], row['case_id'] + if identity in actual or identity not in expected: + raise ValueError('Native route result inventory differs') + actual.add(identity) + case = cases[row['case_id']] + media = declared_input_media_kind(case) + if (_digest(case['url']) != inventory[row['case_id']]['input_url_sha256'] + or inventory[row['case_id']] != {'case_id': row['case_id'], 'track': case.get('track', 'web'), + 'task_kind': case.get('task_kind', 'body'), 'input_media_kind': media, + 'input_url_sha256': _digest(row['url'])} + or datetime.fromisoformat(doc['frozen_at']) > datetime.fromisoformat(row['started_at'])): + raise ValueError('Native input type or pre-call freeze differs') + key = row['provider'], case.get('track', 'web'), case.get('task_kind', 'body'), media + metadata = row.get('provider_metadata', {}) + final_receipt = metadata.get('native_route_receipt', {}) + attempts = metadata.get('retry', {}).get('attempts', []) + if attempts and attempts[-1].get('native_route_receipt') != final_receipt: + raise ValueError('Native final route receipt differs from its last physical call') + for ordinal, physical in enumerate(attempts or [row], 1): + receipt = (physical.get('native_route_receipt', {}) if attempts else final_receipt) + if attempts and receipt.get('physical_attempt') != ordinal: + raise ValueError('Native retry lacks its consecutive physical route receipt') + if receipt.get('protocol_errors'): + raise ValueError('Native request violated its route contract') + if receipt.get('contract_sha256') != binding['sha256'] or receipt.get('cell') != list(key) or key not in cells: + raise ValueError('Native result lacks its bound route receipt') + if receipt.get('answer_fields') != cells[key]['answer_fields']: + raise ValueError('Native answer fields differ from the frozen route') + requests = receipt.get('requests', []) + state = receipt.get('transport_state') + if state not in {'started', 'not_started'}: + raise ValueError('Native result lacks its transport stage') + if bool(requests) != (state == 'started'): + raise ValueError('Native transport stage lacks its invocation signature') + if not physical['ok'] and receipt.get('failure_stage') != ( + 'after_transport' if requests else 'before_transport'): + raise ValueError('Native failed result lacks an explicit failure stage') + if len(requests) > 1 or (physical['ok'] and not requests and cells[key]['transport'] != 'fixture'): + raise ValueError('Native result lacks one complete invocation signature') + for item in requests: + signature = item['signature'] + if (_digest(signature) != item.get('sha256') + or signature['transport'] != cells[key]['transport'] + or signature['interface'] != cells[key]['interface'] + or signature['endpoint_sha256'] != cells[key]['endpoint_sha256'] + or signature.get('parameters') != cells[key]['request_parameters']): + raise ValueError('Native invocation differs from its frozen interface') + if key in signatures and signatures[key] != item['sha256']: + raise ValueError('Native request signature differs within its declared class') + signatures[key] = item['sha256'] + if actual != expected: + raise ValueError('Native route contract has an incomplete result scope') + return doc + + +def verify_native_execution_manifests(source, manifests, dataset, records): + """Check native contracts at every existing aggregate/report boundary.""" + checked = [] + for manifest in manifests: + if not manifest.get('native_routes_required') and not manifest.get('native_route_contract'): + continue + if source is None or dataset is None or records is None: + raise ValueError('Native route verification requires retained inputs and complete records') + identifiers = manifest.get('attempt_ids', []) + if len(identifiers) != len(set(identifiers)): + raise ValueError('Native execution repeats attempt identities') + identifier_set = set(identifiers) + selected = [row for row in records if row['attempt_id'] in identifier_set] + if {r['attempt_id'] for r in selected} != set(identifiers): + raise ValueError('Native execution omits retained attempt evidence') + checked.append(verify_native_route_contract(source, manifest, dataset, selected)) + return checked + + +def materialize_native_route_contracts(output, source, manifests): + """Copy hash-bound frozen contracts, deduplicating multiple cohort roots.""" + import hashlib + import re + import shutil + from lexbench_webfetch.paths import contained_path + for manifest in manifests: + binding = manifest.get('native_route_contract') + if not binding: + if manifest.get('native_routes_required'): + raise ValueError('Native cohort is missing its frozen route contract') + continue + digest = binding.get('sha256', '') + if not isinstance(digest, str) or re.fullmatch('[0-9a-f]{64}', digest) is None: + raise ValueError('Invalid native route contract digest') + relative = binding.get('path') + if relative not in {'native-route-contract.json', 'native-route-contracts/'+digest+'.json'}: + raise ValueError('Invalid native route contract path') + path = contained_path(source, relative, reject_symlinks=True) + if not path.is_file() or hashlib.sha256(path.read_bytes()).hexdigest() != digest: + raise ValueError('Native route contract bytes changed') + target_relative = 'native-route-contracts/'+digest+'.json' + target = contained_path(output, target_relative, reject_symlinks=True) + target.parent.mkdir(parents=True, exist_ok=True) + if target.exists(): + if not target.is_file() or hashlib.sha256(target.read_bytes()).hexdigest() != digest: + raise ValueError('Conflicting native route contract bytes') + else: + shutil.copy2(path, target) + if hashlib.sha256(target.read_bytes()).hexdigest() != digest: + raise ValueError('Copied native route contract bytes changed') + manifest['native_route_contract'] = {'path': target_relative, 'sha256': digest} + + +def verify_saved_native_answer(root, record): + """Validate newly declared saved answers at aggregation/release boundaries. + + Older calls with no declaration need a separately recorded historical audit; + their missing acquisition declarations must never be invented in place. + """ + from .publication_privacy import read_evidence_binding + from lexbench_webfetch.paths import contained_path + metadata = record.get('provider_metadata', {}) + declaration = metadata.get('native_answer_projection') + if not declaration: + if metadata.get('native_answer_required'): + raise ValueError('Scored native answer has no projection declaration') + return + require_native_projection_schema(declaration) + answer = read_evidence_binding(root, {'path': record['artifact_path'], + 'sha256': record['metrics']['content_sha256']}).read_bytes() + # Deadlines discard answers while preserving late raw evidence. + if not record['ok'] and record.get('error_type') == 'timeout' and not answer: + if any(metadata.get(key) for key in SCORE_ANSWER_FIELDS) or metadata.get('file_evaluation', {}).get('files'): + raise ValueError('Discarded late native answer retains score-bearing fields') + return + raw_path = metadata.get('raw_stdout_path') if declaration.get('source_format') == 'bytes' else metadata.get('raw_response_path') + raw_hash = metadata.get('raw_stdout_sha256') if declaration.get('source_format') == 'bytes' else metadata.get('raw_response_sha256') + if not raw_path or not raw_hash: + raise ValueError('Declared native answer lacks its retained raw response') + raw = read_evidence_binding(root, {'path': raw_path, 'sha256': raw_hash}).read_bytes() + files = [{'data': contained_path(root, item['artifact_path'], reject_symlinks=True).read_bytes()} + for item in metadata.get('file_evaluation', {}).get('files', [])] + diagnostic = None + if any(spec.get('format') == 'fetch_log_final_url' + for spec in declaration.get('result_fields', {}).values()): + diagnostic = read_evidence_binding(root, {'path': metadata['diagnostic_log_path'], + 'sha256': metadata['diagnostic_log_sha256']}).read_bytes() + require_native_answer_projection(answer.decode('utf-8'), metadata, raw, files=files, + title=record.get('title') if declaration.get('schema_version') == 2 else None, + final_url=record.get('final_url') if declaration.get('schema_version') == 2 else None, + request_url=record.get('url'), + diagnostic=diagnostic, transport_receipt=metadata.get('source_transport_receipt')) + if files or 'file_evaluation' in metadata: + from lexbench_webfetch.file_acceptance import evaluate_files + source_verification = {} + dataset_path = root/'dataset.json' + if dataset_path.is_file(): + cases = json.loads(dataset_path.read_bytes()).get('cases', []) + case = next((case for case in cases if case.get('id') == record.get('case_id')), None) + if case is None: + raise ValueError('Native file result lacks its frozen case') + source_verification = case.get('source_verification') or {} + expected = evaluate_files({'source_verification': source_verification}, [item['data'] for item in files]) + saved = metadata.get('file_evaluation', {}) + compared = {**saved, 'files': [{k: v for k, v in item.items() if k != 'artifact_path'} + for item in saved.get('files', [])]} + if compared != expected: + raise ValueError('Native file evaluation differs from its retained bytes') diff --git a/harness/web_fetch/network_preflight.py b/harness/web_fetch/network_preflight.py index ec9206672..b2de69826 100644 --- a/harness/web_fetch/network_preflight.py +++ b/harness/web_fetch/network_preflight.py @@ -1,9 +1,11 @@ -"""Reject synthetic DNS addresses before a local/remote comparison spends calls.""" +"""Reject non-public DNS addresses before a public local/remote comparison spends calls.""" import ipaddress import socket from concurrent.futures import ThreadPoolExecutor from urllib.parse import urlsplit +GUARDED_NATIVE_TYPES = {"openclaw_builtin", "deepseek_harness_builtin", "cowagent_builtin", "langchain_webbase", "moli_webfetch", "crawl4ai"} + _FAKE_IP_RANGE = ipaddress.ip_network("198.18.0.0/15") @@ -53,6 +55,16 @@ def require_usable_public_dns(audit, *, allow_partial_resolution_failures=False) raise ValueError("DNS resolution failed for " + scope + ": " + ", ".join(unresolved)) + # Recompute from retained addresses as well, so older preflight receipts + # without the diagnostic field cannot conceal a private-address answer. + non_public = [row["host"] for row in audit["hosts"] + if any(not ipaddress.ip_address(address).is_global + for address in row.get("addresses", []))] + if non_public: + raise ValueError("System DNS returned non-public addresses for: " + ", ".join(non_public) + + "; use a real-public-address DNS environment before comparison. Do not disable product safety guards.") + + def preflight_guarded_providers(providers, cases): """Require one declared DNS environment for every local comparison method.""" from .network_dns import lookup_public_dns, validate_dns_mode diff --git a/harness/web_fetch/providers/base.py b/harness/web_fetch/providers/base.py index a78a6f5b8..fa08b778e 100644 --- a/harness/web_fetch/providers/base.py +++ b/harness/web_fetch/providers/base.py @@ -83,26 +83,46 @@ def __init__(self, name: str, config: dict[str, Any]): self.type = str(config.get("type", name)) def check_native_call(self, case: EvalCase) -> None: - """Allow only shared, answer-free capabilities on fixed native calls.""" + """Permit native task outputs, never case-specific retrieval assistance.""" if not self.config.get("uniform_native_parameters"): return - if case.retrieval_hints: - if ( - case.track != "dynamic" - or case.retrieval_hints != {"version": 3, "mode": "native"} - ): - raise ValueError("fixed native calls prohibit per-case retrieval hints") - supported = { - "web": {"body", "structure"}, - "dynamic": {"body", "structure"}, - "challenge": {"body", "structure"}, - "risk_control": {"body", "structure"}, - "document": {"body", "file"}, - "image": {"body", "resource"}, - "video": {"body", "resource"}, - } - if case.task_kind not in supported.get(case.track, set()): - raise ValueError("fixed native calls do not support this declared task route") + if case.retrieval_hints and not (case.track == "dynamic" and case.retrieval_hints == {"version": 3, "mode": "native"}): + raise ValueError("fixed native calls prohibit per-case retrieval hints") + if case.track not in {"web", "dynamic", "challenge", "risk_control", "document", "image", "video", "file"}: + raise ValueError("fixed native calls require a declared task route") + if case.track == "document" and case.task_kind == "resource": + raise ValueError("fixed native calls require a declared task route") + if case.task_kind not in {"body", "structure", "resource", "file"}: + raise ValueError("fixed native calls require a declared native output kind") + + def capture_native_request(self, case, *, transport, parameters, endpoint=None, interface=None): + ledger = getattr(self, '_native_route_ledger', None) + if ledger is not None: + ledger.observe(self, case, transport=transport, parameters=parameters, + endpoint=endpoint, interface=interface) + + def native_http_request(self, case, callback, endpoint, **kwargs): + """Observe exact transport arguments before a fixed-native HTTP call.""" + ledger = getattr(self, '_native_route_ledger', None) + if self.config.get('uniform_native_parameters'): + native_endpoint, native_arguments = self.native_http_arguments(case, kwargs.get('timeout')) + if endpoint != native_endpoint or kwargs != native_arguments: + if ledger is not None: + ledger.reject(self, case, 'Native adapter arguments differ from its canonical request builder') + raise ValueError('Native adapter arguments differ from its canonical request builder') + endpoint, kwargs = native_endpoint, native_arguments + if ledger is not None: + from ..native_routes import normalized_http_parameters + self.capture_native_request(case, transport='http', endpoint=endpoint, + parameters=normalized_http_parameters(case, kwargs)) + return callback(endpoint, **kwargs) + + def native_http_arguments(self, case, timeout): + raise ValueError('Provider has no fixed-native HTTP request builder') + + def native_http_options(self, body, headers, timeout): + return {'method': 'POST', 'json_body': body, 'headers': headers, + 'timeout': timeout, 'max_bytes': int(self.config.get('max_bytes', 20_000_000))} @property def min_interval_seconds(self) -> float: diff --git a/harness/web_fetch/providers/cloudflare.py b/harness/web_fetch/providers/cloudflare.py index d0fb0e921..ae36ba82f 100644 --- a/harness/web_fetch/providers/cloudflare.py +++ b/harness/web_fetch/providers/cloudflare.py @@ -8,7 +8,7 @@ import urllib.request from pathlib import Path from pathlib import PurePosixPath -from urllib.parse import urlencode, urljoin, urlsplit +from urllib.parse import urlencode, urlsplit from lexbench_webfetch.models import EvalCase from harness.web_fetch.models import ProviderResult @@ -27,22 +27,6 @@ ".xls", ".xlsx", ) -_MEDIA_EXTENSIONS = (".mp4", ".webm", ".mov", ".m4v", ".mp3", ".wav", ".ogg") -_VIDEO_ARCHIVE_EXTENSIONS = (".mp4.zip", ".webm.zip", ".mov.zip", ".m4v.zip") - - -def _video_page_href(path: str) -> bool: - """Classify a page link by its URL route, not by the expected answer. - - A video-link task needs detail-page hrefs, but ordinary site navigation - must not be reported as a video merely because it was in ``a[href]``. - This is intentionally a conservative candidate filter; it does not claim - to discover every provider's opaque video URL scheme. - """ - return any(segment in {"video", "videos", "watch"} - for segment in path.lower().split("/") if segment) - - def _uses_direct_document_route(case: EvalCase) -> bool: """Choose file conversion only when the supplied URL itself names a file. @@ -203,65 +187,6 @@ def _markdown_conversion_content(payload: dict[str, object]) -> str: return result if isinstance(result, str) else "" -def _resource_urls( - payload: dict[str, object], - *, - base_url: str, - track: str, - include_video_page_links: bool = False, -) -> list[str]: - result = payload.get("result") - if not isinstance(result, list): - return [] - raw_values: list[tuple[str, str]] = [] - for group in result: - if not isinstance(group, dict) or not isinstance(group.get("results"), list): - continue - if include_video_page_links and group.get("selector") != "a[href]": - # A declared detail-link task does not ask for player/audio/iframe - # sources. Those can be ordinary utility embeds on a homepage. - continue - for element in group["results"]: - if not isinstance(element, dict) or not isinstance(element.get("attributes"), list): - continue - for attribute in element["attributes"]: - if not isinstance(attribute, dict): - continue - name, value = attribute.get("name"), attribute.get("value") - if isinstance(name, str) and isinstance(value, str): - folded = name.casefold() - if folded in {"src", "srcset", "href", "data-src", "data-original"}: - raw_values.append((folded, value)) - urls: set[str] = set() - for name, value in raw_values: - candidates = [value] - if name == "srcset": - candidates = [part.strip().split()[0] for part in value.split(",") if part.strip()] - for candidate in candidates: - try: - absolute = urljoin(base_url, candidate.strip()) - parsed = urlsplit(absolute) - except ValueError: - continue - if parsed.scheme not in {"http", "https"} or not parsed.hostname: - continue - # A downloadable video may be packaged as a ZIP (for example, - # ``movie.mp4.zip``). Keep those media archives without admitting - # unrelated page/navigation links or arbitrary ZIP files. - if name == "href" and track == "video": - media_file = parsed.path.lower().endswith( - _MEDIA_EXTENSIONS + _VIDEO_ARCHIVE_EXTENSIONS - ) - video_page = (include_video_page_links - and _video_page_href(parsed.path)) - if not (media_file or video_page): - continue - if name == "href" and track == "image": - continue - urls.add(absolute) - return sorted(urls) - - def _origin_status(value: object) -> int | None: """Normalize Cloudflare's target-page status without accepting booleans.""" if isinstance(value, bool): @@ -291,6 +216,22 @@ def _interface_error(*, response_status: int, class CloudflareMarkdownProvider(Provider): + def native_http_interface(self, case): + return ('/content' if self.type == 'cloudflare_content' else '/scrape' + if case.task_kind == 'resource' and case.track in {'image', 'video'} else '/markdown') + + def native_http_arguments(self, case, timeout): + interface = self.native_http_interface(case) + body = {'url': case.url, 'gotoOptions': {'waitUntil': self.config.get('wait_until', 'networkidle2')}} + if interface == '/scrape': + selectors = (['img', 'picture source'] if case.track == 'image' else + ['video', 'video source', 'audio', 'audio source', 'iframe', 'a[href]']) + body['elements'] = [{'selector': value} for value in selectors] + endpoint = 'https://api.cloudflare.com/client/v4/accounts/'+str(self.config.get('account_id', ''))+'/browser-rendering'+interface + if interface != '/content' and self.config.get('browser'): + endpoint += '?' + urlencode({'browser': self.config['browser']}) + return endpoint, self.native_http_options(body, {'Authorization': 'Bearer '+str(self.config.get('api_token', ''))}, timeout) + def validate(self) -> list[str]: return [name for name in ("api_token", "account_id") if not self.config.get(name)] @@ -330,7 +271,7 @@ def _fetch_payload(self, case: EvalCase, timeout: float, request_body: dict) -> account_id=account_id, http_timeout=http_timeout, ) - if _uses_direct_document_route(case) and self.config.get("browser") != "kitesurf": + if not self.config.get("uniform_native_parameters") and self.config.get("browser") != "kitesurf" and _uses_direct_document_route(case): return self._fetch_direct_document( case, account_id=account_id, @@ -355,7 +296,7 @@ def _fetch_payload(self, case: EvalCase, timeout: float, request_body: dict) -> wait_metadata = None plan = None native_contract = None - if case.track == "dynamic" and not native_default_case(case) and "url" in request_body: + if case.track == "dynamic" and not self.config.get("uniform_native_parameters") and not native_default_case(case) and "url" in request_body: try: if not math.isfinite(timeout) or timeout <= 0: raise ValueError @@ -445,7 +386,7 @@ def _fetch_payload(self, case: EvalCase, timeout: float, request_body: dict) -> "support": "unsupported", "harness_content_assembly": False, } - response = request( + response = self.native_http_request(case, request, endpoint + ("?" + query if query else ""), method="POST", timeout=http_timeout, @@ -481,7 +422,7 @@ def _fetch_payload(self, case: EvalCase, timeout: float, request_body: dict) -> # Task scoring, not request classification, decides its usefulness. content=content if envelope_success else "", status_code=response.status, - final_url=meta.get("finalUrl"), + final_url=meta.get("finalUrl") or case.url, title=meta.get("title"), error_type=error_type, error_message=error_message, @@ -491,6 +432,11 @@ def _fetch_payload(self, case: EvalCase, timeout: float, request_body: dict) -> "_raw_response_b64": base64.b64encode(response.body).decode("ascii"), "raw_response_sha256": hashlib.sha256(response.body).hexdigest(), "raw_response_bytes": len(response.body), + **({'native_answer_projection': {'schema_version': 2, + 'content_pointer': ['result'], 'content_format': 'text' if envelope_success and isinstance(payload.get('result', ''), str) else 'empty', 'optional_content': True, + 'result_fields': {'title': {'pointer': ['meta', 'title']}, + 'final_url': {'pointer': ['meta', 'finalUrl'], 'fallback': 'request_url'}}}} + if (envelope_success and isinstance(payload.get('result', ''), str)) or meta else {}), "route": route, "wait": wait_metadata, "browser": self.config.get("browser"), @@ -517,17 +463,20 @@ def _fetch_media_resources( http_timeout: float, ) -> ProviderResult: service_timeout_ms = min(max(1, int(http_timeout * 1000)), 60_000) - include_video_page_links = case.track == "video" and any( + native = bool(self.config.get("uniform_native_parameters")) + include_video_page_links = case.track == "video" and (native or any( isinstance(resource, dict) and resource.get("kind") == "video-link" for resource in case.expected_resources - ) - selectors = ( - ["img", "picture source"] - if case.track == "image" - else (["a[href]"] if include_video_page_links else - ["video", "video source", "audio", "audio source", "iframe", "a[href]"]) - ) - response = request( + )) + if case.track == "image": + selectors = ["img", "picture source"] + elif include_video_page_links and not native: + selectors = ["a[href]"] + else: + # One native media request for every video-resource task; never + # choose selectors from expected URLs or reference answers. + selectors = ["video", "video source", "audio", "audio source", "iframe", "a[href]"] + response = self.native_http_request(case, request, f"https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/scrape" + ("?" + urlencode({"browser": self.config["browser"]}) if self.config.get("browser") else ""), method="POST", @@ -539,9 +488,9 @@ def _fetch_media_resources( "elements": [{"selector": selector} for selector in selectors], "gotoOptions": { "waitUntil": self.config.get("wait_until", "networkidle2"), - "timeout": service_timeout_ms, + **({} if self.config.get("uniform_native_parameters") else {"timeout": service_timeout_ms}), }, - "actionTimeout": service_timeout_ms, + **({} if self.config.get("uniform_native_parameters") else {"actionTimeout": service_timeout_ms}), }, ) try: @@ -551,15 +500,20 @@ def _fetch_media_resources( if not isinstance(payload, dict): payload = {} envelope_success = payload.get("success") is True - # The declared output *kind* is part of the public task contract, not - # answer/reference data. A video-link task needs page hrefs; playback - # resource tasks still exclude ordinary navigation links. - urls = _resource_urls( - payload, - base_url=case.url, - track=case.track, - include_video_page_links=include_video_page_links, - ) + from lexbench_webfetch.answer_projection import project_native_answer + answer_key = 'images' if case.track == 'image' else 'videos' + meta = payload.get('meta') if isinstance(payload.get('meta'), dict) else {} + effective_url = meta.get('finalUrl') or case.url + projection = {'schema_version': 2, 'content_pointer': ['result'], + 'content_format': 'json', 'result_fields': {'final_url': { + 'pointer': ['meta', 'finalUrl'], 'fallback': 'request_url'}}, + 'answer_fields': {answer_key: { + 'pointer': ['result'], 'format': 'html_attribute_urls', + 'base_url': effective_url}}} + answer, fields = ('', {}) + if envelope_success: + answer, fields = project_native_answer(payload, projection) + urls = fields.get(answer_key, []) error_type, error_message = _interface_error( response_status=response.status, envelope_success=envelope_success, @@ -570,9 +524,9 @@ def _fetch_media_resources( answer_key = "images" if case.track == "image" else "videos" return ProviderResult( ok=error_type is None, - content="\n".join(urls) if envelope_success else "", + content=answer, status_code=response.status, - final_url=case.url, + final_url=effective_url, error_type=error_type, error_message=error_message, metadata={ @@ -580,6 +534,9 @@ def _fetch_media_resources( "raw_response_sha256": hashlib.sha256(response.body).hexdigest(), "raw_response_bytes": len(response.body), "route": "browser_rendering_scrape_resources", + **({'native_answer_projection': (projection if envelope_success else { + **projection, 'content_format': 'empty', 'answer_fields': {}})} + if envelope_success or meta else {}), "browser": self.config.get("browser"), "endpoint_route": "/browser-rendering/scrape" + ("?" + urlencode({"browser": self.config["browser"]}) if self.config.get("browser") else ""), "adapter_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), @@ -755,12 +712,18 @@ def _fetch_direct_document( "_raw_response_b64": base64.b64encode(response.body).decode("ascii"), "raw_response_sha256": hashlib.sha256(response.body).hexdigest(), "raw_response_bytes": len(response.body), + **({'native_answer_projection': {'schema_version': 2, + 'content_pointer': ['result'], 'content_format': 'text', 'optional_content': True, + 'result_fields': {'final_url': {'format': 'transport_response_url'}}}} + if envelope_success and isinstance(payload.get('result', ''), str) else {}), "route": "workers_ai_tomarkdown", "source_status": source.status, "source_sha256": hashlib.sha256(source.body).hexdigest(), "source_bytes": len(source.body), "source_content_type": content_type, "source_final_url": source.url, + "source_transport_receipt": {"response_url": source.url, + "body_sha256": hashlib.sha256(source.body).hexdigest()}, "source_elapsed_ms": source_elapsed_ms, "conversion_elapsed_ms": conversion_elapsed_ms, "conversion_attempted": True, @@ -774,6 +737,9 @@ def _fetch_direct_document( class CloudflareContentProvider(Provider): + native_http_interface = CloudflareMarkdownProvider.native_http_interface + native_http_arguments = CloudflareMarkdownProvider.native_http_arguments + """Rendered HTML diagnostic. It is deliberately excluded from the readable-text leaderboard.""" def validate(self) -> list[str]: @@ -784,7 +750,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: account_id = str(self.config.get("account_id", "")) http_timeout = inner_timeout_budget(timeout, native_default=bool(self.config.get("uniform_native_parameters"))) service_timeout_ms = min(max(1, int(http_timeout * 1000)), 60_000) - response = request( + response = self.native_http_request(case, request, f"https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/content", method="POST", timeout=http_timeout, @@ -822,13 +788,17 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: ok=ok, content=content if envelope_success else "", status_code=response.status, - final_url=case.url, + final_url=meta.get("finalUrl") or case.url, error_type=error_type, error_message=error_message, metadata={ "_raw_response_b64": base64.b64encode(response.body).decode("ascii"), "raw_response_sha256": hashlib.sha256(response.body).hexdigest(), "raw_response_bytes": len(response.body), + **({'native_answer_projection': {'schema_version': 2, + 'content_pointer': ['result'], 'content_format': 'text' if envelope_success and isinstance(payload.get('result', ''), str) else 'empty', 'optional_content': True, + 'result_fields': {'final_url': {'pointer': ['meta', 'finalUrl'], 'fallback': 'request_url'}}}} + if (envelope_success and isinstance(payload.get('result', ''), str)) or meta else {}), "output_contract": "rendered_html", "upstream_status": upstream_status, "empty_content": envelope_success and not content, diff --git a/harness/web_fetch/providers/cowagent.py b/harness/web_fetch/providers/cowagent.py index f90af25f1..4dc22d5f5 100644 --- a/harness/web_fetch/providers/cowagent.py +++ b/harness/web_fetch/providers/cowagent.py @@ -8,20 +8,9 @@ import sys from pathlib import Path -from lexbench_webfetch.models import EvalCase -from harness.web_fetch.models import ProviderResult from .subprocess_jsonl import SubprocessJsonlProvider -def _sanitize_public_content(content: str) -> str: - """Remove CowAgent workspace destinations while retaining fetched text.""" - return re.sub( - r"\s*\|\s*Saved to:\s*(?:/Users/|/home/|/private/|/tmp/)[^\]\r\n]+", - "", - content, - ) - - class CowAgentBuiltinProvider(SubprocessJsonlProvider): """Invoke CowAgent's repository-local WebFetch implementation unchanged.""" @@ -42,12 +31,6 @@ def __init__(self, name, config): '{harness_root}/harness/web_fetch/workers/cowagent_worker.py'] super().__init__(name, derived) - def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: - result = super().fetch(case, timeout) - result.content = _sanitize_public_content(result.content) - if result.error_message: - result.error_message = _sanitize_public_content(result.error_message) - return result EXTRACTOR_FIXTURE = '''Fixture diff --git a/harness/web_fetch/providers/exa.py b/harness/web_fetch/providers/exa.py index fd1bf6816..67cadb95f 100644 --- a/harness/web_fetch/providers/exa.py +++ b/harness/web_fetch/providers/exa.py @@ -56,6 +56,17 @@ def _public_url_error(value: object) -> dict[str, object]: class ExaContentsProvider(Provider): + def native_http_arguments(self, case, timeout): + body = {'urls': [case.url], 'text': {'verbosity': self.config.get('text_verbosity', 'full')}, + 'maxAgeHours': int(self.config.get('max_age_hours', 0))} + extras = {'links': 20, 'imageLinks': 20} if self.config.get('include_resource_links') else {} + if self.config.get('rich_links_limit'): + extras['richLinks'] = int(self.config.get('rich_links_limit', 100)) + if extras: + body['extras'] = extras + return (str(self.config.get('base_url', 'https://api.exa.ai')).rstrip('/') + '/contents', + self.native_http_options(body, {'x-api-key': str(self.config.get('api_key', ''))}, timeout)) + def validate(self) -> list[str]: return [] if self.config.get("api_key") else ["api_key"] @@ -73,7 +84,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: # Exa's richLinks route returns the explicit anchor/URL pair needed # by the shared named-link answer contract. extras["richLinks"] = int(self.config.get("rich_links_limit", 100)) - response = request( + response = self.native_http_request(case, request, f"{str(self.config.get('base_url', 'https://api.exa.ai')).rstrip('/')}/contents", method="POST", timeout=http_timeout, @@ -133,7 +144,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: ok=ok, content=content, status_code=response.status, - final_url=(result.get("url") or case.url) if ok else case.url, + final_url=result.get("url") or case.url, title=result.get("title") if ok else None, error_type=error_type, error_message=error_message, @@ -157,5 +168,15 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: ), "extras": result_extras, "links": _rich_answer_links(result_extras), + **({'native_answer_projection': { + 'schema_version': 2, 'content_pointer': ['results', 0, 'text'], + 'content_format': 'text', 'optional_content': True, + 'result_fields': {'final_url': {'pointer': ['results', 0, 'url'], 'fallback': 'request_url'}, + **({'title': {'pointer': ['results', 0, 'title']}} if ok else {})}, + 'answer_fields': ({ + 'extras': {'pointer': ['results', 0, 'extras'], 'format': 'identity', 'default': {}}, + 'links': {'pointer': ['results', 0, 'extras', 'richLinks'], + 'format': 'named_links', 'default': []}} if ok else {})}} + if ok or content or result.get('url') else {}), }, ) diff --git a/harness/web_fetch/providers/firecrawl.py b/harness/web_fetch/providers/firecrawl.py index 6812a4489..e84be10e7 100644 --- a/harness/web_fetch/providers/firecrawl.py +++ b/harness/web_fetch/providers/firecrawl.py @@ -17,6 +17,20 @@ class FirecrawlProvider(Provider): _PROXY_MAX_CREDITS = {"basic": 1, "enhanced": 1, "auto": 1} _VIDEO_FORMAT_CREDITS = 5 + def native_http_arguments(self, case, timeout): + formats = ['rawBase64'] if case.task_kind == 'file' else list(self.config.get('formats', ['markdown'])) + if case.track == 'video' and case.task_kind == 'resource' and 'video' not in formats: + formats.append('video') + body = {'url': case.url, 'formats': formats, 'maxAge': int(self.config.get('max_age', 0)), + 'proxy': str(self.config.get('proxy', 'auto'))} + if 'only_main_content' in self.config: + body['onlyMainContent'] = self.config['only_main_content'] + if case.task_kind == 'file': + body['parsers'] = [] + headers = {'Authorization': 'Bearer '+str(self.config['api_key'])} if self.config.get('api_key') else {} + return (str(self.config.get('base_url', 'https://api.firecrawl.dev')).rstrip('/') + '/v2/scrape', + self.native_http_options(body, headers, timeout)) + def validate(self) -> list[str]: if "only_main_content" in self.config and not isinstance(self.config["only_main_content"], bool): return ["only_main_content must be boolean"] @@ -289,7 +303,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: "Firecrawl dynamic_wait_strategy must be actions or wait_for" ), ) - response = request( + response = self.native_http_request(case, request, f"{base_url}/v2/scrape", method="POST", timeout=http_timeout, @@ -384,6 +398,9 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: source_url = ( metadata.get("sourceURL") if isinstance(metadata.get("sourceURL"), str) else None ) + # sourceURL echoes the input; url records the observed destination. + final_url = metadata.get("url") + final_url = final_url if isinstance(final_url, str) and final_url else None content_type = ( metadata.get("contentType") if isinstance(metadata.get("contentType"), str) else None @@ -392,7 +409,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: ok=ok, content=content if ok and not file_route else "", status_code=response.status, - final_url=source_url or case.url, + final_url=final_url, title=metadata.get("title") if ok and isinstance(metadata.get("title"), str) else None, error_type=error_type, error_message=error_message, @@ -436,6 +453,21 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: "links": data.get("links") if ok and isinstance(data.get("links"), list) else [], "images": data.get("images") if ok and isinstance(data.get("images"), list) else [], "videos": videos if ok else [], + **({'native_answer_projection': { + 'schema_version': 2, + 'result_fields': { + 'final_url': {'pointer': ['data', 'metadata', 'url'], 'format': 'text_or_none', 'nonempty': True}, + **({'title': {'pointer': ['data', 'metadata', 'title'], 'format': 'text_or_none'}} if ok else {})}, + 'content_pointer': ['data', 'markdown'], + 'content_format': 'empty' if file_route or not ok else 'text', 'optional_content': True, + 'answer_fields': ({} if file_route or not ok else { + 'links': {'pointer': ['data', 'links'], 'format': 'identity', 'default': []}, + 'images': {'pointer': ['data', 'images'], 'format': 'identity', 'default': []}, + **({'videos': {'pointer': ['data', 'videos'], 'format': 'video_values', 'default': [], + 'legacy': {'pointer': ['data', 'video'], 'default': ''}}} + if native_video_route else {})}), + **({'files': {'pointer': ['data', 'rawBase64'], 'format': 'base64_file'}} + if file_bytes else {})}} if ok or final_url is not None else {}), **( { "raw_file": { diff --git a/harness/web_fetch/providers/fixture.py b/harness/web_fetch/providers/fixture.py index 36f61600c..8021e3014 100644 --- a/harness/web_fetch/providers/fixture.py +++ b/harness/web_fetch/providers/fixture.py @@ -1,5 +1,9 @@ from __future__ import annotations +import base64 +import hashlib +import json + from lexbench_webfetch.models import EvalCase from harness.web_fetch.models import ProviderResult from .base import Provider @@ -8,10 +12,20 @@ class FixtureProvider(Provider): """Deterministic provider used to verify a fresh installation offline.""" + def _result(self, **kwargs): + result = ProviderResult(**kwargs) + if self.config.get('uniform_native_parameters'): + raw = json.dumps({'content': result.content, 'title': result.title, 'final_url': result.final_url}).encode() + result.metadata.update(_raw_response_b64=base64.b64encode(raw).decode(), + raw_response_sha256=hashlib.sha256(raw).hexdigest(), raw_response_bytes=len(raw), + native_answer_projection={'schema_version': 2, 'content_pointer': ['content'], 'content_format': 'text', + 'result_fields': {'title': {'pointer': ['title']}, 'final_url': {'pointer': ['final_url']}}}) + return result + def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: fail_cases = set(self.config.get("fail_cases", [])) if case.id in fail_cases: - return ProviderResult( + return self._result( ok=False, content="Fixture failure: configured test failure.", status_code=502, @@ -20,7 +34,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: error_message="Configured fixture failure", ) markers = "\n".join(case.expected_any) - return ProviderResult( + return self._result( ok=True, content=f"# {case.name}\n\n{markers}\n\nFixture content for {case.url}", status_code=200, diff --git a/harness/web_fetch/providers/moli.py b/harness/web_fetch/providers/moli.py index 01ae598ca..f72c140bb 100644 --- a/harness/web_fetch/providers/moli.py +++ b/harness/web_fetch/providers/moli.py @@ -14,7 +14,9 @@ import threading import time from pathlib import Path -from urllib.parse import parse_qsl, quote, urlsplit +from urllib.parse import urlsplit + +from lexbench_webfetch.answer_projection import main_document_url_from_fetch_log as _diagnostic_main_document_url from lexbench_webfetch.models import EvalCase, TRACKS from harness.web_fetch.models import ProviderResult @@ -138,76 +140,27 @@ def _native_fetch_diagnostics(stderr: bytes, requested_url: str) -> tuple[int | return status, any(row.group("url") == requested for row in waits) -def _diagnostic_main_document_url(stderr: str | bytes, requested_url: str) -> str | None: - """Read Moli's effective main-document URL from the same call's debug log.""" - pattern = re.compile( - rb"fetch request complete method=(?:GET|POST) url=(?P\S+) " - rb"final_url=(?P\S+) status=(?P\d{3})\b" - ) - - diagnostic = stderr.encode("utf-8") if isinstance(stderr, str) else stderr - navigation_pattern = re.compile( - rb"following pending location navigation asynchronously[^\n]* url=(?P\S+)" - ) - def field(row: re.Match[bytes], name: str) -> str: - return row.group(name).decode("utf-8", "replace") - def identity(value: str) -> tuple | None: - try: - validate_http_url(value) - parts = urlsplit(value) - return ( - parts.scheme.casefold(), - (parts.hostname or "").casefold(), - parts.port if parts.port is not None else (443 if parts.scheme.casefold() == "https" else 80), - re.sub(r"%[0-9a-fA-F]{2}", lambda match: match[0].upper(), - quote(parts.path, safe="/%:@!$&'()*+,;=")).rstrip("/") or "/", - sorted(parse_qsl(parts.query, keep_blank_values=True)), - ) - except ValueError: - return None - - requested_identity = identity(requested_url) - if requested_identity is None: - return None - rows = list(pattern.finditer(diagnostic)) - - # Legacy completion records do not identify redirect hops or request owners. - # Never infer the document from GET ordering or challenge query parameters. - row = next((candidate for candidate in rows - if identity(field(candidate, "url")) == requested_identity), None) - if row is None or 300 <= int(field(row, "status")) < 400: - return None - navigations = list(navigation_pattern.finditer(diagnostic)) - for index, navigation in enumerate(navigations): - boundary = navigations[index + 1].start() if index + 1 < len(navigations) else len(diagnostic) - commits = [diagnostic.find(marker, navigation.end(), boundary) for marker in ( - b"committed main navigation before replacement realm bootstrap", - b"detached committed main WindowProxy for replacement context", - )] - committed = next((position for position in commits if position >= 0), -1) - if committed < 0: - continue - navigation_identity = identity(field(navigation, "url")) - if navigation_identity is None: - return None - matches = [candidate for candidate in rows - if navigation.end() <= candidate.start() < boundary - and identity(field(candidate, "url")) == navigation_identity] - if len(matches) != 1 or 300 <= int(field(matches[0], "status")) < 400: - return None - row = matches[0] - candidate = field(row, "final_url") - if identity(candidate) is None: - return None - try: - validate_http_url(candidate) - except ValueError: - return None - return candidate +def _route_for_case(case: EvalCase, *, native_default: bool = False) -> str: + """Select one output contract before applying page-family extraction rules.""" + if case.task_kind == "file": + return "raw_file" + if native_default: + if case.task_kind == "resource": + return "native_resource_html" + from ..native_routes import declared_input_media_kind + media = declared_input_media_kind(case, required=case.track == "document") + if case.track == "document" and media in {"direct_pdf", "direct_docx"}: + return "native_document_bytes" + return "dynamic_native_markdown" if case.track == "dynamic" else "ordinary_markdown" + return "resource_native_markdown" if case.task_kind == "resource" else "dynamic_native_markdown" if case.track == "dynamic" else "ordinary_markdown" +def _family_wait_until(config, track): + default = (config.get('dynamic_wait_until', 'domstable') + if track == 'dynamic' else config.get('wait_until', 'done')) + return config.get('wait_until_by_track', {}).get(track, default) class MoliWebFetchProvider(Provider): @@ -356,6 +309,21 @@ def _validate_resolved_binary(self, binary: str | None) -> list[str]: self._implementation_identity["native_dns"] = binding return errors + def native_cli_command(self, case): + route = _route_for_case(case, native_default=True) + argv = [str(self.config.get('binary', 'moli')), 'fetch'] + native_dynamic = case.track == 'dynamic' and case.retrieval_hints == {'version': 3, 'mode': 'native'} + if route not in {'raw_file', 'native_document_bytes'}: + wait = _family_wait_until(self.config, case.track) + argv += ['--dump', 'html' if case.task_kind == 'resource' else 'markdown', '--wait-until', wait] + if native_dynamic: + argv += ['--layout', '--with-frames'] + if self.config.get('block_private_networks', True): + argv.append('--block-private-networks') + if self.config.get('log_level'): + argv += ['--log-level', str(self.config['log_level'])] + return [*argv, case.url] + def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: self.check_native_call(case) validate_http_url(case.url) @@ -406,7 +374,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: # task. Only a direct-media-file deliverable takes the raw-byte path. # This avoids changing the participant's output format within a report # family based on the URL suffix or the expected answer. - if case.task_kind == 'file': + if case.task_kind == 'file' and not self.config.get('uniform_native_parameters'): native = HeadlessProvider(self.name, {**self.config, 'type':'headless_markdown', 'engine':'moli', 'binary':binary, 'max_original_file_bytes':int(self.config.get('max_bytes',20_000_000))}) @@ -420,14 +388,8 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: return result finally: with self._process_lock: self._active_native_provider = None - wait = self.config.get("wait_until", "done") - family_wait = ( - self.config.get("dynamic_wait_until", "domstable") - if case.track == "dynamic" - else wait - ) + family_wait = _family_wait_until(self.config, case.track) track_waits = self.config.get("wait_until_by_track", {}) - family_wait = track_waits.get(case.track, family_wait) native_dynamic_task = ( case.track == "dynamic" and case.retrieval_hints == {"version": 3, "mode": "native"} @@ -547,8 +509,21 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: if log_level: command.extend(["--log-level", str(log_level)]) command.append(case.url) + fixed_native = bool(self.config.get('uniform_native_parameters')) + route = _route_for_case(case, native_default=fixed_native) + raw_task = route in {'raw_file', 'native_document_bytes'} + if fixed_native: + command = [binary, *self.native_cli_command(case)[1:]] + dump = None if raw_task else 'html' if case.task_kind == 'resource' else 'markdown' + if not native_dynamic_task: + layout = include_subdocuments = False + wait_metadata = {'wait_until': family_wait, + 'strategy': 'native_default_completion'} + if case.track in track_waits: + wait_metadata.update(scope=f'{case.track}_task_family', + strategy='native_task_family_completion') limit = int(self.config.get("max_bytes", 20_000_000)) - retain_diagnostic = bool(self.config.get("retain_complete_diagnostic_log", False)) + retain_diagnostic = bool(self.config.get("retain_complete_diagnostic_log", False) or fixed_native) diagnostic_limit = int(self.config.get("max_diagnostic_bytes", 100_000_000)) diagnostic_read_limit = diagnostic_limit if retain_diagnostic else limit metadata = { @@ -559,7 +534,8 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: "internal_timeout_policy": "native_default" if self.config.get("uniform_native_parameters") else "explicit", "http_max_response_bytes": None, "retained_output_max_bytes": limit, - "route": ( + "route": route, + "native_route_label": ( "resource_native_markdown" if case.task_kind == "resource" else "dynamic_native_markdown" @@ -645,6 +621,9 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: process_environment = dns_broker.environment(library) metadata["native_dns_binding"] = binding try: + self.capture_native_request(case, transport='cli', interface=route, + parameters={'argv': ['', *command[1:-1], ''], + 'binary_sha256': getattr(self, '_binary_sha256', None)}) process = subprocess.Popen( command, stdin=subprocess.DEVNULL, @@ -751,6 +730,23 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: error_message="Moli fetch exceeded time budget", metadata=metadata, ) + if raw_task: + if self.config.get('uniform_native_parameters'): + metadata.update(_raw_stdout_b64=base64.b64encode(raw).decode('ascii'), + raw_stdout_sha256=hashlib.sha256(raw).hexdigest(), + native_answer_projection={'schema_version': 2, 'source_format': 'bytes', + 'content_format': 'empty', 'files': {'format': 'raw_bytes'}}) + if route == "native_document_bytes": + metadata["native_output_contract"] = "original_document_bytes_without_extracted_text" + ok = process.returncode == 0 + return ProviderResult( + ok, + "", + files=[{"data": raw}] if raw else [], + error_type=None if ok else "tool_error", + error_message=None if ok else stderr or "Moli file request failed", + metadata=metadata, + ) try: content = raw.decode("utf-8") except UnicodeDecodeError: @@ -761,6 +757,11 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: error_message="Moli markdown output is not UTF-8", metadata=metadata, ) + if fixed_native: + metadata.update(_raw_stdout_b64=base64.b64encode(raw).decode('ascii'), + raw_stdout_sha256=hashlib.sha256(raw).hexdigest(), + native_answer_projection={'schema_version': 2, 'source_format': 'bytes', 'content_format': 'utf8', + 'result_fields': {'final_url': {'format': 'fetch_log_final_url'}}}) native_status, readiness_timed_out = _native_fetch_diagnostics(diagnostic_raw, case.url) if native_status is not None: metadata["http_status_source"] = "same_call_native_document_event" diff --git a/harness/web_fetch/providers/subprocess_jsonl.py b/harness/web_fetch/providers/subprocess_jsonl.py index f250503ec..d1b811958 100644 --- a/harness/web_fetch/providers/subprocess_jsonl.py +++ b/harness/web_fetch/providers/subprocess_jsonl.py @@ -194,8 +194,23 @@ def _request_stderr(self, cursor: tuple[int, int]) -> dict[str, Any]: "stderr_truncated": retained_bytes < actual_bytes, } + def native_worker_request(self, case, timeout, diagnostic_path=None): + from ..native_routes import declared_input_media_kind + media = declared_input_media_kind(case, required=False) + return {'url': case.url, 'log_level': self.config.get('log_level'), + 'diagnostic_path': diagnostic_path, 'timeout_seconds': timeout, + 'uniform_native_parameters': True, + 'max_bytes': int(self.config.get('max_bytes', 20_000_000)), + 'dns_mode': self.config.get('dns_mode', 'system'), + 'task_kind': case.task_kind, 'track': case.track, + **({'input_media_kind': media} if media is not None else {}), 'retrieval_plan': None, + 'retrieval_mode': 'native_default' if case.track == 'dynamic' else 'declared_plan'} + def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: self.check_native_call(case) + from ..native_routes import declared_input_media_kind + input_media = declared_input_media_kind(case, required=bool( + self.config.get("uniform_native_parameters") and case.track == "document")) answer_begin = time.perf_counter() try: retrieval_plan = plan_for_case(case) @@ -226,9 +241,19 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: "dns_mode": self.config.get("dns_mode", "system"), "task_kind": case.task_kind, "track": case.track, + **({"input_media_kind": input_media} if input_media is not None else {}), "retrieval_plan": retrieval_plan.worker_dict() if retrieval_plan else None, "retrieval_mode": "native_default" if native_default_case(case) else "declared_plan", } + if self.config.get('uniform_native_parameters'): + request = self.native_worker_request(case, timeout, request['diagnostic_path']) + if getattr(self, '_native_route_ledger', None) is not None: + from ..native_routes import _digest + projected = {**request, 'url': '', + 'diagnostic_path': '' if request['diagnostic_path'] else None} + self.capture_native_request(case, transport='jsonl', interface='native_worker_command', + parameters={'request': projected, + 'command_sha256': _digest([self._resolve_token(part) for part in self.config['command']])}) if self._active_diagnostics: self._active_diagnostics.emit("caller", "worker_request_sent", request=request) try: process.stdin.write(json.dumps(request, ensure_ascii=False) + "\n") @@ -302,6 +327,8 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: worker_logs_truncated = True continue payload = candidate + if payload.get("content") is not None and not isinstance(payload["content"], str): + raise ValueError("worker content must be text or absent") ok = bool(payload.get("ok")) files = [] for item in payload.get("files") or []: @@ -320,6 +347,17 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: error_message=None if ok else str(payload.get("error_message") or "worker failed"), metadata={ **dict(payload.get("metadata") or {}), + **({'native_answer_projection': {'schema_version': 2, + 'content_pointer': ['content'], + 'result_fields': {'title': {'pointer': ['title']}, + 'final_url': {'pointer': ['final_url'], 'fallback': 'request_url'}}, + 'content_format': 'text' if isinstance(payload.get('content'), str) else 'empty', + 'answer_fields': {key: {'pointer': ['metadata', key], 'format': 'identity'} + for key in ('links', 'images', 'videos', 'media', 'extras') + if key in (payload.get('metadata') or {})}, + **({'files': {'pointer': ['files'], 'format': 'worker_files'}} + if payload.get('files') else {})}} + ), "_raw_response_b64": base64.b64encode(line.encode("utf-8")).decode("ascii"), "raw_response_sha256": hashlib.sha256(line.encode("utf-8")).hexdigest(), "raw_response_bytes": len(line.encode("utf-8")), diff --git a/harness/web_fetch/providers/tavily.py b/harness/web_fetch/providers/tavily.py index 83e9be26f..97e0c5b55 100644 --- a/harness/web_fetch/providers/tavily.py +++ b/harness/web_fetch/providers/tavily.py @@ -28,6 +28,12 @@ def _public_failure_errors(values: list[object]) -> list[str]: class TavilyExtractProvider(Provider): + def native_http_arguments(self, case, timeout): + body = {'urls': [case.url], 'extract_depth': self.config.get('extract_depth', 'advanced'), + 'format': 'markdown', 'include_images': bool(self.config.get('include_images', False))} + return (str(self.config.get('base_url', 'https://api.tavily.com')).rstrip('/') + '/extract', + self.native_http_options(body, {'Authorization': 'Bearer '+str(self.config.get('api_key', ''))}, timeout)) + def validate(self) -> list[str]: return [] if self.config.get("api_key") else ["api_key"] @@ -49,7 +55,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: # deadline remains authoritative and we omit an invalid API value. if extract_timeout >= 1.0 and not self.config.get("uniform_native_parameters"): request_body["timeout"] = round(extract_timeout, 3) - response = request( + response = self.native_http_request(case, request, f"{str(self.config.get('base_url', 'https://api.tavily.com')).rstrip('/')}/extract", method="POST", timeout=http_timeout, @@ -84,7 +90,7 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: ok=ok, content=content, status_code=response.status, - final_url=(result.get("url") or case.url) if ok else case.url, + final_url=result.get("url") or case.url, error_type=error_type, error_message=error_message, metadata={ @@ -110,5 +116,12 @@ def fetch(self, case: EvalCase, timeout: float) -> ProviderResult: request_body.get("timeout") ), "images": (result.get("images") or payload.get("images") or []) if ok else [], + **({'native_answer_projection': { + 'schema_version': 2, 'content_pointer': ['results', 0, 'raw_content'], + 'content_format': 'text', 'optional_content': True, + 'result_fields': {'final_url': {'pointer': ['results', 0, 'url'], 'fallback': 'request_url'}}, + 'answer_fields': ({'images': { + 'pointer': ['results', 0, 'images'] if result.get('images') else ['images'], + 'format': 'identity', 'default': []}} if ok else {})}} if ok or content or result.get('url') else {}), }, ) diff --git a/harness/web_fetch/publication_privacy.py b/harness/web_fetch/publication_privacy.py index d3eaf112f..0dadd1f21 100644 --- a/harness/web_fetch/publication_privacy.py +++ b/harness/web_fetch/publication_privacy.py @@ -4,6 +4,7 @@ import json import re from copy import deepcopy +from functools import lru_cache from pathlib import PurePosixPath, PureWindowsPath from urllib.parse import unquote_plus @@ -153,7 +154,7 @@ def _public_cors_policy(data): ) JSON_SENSITIVE_VALUE = re.compile( rb'(?i)(["\']' + SENSITIVE_KEY + rb'["\']\s*:\s*["\'])(?!<|\[)' - rb'((?:\\.|[^"\'\r\n])*)(["\'])' + rb'((?:\\.|[^"\'\r\n])+)(["\'])' ) LINE_SENSITIVE_VALUE = re.compile( rb'(?im)(^|[ \t])(' + SENSITIVE_KEY + rb'[ \t]*[:=])' @@ -188,63 +189,138 @@ def _public_cors_policy(data): ) -def _redact_plain_url_queries(text): - """Redact sensitive parameters in URLs embedded in diagnostic prose.""" - replacements = 0 +_URL_ENTITY = re.compile(r'&(?:\#[xX][0-9A-Fa-f]+;?|\#[0-9]+;?|[A-Za-z][A-Za-z0-9]+;)') - def redact_url(match): - nonlocal replacements - original = match.group() - decoded = original - while True: - unescaped = html.unescape(decoded) - if unescaped == decoded: - break - decoded = unescaped - bounded = EMBEDDED_URL.match(decoded) - if bounded is None: - return original - url = bounded.group() - suffix = decoded[len(url):] - query = url.partition('?')[2].partition('#')[0] - if not query: - return original - prior_replacements = replacements - parts = [] - for item in query.split('&'): - key, separator, value = item.partition('=') - # HTML attributes can spell a separator as &. - logical_key = key.removeprefix('amp;') - decoded_key = unquote_plus(logical_key).casefold() - decoded_value = unquote_plus(value) - sensitive = bool(QUERY_SENSITIVE_KEY.fullmatch(decoded_key)) - placeholder = (decoded_value.startswith('<') and decoded_value.endswith('>') - or decoded_value == '[REDACTED]') - if separator and sensitive and value and not placeholder: - item = key + '=[REDACTED]' - replacements += 1 - parts.append(item) - if replacements == prior_replacements: - return original - projected = url.replace('?' + query, '?' + '&'.join(parts), 1) + suffix - return html.escape(projected, quote=True) if decoded != original else projected - - return EMBEDDED_URL.sub(redact_url, text), replacements + +def _url_entity_surface(raw): + """Decode URL entities while retaining each character's original span.""" + units = [(char, index, index+1) for index, char in enumerate(raw)] + for _ in range(8): + text = ''.join(unit[0] for unit in units) + matches = [m for m in _URL_ENTITY.finditer(text) if html.unescape(m.group()) != m.group()] + if not matches: + return text, units + result, cursor = [], 0 + for match in matches: + result.extend(units[cursor:match.start()]) + start, end = units[match.start()][1], units[match.end()-1][2] + result.extend((char, start, end) for char in html.unescape(match.group())) + cursor = match.end() + result.extend(units[cursor:]) + units = result + raise ValueError('Excessively nested URL entities require explicit privacy review') + + +def _redact_plain_url_queries(text): + """Redact actual query values, preserving URL/Markdown/entity syntax.""" + replacements, cursor, output = 0, 0, [] + while match := EMBEDDED_URL.search(text, cursor): + start, end = match.span() + markdown = bool(re.search(r'\]\([ \t]*$', text[max(0, start-100):start])) + if markdown: + depth = 0 + for index in range(start, end): + if text[index] == '(': + depth += 1 + elif text[index] == ')': + if depth == 0: + end = index + break + depth -= 1 + raw = text[start:end] + suffix = '' + if not markdown: + for opening, closing in (('(', ')'), ('[', ']'), ('{', '}')): + while raw.endswith(closing) and raw.count(closing) > raw.count(opening): + raw, suffix = raw[:-1], closing + suffix + if _URL_ENTITY.search(raw): + url, units = _url_entity_surface(raw) + stop = next((i for i, ch in enumerate(url) if ch in '\"\'<>\\\x00' or ch.isspace()), len(url)) + if stop < len(url): + # An encoded HTML attribute delimiter is not query content. + used = units[stop][1] + raw, end, suffix = raw[:used], start+used, '' + url, units = url[:stop], units[:stop] + else: + url, units = raw, None + query_start = url.find('?') + changes = [] + if query_start >= 0: + query_end = url.find('#', query_start) + if query_end < 0: + query_end = len(url) + position = query_start+1 + for item in url[position:query_end].split('&'): + key, separator, value = item.partition('=') + logical_key = key + while logical_key.startswith('amp;'): + logical_key = logical_key[4:] + decoded_key, decoded_value = unquote_plus(logical_key).casefold(), unquote_plus(value) + sensitive = bool(QUERY_SENSITIVE_KEY.fullmatch(decoded_key)) + placeholder = (decoded_value.startswith('<') and decoded_value.endswith('>') + or decoded_value == '[REDACTED]') + if separator and sensitive and value and not placeholder: + left, right = position+len(key)+1, position+len(item) + changes.append((units[left][1] if units else left, + units[right-1][2] if units else right)) + replacements += 1 + position += len(item)+1 + for left, right in reversed(changes): + raw = raw[:left]+'[REDACTED]'+raw[right:] + output.extend((text[cursor:start], raw+suffix)) + cursor = end + output.append(text[cursor:]) + return ''.join(output), replacements _JSON_STRING = re.compile(r'"(?:\\.|[^"\\\x00-\x1f])*"') +def _json_string_spans(text): + """Match the JSON-string grammar without retrying long invalid suffixes. + + An unterminated escaped string can make an unanchored regex rescan the same + suffix at every quote. Backward states remember the next valid closing + quote; forward selection preserves ordinary finditer's nonoverlap rule. + """ + if '"' not in text: + return + if len(text) <= 4096: + yield from (match.span() for match in _JSON_STRING.finditer(text)) + return + candidates = [] + next_end = after_next_end = None + for index in range(len(text)-1, -1, -1): + char = text[index] + if char == '"': + if next_end is not None: + candidates.append((index, next_end+1)) + end = index + elif char == '\\': + end = (after_next_end if index+1 < len(text) and text[index+1] != '\n' + else None) + elif ord(char) < 32: + end = None + else: + end = next_end + after_next_end, next_end = next_end, end + cursor = 0 + for start, end in reversed(candidates): + if start >= cursor: + yield start, end + cursor = end + + def redact_url_queries(text): """Decode JSON string escapes before checking URLs; preserve clean bytes.""" parts = [] cursor = 0 replacements = 0 - for match in _JSON_STRING.finditer(text): - prefix, count = _redact_plain_url_queries(text[cursor:match.start()]) + for start, end in _json_string_spans(text): + prefix, count = _redact_plain_url_queries(text[cursor:start]) parts.append(prefix) replacements += count - raw = match.group() + raw = text[start:end] try: decoded = json.loads(raw) except json.JSONDecodeError: @@ -254,7 +330,7 @@ def redact_url_queries(text): projected = json.dumps(projected, ensure_ascii=True) if count else raw parts.append(projected) replacements += count - cursor = match.end() + cursor = end suffix, count = _redact_plain_url_queries(text[cursor:]) parts.append(suffix) return ''.join(parts), replacements + count @@ -280,20 +356,75 @@ def auth_matches(data): return distinct -def publication_risk(data): - """Return a public-bundle risk class without exposing the matched value.""" +def _plain_publication_risk(data): + """Inspect actual text, excluding serialization delimiters.""" decoded=html.unescape(data.decode('utf-8',errors='surrogateescape')).encode('utf-8',errors='surrogateescape') if GOOGLE_KEY.search(decoded) or OTHER_SECRET.search(decoded): return 'credential-shaped value' if auth_matches(decoded): return 'sensitive field value' - if redact_url_queries(decoded.decode('utf-8', errors='surrogateescape'))[1]: + # Bare query parameters such as ×tamp must not be interpreted as the + # semicolon-less HTML entity ×. Decode explicit named entities and + # numeric references for URL inspection while preserving raw separators. + text = data.decode('utf-8', errors='surrogateescape') + url_surface = re.sub(r'&(?:\#[xX][0-9A-Fa-f]+;?|\#[0-9]+;?|[A-Za-z][A-Za-z0-9]+;)', + lambda match: html.unescape(match.group()), text) + if redact_url_queries(url_surface)[1]: return 'sensitive URL query value' if LOCAL_USER_PATH.search(decoded): return 'machine-local user path' return None +def _scan_publication_risk(data): + """Inspect JSON values structurally, otherwise inspect the actual text. + + A serialized quote after a string such as ``Authorization:`` is not an + authorization value. Nonempty sensitive fields and real header values still + fail closed, including values hidden by JSON string escapes. + """ + try: + payload = json.loads(data) + except (ValueError, UnicodeDecodeError): + return _plain_publication_risk(data) + + def inspect(value, *, secret=False): + if isinstance(value, dict): + for key, item in value.items(): + risk = _plain_publication_risk(key.encode('utf-8')) + if risk: + return risk + sensitive = secret or bool(re.fullmatch(SENSITIVE_KEY.decode('ascii'), key, re.I)) + risk = inspect(item, secret=sensitive) + if risk: + return risk + elif isinstance(value, list): + for item in value: + risk = inspect(item, secret=secret) + if risk: + return risk + elif isinstance(value, str): + if (secret and value and not ((value.startswith('<') and value.endswith('>')) + or (value.startswith('[') and value.endswith(']')))): + return 'sensitive field value' + return _plain_publication_risk(value.encode('utf-8')) + return None + + return inspect(payload) + + +@lru_cache(maxsize=8) +def _cached_publication_risk(data): + return _scan_publication_risk(data) + + +def publication_risk(data): + """Reuse checks only for identical immutable bytes; bound retained memory.""" + data = bytes(data) + return (_cached_publication_risk(data) if len(data) <= 2_000_000 + else _scan_publication_risk(data)) + + def project_file(name, data): """Project public copies and bind their hashes to retained original evidence.""" path = PurePosixPath(name) @@ -358,6 +489,439 @@ def project_pre(match): } +def _host_metadata_projection_path(name): + path = PurePosixPath(name) + if path.is_absolute() or '..' in path.parts or any(p.startswith('.') for p in path.parts): + return False + metadata = (path.suffix in {'.json', '.jsonl'} + and path.parts[0] not in {'artifacts', 'sources', 'logs', 'references'}) + auxiliary_log = path.parts[0] == 'logs' and path.name.endswith(('.debug.txt', '.debug.html')) + return metadata or auxiliary_log + + +def project_host_metadata_files(files, host_prefixes, *, host_names=(), public_excerpt_views=False, output_store=None): + """Project host metadata/auxiliary logs and explicitly rebind dependencies. + + Inputs and measured payloads are immutable. Returned receipts distinguish + original file identities from the public copies; public hashes never claim + to be original call hashes. Only known whole-file digest references change. + """ + if not host_prefixes or any(not p.startswith('/') or p == '/' for p in host_prefixes): + raise ValueError('Explicit non-root host prefixes are required') + patterns = [re.compile(re.escape(p.rstrip('/')) + r"(?=/|$)[^\s\"'<>\\\x00]*") + for p in sorted(set(host_prefixes), key=len, reverse=True)] + json_names = {name for name in files if name.endswith(('.json', '.jsonl'))} + original_hashes = {name: hashlib.sha256(files[name]).hexdigest() for name in files} + by_hash = {digest: name for name, digest in original_hashes.items()} + # Prewarm hash dependencies before parsing a large parent. Disk-backed + # mappings can therefore project multi-GB histories one document at a time. + dependencies = {} + for name in json_names: + hashes = {match.group(1).decode('ascii') for match in + re.finditer(rb'"([0-9a-f]{64})"(?!\s*:)', files[name])} + dependencies[name] = {by_hash[digest] for digest in hashes + if digest in by_hash and by_hash[digest] != name} + projected = output_store if output_store is not None else {} + published_hashes, changed_fields, visiting = {}, {}, set() + original_sizes, published_sizes, changed_names = {}, {}, set() + + def project(name): + if name in published_hashes: + return + if name in visiting: + raise ValueError('Cyclic publication metadata dependencies') + visiting.add(name) + for dependency in sorted(dependencies.get(name, set())): + project(dependency) + original = raw = files[name] + original_sizes[name] = len(original) + fields = [] + def value(node, path=''): + if isinstance(node, dict): + if 'hostname' in node and path.endswith('.environment'): + fields.append(path + '.hostname') + return {key: value(item, path + '.' + key) for key, item in node.items() + if not (key == 'hostname' and path.endswith('.environment'))} + if isinstance(node, list): + return [value(item, path + '[]') for item in node] + if not isinstance(node, str): + return node + if node in by_hash: + dependency = by_hash[node] + if dependency != name: + project(dependency) + return published_hashes[dependency] + result = node + for pattern in patterns: + result = pattern.sub('[HOST_PATH_OMITTED]', result) + for host in host_names: + if not host: + raise ValueError('Host names must be explicit nonempty identities') + result = result.replace(host, '[HOSTNAME_OMITTED]') + if public_excerpt_views and (path.endswith('.samples[].text') + or path.endswith('.agent_evidence_quotes[]') or path.endswith('.evidence_quotes[]') + or ('.metrics.' in path and path.endswith('.context'))): + public, _ = project_evidence_view('public-evidence/excerpt.txt', result.encode('utf-8'), + role='scored_answer', host_prefixes=host_prefixes, host_names=host_names, + content_format='text') + result = public.decode('utf-8') + if result != node: + fields.append(path) + return result + if name in json_names: + text = raw.decode('utf-8') + parsed = ([json.loads(line) for line in text.splitlines() if line.strip()] + if name.endswith('.jsonl') else json.loads(text)) + result = value(parsed) + if result != parsed: + raw = ((''.join(json.dumps(row, ensure_ascii=False, separators=(',', ':')) + '\n' + for row in result)) if name.endswith('.jsonl') + else json.dumps(result, ensure_ascii=False, indent=2) + '\n').encode() + elif host_names: + raw = value(raw.decode('utf-8'), 'auxiliary_log').encode('utf-8') + if raw != original and not _host_metadata_projection_path(name): + raise ValueError('Host projection must not alter scored or response payloads') + if raw != original: + changed_names.add(name) + projected[name] = raw + published_sizes[name] = len(raw) + published_hashes[name] = hashlib.sha256(raw).hexdigest() + changed_fields[name] = sorted(set(fields)) + visiting.remove(name) + + for name in files: + project(name) + receipt = {'kind': 'host_metadata_publication_projection', + 'provider_calls_added': 0, + 'meaning': 'Original identities are retained below; published hashes identify redacted copies only.', + 'files': [{'path': name, 'original_sha256': original_hashes[name], + 'published_sha256': published_hashes[name], + 'original_bytes': original_sizes[name], 'published_bytes': published_sizes[name], + 'redacted_fields': changed_fields[name]} + for name in sorted(changed_names)]} + return projected, receipt + + +def require_host_metadata_projection(source, binding): + """Validate public metadata hashes without certifying unavailable originals.""" + from .local_revision import read_bound + receipt = json.loads(read_bound(source, binding).read_text()) + rows = receipt.get('files', []) + names = [item['path'] for item in rows] + if (receipt.get('kind') != 'host_metadata_publication_projection' + or receipt.get('provider_calls_added') != 0 or len(names) != len(set(names))): + raise ValueError('Invalid host metadata projection receipt') + for item in rows: + if (not _host_metadata_projection_path(item['path']) + or not re.fullmatch('[0-9a-f]{64}', item.get('original_sha256', '')) + or item['original_sha256'] == item.get('published_sha256')): + raise ValueError('Projection must preserve scored payloads and original identities') + public = read_bound(source, {'path': item['path'], 'sha256': item['published_sha256']}) + if public.stat().st_size != item['published_bytes']: + raise ValueError('Published metadata size differs from projection receipt') + return receipt + + +def project_evidence_view(name, data, *, role, host_prefixes=(), host_names=(), content_format="auto"): + """Create a separately identified public view after scoring. + + The caller retains original bytes privately. This function never changes a + result, score, original binding or source file. Its receipt identifies both + versions; verifying the public hash does not verify unavailable originals. + JSON envelopes are traversed structurally so escaped paths are redacted + without corrupting strings or rewriting numeric measurements. + """ + path = PurePosixPath(name) + if (path.is_absolute() or not path.parts or path.parts[0] != 'public-evidence' + or '..' in path.parts or any(part.startswith('.') for part in path.parts)): + raise ValueError('Public evidence views require a separate safe public-evidence path') + if role not in {'scored_answer', 'native_response'}: + raise ValueError('Unknown public evidence role') + if content_format not in {'auto', 'text', 'json'}: + raise ValueError('Unknown public evidence content format') + if any(not p.startswith('/') or p == '/' for p in host_prefixes): + raise ValueError('Public evidence host prefixes must be explicit non-root paths') + if any(not host for host in host_names): + raise ValueError('Public evidence host names must be explicit nonempty identities') + counts = {} + patterns = [re.compile(re.escape(p.rstrip('/')) + r'''(?=/|$)[^\s"'<>\\\x00]*''') + for p in sorted(set(host_prefixes), key=len, reverse=True)] + # User-home paths in retained diagnostics need a complete replacement, not + # just a username replacement that would disclose the remaining directories. + local = re.compile(LOCAL_USER_PATH.pattern + rb'''[^\s"'<>\\\x00]*''') + + def text_view(value): + before = value + for pattern in patterns: + value, count = pattern.subn('[HOST_PATH_OMITTED]', value) + counts['local_path'] = counts.get('local_path', 0) + count + for host in host_names: + count = value.count(host) + value = value.replace(host, '[HOSTNAME_OMITTED]') + counts['hostname'] = counts.get('hostname', 0) + count + raw = value.encode('utf-8') + raw, count = local.subn(b'[HOST_PATH_OMITTED]', raw) + counts['local_path'] = counts.get('local_path', 0) + count + raw, count = OTHER_SECRET.subn(b'[REDACTED_CREDENTIAL]', raw) + counts['credential_shape'] = counts.get('credential_shape', 0) + count + projected, receipt = project_file(name, raw) + if receipt: + counts['sensitive_values'] = counts.get('sensitive_values', 0) + receipt['replacements'] + result = projected.decode('utf-8') + return result if result != before else before + + def node_view(value): + if isinstance(value, dict): + result = {} + for key, item in value.items(): + public_key = text_view(key) + if public_key in result: + raise ValueError('Public view would merge distinct native fields') + if (re.fullmatch(SENSITIVE_KEY.decode('ascii'), key, re.I) + and isinstance(item, str) and item + and not ((item.startswith('<') and item.endswith('>')) + or (item.startswith('[') and item.endswith(']')))): + result[public_key] = '[REDACTED]' + counts['sensitive_values'] = counts.get('sensitive_values', 0) + 1 + else: + result[public_key] = node_view(item) + return result + if isinstance(value, list): + return [node_view(item) for item in value] + return text_view(value) if isinstance(value, str) else value + + text = data.decode('utf-8') + if content_format == 'text': + projected = text_view(text).encode('utf-8') + encoding = 'utf-8-text' + else: + try: + native = json.loads(text) + except ValueError: + if content_format == 'json': + raise ValueError('Declared JSON public evidence is not JSON') + projected = text_view(text).encode('utf-8') + encoding = 'utf-8-text' + else: + public = node_view(native) + projected = (json.dumps(public, ensure_ascii=False, separators=(',', ':')).encode('utf-8') + if public != native else data) + encoding = 'json' + risk = publication_risk(projected) + if risk: + raise ValueError('Public evidence still has a publication risk: '+risk) + receipt = {'kind': 'post_score_public_evidence_view', 'role': role, 'path': name, + 'encoding': encoding, 'provider_calls_added': 0, + 'original_sha256': hashlib.sha256(data).hexdigest(), + 'original_bytes': len(data), + 'published_sha256': hashlib.sha256(projected).hexdigest(), + 'published_bytes': len(projected), + 'redactions': {key: count for key, count in counts.items() if count}, + 'original_retained_privately': True, + 'meaning': 'Public-view bytes are separately identified; original scoring bytes and measurements are unchanged. The public view cannot reconstruct omitted private values.'} + return projected, receipt + + +def require_public_evidence_view(source, binding): + """Verify a declared public view, without claiming original bytes are public.""" + from .local_revision import read_bound + receipt = json.loads(read_bound(source, binding).read_text()) + name = receipt.get('path', '') + path = PurePosixPath(name) + redactions = receipt.get('redactions') + if (receipt.get('kind') != 'post_score_public_evidence_view' + or receipt.get('role') not in {'scored_answer', 'native_response'} + or receipt.get('encoding') not in {'json', 'utf-8-text'} + or receipt.get('provider_calls_added') != 0 + or receipt.get('original_retained_privately') is not True + or path.is_absolute() or not path.parts or path.parts[0] != 'public-evidence' + or '..' in path.parts or any(p.startswith('.') for p in path.parts) + or not isinstance(redactions, dict) + or any(key not in {'local_path', 'hostname', 'credential_shape', 'sensitive_values'} + or type(count) is not int or count < 1 for key, count in redactions.items())): + raise ValueError('Invalid separately identified public evidence view') + for label in ('original', 'published'): + if (not re.fullmatch('[0-9a-f]{64}', str(receipt.get(label+'_sha256', ''))) + or type(receipt.get(label+'_bytes')) is not int or receipt[label+'_bytes'] < 0): + raise ValueError('Invalid original/public evidence identity') + same = receipt['original_sha256'] == receipt['published_sha256'] + if (same and (redactions or receipt['original_bytes'] != receipt['published_bytes']) + or not same and not redactions): + raise ValueError('Public view identity does not match its declared redactions') + public = read_bound(source, {'path': name, 'sha256': receipt['published_sha256']}) + raw = public.read_bytes() + if len(raw) != receipt['published_bytes'] or publication_risk(raw): + raise ValueError('Public evidence view bytes or privacy state differ') + if receipt['encoding'] == 'json': + json.loads(raw) + else: + raw.decode('utf-8') + return receipt + + +@lru_cache(maxsize=4) +def _public_view_inventory_index(raw): + # Cache parsing by exact file bytes, never by a path or timestamp. Every + # caller still reads and checks the inventory's declared SHA-256. + index = json.loads(raw) + if (index.get('kind') != 'post_score_public_evidence_view_inventory' + or index.get('provider_calls_added') != 0): + raise ValueError('Invalid public evidence view inventory') + entries = {} + for entry in index.get('files', []): + key = entry.get('original_path'), entry.get('original_sha256') + if key in entries: + raise ValueError('Ambiguous public evidence view identity') + entries[key] = entry + return entries + + +def bound_public_view(source, binding): + """Resolve only a view explicitly bound by this snapshot's public profile. + + A returned view verifies public bytes plus the recorded original identity; + it does not claim that omitted original bytes have been reconstructed. + """ + from pathlib import Path + from lexbench_webfetch.paths import contained_path + source = Path(source) + profile_path = contained_path(source, 'presentation-binding.json', reject_symlinks=True) + if not profile_path.is_file(): + return None + profile = json.loads(profile_path.read_text()) + declared = profile.get('publication_evidence_views') + if not declared: + return None + index_path = contained_path(source, declared['path'], reject_symlinks=True) + raw = index_path.read_bytes() + if hashlib.sha256(raw).hexdigest() != declared.get('sha256'): + raise ValueError('Public evidence view inventory hash changed') + entries = _public_view_inventory_index(raw) + entry = entries.get((binding.get('path'), binding.get('sha256'))) + if entry is None: + return None + receipt = require_public_evidence_view(source, entry['view_receipt']) + original_root = PurePosixPath(binding['path']).parts[0] + expected_root = {'scored_answer': 'artifacts', 'native_response': 'private_artifacts'}[receipt['role']] + if original_root != expected_root: + raise ValueError('Public payload views cannot replace metadata or source authority') + if receipt['original_sha256'] != binding['sha256']: + raise ValueError('Public view does not identify the requested original') + return contained_path(source, receipt['path'], reject_symlinks=True), receipt + + +def read_evidence_binding(source, binding): + """Verify original bytes, or return the explicitly disclosed public view.""" + from lexbench_webfetch.paths import contained_path + path = contained_path(source, binding['path'], reject_symlinks=True) + if path.is_file() and hashlib.sha256(path.read_bytes()).hexdigest() == binding.get('sha256'): + return path + public = bound_public_view(source, binding) + if public is None: + raise ValueError('Local revision evidence binding changed: ' + binding['path']) + return public[0] + + +def original_evidence_size(source, binding): + """Return verified original size, or its explicitly recorded private size.""" + path = read_evidence_binding(source, binding) + if hashlib.sha256(path.read_bytes()).hexdigest() == binding['sha256']: + return path.stat().st_size + public = bound_public_view(source, binding) + if public is None: + raise ValueError('Public evidence lacks its original-size disclosure') + return public[1]['original_bytes'] + + +def evidence_excerpt_matches(source, binding, sample, text): + """Validate exact original offsets, or a disclosed redacted-view excerpt. + + Original coordinates remain recorded. A redacted public view can verify the + visible excerpt's presence, not its former private-byte/character position. + """ + start, end = sample.get('start'), sample.get('end') + if type(start) is not int or type(end) is not int or not 0 <= start < end: + return False + excerpt = sample.get('text') + if not isinstance(excerpt, str) or not excerpt: + return False + if end <= len(text) and text[start:end] == excerpt: + return True + public = bound_public_view(source, binding) + return bool(public and public[1]['redactions'] and excerpt in text) + + +def require_public_evidence_view_inventory(source, binding, records): + """Require complete current-answer/native-response view coverage at release.""" + from .local_revision import read_bound + from lexbench_webfetch.paths import contained_path + if (source/'private_artifacts').exists(): + raise ValueError('Public evidence snapshot must not contain private artifact trees') + index = json.loads(read_bound(source, binding).read_text()) + if (index.get('kind') != 'post_score_public_evidence_view_inventory' + or index.get('provider_calls_added') != 0): + raise ValueError('Invalid public evidence view inventory') + entries = index.get('files', []) + wanted = {(row['artifact_path'], row['metrics']['content_sha256']) for row in records} + for row in records: + metadata = row.get('provider_metadata', {}) + if metadata.get('raw_response_path'): + wanted.add((metadata['raw_response_path'], metadata['raw_response_sha256'])) + identities = set() + for entry in entries: + identity = entry.get('original_path'), entry.get('original_sha256') + if identity in identities: + raise ValueError('Public evidence view inventory repeats an original') + identities.add(identity) + receipt = require_public_evidence_view(source, entry['view_receipt']) + root = PurePosixPath(identity[0]).parts + expected_root = {'scored_answer': 'artifacts', 'native_response': 'private_artifacts'}[receipt['role']] + if (not root or root[0] != expected_root or '..' in root + or receipt['original_sha256'] != identity[1]): + raise ValueError('Public evidence view inventory changes an original authority') + if identity in wanted and receipt['role'] == 'scored_answer': + alias = contained_path(source, identity[0], reject_symlinks=True) + if (not alias.is_file() + or hashlib.sha256(alias.read_bytes()).hexdigest() != receipt['published_sha256']): + raise ValueError('Displayed answer differs from its declared public view') + if not wanted <= identities: + raise ValueError('Public evidence view inventory omits current evidence') + return index + + +def public_render_views(source, records): + """Read bound display annotations; never rewrite saved result records.""" + from .local_revision import read_bound + profile_path = source/'presentation-binding.json' + if not profile_path.is_file(): + return {} + profile = json.loads(profile_path.read_text()) + declared = profile.get('publication_evidence_views') + if not declared: + return {} + index = json.loads(read_bound(source, declared).read_text()) + entries = {(e['original_path'], e['original_sha256']): e for e in index['files']} + result = {} + for row in records: + metadata = row.get('provider_metadata', {}) + wanted = {'scored_answer': (row['artifact_path'], row['metrics']['content_sha256'])} + if metadata.get('raw_response_path'): + wanted['native_response'] = (metadata['raw_response_path'], metadata['raw_response_sha256']) + views = {} + for role, identity in wanted.items(): + if identity not in entries: + raise ValueError('Public rendering omits a declared evidence view') + binding = entries[identity]['view_receipt'] + receipt = json.loads(read_bound(source, binding).read_text()) + if receipt['role'] != role or receipt['original_sha256'] != identity[1]: + raise ValueError('Public rendering changed its original evidence identity') + views[role] = {key: receipt[key] for key in ('path', 'original_sha256', 'published_sha256', + 'original_bytes', 'published_bytes', 'redactions')} + views[role]['receipt'] = binding['path'] + result[row['attempt_id']] = views + return result + + def _project_json(name, data): """Project decoded values so header text cannot consume JSON delimiters.""" lines = data.splitlines(keepends=True) if name.endswith('.jsonl') else [data] diff --git a/harness/web_fetch/reference_lifecycle.py b/harness/web_fetch/reference_lifecycle.py index e7f43c5b0..0a021671c 100644 --- a/harness/web_fetch/reference_lifecycle.py +++ b/harness/web_fetch/reference_lifecycle.py @@ -264,8 +264,59 @@ def _resource_target_matches(case, candidates: list[dict]) -> list[dict]: return results +def _resource_observation_scope(case, root, evidence): + """Retain an observation's old scope across an explicit score-only alias revision.""" + current = resource_source_scope_identity(case) + binding = current['resource_output_contract'].pop('equivalence_revision', None) + if binding is None: + return current + identifier = safe_identifier(_get(case, 'id')) + if (not isinstance(binding, dict) + or re.fullmatch(r'resource-equivalence-(?:revision|[0-9a-f]{64})\.json', + str(binding.get('path', ''))) is None): + raise ReferenceLifecycleError('Invalid resource equivalence revision binding') + try: + raw = contained_path(root, identifier + '/' + binding['path']).read_bytes() + if sha256(raw) != binding.get('sha256'): + raise ReferenceLifecycleError('Resource equivalence revision hash changed') + receipt = json.loads(raw) + before = receipt['before_scope'] + canonical = json.dumps(current, ensure_ascii=False, sort_keys=True, separators=(',', ':')) + if (receipt.get('kind') != 'offline_resource_equivalence_revision' + or receipt.get('provider_calls_added') != 0 + or receipt.get('new_source_observation') is not False + or receipt.get('after_scope_sha256') != sha256(canonical) + or receipt.get('source_evidence_sha256') != _get(case, 'source_verification', {}).get('evidence_sha256') + or receipt.get('browser_observation_sha256') != evidence.get('independent_browser_observation', {}).get('sha256') + or not receipt.get('authority', {}).get('description')): + raise ReferenceLifecycleError('Resource equivalence revision authority/scope differs') + # Only add declared equivalents. Never replace primary source targets, + # their identity, candidate policy, task URL, or historical observations. + old_targets, new_targets = before['expected_resources'], current['expected_resources'] + if ({k: v for k, v in before.items() if k != 'expected_resources'} + != {k: v for k, v in current.items() if k != 'expected_resources'} + or len(old_targets) != len(new_targets)): + raise ReferenceLifecycleError('Resource equivalence revision changed source scope') + for old, new in zip(old_targets, new_targets): + if ({k: v for k, v in old.items() if k not in {'aliases', 'url_patterns'}} + != {k: v for k, v in new.items() if k not in {'aliases', 'url_patterns'}} + or any(not set(old.get(key, [])) <= set(new.get(key, [])) + for key in ('aliases', 'url_patterns'))): + raise ReferenceLifecycleError('Resource equivalence revision replaced a primary target') + from lexbench_webfetch.scoring_regressions import require_scoring_regressions + controls = receipt.get('regression_ids') + if not isinstance(controls, list) or not controls: + raise ReferenceLifecycleError('Resource equivalence revision lacks executable controls') + require_scoring_regressions(controls) + except ReferenceLifecycleError: + raise + except (OSError, ValueError, TypeError, KeyError) as error: + raise ReferenceLifecycleError('Invalid resource equivalence revision') from error + return before + + def validate_resource_source_evidence(case, dataset_path: Path, *, execution: bool = False) -> dict | None: - """Qualify resource sources, or reuse fixed targets during declared execution. + """Fail closed unless every resource target has independent, hash-bound evidence. HTTP probes remain useful diagnostics, but a blocked probe cannot become a valid source merely because an unbound human-review note says the page was @@ -327,6 +378,7 @@ def validate_resource_source_evidence(case, dataset_path: Path, *, execution: bo candidates = [*candidates, *observed_candidates] source_qualified = evidence.get("state") == "captured" and evidence.get("status") == 200 + observed_scope = _resource_observation_scope(case, root, evidence) browser_binding = evidence.get("independent_browser_observation") if browser_binding is not None: if (not isinstance(browser_binding, dict) @@ -351,12 +403,13 @@ def validate_resource_source_evidence(case, dataset_path: Path, *, execution: bo if (observation.get("case_id") != identifier or observation.get("url") != _get(case, "url") or observation.get("independent") is not True - or observation.get("resource_scope_identity") != resource_source_scope_identity(case) + or observation.get("resource_scope_identity") != observed_scope or not isinstance(observed_candidates, list) or len(observed_candidates) > 512 or observation.get("resource_candidates_sha256") != sha256(canonical)): raise ReferenceLifecycleError(f"{identifier}: browser observation scope/hash mismatch") timestamp(observation.get("checked_at")) - browser_matches = _resource_target_matches(case, observed_candidates) + browser_matches = _resource_target_matches( + {**observed_scope, "id": observed_scope["case_id"]}, observed_candidates) browser_qualified = ( observation.get("status") == "success" and observation.get("completion", {}).get("satisfied") is True @@ -1039,6 +1092,7 @@ def valid_assembled_binding(row, index): if not isinstance(rows, list) or not rows: return False allowed_fields = { + "page_identity_required", 'body_requirements', 'required_patterns', 'required_facts', 'acceptance_contract', 'composite_acceptance', 'forbidden_any', 'expected_resources', 'resource_output_contract', 'task_prompt', 'reference_answer', @@ -1460,6 +1514,10 @@ def current_protocol(batch): def valid_hash(value): return isinstance(value, str) and re.fullmatch(r"[0-9a-f]{64}", value) is not None + from .native_routes import verify_native_execution_manifests + verify_native_execution_manifests(source, manifests, + {'cases': [dict(c) if isinstance(c, dict) else vars(c) for c in case_map.values()]} + if case_map is not None else None, records) allow_scoring_rule_revision = (allow_scoring_rule_revision or valid_scoring_revision(source)) revised_references = reference_revision_cases(source) original_latency = latency_correction_original_elapsed(source, records=records) diff --git a/harness/web_fetch/report.py b/harness/web_fetch/report.py index 4b7843847..7845fbb94 100644 --- a/harness/web_fetch/report.py +++ b/harness/web_fetch/report.py @@ -309,6 +309,8 @@ def case_track_label(case): if case.get('track','web')=='web': return {'file':'文件交付','resource':'资源发现','structure':'结构化内容'}.get(case.get('task_kind','body'),TRACKS['web']) return TRACKS[case['track']] +REPORT_TRACKS = {key: TRACKS[key] for key in ('web', 'dynamic', 'challenge', 'risk_control', 'document', 'image', 'video')} + def active_tracks(cases): rows=[] for key,label in REPORT_FAMILIES.items(): @@ -350,7 +352,7 @@ def provider_page_body(name,records,cases,headers,tracks,*,native_default=False) function filterMatch(r,s){return !s.value||(s.dataset.filter==='query'?(r.textContent+' '+(r.dataset.search||'')).toLowerCase().includes(s.value.trim().toLowerCase()):filterValues(r,s.dataset.filter).includes(s.value))} function filterRows(){let count=0;document.querySelectorAll('#rows tr').forEach(r=>{let show=[...document.querySelectorAll('[data-filter]')].every(s=>filterMatch(r,s));r.hidden=!show;if(show)count++});let c=document.querySelector('#count');if(c)c.textContent=tr(`显示 ${count} 条`,`Showing ${count}`)} function paintSort(t){let col=Number(t.dataset.col);if(!Number.isFinite(col)||!t.tHead?.rows[0]?.cells[col])return;[...t.tHead.rows[0].cells].forEach((h,i)=>{h.classList.toggle('sorted',i===col);h.removeAttribute('aria-sort')});t.tHead.rows[0].cells[col].setAttribute('aria-sort',t.dataset.dir==='asc'?'ascending':'descending')} -function sortTable(id,col){let t=document.getElementById(id),dir=t.dataset.col===String(col)&&t.dataset.dir==='desc'?'asc':'desc';let rows=[...t.tBodies[0].rows];rows.sort((a,b)=>{let av=a.cells[col].dataset.sort,bv=b.cells[col].dataset.sort;if(av==='-1'||bv==='-1')return av===bv?0:av==='-1'?1:-1;let result=(av==null||bv==null)?a.cells[col].textContent.localeCompare(b.cells[col].textContent,document.documentElement.lang):Number(av)-Number(bv);return dir==='asc'?result:-result});rows.forEach((r,i)=>{r.cells[0].textContent=i+1;t.tBodies[0].append(r)});t.dataset.col=String(col);t.dataset.dir=dir;paintSort(t);if(t.hasAttribute?.('data-wmb-table'))wmbFilters();if(t.id==='wmb-slices')wmbSliceFilters()} +function sortTable(id,col){let t=document.getElementById(id),dir=t.dataset.col===String(col)&&t.dataset.dir==='desc'?'asc':'desc';let rows=[...t.tBodies[0].rows];rows.sort((a,b)=>{let av=a.cells[col].dataset.sort,bv=b.cells[col].dataset.sort;if(av==='-1'||bv==='-1')return av===bv?0:av==='-1'?1:-1;let result=(av==null||bv==null)?a.cells[col].textContent.localeCompare(b.cells[col].textContent,document.documentElement.lang):Number(av)-Number(bv);return dir==='asc'?result:-result});rows.forEach((r,i)=>{if(t.dataset.rowIndex!=='false')r.cells[0].textContent=i+1;t.tBodies[0].append(r)});t.dataset.col=String(col);t.dataset.dir=dir;paintSort(t);if(t.hasAttribute?.('data-wmb-table'))wmbFilters();if(t.id==='wmb-slices')wmbSliceFilters()} function chooseTab(group,key){let box=document.getElementById(group);if(!box)return;box.querySelectorAll(':scope > .tab-bar > button').forEach(b=>{let on=b.dataset.tab===key;b.setAttribute('aria-selected',String(on));b.tabIndex=on?0:-1});box.querySelectorAll(':scope > [role="tabpanel"]').forEach(p=>p.hidden=p.dataset.panel!==key)} function tabKeys(event,group){if(!['ArrowRight','ArrowLeft','Home','End'].includes(event.key))return;let bs=[...document.getElementById(group).querySelectorAll(':scope > .tab-bar > button')],i=bs.indexOf(event.target);i=event.key==='Home'?0:event.key==='End'?bs.length-1:(i+(event.key==='ArrowRight'?1:-1)+bs.length)%bs.length;event.preventDefault();chooseTab(group,bs[i].dataset.tab);bs[i].focus()} function cascadeFilters(changed){let rows=[...document.querySelectorAll('#rows tr')],selects=[...document.querySelectorAll('select[data-filter]')],prior=[];selects.forEach(s=>{let available=new Set(rows.filter(r=>prior.every(p=>filterMatch(r,p))).flatMap(r=>filterValues(r,s.dataset.filter)));if(s.dataset.filter!=='result'){let failureFilter=['failureStage','failureCategory','issue'].includes(s.dataset.filter);[...s.options].forEach(o=>o.disabled=!!o.value&&!available.has(o.value));if(s.value&&!available.has(s.value))s.value=failureFilter?'':([...s.options].find(o=>o.value&&!o.disabled)?.value||'');if(failureFilter)s.disabled=![...s.options].some(o=>o.value&&!o.disabled);}prior.push(s)});filterRows()} @@ -379,6 +381,7 @@ def provider_page_body(name,records,cases,headers,tracks,*,native_default=False) document.addEventListener('click',event=>{let jump=event.target.closest('[data-log-line]');if(jump){let line=document.getElementById(jump.dataset.logLine);line?.scrollIntoView({block:'center'});line?.focus({preventScroll:true});return}let open=event.target.closest('[data-call-log-open]');if(open){openCallLog(open);return}let close=event.target.closest('[data-call-log-close]');if(close)closeCallLog(close)}); ''' EXTRA=''' +.wrap{min-width:0;overflow-wrap:anywhere}.table-wrap{max-width:100%} .wmb-log-focus{background:#684f0a;color:#fff3bb;outline:2px solid #b98b1d;display:block}.wmb-log>span{scroll-margin:20px}.wmb-detail-dialog{width:min(1120px,calc(100vw - 24px));max-height:90vh;padding:0;border:1px solid var(--line);border-radius:16px;background:var(--panel);color:var(--ink)}.wmb-detail-dialog::backdrop{background:#0009}.wmb-detail-dialog .call-log-shell>header{display:flex;align-items:center;justify-content:space-between;gap:16px;padding:20px;border-bottom:1px solid var(--line);position:sticky;top:0;background:var(--panel);z-index:1}.wmb-detail-dialog h2{font-size:24px;margin:0;overflow-wrap:anywhere}.wmb-dialog-content{padding:20px;min-width:0}.wmb-detail-dialog pre{max-height:60vh;overflow:auto;white-space:pre-wrap;overflow-wrap:anywhere;background:var(--code-bg);color:var(--code-ink);padding:16px;border-radius:8px}.call-log-button,.wmb-detail-dialog [data-call-log-close]{font:inherit;color:var(--accent);background:var(--panel);border:1px solid var(--line);padding:8px 12px;border-radius:8px;cursor:pointer}.call-log-button:focus-visible,.wmb-detail-dialog button:focus-visible{outline:3px solid var(--accent);outline-offset:2px}.wmb-evidence-actions{display:flex;gap:12px;flex-wrap:wrap}.wmb-diagnosis{display:grid;grid-template-columns:150px 1fr;gap:12px}.wmb-diagnosis dt{font-weight:700}.wmb-diagnosis dd{margin:0;overflow-wrap:anywhere}.wmb-detail-scores{min-width:650px}.wmb-detail-scores th,.wmb-detail-scores td{white-space:nowrap}.wmb-detail-dialog .evidence-compare{min-width:0}.wmb-detail-dialog .evidence-pane{min-width:0}@media(max-width:700px){.wmb-diagnosis{grid-template-columns:1fr;gap:6px}.wmb-diagnosis dd{margin-bottom:14px}.wmb-detail-dialog .evidence-compare{display:grid;grid-template-columns:1fr}.wmb-dialog-content{padding:12px}.wmb-detail-dialog .call-log-shell>header{padding:12px}.wmb-detail-dialog h2{font-size:20px}} .table-wrap{margin-top:16px}table{width:100%;border-collapse:collapse;min-width:760px}th,td{vertical-align:top;padding:12px 14px;border-bottom:1px solid var(--line);text-align:left}th{background:var(--panel2);font-size:12px}td:first-child,th:first-child{width:48px;white-space:nowrap}.ranking{min-width:1240px}.ranking td{white-space:nowrap}.ranking td:nth-child(2){white-space:nowrap;min-width:220px;max-width:none;overflow-wrap:normal;word-break:normal}.sort-button{border:0;background:none;color:inherit;font:inherit;font-weight:700;cursor:pointer;padding:0;text-align:left}.grid .card{grid-column:span 4}.card p{margin-bottom:0}.filters{display:flex;gap:12px;flex-wrap:wrap;align-items:center}.filters select,.filters input{min-width:150px;padding:10px;border:1px solid var(--line);background:var(--panel);color:var(--ink);border-radius:8px}.compare{display:grid;grid-template-columns:1fr 1fr;gap:20px}.compare>div{min-width:0}pre{white-space:pre-wrap;overflow-wrap:anywhere;max-height:620px;overflow:auto}.break{overflow-wrap:anywhere}.status{white-space:nowrap}.source-note{color:var(--muted);font-size:13px}details{padding:10px 0}.accounts-table{min-width:1300px}.accounts-table code{white-space:nowrap}.hero h1{font-size:clamp(30px,4vw,48px)}section[id]{scroll-margin-top:75px}.actions{display:flex;gap:16px;flex-wrap:wrap}nav{gap:16px}.tiny{font-size:12px} @media(max-width:700px){.grid .card{grid-column:span 12}.compare{grid-template-columns:1fr}.wrap{width:calc(100% - 28px)}.filters>*{width:100%}} ''' @@ -593,6 +596,8 @@ def run_log_section(out,record,output): payload=logs[-1]['payload'] if logs else {} process=payload.get('process') if isinstance(payload.get('process'),dict) else {} timing=payload.get('timing') if isinstance(payload.get('timing'),dict) else {} + native_public=metadata.get('publication_views',{}).get('native_response') + native_public_link=('
'+link('../'+native_public['path'],'下载完整原生响应（公开副本）','Download complete native response (public view)')+' · '+link('../'+native_public['receipt'],'原件与公开副本校验记录','Original/public identity record')) if native_public else '' artifact_relative=record.get('artifact_path') artifact=contained_path(out,artifact_relative) if artifact_relative else None output_data=artifact.read_bytes() if artifact and artifact.is_file() else output.encode('utf-8') @@ -661,7 +666,12 @@ def run_log_section(out,record,output): raw_record=' · '.join(f'下载{esc(log["label"])}记录' for log in logs) if artifact_relative: if raw_record:raw_record+=' · ' - raw_record+=f'下载本次保存的原始输出' + _answer_view=metadata.get('publication_views',{}).get('scored_answer') + if _answer_view and _answer_view['original_sha256']!=_answer_view['published_sha256']: + raw_record+=f''+model_text('下载本次输出公开副本','Download public output view')+'' + else: + raw_record+=f'下载本次保存的原始输出' + raw_record+=native_public_link public_response=metadata.get('public_response_path') if public_response: response_view=diagnostic_view_page(out,public_response,'完整原始响应','产品实际返回；公开副本已隐藏凭据与本机路径。') @@ -682,17 +692,24 @@ def run_log_section(out,record,output): raw_record+=(f'下载完整原始 debug 日志' f'（{diagnostic_size:,} 字节 · SHA-256：{esc(diagnostic_sha)} · 截断：{diagnostic_truncated}）') if not raw_record:raw_record='这次历史调用没有可下载的原始记录' + evidence_identity_note=(model_text('评分原件私下保留；公开下载隐藏目录或敏感值，校验记录分别标明原件与公开副本。','Scoring originals remain private; public downloads hide paths or sensitive values and bind original/public identities separately.') if any(view.get('redactions') for view in metadata.get('publication_views',{}).values()) else '原始输出保持调用后保存的字节。') return (f'

运行日志与证据

' f'
{fields}
{runtime_error}' f'
查看完整运行字段
{full_fields}
' f'

结论说明：{result_note}

' f'
查看本次真实日志
{diagnostic}
' f'

原始运行证据

' - f'

{raw_record}。运行记录保留请求状态、耗时和证据路径；原始输出保持调用后保存的字节。

') + f'

{raw_record}。运行记录保留请求状态、耗时和证据路径；{evidence_identity_note}

') def pct(n):return '—' if n is None else f'{n*100:.1f}%' def ms(n):return '—' if n is None else f'{n:,.0f} ms' -def link(url,label):return f'{esc(label)}' +def link(url,label,english=None):return f'{model_text(label,english)}' def cell(value,sort=None):return '{value}' +EXTRA+=''' +table.ranking:not([data-row-index="false"]) tbody td:nth-child(4), +table.ranking[data-row-index="false"] tbody td:nth-child(3){color:var(--good);font-weight:800} +''' +def model_text(value,english=None): + return esc(value) if english is None else ''+esc(value)+'' def rate(a,b):return pct(a/b) if b else '—' def section(title,body,id=''):return '

{title}

{body}
' def hero(title,desc): @@ -726,13 +743,20 @@ def comparative_rows(rows, directions): return result -def table(headers,rows,id='',sortable=False,cls='',metrics=None,index_header='序号',default_sort=None): - head=f'{esc(index_header)}'+''.join(f'' if sortable else f'{h}' for i,h in enumerate(headers)) - body=''.join(f'{i+1}{values}' for i,(attrs,values) in enumerate(rows)) +def table(headers,rows,id='',sortable=False,cls='',metrics=None,index_header='序号',default_sort=None,include_index=True): + head=(f'{esc(index_header)}' if include_index else '')+''.join(f'' if sortable else f'{h}' for i,h in enumerate(headers)) + body=''.join(f''+ (f'{i+1}' if include_index else '')+f'{values}' for i,(attrs,values) in enumerate(rows)) sort_attrs=(f' data-col="{default_sort[0]}" data-dir="{default_sort[1]}"' if default_sort else '') + if not include_index:sort_attrs+=' data-row-index="false"' return metric_legend(headers if metrics is None else metrics)+f'
{head}{body}
' EXTRA+='\n.change-log section{padding:24px 0}.change-log h2{font-size:24px}.change-log p{margin:10px 0;line-height:1.75}.change-log summary{font-weight:650;color:var(--accent)}.change-log .table-wrap{overflow-x:auto}\n' EXTRA+=''' +#filtered-table.all-method-results{min-width:1280px} +#filtered-table.all-method-results td:nth-child(2){min-width:180px;white-space:nowrap} +#filtered-table.all-method-results td:nth-child(3){min-width:260px} +#filtered-table.all-method-results td:nth-child(4){min-width:120px;white-space:nowrap} +''' +EXTRA+=''' .anchor-grid[data-scroll-after="5"],.answer-list[data-scroll-after="5"],.record-scroll{max-height:620px;overflow:auto;overscroll-behavior:contain;scrollbar-gutter:stable;padding-right:6px} .region-pair pre{max-height:360px;overflow:auto;scrollbar-gutter:stable} .diagnostic-log{margin-top:18px;padding:0;border:1px solid var(--line);border-radius:12px;background:var(--panel)} @@ -759,7 +783,7 @@ def write(out,path,title,body,*,root_links=False): counterpart=(base+'en/'+path if _ENGLISH_GENERATED else base+'en/index.html') language=f'English' footer='所有分数由本轮调用记录计算。网页与服务可能变化；结论仅适用于本轮样本和配置。' - page=f'{document_base}{esc(title)} · Web Fetch
{_RELIABILITY_BANNER}{body}

{footer}

' + page=f'{document_base}{esc(title)} · Web Fetch
{_RELIABILITY_BANNER}{body}

{footer}

' target=contained_path(out,path);target.parent.mkdir(parents=True,exist_ok=True);target.write_text(page,encoding='utf-8') _RESEARCH_ROUTES=[ @@ -1083,7 +1107,8 @@ def measured(r,c): return task_measured(r['metrics'],c.get('task_kind','body')) def result_label(r,c): - if not measured(r,c):return '未测' + if r['metrics'].get('evaluation_excluded'):return '排除比较' + if not measured(r,c):return '无法判定' state='通过' if succeeded(r,c) else '未通过' return '列表任务：'+state if c.get('task_kind')=='structure' else state @@ -1092,20 +1117,32 @@ def case_rate(rs,c): value=rate(sum(succeeded(r,c) for r in rs),len(rs)) return value -def overview_card(records,cases,track,label,names): +def overview_card(records,cases,track,label,names,*,diagnostic=False): selected=[c for c in cases.values() if report_family(c)==track] if not selected:return '' rs=[r for r in records if report_family(cases[r['case_id']])==track] eligible=[r for r in rs if measured(r,cases[r['case_id']])] + if diagnostic: + return f'
{len(selected)} 个任务

{link("#home-rankings-"+track,label)}

本地诊断结果，不提供名次。

' rates={n:sum(succeeded(r,cases[r['case_id']]) for r in eligible if r['provider']==n)/sum(r['provider']==n for r in eligible) for n in names if any(r['provider']==n for r in eligible)} maximum=max(rates.values(),default=None) winners='、'.join(provider_label(n) for n in names if rates.get(n)==maximum) if maximum else '本轮无通过任务' if rates else '暂无调用记录' return f'
{len(selected)} 个任务

{link("#home-rankings-"+track,label)}

本类最高：{pct(maximum)}

{esc(winners)}

' -def review_summary(out, prefix=''): +def review_summary(out, prefix='', *, highlight=True): path=out/'result-review.json' if not path.is_file():return '' audit=json.loads(path.read_text()) + if audit.get('retained_reviews'): + mode_note = (model_text(' 本次为已知程序结论的证据复核，不是独立盲审。', ' This round checked evidence with known machine outcomes; it was not an independent blind review.') if audit.get('review_protocol',{}).get('blind_initial_verdict') is False else '') + return (''+model_text( + f'当前复核覆盖 {audit.get("reviewed_attempts",0)}/{audit.get("expected_attempts",0)}：本次重新检查 {audit.get("fresh_reviewed_attempts",0)} 条，{audit.get("retained_reviewed_attempts",0)} 条沿用检查依据未变的旧复核。', + f'Current review coverage: {audit.get("reviewed_attempts",0)}/{audit.get("expected_attempts",0)}: {audit.get("fresh_reviewed_attempts",0)} freshly reviewed; {audit.get("retained_reviewed_attempts",0)} prior reviews retained after verifying unchanged decision inputs.')+mode_note+' '+link(prefix+'review/index.html','查看核验轮次 →')+'

') + if audit.get('adjudication_revision'): + count=len(audit['adjudication_revision']['adjudicated_attempt_ids']) + return (''+model_text( + f'当前复核覆盖 {audit.get("reviewed_attempts",0)}/{audit.get("expected_attempts",0)}；本次补充逐案裁定 {count} 条。未新增全量盲审，也不构成正式认证。', + f'Current review coverage: {audit.get("reviewed_attempts",0)}/{audit.get("expected_attempts",0)}; {count} case-specific adjudications supplemented. No new full blind review or certification is claimed.')+' '+link(prefix+'review/index.html','查看核验轮次 →')+'

') scope=audit.get('review_protocol',{}).get('scope','all') if scope=='supplement-25-local-audit': return (f'

新增25例结果审核 ' @@ -1125,7 +1162,7 @@ def review_cell(audit, rows, base=''): if not audit:return '' indexed=review_index(audit);items=[indexed[r['attempt_id']] for r in rows if r['attempt_id'] in indexed] unresolved=sum(not x.get('final_verdict') for x in items) - label=f'{len(items)}/{len(rows)} · Codex 已核验' + label=f'{len(items)}/{len(rows)} · 复核记录' if unresolved:label+=f' · 未决 {unresolved}' return cell(link(base+'review/index.html',label),len(items)/len(rows) if rows else -1) @@ -1149,15 +1186,17 @@ def _best_evidence_excerpt(expected, output): if score>best_score:best,best_score=chunk,score return best if best_score>=min(.25,3/max(1,sum(wanted.values()))) else '' -def _exact_evidence_lines(expected, output): +def _normalized_evidence_lines(output): + from lexbench_webfetch.metrics import normalize_for_match + return [(line,normalize_for_match(line)) for raw in output.splitlines() if (line:=raw.strip())] + +def _exact_evidence_lines(expected, output, *, normalized_lines=None): """Return complete output lines containing the normalized expected text.""" from lexbench_webfetch.metrics import normalize_for_match normalized_expected=normalize_for_match(expected) selected=[] - for raw in output.splitlines(): - line=raw.strip() - if not line:continue - normalized_line=normalize_for_match(line) + lines=_normalized_evidence_lines(output) if normalized_lines is None else normalized_lines + for line,normalized_line in lines: if normalized_expected and normalized_expected in normalized_line: if line not in selected:selected.append(line) return '\n\n'.join(selected) @@ -1254,20 +1293,38 @@ def _body_scoring_text(record, output): from lexbench_webfetch.response_input import answer_input content=answer_input(output.encode('utf-8'),record.get('provider_metadata'), ok=record.get('ok',False),status_code=record.get('status_code'))[0] - from lexbench_webfetch.reference_quality import is_html_document_or_fragment, visible_mixed_html_text + from lexbench_webfetch.reference_quality import is_html_document_or_fragment if is_html_document_or_fragment(content): from lexbench_webfetch.dynamic_lists import visible_html_text content=visible_html_text(content) else: - content=visible_mixed_html_text(content) + from lexbench_webfetch.body_coverage import native_markup_body_text + fallback=html.unescape(re.sub(r"<(script|style|template)\b[^>]*>.*?","",content,flags=re.I|re.S)) + content=native_markup_body_text(content,fallback) return content -def body_region_rows(case, metrics, output): +def body_region_rows(case, metrics, output, *, frozen_scores=False): from lexbench_webfetch.body_coverage import evaluate_body_requirements contract=case.get('body_requirements') or {} if not contract:return [],'' saved_regions=metrics.get('body_region_results',[]) - if saved_regions and all('matched_output_passages' in item for item in saved_regions): + recalculated={} + if frozen_scores: + # The frozen-build boundary validates the saved metric and answer hashes. + # A newer normalizer or a redacted public answer cannot reproduce the old + # matching-text identity. Render the saved allocation without rescoring it. + from .publication_privacy import project_evidence_view + evidence={} + for item in saved_regions: + passages=[] + for passage in item.get('matched_output_passages',[]): + if passage.get('text'): + public,_=project_evidence_view('public-evidence/excerpt.txt',passage['text'].encode(),role='scored_answer',content_format='text') + passages.append(public.decode()) + evidence[item['id']]=passages + evidence_counts={item['id']:item.get('matched_units',item.get('matched_tokens',0)) for item in saved_regions} + recalculated={item['id']:item for item in saved_regions} + elif saved_regions and all('matched_output_passages' in item for item in saved_regions): from lexbench_webfetch.body_coverage import _compact_with_positions matching_text=_compact_with_positions(output)[0] digest=hashlib.sha256(matching_text.encode()).hexdigest() @@ -1284,7 +1341,7 @@ def body_region_rows(case, metrics, output): rows=[];cards=[];total=len(regions) for index,item in enumerate(metrics.get('body_region_results',[]),1): actual={**recalculated.get(item['id'],{}),**item} - if evidence_counts.get(item['id'])!=recalculated.get(item['id'],{}).get('matched_units'): + if recalculated and evidence_counts.get(item['id'])!=recalculated.get(item['id'],{}).get('matched_units'): raise ValueError('Body evidence allocation differs from scorer: '+item['id']) expected=regions.get(item['id'],{}).get('text','') expected_count=actual.get('expected_units',actual.get('expected_tokens',len(expected.split()))) @@ -1296,8 +1353,10 @@ def body_region_rows(case, metrics, output): rows.append(('',cell(esc(label))+cell(f'{expected_count:,}')+cell(f'{matched:,}')+ cell(pct(actual['coverage']))+cell(esc(judgement)))) passage='\n\n'.join(evidence.get(item['id'],[])) + missing=(model_text('历史评分未保存本段命中文字；请对照完整输出核对。','The historical score has no saved matching passage for this region; inspect the complete output.') + if frozen_scores and 'matched_output_passages' not in item else '评分器没有向这一段分配可展示的实际返回文字。') actual_html=(f'

{esc(passage)}
' if passage else - '

评分器没有向这一段分配可展示的实际返回文字。

') + '

'+missing+'

') cards.append(f'

{index}. {esc(label)}

本段覆盖率为 {pct(actual["coverage"])}。每段至少达到 {pct(actual.get("floor",actual["minimum"]))}；达到 {pct(actual["minimum"])} 时算作主要完整段落。

{esc(judgement)}
这一位置的完整标准正文
{esc(expected)}
参与本段计分的完整实际返回行{actual_html}
') return rows,cards @@ -1308,8 +1367,8 @@ def stable_answer_panel(contract): items=[] for index,anchor in enumerate(anchors,1): alternatives=anchor.get('alternatives',[]) - text='\n或\n'.join(alternatives) - items.append(f'
{anchor_label} {index}（任一写法即可）
{esc(text)}
') + text='\n或\n'.join(f'{esc(value)}' for value in alternatives) + items.append(f'
{anchor_label} {index}（任一写法即可）
{text}
') minimum=contract.get('minimum_records') maximum=contract.get('maximum_records') count=(f'至少 {minimum} 条完整且不重复的记录' if maximum is None else @@ -1320,6 +1379,17 @@ def stable_answer_panel(contract): if len(anchors)>5 else '') return note+f'
结构要求
{esc(count)}
{"".join(items)}
' +def required_text_answer_panel(case): + items=[f'
正文长度要求
至少 {case.get("min_content_chars",1):,} 字符
'] + for index,fact in enumerate(case.get('required_facts',[]),1): + text='\n或\n'.join(f'{esc(value)}' for value in fact.get('alternatives',[])) + items.append(f'
必需文字组 {index}（任一写法即可）
{text}
') + threshold=case.get('fact_coverage_threshold',1.0) + note=('

全部文字组都须命中。

' if threshold==1 else + f'

文字组覆盖至少达到 {pct(threshold)}。

') + return note+'
'+''.join(items)+'
' + + def machine_contract(case): contract={key:case[key] for key in ('acceptance_contract','composite_acceptance','body_requirements') if case.get(key)} return contract @@ -1405,14 +1475,15 @@ def _quote_field_issue(metrics): return '；'.join(parts)+'。任务要求标签列表保留字段边界，因此仍判未通过' def stable_anchor_evidence(metrics,output): - cards=[] + cards=[];normalized_lines=None anchors_required=metrics.get('stable_anchors_required',True) expected_label='标准答案' if anchors_required else '范围核对文字' for index,item in enumerate(metrics.get('stable_anchor_results',[]),1): - alternatives=item.get('alternatives',[]);expected='\n或\n'.join(alternatives) + alternatives=item.get('alternatives',[]);expected='\n或\n'.join(f'{esc(value)}' for value in alternatives) excerpt='' for alternative in alternatives: - candidate=_exact_evidence_lines(alternative,output) + if normalized_lines is None:normalized_lines=_normalized_evidence_lines(output) + candidate=_exact_evidence_lines(alternative,output,normalized_lines=normalized_lines) if candidate:excerpt=candidate;break passed=bool(item.get('pass'));css='pass' if passed else 'fail' if passed and not excerpt: @@ -1428,11 +1499,11 @@ def stable_anchor_evidence(metrics,output): wanted=expected_fields.get(field,'');got=returned_fields.get(field,'') compact=(normalize_for_match(wanted).replace(' ','')==normalize_for_match(got).replace(' ','')) verdict='命中' if exact[field] else '文字齐全，但分隔丢失' if field=='tags' and compact else '不一致' - rows.append(('',cell(esc(labels[field]))+cell(esc(wanted))+cell(esc(got))+cell(esc(verdict)))) + rows.append(('',cell(esc(labels[field]))+cell(f'{esc(wanted)}')+cell(f'{esc(got)}')+cell(esc(verdict)))) field_detail='

整条记录按字段判定；文字出现不等于列表边界完整。

'+table(['字段','标准答案','实际解析','判定'],rows,metrics=[]) badge='字段边界不合格' - actual=(f'

{esc(excerpt)}

' if excerpt else '

实际返回中未找到可供定位的连续原文。

') - cards.append(f'

稳定文字 {index}

{badge}
{expected_label}

{esc(expected)}

实际返回证据{actual}
{field_detail}
') + actual=(f'

{esc(excerpt)}

' if excerpt else '

实际返回中未找到可供定位的连续原文。

') + cards.append(f'

稳定文字 {index}

{badge}
{expected_label}

{expected}

实际返回证据{actual}
{field_detail}
') if not cards:return '' role_note=('全部参与评分' if anchors_required else '只用于范围核对，不影响是否通过') @@ -1442,14 +1513,18 @@ def stable_anchor_evidence(metrics,output): return note+f'
'+''.join(cards)+'
' def failure_category(r,c): - if not measured(r,c):return '未测：缺少验收证据' + if r['metrics'].get('evaluation_excluded'):return '评测环境异常：统一排除比较' + if not measured(r,c):return '无法判定：最终网页身份缺少证据' if r['metrics'].get('page_identity_required') and r['metrics'].get('page_identity_status')=='unavailable' else '未测：缺少验收证据' if succeeded(r,c):return '—' + # Classification describes an already failed result; hints never veto success. + metrics=r['metrics'] + if (r.get('ok',False) and metrics.get('request_ok',True) + and (metrics.get('blocked_page') or metrics.get('stable_blocked_page'))): + return '返回验证或拦截页面' if c.get('track') == 'risk_control': metrics=r['metrics'] if not metrics.get('request_ok',r.get('ok',False)): return '请求或程序执行失败' - if metrics.get('blocked_page') or metrics.get('stable_blocked_page'): - return '返回验证或拦截页面' content_chars=metrics.get('content_chars') if content_chars == 0: return '完全空返回' @@ -1489,15 +1564,21 @@ def stable_failure_summary(metrics): ('content_too_short','正文短于任务声明的最低长度'), ('duplicate_records','返回记录存在重复'),('incomplete_records','部分记录缺少必需字段'), ('too_many_complete_records','完整记录数超过任务声明范围'), - ('blocked_page','实际返回是访问限制或验证页面'), ] for key,label in labels: if key in errors and label not in parts:parts.append(label) return '；'.join(parts) or '稳定文字或记录结构没有满足任务要求' def reason(r,c): - if not measured(r,c):return '未测：历史调用未记录原文件字节验收证据，不计入任务通过率或失败数。' + if r['metrics'].get('evaluation_excluded'): + return r['metrics']['evaluation_exclusion_reason'] + if not measured(r,c): + if r['metrics'].get('page_identity_required') and r['metrics'].get('page_identity_status')=='unavailable': + return '无法判定：已保存实际返回，但缺少可核实的最终网页身份，不计入通过数、失败数或任务通过率的分母。' + return '未测：历史调用未记录原文件字节验收证据，不计入任务通过率或失败数。' if r['metrics'].get('deadline_overrun'):return '超过该调用登记的端到端时间预算；保留实际耗时和返回内容，但不计为按时交付。' + if r['metrics'].get('forbidden_requirement_hits'): + return '命中题目明确禁止的内容：'+'、'.join(r['metrics']['forbidden_requirement_hits'])+'。' if r['ok'] and 'stable_task_pass' in r['metrics']: m=r['metrics'] if m.get('composite_acceptance_pass') is False: @@ -1537,7 +1618,6 @@ def reason(r,c): m=r['metrics'];parts=[] if m.get('declared_truncation_hits'):parts.append('供应商声明正文已截断') if not m.get('length_pass'):parts.append('正文过短') - if m.get('forbidden_hits'):parts.append('命中排除文本：'+', '.join(m['forbidden_hits'])) missing=[x['id'] for x in m.get('fact_results',[]) if not x['pass']] if missing:parts.append(('条目字段未完整匹配：' if any(x.get('match_scope')=='entry' for x in m.get('fact_results',[])) else '缺少核对点：')+', '.join(missing)) for x in m.get('pattern_results',[]): @@ -1556,16 +1636,19 @@ def stable_checks(m): if len(records)>5:record_table=f'
{record_table}
' return content+record_table+('

未通过原因：'+esc(stable_failure_summary(m))+'。

' if m.get('stable_task_errors') else '') -def ranking(records,cases,track,base='',id=None,compact=False,audit=None): +def ranking(records,cases,track,base='',id=None,compact=False,audit=None,*,diagnostic=False): + excluded={r['case_id'] for r in records if r['metrics'].get('evaluation_excluded')} + records=[r for r in records if not r['metrics'].get('evaluation_excluded')] from .report_components import comparable_body_reference title_scope=track=='challenge' and any('title_quality' in r for r in records if report_family(cases[r['case_id']])==track) - selected={k:c for k,c in cases.items() if report_family(c)==track} + selected={k:c for k,c in cases.items() if report_family(c)==track and k not in excluded} text_track=track in {'web','dynamic','document','challenge','risk_control'} and any(c.get('task_kind','body') in {'body','structure'} for c in selected.values()) groups=defaultdict(list) for r in records: if report_family(cases[r['case_id']])==track:groups[r['provider']].append(r) def sort_key(item): name,rs=item + if diagnostic:return name n=sum(measured(r,cases[r['case_id']]) for r in rs) return (-sum(succeeded(r,cases[r['case_id']]) for r in rs)/n if n else 1,name) rows=[];text_scopes={} @@ -1606,7 +1689,7 @@ def avg(key): else:heads+=['任务通过率','资源命中率','文件取得率','信息通过率'] heads+=['均值','P50','P95','判分完整性'] if audit:heads+=['审查结果'] - note='请求成功率和任务通过率以全部任务为分母。' + note='请求成功率以全部调用为分母；任务通过率以可判定结果为分母，缺少验收证据不计为通过或失败。' if text_track: covered=sum(comparable_body_reference(c) for c in selected.values()) note=f'完整参考覆盖 {covered}/{len(selected)}；缺少参考只影响全文指标，不影响任务结果。有参考时，请求失败计 0；请求成功但任务未通过仍按实际正文评分。' @@ -1622,7 +1705,8 @@ def avg(key): else:note='直接文件必须有字节验收才能判分；任何缺失都会使整份报告标为不可靠。资源、文件、信息三列分别按对应任务计算。' note+=' 判分完整性是已有明确判定的调用数/计划调用数，不是通过率；任何方法未满即表示整份报告不完整，禁止发布。均值、P50、P95 仅统计各方法自身通过项，不能直接视为同题速度排名。' unmeasured=sorted({r['case_id'] for rs in groups.values() for r in rs if not measured(r,cases[r['case_id']])}) - if unmeasured:note+=' 缺少验收证据的案例：'+'、'.join(link(base+'datasets/'+key+'.html',cases[key]['name']) for key in unmeasured)+'。'+link(base+'changes/index.html#file-gap','为什么未测、补测进展 →') + if unmeasured: + note+=' 缺少验收证据的案例：'+'、'.join(link(base+'datasets/'+key+'.html',cases[key]['name']) for key in unmeasured)+'。不计为通过或失败，任务通过率的分母仅包含可判定结果。' if track=='dynamic': missing=sorted({r['provider'] for r in records}-set(groups)) note+=' 各类型按各自任务范围比较，不合并为跨类型总分。' @@ -1630,7 +1714,7 @@ def avg(key): directions={i:('quality_'+str(i),'max') for i in range(1,len(heads)) if heads[i] not in {'均值','P50','P95','判分完整性','审查结果'}} directions.update({i:('latency_'+str(i),'min') for i,h in enumerate(heads) if h in {'均值','P50','P95'}}) rows=comparative_rows(rows,directions) - result=table(heads,rows,id or 'ranking-'+track,True,'ranking',metrics=[] if compact else None) + result=table(heads,rows,id or 'ranking-'+track,True,'ranking',metrics=[] if compact else None,include_index=not diagnostic) if text_scopes and len(set(text_scopes.values()))>1: result='
文字核对覆盖范围
    '+''.join('
  • '+esc(name)+f'：{len(scope)}/{len(selected)} 例；'+esc('、'.join(selected[k]['name'] for k in sorted(scope)) or '无文字核对记录')+'
    • ' for name,scope in sorted(text_scopes.items()))+'
'+result if compact: @@ -1638,7 +1722,7 @@ def avg(key): if title_scope:result+=title_diagnostics(groups,selected,base,id or 'ranking-'+track) result+=matched_latency(groups,cases,base) from .report_components import task_pass_guide - return task_pass_guide(track,base,selected)+'

'+note+'

'+result.replace(''+note+'

'+(result if diagnostic else result.replace('
同题耗时 · 全部方法共同通过子集

'+note+'

'+ (table(['方法','同题均值','同题 P50'],rows,metrics=[])+ '

'+links+'

' if rows else '')+'' def overview_all_call_latency(records,cases): + excluded={r['case_id'] for r in records if r['metrics'].get('evaluation_excluded')} + records=[r for r in records if r['case_id'] not in excluded] + cases={key:case for key,case in cases.items() if key not in excluded} names=sorted({r['provider'] for r in records}) keys={(r['provider'],r['case_id']) for r in records} if len(keys)!=len(records) or any((name,cid) not in keys for name in names for cid in cases): @@ -1665,9 +1752,20 @@ def overview_all_call_latency(records,cases): times=[r['elapsed_ms'] for r in records if r['provider']==name] rows.append(('',cell(link('providers/'+name+'.html',provider_label(name)))+cell(str(len(times)))+ cell(ms(statistics.mean(times)))+cell(ms(percentile(times,.5)))+cell(ms(percentile(times,.95))))) - return (f'

每种方法均使用同一组 {len(cases)} 个任务，包含通过、未通过、错误和超时。耗时是单次调用从输入到返回的时间，不包含调用之间的统一冷却间隔。各方法串行执行，仍受本地与远程环境、网络和产品内部限制影响。

'+ + return ('

'+model_text(f'每种方法均使用同一组 {len(cases)} 个任务，包含通过、未通过、错误和超时。耗时是单次调用从输入到返回的时间，不包含调用之间的统一冷却间隔。每种方法内部按案例依次执行；方法之间的并发安排和限制见调用链检查。耗时仍受本地与远程环境、网络和产品内部限制影响。',f'Each method uses the same {len(cases)} tasks, including passes, failures, errors and timeouts. Latency covers one call from input to output, excluding cooldown intervals between calls. Each method executes cases sequentially; method concurrency and limits are documented in the call-chain audit. Local/remote execution, network conditions and native product limits still affect latency.')+'

'+ table(['方法','调用数','全部调用均值','全部调用 P50','全部调用 P95'],rows,'all-call-latency-table',metrics=[])) + +def result_ledger_section(out, records, prefix=''): + path=out/'local-evidence/result-ledger.json' + if not path.is_file():return '' + from .result_ledger import validate_ledger + ledger=validate_ledger(json.loads(path.read_text()),records,source=out) + rows=[('',cell(esc(row['id']))+cell(str(row['before_passes']))+cell(str(row['added_passes']))+ + cell(str(row['removed_passes']))+cell(str(row['totals']['pass']))+ + cell(str(row['totals']['excluded']))) for row in ledger['rounds']] + return section('评分修正账目','

每轮增加、减少与最终通过数均从逐条状态记录计算；排除结果保留原始调用，不计入比较分母。

'+table(['阶段','修正前通过','新增通过','减少通过','修正后通过','排除比较'],rows,metrics=[])+link(prefix+'local-evidence/result-ledger.json','下载逐条增减账目'),'result-ledger') + def overview_matched_latency(records,cases): from .report_explainer import _article_speed_population names=sorted({r['provider'] for r in records}) @@ -1731,14 +1829,19 @@ def title_diagnostics(groups,selected,base,identifier): '

该诊断回答“已经返回的条目标题是否完整、干净”，不衡量榜单召回率、实时性或整页正文质量。

'+steps +table(['方法','标题完整度','标题纯净度','标题 F1','标题参考覆盖（可评分 / 返回）'],rows,identifier+'-titles',True,metrics=[])+ '') -def overall_ranking(records,cases,audit=None,*,call_budget_seconds=None): +def overall_ranking(records,cases,audit=None,*,diagnostic=False,call_budget_seconds=None): + excluded={r['case_id'] for r in records if r['metrics'].get('evaluation_excluded')} + records=[r for r in records if not r['metrics'].get('evaluation_excluded')] + cases={k:c for k,c in cases.items() if k not in excluded} + has_unknown=any(not measured(r,cases[r["case_id"]]) for r in records) groups=defaultdict(list) for r in records:groups[r['provider']].append(r) rows=[] for name,rs in groups.items(): n=sum(measured(r,cases[r['case_id']]) for r in rs) passed=sum(succeeded(r,cases[r['case_id']]) for r in rs) - values=cell(link(f'providers/{name}.html',provider_label(name)))+cell(rate(sum(bool(r['ok']) for r in rs),len(rs)),sum(bool(r['ok']) for r in rs)/len(rs))+cell(rate(passed,n),passed/n if n else -1) + values=cell(link(f'providers/{name}.html',provider_label(name)))+cell(rate(sum(bool(r['ok']) for r in rs),len(rs)),sum(bool(r['ok']) for r in rs)/len(rs))+cell(rate(passed,n)+(f'
{passed}/{n}' if has_unknown else ''),passed/n if n else -1) + if has_unknown:values+=cell(str(len(rs)-n)) for track,_ in active_tracks(cases): subset=[r for r in rs if report_family(cases[r['case_id']])==track and measured(r,cases[r['case_id']])] values+=cell(f'{sum(succeeded(r,cases[r["case_id"]]) for r in subset)}/{len(subset)}',sum(succeeded(r,cases[r['case_id']]) for r in subset)/len(subset) if subset else -1) @@ -1748,29 +1851,36 @@ def overall_ranking(records,cases,audit=None,*,call_budget_seconds=None): values+=cell(ms(percentile(times,.95)),percentile(times,.95)) if audit:values+=review_cell(audit,rs) rows.append((passed,name,('',values))) - rows=[row for _,_,row in sorted(rows,key=lambda x:(-x[0],x[1]))] + rows=[row for _,_,row in sorted(rows,key=lambda x:x[1] if diagnostic else (-x[0],x[1]))] tracks=active_tracks(cases) - heads=['方法','请求成功率','综合任务通过率']+[label for _,label in tracks]+['总耗时均值','总耗时 P50','总耗时 P95'] + heads=['方法','请求成功率','综合任务通过率']+(['无法判定'] if has_unknown else [])+[label for _,label in tracks]+['总耗时均值','总耗时 P50','总耗时 P95'] if audit:heads+=['审查结果'] case_count=len(cases);track_count=len(tracks);track_label=track_count_label(track_count) if case_count==150 and track_count==6: intro='

总榜口径：每种方法都执行同一组 150 个任务，每题计一次通过或失败，直接相加；耗时单列展示，不参与通过率排序。总榜适合看整体交付覆盖，具体能力仍应回到六个分榜。

' else: intro=(f'

总榜口径：每种方法都执行当前评测集计划的 {case_count} 个任务，每题计一次通过或失败，直接相加；耗时单列展示，不参与通过率排序。总榜适合看整体交付覆盖，具体能力仍应回到 {track_label} 个分榜。

') + if has_unknown: + intro=f'

总榜口径：每种方法都调用同一组 {case_count} 个任务。通过率为通过数 / 可判定数；无法判定数量另列，不计入通过数或失败数。各方法可判定分母不同，比例只是已能验收的范围，不能代表全部任务的确定通过率。耗时不参与通过率排序。

' help=('
总榜指标说明
' - f'
综合任务通过率
{case_count} 个任务中满足各自验收规则的数量；每题等权。
' + f'
综合任务通过率
{case_count} 个计划任务中的通过数 / 可判定数；每题等权，无法判定结果不参与分母。
' f'
{track_label} 类通过数
各分榜通过数 / 本类任务数，用于定位总分来自哪些能力。
' f'
总耗时均值 / P50 / P95
每种方法在同一组 {case_count} 个计划任务上的端到端耗时，包含通过与失败；三项不参与通过率排名。主矩阵调用跨日期、跨批次，不等同于同窗受控速度比较。
' '
审查结果
独立盲审覆盖数、审查方法和未决数。
' '
') - directions={i:('acceptance_'+str(i),'max') for i in range(1,3+len(tracks))} - directions.update({i:('all_latency_'+str(i),'min') for i in range(3+len(tracks),6+len(tracks))}) + directions={i:('acceptance_'+str(i),'max') for i in range(1,3+len(tracks)+int(has_unknown))} + directions.update({i:('all_latency_'+str(i),'min') for i in range(3+len(tracks)+int(has_unknown),6+len(tracks)+int(has_unknown))}) + if has_unknown:directions.pop(3,None) rows=comparative_rows(rows,directions) timing_note='

总耗时按每题所有物理调用累计，包含失败重试和重试等待；每题仍只计一个样本。' if call_budget_seconds is not None: timing_note+=f'每次物理调用预算为 {call_budget_seconds:g} 秒，重试后的累计耗时可能超过该预算。' timing_note+='

' - return intro+timing_note+help+table(heads,rows,'ranking-overall',True,'ranking',metrics=[]).replace('
至少需要两种方法才能比较任务交集。

' a=ordered[0];b=ordered[1] options=lambda selected:''.join(f'' for name in ordered) - tracks=''+''.join(f'' for key,label in TRACKS.items()) + tracks=''+''.join(f'' for key,label in REPORT_TRACKS.items()) rows=comparison_rows(cases,bycase,names) checks=''.join(f'' for name in ordered) return (f'

同题结果组合

以方法 A 为基准，可选择一个或多个方法共同比较；方法名可直达该次完整证据。

' @@ -1825,12 +1935,12 @@ def case_attrs(r,c): resources='na' if not c.get('expected_resources') else 'pass' if r['ok'] and r['metrics'].get('resource_pass') else 'fail' stage='' if succeeded(r,c) or not measured(r,c) else 'task' if r['ok'] else 'request' category='' if not stage else failure_category(r,c) - result='unmeasured' if not measured(r,c) else 'pass' if succeeded(r,c) else 'fail' + result='excluded' if r['metrics'].get('evaluation_excluded') else 'unmeasured' if not measured(r,c) else 'pass' if succeeded(r,c) else 'fail' return dataset_attrs(c)+f' data-provider="{esc(r["provider"])}" data-transport="{"pass" if r["ok"] else "fail"}" data-result="{result}" data-content="{content}" data-resources="{resources}" data-failure-stage="{stage}" data-failure-category="{esc(category)}" data-search="{esc(str(c.get("id",r.get("case_id","")))+" "+str(c.get("name",""))+" "+str(c.get("url",""))+" "+reason(r,c))}"' def attempt_row(r,c,base='../',*,label=None): state=result_label(r,c) - values=cell(link(base+'attempts/'+r['attempt_id']+'.html',c['name'] if label is None else label))+cell(case_track_label(c))+cell(f'{state}')+cell(esc(failure_category(r,c)))+cell(ms(r['elapsed_ms'])) + values=cell(link(base+'attempts/'+r['attempt_id']+'.html',c['name'] if label is None else label))+cell(case_track_label(c))+cell(f'{state}')+cell(esc(failure_category(r,c)))+cell(ms(r['elapsed_ms'])) values+=cell(f'{esc(r.get("run_id") or execution_id(r))}') return case_attrs(r,c),values @@ -1839,22 +1949,37 @@ def local_call_chain_section(dataset, prefix="../"): if not audit:return '' rows=[] for item in audit.get('methods',[]): - rows.append(('',cell(link(prefix+'providers/'+item['provider']+'.html',provider_label(item['provider'])))+cell(str(item['calls']))+cell(str(item['request_ok']))+cell(str(item['task_pass']))+cell(str(item['logs']))+cell(esc(item['native_log_note']))+cell(esc(item['assessment'])))) - findings=''.join('

'+esc(item['text'])+'

' for item in audit.get('findings',[])) - findings+='

'+' · '.join(link(prefix+item['path'],item['label']) for item in audit.get('evidence_links',[]))+'

' - return section('调用链与日志检查','

'+esc(audit.get('summary',''))+'

'+table(['方法','调用数','请求成功','题目通过','诊断日志','原生日志范围','检查结论'],rows,metrics=[])+findings,'call-chain-audit') + note=re.sub(r'；旧\s*\d+\s*条不计入本报告。$', '。', item['native_log_note']).replace('新批次启用', '本轮启用') + rows.append(('',cell(link(prefix+'providers/'+item['provider']+'.html',provider_label(item['provider'])))+cell(esc(note)))) + intro='

每次调用都保存了测试程序的调用记录：何时开始、传入什么参数、何时返回，以及是否发生错误。各案例详情中的“调用方事件”展示这些记录。

“产品与依赖库日志”展示产品或底层库实际输出的运行信息。云端服务没有开放的内部日志，不会用测试程序的记录代替。下表说明各方法留下了哪些日志。

' + return section('本轮保存了哪些日志',intro+table(['方法','产品与依赖库日志范围'],rows,metrics=[])+'

完整日志在各案例详情中查看。日志可以帮助定位问题，但没有记录的内部过程不能据此推断；日志文件存在，也不等于产品的全部内部过程都有记录。

','call-chain-audit') + + +def local_pass_rate_ranking(dataset): + return dataset.get('comparison',{}).get('presentation',{}).get('ranking') == 'saved_task_pass_rate' + def local_audit_banner(dataset): if dataset.get('comparison',{}).get('publication_state')!='local_audit_only_not_for_release': return '' if dataset.get('comparison',{}).get('kind')=='uniform_native_local_measurement': - return '
本地实测版

全部方法使用同一组已提交案例和判分规则，每例一次调用，统一45秒外部时限，不设逐例等待或提取参数，不重试。结果为机器判定，未进行独立人工审核；本报告仅供本地查看，不代表正式发布。

' + if local_pass_rate_ranking(dataset): + return '

'+model_text('按本次所列任务的通过率排序；未完成仓库认证流程。测试日期和核验范围见详情。','Ordered by pass rate on the listed tasks; repository certification is incomplete. See details for test dates and verification scope.')+'

' + note='已有复核记录；覆盖和裁定依据见核验页面。' if dataset['comparison'].get('independent_review') else '结果为机器判定，未进行独立人工审核。' + return '' return '
本地审核版

本页合并两个不同日期的 Moli 调用批次，保留每例的真实调用时间。本次仅更换报告展示代码，没有新增 Moli 调用，也不代表正式发布。

' from .methods import with_measured_method_labels @with_measured_method_labels -def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english:bool=False,authored_interpretation:bool=True,presentation_only:bool=False,detail_attempt_ids:set[str]|None=None): +def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english:bool=False,authored_interpretation:bool=True,presentation_only:bool=False,show_local_status_banners:bool=True,detail_attempt_ids:set[str]|None=None,frozen_scores:bool=False,english_routes=None): + from .publication_privacy import public_render_views + _public_views=public_render_views(out,records) + if english_routes is not None and not presentation_only: + raise ValueError('Scoped English refresh requires presentation_only=True') + for _record in records: + if _record['attempt_id'] in _public_views: + _record.setdefault('provider_metadata',{})['publication_views']=_public_views[_record['attempt_id']] global _RELIABILITY_BANNER,_ENGLISH_GENERATED if detail_attempt_ids is not None: if not presentation_only: @@ -1862,13 +1987,30 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english unknown=set(detail_attempt_ids)-{row['attempt_id'] for row in records} if unknown:raise ValueError('Unknown detail refresh attempts: '+', '.join(sorted(unknown))) _ENGLISH_GENERATED=include_english + local_snapshot=dataset.get('comparison',{}).get('publication_state')=='local_audit_only_not_for_release' + diagnostic=local_snapshot and not local_pass_rate_ranking(dataset) + if local_snapshot and not show_local_status_banners: + raise ValueError('Uncertified local diagnostic status cannot be hidden') from .reliability import assess_reliability reliability=assess_reliability(dataset,records) if reliability['complete']: _RELIABILITY_BANNER='' + elif (dataset.get('comparison',{}).get('kind')=='uniform_native_local_measurement' + and dataset.get('comparison',{}).get('publication_state')=='local_audit_only_not_for_release' + and not any(reliability[key] for key in ('missing','extra','duplicates'))): + excluded=[r for r in records if r['metrics'].get('evaluation_excluded')] + if len(excluded)==len(reliability['unjudged']): + _RELIABILITY_BANNER='
'+model_text( + f'{len(excluded)} 条结果因已确认的评测环境异常统一排除比较；原始调用和输出保留。', + f'{len(excluded)} results are excluded across all methods for a verified evaluation-environment defect; original calls and outputs are retained.')+'
' + else: + _RELIABILITY_BANNER='' else: _RELIABILITY_BANNER='' - _RELIABILITY_BANNER+=local_audit_banner(dataset) + if not show_local_status_banners and dataset.get('comparison',{}).get('publication_state')=='local_audit_only_not_for_release' and not any(reliability[key] for key in ('missing','extra','duplicates')): + _RELIABILITY_BANNER='' + if show_local_status_banners: + _RELIABILITY_BANNER+=local_audit_banner(dataset) known=dataset.get('comparison',{}).get('call_chain_audit',{}).get('known_scoring_false_negatives',[]) if known: _RELIABILITY_BANNER+='' @@ -1897,38 +2039,48 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english for r in records:bycase[r['case_id']].append(r);byprovider[r['provider']].append(r) cards='' for track,label in tracks: - cards+=overview_card(records,cases,track,label,names) - home='
'+hero('Web Fetch 评测报告',f'{len(cases)} 个案例 · {len(names)} 种方法 · {track_label} 类分榜与一个综合通过率总榜。') - if dataset.get('comparison',{}).get('call_chain_audit'): - source_note='本轮未重新核验来源。' if dataset.get('comparison',{}).get('source_reverified') is False else '' - home+=f'

{len(cases)} 个案例 · {len(records)} 次调用，机器判分。'+source_note+link('guide/index.html#call-chain-audit','查看调用链问题和日志留存范围')+'

' + cards+=overview_card(records,cases,track,label,names,diagnostic=diagnostic) + description=(f'{len(cases)} 个案例 · {len(names)} 种方法 · 本地未认证诊断快照。' if diagnostic else f'{len(cases)} 个案例 · {len(names)} 种方法 · {track_label} 类分榜与一个综合通过率总榜。') + if local_pass_rate_ranking(dataset): + description=model_text(f'{len(cases)} 个案例 · {len(names)} 种方法 · 按本轮任务通过率排序。',f'{len(cases)} cases · {len(names)} methods · Ordered by task pass rate for this sample.') + home='
'+hero('Web Fetch 评测报告',description) from .report_explainer import article_title,render_explainer,supports_authored_explainer TITLE='本轮调用链检查与日志范围' if dataset.get('comparison',{}).get('call_chain_audit') else article_title(records) if authored_interpretation and supports_authored_explainer(cases) and not all(row.get('provider_type') == 'fixture' for row in records) else '本报告的任务范围与验收规则' if authored_interpretation and dataset.get('comparison',{}).get('report_interpretation'): binding=dataset['comparison']['report_interpretation'] TITLE=json.loads(contained_path(out,binding['path']).read_text())['title'] reading_note='从分数差异、失败案例和实现机制，理解如何选择抓取方法。' if authored_interpretation else '查看本报告的任务范围、结果数量与验收规则。' - home+='' + if dataset.get('comparison',{}).get('call_chain_audit'): + reading_url='guide/index.html' + home+='' + if not dataset.get('comparison',{}).get('call_chain_audit'):home+='' if dataset.get('comparison',{}).get('call_chain_audit'): TITLE='本轮调用链检查与日志范围' - guide=local_call_chain_section(dataset)+section('如何读取本轮结果','

本轮按保存的实际返回进行机器判分，比例只对应本报告列出的案例与配置。日志可用于复查调用和判分；未经过独立人工审核，不能将失败一概归因于产品内部。

'+link('../analysis/index.html','逐次结果与筛选')+' · '+link('../methodology/index.html','题目与计时规则')+' · '+link('../reproduce/index.html','配置、版本和复现说明')+'

') + guide=section('如何读取本轮结果','

比例只对应本报告列出的案例与配置。先看总榜和分榜，再打开具体案例，对照验收要求与实际返回。请求成功只表示方法正常返回；任务通过表示返回内容满足该案例的全部要求。

评分规则与计时范围见“评测方法”，检查和修正记录见“核验记录”，方法版本、固定配置与日志范围见“环境与复现”。

') else: guide=render_explainer(out,cases,records,table=table,cell=cell,link=link,authored=authored_interpretation) + from .report_reading import render_archived_reading + guide=render_archived_reading(out) or guide write(out,'guide/index.html','报告解读',guide) panels=[] for track,label in tracks: selected=[c for c in cases.values() if report_family(c)==track] - coverage=f'{len(selected)} 个案例 · {len({r["provider"] for r in records if report_family(cases[r["case_id"]])==track})} 种已测方法' - panels.append((track,label,f'

{label}

{coverage}
'+ranking(records,cases,track,compact=True,audit=audit)+f'

{link("datasets/index.html#track-"+track,"案例与参考 →")}

')) + coverage=f'{len(selected)} 个案例 · {len({r["provider"] for r in records if cases[r["case_id"]].get("track","web")==track})} 种已测方法' + panels.append((track,label,f'

{label}

{coverage}
'+ranking(records,cases,track,compact=True,audit=audit,diagnostic=diagnostic)+f'

{link("datasets/index.html#track-"+track,"案例与参考 →")}

')) + overall_title=('任务判定汇总（非排名）' if diagnostic else model_text('本轮任务通过率排名','Task pass-rate ranking for this sample') if local_pass_rate_ranking(dataset) else '综合通过率总榜') budgets={m['timeout_seconds'] for m in manifests if m.get('timeout_seconds') is not None} - home+=section('综合通过率总榜',overall_ranking(records,cases,audit,call_budget_seconds=next(iter(budgets)) if len(budgets)==1 else None),'overall-ranking') + home+=section(overall_title,overall_ranking(records,cases,audit,diagnostic=diagnostic,call_budget_seconds=next(iter(budgets)) if len(budgets)==1 else None),'overall-ranking') + if diagnostic: + home+=result_ledger_section(out,records) + low=[r for r in records if not r['metrics'].get('evaluation_excluded') and succeeded(r,cases[r['case_id']]) and cases[r['case_id']].get('ground_truth_main_content') and r['metrics'].get('word_precision') is not None and r['metrics']['word_precision']<.3] + home+=section('通过结果的内容诊断',f'

{len(low)} 条通过结果的参考词精确率低于 30%。此比例用于定位检查对象，不是统一失败线；通过不保证干净正文，正文参考的范围和时间也会影响该指标。

','coverage-quality') if dataset.get('comparison',{}).get('kind')=='uniform_native_local_measurement': home+=section('同一任务集合的全部调用耗时',overview_all_call_latency(records,cases),'all-call-latency') home+=section(f'{track_label} 类任务概览',review_summary(out)+'

调用完整：'+str(reliability['expected_calls'])+'/'+str(reliability['expected_calls'])+' 次均有结果。

只在同一任务类别内比较。

'+cards+'
','overview') home+=section('全部方法共同成功任务的耗时',overview_matched_latency(records,cases),'overall-matched-latency') if (out/'resource-cohort/resource-comparison.json').is_file(): home+=section('CPU、内存与网络',overview_resources(out),'overall-resources') - home+=section(f'{track_label} 类分榜','

分榜用于判断具体能力。默认按任务通过率排序，点击列名可切换；质量分数与通过率分别计算。

'+tabs('home-rankings',panels),'results') + home+=section('按任务类别查看诊断结果' if diagnostic else f'{track_label} 类分榜',('

方法按名称排列，不提供名次。任务判定、内容质量和耗时分别展示。

' if diagnostic else '

分榜用于判断具体能力。默认按任务通过率排序，点击列名可切换；质量分数与通过率分别计算。

')+tabs('home-rankings',panels),'results') write(out,'index.html','评测报告',home+'
') controls=filter_controls([('track','类型',tracks),('category','类别',[(x,x) for x in dict.fromkeys(c['category'] for c in cases.values())])]) rows=[] @@ -2000,7 +2152,7 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english for r in detail_records: c=cases[r['case_id']];m=r['metrics'];ref=reference_text(c) output=contained_path(out,r['artifact_path']).read_text(encoding='utf-8',errors='replace') - rows=[('',cell(k)+cell(v)) for k,v in [('任务 ID',f'{esc(c["id"])}'),('方法',link('../providers/'+r['provider']+'.html',provider_label(r['provider']))),('请求','成功' if r['ok'] else '失败'),('任务结果','通过' if succeeded(r,c) else '未通过'),('HTTP 状态',esc(r.get('status_code'))),('耗时',ms(r['elapsed_ms'])),('全文完整度 / 纯净度 / F1',f'{pct(m.get("word_recall"))} / {pct(m.get("word_precision"))} / {pct(m.get("word_f1"))}'),('输出字符',f'{len(output):,}'),('输出校验值',f'{esc(m.get("content_sha256"))}')]] + rows=[('',cell(k)+cell(v)) for k,v in [('任务 ID',f'{esc(c["id"])}'),('方法',link('../providers/'+r['provider']+'.html',provider_label(r['provider']))),('请求','成功' if r['ok'] else '失败'),('任务结果',result_label(r,c)),('HTTP 状态',esc(r.get('status_code'))),('耗时',ms(r['elapsed_ms'])),('全文完整度 / 纯净度 / F1',f'{pct(m.get("word_recall"))} / {pct(m.get("word_precision"))} / {pct(m.get("word_f1"))}'),('输出字符',f'{len(output):,}'),('输出校验值',f'{esc(m.get("content_sha256"))}')]] if not c.get('ground_truth_main_content'): rows[6]=('',cell('全文完整度 / 纯净度 / F1')+cell('— / — / —（全文指标无完整参考，未评测）')) else: @@ -2012,6 +2164,11 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english if c.get('task_kind')=='file': rows[7]=('',cell('文本输出字符')+cell(f'{len(output):,}（文件字节另列）')) rows[8]=('',cell('文本输出校验值')+cell(f'{esc(m.get("content_sha256"))}')) + _answer_view=r.get('provider_metadata',{}).get('publication_views',{}).get('scored_answer') + if _answer_view and _answer_view['original_sha256']!=_answer_view['published_sha256']: + rows[7]=('',cell(model_text('公开副本字符','Public-view characters'))+cell(f'{len(output):,}')) + rows[8]=('',cell(model_text('评分时输出校验值','Original scored output hash'))+cell(f'{esc(m.get("content_sha256"))}')) + rows.append(('',cell(model_text('公开副本校验值','Public-view hash'))+cell(f'{esc(_answer_view["published_sha256"])}'))) if 'content_check_total' in m: rows.append(('',cell('文字核对覆盖')+cell(pct(m.get('content_check_coverage'))+f' · {m["content_check_hits"]}/{m["content_check_total"]} 组'))) if c.get('task_kind')=='resource': @@ -2095,8 +2252,8 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english hero_meta=(f'任务 ID：{esc(c["id"])}
{esc(task_instruction(c))}' f'方法：{esc(provider_label(r["provider"]))}' f'运行 ID：{esc(r.get("run_id") or execution_id(r))}') - summary=(f'
' - f'{esc(state)}

{esc(reason(r,c) if passed else (r.get("diagnosis") or {}).get("category") or "任务规则未满足")}

' + summary=(f'
' + f'{esc(state)}

{esc(reason(r,c) if passed or not measured(r,c) else (r.get("diagnosis") or {}).get("category") or "任务规则未满足")}

' f'

方法：{link("../providers/"+r["provider"]+".html",provider_label(r["provider"]))} · ' f'请求：{"成功" if r["ok"] else "失败"} · 耗时：{ms(r["elapsed_ms"])} · {actual_summary}

') summary+=f'

{esc(route_label)}

' @@ -2104,7 +2261,7 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english '查看本次完整实际返回' '查看逐项判定' '查看运行日志与证据

') - if r.get('diagnosis') and not passed: + if r.get('diagnosis') and not passed and measured(r,c): d=r['diagnosis'];s=d.get('independent_source_check') or {} quote_issue=_quote_field_issue(m) facts=[('失败分类','字段边界丢失' if quote_issue else d['category']), @@ -2113,21 +2270,28 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english if s:facts.extend([('独立源页复查',f"{s.get('checked_at','')} · HTTP {s.get('status','未取得')} · {s.get('title') or s.get('error','')}"),('复查边界',d['source_check_boundary'])]) summary+='
查看失败分类与归因边界'+table(['项目','依据'],[('',cell(esc(k))+cell(esc(v))) for k,v in facts],metrics=[])+"
" body=hero(esc(c['name']),hero_meta)+section('结论',summary,'overview') + from .report_quality import render_quality_observations + body+=section('目标覆盖与附加内容',render_quality_observations(r,c,output,esc=esc),'quality-diagnostics') reviewed=audited_attempts.get(r['attempt_id']) if reviewed: - independent='通过' if reviewed.get('agent_verdict')=='pass' else '未通过' - final='通过' if reviewed.get('final_verdict')=='pass' else '未通过' + independent={'pass':'通过','fail':'未通过','insufficient':'证据不足'}.get(reviewed.get('agent_verdict'),'未记录') + final={'pass':'通过','fail':'未通过','unavailable':'无法判定'}.get(reviewed.get('final_verdict'),'未裁定；保留现有评分') review_rows=[ ('独立核验结论',independent), ('直接证据',reviewed.get('agent_evidence') or '—'), ('最终结论',final), ] - if reviewed.get('adjudication'): - review_rows.append(('分歧裁定',reviewed['adjudication'])) + if reviewed.get('adjudication_evidence'): + proof=reviewed['adjudication_evidence'] + review_rows.append(('分歧裁定',proof.get('summary_zh',reviewed['adjudication']))) body+=section('独立结果核验',table( ['项目','核验记录'], - [('',cell(esc(label))+cell(esc(value))) for label,value in review_rows], + [('',cell(esc(label))+cell( + model_text(value,reviewed.get('adjudication_evidence',{}).get('reason_en')) if label=='分歧裁定' + else ''+esc(value)+'' if label=='直接证据' else esc(value))) for label,value in review_rows], metrics=[]),'independent-review') + if reviewed.get('adjudication_evidence',{}).get('summary_zh'): + body+='
'+model_text('完整裁定记录（原文）','Full adjudication record (original)')+'
'+esc(reviewed['adjudication'])+'
' failure_review=failure_reviews.get(r['attempt_id']) if failure_review: cause_body='

'+esc(failure_review['category_label'])+'

'+esc(failure_review['reason'])+'

' @@ -2141,6 +2305,15 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english if not (c.get('ground_truth_main_content') or c.get('reference_answer')) and c.get('task_kind','body')=='body': reference_heading='验收规则（无全文参考）' source_meta=f'{link(reference_download,"下载验收规则")}；全文完整度、纯净度和 F1 不适用。' + if (c.get('track','web')=='web' and c.get('task_kind','body')=='body' + and c.get('required_facts') and not c.get('acceptance_contract')): + rules={key:c[key] for key in ('min_content_chars','expected_any','required_facts', + 'fact_coverage_threshold','required_patterns','body_requirements') if key in c} + source_body=required_text_answer_panel(c) + if c.get('ground_truth_main_content') or c.get('reference_answer'): + source_body+='

完整参考正文

'+f'
{esc(ref)}
' + source_body+=('
查看原始验收规则' + f'
{esc(json.dumps(rules,ensure_ascii=False,indent=2))}
') delivered=r.get('provider_metadata',{}).get('file_evaluation',{}).get('files',[]) if 'stable_task_pass' in m and c.get('acceptance_contract'): reference_download='../contracts/'+c['id']+'.json' @@ -2157,7 +2330,11 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english corruption=(f'

实际输出存在解码损坏：' f'含 {m.get("replacement_character_count",0):,} 个 �，占 {pct(m.get("replacement_character_ratio"))}。' '这些字符在保存前已丢失，不能从当前 Markdown 还原。

') + answer_view=r.get('provider_metadata',{}).get('publication_views',{}).get('scored_answer') output_note=(f'{len(output):,} 字符 · '+link('../'+r['artifact_path'],'下载原始输出')) + if answer_view and answer_view['original_sha256']!=answer_view['published_sha256']: + output_note=(f'{len(output):,} 字符 · '+link('../'+answer_view['path'],'下载公开副本','Download public view')+' · '+link('../'+answer_view['receipt'],'查看脱敏记录','View redaction record')+ + '
'+model_text('目录或敏感值已隐藏；成绩来自评分原件。原件与公开副本的校验值见运行字段。','Paths or sensitive values are hidden; scores come from the original. Original/public hashes are listed in the run fields.')) if readable_output(output)!=output:output_note+=' · 控制字节在页面中显示为 \\xNN' actual_body=(corruption+f'
{esc(readable_output(output))}
' if output else f'
{esc(empty_output_reason(r,c,output))}
') @@ -2165,7 +2342,7 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english reference_heading='目标文件' expected_hash=c.get('source_verification',{}).get('file_sha256') or '未登记' source_meta='本任务验收原始文件字节，不要求 Markdown。' - source_body=(f'
文件要求
{esc(ref)}
' + source_body=(f'
文件要求
{esc(ref)}
' f'
目标 SHA-256{esc(expected_hash)}
') if delivered: actual_rows=[('',cell(esc(f.get('mime') or '未知格式'))+cell(f'{f.get("bytes",0):,}')+ @@ -2177,7 +2354,10 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english output_note='实际交付 0 个文件' reference_scope=('' if not (c.get('track')=='document' and 0 < len(ref) < 100) else f'

参考范围：这是最小文档解析样本。冻结源文件的全部可提取正文共 {len(ref):,} 个字符，标准答案没有省略。它只验证短文档文字提取，不用于证明多页、表格或附录完整性。

') - evidence=('

左边是判定时使用的标准答案或明确验收规则，右边是该次调用原样保存的完整输出。两栏都是真实内容，不是摘要；长内容可在栏内滚动。

'+reference_scope+ + evidence_intro='左边是判定时使用的标准答案或明确验收规则，右边是该次调用原样保存的完整输出。两栏都是真实内容，不是摘要；长内容可在栏内滚动。' + if answer_view and answer_view['original_sha256']!=answer_view['published_sha256']: + evidence_intro=model_text('左边是判定时的标准答案或验收规则，右边是本次输出的公开副本，目录或敏感值已隐藏。成绩保留评分原件的结论；副本与原件分别标明校验值。长内容可在栏内滚动。','The left pane contains the scoring reference or acceptance rules. The right pane contains the public output view with paths or sensitive values hidden. Scores retain the original verdict; original and public hashes are identified separately. Long content scrolls within each pane.') + evidence=('

'+evidence_intro+'

'+reference_scope+ f'

{esc(reference_heading)}

{source_meta}

{source_body}
' f'

实际返回

{output_note}

{actual_body}
') body+=section('标准答案与实际返回',evidence,'evidence') @@ -2204,13 +2384,15 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english body+=section('为什么这样判',caveat+checks_html,'decision') if m.get('body_region_results'): scoring_text=_body_scoring_text(r,output) - region_rows,region_cards=body_region_rows(c,m,scoring_text);rules=c.get('body_requirements',{}) + region_rows,region_cards=body_region_rows(c,m,scoring_text,frozen_scores=frozen_scores);rules=c.get('body_requirements',{}) required=rules.get('minimum_region_passes',len(rules.get('regions',[]))) region_count=len(rules.get('regions',[])) region_intro=(f'
正文判定：本例分为 {region_count} 段。' f'每段至少达到 {pct(rules.get("minimum_floor",rules.get("minimum_coverage")))}，' f'其中至少 {required} 段达到 {pct(rules.get("minimum_coverage"))}。' '先看下表；需要核对原文时，再展开下面的左右对照。
') + if frozen_scores: + region_intro+='

'+model_text('本节沿用该次评分保存的数值和片段；公开片段已隐藏敏感值。未保存的历史片段不使用当前规则补算，公开副本也不用于重算原件指标。','This section retains the values and passages saved with the score. Sensitive values are hidden in public passages. Missing historical passages are not reconstructed with current rules, and public copies are not used to recalculate original metrics.')+'

' body+=section('正文是否抓全',region_intro+table(['标准答案位置','标准片段数','命中片段数','覆盖率','判定'],region_rows,metrics=[])+ '
查看各段完整标准正文和计分行'+''.join(region_cards)+'
','body-coverage') if title_details:body+=section('标题文字质量',title_details) @@ -2234,18 +2416,21 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english write(out,'attempts/'+r['attempt_id']+'.html',c['name']+' · '+r['provider'],body) controls=filter_controls([('provider','方法',[(n,provider_label(n)) for n in names]),('track','类型',tracks),('result','结果',[('pass','任务通过'),('fail','任务失败')])]+failure_filters(records,cases)) comparison=section('方法组合对比','

按同一任务逐项比较，不用各方法不同的成功样本拼接结论。成功与失败均沿用主榜的程序判定；点击方法可查看标准答案、实际返回和原因。

'+method_comparisons(cases,bycase,names),'method-comparison') - write(out,'analysis/index.html','结果分析',hero('结果分析','先比较方法在同一任务上的成败组合，再筛选全部执行记录。')+comparison+section('全部执行记录',controls+table(['方法']+headers,[(case_attrs(r,cases[r['case_id']]),cell(link('../providers/'+r['provider']+'.html',provider_label(r['provider'])))+attempt_row(r,cases[r['case_id']])[1]) for r in records],'filtered-table'))) + write(out,'analysis/index.html','结果分析',hero('结果分析','先比较方法在同一任务上的成败组合，再筛选全部执行记录。')+comparison+section('全部执行记录',controls+table(['方法']+headers,[(case_attrs(r,cases[r['case_id']]),cell(link('../providers/'+r['provider']+'.html',provider_label(r['provider'])))+attempt_row(r,cases[r['case_id']])[1]) for r in records],'filtered-table',cls='all-method-results'))) write_support(out,records,manifests,authored_interpretation=authored_interpretation) from .report_changes import render_changes if records and all(row.get('provider_type') == 'fixture' for row in records): write(out,'changes/index.html','示例说明',hero('合成示例','本报告由本地测试适配器生成。没有正式评测发布记录。')) elif not authored_interpretation: - write(out,'changes/index.html','本轮结果说明',hero('本轮结果说明','本报告从保存的调用、评分与证据构建，不属于仓库历史报告的发布版本。')+section('查看本轮证据','

'+link('../analysis/index.html','逐任务结果与证据')+' · '+link('../reproduce/index.html','运行参数与版本')+'

')) + review_path=out/'result-review.json' + review=json.loads(review_path.read_text()) if review_path.is_file() else {} + updates=''.join('
'+esc(row.get('date',''))+'

'+model_text(row.get('title',''),row.get('title_en'))+'

'+model_text(row.get('fix',''),row.get('fix_en'))+'

'+model_text(row.get('scope',''),row.get('scope_en'))+'

' for row in review.get('review_rounds',[])) + write(out,'changes/index.html','变更记录',hero('变更记录','列出本轮核验后的修正；修正读取已保存的返回，不代表重新调用产品。')+section('本轮结果修正','
'+updates+'
')+section('详细依据','

'+link('../review/index.html','核验范围与修正说明')+' · '+link('../analysis/index.html','逐任务结果与证据')+' · '+link('../reproduce/index.html','运行参数与版本')+'

')) else: render_changes(out,cases,records,dataset) if include_english: from .report_i18n import write_english_report - write_english_report(out) + write_english_report(out,source_routes=english_routes) from harness.report_languages import write_language_contract if presentation_only and (out/'report-languages.json').is_file(): from harness.report_languages import check_language_contract, generated_languages @@ -2264,26 +2449,73 @@ def generate(out:Path,dataset:dict,records:list,manifests:list,*,include_english return out/'index.html' +def current_local_review_findings(out,dataset,records,audit): + """Describe the current snapshot from bound calls, never cached release prose.""" + from .local_revision import read_bound + receipt=json.loads(read_bound(out,dataset['comparison']['task_replacement']).read_text()) + added=receipt['provider_calls_added'] + cases={c['id']:c for c in dataset['cases']} + eligible=[r for r in records if measured(r,cases[r['case_id']])] + passed=sum(succeeded(r,cases[r['case_id']]) for r in eligible) + adjudicated=len(audit.get('adjudication_revision',{}).get('adjudicated_attempt_ids',[])) + unreviewed=len(audit.get('unreviewed_attempt_ids',[])) + rows=[ + (f"当前报告包含 {len(records)} 条结果：{receipt['inherited_results']} 条沿用历史调用，{added} 条来自新增调用。原始返回、调用时间和耗时均保留。", + f"The current report contains {len(records)} results: {receipt['inherited_results']} inherited calls and {added} new calls. Original answers, timestamps and latencies are retained."), + (f"当前通过 {passed} 条，可比较 {len(eligible)} 条，排除 {len(records)-len(eligible)} 条。页面与可下载汇总使用同一组保存结果。", + f"Currently {passed} pass among {len(eligible)} eligible results; {len(records)-len(eligible)} are excluded. Pages and downloadable summaries use the same saved results."), + (f"保留独立复核记录 {audit.get('reviewed_attempts',0)}/{len(records)}；另有 {unreviewed} 条未独立复核，其中机器通过 {audit.get('accepted_unreviewed_passes',0)} 条。补充裁定 {adjudicated} 条，不称为新一轮全量盲审。", + f"Retained independent review covers {audit.get('reviewed_attempts',0)}/{len(records)} results. Another {unreviewed} are not independently reviewed, including {audit.get('accepted_unreviewed_passes',0)} machine passes. There are {adjudicated} supplemental adjudications; this is not a new full blind review."), + ("目标内容要求、正文质量和诊断文字分别展示；具体评分实现哈希见各结果详情，诊断词不单独决定任务是否通过。", + "Target requirements, body quality and diagnostic text are reported separately. Each result binds its scorer hash; diagnostic words alone do not determine acceptance."), + ] + if audit.get('retained_reviews'): + rows[2] = ( + f"当前复核覆盖 {audit['reviewed_attempts']}/{len(records)}：本次按当前要求重新检查 {audit['fresh_reviewed_attempts']} 条；{audit['retained_reviewed_attempts']} 条沿用原始复核。沿用前逐条确认正向要求、保存的返回和结论未变，且旧判断不依赖已删除的限制。沿用不算作新一轮盲审。", + f"Current review covers {audit['reviewed_attempts']}/{len(records)} results: {audit['fresh_reviewed_attempts']} newly checked under current requirements; {audit['retained_reviewed_attempts']} original reviews retained. Retention verifies unchanged positive requirements, saved answers and verdicts, with no dependency on removed restrictions. Retention is not a fresh blind review.") + supplements = audit.get('supplemental_reviews', []) + if supplements and audit.get('retained_reviews'): + fresh = [row for row in audit.get('attempts', []) if row.get('review_origin') == 'fresh'] + conflicts = sum(row.get('agent_verdict') != row.get('final_verdict') for row in fresh) + if audit.get('review_protocol',{}).get('blind_initial_verdict') is False: + rows.append(('本次复核时已知程序结论，逐条检查了保存的返回、任务要求和原生字段；不称为独立盲审，也未新增被测网页调用。旧复核的原始意见保留。', + 'This round inspected saved answers, task requirements and native fields with known machine outcomes. It was not an independent blind review and added no benchmark webpage calls. Original opinions from retained reviews are preserved.')) + else: + rows.append((f'本次新复核先隐藏程序结论；初判与程序有 {conflicts} 条分歧，已对照原始返回和题目要求裁定。原始独立意见保留，未新增网页调用。', + f'Fresh review withheld machine verdicts. {conflicts} initial disagreements were adjudicated against the original answers and task requirements. Initial independent opinions are preserved; no new webpage calls were added.')) + elif supplements: + supplement_rows = [json.loads(read_bound(out, item).read_text()) for item in supplements] + reviewed = sum(item['reviewed_attempts'] for item in supplement_rows) + rows.append((f'新增调用中 {reviewed} 条已逐条补充复核，判定与程序一致；复核前已知机器汇总，不称为盲审。内容通过不保证代码可直接运行，具体质量问题见复核记录。', + f'{reviewed} new-call results received supplemental per-answer review and agree with the scorer. The machine summary was already known; this was not blind review. A content pass does not guarantee executable code; quality issues are retained in the review records.')) + if local_pass_rate_ranking(dataset): + rows.append(('名次按当前任务通过率计算，耗时单独展示，不参与名次。历史调用未完成调用前后的全量来源核验，保留实际检查日期和范围，不声称已完成来源认证。', + 'Ranks use current task pass rates. Latency is separate and does not determine rank. Historical calls lack complete before/after source verification; actual check dates and scope are retained without claiming source certification.')) + return [{'text':zh,'text_en':en} for zh,en in rows] + + def write_review_page(out,records): path=out/'result-review.json' if not path.is_file(): write(out,'review/index.html','核验记录',hero('核验记录','本轮没有绑定独立核验记录。')) return audit=json.loads(path.read_text());reviewer=audit.get('reviewer',{}) + dataset_path=out/'dataset.json' + dataset=json.loads(dataset_path.read_text()) if dataset_path.is_file() else {} metadata_path=Path(__file__).resolve().parents[2]/'research/report-changes.json' if not metadata_path.is_file(): raise ValueError('A reviewed report requires research/report-changes.json') changes=json.loads(metadata_path.read_text()) from .report_changes import require_review_rounds - rounds=require_review_rounds(changes) + rounds=(audit['review_rounds'] if audit.get('kind')=='local_independent_review' else require_review_rounds(changes)) rounds_html=''.join( - f'
第 {i} 轮
{esc(row.get("date",""))}

{esc(row.get("title",""))}

范围：{esc(row.get("scope",""))}

发现：{esc(row.get("finding",""))}

纠正：{esc(row.get("fix",""))}

未完成：{esc(row.get("pending",""))}

' + f'
第 {i} 轮
{esc(row.get("date",""))}

{model_text(row.get("title",""),row.get("title_en"))}

范围：{model_text(row.get("scope",""),row.get("scope_en"))}

发现：{model_text(row.get("finding",""),row.get("finding_en"))}

纠正：{model_text(row.get("fix",""),row.get("fix_en"))}

未完成：{model_text(row.get("pending",""),row.get("pending_en"))}

' for i,row in enumerate(rounds,1)) page=hero('核验记录','记录每轮核验的范围、发现、纠正和未决事项。') reviewed_attempts=audit.get('attempts',[]) disagreements=[row for row in reviewed_attempts if row.get('agent_verdict') != row.get('machine_verdict')] - adjudicated=[row for row in disagreements if row.get('final_verdict') in {'pass','fail'}] + adjudicated=[row for row in disagreements if row.get('final_verdict') in {'pass','fail','unavailable'}] unresolved=audit.get('unresolved',0) def coverage(label,item): scope=item.get('review_protocol',{}).get('scope','all') @@ -2293,13 +2525,59 @@ def coverage(label,item): return f'{label}全量 {item.get("reviewed_attempts",0):,}/{item.get("total_attempts",item.get("expected_attempts",0)):,}' conclusion=(f'

{coverage("主结果",audit)}；' f'代理分歧 {len(disagreements)}；已裁定 {len(adjudicated)}；未决 {unresolved}。

') + if audit.get('adjudication_revision'): + conclusion=review_summary(out,'../',highlight=not local_pass_rate_ranking(dataset)) page+=section('当前状态',conclusion) - page+=section('核验轮次','
'+rounds_html+'
') + page+=result_ledger_section(out,records,'../') + page+=section('核验轮次','

'+model_text('以下历史轮次保留当时的处理和限制；当前成绩、案例范围和排名状态以本页当前说明为准。','Historical rounds retain the treatment and limits recorded at the time. Current scores, task scope and ranking state are described by the current summary on this page.')+'

'+rounds_html+'
') method=(f'
核验协议与执行者

执行者：{esc(reviewer.get("agent","独立代理"))}；模型：{esc(reviewer.get("model","—"))}；推理强度：{esc(reviewer.get("effort","—"))}。' '核验时隐藏机器结论和分数，只提供任务契约、标准答案与完整实际输出；分歧回到原始证据裁定，未决非零则禁止发布。任务和输出未变化时沿用原核验；发生变化的调用必须重新盲审。

') + if audit.get('adjudication_revision'): + method='

'+model_text('原独立判断和补充判断保留在逐条记录中。本次补充裁定查看了已有评分、契约和原始输出，不是新一轮全量盲审。每条裁定列出具体规则和原始证据片段；本地结果不因裁定完成而获得来源认证。','Original independent and supplemental opinions remain in the per-result records. This supplemental adjudication inspected saved scores, contracts and original answers; it is not a new full blind review. Each adjudication records specific rules and original excerpts. Completing adjudication does not certify source validity.')+'

' + if audit.get('review_protocol',{}).get('blind_initial_verdict') is False and audit.get('retained_reviews'): + method='

'+model_text('本次重新检查的记录采用已知程序结论的证据复核，逐条核对完整保存答案、任务要求和原生字段。其余记录经依赖检查沿用旧意见。两类范围分别统计；本次证据复核不算作新的独立盲审，也不补足历史来源认证。','The newly checked records received evidence review with known machine outcomes, checking complete saved answers, task requirements and native fields. Other records retain prior opinions after dependency checks. The two scopes are reported separately; this evidence review is not a new independent blind review and does not complete historical source certification.')+'

' page+=section('核验口径',method) + dataset_path=out/'dataset.json' + if dataset_path.is_file(): + dataset=json.loads(dataset_path.read_text()) + findings=dataset.get('comparison',{}).get('call_chain_audit',{}).get('findings',[]) + if dataset.get('comparison',{}).get('task_replacement'): + findings=current_local_review_findings(out,dataset,records,audit) + if findings: + page+=section('本轮修正与检查限制',''.join('

'+model_text(item['text'],item.get('text_en'))+'

' for item in findings)) write(out,'review/index.html','核验记录',page) +NATIVE_OPTION_LABELS={'wait_until':'页面完成条件','block_private_networks':'阻止访问私网','max_bytes':'返回内容上限（字节）','max_age':'缓存最长年龄','proxy':'代理选择','formats':'请求输出格式','only_main_content':'只取主要正文','extract_depth':'提取深度','include_images':'包含图片','text_verbosity':'正文详细程度','max_age_hours':'缓存最长年龄（小时）','include_resource_links':'包含资源链接','rich_links_limit':'具名链接数量上限','document_max_bytes':'文档大小上限（字节）','min_interval_seconds':'调用间隔（秒）','log_level':'日志级别','retain_complete_diagnostic_log':'完整保存诊断日志','max_diagnostic_bytes':'日志保存上限（字节）','uniform_native_parameters':'使用统一固定原生选项','dns_mode':'域名解析方式'} + +def load_frozen_method_options(out,records): + """Bind displayed controls to their actual configuration and call scope.""" + from .local_revision import read_bound + document=json.loads((out/'frozen-method-options.json').read_text()) + if document.get('schema_version')!=2 or not document.get('source_config'): + raise ValueError('Frozen option display requires a bound configuration and call scope') + binding=document['source_config'] + configuration=json.loads(read_bound(out,binding).read_text())['providers'] + identifiers=document.get('case_ids',[]) + selected=[row for row in records if row['case_id'] in identifiers] + providers=document.get('providers',{}) + if (not identifiers or len(identifiers)!=len(set(identifiers)) or not providers + or document.get('provider_config_sha256')!=binding['sha256'] + or document.get('result_count')!=len(selected) + or len(selected)!=len(identifiers)*len(providers) + or {(r['provider'],r['case_id']) for r in selected} + !={(p,c) for p in providers for c in identifiers} + or any(not document.get(key) for key in + ('scope_title','scope_title_en','scope_description','scope_description_en'))): + raise ValueError('Frozen option display scope differs from saved calls') + for name,options in providers.items(): + if (name not in configuration or not options + or any(key not in NATIVE_OPTION_LABELS or key not in configuration[name] + or type(configuration[name][key]) is not type(value) + or configuration[name][key]!=value for key,value in options.items())): + raise ValueError('Displayed option differs from the frozen configuration') + return document + + def write_support(out,records,manifests,*,authored_interpretation=True): dataset=json.loads((out/'dataset.json').read_text()) if (out/'dataset.json').is_file() else {} local_matrix=(out/'dataset.json').is_file() and json.loads((out/'dataset.json').read_text()).get('comparison',{}).get('kind')=='uniform_native_local_measurement' @@ -2329,7 +2607,13 @@ def write_support(out,records,manifests,*,authored_interpretation=True): '旧耗时和原始物理调用保留在离线证据，不能把跨批次视图当成同时重跑。') methodology+=section('评分来源与修订','

'+detail+' '+' · '.join(revision_links)+'

') methodology+=section('稳定任务契约与全文诊断',task_contract_guide(),'task-contracts') - figures=''.join(f'
{label}
{label}
' for name,label in [('annotation','参考标注'),('validation','任务验收'),('scoring','评分口径')] if (out/'assets/methodology'/f'{name}.png').is_file()) + figures='' + for name,label in [('annotation','参考标注'),('validation','任务验收'),('scoring','评分口径')]: + if not (out/'assets/methodology'/f'{name}.png').is_file(): + continue + english_asset=(f' data-en-src="../assets/methodology/{name}.en.png"' + if (out/'assets/methodology'/f'{name}.en.png').is_file() else '') + figures+=f'
{label}
{label}
' if figures:methodology+=section('方法图解',figures) methodology+=section('参考与匹配','

原始快照保留不变；评分参考清除零宽空格、BOM 和软连字符，并检查空正文、乱码、重复核对点及核对点是否存在于参考中。通过检查不等于语义完整性已获证明，正文范围仍以来源记录为准。

关键事实按规范化后的可见文本机械匹配：统一 Unicode、大小写及排版空白，菜单路径如 Settings > AI 与 Settings>AI 等价；忽略 Markdown 链接地址、行内代码与加粗标记。一个已被完整记录边界约束的字段允许丢失内联元素间空白，例如标签列表 change deep-thoughts 与 changedeep-thoughts 内容等价；相邻独立链接仍按独立词段核对，不猜测跨记录的粘连分词。保留数字、正负号、否定词、标点和英文词边界。不使用大模型或模糊相似度判定通过。

') methodology+=section('调用时间','

各方法使用同一套冻结任务与参考。不同批次的耗时不直接排名；同窗速度子集单独标识。实时网页内容、登录状态和站点策略可能变化，成绩只代表所记录的调用。环境、版本与命令见“环境与复现”。

') @@ -2371,6 +2655,7 @@ def write_support(out,records,manifests,*,authored_interpretation=True): subset=[r for r in records if r['provider']==name] versions=sorted({str(value) for r in subset if (value:=method_version(r,provenance))}) identities=[p.get('implementation_identity',{}) for m in manifests for p in m.get('providers',[]) if p.get('name')==name] + versions=versions or sorted({str(identity['reported_version']) for identity in identities if identity.get('reported_version')}) versions=versions or sorted({str(identity['repository']['head']) for identity in identities if identity.get('repository',{}).get('head')}) hashes=sorted({str(r.get('provider_metadata',{}).get('binary_sha256')) for r in subset if r.get('provider_metadata',{}).get('binary_sha256')}) versions=versions or sorted({provenance[h].get('version') or provenance[h].get('identity') for h in hashes if h in provenance}) @@ -2384,7 +2669,13 @@ def write_support(out,records,manifests,*,authored_interpretation=True): basis='运行时版本与 SHA-256' else: basis='运行时 SHA-256；nightly 源修订不可得' - provider_rows.append(('',cell(link('../providers/'+name+'.html',provider_label(name)))+cell(esc('；'.join(versions) or '版本未记录'))+cell(esc(basis))+cell(esc('；'.join(deployments)))+cell('
'.join(''+esc(x)+'' for x in hashes) or '—')+cell(str(len(subset))))) + if local_matrix: + version='；'.join(versions) or ('云端服务未公开后端版本' if not hashes and not any(identities) else '未取得可确认的版本信息') + if versions and all(re.fullmatch(r'[0-9a-f]{40}', value) for value in versions): + version='源码提交：'+version + provider_rows.append(('',cell(link('../providers/'+name+'.html',provider_label(name)))+cell(''+esc(version)+''))) + else: + provider_rows.append(('',cell(link('../providers/'+name+'.html',provider_label(name)))+cell(esc('；'.join(versions) or '版本未记录'))+cell(esc(basis))+cell(esc('；'.join(deployments)))+cell('
'.join(''+esc(x)+'' for x in hashes) or '—')+cell(str(len(subset))))) write_review_page(out,records) repo='https://github.com/lexmount/browser-eval' support=hero('环境与复现','代码、评测集和复现说明以 browser-eval 仓库为准；本页只列本轮必要配置。') @@ -2395,8 +2686,25 @@ def write_support(out,records,manifests,*,authored_interpretation=True): support+=section('隔离资源证据','

'+link('../resource-cohort/resource-comparison.json','CPU、内存、网络汇总与共同成功样本')+' · '+link('../resource-cohort/results.jsonl','资源子集逐次记录')+' · '+link('../resource-cohort/execution.json','容器环境与镜像身份')+'

') conditions=(f'最终报告纳入 {len(records):,} 次调用。任务与参考冻结后执行；每个方法内部串行，不复用个人登录状态。' if authored_interpretation else f'本报告纳入 {len(records):,} 个逻辑结果。运行参数见下表；保留的物理调用与输出见各结果详情。') support+=section('公共条件','

'+conditions+'

'+table(['参数','值'],common,metrics=[])) - support+=section('方法版本','

“事后核验”只说明留存二进制与官方归档完全一致，不改写成当时已经记录了版本。移动 nightly 无法追到不可得的源修订时，只用 SHA-256 标识精确可执行文件。

'+table(['方法','版本','身份依据','部署路径','二进制 SHA-256','调用数'],provider_rows,metrics=[])) - support+=section('发布与加载','

本地构建为每个任务、方法和证据生成可独立检查的逻辑路由；这不等于线上部署同样数量的独立 HTML。发布器会把整份报告编译成一个通用 HTML 外壳、共享样式与脚本，以及按内容寻址的压缩数据块。页面切换由前端路由完成，证据只在打开对应页面时读取，超过 256 KiB 的完整文本还需用户主动加载。

发布清单记录逻辑路由数、源文件数、压缩块数和逐文件 SHA-256；发布门禁会重建每个文件并验证内部链接与原始下载字节。这样保留逐项可审计性，同时避免重复页面骨架和一次性加载全部证据。

') + if local_matrix: + support+=section('方法版本','

显示本轮使用的软件版本或源码提交。云端服务未公开可确认的后端版本号时，直接说明；这不代表服务没有版本。

'+table(['方法','本轮使用版本／源码提交'],provider_rows,metrics=[])) + else: + support+=section('方法版本','

版本优先使用调用时记录的版本号；源码实现使用冻结的提交号。云端服务没有公开后端版本时如实标注。二进制哈希用于确认具体执行文件；事后核验的版本会说明依据。

'+table(['方法','版本','身份依据','部署路径','二进制 SHA-256','调用数'],provider_rows,metrics=[])) + if local_matrix: + options_path=out/'frozen-method-options.json' + if options_path.is_file(): + options_document=load_frozen_method_options(out,records) + options=options_document['providers'] + labels=NATIVE_OPTION_LABELS + def option_text(value): + return '开启' if value is True else '关闭' if value is False else '、'.join(map(str,value)) if isinstance(value,list) else str(value) + config_rows=[('',cell(link('../providers/'+name+'.html',provider_label(name)))+cell(''.join('
'+esc(labels.get(k,k))+'：'+esc(option_text(v))+'
' for k,v in value.items()))) for name,value in options.items()] + support+=section(model_text(options_document['scope_title'],options_document['scope_title_en']), + '

'+model_text(options_document['scope_description'],options_document['scope_description_en'])+'

' + +table(['方法','固定选项'],config_rows,metrics=[])) + support+=local_call_chain_section(json.loads((out/'dataset.json').read_text())) + if not local_matrix: + support+=section('发布与加载','

本地构建为每个任务、方法和证据生成可独立检查的逻辑路由；这不等于线上部署同样数量的独立 HTML。发布器会把整份报告编译成一个通用 HTML 外壳、共享样式与脚本，以及按内容寻址的压缩数据块。页面切换由前端路由完成，证据只在打开对应页面时读取，超过 256 KiB 的完整文本还需用户主动加载。

发布清单记录逻辑路由数、源文件数、压缩块数和逐文件 SHA-256；发布门禁会重建每个文件并验证内部链接与原始下载字节。这样保留逐项可审计性，同时避免重复页面骨架和一次性加载全部证据。

') if not authored_interpretation: reproduce='

使用本轮任务配置与方法配置运行，或从保存的结果重新构建报告：

browser-harness-eval run --dataset DATASET --provider-config CONFIG --providers METHODS --output NEW_EVIDENCE\nbrowser-harness-eval report EVIDENCE

重新运行会调用配置中的方法；重建报告只读取已有结果。本轮参数、版本与可执行文件哈希见上表。

' elif headless: @@ -2404,10 +2712,19 @@ def write_support(out,records,manifests,*,authored_interpretation=True): else: reproduce+='

方法配置、凭据变量和本轮实际版本分别见“方法介绍”“账号”和上方方法版本表；不存在的 headless 专用配置不属于本 cohort。

' if local_matrix: - reproduce+='

重建历史成绩时，先使用本轮绑定的评分提交；当前分支的评分规则可能已经更新。提交号和评分文件哈希见仓库 evaluation/dynamic-content-100-moli-webfetch/current-local-results.json 的 measured_scoring_git_revision 与 measured_scoring_source_sha256。先在该提交的代码目录执行上面的构建命令。只更新已有报告说明时，可在当前代码目录执行以下命令；它保留原评分和实测文件。

uv run --frozen python scripts/build_report.py --refresh-presentation evaluation/dynamic-content-100-moli-webfetch/current-local-results.json --output /absolute/path/to/report
' + frozen_binding=(out/'presentation-binding.json').is_file() + build_binding=' --presentation-binding /absolute/path/to/report-evidence/presentation-binding.json' if frozen_binding else '' + check_binding=' --presentation-binding /absolute/path/to/report/presentation-binding.json' if frozen_binding else '' + reproduce='

以下命令从本轮保存的证据构建本地报告，不重新调用产品：

uv run --frozen python scripts/build_report.py --local-audit --language '+('zh --include-english' if _ENGLISH_GENERATED else 'zh')+' --evidence /absolute/path/to/report-evidence --output /absolute/path/to/report'+build_binding+'\nuv run --frozen python scripts/check_report.py /absolute/path/to/report'+check_binding+'

使用与本轮结果对应的证据目录。调用时的版本、配置和评分器哈希见 '+link('../local-run-contract.json','调用前冻结协议')+' 与 '+link('../execution.json','执行批次')+'。上面的构建命令读取保存的评分；修改评分规则后，应先按声明范围复评分，不能仅重建页面就称成绩已经更新。

' + if local_matrix and frozen_binding: + public_profile=json.loads((out/'presentation-binding.json').read_text()) + if public_profile.get('publication_projection'): + reproduce+='

'+model_text('公开副本已去掉本机目录信息。清单分别记录原始文件与公开副本的校验值；公开校验值不冒充原始调用标识。成绩、耗时和复核结论保持不变。','This public copy omits host paths. The manifest distinguishes original and public file hashes; public hashes do not claim to be original call identities. Scores, timings and review decisions are unchanged.')+' '+link('../'+public_profile['publication_projection']['path'],'PUBLICATION-PROJECTION.json')+'

' + if public_profile.get('publication_evidence_views'): + reproduce+='

'+model_text('评分时的原件私下保留；公开回答和完整原生响应隐藏了目录或敏感值。清单保留评分原件的标识，并单独记录公开副本。公开文件可重建本报告，不能还原隐藏内容；正文质量等指标保留原件的计算值，不保证用脱敏文字逐字重算一致。复核片段的坐标记录原件位置；公开副本仅能核对仍可见的片段。','Scoring originals remain private. Public answers and complete native responses hide paths or sensitive values. The inventory preserves original scoring identities and separately binds public views. Public files rebuild this report but cannot reconstruct omitted content. Text-quality diagnostics retain the original calculations and may not reproduce exactly from redacted text. Review coordinates refer to originals; public views verify only the visible excerpts.')+' '+link('../'+public_profile['publication_evidence_views']['path'],'PUBLIC-EVIDENCE-VIEWS.json')+'

' support+=section('核心复现指令',reproduce) write(out,'reproduce/index.html','环境与复现',support) - if not authored_interpretation: + if not authored_interpretation and not local_matrix: rows=[] for name in sorted({r['provider'] for r in records}): selected=[r for r in records if r['provider']==name] @@ -2435,8 +2752,12 @@ def write_support(out,records,manifests,*,authored_interpretation=True): {r['provider'] for r in records}, fixture_providers={r['provider'] for r in records if r.get('provider_type')=='fixture'}, ) - accountrows=[('',cell(esc(a['provider']))+cell(link(a['registration_url'],'注册 / 登录')+'
'+link(a['key_url'],'获取凭据'))+cell('
'.join(''+esc(x)+'' for x in a['env_vars']))+cell(esc(a['steps']))+cell(esc(a['free_note']))) for a in accounts] - write(out,'accounts/index.html','账号',hero('账号与凭据','只列官方入口和环境变量名称，不保存实际凭据。')+section('注册清单',table(['提供方','官方入口','环境变量','取得方式','免费 / 准入说明'],accountrows,cls='accounts-table'))) + accountrows=[] + for a in accounts: + official=link(a['registration_url'],'官方获取入口' if local_matrix else '注册 / 登录') + if a['key_url']!=a['registration_url']:official+='
'+link(a['key_url'],'获取凭据') + accountrows.append(('',cell(esc(a['provider']))+cell(official)+cell('
'.join(''+esc(x)+'' for x in a['env_vars']))+cell(esc(a['steps']))+cell(esc(a['free_note'])))) + write(out,'accounts/index.html','账号与安装',hero('账号与安装','只列官方入口和环境变量名称，不保存实际凭据。')+section('安装与账号准备',table(['提供方','官方入口','环境变量','取得方式','免费 / 准入说明'],accountrows,cls='accounts-table'))) from .report_methods import render_methods write(out,'adoption/index.html','方法介绍',render_methods(out,records,hero=hero,section=section,table=table,cell=cell,link=link,esc=esc,authored=authored_interpretation)) diff --git a/harness/web_fetch/report_article_cases.py b/harness/web_fetch/report_article_cases.py index f14934dc6..671422be7 100644 --- a/harness/web_fetch/report_article_cases.py +++ b/harness/web_fetch/report_article_cases.py @@ -121,11 +121,6 @@ def remaining_fact_lines(text, excerpt): f'至少 {required}/{len(regions)} 区达到 {threshold:.0%}；每区不低于 {floor:.0%}', ' / '.join(f'{r["coverage"]:.0%}' for r in regions), metrics['body_coverage_pass'])) - if 'forbidden_hits' in metrics: - forbidden = metrics['forbidden_hits'] - checks.append(_row('排除错误页', '排除：' + ' / '.join(case.get('forbidden_any', [])), - ' / '.join(forbidden) if forbidden else '未命中排除项', - not forbidden)) if 'text_encoding_corruption' in metrics: checks.append(_row('文字解码', '替换字符数 ≥20 且占比 ≥2% 时拒绝', f'{metrics.get("replacement_character_count", 0)} 个替换字符', @@ -168,12 +163,6 @@ def _list_example(case, record, output): f'{metrics["stable_complete_record_count"]} 条', metrics['stable_structure_pass']), _row('唯一记录', f'去重后至少 {contract["minimum_records"]} 条', f'{metrics["stable_unique_record_count"]} 条', metrics['stable_structure_pass'])] - forbidden = metrics['forbidden_hits'] - blocked = metrics['stable_blocked_page'] - checks.append(_row('排除错误页', - '排除：' + ' / '.join(case.get('forbidden_any', [])) + '；不得返回安全验证或拒绝访问页', - ' / '.join(forbidden) if forbidden else ('命中拦截页特征' if blocked else '未命中排除项或拦截页特征'), - not forbidden and not blocked)) checks.append(_row('文字解码', '替换字符数 ≥20 且占比 ≥2% 时拒绝', f'{metrics["replacement_character_count"]} 个替换字符', not metrics['text_encoding_corruption'])) @@ -204,10 +193,6 @@ def _dynamic_example(case, record, output): metrics['stable_structure_pass']), _row('全部固定名言', '10 组原句、作者与标签均须匹配独立参考', f'{metrics["stable_anchor_hits"]}/{metrics["stable_anchor_total"]} 组命中', metrics['stable_content_pass']), - _row('排除错误页', '排除：' + ' / '.join(case['forbidden_any']) + '；不得返回安全验证或拒绝访问页', - ' / '.join(metrics['forbidden_hits']) if metrics['forbidden_hits'] else - ('命中拦截页特征' if metrics['stable_blocked_page'] else '未命中排除项或拦截页特征'), - not metrics['forbidden_hits'] and not metrics['stable_blocked_page']), _row('文字解码', '替换字符数 ≥20 且占比 ≥2% 时拒绝', f'{metrics["replacement_character_count"]} 个替换字符', not metrics['text_encoding_corruption']), ] diff --git a/harness/web_fetch/report_build.py b/harness/web_fetch/report_build.py index db685bd29..4f2bf7626 100644 --- a/harness/web_fetch/report_build.py +++ b/harness/web_fetch/report_build.py @@ -248,9 +248,111 @@ def _install_report_tree(staging: Path, output: Path): staging.rename(output) +def _build_frozen_presentation(source, output, presentation_binding, *, + include_english, language, authored_interpretation, + show_local_status_banners): + """Render a bound historical report using saved scores and public evidence.""" + from harness.report_languages import select_report_language + from .safety import evidence_tree_identity, copy_evidence_file + if source in output.parents: + raise ValueError("Frozen presentation output must be outside its source tree") + before = evidence_tree_identity(source) + _, dataset, records, manifest = validate_local_presentation(source, presentation_binding) + if authored_interpretation is None: + authored_interpretation = manifest.get('presentation', {}).get( + 'authored_interpretation', bool(dataset.get('comparison', {}).get('report_interpretation'))) + review_path = source / "result-review.json" + binding = json.loads(presentation_binding.read_text()) + required = ["local-run-contract.json", "execution.json"] + if review_path.is_file(): + required.append("result-review.json") + for relative in required: + if not any(item.get("path") == relative + and item.get("sha256") == before.get(relative) + for item in binding.get("evidence_bindings", [])): + raise ValueError("Frozen presentation input lacks a hash binding: " + relative) + output.parent.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(prefix=".frozen-presentation-", dir=output.parent) as work: + staging = Path(work) / "report" + shutil.copytree(source, staging, copy_function=copy_evidence_file) + shutil.rmtree(staging / "en", ignore_errors=True) + shutil.copy2(presentation_binding, staging / "presentation-binding.json") + batches = json.loads((source / "execution.json").read_text())["batches"] + for record in records: + case = next(item for item in dataset['cases'] if item['id'] == record['case_id']) + if not record.get('diagnosis'): + text = contained_path(source, record['artifact_path']).read_text(errors='replace') + record['diagnosis'] = diagnose(record, case, text) + metadata = record.setdefault("provider_metadata", {}) + metadata["public_log_note"] = dataset.get("comparison", {}).get( + "call_chain_audit", {}).get("call_log_notes", {}).get(record["attempt_id"], "") + for kind, suffix in (("debug_log", "debug"), ("response", "response")): + relative = "logs/" + record["attempt_id"] + "." + suffix + ".txt" + if relative in before: + metadata["public_" + kind + "_path"] = relative + metadata["public_" + kind + "_sha256"] = before[relative] + generate(staging, dataset, records, batches, include_english=include_english, + authored_interpretation=authored_interpretation, + show_local_status_banners=show_local_status_banners, frozen_scores=True) + # Scores remain frozen, but cached inventory/summary files must describe + # those exact current scores after an explicitly bound offline revision. + write_frozen_result_summary(staging, dataset, records) + if language is not None: + select_report_language(staging, language) + write_frozen_presentation_manifest(staging, manifest, show_local_status_banners=show_local_status_banners, + authored_interpretation=authored_interpretation) + validate_local_presentation(staging, presentation_binding) + if evidence_tree_identity(source) != before: + raise ValueError("Presentation source changed during rendering") + _install_report_tree(staging, output) + return output / "index.html" + + +def write_frozen_presentation_manifest(output, manifest, *, show_local_status_banners=True, authored_interpretation=False): + """Refresh renderer metadata while retaining the frozen measurement bindings.""" + from harness.report_languages import generated_languages + from harness.reporting.templates import resolve_template + current = dict(manifest) + languages = list(generated_languages(output)) + current.update(template=resolve_template('webfetch'), languages=languages, generated_languages=languages, + presentation={"show_local_status_banners": show_local_status_banners, + "frozen_scores": True, "authored_interpretation": authored_interpretation}) + (output / "report-manifest.json").write_text(json.dumps(current, ensure_ascii=False, indent=2) + "\n") + + +def write_frozen_result_summary(output, dataset, records): + cases = {case['id']: case for case in dataset['cases']} + summary = {} + for track in TRACKS: + providers = {} + for provider in sorted({row['provider'] for row in records}): + rows = [row for row in records if row['provider'] == provider + and cases[row['case_id']].get('track', 'web') == track] + if not rows: + continue + eligible = [row for row in rows if measured(row, cases[row['case_id']])] + text = [row for row in eligible if row['metrics'].get('content_check_coverage') is not None] + item = {'attempts': len(rows), 'task_pass': sum(succeeded(row, cases[row['case_id']]) for row in rows), + 'task_evaluated_attempts': len(eligible), 'task_unmeasured_attempts': len(rows) - len(eligible), + 'task_excluded_attempts': sum(bool(row['metrics'].get('evaluation_excluded')) for row in rows), + 'scoring_scope': scoring_scope(eligible, cases), 'text_checked_attempts': len(text), + 'text_checked_cases': len({row['case_id'] for row in text})} + if track == 'challenge': + body = [row for row in eligible if cases[row['case_id']].get('task_kind', 'body') == 'body' + and cases[row['case_id']].get('ground_truth_main_content')] + item.update(body_evaluated_attempts=len(body), + body_task_pass=sum(succeeded(row, cases[row['case_id']]) for row in body) if body else None) + providers[provider] = item + summary[track] = providers + for name, value in [('summary.json', summary), ('reliability.json', assess_reliability(dataset, records))]: + (output / name).write_text(json.dumps(value, ensure_ascii=False, indent=2) + '\n') + + def build_report(source: Path, output: Path, *, include_english: bool = False, local_audit: bool = False, authored_interpretation: bool | None = None, - diagnostic_preview: bool = False, language: str | None = None) -> Path: + diagnostic_preview: bool = False, language: str | None = None, + show_local_status_banners: bool = True, + presentation_binding: Path | None = None) -> Path: from harness.report_languages import includes_english, select_report_language include_english = includes_english(language, include_english) source, output = source.resolve(), output.resolve() @@ -259,6 +361,13 @@ def build_report(source: Path, output: Path, *, include_english: bool = False, marker = '.generated-report' if output.exists() and not (output/marker).is_file(): raise ValueError(f'Refusing to replace an unmarked directory: {output}') + if presentation_binding is not None: + if not local_audit: + raise ValueError("A frozen presentation binding requires local audit mode") + return _build_frozen_presentation( + source, output, presentation_binding, include_english=include_english, + language=language, authored_interpretation=authored_interpretation, + show_local_status_banners=show_local_status_banners) dataset_probe = json.loads(contained_path(source, 'dataset.json').read_text()) diagnostic_preview = (diagnostic_preview and dataset_probe.get('comparison', {}).get('kind') == 'web_fetch_diagnostic') @@ -362,6 +471,8 @@ def build_report(source: Path, output: Path, *, include_english: bool = False, raise ValueError('A report requires tasks and measured records') execution = source/'execution.json' manifests = json.loads(execution.read_text())['batches'] if execution.exists() else [json.loads((source/'manifest.json').read_text())] + from .native_routes import verify_native_execution_manifests + verify_native_execution_manifests(source, manifests, dataset, records) if not local_audit and not diagnostic_preview: require_verified_reference_checks(manifests, cases=dataset, records=records, source=source, equivalent_capture_bindings=dataset.get('comparison',{}).get('reference_binding_mode') == @@ -605,14 +716,31 @@ def build_report(source: Path, output: Path, *, include_english: bool = False, shutil.copytree(source/'history/file-baseline', staging/'history/file-baseline') if dataset.get("comparison", {}).get("kind") == "uniform_native_local_measurement": shutil.copy2(source / "local-run-contract.json", staging / "local-run-contract.json") + if (source/"providers.json").is_file(): + # Publish only an explicit allowlist from the already validated frozen config. + config=json.loads((source/'providers.json').read_text())['providers'] + allowed={'wait_until','block_private_networks','max_bytes','max_age','proxy','formats','only_main_content','extract_depth','include_images','text_verbosity','max_age_hours','include_resource_links','rich_links_limit','document_max_bytes','min_interval_seconds','log_level','retain_complete_diagnostic_log','max_diagnostic_bytes','uniform_native_parameters','dns_mode'} + public_options={name:{k:v for k,v in options.items() if k in allowed} for name,options in config.items()} + (staging/'frozen-method-options.json').write_text(json.dumps({'schema_version':1,'provider_config_sha256':hashlib.sha256((source/'providers.json').read_bytes()).hexdigest(),'providers':public_options},ensure_ascii=False,indent=2)) + from harness.reporting.templates import resolve_template + (staging / "report-manifest.json").write_text(json.dumps({ + "schema_version": 1, "template": resolve_template("webfetch"), + "evidence_state": "local_measurement", "certified": False, + "languages": ["zh", "en"] if include_english else ["zh"], "provider_calls_added": 0, + "dataset_sha256": hashlib.sha256((source / "dataset.json").read_bytes()).hexdigest(), + "results_sha256": hashlib.sha256((source / "results.jsonl").read_bytes()).hexdigest(), + "presentation": {"show_local_status_banners": show_local_status_banners}, + }, ensure_ascii=False, indent=2)) + from .report_reading import copy_archived_reading + copy_archived_reading(source,staging) generate(staging,dataset,records,manifests,include_english=include_english, - authored_interpretation=authored_interpretation) + authored_interpretation=authored_interpretation, + show_local_status_banners=show_local_status_banners) if language is not None: select_report_language(staging, language) from harness.reporting.templates import resolve_template from harness.report_languages import generated_languages - nonpublication = diagnostic_preview or dataset.get('comparison', {}).get('kind') == 'uniform_native_local_measurement' - template_manifest = {'schema_version': 1, 'template': None if nonpublication else resolve_template('webfetch'), + template_manifest = {'schema_version': 1, 'template': None if diagnostic_preview else resolve_template('webfetch'), 'dataset_sha256': hashlib.sha256((staging/'dataset.json').read_bytes()).hexdigest(), 'results_sha256': hashlib.sha256((staging/'results.jsonl').read_bytes()).hexdigest(), 'generated_languages': generated_languages(staging)} @@ -1287,6 +1415,7 @@ def validate_local_presentation(output: Path, binding_path: Path): not a fresh build or certification. All measured files remain untouched. """ from lexbench_webfetch.paths import contained_path + from harness.report_languages import generated_languages output = output.resolve() binding = json.loads(binding_path.read_text()) manifest_path = contained_path(output, 'report-manifest.json') @@ -1294,7 +1423,7 @@ def validate_local_presentation(output: Path, binding_path: Path): if (binding.get('comparison_kind') != 'uniform_native_local_measurement' or manifest.get('evidence_state') != 'local_measurement' or manifest.get('certified') is not False - or manifest.get('languages') not in (['zh'], ['zh', 'en']) + or manifest.get('languages') != list(generated_languages(output)) or not (output / '.generated-report').is_file()): raise ValueError('Presentation refresh requires a bound diagnostic snapshot') for name, key in [('dataset.json', 'frozen_dataset_sha256'), ('results.jsonl', 'results_sha256')]: @@ -1303,33 +1432,72 @@ def validate_local_presentation(output: Path, binding_path: Path): if (not expected or manifest.get(manifest_key) != expected or hashlib.sha256(contained_path(output, name).read_bytes()).hexdigest() != expected): raise ValueError('Presentation input binding changed: ' + name) + for item in binding.get('evidence_bindings', []): + path = contained_path(output, item['path'], reject_symlinks=True) + if hashlib.sha256(path.read_bytes()).hexdigest() != item['sha256']: + raise ValueError('Presentation evidence binding changed: ' + item['path']) + if binding.get('publication_projection'): + from .publication_privacy import require_host_metadata_projection + require_host_metadata_projection(output, binding['publication_projection']) dataset = json.loads((output / 'dataset.json').read_text()) records = load_jsonl(contained_path(output, 'results.jsonl')) + if binding.get('publication_evidence_views'): + from .publication_privacy import require_public_evidence_view_inventory + require_public_evidence_view_inventory(output, binding['publication_evidence_views'], records) + from .native_routes import verify_native_execution_manifests + execution = output/'execution.json' + if execution.is_file(): + verify_native_execution_manifests(output, json.loads(execution.read_text())['batches'], dataset, records) + scoring_sources = {row['attempt_id']: binding.get('measured_scoring_source_sha256') for row in records} + if binding.get('offline_revision'): + from .local_revision import validate_revision_chain, read_bound + scoring_sources, input_dataset, input_results = validate_revision_chain( + output, binding['offline_revision'], dataset, records) + if binding.get('task_replacement'): + from .local_revision import require_local_task_replacement + previous_dataset = json.loads(read_bound(output, input_dataset).read_text()) + previous_records = load_jsonl(read_bound(output, input_results)) + require_local_task_replacement(output, binding['task_replacement'], previous_dataset, previous_records) + elif binding.get('task_replacement'): + from .local_revision import require_local_task_replacement + scoring_sources = require_local_task_replacement(output, binding['task_replacement'], dataset, records) if (len(records) != binding.get('result_count') or len(dataset['cases']) != binding.get('case_count') or sorted({row['provider'] for row in records}) != sorted(binding['providers']) - or any(row['metrics'].get('scoring_source_sha256') != binding.get('measured_scoring_source_sha256') for row in records)): + or any(row['metrics'].get('scoring_source_sha256') != scoring_sources[row['attempt_id']] for row in records)): raise ValueError('Presentation refresh cannot change the measured matrix or scoring revision') if any(path.is_symlink() for path in output.rglob("*")): raise ValueError("Presentation snapshot must not contain symlinks") validate_report_ids(dataset, records) - if not assess_reliability(dataset, records)['complete']: + reliability = assess_reliability(dataset, records) + from .cohort_exclusions import require_cohort_exclusions + excluded = require_cohort_exclusions(output, dataset, records) + if (not reliability['complete'] and + (any(reliability[key] for key in ('missing', 'extra', 'duplicates')) + or any(item['case_id'] not in excluded for item in reliability['unjudged']))): raise ValueError('Presentation refresh requires the complete declared matrix') for row in records: - artifact = contained_path(output, row['artifact_path']) - if hashlib.sha256(artifact.read_bytes()).hexdigest() != row['metrics']['content_sha256']: - raise ValueError('Saved output changed: ' + row['attempt_id']) - verify_record_public_call_logs(output, row) + from .publication_privacy import read_evidence_binding + try: + read_evidence_binding(output, {'path': row['artifact_path'], + 'sha256': row['metrics']['content_sha256']}) + except ValueError as error: + if binding.get('publication_evidence_views'): + raise + raise ValueError('Saved output changed: ' + row['attempt_id']) from error + verify_record_public_call_logs(output, row, + allow_public_views=bool(binding.get('publication_evidence_views'))) return binding, dataset, records, manifest def refresh_local_presentation(output: Path, binding_path: Path) -> Path: """Install explanatory routes from the single report owner.""" - from .safety import unlinked_path, evidence_tree_identity + from .safety import unlinked_path, evidence_tree_identity, copy_evidence_file output = unlinked_path(output) before_tree = evidence_tree_identity(output) binding, dataset, records, manifest = validate_local_presentation(output, binding_path) - manifest_path = output / "report-manifest.json" + authored_interpretation = manifest.get('presentation', {}).get( + 'authored_interpretation', bool(dataset.get('comparison', {}).get('report_interpretation'))) # The existing owner renders from saved metrics. Only these explanatory # routes may be installed; result pages, logs and every measured byte stay. routes = ['index.html', 'methodology/index.html', 'adoption/index.html', 'reproduce/index.html', 'changes/index.html'] @@ -1340,10 +1508,11 @@ def refresh_local_presentation(output: Path, binding_path: Path) -> Path: routes += ['en/' + relative for relative in routes if relative.endswith('.html')] with tempfile.TemporaryDirectory(prefix='.presentation-', dir=output.parent) as scratch: staging = Path(scratch) / 'report' - shutil.copytree(output, staging) - shutil.rmtree(staging / 'en', ignore_errors=True) + shutil.copytree(output, staging, copy_function=copy_evidence_file) batches = json.loads((output / 'execution.json').read_text())['batches'] - generate(staging, dataset, records, batches, include_english=include_english, presentation_only=True) + generate(staging, dataset, records, batches, include_english=include_english, presentation_only=True, + frozen_scores=True, authored_interpretation=authored_interpretation, + english_routes=[r for r in routes if r.endswith('.html') and not r.startswith('en/')]) changed = [] for relative in routes: before, after = contained_path(output, relative), contained_path(staging, relative) @@ -1353,7 +1522,7 @@ def refresh_local_presentation(output: Path, binding_path: Path) -> Path: changed.append(relative) manifest['presentation_refresh'] = { 'provider_calls_added': 0, 'scores_changed': False, - 'scoring_git_revision': binding['measured_scoring_git_revision'], + 'scoring_git_revision': binding.get('measured_scoring_git_revision'), 'scoring_source_sha256': binding['measured_scoring_source_sha256'], 'renderer_sha256': hashlib.sha256(Path(__file__).with_name('report.py').read_bytes()).hexdigest(), 'changed_routes': sorted(set(changed) | set(manifest.get('presentation_refresh', {}).get('changed_routes', []))), @@ -1361,10 +1530,12 @@ def refresh_local_presentation(output: Path, binding_path: Path) -> Path: # Stage the permitted pages into a copy of the original snapshot so no # renderer side effects can change evidence or unrelated routes. selected = Path(scratch) / 'selected' - shutil.copytree(output, selected) + shutil.copytree(output, selected, copy_function=copy_evidence_file) for relative in changed: shutil.copy2(staging / relative, selected / relative) - (selected / manifest_path.name).write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + '\n') + write_frozen_presentation_manifest(selected, manifest, + show_local_status_banners=manifest.get('presentation', {}).get('show_local_status_banners', True), + authored_interpretation=authored_interpretation) if evidence_tree_identity(output) != before_tree: raise ValueError('Presentation snapshot changed during rendering') _install_report_tree(selected, output) diff --git a/harness/web_fetch/report_check.py b/harness/web_fetch/report_check.py index f7379ae0b..4ac494d86 100644 --- a/harness/web_fetch/report_check.py +++ b/harness/web_fetch/report_check.py @@ -20,20 +20,67 @@ from harness.web_fetch.storage import require_canonical_inputs, require_current_report_profile +def require_public_evidence_page(content, relative, views): + """Require inspectable public downloads and honest original/public hashes.""" + from bs4 import BeautifulSoup + from urllib.parse import unquote, urljoin + content = re.sub(r'<(pre|script|style)\b[^>]*>.*?', '', content, flags=re.I | re.S) + soup = BeautifulSoup(content, 'html.parser') + visible = soup.get_text(' ', strip=True) + if ']+>', '', value) for value in re.findall(r']*>(.*?)', home, re.S)] + if any(marker not in headings for marker in markers) or not ( + headings.index(markers[1]) < headings.index(markers[0]) < headings.index(markers[2])): + raise ValueError('Task overview and comparison must precede detailed results') + if '判分完整性' not in home or ('本类最高：' in home) == diagnostic: + raise ValueError('Comparison winner disclosure differs from evidence state') + + class VisibleText(HTMLParser): def __init__(self): super().__init__() @@ -100,6 +169,7 @@ def __init__(self, page=None, assets=None): self.assets = assets self.figures = [] self.images = [] + self.english_images = [] def handle_starttag(self, tag, attrs): if tag == "figure": @@ -115,12 +185,15 @@ def handle_starttag(self, tag, attrs): (key, value) for key, value in attrs if key.startswith("data-") and key != "data-i18n-attrs" + and not (key == "data-en-src" and self.page + and self.page.parts[-2:] == ("methodology", "index.html")) ) ) if values.get("href") and (not values.get("hreflang")): self.links.append(values["href"]) if tag == "img" and values.get("src"): self.images.append(values["src"]) + self.english_images.append(values.get("data-en-src", values["src"])) if tag in {"pre", "code", "script", "style"}: self.skip += 1 @@ -135,6 +208,93 @@ def handle_data(self, text): self.numbers.extend(visible_numbers(text)) +def check_localized_image_sources(zh, en, canonical): + """Require each declared English companion, never silently use the old image.""" + expected = Counter(canonical(zh.page, value) for value in zh.english_images) + actual = Counter(canonical(en.page, value) for value in en.images) + if expected != actual: + raise ValueError('Localized image sources differ from the declared companions') + + +def check_saved_result_summary(root, dataset, records): + from lexbench_webfetch.metrics import task_measured, task_passed + cases = {case['id']: case for case in dataset['cases']} + summary = json.loads((root / 'summary.json').read_text()) + groups = {} + for row in records: + key = (cases[row['case_id']].get('track', 'web'), row['provider']) + groups.setdefault(key, []).append(row) + for (track, provider), rows in groups.items(): + saved = summary.get(track, {}).get(provider, {}) + expected = {'attempts': len(rows), + 'task_pass': sum(task_passed(row['ok'], row['metrics']) for row in rows), + 'task_evaluated_attempts': sum(task_measured(row['metrics']) for row in rows), + 'task_excluded_attempts': sum(bool(row['metrics'].get('evaluation_excluded')) for row in rows)} + if any(saved.get(key) != value for key, value in expected.items()): + raise ValueError('Saved result summary differs from current evidence: ' + provider + '/' + track) + + +def require_replacement_execution(root, binding, physical, logical): + """Keep superseded task calls as history without counting them as current rows.""" + from .local_revision import read_bound + receipt = json.loads(read_bound(root, binding).read_text()) + previous = load_jsonl(read_bound(root, receipt['baseline_results'])) + calls = load_jsonl(read_bound(root, receipt['cohort_results'])) + expected_physical = Counter(row['attempt_id'] for row in previous + calls) + current = [row for row in previous if row['case_id'] != receipt['removed_case_id']] + expected_logical = Counter(row['attempt_id'] for row in current + calls) + if physical != expected_physical or logical != expected_logical: + raise ValueError('Task replacement physical/logical call inventory differs') + + +class VisibleLocalScope(HTMLParser): + """Disclosure attributes count only when attached to visible explanatory text.""" + def __init__(self): + super().__init__() + self.stack = [] + self.visible = set() + + def handle_starttag(self, tag, attrs): + attrs = dict(attrs) + inherited = self.stack[-1][1:] if self.stack else (False, set()) + style = re.sub(r'\s+', '', attrs.get('style', '')).lower() + hidden = (inherited[0] or tag in {'script', 'style', 'template'} + or 'hidden' in attrs or attrs.get('aria-hidden') == 'true' + or 'display:none' in style or 'visibility:hidden' in style) + kinds = set(inherited[1]) + if attrs.get('data-evidence-state') == 'local_measurement': kinds.add('evidence') + if attrs.get('data-ranking-scope') == 'saved-task-pass-rate': kinds.add('ranking') + if tag not in {'area','base','br','col','embed','hr','img','input','link','meta','param','source','track','wbr'}: + self.stack.append((tag, hidden, kinds)) + + def handle_endtag(self, tag): + for index in range(len(self.stack) - 1, -1, -1): + if self.stack[index][0] == tag: + del self.stack[index:] + break + + def handle_data(self, data): + if data.strip() and self.stack and not self.stack[-1][1]: + self.visible.update(self.stack[-1][2]) + + +def check_local_diagnostic_presentation(root: Path, *, ranked=False): + """Track evidence state and scope any explicitly requested sample ranking.""" + for path in root.rglob('*.html'): + page = path.read_text(encoding='utf-8') + if '' not in page: + continue # Original response/source HTML is evidence, not report prose. + disclosure = VisibleLocalScope() + disclosure.feed(page) + if 'evidence' not in disclosure.visible: + raise ValueError('Local diagnostic disclosure missing: ' + str(path.relative_to(root))) + if ranked and 'ranking' not in disclosure.visible: + raise ValueError('Sample ranking scope missing: ' + str(path.relative_to(root))) + for table in re.findall(r']*\bid="ranking-[^"]+".*?
', page, re.S): + if not ranked and ('data-dir="desc"' in table or re.search(r'\s*(?:序号|Rank|No\.)\s*', table)): + raise ValueError('Uncertified diagnostic comparison assigns ranks: ' + str(path.relative_to(root))) + + def check(root: Path, *, authority_root: Path, profile: Path | None = None, presentation_binding: Path | None = None, source_dataset: Path | None = None): reject_evidence_symlinks(root) root = root.resolve() @@ -167,7 +327,7 @@ def check(root: Path, *, authority_root: Path, profile: Path | None = None, pres if manifest_path.is_file(): from harness.reporting.templates import resolve_template manifest = json.loads(manifest_path.read_text()) - expected_template = None if d.get('comparison', {}).get('kind') == 'uniform_native_local_measurement' else resolve_template('webfetch', root=repository) + expected_template = resolve_template('webfetch', root=repository) if manifest.get('template') != expected_template: raise ValueError('Report manifest differs from the maintained template') for name in ('dataset', 'results'): @@ -198,22 +358,42 @@ def check(root: Path, *, authority_root: Path, profile: Path | None = None, pres "PASS: WebMainBench preselected-200 matrix, sample, scores, retries, " "declared-language routes and reference assets verified" ) + public_payload_views = False names = {r["provider"] for r in rs} local_measurement = d.get("comparison", {}).get("kind") == "uniform_native_local_measurement" if presentation_binding and not local_measurement: raise ValueError("Presentation binding requires a frozen uniform-native diagnostic snapshot") + from .report import local_pass_rate_ranking + sample_ranking = local_measurement and local_pass_rate_ranking(d) if local_measurement: - from .local_audit import require_local_measurement + check_local_diagnostic_presentation(root, ranked=sample_ranking) + if (root / 'result-review.json').is_file(): + from .adjudication_evidence import require_adjudication_evidence, require_current_review_state + review=json.loads((root / 'result-review.json').read_text()) + require_current_review_state(rs,review) + require_adjudication_evidence(root,d,rs,review) + if (root / 'local-evidence/result-ledger.json').is_file(): + from .result_ledger import validate_ledger + ledger = validate_ledger(json.loads((root / 'local-evidence/result-ledger.json').read_text()), rs, source=root) + if ledger['states'][-1]['source_results_sha256'] != hashlib.sha256((root / 'results.jsonl').read_bytes()).hexdigest(): + raise ValueError('Result ledger source hash differs from current results') + from .local_audit import require_local_measurement, local_measurement_cases if presentation_binding: if profile: raise ValueError("Presentation binding cannot certify a release profile") from .report_build import validate_local_presentation - validate_local_presentation(root, presentation_binding) + checked_binding = validate_local_presentation(root, presentation_binding)[0] + public_payload_views = bool(checked_binding.get("publication_evidence_views")) else: require_local_measurement(root, d, rs) inventory = {track: {"case_ids": [case["id"] for case in d["cases"] if case.get("track", "web") == track], "providers": providers} for track, providers in d["coverage_plan"].items()} - canonical_dataset = repository / json.loads((root / "local-run-contract.json").read_text())["dataset_path"] - elif source_dataset or (profile_path and (json.loads(profile_path.read_text()).get('frozen_inputs') or (profile_path.parent / "dataset.json").is_file())): + local_contract = json.loads((root / "local-run-contract.json").read_text()) + canonical_cases = ({case["id"]: case for case in d["cases"]} + if presentation_binding and d.get('comparison', {}).get('task_replacement') + else {case["id"]: case for case in local_measurement_cases(local_contract)}) + canonical_dataset = (root / "dataset.json" if local_contract.get("dataset_sources") + else repository / local_contract["dataset_path"]) + elif source_dataset or (profile_path and (json.loads(profile_path.read_text()).get('frozen_inputs') or (profile_path.parent / 'dataset.json').is_file())): inventory = { track: { "case_ids": [ @@ -277,21 +457,34 @@ def check(root: Path, *, authority_root: Path, profile: Path | None = None, pres raise ValueError("Report acceptance check failed") if not len({(r["provider"], r["case_id"]) for r in rs}) == expected: raise ValueError("Report acceptance check failed") + if public_payload_views: + check_public_evidence_pages(root, rs) case_by_id = {c["id"]: c for c in d["cases"]} for case in d["cases"]: safe_identifier(case["id"]) - validate_resource_source_evidence(case, canonical_dataset, execution=True) - reliability = require_reliable(d, rs) + validate_resource_source_evidence(case, canonical_dataset) + if local_measurement: + from .local_audit import require_complete_local_inventory + reliability = require_complete_local_inventory(d, rs) + else: + reliability = require_reliable(d, rs) saved_reliability = json.loads((root / "reliability.json").read_text()) if not saved_reliability == reliability: raise ValueError("Saved reliability status differs from report evidence") + if local_measurement and d.get('comparison', {}).get('result_ledger'): + check_saved_result_summary(root, d, rs) for r in rs: safe_identifier(r["attempt_id"]) safe_identifier(r["provider"]) safe_identifier(r["case_id"]) - p = contained_path(root, r["artifact_path"], reject_symlinks=True) - if not hashlib.sha256(p.read_bytes()).hexdigest() == r["metrics"]["content_sha256"]: - raise ValueError(p) + if public_payload_views: + from .publication_privacy import read_evidence_binding + read_evidence_binding(root, {'path': r['artifact_path'], + 'sha256': r['metrics']['content_sha256']}) + else: + p = contained_path(root, r['artifact_path'], reject_symlinks=True) + if hashlib.sha256(p.read_bytes()).hexdigest() != r['metrics']['content_sha256']: + raise ValueError(p) if not r["ok"]: if not not r["metrics"]["content_pass"]: raise ValueError("Report acceptance check failed") @@ -309,14 +502,15 @@ def check(root: Path, *, authority_root: Path, profile: Path | None = None, pres ) if local_measurement: from lexbench_webfetch.stable_tasks import validate_acceptance_contract - validate_acceptance_contract(c["acceptance_contract"]) + if c.get("acceptance_contract"): + validate_acceptance_contract(c["acceptance_contract"]) + elif not (stable_checks or c.get("task_kind") == "file"): + raise ValueError(c["id"]) elif not stable_checks: raise ValueError(c["id"]) if c.get("ground_truth_main_content"): expected = c["ground_truth_main_content"].encode() if local_measurement: - from .reference_inputs import read_dataset_bytes - canonical_cases = {case["id"]: case for case in json.loads(read_dataset_bytes(canonical_dataset))["cases"]} valid_reference = expected == canonical_cases[c["id"]]["ground_truth_main_content"].encode() elif source_dataset: digest = hashlib.sha256(expected).hexdigest() @@ -392,6 +586,8 @@ def canonical_link(page, value): en_links = Counter(canonical_link(en_path, value) for value in en.links) if zh_links != en_links: bilingual_issues.append((relative, "evidence or navigation links differ")) + if relative == Path("methodology/index.html"): + check_localized_image_sources(zh, en, canonical_link) if relative in {Path("guide/index.html"), Path("guide/article.html")}: if Counter(canonical_link(zh_path, value) for value in zh.images) != Counter( canonical_link(en_path, value) for value in en.images @@ -417,6 +613,12 @@ def canonical_link(page, value): allowed_english_source_text = {"中文"} | { case["name"] for case in d["cases"] if case.get("name") } + archived_manifest = root / "archived-reading.json" + if archived_manifest.is_file(): + from .report_reading import render_archived_reading + # Validate the archived source before allowing its original case names. + render_archived_reading(root, "en" if has_english else "zh") + allowed_english_source_text.update(json.loads(archived_manifest.read_text()).get("preserved_source_names", [])) if has_english: for relative in english_primary: if relative not in en_pages: @@ -461,9 +663,8 @@ def canonical_link(page, value): parser.feed(text) relative_parts = p.relative_to(root).parts if "browser-use" not in relative_parts: - expected_index_header = "No." if relative_parts[0] == "en" else "序号" - if not all(h == expected_index_header for h in parser.headers): - raise ValueError((p, parser.headers)) + require_table_index_headers(parser, english=relative_parts[0] == 'en', + diagnostic=local_measurement) if p.parent.name == "attempts": record = next(r for r in rs if r["attempt_id"] == p.stem) case = case_by_id[record["case_id"]] @@ -473,7 +674,7 @@ def canonical_link(page, value): else "标准答案与验收规则" if case.get("task_kind") == "structure" else "验收规则（无全文参考）" - if case.get('task_kind', 'body') == 'body' and not (case.get('ground_truth_main_content') or case.get('reference_answer')) + if case.get("task_kind", "body") == "body" and not (case.get("ground_truth_main_content") or case.get("reference_answer")) else "完整标准答案" ) actual_heading = ( @@ -505,7 +706,10 @@ def canonical_link(page, value): execution = json.loads((root / "execution.json").read_text())["batches"] physical_attempts = Counter(i for b in execution for i in b["attempt_ids"]) logical_attempts = Counter(r["attempt_id"] for r in rs) - if d.get("comparison", {}).get("superseded_formal_provider_calls"): + if d.get('comparison', {}).get('task_replacement'): + require_replacement_execution(root, d['comparison']['task_replacement'], + physical_attempts, logical_attempts) + elif d.get("comparison", {}).get("superseded_formal_provider_calls"): if not set(physical_attempts) == set(logical_attempts): raise ValueError("Report acceptance check failed") if ( @@ -517,14 +721,7 @@ def canonical_link(page, value): raise ValueError("Report acceptance check failed") home = (root / "index.html").read_text() track_count = len({case.get("track", "web") for case in d["cases"]}) - if ( - not home.index("

综合通过率总榜

") - < home.index(f"

{track_count} 类任务概览

") - < home.index(f"

{track_count} 类分榜

") - ): - raise ValueError("Core comparison must precede the overview and detailed rankings") - if not ("本类最高：" in home and "判分完整性" in home): - raise ValueError("Report acceptance check failed") + require_home_comparison_layout(home, track_count, diagnostic=local_measurement and not sample_ranking, sample_ranking=sample_ranking) if (root / "resource-cohort/resource-comparison.json").is_file(): from .report_resources import check_resource_presentation resource_summary = json.loads((root / "resource-cohort/resource-comparison.json").read_text()) @@ -716,7 +913,11 @@ def check_revision(out: Path, *, authority_root: Path): if expected_failure not in [h.get_text() for h in soup.select("#filtered-table th")]: raise ValueError(path) records = load_jsonl(out / 'results.jsonl') - if records and all(row.get('provider_type') == 'fixture' for row in records): + from .methods import native_browser_engine + synthetic = dataset.get('comparison', {}).get('kind') == 'synthetic_browser_report' + if records and all(row.get('provider_type') == 'fixture' for row in records) and ( + synthetic or all(not native_browser_engine(row['provider']) for row in records) + ): methods = BeautifulSoup((out / 'adoption/index.html').read_text(), 'html.parser') for name in {row['provider'] for row in records}: if methods.find(id='interface-' + name) is None: diff --git a/harness/web_fetch/report_components.py b/harness/web_fetch/report_components.py index 726e2e71d..dec564d5f 100644 --- a/harness/web_fetch/report_components.py +++ b/harness/web_fetch/report_components.py @@ -11,7 +11,7 @@ def reference_text(case): return '本例按资源或结构规则评分，不以页面短摘要计算全文 F1。' rules=[('最少字符数',case.get('min_content_chars',1))] for key,label in [('expected_any','至少命中一项文字'),('required_facts','必需文字组'), - ('forbidden_any','不得出现的文字'),('required_patterns','格式与数量要求'), + ('required_patterns','格式与数量要求'), ('body_requirements','正文范围要求'),('acceptance_contract','验收契约')]: if case.get(key):rules.append((label,case[key])) if case.get('required_facts'): @@ -35,7 +35,7 @@ def cards(items): def metric_guide(): return '

静态网站正文、动态网站正文、热门网站正文、风控网页正文和文档正文使用同一评分流程。页面来源与加载方式不同，改变的是案例要求，不改变指标定义。

'+cards([ - ('统一验收流程','接口返回 → 排除空内容与错误页 → 核对题目要求的文字和字段 → 检查列表数量、类型与唯一性 → 汇总任务结果。只执行该例适用的检查；失败详情保留具体未满足项。图片与视频沿用请求和交付检查，资源地址或文件字节另按格式核验。'), + ('统一验收流程','接口返回 → 检查目标内容是否为空 → 核对题目要求的文字和字段 → 检查列表数量、类型与唯一性 → 汇总任务结果。只执行该例适用的检查；失败详情保留具体未满足项。图片与视频沿用请求和交付检查，资源地址或文件字节另按格式核验。'), ('任务通过率','满足该例全部验收规则的次数 ÷ 已有验收证据的任务。段落、列表、字段都属于内容要求；没有 F1 最低分门槛。请求失败计入分母；任何计划调用缺失或无法判分时，整份报告标为不可靠并禁止发布。'), ('文字核对覆盖','每例命中的必需文字组数 ÷ 登记组数，再按案例取均值。固定标题、说明、字段标签或事实句均可作为核对组；易变名单和价格不固定为答案。没有文字依据的案例不记作满分，覆盖案例数另列。'), ('请求成功率','接口正常返回的次数占全部调用的比例。接口返回成功，不代表抓到了目标内容。'), @@ -91,9 +91,9 @@ def metric_legend(headers): TRACK_CONTEXT = { 'web': { - 'flow': ['请求网页', '定位正文区域', '去除导航等杂文', '输出正文'], + 'flow': ['请求网页', '定位正文区域', '按题目范围保留文字', '输出正文'], 'flow_note': '目标是读出页面内容，而不是保存整份网页源码。正文若已在响应中，普通 HTTP 加解析器就可能完成。', - 'meaning': '常见网页的正文提取：从文章、文档站、百科或项目主页中取回要读的内容，剔除导航、广告等干扰。难点是找对正文边界，既不漏段落，也不混入大量杂文。', + 'meaning': '从文章、文档站、百科或项目主页中取回题目要求的内容。导航和页脚是否需要保留，由每道题登记的范围决定；它们不能替代要求的正文、列表或表格。', 'example': '项目仓库页以 README 为正文：只返回仓库名称和简介不够；把文件目录、导航一起带回来，也可能降低纯净度。', 'boundary': '这组按正文提取任务选样，没有逐页按渲染方式分组，不能等同于“纯静态网页”。请求成功、任务通过、全文质量分别衡量；不测站内搜索、点击或多页浏览。', }, @@ -166,8 +166,8 @@ def metric_legend(headers): def task_pass_guide(track, base='', cases=None): rules = { 'dynamic': '核验初次加载范围：稳定内容锚点（稳定标题、栏目、说明）与条目的必需字段、数量、唯一性和类型共同决定任务通过。频繁变化的名单、条目标题和价格按声明的结构与类型核验，不固定旧快照取值。HTTP 页面壳不通过；渲染、内嵌数据和公开 API 均可作为实现路径。全文质量仅在有完整、可比的参考时另算。', - 'web': '请求成功，并同时满足本例最低字符数、全部必需核对点、显著标记及结构要求；不得命中非参考正文自身包含的错误页标记。完整度、纯净度和 F1 另算，不作为通过门槛。', - 'document': '文档正文采用与网页相同的文字验收规则：请求、最低长度、全部核对点及错误页检查满足；不是只拿到下载链接。F1 另算，不作为通过门槛。', + 'web': '请求成功，并同时满足本例最低字符数、全部必需核对点、显著标记及结构要求。验证码或错误提示文字不单独决定任务是否通过。完整度、纯净度和 F1 另算，不作为通过门槛。', + 'document': '文档正文按请求状态、最低长度与全部内容核对点验收；只拿到下载链接不能通过。验证或错误提示只作诊断。F1另算，不作为通过门槛。', 'image': '含图页面须在实际返回值中找齐指定图片地址；直接 PNG 须交付文件字节并通过格式、长度与目标 SHA-256 验收。', 'video': '按案例区分：找齐播放入口、核对标题与发布者，或交付通过格式、长度与目标 SHA-256 验收的原文件。链接不等于文件，均不测画面理解。', 'web_media': '输入是网页，共 17 题：16 题返回指定图片、视频或视频入口地址，1 题返回视频页登记的文字信息。地址题不下载媒体文件；文字题也不据此推断已取得视频字节。', @@ -176,9 +176,16 @@ def task_pass_guide(track, base='', cases=None): 'risk_control': '请求成功并取得指定目标页面；正文分区、组合规则和登记为必需的稳定文字均须满足。登记为诊断的稳定文字只帮助核对抓取范围，不影响是否通过。文章不能只返回标题或开头，列表须达到登记的字段、真实链接、路径、去重和数量要求；验证页、空返回和页面外壳均不通过。', } text = rules.get(track, '按案例中登记的验收规则逐项检查。') + from .report_i18n import TEXT + missing_translations = { + 'dynamic': 'Validate the initial-load scope: stable text and required record fields, counts, uniqueness and types jointly determine acceptance. Volatile members, titles and prices are checked structurally rather than frozen to old values. An HTTP shell is insufficient; rendering, embedded data and public APIs are valid routes. Full-text quality is separate and requires a complete comparable reference.', + 'document': 'Document text must satisfy request status, minimum length and every registered content check; a download link alone is insufficient. Verification and error hints are diagnostic. F1 is reported separately and is not a pass threshold.', + } + text_en = TEXT.get(text, missing_translations.get(track, 'Check each registered task requirement.')) coverage=[c['body_requirements'] for c in (cases or {}).values() if c.get('body_requirements')] + coverage_note = '' if coverage: - text+=' 已登记正文覆盖规则的案例，按各例指定的比较单位、区域门槛和通过区域数验收，不能只返回标题或开头。' + coverage_note = ' 已登记正文覆盖规则的案例，按各例指定的比较单位、区域门槛和通过区域数验收，不能只返回标题或开头。' if any(r.get('unit')=='region_compact_hexagram_recall' for r in coverage): text+=' 字符六元组规则忽略排版空白，保留文字顺序；各区域的要求见题目详情。' for c in (cases or {}).values(): @@ -186,6 +193,7 @@ def task_pass_guide(track, base='', cases=None): if fields: labels={'financing':'分期','shipping':'配送'} text+=' '+c['name']+' 的字段覆盖要求：'+'、'.join(f'{labels.get(k,k)}至少 {v} 条' for k,v in fields.items())+'，各归属于完整且唯一的记录。' + text_en+=' '+c['name']+' field coverage: '+', '.join(f'{k}: at least {v} records' for k,v in fields.items())+'; every field must belong to a complete, unique record.' context = TRACK_CONTEXT.get(track) kinds = {c.get('task_kind', 'body') for c in (cases or {}).values() if c.get('track', 'web') == track} @@ -197,9 +205,16 @@ def task_pass_guide(track, base='', cases=None): 'file': '文件交付须返回原始字节，并通过格式、长度及已登记目标 SHA-256 校验；文字或链接不能替代文件。', } text = ' '.join(output_rules[kind] for kind in sorted(kinds) if kind in output_rules) + english_outputs = { + 'body': TEXT[rules['web']], + 'structure': 'Structured output must satisfy registered fields, types, counts, uniqueness and link rules.', + 'resource': 'Resource discovery must return the registered target addresses and answer fields.', + 'file': 'File delivery requires original bytes with valid format, length and any registered SHA-256; text or links cannot replace the file.', + } + text_en = ' '.join(english_outputs[k] for k in sorted(kinds) if k in english_outputs) context = None if not context: - return '
怎样才算任务通过？

'+escape(text)+'

' + return '
怎样才算任务通过？

'+''+escape(text)+''+coverage_note+'

' example_link = '' if context.get('case_id') in (cases or {}): example_link = ' 查看案例 →' @@ -208,7 +223,7 @@ def task_pass_guide(track, base='', cases=None): '
典型处理流程
    ' + ''.join('
  1. '+escape(step)+'
    2. ' for step in context['flow'])+'
' '

'+escape(context['flow_note'])+'

' - '
怎样才算任务通过？

'+escape(text)+'

' + '
怎样才算任务通过？

'+''+escape(text)+''+coverage_note+'

' '
任务例子与评测边界
' '
一个例子
'+escape(context['example'])+example_link+'
' '
范围与边界
'+escape(context['boundary'])+'
' @@ -256,7 +271,7 @@ def examples(): ('risk_control','风控网页正文',[ ('任务','打开指定网址，取得首次自动加载出的目标正文、列表和真实链接，不进行点击、输入、滚动、翻页或登录。'), ('核对依据','按独立保存的目标页面证据登记稳定文字、完整正文分区，以及名称—链接、路径、字段、去重和数量要求。稳定文字会明确标为必需或诊断。'), - ('怎么测','先排除请求失败、空返回、验证或拦截页面和页面外壳，再逐项检查本例声明的稳定文字、正文覆盖与组合规则；诊断文字只显示命中情况。'), + ('怎么测','先检查请求与返回内容，再逐项检查本例声明的稳定文字、正文覆盖与组合规则；诊断文字只显示命中情况。'), ('怎样算通过','正文与列表规则及标为必需的稳定文字均通过；诊断文字不影响结论。文章只返回标题或开头、列表数量不足、链接不符合真实路径，均不通过。'), ]), ] diff --git a/harness/web_fetch/report_i18n.py b/harness/web_fetch/report_i18n.py index 801899511..ae8c7981b 100644 --- a/harness/web_fetch/report_i18n.py +++ b/harness/web_fetch/report_i18n.py @@ -37,6 +37,7 @@ # Exact editorial translations take priority. Dynamic evidence (task names, # references and provider output) is deliberately absent from this catalog. TEXT = { + "先检查请求与返回内容，再逐项检查本例声明的稳定文字、正文覆盖与组合规则；诊断文字只显示命中情况。": "Check the request and returned content, then apply the declared stable text, body coverage and composite requirements. Diagnostic text reports matches without changing the verdict.", "本轮所有本地方法的每个主机名和跳转都使用固定的 Cloudflare HTTPS DNS，且不回退到系统 DNS。Moli、Lightpanda 和 Obscura 通过进程内解析器取得同一组公网地址；Chromium 使用自身的严格安全 DNS 配置。各浏览器原有的私网访问保护保持不变。": "Every local method uses the fixed Cloudflare HTTPS DNS service for each hostname and redirect, with no fallback to system DNS. Moli, Lightpanda, and Obscura receive the same public addresses through a process-scoped resolver; Chromium uses its own strict secure-DNS profile. Each browser keeps its existing private-network protection.", "Kitesurf 在 Cloudflare Browser Run 中执行，使用云端网络。它的任务结果可以按同一验收规则比较，但耗时包含不同的部署位置和公网路径，不能据此判断本地浏览器引擎谁更快。": "Kitesurf runs in Cloudflare Browser Run and uses its cloud network. Its task outcomes use the same acceptance rules, but its latency includes a different deployment location and public-network path and therefore cannot rank local browser-engine speed.", "CLI · 原生 Markdown 或原始响应路线": "CLI · native Markdown or raw-response route", @@ -66,8 +67,9 @@ "CLI · 按任务预选原生 Markdown、HTML 或原始响应路线": "CLI · preselects native Markdown, HTML, or raw-response routes by task", "网页内嵌媒体": "Embedded webpage media", "直接媒体文件": "Direct media files", - "输入是网页，共 17 题：16 题返回指定图片、视频或视频入口地址，1 题返回视频页登记的文字信息。地址题不下载媒体文件；文字题也不据此推断已取得视频字节。": "The input is a webpage. Across 17 tasks, 16 require specified image, video, or video-entry URLs and one requires registered text from a video page. URL tasks do not download the media file, and the text task does not imply that video bytes were obtained.", - "输入直接指向媒体文件，共 3 题：必须取得原始字节，并通过格式特征、长度与目标 SHA-256 验收。只复述输入网址不通过，也不要求把二进制内容转换成 Markdown。": "The input points directly to a media file. Across three tasks, the method must obtain the original bytes and pass format-signature, length, and target SHA-256 checks. Repeating the input URL fails, and binary content does not need conversion to Markdown.", + "输入是网页，共 17 题：16 题返回指定图片、视频或视频入口地址，1 题返回视频页登记的文字信息。地址题不下载媒体文件；文字题也不据此推断已取得视频字节。": "The input is a webpage. Across 17 tasks, 16 require specified image, video, or video-entry URLs and 1 requires registered text from a video page. URL tasks do not download the media file, and the text task does not imply that video bytes were obtained.", + "输入直接指向媒体文件，共 3 题：必须取得原始字节，并通过格式特征、长度与目标 SHA-256 验收。只复述输入网址不通过，也不要求把二进制内容转换成 Markdown。": "The input points directly to a media file. Across 3 tasks, the method must obtain the original bytes and pass format-signature, length, and target SHA-256 checks. Repeating the input URL fails, and binary content does not need conversion to Markdown.", + "16 道题只要求网页中的资源地址，另 1 道题要求视频页标题与发布者；这组不下载媒体文件。": "16 tasks require resource URLs from a webpage; 1 requires the video-page title and publisher. This group does not download media files.", "直接取得 Moli 原生 HTML；共同评分器只识别其中的资源 URL": "Returns Moli's native HTML directly; the shared scorer only identifies resource URLs in it", "使用 Moli 官方等待能力；只等待，不抽取或拼接正文": "Uses Moli's official wait capability for readiness only; it neither extracts nor assembles body text", "所有正文与结构任务使用 Moli 原生 Markdown。旧版 selector 合同只映射到官方等待参数，不参与正文抽取。图片、视频资源页使用原生 HTML，共同评分器只识别其中的资源 URL。直接 PDF/DOCX 仍调用同一原生接口；接口没有交付可读正文时按失败记录，评测端不下载后解析。文件使用原生 CDP 响应或同一二进制的原始附件接口。最终网址仅作诊断，不决定正文是否通过；失败后不切换格式或借用其他 provider。": "All body and structured tasks use Moli's native Markdown. Legacy selector contracts map only to official wait parameters and do not extract body text. Image and video resource pages use native HTML, from which the shared scorer only identifies resource URLs. Direct PDF and DOCX inputs still use the same native interface; if it does not deliver readable text, the task fails and the benchmark does not download and parse the file. File tasks use native CDP responses or the same binary's raw attachment interface. The final URL is diagnostic only and does not determine body-task acceptance. A failure never triggers a format switch or another provider.", @@ -101,7 +103,45 @@ "下载最终物理调用记录": "Download the final physical-call record", "。运行记录保留请求状态、耗时和证据路径；原始输出保持调用后保存的字节。": ". The record retains request status, duration and evidence paths. Raw output preserves the saved response bytes.", "请求成功，并同时满足本例最低字符数、全部必需核对点、显著标记及结构要求；不得命中非参考正文自身包含的错误页标记。完整度、纯净度和 F1 另算，不作为通过门槛。": "A successful request must satisfy the task's minimum length, all required text checks, salient markers and structure requirements. Error-page markers absent from the reference must not appear. Recall, precision and F1 are calculated separately and do not determine acceptance.", + "云端服务未公开后端版本": "The cloud service does not disclose its backend version", + "未取得可确认的版本信息": "No verifiable version information is available", + "本轮使用版本／源码提交": "Version / source commit used in this run", + "显示本轮使用的软件版本或源码提交。云端服务未公开可确认的后端版本号时，直接说明；这不代表服务没有版本。": "Software versions or source commits used for this run. Undisclosed cloud backend versions are stated explicitly; this does not mean the service has no version.", + "本轮启用自身 DEBUG 并捕获网络库日志。": "This run enabled native DEBUG output and captured network-library logs.", + "列出本轮核验后的修正；修正读取已保存的返回，不代表重新调用产品。": "Corrections after verification of this run use saved responses; they do not represent new product calls.", + "本轮结果修正": "Corrections to this run", + "详细依据": "Supporting evidence", + "核验范围与修正说明": "Review scope and corrections", + "产品原生返回；保留实际输出与日志": "Native product output; actual responses and logs are retained", + "缓存与等待选项见本轮固定配置": "See the frozen configuration for cache and wait settings", + "Firecrawl、Tavily、Exa、Cloudflare；通过各自 API 调用。": "Firecrawl, Tavily, Exa and Cloudflare, called through their respective APIs.", + "本轮实际固定配置": "Frozen settings used in this run", + "本轮保存了哪些日志": "Logs retained in this run", + "产品与依赖库日志范围": "Product and dependency log coverage", + + "比例只对应本报告列出的案例与配置。先看总榜和分榜，再打开具体案例，对照验收要求与实际返回。请求成功只表示方法正常返回；任务通过表示返回内容满足该案例的全部要求。": "Rates apply only to the cases and configurations in this report. Start with the overall and category rankings, then open individual cases to compare requirements with actual output. Request success means the method returned normally; task success means the returned content met every requirement for that case.", + "评分规则与计时范围见“评测方法”，检查和修正记录见“核验记录”，方法版本、固定配置与日志范围见“环境与复现”。": "See Methodology for scoring and timing, Review history for checks and corrections, and Reproduce for method versions, fixed options and log coverage.", "已保存的诊断日志": "Saved diagnostic logs", + "已保存的诊断日志 · Web Fetch": "Saved diagnostic logs · Web Fetch", + "完整原始响应": "Complete original response", + "完整原始响应 · Web Fetch": "Complete original response · Web Fetch", + "查看完整 API 响应（已隐藏凭据与本机路径）": "View complete API response (credentials and local paths hidden)", + "查看已保存的诊断日志（已隐藏凭据与本机路径）": "View saved diagnostic logs (credentials and local paths hidden)", + "已隐藏凭据与本机路径": "Credentials and local paths hidden", + "真实保存记录；仅为浏览显示，未增加产品调用。": "Actual saved records, formatted for browsing without additional product calls.", + "记录来源": "Record source", + "下载原始公开文件": "Download original public file", + "仅包含实际观测的日志。": "Contains only observed logs.", + "产品实际返回；公开副本已隐藏凭据与本机路径。": "Actual product response; credentials and local paths are hidden in this public copy.", + "或": "or", + "稳定内容与结构检查通过。": "Stable content and structure checks passed.", + "条目结构检查通过；本例未设置稳定文字核对点。": "Record structure passed; this task has no stable-text checks.", + "稳定文字或记录结构没有满足任务要求。": "Stable text or record structure did not meet the task requirements.", + "全部文字组都须命中。": "Every text group must match.", + "完整参考正文": "Complete reference body", + "查看原始验收规则": "View raw acceptance rules", + "正文长度要求": "Body length requirement", + "· 公开 SHA-256：": "· Public SHA-256:", "已保存文件截断；原生日志范围见说明": "Saved file truncated; see native log coverage notes", "已保存文件未截断；原生日志范围见说明": "Saved file not truncated; see native log coverage notes", "本次没有取得产品或依赖库的原生日志。已保存的调用方记录见下方。": "No native product or dependency logs were obtained for this call. Saved caller records appear below.", @@ -113,6 +153,35 @@ "已保存真实诊断（原生日志范围见说明）": "Observed diagnostics saved (see native log coverage notes)", "本轮未重新核验来源。": "Sources were not reverified for this run.", "本地实测版": "Local measured comparison", + "本地未认证诊断快照 · 不可用于排名": "Local uncertified diagnostic snapshot · not for ranking", + "来源未在调用前后完成全量核验。保存的评分、质量指标和复核记录仅供诊断，不代表正式发布或认证结果。": "Sources were not fully verified before and after calls. Saved scores, quality metrics and review records are diagnostic evidence, not a certified release.", + "本地诊断结果，不提供名次。": "Local diagnostic results; no ranks are assigned.", + "任务判定汇总（非排名）": "Task judgments (not a ranking)", + "按任务类别查看诊断结果": "Diagnostic results by task family", + "方法按名称排列，不提供名次。任务判定、内容质量和耗时分别展示。": "Methods are listed by name without ranks. Task judgments, content quality and latency are shown separately.", + "本地未认证诊断结果，仅展示保存的任务判定与耗时，不用于产品排名。通过不代表返回内容干净；请同时查看全文指标和挑战提示。": "Local uncertified diagnostic results show saved task judgments and latency, not product rankings. Passing does not establish clean content; also inspect full-text metrics and challenge indicators.", + "诊断指标说明": "Diagnostic metric definitions", + "无命中": "No matches", + "评分修正账目": "Scoring revision ledger", + "每轮增加、减少与最终通过数均从逐条状态记录计算；排除结果保留原始调用，不计入比较分母。": "Additions, removals and final passes are derived from individual result states. Excluded results retain their original calls and stay outside comparison denominators.", + "阶段": "Stage", + "修正前通过": "Passes before", + "新增通过": "Passes added", + "减少通过": "Passes removed", + "修正后通过": "Passes after", + "下载逐条增减账目": "Download per-result revision ledger", + "通过结果的内容诊断": "Content diagnostics for passing results", + "目标覆盖与附加内容": "Target coverage and additional content", + "目标要求：": "Target requirements:", + "附加内容诊断：": "Additional-content diagnostics:", + "参考词精确率：": "Reference word precision:", + "包含验证码组件文字": "Challenge-widget text is present", + "未检测到该组件文字特征": "No matching widget-text signature detected", + "排除比较": "Excluded from comparison", + "通过只表示满足登记的目标要求，不保证干净正文。提示词命中不证明整页被拦截；词精确率也受参考范围和时间影响。低于 30% 仅用于定位检查对象，不是统一失败线。": "Passing means the registered target requirements were met, not that the body is clean. Hint words do not establish that the whole page was blocked; word precision also depends on reference scope and time. Below 30% identifies inspection candidates, not a universal failure threshold.", + "评测环境异常：统一排除比较": "Evaluation environment defect: excluded across all methods", + "已有复核记录；覆盖和裁定依据见核验页面。": "Review records are available; see the review page for coverage and adjudication evidence.", + "结果为机器判定，未进行独立人工审核。": "Results are machine judgments without independent human review.", "全部方法使用同一组已提交案例和判分规则，每例一次调用，统一45秒外部时限，不设逐例等待或提取参数，不重试。结果为机器判定，未进行独立人工审核；本报告仅供本地查看，不代表正式发布。": "All methods use the same committed cases and scoring rules: one call per case, a shared 45-second caller deadline, no per-case wait or extraction parameters, and no retries. Results are machine-scored without independent human review. This report is for local inspection and is not a formal release.", "调用链与日志检查": "Call-chain and log audit", "题目通过": "Task passes", @@ -120,6 +189,7 @@ "原生日志范围": "Native log coverage", "检查结论": "Audit findings", "如何读取本轮结果": "How to read these results", + "本轮按保存的实际返回进行机器判分，比例只对应本报告列出的案例与配置。日志可用于复查调用和判分；失败原因须结合原始响应、调用记录及核验记录判断，不能一概归因于产品内部。": "Saved outputs are machine-scored. Rates apply only to the cases and configurations listed here. Assess failure causes using the native response, call log and review record; failures cannot all be attributed to product internals.", "本轮按保存的实际返回进行机器判分，比例只对应本报告列出的案例与配置。日志可用于复查调用和判分；未经过独立人工审核，不能将失败一概归因于产品内部。": "Saved outputs are machine-scored. Rates apply only to the cases and configurations listed here. Logs support review of calls and scoring; without independent human review, failures cannot all be attributed to product internals.", "逐次结果与筛选": "Per-call results and filters", "题目与计时规则": "Tasks and timing rules", @@ -207,6 +277,8 @@ "从官方 v1.1.7 release 获取二进制，配置 MOLI_1_1_7_BINARY。": "Obtain the binary from the official v1.1.7 release and configure MOLI_1_1_7_BINARY.", "Moli 1.1.7 正式发布": "Moli 1.1.7 official release", "本方法只使用官方 1.1.7 发行二进制，与其他 Moli 版本分开计分。普通网页使用原生 Markdown；直接文档与媒体的裸 fetch 路线在正式二进制上实测确认。正文任务若取得受支持的原始文档，会在同一计时预算内解析文字；文件任务直接验收字节。不在失败后切换引擎或输出模式。": "This method uses only the official 1.1.7 release binary and is scored separately from other Moli versions. Ordinary web pages use native Markdown; bare-fetch routes for direct documents and media were verified on the release binary. If a body-text task retrieves a supported original document, text is parsed within the same timing budget; file tasks validate bytes directly. The method does not switch engines or output modes after a failure.", + "风控网站": "Risk-control websites", + "历史访问受限网址内容": "Historically access-restricted URL content", "风控网站正文": "Risk-control website body text", "请求指定网址": "Request the specified URL", "识别目标页或拦截页": "Identify the target page or a blocked page", @@ -1736,6 +1808,8 @@ "请求网页": "Request page", "定位正文区域": "Locate body region", "去除导航等杂文": "Remove navigation and noise", + "按题目范围保留文字": "Retain text within the task scope", + "从文章、文档站、百科或项目主页中取回题目要求的内容。导航和页脚是否需要保留，由每道题登记的范围决定；它们不能替代要求的正文、列表或表格。": "Retrieve the required content from articles, documentation, encyclopedias or project pages. Each task declares whether navigation and footer text must be retained; they cannot substitute for the required body, lists or tables.", "输出正文": "Return body text", "目标是读出页面内容，而不是保存整份网页源码。正文若已在响应中，普通 HTTP 加解析器就可能完成。": "The goal is readable page content, not a copy of the complete page source. When the body is already present in the response, ordinary HTTP plus a parser may be sufficient.", "项目仓库页以 README 为正文：只返回仓库名称和简介不够；把文件目录、导航一起带回来，也可能降低纯净度。": "For a repository page, the README is the body target. A repository name and short description are insufficient, while including the file tree and navigation may reduce precision.", @@ -2326,11 +2400,26 @@ # rendered on the methodology page and therefore need full English editions; # measured task names and prompts remain unchanged. TEXT.update({ - "输入网页，从页面里找到指定图片、视频或视频入口地址；另有 1 题读取视频页标题与发布者。": "Given a web page, find the specified image, video, or playback URL. One additional task asks for the video-page title and publisher.", - "16 道地址题使用独立保存的资源地址或地址规则；1 道文字题使用可见标题与发布者。": "Sixteen URL tasks use independently saved resource URLs or URL rules. One text task uses the visible title and publisher.", + "从网页中找到指定图片、视频资源或视频入口地址，或读取视频页登记的文字信息。这里测的是网页取数，不是图片或视频理解。": "Find the specified image, video, or video-entry URLs on a webpage, or read its registered video information. These tasks measure webpage data retrieval; image and video understanding are outside their scope.", + "请求媒体网页": "Request the media webpage", + "定位指定资源或信息": "Locate the required resource or information", + "提取地址或文字": "Extract URLs or text", + "交付题目要求的结果": "Return the required result", + "含图页面须返回登记的图片地址；视频信息页须返回标题与发布者。拿到地址不代表已经取得对应文件。": "Image pages must return the registered image URLs; video-information pages must return the title and publisher. Returning a URL does not establish that its file was obtained.", + "地址题只核对目标地址，文字题只核对登记信息；不检查画面、字幕、声音，也不要求下载媒体字节。": "URL tasks check the target URLs and text tasks check the registered information. They do not assess frames, captions, or audio, and do not require downloading media bytes.", + "取得直接媒体文件。必须返回可验证的图片或视频字节；复述网址、返回文件名或网页文字都不能代替文件。": "Retrieve a direct media file and return verifiable image or video bytes. Repeating a URL, filename, or webpage text cannot substitute for the file.", + "请求直接文件地址": "Request the direct file URL", + "接收响应字节": "Receive response bytes", + "识别文件格式": "Identify the file format", + "核对长度与 SHA-256": "Verify length and SHA-256", + "输入直接指向图片或视频文件，评测对象是返回的原始字节，不进行 Markdown 转换。": "The input points directly to an image or video file. Evaluation uses the returned original bytes without Markdown conversion.", + "直接 PNG 地址须交付与独立保存样本一致的文件格式、长度和 SHA-256；只返回该 PNG 的网址不通过。": "A direct PNG task must deliver a file matching the independently saved sample's format, length, and SHA-256. Returning only the PNG URL fails.", + "这组只验证文件交付，不测试图片描述、视频理解、语音转写或字幕质量。": "These tasks assess file delivery only. Image descriptions, video understanding, speech transcription, and caption quality are outside their scope.", + "输入网页，从页面里找到指定图片、视频或视频入口地址；另有 1 题读取视频页标题与发布者。": "Given a web page, find the specified image, video, or playback URL. 1 additional task asks for the video-page title and publisher.", + "16 道地址题使用独立保存的资源地址或地址规则；1 道文字题使用可见标题与发布者。": "16 URL tasks use independently saved resource URLs or URL rules. 1 text task uses the visible title and publisher.", "地址题只核对返回值里是否出现目标地址，不要求另行下载媒体文件；文字题按登记事实核对。": "URL tasks check whether the returned value contains the target URL and do not require a separate media download. The text task checks the registered facts.", "完成该题声明的地址或文字交付。拿到地址不等于取得文件，也不表示理解了图片、画面、字幕或声音。": "Return the URL or text required by the task. A URL alone does not deliver the file or show understanding of images, video, captions, or audio.", - "输入 1 个直接图片地址或 2 个直接视频地址，取得原始文件。": "Retrieve the original file from one direct image URL or either of two direct video URLs.", + "输入 1 个直接图片地址或 2 个直接视频地址，取得原始文件。": "Retrieve the original file from 1 direct image URL or either of 2 direct video URLs.", "独立保存的原文件格式、长度和 SHA-256。": "Independently saved file format, length, and SHA-256.", "核对实际返回字节的格式特征、长度和目标 SHA-256；不对二进制文件做 Markdown 转换。": "Check the returned bytes against the required file signature, length, and SHA-256. Binary files are not converted to Markdown.", "必须实际交付匹配的文件字节；复述输入网址或只返回网页文字不通过。本组不测试媒体理解。": "The method must return the matching file bytes. Repeating the input URL or returning page text does not pass. These tasks do not test media understanding.", @@ -2489,6 +2578,7 @@ # are deliberately narrow report phrases; they never include task names, # reference answers, or provider output. PHRASES = { + "处理路线：": "Retrieval path: ", "Moli 1.1.15 · 独立资源批次": "Moli 1.1.15 · independent resource cohort", "的 Moli 已保存输出仍包含": "'s saved Moli output still contains ", "，内容规则未通过。请求成功在这里记录了一个正常返回的结果，却没有等来任务要求的内容。这比方法是否具备浏览器功能更接近本次接入要解决的问题。": ", but failed the content rule. A normal response did not wait for the required content. This is closer to the integration problem than whether the method has browser features.", @@ -2626,6 +2716,17 @@ TEXT.update(adoption_text) +def _body_region_summary(match): + regions = match[1] + for source, target in (("开头", "Opening"), ("前部", "Early section"), + ("中部", "Middle"), ("后部", "Late section"), + ("结尾", "Ending"), ("位置", "Region")): + regions = regions.replace(source + " ", target + " ") + return ("Body coverage is incomplete. Regions below target: " + + regions.replace("、", ", ") + + ". The complete reference and actual output are shown side by side below.") + + REGEX = [ (re.compile(r"^每次物理调用预算为 ([\d.]+) 秒，重试后的累计耗时可能超过该预算。$"), r"Each physical call has a \1-second budget; accumulated latency after retries can exceed it."), (re.compile(r"^本报告包含 (\d+) 个任务、(\d+) 个逻辑结果，任务通过 (\d+)/(\d+)。结论仅适用于已登记的任务和规则。$"), r"This report contains \1 tasks and \2 logical results, with \3/\4 task passes. Conclusions apply only to the registered tasks and rules."), @@ -2682,6 +2783,7 @@ lambda match: f"The evaluation does not cover its plan; {match.group(1)} unplanned calls are present" + (f"; {match.group(2)} calls have no pass/fail judgement" if match.group(2) else "") + ". Rankings on this page must not be used for conclusions or publication."), + (re.compile(r"^(\d+) 个案例 · (\d+) 种方法 · 本地未认证诊断快照。$"), r"\1 tasks · \2 methods · local uncertified diagnostic snapshot."), (re.compile(r"^(\d+) 个案例 · (\d+) 种方法 · (\d+)\s*类分榜与一个综合通过率总榜。$"), r"\1 tasks · \2 methods · \3 family rankings and one overall coverage ranking."), (re.compile(r"^(\d+)\s*类任务概览$"), r"\1 task families"), (re.compile(r"^(\d+)\s*类分榜$"), r"Rankings for \1 task families"), @@ -2793,7 +2895,7 @@ (re.compile(r"^实际解析出 (\d+) 条完整且不重复的记录，要求至少 (\d+) 条；结构检查未通过，稳定文字检查未通过。$"), r"Parsed \1 complete, unique records against a minimum of \2; structure and stable-text checks failed."), (re.compile(r"^实际解析出 (\d+) 条完整且不重复的记录，要求至少 (\d+) 条；结构检查(通过|未通过)，稳定文字检查(通过|未通过)。$"), lambda m: f"Parsed {m.group(1)} complete, unique records against a minimum of {m.group(2)}; structure {'passed' if m.group(3) == '通过' else 'failed'} and stable-text checks {'passed' if m.group(4) == '通过' else 'failed'}."), (re.compile(r"^完整度 / 纯净度 / F1 为 ([^。]+)。$"), r"Recall / precision / F1: \1."), - (re.compile(r"^正文没有抓全；低于目标的位置为：(.+)。下方会把完整标准答案与实际返回并排展示。$"), r"Body coverage is incomplete. Regions below target: \1. The complete reference and actual output are shown side by side below."), + (re.compile(r"^正文没有抓全；低于目标的位置为：(.+)。下方会把完整标准答案与实际返回并排展示。$"), _body_region_summary), (re.compile(r"^这一段标准答案产生 ([\d,]+) 个连续六字符片段；在上方完整实际返回（([\d,]+) 字符）中命中 ([\d,]+) 个，所以覆盖率为 ([\d,]+) ÷ ([\d,]+) = ([\d.]+%)。目标 ([\d.]+%)，硬下限 ([\d.]+%)。$"), r"This reference region produces \1 contiguous six-character shingles. \3 are found in the complete output above (\2 characters), so coverage is \4 ÷ \5 = \6. Target: \7; hard floor: \8."), (re.compile(r"^· 采集时间 (.+)$"), r"· captured at \1"), ] @@ -3558,6 +3660,12 @@ def add(source, target): }) REGEX.extend([ + (re.compile(r"^核验于 (\d{4}-\d{2}-\d{2})。算力、网络、存储、代理和维护成本自担；没有测量这些成本，不把本地调用等同于零成本。官方托管产品或服务价格不属于本地执行模式。$"), + r"Checked on \1. The caller pays compute, network, storage, proxy, and maintenance costs. These costs were not measured, so local calls are not treated as free. Official managed-product or service prices do not apply to local execution."), + (re.compile(r"^本方法只使用官方 (\d+(?:\.\d+)+) 发行二进制，与其他 Moli 版本分开计分。网页正文、文档正文、结构题和网页内嵌媒体题统一使用原生 Markdown；动态网站正文整类统一启用 --layout、--with-frames 和 domstable，不按题目设置 selector。网页媒体只核对这份输出中的目标地址，不下载，也不切换 HTML、JSON 或网络跟踪接口。直接 PDF/DOCX 没有交付可读正文时按失败记录；直接媒体文件验收原始字节。评测端不使用外部 PDF/DOCX 解析器补正文，也不在失败后切换引擎或输出模式。$"), + r"This method uses only the official \1 release binary and is scored separately from other Moli versions. Webpage body text, document body text, structured tasks, and webpage media tasks use native Markdown. Every dynamic-body task enables --layout, --with-frames, and domstable, without task-specific selectors. Webpage media tasks check only target URLs in that output; they do not download media or switch to HTML, JSON, or network-tracing interfaces. Direct PDF/DOCX tasks fail without readable body text; direct media files are evaluated as original bytes. The benchmark does not supply external PDF/DOCX parsing or switch engines or output modes after a failure."), + (re.compile(r"^必需文字组 (\d+)（任一写法即可）$"), r"Required text group \1 (any alternative may match)"), + (re.compile(r"^· 原始采集记录 ([\d,]+) 字节（可含协议返回） · 原始 SHA-256：$"), r"· Original capture: \1 bytes (may include protocol responses) · Original SHA-256:"), (re.compile(r"^本报告纳入 ([\d,]+) 个逻辑结果。运行参数见下表；保留的物理调用与输出见各结果详情。$"), r"This report includes \1 logical results. Run parameters are listed below; each result detail contains its retained physical calls and outputs."), (re.compile(r"^· 请求：(成功|失败) · 耗时：([\d,.]+) ms · 实际输出：([\d,]+) 字符$"), @@ -3606,10 +3714,19 @@ def __init__(self, source: PurePosixPath | None = None, translations=None) -> No self.stack: list[str] = [] self.source = source self.translations = translations + self.translate_stack = [] + self.model_text_stack = [] + + def _allows_translation(self, attrs): + mode = (dict(attrs).get("translate") or "").lower() + if mode in {"yes", "no"}: + return mode == "yes" + return self.translate_stack[-1] if self.translate_stack else True def _start_tag(self, attrs) -> str: original = self.get_starttag_text() - if any(tag in self.stack for tag in ("pre", "code", "script", "style")): + if (not self._allows_translation(attrs) + or any(tag in self.stack for tag in ("pre", "code", "script", "style"))): return original values = dict(attrs) article = self.source in {PurePosixPath("guide/index.html"), PurePosixPath("guide/article.html")} @@ -3620,7 +3737,7 @@ def replace_attribute(markup, name, value): return pattern.sub(lambda match: match.group(1) + '"' + html.escape(value, quote=True) + '"', markup) - if article: + if article or self.source == PurePosixPath("methodology/index.html"): # The renderer supplies both language assets. Localization only selects # an existing URL; it never reads files or modifies measured evidence. for name in ("src", "href"): @@ -3641,8 +3758,14 @@ def handle_decl(self, decl: str) -> None: self.parts.append(f"") def handle_starttag(self, tag: str, attrs) -> None: + localized = dict(attrs).get('data-report-en') if tag == 'span' else None + parent_model = any(self.model_text_stack) + self.model_text_stack.append(localized is not None or parent_model) + self.translate_stack.append(self._allows_translation(attrs)) self.stack.append(tag) self.parts.append(self._start_tag(attrs)) + if localized is not None and not parent_model: + self.parts.append(html.escape(localized)) def handle_startendtag(self, tag: str, attrs) -> None: self.parts.append(self._start_tag(attrs)) @@ -3650,18 +3773,27 @@ def handle_startendtag(self, tag: str, attrs) -> None: def handle_endtag(self, tag: str) -> None: if tag in self.stack: self.stack = self.stack[:len(self.stack) - 1 - self.stack[::-1].index(tag)] + self.translate_stack = self.translate_stack[:len(self.stack)] + self.model_text_stack = self.model_text_stack[:len(self.stack)] self.parts.append(f"") def handle_data(self, data: str) -> None: - if any(tag in self.stack for tag in ("pre", "code", "script", "style")): + if any(self.model_text_stack): + return + if ((self.translate_stack and not self.translate_stack[-1]) + or any(tag in self.stack for tag in ("pre", "code", "script", "style"))): self.parts.append(data) else: self.parts.append(translate_text(data, self.translations)) def handle_entityref(self, name: str) -> None: + if any(self.model_text_stack): + return self.parts.append(f"&{name};") def handle_charref(self, name: str) -> None: + if any(self.model_text_stack): + return self.parts.append(f"&#{name};") def handle_comment(self, data: str) -> None: @@ -3766,18 +3898,451 @@ def localize_html(page: str, source: PurePosixPath, *, translations=None) -> str english = PurePosixPath("en") / source page = page.replace('', '', 1) page = re.sub(r'
.*?
', _header(source, english), page, count=1, flags=re.S) + # Raw evidence and executable/style blocks are preserved, never translated. + # Mask them before parsing so large responses do not incur entity-by-entity work. + preserved = [] + def preserve(match): + preserved.append(match.group(0)) + return f"" + page = re.sub(r"<(pre|script|style)\b[^>]*>.*?", preserve, page, flags=re.S | re.I) parser = Localizer(source, translations) parser.feed(page) parser.close() - return _rewrite_links("".join(parser.parts), source, english) + localized = "".join(parser.parts) + for index, block in enumerate(preserved): + localized = localized.replace(f"", block) + return _rewrite_links(localized, source, english) + + +TEXT.update({ + "安装与账号准备": "Installation and account setup", + "账号与安装 · Web Fetch": "Access & setup · Web Fetch", +}) +TEXT.update({'内容选择': 'Content selection', + '典型处理流程': 'Typical processing flow', + '变更记录 · Web Fetch': 'Changelog · Web Fetch', + '已知 URL → 网页正文、页面资源地址或直接文档文字': 'Known URL → page body, resource URLs, or direct document text', + '已知 URL → 可读文本/文档文字': 'Known URL → readable text or document text', + '本地浏览器渲染 + Markdown 抽取': 'Local browser rendering + Markdown extraction', + '匿名 HTTP provider + Turndown': 'Anonymous HTTP provider + Turndown', + '已知 URL → 正文文本': 'Known URL → body text', + 'HTTP + BeautifulSoup 网页文本加载器': 'HTTP + BeautifulSoup page-text loader', + '已知 URL → 渲染正文、结构化资源 URL 或原文件字节': 'Known URL → rendered body, structured resource URLs, or original file ' + 'bytes', + 'HTTP GET + Readability 正文抽取（关闭 Firecrawl 回退）': 'HTTP GET + Readability body extraction (Firecrawl fallback ' + 'disabled)', + '已知 URL → raw_content': 'Known URL → raw_content', + '逐任务结果与证据': 'Per-task results and evidence', + '运行参数与版本': 'Run parameters and versions', + '本报告纳入 4,300 个逻辑结果。运行参数见下表；保留的物理调用与输出见各结果详情。': 'This report includes 4,300 logical results. Run parameters ' + 'are listed below; retained physical calls and outputs are ' + 'available in each result detail.', + '以下选项来自调用前冻结的配置。每题调用一次，外层截止时间45秒；产品内部超时保留默认值。调用间隔不计入单次耗时。这里只展示影响抓取与日志的选项，具体请求和按任务选择的输出格式见各案例的调用日志。': 'These ' + 'options ' + 'come ' + 'from ' + 'configuration ' + 'frozen ' + 'before ' + 'calls. ' + 'Each ' + 'task ' + 'is ' + 'called ' + 'once ' + 'with ' + 'an ' + 'outer ' + 'deadline ' + 'of 45 ' + 'seconds; ' + 'product-internal ' + 'timeouts ' + 'keep ' + 'their ' + 'defaults. ' + 'Inter-call ' + 'intervals ' + 'are ' + 'excluded ' + 'from ' + 'call ' + 'latency. ' + 'Only ' + 'retrieval ' + 'and ' + 'logging ' + 'options ' + 'are ' + 'shown ' + 'here; ' + 'see ' + 'each ' + 'call ' + 'log ' + 'for ' + 'actual ' + 'requests ' + 'and ' + 'task-dependent ' + 'output ' + 'formats.', + '固定选项': 'Fixed options', + '页面完成条件：': 'Page completion condition:', + '阻止访问私网：': 'Block private-network access:', + '返回内容上限（字节）：': 'Maximum returned bytes:', + '调用间隔（秒）：': 'Inter-call interval (seconds):', + '日志级别：': 'Log level:', + '完整保存诊断日志：': 'Retain full diagnostic logs:', + '日志保存上限（字节）：': 'Maximum retained log bytes:', + '使用统一固定原生选项：': 'Use uniform fixed native options:', + '域名解析方式：': 'DNS resolution mode:', + '缓存最长年龄：': 'Maximum cache age:', + '代理选择：': 'Proxy selection:', + '请求输出格式：': 'Requested output formats:', + '只取主要正文：': 'Main content only:', + '提取深度：': 'Extraction depth:', + '包含图片：': 'Include images:', + '正文详细程度：': 'Text verbosity:', + '缓存最长年龄（小时）：': 'Maximum cache age (hours):', + '包含资源链接：': 'Include resource links:', + '具名链接数量上限：': 'Maximum named links:', + '文档大小上限（字节）：': 'Maximum document bytes:', + '每次调用都保存了测试程序的调用记录：何时开始、传入什么参数、何时返回，以及是否发生错误。各案例详情中的“调用方事件”展示这些记录。': 'Every call retains harness events: ' + 'start time, input parameters, return ' + 'time, and errors. These appear under ' + 'Caller events in each case detail.', + '“产品与依赖库日志”展示产品或底层库实际输出的运行信息。云端服务没有开放的内部日志，不会用测试程序的记录代替。下表说明各方法留下了哪些日志。': 'Product and dependency logs ' + 'contain runtime information ' + 'actually emitted by the product ' + 'or underlying libraries. Harness ' + 'events do not substitute for ' + 'unavailable cloud-internal logs. ' + 'The table lists what was ' + 'retained for each method.', + '已启用产品 debug；日志文件另含调用方记录。': 'Product debug enabled; the log file also contains caller events.', + '保留真实请求、返回错误及可用请求标识；未取得云服务内部 debug，后台能力尚未确认。': 'Actual requests, returned errors, and available request IDs ' + 'retained; cloud-internal debug was not obtained, and ' + 'backend capabilities remain unconfirmed.', + '已启用 debug；实际原生日志可能很少，不能以日志文件存在推断内部过程完整。': 'Debug enabled; native logs may be sparse, and the presence of a ' + 'log file does not prove complete internal-process coverage.', + '保留真实调用、返回和错误；此抓取路径未确认可开启内部 debug。': 'Actual calls, responses, and errors retained; internal debug ' + 'availability for this retrieval path is unconfirmed.', + '保留底层 HTTP 库日志；不等同于完整产品内部执行日志。': 'Underlying HTTP-library logs retained; these are not complete ' + 'product-internal execution logs.', + '启用原生 verbose 运行日志，并捕获进程输出。': 'Native verbose logging enabled, with process output captured.', + '完整日志在各案例详情中查看。日志可以帮助定位问题，但没有记录的内部过程不能据此推断；日志文件存在，也不等于产品的全部内部过程都有记录。': 'Full retained logs are available in ' + 'each case detail. They help locate ' + 'problems, but unrecorded internal ' + 'processes cannot be inferred; a log ' + 'file does not mean every internal ' + 'process was recorded.', + '以下命令从本轮保存的证据构建本地报告，不重新调用产品：': 'These commands build the local report from saved evidence without ' + 'additional product calls:', + '使用与本轮结果对应的证据目录。调用时的版本、配置和评分器哈希见': 'Use the evidence directory corresponding to these results. Call-time ' + 'versions, configuration, and scorer hashes are in', + '调用前冻结协议': 'Pre-call frozen protocol', + '。上面的构建命令读取保存的评分；修改评分规则后，应先按声明范围复评分，不能仅重建页面就称成绩已经更新。': '. The build commands read saved scores. After ' + 'changing scoring rules, rescore the declared scope ' + 'before rebuilding; rebuilding pages alone does not ' + 'update scores.', + '430 个计划任务中的通过数 / 可判定数；每题等权，无法判定结果不参与分母。': 'Passed / decidable results among 430 planned tasks; each task ' + 'has equal weight, and undecidable results are excluded from the ' + 'denominator.', + '每种方法均使用同一组 430 个任务，包含通过、未通过、错误和超时。耗时是单次调用从输入到返回的时间，不包含调用之间的统一冷却间隔。每种方法内部按案例依次执行；方法之间的并发安排和限制见调用链检查。耗时仍受本地与远程环境、网络和产品内部限制影响。': 'Each ' + 'method ' + 'uses ' + 'the ' + 'same ' + '430 ' + 'tasks, ' + 'including ' + 'passes, ' + 'failures, ' + 'errors, ' + 'and ' + 'timeouts. ' + 'Latency ' + 'runs ' + 'from ' + 'input ' + 'to ' + 'return ' + 'and ' + 'excludes ' + 'the ' + 'common ' + 'cooldown ' + 'between ' + 'calls. ' + 'Cases ' + 'execute ' + 'sequentially ' + 'within ' + 'each ' + 'method; ' + 'inter-method ' + 'scheduling ' + 'and ' + 'limits ' + 'are ' + 'documented ' + 'in ' + 'the ' + 'call-chain ' + 'audit. ' + 'Local ' + 'and ' + 'remote ' + 'environments, ' + 'networks, ' + 'and ' + 'product ' + 'limits ' + 'still ' + 'affect ' + 'latency.', + '全部调用均值': 'All-call mean', + '全部调用 P50': 'All-call P50', + '全部调用 P95': 'All-call P95', + '请求成功，并同时满足本例最低字符数、全部必需核对点、显著标记及结构要求。验证码或错误提示文字不单独决定任务是否通过。完整度、纯净度和 F1 另算，不作为通过门槛。 已登记正文覆盖规则的案例，还须在最多 5 个参考区域中逐区达到 80% 词语计数覆盖，不能只返回标题或开头。': 'The ' + 'request ' + 'must ' + 'succeed ' + 'and ' + 'satisfy ' + 'the ' + 'case ' + 'minimum ' + 'length, ' + 'every ' + 'required ' + 'checkpoint, ' + 'significant ' + 'marker, ' + 'and ' + 'structure ' + 'requirement. ' + 'Verification ' + 'or ' + 'error ' + 'text ' + 'does ' + 'not ' + 'independently ' + 'determine ' + 'acceptance. ' + 'Completeness, ' + 'purity, ' + 'and ' + 'F1 ' + 'are ' + 'separate ' + 'metrics, ' + 'not ' + 'pass ' + 'thresholds. ' + 'Cases ' + 'with ' + 'body-coverage ' + 'rules ' + 'must ' + 'reach ' + '80% ' + 'token-count ' + 'coverage ' + 'in ' + 'each ' + 'of ' + 'up ' + 'to ' + '5 ' + 'reference ' + 'regions; ' + 'a ' + 'title ' + 'or ' + 'opening ' + 'alone ' + 'is ' + 'insufficient.', + '文档正文按请求状态、最低长度与全部内容核对点验收；只拿到下载链接不能通过。验证或错误提示只作诊断。F1另算，不作为通过门槛。 已登记正文覆盖规则的案例，还须在最多 5 个参考区域中逐区达到 80% 词语计数覆盖，不能只返回标题或开头。': 'Document ' + 'text ' + 'is ' + 'accepted ' + 'by ' + 'request ' + 'status, ' + 'minimum ' + 'length, ' + 'and ' + 'all ' + 'content ' + 'checkpoints; ' + 'a ' + 'download ' + 'link ' + 'alone ' + 'is ' + 'insufficient. ' + 'Verification ' + 'and ' + 'error ' + 'notices ' + 'are ' + 'diagnostic ' + 'only. ' + 'F1 ' + 'is ' + 'separate, ' + 'not ' + 'a ' + 'pass ' + 'threshold. ' + 'Cases ' + 'with ' + 'body-coverage ' + 'rules ' + 'must ' + 'reach ' + '80% ' + 'token-count ' + 'coverage ' + 'in ' + 'each ' + 'of ' + 'up ' + 'to ' + '5 ' + 'reference ' + 'regions; ' + 'a ' + 'title ' + 'or ' + 'opening ' + 'alone ' + 'is ' + 'insufficient.', + '不支持输入类型': 'Unsupported input type', + '任务规则未满足': 'Task requirements not met', + '取页返回 404': 'Retrieval returned 404', + '取页连接失败': 'Retrieval connection failed', + '方法安全限制': 'Method safety restriction', + '方法禁止跨站跳转': 'Method blocks cross-site redirects', + '方法空页规则拦截': 'Method empty-page rule blocked retrieval', + '正文提取为空': 'Body extraction empty', + '源站要求验证码': 'Source site requires verification', + '搜索': 'Search', + '搜索案例、方法或原因': 'Search cases, methods, or reasons', + '参考标注': 'Reference annotations', + '任务验收': 'Task acceptance', + '评分口径': 'Scoring scope', + '核对请求是否成功，再逐项检查本例要求的稳定文字、正文覆盖与完整条目。验证或拦截提示只作诊断，出现提示不单独导致失败；只有页面外壳而没有合格目标内容仍不通过。': 'Check request success, ' + 'then validate the ' + 'required stable text, ' + 'body coverage, and ' + 'complete records. ' + 'Verification or blocking ' + 'notices are diagnostic ' + 'only and do not ' + 'independently cause ' + 'failure; a page shell ' + 'without qualifying ' + 'target content still ' + 'fails.', + '接口返回 → 检查目标内容是否为空 → 核对题目要求的文字和字段 → 检查列表数量、类型与唯一性 → 汇总任务结果。只执行该例适用的检查；失败详情保留具体未满足项。图片与视频沿用请求和交付检查，资源地址或文件字节另按格式核验。': 'Response ' + '→ ' + 'check ' + 'for ' + 'empty ' + 'target ' + 'content ' + '→ ' + 'verify ' + 'required ' + 'text ' + 'and ' + 'fields ' + '→ ' + 'check ' + 'list ' + 'count, ' + 'types, ' + 'and ' + 'uniqueness ' + '→ ' + 'aggregate ' + 'the ' + 'task ' + 'verdict. ' + 'Only ' + 'checks ' + 'applicable ' + 'to ' + 'that ' + 'case ' + 'run; ' + 'failure ' + 'details ' + 'retain ' + 'unmet ' + 'requirements. ' + 'Images ' + 'and ' + 'videos ' + 'use ' + 'request ' + 'and ' + 'delivery ' + 'checks, ' + 'with ' + 'resource ' + 'URLs ' + 'or ' + 'file ' + 'bytes ' + 'validated ' + 'by ' + 'format.'}) +REGEX.insert(0, (re.compile(r"^源码提交：(.+)$"), r"Source commit: \1")) + +TEXT.update({'本轮修正与检查限制': 'Corrections and review limits'}) +PHRASES.update({"Codex 独立复核": "Codex independent review", "见原始复核材料绑定记录": "see the binding in the original review evidence", "见原始复核材料": "see original review evidence"}) -def write_english_report(out: Path) -> int: - """Mirror every Chinese HTML route and return the mirrored route count.""" +TEXT.update({ + "请求成功，并同时满足本例最低字符数、全部必需核对点、显著标记及结构要求。验证码或错误提示文字不单独决定任务是否通过。完整度、纯净度和 F1 另算，不作为通过门槛。": "The request must succeed and satisfy minimum length, every required checkpoint, significant marker, and structure requirement. Verification or error notices do not independently determine acceptance. Completeness, purity, and F1 are separate metrics, not pass thresholds.", +}) +REGEX.insert(0, (re.compile(r"^(\d+) 个计划任务中的通过数 / 可判定数；每题等权，无法判定结果不参与分母。$"), r"Passed / decidable results among \1 planned tasks; every task has equal weight, and undecidable results are excluded from the denominator.")) +REGEX.insert(0, (re.compile(r"^(\d+)/(\d+) · 复核记录$"), r"\1/\2 · review records")) +REGEX.insert(0, (re.compile(r"^(\d+) 条通过结果的参考词精确率低于 30%。此比例用于定位检查对象，不是统一失败线；通过不保证干净正文，正文参考的范围和时间也会影响该指标。$"), r"\1 passing results have reference word precision below 30%. This locates inspection candidates, not a universal failure threshold. Passing does not establish clean body text; reference scope and time also affect this metric.")) + +TEXT.update({ + "版本优先使用调用时记录的版本号；源码实现使用冻结的提交号。云端服务没有公开后端版本时如实标注。二进制哈希用于确认具体执行文件；事后核验的版本会说明依据。": "Versions use call-time records where available; source implementations use frozen commits. Undisclosed cloud backend versions are stated explicitly. Binary hashes identify the executable, and retrospectively verified versions state their evidence.", +}) + +def write_english_report(out: Path, *, source_routes=None) -> int: + """Mirror Chinese HTML routes; a scoped refresh preserves other editions.""" + if source_routes is None: + sources = sorted(path for path in out.rglob("*.html") if "en" not in path.relative_to(out).parts) + else: + from lexbench_webfetch.paths import contained_path + sources = [] + for name in sorted(set(source_routes)): + route = PurePosixPath(name) + if route.is_absolute() or '..' in route.parts or not route.parts or route.parts[0] == 'en' or route.suffix != '.html': + raise ValueError('Scoped English routes must be relative Chinese HTML paths') + path = contained_path(out, str(route), reject_symlinks=True) + contained_path(out, 'en/' + str(route), reject_symlinks=True) + if not path.is_file(): + raise FileNotFoundError('Missing Chinese route: ' + str(route)) + sources.append(out / str(route)) history_path = out / "report-changes.json" translations = release_translations(json.loads(history_path.read_text())) if history_path.is_file() else {} + from .report_reading import archived_translations + article_translations = archived_translations(out) english_root = out / "en" - if english_root.exists(): + if source_routes is None and english_root.exists(): shutil.rmtree(english_root) dataset_path = out / "dataset.json" allowed_source_text = {"中文"} @@ -3786,12 +4351,28 @@ def write_english_report(out: Path) -> int: allowed_source_text.update( case.get("name") for case in dataset.get("cases", []) if case.get("name") ) - sources = sorted(path for path in out.rglob("*.html") if "en" not in path.relative_to(out).parts) + archived_manifest = out / "archived-reading.json" + if archived_manifest.is_file(): + allowed_source_text.update(json.loads(archived_manifest.read_text()).get("preserved_source_names", [])) for source_path in sources: source = PurePosixPath(source_path.relative_to(out).as_posix()) target = out / "en" / Path(str(source)) target.parent.mkdir(parents=True, exist_ok=True) - localized = localize_html(source_path.read_text(encoding="utf-8"), source, translations=translations) + raw=source_path.read_text(encoding="utf-8") + if str(source)=='guide/index.html' and (out/'archived-reading.json').is_file(): + from .report_reading import render_archived_reading + from bs4 import BeautifulSoup, Comment + parsed = BeautifulSoup(raw, 'html.parser') + article = parsed.select_one('[data-archived-reading="true"]') + if article is None: + raise ValueError('Historical article route lacks its scoped wrapper') + article.replace_with(Comment('ARCHIVED-ARTICLE')) + shell = localize_html(str(parsed), source, translations=translations) + body = localize_html(render_archived_reading(out,'en'), source, + translations=article_translations) + localized = shell.replace('', body) + else: + localized=localize_html(raw,source,translations=translations) if source in {PurePosixPath("guide/index.html"), PurePosixPath("guide/article.html")}: untranslated = untranslated_english_narrative(localized, allowed_source_text) if untranslated: diff --git a/harness/web_fetch/report_i18n_adoption_d.py b/harness/web_fetch/report_i18n_adoption_d.py index e4fb34753..1bf8e4633 100644 --- a/harness/web_fetch/report_i18n_adoption_d.py +++ b/harness/web_fetch/report_i18n_adoption_d.py @@ -1,6 +1,31 @@ """English translations for the adoption report, shard D.""" TEXT = { + "2026-10-09 核验：Kitesurf 专项文档仍标明 Beta 期免费，受账户用量限制。通用 Browser Run 另有浏览器时长计价；在专项文档改口前，不把通用超额价格当成 Kitesurf Beta 的已证实单价。": "Checked on 2026-10-09: the Kitesurf-specific documentation still describes a free Beta subject to account usage limits. General Browser Run has separate browser-duration pricing; until the specific documentation changes, its overage prices are not treated as confirmed Kitesurf Beta rates.", + "所有网页、文档正文与网页内嵌媒体任务": "All webpage, document-body, and embedded-media tasks", + "本地浏览器；无托管调用单价": "Local browser; no managed per-call price", + "网页与文档任务由 --dump markdown 返回原生 Markdown；直接媒体文件返回原始字节；等待条件不参与内容抽取。": "Webpage and document tasks use --dump markdown to return native Markdown. Direct media files return original bytes; wait conditions do not extract content.", + "Advanced Extract 模式下，套餐表按 Basic 满额换算；相同用量的 Advanced Extract 积分翻倍、每千页价格翻倍，不能把搜索的一次调用价格套给 Extract。": "The plan table assumes full use of Basic Extract. At the same volume, Advanced Extract doubles credits and the price per thousand pages. Search request prices do not apply to Extract.", + "给定 URL 的 Contents 模式使用 text=true，不是 Search。每页每种内容类型单独计费，不能把一次批量请求当成一页；链接和图片地址不等于下载原文件。": "For a supplied URL, Contents uses text=true rather than Search. Each page and content type is billed separately; a batch request does not count as a single page. Links and image URLs do not deliver the original files.", + "所查普通 API Key 计费文档没有明确的所有失败均免费的承诺；应核验逐 URL 状态与 costDollars。x402 按请求参数计算付款，是不同支付方式，不能套用为 Contents 的结算规则。": "The reviewed billing documentation for ordinary API keys does not explicitly promise that every failed request is free; check each URL's status and costDollars. x402 calculates payment from request parameters and is a separate payment method, so its rules must not be applied to Contents billing.", + "公开定价按内容类型分别收费。泛称“包含内容”的搜索描述不意味着独立 Contents 免费；年付折扣、免费积分结转及extras 字段的独立加价未获公开确认。": "Public pricing charges separately for each content type. A general search description that includes content does not make standalone Contents free. Annual discounts, rollover of free credits, and separate extras surcharges have not been publicly confirmed.", + "官方现称 Browser Run。/markdown 属于 Quick Actions，只收浏览器时长费用，不收 Browser Sessions 的并发费用；不是 Workers AI toMarkdown 或 Markdown for Agents。": "The official name is now Browser Run. /markdown is a Quick Action billed for browser duration, without the Browser Sessions concurrency charge. It is separate from Workers AI toMarkdown and Markdown for Agents.", + "本地 CLI；无托管 API 单价": "Local CLI; no managed API unit price", + "本地开源库；非云套餐": "Local open-source library; no cloud plan", + "计算、浏览器、网络及代理成本自担；商业云服务价格不能当成本地调用的费用。": "The caller pays compute, browser, network, and proxy costs. Commercial cloud prices do not represent local-call costs.", + "内置或本地实现": "Built-in or local implementation", + "接口与实现 ↗": "Interface and implementation ↗", + "已知 URL → 原生 Markdown 或原文件字节": "Known URL → native Markdown or original file bytes", + "实际版本和接口见运行记录；下方链接指向官方说明。": "See the run records for the actual version and interface. The links below point to official documentation.", + "所有页面类任务都取得完整渲染 DOM，再使用 node-html-markdown 2.0.0 转为 Markdown；不运行资源提取脚本，不读取 PDF.js 等查看器私有对象，也不使用外部文档解析器。页面获取、DOM 转换与原始文件读取均计入耗时；原始字节、HTML 与转换器哈希均保留。": "Every webpage task obtains the complete rendered DOM and converts it to Markdown with node-html-markdown 2.0.0. No resource-extraction script, private viewer object such as PDF.js, or external document parser is used. Page retrieval, DOM conversion, and original-file reads all count toward latency. Original bytes, HTML, and converter hashes are retained.", + "网页正文、嵌入文档以及网页中的图片或视频信息都使用指定 Kitesurf 的 Browser Run 原生 Markdown。直接文档也使用官方 Markdown 接口，直接文件通过官方 CDP 获取原始响应。Workers AI 文档转换不计入 Kitesurf 成绩。客户端耗时包含公网往返、排队与执行。": "Webpage body text, embedded documents, and image or video information use native Browser Run Markdown from the specified Kitesurf. Direct documents also use the official Markdown interface; direct files use official CDP to obtain original responses. Workers AI document conversion is excluded from Kitesurf results. Client latency includes public-network round trips, queuing, and execution.", + "网页正文、嵌入文档以及网页中的图片或视频信息都使用浏览器原生 Markdown。直接文件使用浏览器已有的原始响应接口。不在失败后切换输出方式、调用共同 DOM 脚本，或请求外部服务补正文。接口未提供 HTTP 状态或最终网址时保留空值。": "Webpage body text, embedded documents, and image or video information use the browser's native Markdown. Direct files use its existing original-response interface. Failures do not trigger output switching, a shared DOM script, or an external service to supply missing text. HTTP status and final URL remain empty when the interface does not provide them.", + "网页、文档正文、结构与网页内嵌媒体任务": "Webpage, document-body, structured, and embedded-media tasks", + "直接取得 Markdown；直接 PDF/DOCX 不提供可读正文时失败": "Obtain Markdown directly; direct PDF/DOCX tasks fail without readable body text", + "旧版预声明完成条件": "Previously declared completion conditions", + "只等待，不抽取或拼接正文": "Wait only; do not extract or assemble body text", + "网页与文档任务为原生 Markdown；直接文件为原始字节": "Native Markdown for webpage and document tasks; original bytes for direct files", + "使用本轮绑定提交构建的二进制；提交、构建环境与 SHA-256 见环境与复现。": "Use the binary built from this run's bound commit. See Reproduction for the commit, build environment, and SHA-256.", "未查到公开年付或 Browser Run 时长量阶折扣。Workers 计算、存储等其他产品用量另算；本表不为 Quick Actions 叠加 Sessions 并发费。": "No public volume discounts were found for annual payment or Browser Run duration. Workers compute, storage, and other product usage are billed separately; this table does not add Sessions concurrency fees on top of Quick Actions.", "假设仅使用 /markdown，全部用量可计费；按响应头 X-Browser-Ms-Used 计时。每千页边际价＝每页秒数 × $0.025，不含最低月费、赠送时长和月度取整。": "Assumes exclusive use of /markdown and that all usage is billable, timed by the X-Browser-Ms-Used response header. Marginal price per thousand pages = seconds per page × $0.025, excluding the monthly minimum, included duration, and monthly rounding.", "5 秒": "5 seconds", diff --git a/harness/web_fetch/report_interfaces.py b/harness/web_fetch/report_interfaces.py index 450824982..4cbaee2ea 100644 --- a/harness/web_fetch/report_interfaces.py +++ b/harness/web_fetch/report_interfaces.py @@ -409,7 +409,9 @@ def fields(rows):return '
'+''.join('
if not bindings: raise ValueError("Measured native interface requires a saved call log: "+name) request=json.loads(verify_public_call_log(out,bindings[-1]).read_text())["request"] - s={**s,"params":[],"request":request,"notes":"此处展示该次公开调用记录保存的请求字段；完整配置和日志见环境与调用详情。产品内部超时以冻结配置和实际请求为准。"} + from .methods import method_page_model + model=method_page_model(name,name,[x for x in records if x["provider"]==name]) + s={**s,"entry":model["endpoint"],"params":[],"request":request,"notes":"此处展示该次公开调用记录保存的请求字段；完整配置和日志见环境与调用详情。产品内部超时以冻结配置和实际请求为准。"} access=method_access(name,records) if r: content=contained_path(out,r['artifact_path']).read_text();sample={k:r.get(k) for k in ['ok','status_code','title','final_url']};sample['content_excerpt']=content[:450];sample['content_characters']=len(content) diff --git a/harness/web_fetch/report_methods.py b/harness/web_fetch/report_methods.py index ba836318c..79c3e21c5 100644 --- a/harness/web_fetch/report_methods.py +++ b/harness/web_fetch/report_methods.py @@ -63,7 +63,8 @@ def render_methods(out,records,*,hero,section,table,cell,link,esc,authored=True) '

输入为任务定义与超时预算，输出为预设的内容或错误。完整输出见该方法的结果详情。

', 'interface-' + name) return body - if not authored: + native=(out/'dataset.json').is_file() and json.loads((out/'dataset.json').read_text()).get('comparison',{}).get('kind')=='uniform_native_local_measurement' + if not authored and not native: body=hero('本轮方法与接口','以下内容来自本轮结果记录与适配器契约。') anchors={'method-guide-pricing'} for name in sorted({row['provider'] for row in records}): @@ -98,6 +99,11 @@ def render_methods(out,records,*,hero,section,table,cell,link,esc,authored=True) kind,desc,url=PROFILES.get(name,PROFILES.get(engine,('测试适配器','本地可重复测试实现。','https://example.com'))) if engine and engine!=name:url=browser_method_registry()[name].get('source_url',url) r=next(r for r in records if r['provider']==name);info=method_info(r['provider_type']) + if native: + model=method_page_model(name,name,[row for row in records if row['provider']==name]) + desc=model['contract'] + if name=='moli-webfetch':url='https://github.com/lexmount/moli' + info={**info,'output':'产品原生返回；保留实际输出与日志','freshness':'缓存与等待选项见本轮固定配置'} intro.append(('',cell(link('../providers/'+name+'.html',display_name(name)))+cell(kind)+cell(desc)+cell(esc(info['output']))+cell(esc(info['freshness']))+cell(link(url,'文档 / 源码 ↗')))) for item in counts: name=item['provider'];n=item['repository_count'];known=n is not None @@ -122,7 +128,7 @@ def render_methods(out,records,*,hero,section,table,cell,link,esc,authored=True) body+=section('参评方法',cards([('远程接口','Kitesurf 使用指定 browser=kitesurf 的 Cloudflare Browser Run API；耗时包含公网往返、远端排队与执行。'),('本地原生接口','Lightpanda、Obscura 与 Moli 按任务使用各自官方接口。正文、文档和媒体任务的具体调用及输出格式见报告解读中的接口表。'),('Chromium 的正文转换','Chromium 通过原生接口读取渲染后的 DOM，再用 node-html-markdown 2.0.0 转成 Markdown。其他浏览器的正文不经过共同转换；文档任务不使用外部 PDF / DOCX 解析器。所有方法均不调用模型。')])) body+=section('Kitesurf 能否本地部署',cards([('当前结论','截至 2026-09-07，Cloudflare 尚未发布 Kitesurf 源码或自行部署包；本轮只能通过 Cloudflare 账号版 Browser Run 远程测试，不能把它当作本地开源浏览器。'),('后续意向','Cloudflare 官方博客表示计划在准备好后开源，并希望允许客户部署到自己的账户。这是未来计划，不是当前已有能力。')])+'

当前官方接入说明 · 官方开源计划说明

') else: - body+=section('参评方法',cards([('托管 API','Lexmount 两个区域、Firecrawl、Tavily、Exa、Cloudflare；通过各自 API 调用。'),('项目内置','CowAgent、OpenClaw、DeepSeek Harness；按本轮保留的实现与配置执行。'),('本地运行','LangChain WebBaseLoader、Crawl4AI、Moli WebFetch；使用本机网络与运行环境。')])) + body+=section('参评方法',cards([('托管 API','Firecrawl、Tavily、Exa、Cloudflare；通过各自 API 调用。'),('项目内置','CowAgent、OpenClaw、DeepSeek Harness；按本轮保留的实现与配置执行。'),('本地运行','LangChain WebBaseLoader、Crawl4AI、Moli WebFetch；使用本机网络与运行环境。')])) body+=section('能力、价格与接口',tabs('method-guide',[ ('capabilities','官方能力',render_capabilities(out,names,table=table,cell=cell,link=link,records=records)), ('pricing','价格与计费',render_pricing(out,names,table=table,cell=cell,link=link,esc=esc)), diff --git a/harness/web_fetch/report_quality.py b/harness/web_fetch/report_quality.py new file mode 100644 index 000000000..a28f0cf82 --- /dev/null +++ b/harness/web_fetch/report_quality.py @@ -0,0 +1,31 @@ +"""Present coverage and extra-content diagnostics as separate observations.""" +import re + +from lexbench_webfetch.metrics import task_passed + + +def quality_observations(record, case, output=''): + metrics = record['metrics'] + precision = metrics.get('word_precision') if case.get('ground_truth_main_content') else None + # A bare word in ordinary documentation is not a captured challenge widget. + component = bool(re.search(r'hcaptcha', output, re.I) + and re.search(r'Please try again', output, re.I) + and re.search(r'\bVerify\b', output) + and re.search(r'hcaptcha\.com/(?:privacy|what-is-hcaptcha)', output, re.I)) + return {'task_pass': task_passed(record['ok'], metrics, case.get('task_kind')), + 'word_precision': precision, 'low_precision': precision is not None and precision < .3, + 'hint_words': list(dict.fromkeys(metrics.get('diagnostic_hits', []) + metrics.get('blocked_page_hits', []))), + 'challenge_component_text': component} + + +def render_quality_observations(record, case, output, *, esc): + info = quality_observations(record, case, output) + verdict = '排除比较' if record['metrics'].get('evaluation_excluded') else '通过' if info['task_pass'] else '未通过' + precision = '未评测' if info['word_precision'] is None else f'{info["word_precision"]:.2%}' + component = '包含验证码组件文字' if info['challenge_component_text'] else '未检测到该组件文字特征' + hints = '、'.join(info['hint_words']) or '无命中' + hint_html=''+esc(hints)+'' if info['hint_words'] else '无命中' + return ('

目标要求：'+verdict+'

' + '

附加内容诊断：'+component+'；'+hint_html+'

' + '

参考词精确率：'+precision+'

' + '

通过只表示满足登记的目标要求，不保证干净正文。提示词命中不证明整页被拦截；词精确率也受参考范围和时间影响。低于 30% 仅用于定位检查对象，不是统一失败线。

') diff --git a/harness/web_fetch/report_reading.py b/harness/web_fetch/report_reading.py new file mode 100644 index 000000000..b8531a630 --- /dev/null +++ b/harness/web_fetch/report_reading.py @@ -0,0 +1,121 @@ +"""An explicitly supplied historical article, separate from current measurements.""" +import hashlib +import json +import re +import shutil +import base64 +from pathlib import Path +from urllib.parse import urlsplit + +from lexbench_webfetch.paths import contained_path + + +def sanitize_article_html(fragment): + """Keep static article content; never import executable HTML or global CSS.""" + from bs4 import BeautifulSoup + allowed = {'article', 'section', 'div', 'span', 'p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', + 'table', 'thead', 'tbody', 'tfoot', 'tr', 'th', 'td', 'ul', 'ol', 'li', 'a', 'img', + 'figure', 'figcaption', 'pre', 'code', 'blockquote', 'strong', 'em', 'b', 'i', + 'small', 'sup', 'sub', 'br', 'hr', 'details', 'summary'} + soup = BeautifulSoup(fragment, 'html.parser') + for element in list(soup.find_all(['script', 'style', 'iframe', 'form', 'input', 'button', + 'object', 'embed', 'link', 'meta', 'base', 'svg', 'math', 'template'])): + element.decompose() + for element in list(soup.find_all(True)): + if element.name not in allowed: + element.unwrap() + continue + attrs = {} + for key in ('title', 'alt', 'colspan', 'rowspan'): + if key in element.attrs: + attrs[key] = str(element.attrs[key]) + if element.get('id'): + attrs['id'] = 'archived-' + str(element['id']) + for key in ('href', 'src'): + if key not in element.attrs or (key == 'src' and element.name != 'img'): + continue + value = str(element[key]).strip() + compact = re.sub(r'[\x00-\x20\x7f]', '', value) + if key == 'src' and compact.startswith('data:'): + raster = re.fullmatch(r'data:image/(png|jpeg|webp);base64,([A-Za-z0-9+/]*={0,2})', compact) + if raster and len(raster[2]) <= 16_000_000: + try: + raw = base64.b64decode(raster[2], validate=True) + except ValueError: + raw = b'' + kind = raster[1] + if ((kind == 'png' and raw.startswith(b'\x89PNG\r\n\x1a\n')) + or (kind == 'jpeg' and raw.startswith(b'\xff\xd8\xff')) + or (kind == 'webp' and raw.startswith(b'RIFF') and raw[8:12] == b'WEBP')): + attrs[key] = value + continue + try: + parts = urlsplit(compact) + except ValueError: + continue + if '\\' in compact or parts.scheme.lower() not in {'', 'http', 'https', 'mailto'}: + continue + if key == 'src' and parts.scheme == 'mailto': + continue + attrs[key] = '#archived-' + value[1:] if value.startswith('#') else value + if element.name == 'a' and 'href' in attrs: + attrs['rel'] = 'noopener noreferrer' + element.attrs = attrs + for table in soup.find_all('table'): + table.wrap(soup.new_tag('div', attrs={'class': 'table-wrap'})) + return str(soup) + + +def copy_archived_reading(source: Path, out: Path) -> None: + manifest = source / "archived-reading.json" + if not manifest.is_file(): + return + data = json.loads(manifest.read_text()) + for item in [*data["languages"].values(), *([data["translations"]] if data.get("translations") else [])]: + origin = contained_path(source, item["path"]) + if origin.is_symlink() or hashlib.sha256(origin.read_bytes()).hexdigest() != item["sha256"]: + raise ValueError("Archived article content differs from its manifest") + target = contained_path(out, item["path"]) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(origin, target) + shutil.copy2(manifest, out / "archived-reading.json") + + +def render_archived_reading(out: Path, language: str = "zh") -> str | None: + manifest = out / "archived-reading.json" + if not manifest.is_file(): + return None + data = json.loads(manifest.read_text()) + item = data["languages"][language] + raw = contained_path(out, item["path"]).read_bytes() + if hashlib.sha256(raw).hexdigest() != item["sha256"]: + raise ValueError("Archived article hash mismatch") + text = raw.decode() + body = re.search(r"]*>(.*?)", text, re.S) + if not body: + raise ValueError("Archived article has no main content") + date = data.get("source_data_date") + if language == "zh": + date_note = f"（截至{date}）" if date else "" + note = f"本文沿用旧报告文章与原有数据{date_note}，不代表本轮评测结果。" + else: + date_note = f" through {date}" if date else "" + note = f"This article retains the older report and its data{date_note}; it does not describe the current benchmark results." + return ('

' + note + + '

' + sanitize_article_html(body[1]) + '
') + + +def archived_translations(out: Path) -> dict[str, str]: + manifest = out / "archived-reading.json" + if not manifest.is_file(): + return {} + item = json.loads(manifest.read_text()).get("translations") + if not item: + return {} + raw = contained_path(out, item["path"], reject_symlinks=True).read_bytes() + if hashlib.sha256(raw).hexdigest() != item["sha256"]: + raise ValueError("Archived translations hash mismatch") + text = json.loads(raw)["text"] + if not isinstance(text, dict) or any(not isinstance(k, str) or not isinstance(v, str) for k, v in text.items()): + raise ValueError("Report translations require text pairs") + return text diff --git a/harness/web_fetch/report_style.py b/harness/web_fetch/report_style.py index af91d7efe..0ba8ace8d 100644 --- a/harness/web_fetch/report_style.py +++ b/harness/web_fetch/report_style.py @@ -1,5 +1,6 @@ """Shared visual styling for the current report and standalone editorial export.""" STYLE = r""" +[data-archived-reading] img{max-width:100%;height:auto} .browser-interface-matrix table{min-width:1440px;table-layout:fixed}.browser-interface-matrix th:first-child{width:140px}.browser-interface-matrix td{overflow-wrap:anywhere}.browser-interface-matrix th{white-space:nowrap} .browser-interpretation .browser-interface-matrix table{min-width:1100px;font-size:13px}.browser-interpretation .browser-interface-matrix th:first-child{width:32px}.browser-interpretation .browser-interface-matrix th:nth-child(2){width:150px}.browser-interpretation pre{max-height:24rem;overflow:auto;white-space:pre-wrap;overflow-wrap:anywhere} :root{color-scheme:light;--bg:#f6f8fb;--panel:#fff;--panel2:#eef3f8;--ink:#17212b;--muted:#5d6b78;--line:#d9e2ea;--accent:#1268d8;--accent-soft:#e9f2ff;--violet:#6756d8;--lime:#3f7b26;--coral:#bd402f;--good:#117a47;--bad:#c22e43;--warn:#8a5a00;--callout:#fff6d8;--failure-bg:#fff0f2;--success-bg:#eaf8f0;--code-bg:#101820;--code-ink:#e9f1f5;--shadow:0 8px 28px rgba(30,55,80,.08);--max:1180px} diff --git a/harness/web_fetch/result_ledger.py b/harness/web_fetch/result_ledger.py new file mode 100644 index 000000000..3f884f1d4 --- /dev/null +++ b/harness/web_fetch/result_ledger.py @@ -0,0 +1,116 @@ +"""Conserve task judgments across explicit offline corrections and exclusions.""" +from collections import Counter + + +VERDICTS = {'pass', 'fail', 'excluded', 'unavailable'} + + +def summarize_state(state): + if not isinstance(state, dict) or not state or any(value not in VERDICTS for value in state.values()): + raise ValueError('Invalid result ledger state') + counts = Counter(state.values()) + return {key: counts[key] for key in ('pass', 'fail', 'excluded', 'unavailable')} + + +def build_ledger(states, *, predecessor=None): + """Derive every total and transition from per-attempt states, never prose.""" + if not states: + raise ValueError('Result ledger requires a baseline') + inventory = set(states[0]['verdicts']) + rounds = [] + previous = None + for state in states: + current_inventory = set(state['verdicts']) + replacement = state.get('replacement') + if not state.get('id') or not state.get('source_results_sha256'): + raise ValueError('Result ledger inventory or source binding differs') + if current_inventory != inventory: + removed, added = inventory - current_inventory, current_inventory - inventory + if (previous is None or not replacement + or set(replacement.get('removed_attempt_ids', [])) != removed + or set(replacement.get('added_attempt_ids', [])) != added + or not removed or len(removed) != len(added) + or {key.split('__')[0] for key in removed} != {key.split('__')[0] for key in added} + or any(previous[key] != state['verdicts'][key] + for key in inventory & current_inventory)): + raise ValueError('Result ledger replacement is not an atomic complete-method transition') + elif replacement: + raise ValueError('Result ledger replacement must change the declared inventory') + totals = summarize_state(state['verdicts']) + changes = [] if previous is None else [ + {'attempt_id': identifier, 'before': previous.get(identifier), 'after': state['verdicts'].get(identifier)} + for identifier in sorted(set(previous) | current_inventory) + if previous.get(identifier) != state['verdicts'].get(identifier)] + added = sum(row['after'] == 'pass' and row['before'] != 'pass' for row in changes) + removed = sum(row['before'] == 'pass' and row['after'] != 'pass' for row in changes) + before_passes = totals['pass'] if previous is None else summarize_state(previous)['pass'] + if before_passes + added - removed != totals['pass']: + raise ValueError('Result ledger pass totals do not conserve') + rounds.append({'id': state['id'], 'source_results_sha256': state['source_results_sha256'], + 'before_passes': before_passes, 'added_passes': added, + 'removed_passes': removed, 'totals': totals, 'changes': changes}) + previous = state['verdicts'] + inventory = current_inventory + result = {'schema_version': 2 if predecessor else 1, 'logical_results': len(inventory), + 'states': states, 'rounds': rounds, 'current': rounds[-1]['totals']} + if predecessor: + result['predecessor'] = predecessor + return result + + +def validate_ledger(ledger, records=None, *, source=None): + if ledger.get('schema_version') not in {1, 2}: + raise ValueError('Unsupported result ledger') + rebuilt = build_ledger(ledger['states'], predecessor=ledger.get('predecessor')) + if ledger != rebuilt: + raise ValueError('Result ledger counts or transitions differ from its states') + if (any(state.get('replacement') for state in ledger['states']) + and not ledger.get('predecessor')): + raise ValueError('Task replacement requires bound predecessor evidence') + if ledger.get('predecessor'): + if source is None: + raise ValueError('Result ledger needs its bound predecessor evidence') + import json + from .local_revision import read_bound, require_local_task_replacement + from harness.record_io import load_jsonl + prior = json.loads(read_bound(source, ledger['predecessor']).read_text()) + validate_ledger(prior, source=source) + count = len(prior['states']) + if ledger['states'][:count] != prior['states'] or len(ledger['states']) <= count: + raise ValueError('Result ledger does not retain the complete predecessor chain') + for previous, state in zip(ledger['states'], ledger['states'][1:]): + event = state.get('replacement') + if not event: + continue + receipt = json.loads(read_bound(source, event['receipt']).read_text()) + if receipt['baseline_results']['sha256'] != previous['source_results_sha256']: + raise ValueError('Task replacement does not continue the predecessor results') + dataset = json.loads(read_bound(source, event['output_dataset']).read_text()) + output = load_jsonl(read_bound(source, event['output_results'])) + require_local_task_replacement(source, event['receipt'], dataset, output) + proof = json.loads(read_bound(source, event['output_verdicts']).read_text()) + if (proof['input_results_sha256'] != state['source_results_sha256'] + or event['output_results']['sha256'] != state['source_results_sha256'] + or set(proof['verdicts']) != set(state['verdicts']) + or any(type(value) is not bool for value in proof['verdicts'].values()) + or {key: 'pass' if value else 'fail' for key, value in proof['verdicts'].items()} + != state['verdicts']): + raise ValueError('Task replacement verdicts differ from frozen output evidence') + read_bound(source, proof['frozen_predicate']) + read_bound(source, proof['snapshot_manifest']) + before = load_jsonl(read_bound(source, receipt['baseline_results'])) + calls = load_jsonl(read_bound(source, receipt['cohort_results'])) + removed = {row['attempt_id'] for row in before if row['case_id'] == receipt['removed_case_id']} + added = {row['attempt_id'] for row in calls} + if (set(event['removed_attempt_ids']) != removed + or set(event['added_attempt_ids']) != added): + raise ValueError('Task replacement inventory differs from its physical calls') + if records is not None: + from lexbench_webfetch.metrics import task_measured, task_passed + actual = {record['attempt_id']: ( + 'excluded' if record['metrics'].get('evaluation_excluded') else + 'unavailable' if not task_measured(record['metrics']) else + 'pass' if task_passed(record['ok'], record['metrics']) else 'fail') for record in records} + if len(actual) != len(records) or actual != ledger['states'][-1]['verdicts']: + raise ValueError('Result ledger does not match current scored results') + return rebuilt diff --git a/harness/web_fetch/retrieval_hints.py b/harness/web_fetch/retrieval_hints.py index 79d505abc..a6d02f121 100644 --- a/harness/web_fetch/retrieval_hints.py +++ b/harness/web_fetch/retrieval_hints.py @@ -546,7 +546,7 @@ def native_contract_for_case(case: Any) -> NativeRetrievalContract | None: def native_default_case(case: Any) -> bool: """Stable acceptance contracts permit a URL-only provider-default route.""" get = case.get if isinstance(case, dict) else lambda key, default=None: getattr(case, key, default) - if get("track", "web") != "dynamic" or get("retrieval_hints"): + if get("track", "web") != "dynamic" or get("retrieval_hints") not in (None, {}, {"version": 3, "mode": "native"}): return False contract = get("acceptance_contract") if not contract: diff --git a/harness/web_fetch/runner.py b/harness/web_fetch/runner.py index 9744af599..673f7b274 100644 --- a/harness/web_fetch/runner.py +++ b/harness/web_fetch/runner.py @@ -384,9 +384,35 @@ def persist_public_call_log( def persist_result_artifacts( - output_dir: Path, artifact_rel: Path, result: ProviderResult + output_dir: Path, artifact_rel: Path, result: ProviderResult, *, request_url: str | None = None ) -> Path: """Persist one physical call without changing its scored answer bytes.""" + discarded_late_answer = (not result.ok and result.error_type == 'timeout' + and result.content == '') + from lexbench_webfetch.answer_projection import require_native_answer_projection, require_native_projection_schema + if result.metadata.get('native_answer_projection'): + require_native_projection_schema(result.metadata['native_answer_projection']) + raw = result.metadata.get('_raw_response_b64') + if result.metadata.get('native_answer_projection', {}).get('source_format') == 'bytes': + raw = result.metadata.get('_raw_stdout_b64') + needs_projection = (result.metadata.get('native_answer_required') + or result.metadata.get('native_answer_projection') + or any(result.metadata.get(key) for key in ('links', 'images', 'videos', 'media', 'extras')) + or (raw is not None and (result.files or result.title is not None or result.final_url is not None))) + if not discarded_late_answer and needs_projection: + if result.metadata.get('native_answer_required') and not result.metadata.get('native_answer_projection'): + raise ValueError('Scored native answer has no projection declaration') + require_native_answer_projection(result.content, result.metadata, + base64.b64decode(raw, validate=True) if isinstance(raw, str) else None, + files=result.files if raw is not None or result.metadata.get('native_answer_projection') else (), + title=result.title, final_url=result.final_url, request_url=request_url, + diagnostic=base64.b64decode(result.metadata['_diagnostic_log_b64'], validate=True) + if isinstance(result.metadata.get('_diagnostic_log_b64'), str) else None, + transport_receipt=result.metadata.get('source_transport_receipt')) + elif discarded_late_answer and (result.files or any(result.metadata.get(key) for key in ('images', 'links', 'videos', 'media', 'extras'))): + raise ValueError('Discarded late native answer retains score-bearing fields') + if result.metadata.get('native_answer_projection'): + result.metadata['native_answer_required'] = True artifact = contained_path(output_dir, artifact_rel) artifact.parent.mkdir(parents=True, exist_ok=True) artifact.write_text(result.content, encoding='utf-8', errors='replace') @@ -468,6 +494,8 @@ def run_evaluation( provider_concurrency: int = 1, max_failure_retries: int = DEFAULT_FAILURE_RETRIES, resource_sample_interval_ms: int | None = None, + reference_policy: str = "fresh", + case_admission: Any | None = None, call_order: str = "provider", ) -> tuple[list[dict[str, Any]], dict[str, Any]]: output_dir = output_dir.resolve() @@ -495,8 +523,10 @@ def run_evaluation( if provider.config.get("uniform_native_parameters"): if runs != 1: raise ValueError("fixed native calls require one run") + from .native_routes import declared_input_media_kind for case in cases: provider.check_native_call(case) + declared_input_media_kind(case) safe_identifier(provider.name) for case in cases: validate_case(case) @@ -504,10 +534,23 @@ def run_evaluation( raise ValueError('Provider names must be unique within an execution') if len({case.id for case in cases}) != len(cases): raise ValueError('Task IDs must be unique within an execution') + from .native_routes import NativeRouteLedger + native_routes = (NativeRouteLedger(cases, providers, timeout_seconds=timeout) + if any(p.config.get('uniform_native_parameters') for p in providers) else None) for filename in ['artifacts', PUBLIC_CALL_LOG_ROOT, 'manifest.json', 'dataset.json', 'results.jsonl', 'summary.json']: contained_path(output_dir, filename) - reference_checks = preflight_reference_checks(cases, dataset_path) + if reference_policy not in {"fresh", "frozen_existing"}: + raise ValueError("unknown reference policy") + if reference_policy == "frozen_existing": + # Explicit local measurement of existing rules; no live-source claim. + from .reference_lifecycle import validate_resource_source_evidence + for case in cases: + validate_resource_source_evidence(case, dataset_path) + frozen_dataset_sha256 = hashlib.sha256(read_dataset_bytes(dataset_path)).hexdigest() + reference_checks = None + else: + reference_checks = preflight_reference_checks(cases, dataset_path) for provider in providers: missing = provider.validate() if missing: @@ -535,6 +578,7 @@ def run_evaluation( "network_preflight": ({"dns_mode": network_audit["dns_mode"], "sha256": hashlib.sha256((output_dir / 'network-preflight.json').read_bytes()).hexdigest()} if network_audit is not None else None), + "reference_policy": reference_policy, "unexecuted": [], "deadline_policy": "end_to_end_answer; non_cancellable_adapter_quarantined", "public_call_log_schema_version": 1, @@ -551,6 +595,7 @@ def run_evaluation( "name": provider.name, "type": provider.type, "min_interval_seconds": provider.min_interval_seconds, + "fixed_native_parameters": bool(provider.config.get("uniform_native_parameters")), **({"implementation_identity": provider._implementation_identity} if isinstance(getattr(provider, '_implementation_identity', None), dict) else {}), } @@ -589,6 +634,11 @@ def run_evaluation( manifest['evidence_state'] = 'diagnostic' manifest['unqualified_reference_cases'] = sorted( set(comparison.get('unqualified_reference_cases', [])) & {case.id for case in cases}) + if native_routes is not None: + manifest['native_routes_required'] = True + manifest['native_route_contract'] = native_routes.write(output_dir, + dataset_sha256=manifest['dataset_sha256'], + runtime_source_sha256=manifest['runtime_source_sha256']) (output_dir / "manifest.json").write_text( json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" ) @@ -596,6 +646,11 @@ def run_evaluation( json.dumps(dataset_meta, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" ) + if case_admission is not None: + manifest["scheduling"] = case_admission.describe() + (output_dir / "manifest.json").write_text( + json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + results_path = output_dir / "results.jsonl" result_lock = threading.Lock() with results_path.open("w", encoding="utf-8") as results_file: @@ -617,194 +672,213 @@ def run_provider(provider: Provider, selected_cases=None, selected_runs=None) -> binding = reference_checks["cases"].get(case.id) if reference_checks else None # Outside provider exception handling: expired references # stop measurement, never create a supplier failure record. - check_case_reference_fresh(case, dataset_path, binding) - started = utc_now() - physical_attempts = [] - stopped_provider = False - for physical_attempt in range(1, max_failure_retries + 2): - retry_throttle_wait_ms = 0.0 - if physical_attempt > 1 and provider.min_interval_seconds: - remaining = provider.min_interval_seconds - (time.monotonic() - previous_request_completed) - if remaining > 0: - retry_throttle_wait_ms = round(remaining * 1000, 2) - time.sleep(remaining) - physical_started = utc_now() - physical_begin = time.perf_counter() - pending = None - caught_exception = None - diagnostics = None - if provider.config.get("log_level") == "debug": - diagnostics = CallDiagnostics(output_dir / "private_artifacts" / "live-logs" / provider.name / case.id / f"run-{run_number:03d}-physical-{physical_attempt:03d}.jsonl", attempt_id) - diagnostics.emit("caller", "call_started", provider=provider.name, case_id=case.id, url=case.url, timeout_seconds=timeout) - provider._call_diagnostics = diagnostics - try: - result, pending = fetch_with_deadline(provider, case, timeout) - except Exception as error: # each physical failure remains inspectable - caught_exception = error - error_type, error_message = classify_exception(error) - public_metadata = getattr(error, "public_metadata", {}) - public_metadata = public_metadata if isinstance(public_metadata, dict) else {} - allowed_diagnostic = { - key: public_metadata[key] - for key in ("_answer_elapsed_ms", "exit_code", "stdout", - "stdout_bytes", "stdout_truncated", - "stderr", "stderr_bytes", "stderr_truncated") - if key in public_metadata - } - result = ProviderResult( - ok=False, - # Adapter exceptions are diagnostics, never a - # supplier answer for the public scorer. - content="", - error_type=error_type, - error_message=_public_text(error_message), - metadata=allowed_diagnostic, + if reference_policy == "frozen_existing": + if hashlib.sha256(read_dataset_bytes(dataset_path)).hexdigest() != frozen_dataset_sha256: + raise ReferenceLifecycleError("Frozen rules changed during the run", "reference_changed") + else: + check_case_reference_fresh(case, dataset_path, binding) + # Admission waits before the answer timer and 45-second budget. + admission_token = (case_admission.acquire(provider, case) + if case_admission is not None else None) + pending = None + record = None + try: + started = utc_now() + physical_attempts = [] + stopped_provider = False + for physical_attempt in range(1, max_failure_retries + 2): + retry_throttle_wait_ms = 0.0 + if physical_attempt > 1 and provider.min_interval_seconds: + remaining = provider.min_interval_seconds - (time.monotonic() - previous_request_completed) + if remaining > 0: + retry_throttle_wait_ms = round(remaining * 1000, 2) + time.sleep(remaining) + physical_started = utc_now() + physical_begin = time.perf_counter() + pending = None + caught_exception = None + diagnostics = None + if provider.config.get("log_level") == "debug": + diagnostics = CallDiagnostics(output_dir / "private_artifacts" / "live-logs" / provider.name / case.id / f"run-{run_number:03d}-physical-{physical_attempt:03d}.jsonl", attempt_id) + diagnostics.emit("caller", "call_started", provider=provider.name, case_id=case.id, url=case.url, timeout_seconds=timeout) + provider._call_diagnostics = diagnostics + if native_routes is not None and provider.name in native_routes.providers: + native_routes.begin_attempt(provider, case, physical_attempt) + try: + result, pending = fetch_with_deadline(provider, case, timeout) + except Exception as error: # each physical failure remains inspectable + caught_exception = error + error_type, error_message = classify_exception(error) + public_metadata = getattr(error, "public_metadata", {}) + public_metadata = public_metadata if isinstance(public_metadata, dict) else {} + allowed_diagnostic = { + key: public_metadata[key] + for key in ("_answer_elapsed_ms", "exit_code", "stdout", + "stdout_bytes", "stdout_truncated", + "stderr", "stderr_bytes", "stderr_truncated") + if key in public_metadata + } + result = ProviderResult( + ok=False, + # Adapter exceptions are diagnostics, never a + # supplier answer for the public scorer. + content="", + error_type=error_type, + error_message=_public_text(error_message), + metadata=allowed_diagnostic, + ) + wall_elapsed_ms = round((time.perf_counter() - physical_begin) * 1000, 2) + if diagnostics is not None: + diagnostics.finish(result) + # Cancellable adapters may need a bounded post-deadline + # unwind. That cleanup determines quarantine safety but + # is not part of the answer latency. + physical_elapsed_ms = result.metadata.pop( + '_answer_elapsed_ms', wall_elapsed_ms ) - wall_elapsed_ms = round((time.perf_counter() - physical_begin) * 1000, 2) - if diagnostics is not None: - diagnostics.finish(result) - # Cancellable adapters may need a bounded post-deadline - # unwind. That cleanup determines quarantine safety but - # is not part of the answer latency. - physical_elapsed_ms = result.metadata.pop( - '_answer_elapsed_ms', wall_elapsed_ms - ) - previous_request_completed = time.monotonic() - completed_by_provider[provider.name] = previous_request_completed - if case.task_kind == 'file': - result.metadata.setdefault('file_evaluation', {'files': [], 'pass': False, - 'scope': 'returned_bytes_signature_length_sha256'}) - suffix = (f'run-{run_number:03d}.txt' if max_failure_retries == 0 else - f'run-{run_number:03d}-physical-{physical_attempt:03d}.txt') - artifact_rel = Path('artifacts') / provider.name / case.id / suffix - artifact = persist_result_artifacts(output_dir, artifact_rel, result) - artifact_bytes = result.content.encode('utf-8', errors='replace') - measured = score_attempt( - case, artifact_bytes, ok=result.ok, metadata=result.metadata, - title=result.title, final_url=result.final_url, - list_reference=load_list_reference(case, dataset_path.parent), - status_code=result.status_code, - prepared_answer=result.prepared_answer, - ) - measured['deadline_overrun'] = physical_elapsed_ms > timeout * 1000 - measured['deadline_budget_ms'] = timeout * 1000 - if binding is not None: - measured['reference_sha256'] = binding['reference_sha256'] - physical_passed = task_passed(result.ok, measured, case.task_kind) - physical_completed = utc_now() - call_log = persist_public_call_log( - output_dir, provider=provider, case=case, attempt_id=attempt_id, - run_number=run_number, physical_attempt=physical_attempt, - started_at=physical_started, completed_at=physical_completed, - elapsed_ms=physical_elapsed_ms, - retry_throttle_wait_ms=retry_throttle_wait_ms, timeout=timeout, - artifact_rel=artifact_rel, - content_sha256=measured['content_sha256'], result=result, - exception=caught_exception, - ) - artifact_paths = [artifact_rel.as_posix()] - for key in ('rendered_html_path', 'raw_stdout_path'): - if result.metadata.get(key): - artifact_paths.append(result.metadata[key]) - artifact_paths.extend( - item['artifact_path'] - for item in result.metadata.get('file_evaluation', {}).get('files', []) - if item.get('artifact_path') - ) - artifact_hashes = { - relative: hashlib.sha256(contained_path( - output_dir, relative, reject_symlinks=True).read_bytes()).hexdigest() - for relative in artifact_paths - } - if artifact_hashes[artifact_rel.as_posix()] != measured['content_sha256']: + previous_request_completed = time.monotonic() + completed_by_provider[provider.name] = previous_request_completed + if case.task_kind == 'file': + result.metadata.setdefault('file_evaluation', {'files': [], 'pass': False, + 'scope': 'returned_bytes_signature_length_sha256'}) + suffix = (f'run-{run_number:03d}.txt' if max_failure_retries == 0 else + f'run-{run_number:03d}-physical-{physical_attempt:03d}.txt') + artifact_rel = Path('artifacts') / provider.name / case.id / suffix + if native_routes is not None and provider.name in native_routes.providers: + result.metadata['native_route_receipt'] = native_routes.receipt(provider, case, ok=result.ok) + if provider.config.get('uniform_native_parameters') and (result.content or result.files or result.title is not None): + result.metadata['native_answer_required'] = True + artifact = persist_result_artifacts(output_dir, artifact_rel, result, request_url=case.url) + artifact_bytes = result.content.encode('utf-8', errors='replace') + measured = score_attempt( + case, artifact_bytes, ok=result.ok, metadata=result.metadata, + title=result.title, final_url=result.final_url, + list_reference=load_list_reference(case, dataset_path.parent), + status_code=result.status_code, + prepared_answer=result.prepared_answer, + ) + measured['deadline_overrun'] = physical_elapsed_ms > timeout * 1000 + measured['deadline_budget_ms'] = timeout * 1000 + if binding is not None: + measured['reference_sha256'] = binding['reference_sha256'] + physical_passed = task_passed(result.ok, measured, case.task_kind) + physical_completed = utc_now() + call_log = persist_public_call_log( + output_dir, provider=provider, case=case, attempt_id=attempt_id, + run_number=run_number, physical_attempt=physical_attempt, + started_at=physical_started, completed_at=physical_completed, + elapsed_ms=physical_elapsed_ms, + retry_throttle_wait_ms=retry_throttle_wait_ms, timeout=timeout, + artifact_rel=artifact_rel, + content_sha256=measured['content_sha256'], result=result, + exception=caught_exception, + ) + artifact_paths = [artifact_rel.as_posix()] + for key in ('rendered_html_path', 'raw_stdout_path'): + if result.metadata.get(key): + artifact_paths.append(result.metadata[key]) + artifact_paths.extend( + item['artifact_path'] + for item in result.metadata.get('file_evaluation', {}).get('files', []) + if item.get('artifact_path') + ) + artifact_hashes = { + relative: hashlib.sha256(contained_path( + output_dir, relative, reject_symlinks=True).read_bytes()).hexdigest() + for relative in artifact_paths + } + if artifact_hashes[artifact_rel.as_posix()] != measured['content_sha256']: + raise ValueError('Persisted output differs from scored bytes') + private_artifacts = [] + private_binding = raw_response_binding(result.metadata) + if private_binding: + private_artifacts.append(private_binding) + diagnostic_binding = diagnostic_log_binding(result.metadata) + if diagnostic_binding: + private_artifacts.append(diagnostic_binding) + physical_attempts.append({ + 'physical_attempt': physical_attempt, + **({'native_route_receipt': result.metadata['native_route_receipt']} + if 'native_route_receipt' in result.metadata else {}), + 'started_at': physical_started, + 'completed_at': physical_completed, + 'elapsed_ms': physical_elapsed_ms, + 'retry_throttle_wait_ms': retry_throttle_wait_ms, + 'ok': result.ok, + 'status_code': result.status_code, + 'error_type': result.error_type, + 'error_message': result.error_message, + 'task_pass': physical_passed, + 'content_sha256': measured['content_sha256'], + 'artifact_path': artifact_rel.as_posix(), + 'artifact_paths': artifact_paths, + 'artifact_sha256': artifact_hashes, + 'call_log': call_log, + **({'private_artifacts': private_artifacts} if private_artifacts else {}), + }) + stopped_provider = (result.error_type == 'timeout' and pending is not None + and pending.is_alive()) + if physical_passed or stopped_provider or physical_attempt > max_failure_retries: + break + elapsed_ms = physical_attempts[-1]['elapsed_ms'] + result.metadata['call_log'] = physical_attempts[-1]['call_log'] + if len(physical_attempts) > 1: + result.metadata['retry'] = { + 'physical_attempts': len(physical_attempts), + 'retries_used': len(physical_attempts) - 1, + 'execution_cost_ms': round(sum(item['elapsed_ms'] + item['retry_throttle_wait_ms'] for item in physical_attempts), 2), + 'trigger': 'previous physical attempt failed the task predicate', + 'selection': 'final physical attempt is authoritative; no best-of selection', + 'attempts': physical_attempts, + } + finished = utc_now() + if hashlib.sha256(artifact.read_bytes()).hexdigest() != measured['content_sha256']: raise ValueError('Persisted output differs from scored bytes') - private_artifacts = [] - private_binding = raw_response_binding(result.metadata) - if private_binding: - private_artifacts.append(private_binding) - diagnostic_binding = diagnostic_log_binding(result.metadata) - if diagnostic_binding: - private_artifacts.append(diagnostic_binding) - physical_attempts.append({ - 'physical_attempt': physical_attempt, - 'started_at': physical_started, - 'completed_at': physical_completed, - 'elapsed_ms': physical_elapsed_ms, - 'retry_throttle_wait_ms': retry_throttle_wait_ms, - 'ok': result.ok, - 'status_code': result.status_code, - 'error_type': result.error_type, - 'error_message': result.error_message, - 'task_pass': physical_passed, - 'content_sha256': measured['content_sha256'], - 'artifact_path': artifact_rel.as_posix(), - 'artifact_paths': artifact_paths, - 'artifact_sha256': artifact_hashes, - 'call_log': call_log, - **({'private_artifacts': private_artifacts} if private_artifacts else {}), - }) - stopped_provider = (result.error_type == 'timeout' and pending is not None - and pending.is_alive()) - if physical_passed or stopped_provider or physical_attempt > max_failure_retries: - break - elapsed_ms = physical_attempts[-1]['elapsed_ms'] - result.metadata['call_log'] = physical_attempts[-1]['call_log'] - if len(physical_attempts) > 1: - result.metadata['retry'] = { - 'physical_attempts': len(physical_attempts), - 'retries_used': len(physical_attempts) - 1, - 'trigger': 'previous physical attempt failed the task predicate', - 'execution_cost_ms': round(sum( - item['elapsed_ms'] + item['retry_throttle_wait_ms'] - for item in physical_attempts), 2), - 'selection': 'final physical attempt is authoritative; no best-of selection', - 'attempts': physical_attempts, - } - finished = utc_now() - if hashlib.sha256(artifact.read_bytes()).hexdigest() != measured['content_sha256']: - raise ValueError('Persisted output differs from scored bytes') - record = AttemptRecord( - attempt_id=attempt_id, - provider=provider.name, - provider_type=provider.type, - case_id=case.id, - case_name=case.name, - category=case.category, - url=case.url, - run=run_number, - started_at=started, - elapsed_ms=elapsed_ms, - ok=result.ok, - status_code=result.status_code, - final_url=result.final_url, - title=result.title, - error_type=result.error_type, - error_message=result.error_message, - artifact_path=artifact_rel.as_posix(), - metrics=measured, - provider_metadata={ - **result.metadata, - "configured_min_interval_seconds": provider.min_interval_seconds, - "client_throttle_wait_ms": throttle_wait_ms, - }, - ).to_dict() - apply_request_status(record) - record["completed_at"] = finished - from .models import execution_id - record['run_id'] = execution_id(record) - passed = task_passed(record['ok'],record['metrics'],case.task_kind) - state = "PASS" if passed else "FAIL" - with result_lock: - records.append(record) - results_file.write(json.dumps(record, ensure_ascii=True) + "\n") - results_file.flush() - print( - f"{state:4} {provider.name:14} {case.id:24} " - f"{elapsed_ms:9.2f} ms {result.error_type or ''}", - flush=True, - ) - if delay: - time.sleep(delay) + record = AttemptRecord( + attempt_id=attempt_id, + provider=provider.name, + provider_type=provider.type, + case_id=case.id, + case_name=case.name, + category=case.category, + url=case.url, + run=run_number, + started_at=started, + elapsed_ms=elapsed_ms, + ok=result.ok, + status_code=result.status_code, + final_url=result.final_url, + title=result.title, + error_type=result.error_type, + error_message=result.error_message, + artifact_path=artifact_rel.as_posix(), + metrics=measured, + provider_metadata={ + **result.metadata, + "configured_min_interval_seconds": provider.min_interval_seconds, + "client_throttle_wait_ms": throttle_wait_ms, + }, + ).to_dict() + apply_request_status(record) + record["completed_at"] = finished + from .models import execution_id + record['run_id'] = execution_id(record) + passed = task_passed(record['ok'],record['metrics'],case.task_kind) + state = "PASS" if passed else "FAIL" + with result_lock: + records.append(record) + results_file.write(json.dumps(record, ensure_ascii=True) + "\n") + results_file.flush() + print( + f"{state:4} {provider.name:14} {case.id:24} " + f"{elapsed_ms:9.2f} ms {result.error_type or ''}", + flush=True, + ) + if delay: + time.sleep(delay) + finally: + if case_admission is not None: + case_admission.release(admission_token, pending=pending, record=record) if stopped_provider: remaining_attempts = ((runs - run_number) * len(cases) + len(cases) - case_index) @@ -862,12 +936,17 @@ def run_provider(provider: Provider, selected_cases=None, selected_runs=None) -> manifest["completed_at"] = utc_now() manifest["attempt_count"] = len(records) manifest["attempt_ids"] = [r['attempt_id'] for r in records] + if native_routes is not None: + manifest['native_route_errors'] = list(native_routes.errors) (output_dir / "manifest.json").write_text( json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" ) (output_dir / "summary.json").write_text( json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8" ) + if native_routes is not None: + from .native_routes import verify_native_route_contract + verify_native_route_contract(output_dir, manifest, dataset_meta, records) expected_attempts = len(cases) * len(providers) * runs if len(records) != expected_attempts or manifest["unexecuted"]: raise IncompleteEvaluationError( diff --git a/harness/web_fetch/safety.py b/harness/web_fetch/safety.py index f00860ffd..618b2f203 100644 --- a/harness/web_fetch/safety.py +++ b/harness/web_fetch/safety.py @@ -108,3 +108,26 @@ def commit_staged_evidence(source: Path, staged: Path, backup: Path) -> None: f"Rollback failed; original data retained at {backup}" ) from recovery_error raise + + +def copy_evidence_file(source, destination): + """Copy independent evidence bytes, using APFS copy-on-write when available. + + Never hard-link frozen inputs: writing a generated copy must not alter its + source. Unsupported filesystems/platforms retain ordinary copy semantics. + """ + import shutil + source, destination = unlinked_path(Path(source)), unlinked_path(Path(destination)) + if sys.platform == 'darwin': + import ctypes + try: + clone = ctypes.CDLL(None, use_errno=True).clonefile + except (AttributeError, OSError): + clone = None + if clone is not None: + clone.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_int] + clone.restype = ctypes.c_int + if clone(os.fsencode(source), os.fsencode(destination), 0) == 0: + shutil.copystat(source, destination) + return str(destination) + return shutil.copy2(source, destination) diff --git a/harness/web_fetch/scoring_revision.py b/harness/web_fetch/scoring_revision.py index e48a80e9f..76c9e5f27 100644 --- a/harness/web_fetch/scoring_revision.py +++ b/harness/web_fetch/scoring_revision.py @@ -28,6 +28,7 @@ from lexbench_webfetch.scoring_regressions import ROOT_CAUSE_CATEGORIES, build_correction_provenance ALLOWED_FIELDS = { + "page_identity_required", "body_requirements", "required_patterns", "required_facts", diff --git a/harness/web_fetch/workers/cowagent_worker.py b/harness/web_fetch/workers/cowagent_worker.py index 803ab7a46..132212746 100644 --- a/harness/web_fetch/workers/cowagent_worker.py +++ b/harness/web_fetch/workers/cowagent_worker.py @@ -17,6 +17,12 @@ def main(): sys.path.insert(0, str(root)) sys.path.insert(0, str(Path(__file__).resolve().parents[3])) from harness.web_fetch.network_dns import public_dns_context, validate_dns_mode + from harness.web_fetch.diagnostics import diagnostic_context, redact + + class NativeLogSink: + def emit(self, source, event, **fields): + print(json.dumps(redact({"source": source, "event": event, **fields}), + ensure_ascii=False, default=str), file=sys.stderr, flush=True) module = None for line in sys.stdin: if not line.strip(): @@ -34,7 +40,10 @@ def main(): module, 'DEFAULT_TIMEOUT', None) tool = module.WebFetch({'cwd': str(workspace)}) dns_mode = validate_dns_mode(request.get('dns_mode', 'system')) - with public_dns_context(dns_mode, request['timeout_seconds']) as dns_trace: + with diagnostic_context( + NativeLogSink() if request.get('log_level') == 'debug' else None, + names=('log', 'urllib3', 'requests', 'langchain_community'), + ), public_dns_context(dns_mode, request['timeout_seconds']) as dns_trace: result = tool.execute({'url': request['url']}) metadata.update(dns_mode=dns_mode, dns_resolutions=dns_trace, effective_timeout_seconds=metadata['cowagent_default_timeout_seconds'], diff --git a/harness/web_fetch/workers/crawl4ai_worker.py b/harness/web_fetch/workers/crawl4ai_worker.py index 1184089cc..043b46e3e 100755 --- a/harness/web_fetch/workers/crawl4ai_worker.py +++ b/harness/web_fetch/workers/crawl4ai_worker.py @@ -145,6 +145,9 @@ def is_direct_pdf_document(request): return False if request.get("task_kind", "body") != "body": return False + if request.get("uniform_native_parameters"): + from harness.web_fetch.native_routes import declared_input_media_kind + return declared_input_media_kind(request) == "direct_pdf" path = urlsplit(request["url"]).path.lower().rstrip("/") return path.endswith(".pdf") or "/pdf/" in path @@ -166,7 +169,7 @@ async def fetch_file(request, directory): ) as crawler: result = await crawler.crawl( request["url"], - config=CrawlerRunConfig(page_timeout=int(request["timeout_seconds"] * 1000)), + config=CrawlerRunConfig(**({} if request.get("uniform_native_parameters") else {"page_timeout": int(request["timeout_seconds"] * 1000)})), ) return { "ok": 200 <= result.status_code < 300, @@ -192,9 +195,9 @@ async def _run_pdf_document(request): url=request["url"], config=CrawlerRunConfig( cache_mode=CacheMode.BYPASS, - page_timeout=int(timeout_seconds * 1000), + **({} if request.get("uniform_native_parameters") else {"page_timeout": int(timeout_seconds * 1000)}), scraping_strategy=scraping_strategy, - verbose=False, + verbose=request.get("log_level") == "debug", ), ) markdown = result.markdown.raw_markdown if result.markdown else "" @@ -408,7 +411,7 @@ async def main() -> None: "ok": bool(result.success), # Preserve HTML, canonical Markdown, headers and native errors # privately; answer selection remains raw_markdown. - "raw_tool_response": result.model_dump(mode="json"), + "raw_tool_response": result.model_dump(mode="json", fallback=str), "content": markdown, "files": files, "status_code": result.redirected_status_code if result.redirected_status_code is not None else result.status_code, diff --git a/pyproject.toml b/pyproject.toml index 1f6be06a6..8798fdc35 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ readme = "README.md" license = {file = "LICENSE"} requires-python = ">=3.11" dependencies = [ - "lexbench-webfetch @ git+https://github.com/lexmount/lexbench-webfetch.git@6d8d72637bb96f4938b9a2f7c46d0c7dffd74945", + "lexbench-webfetch @ git+https://github.com/lexmount/lexbench-webfetch.git@c75efb365c7133d9f82f70fb1044f2ba88a36445", "beautifulsoup4>=4.12,<5", ] diff --git a/research/native-430-local-report.json b/research/native-430-local-report.json new file mode 100644 index 000000000..7e0698721 --- /dev/null +++ b/research/native-430-local-report.json @@ -0,0 +1,327 @@ +{ + "schema_version": 1, + "comparison_kind": "uniform_native_local_measurement", + "case_count": 430, + "result_count": 4300, + "providers": [ + "cloudflare-markdown", + "cowagent-built-in", + "crawl4ai", + "deepseek-harness-built-in", + "exa", + "firecrawl", + "langchain-webbase", + "moli-webfetch", + "openclaw-built-in", + "tavily" + ], + "frozen_dataset_sha256": "640e6dda990e79d8db4f8d9fb9551a9d5317b2f2669f74f121ddf8089c795f84", + "results_sha256": "6f43a3419940bfdb87b09c0652801817722ff38afcf48036da96ffb030903cb9", + "measured_scoring_source_sha256": "4ce40cdf2ecb1af4088915a2b869ffbdcb65dce5222d1429578e97d175f13bfc", + "evidence_state": "local_measurement", + "certified": false, + "evidence_storage": "Local score revision of the retained PR51 public snapshot; original answers were hash-verified privately for scoring and existing public views are unchanged. The predecessor packages remain immutable historical inputs.", + "evidence_bindings": [ + { + "path": "local-run-contract.json", + "sha256": "2de0ae1532a51428b2eb19dac44b5e1ba94e67f1433a0420cbeb038dbd602691" + }, + { + "path": "result-review.json", + "sha256": "4d44137bcf71d546309ca77fdf9457846edfa6762530a31324eb1a2f46b3f9ec" + }, + { + "path": "archived-reading.json", + "sha256": "85da5e52c295eb7fffbcd3e10e41d33294eeadf233ec7a6958de68ed0c9e30fb" + }, + { + "path": "execution.json", + "sha256": "bc6c113a85d5e6bb345e08f39924032ff55b47220e16b338d20e52592317c0d8" + }, + { + "path": "local-evidence/result-ledger.json", + "sha256": "3293b04e245c8a92ba4e35c33f0f4d0254d608539a4d7ff3be0c26f6470a900f" + }, + { + "path": "frozen-method-options.json", + "sha256": "bbaae307a3cadeb75fee4a7f8f81fde41dca718728dad75b8cdf8c911866a2b3" + }, + { + "path": "assets/methodology/annotation.png", + "sha256": "ca37357825ee094e1aee19b7987773f7d1586452215d0be716a41ef55619178d" + }, + { + "path": "assets/methodology/validation.png", + "sha256": "45141f56838bd4416c5a5e374f2cc3f1ca2190b5d4fe3454c9e7b88857ece207" + }, + { + "path": "assets/methodology/scoring.png", + "sha256": "fdd3ae94d91185f1b5e17ce9fca4fd2ae49ac67c1862eb0a8760c6739bb7f030" + }, + { + "path": "assets/methodology/annotation.en.png", + "sha256": "0762cbd955fbd1566cd147599f968df455d07d6236314cf48f1db7fdd3ead456" + }, + { + "path": "assets/methodology/validation.en.png", + "sha256": "7c75067d6a3a163763643ecae319333fc999b0ab9fa455b165b451a4365e865c" + }, + { + "path": "assets/methodology/scoring.en.png", + "sha256": "84660859bbbf002cf4ed1cf285c27e435460beadce34de759ed8800f34249bde" + }, + { + "path": "archived-reading/article.zh.txt", + "sha256": "88ca41eb3a55010532697676ee46279372fbb6389bcde1afab70b275d8d02177" + }, + { + "path": "archived-reading/article.en.txt", + "sha256": "545cae010f4cab40778a2946681dd78864bd30281a63b40c191f6b939db1b254" + }, + { + "path": "archived-reading/translations.json", + "sha256": "fa2fff490354674e2edf19bcee87306b364053df88c88be1ab0fab9d1eb424e5" + }, + { + "path": "local-evidence/code-block-impact.json", + "sha256": "c49297a036fa007bf29117ac5220061b7e7fcd393ad4d2fed7671c2341b9a66d" + }, + { + "path": "report-assets/report-changes.json", + "sha256": "d6e786f05db2843d7d078fc82c5559dd558e5acd5bf8a13f144f8cbfbca82180" + } + ], + "evidence_path": "runs/pr51-main-sync-20261016/evidence", + "task_replacement": { + "path": "local-evidence/task-replacement/receipt.json", + "sha256": "55a21acee65634351cc394c020c9943d320cef2f99c12c2d902aa5e724c6ffd4" + }, + "offline_revision": { + "path": "local-evidence/offline-revisions/f3052148f4f4930a-abead7654b4e78f3/receipt.json", + "sha256": "2eaec8db641953267422e8158f776977d0996be4963a0c886099e306e6b704c6" + }, + "replacement_case": { + "id": "static-sqlite-quickstart", + "name": "SQLite official quick start", + "url": "https://www.sqlite.org/quickstart.html", + "track": "web", + "category": "technical_documentation", + "task_kind": "body", + "task_prompt": "Read the supplied page after initial loading. Preserve the document text, headings, lists and code examples; no login, clicks, pagination or follow-up interaction is required.", + "min_content_chars": 500, + "required_facts": [ + { + "id": "database-command", + "alternatives": [ + "sqlite3 test.db" + ] + }, + { + "id": "open-database", + "alternatives": [ + "sqlite3_open" + ] + }, + { + "id": "execute-sql", + "alternatives": [ + "sqlite3_exec" + ] + }, + { + "id": "close-database", + "alternatives": [ + "sqlite3_close" + ] + } + ], + "body_coverage": { + "version": 3, + "minimum_coverage": 0.8, + "minimum_floor": 0.6, + "minimum_region_passes": 4, + "regions": 5 + }, + "diagnostic_any": [ + "Page Not Found", + "Access denied", + "CAPTCHA" + ], + "code_preservation": { + "match_scope": "code_compact", + "required_patterns": 27, + "authority": "Frozen pre-call official source; preserve code tokens while allowing whitespace and HTML entities." + } + }, + "current_scoring_source_sha256": "a96b88127c304ba035cbe0e52dd0940b181334075feb47cc621bdbc443089295", + "retained_review": { + "path": "local-evidence/main-sync/review-retention.json", + "sha256": "2db81ea76dff03b797bf74255d85ff5f4f60a3e876fc51fde413a6505a869e39" + }, + "review_scope": { + "fresh": 0, + "retained": 2980, + "unreviewed": 1320, + "total": 4300, + "fresh_independent_blind_review": false, + "new_score_revision_independently_reviewed": false + }, + "review_evidence_package": { + "archive_name": "PR-51-public-saved-inputs-20261016.tar.xz", + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-public-saved-inputs-20261016.tar.xz", + "sha256": "6ceb6a8bb35bc0e1d4ba40ff183dbb66e975d37426d7277ec6fca27ef58ad1ea", + "bytes": 295677260, + "self_contained_restore_verified": true, + "full_report_rebuild_verified": true, + "logical_report_routes": 25760, + "anonymous_download_verified": true, + "access": "Public read-only saved inputs; no repository write access or earlier draft assets required.", + "presentation_binding": { + "path": "presentation-binding.json", + "sha256": "89b0622a86b2713bc670903600931b2cc5e3851b757afa4012c10b3f0cc763a4" + }, + "projection_receipt": { + "path": "PUBLICATION-PROJECTION.json", + "sha256": "b836c8da535c5eb106b223099b15bed11023098545ee38209e39b3b96c223b66" + }, + "public_evidence_views": { + "path": "PUBLIC-EVIDENCE-VIEWS.json", + "sha256": "3b4ad900100bfc94cc004ee2f67c0fae54a2199ea78265c60c623d91db6fd79b" + }, + "public_dataset_sha256": "19a102898c9ab3988d6c794d05b9def1db430d29bdf2d137acd811b5051fe1b3", + "public_results_sha256": "08a9020c8ac3523fa6d1c5872e4c59133877ed03b412ae7f3b53339b7ee2fc66", + "verification": { + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-verification-20261016.json", + "sha256": "2fe714e6e8b410890f8be0329da3cb01bb02796c051b5477de0c2f5e3f4a6b40" + }, + "report_renderer_commit": "ffe4a03367dc2b2d2fd168600042c7a3ead10d73", + "validation_commit": "4ef472ff151274e9c526be58bf0e056a1591f420", + "role": "immutable_baseline_for_native_context_correction" + }, + "historical_native_route_audit": { + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-route-audit-20261016.json", + "sha256": "54a8b3a1370192a31d6d606aca21ef84e8f4e57d4ce6052c9d299e2ab5c0b69e", + "cells": 140, + "observed_signature_drift": 0, + "timeouts_without_route_confirmation": 19, + "pre_call_frozen": false + }, + "native_answer_field_audit": { + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-native-answer-field-audit-20261016.json", + "sha256": "4bd896daebca2b479c8e1b84fc41a58465385f063962edba55b3dc0629c156f8", + "bytes": 1468237, + "results_sha256": "ece94c0911a769c84ce9d896b00d97706fd4435d6d194f03cff77391710934a4", + "provider_calls_added": 0 + }, + "historical_timeout_stage_audit": { + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-timeout-stage-audit-20261016.json", + "sha256": "369441bf256f753791ead3f08ead503cf91bc20577ab8cb4e00e88c7bfa202eb", + "bytes": 27272, + "results_sha256": "ece94c0911a769c84ce9d896b00d97706fd4435d6d194f03cff77391710934a4", + "provider_calls_added": 0 + }, + "publication_evidence_views": { + "path": "PUBLIC-EVIDENCE-VIEWS.json", + "sha256": "c1719bd6694c7fd5c6c93471e82916da4b449ca15ef7f5e0533402ff108af519" + }, + "publication_projection": { + "path": "local-evidence/main-sync/diagnostic-context-publication-projection.json", + "sha256": "0f6d2285b55f558c5d8b515f14e0fe2f7a51359e88212ede7132b291ae9d90b1", + "kind": "host_metadata_publication_projection" + }, + "native_answer_context_audit": { + "path": "local-evidence/native-context-audit/PR-51-native-context-audit-20261016.json", + "sha256": "1c676e00c5b2a0f18928f6cc8de7e1674659fc92e70ae9978f4a9a80c3789eab", + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-native-context-audit-20261016.json", + "bytes": 8046875, + "result_count": 4300, + "original_context_differences": 2, + "current_context_differences": 0, + "provider_calls_added": 0, + "anonymous_download_sha256_verified": true + }, + "original_measured_dataset_sha256": "28e3b3e9ec89ce545aae44355bd28eb0dc383ad6f64aed4dc91f327b14a528ab", + "original_measured_results_sha256": "ece94c0911a769c84ce9d896b00d97706fd4435d6d194f03cff77391710934a4", + "original_implementation_commit": "4ef472ff151274e9c526be58bf0e056a1591f420", + "native_context_correction": { + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-native-context-correction-20261016-v2.tar.xz", + "sha256": "f33f5e2f2a8d643ed80bab5ecec57bc7f353924fe4a7ea958b8f0caab6ee3c39", + "bytes": 43947732, + "implementation_commit": "31a00bad2bbbdd33d50f47ef17fd8beb6dd4900d", + "baseline_archive_sha256": "6ceb6a8bb35bc0e1d4ba40ff183dbb66e975d37426d7277ec6fca27ef58ad1ea", + "input_public_results_sha256": "08a9020c8ac3523fa6d1c5872e4c59133877ed03b412ae7f3b53339b7ee2fc66", + "output_public_results_sha256": "b2e3546914de91912ff648184f0df3db6b6d1bb32784d3d3cc4c8b74af206bdb", + "presentation_binding": { + "path": "presentation-binding.json", + "sha256": "b2dc50c1ae3f75638dc10faea09a39ad5fd451611cf8cc04385abcbbd6d879f2" + }, + "manifest_sha256": "fa003f1c01eff9bed30de49fc7bb3b8acac51ef2999c1424d912fd647a950675", + "restore_verified": true, + "restored_input_files": 33352, + "native_revision_chain_verified": true, + "review_bindings_verified": true, + "result_ledger_verified": true, + "anonymous_download_sha256_verified": true, + "provider_calls_added": 0, + "resource_route_results_rescored": 16, + "retained_identity_reviews_carried": 4, + "offline_rescored_results": 20, + "original_context_differences": 2, + "current_context_differences": 0, + "task_verdict_changes": 0, + "passes": 1966, + "failures": 2334, + "report_generated_languages": [ + "zh", + "en" + ], + "page_layout_checks": 136, + "page_layout_issues": 0, + "historical_limits_unchanged": true, + "presentation_revision": "Aligned English numeric copy in the new review round; original opinions and scoring inputs unchanged." + }, + "predecessor_presentation": { + "path": "local-evidence/main-sync/predecessor-presentation-binding.json", + "sha256": "8a134b98183f3ab1b0a2c60e079c84866b5876a8e9a8e0ef3e6f44cfead3f046" + }, + "historical_review": { + "path": "local-evidence/main-sync/predecessor-result-review.json", + "sha256": "12c5874acf32fbec1bd680835779525894447a440ad41aa72b8e8eba02bfd7d9" + }, + "implementation_parents": { + "pr51": "fa7d93a2997f022e89eee0aba5902094563e2363", + "main": "637bd5171b296af6431b7a570f3d8a48ec5c671f" + }, + "score_revision_configurations": [ + { + "path": "local-evidence/main-sync/revision-config.json", + "sha256": "bff6292c9a38ebf4688c61ffb3d599e00b97326ada659554221a05a7191fe8d3" + }, + { + "path": "local-evidence/main-sync/table-scope-revision-config.json", + "sha256": "c49810417427e72a4e2c741bf231418a8801cb43348f44ed6683c4bc564f4f67" + } + ], + "report_changes": { + "path": "report-assets/report-changes.json", + "sha256": "d6e786f05db2843d7d078fc82c5559dd558e5acd5bf8a13f144f8cbfbca82180" + }, + "main_sync_correction": { + "url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-main-sync-correction-20261016.tar.gz", + "sha256": "6656c4b6057a2e7bd966ba4d664de2326229af7a918e0d630256ddd7bd63bd2e", + "bytes": 1174128171, + "manifest_url": "https://github.com/lexmount/browser-eval/releases/download/pr-51-public-evidence-20261016/PR-51-main-sync-manifest-20261016.json", + "manifest_sha256": "83c2f9e6cea1a792fe620c0a65ce9a7e714ee0b3f0594015410ff1d7d2dee20a", + "presentation_binding": { + "path": "presentation-binding.json", + "sha256": "f150a4c339482ff8d0e00394b4df5082a9e1811a8e0f0101242bfeea4a626acc" + }, + "base_dataset_sha256": "e2a5c68fa7c2640414c56c0949a8f344f8091e602e6479f8fd2bb07fec5b9681", + "base_results_sha256": "314a5adb2225fde74cfe0672d25d972c2c99e65b40e337d93c0a1d7418e4fe00", + "output_dataset_sha256": "640e6dda990e79d8db4f8d9fb9551a9d5317b2f2669f74f121ddf8089c795f84", + "output_results_sha256": "6f43a3419940bfdb87b09c0652801817722ff38afcf48036da96ffb030903cb9", + "restored_input_files": 33373, + "restore_verified": true, + "provider_calls_added": 0 + } +} diff --git a/scripts/build_report.py b/scripts/build_report.py index 3d7c36497..98499fec6 100644 --- a/scripts/build_report.py +++ b/scripts/build_report.py @@ -24,6 +24,8 @@ help='generate both editions; --language selects the homepage') parser.add_argument('--refresh-presentation', type=Path, help='refresh an existing local report using a hash-bound current-local-results.json; preserve measured evidence') + parser.add_argument('--presentation-binding', type=Path, + help='rebuild a hash-bound historical local report without rescoring') parser.add_argument('--local-audit',action='store_true', help='build an explicitly local, non-publishable audit report from bound evidence') parser.add_argument('--diagnostic-cohorts',action='store_true', @@ -33,6 +35,8 @@ parser.add_argument('--wcxb',action='store_true',help='Build the independently bound WCXB report') parser.add_argument('--webmainbench-full',action='store_true',help='Build the separately certified complete 545-case report') args=parser.parse_args() + if args.presentation_binding and not args.local_audit: + parser.error('--presentation-binding requires --local-audit') from harness.reporting.templates import resolve_template resolve_template('webfetch') if args.refresh_presentation: @@ -68,7 +72,9 @@ def requested_language(output, supported=SUPPORTED_REPORT_LANGUAGES): evidence=require_evidence(args.evidence) print(build_report(evidence,args.output, include_english=args.include_english, language=requested_language(args.output), - local_audit=True)) + local_audit=True, + show_local_status_banners=True, + presentation_binding=args.presentation_binding)) raise SystemExit(0) if sum(bool(value) for value in (args.profile, args.wcxb, args.webmainbench_full)) > 1: parser.error('--profile, --wcxb and --webmainbench-full are mutually exclusive') diff --git a/tests/test_adjudication_evidence.py b/tests/test_adjudication_evidence.py new file mode 100644 index 000000000..c928ae491 --- /dev/null +++ b/tests/test_adjudication_evidence.py @@ -0,0 +1,179 @@ +import hashlib + +import pytest + +from harness.web_fetch.adjudication_evidence import ( + adjudication_reason, case_hash, current_rule_outcomes, require_adjudication_evidence, +) +from lexbench_webfetch.scoring import score_attempt + + +def fixture(tmp_path): + text = 'Required complete article.' + (tmp_path / 'answer.txt').write_text(text) + case = {'id': 'a', 'url': 'https://example.test/', 'required_facts': [ + {'id': 'article', 'alternatives': [text]}], 'diagnostic_any': ['CAPTCHA']} + metrics = score_attempt(case, text.encode(), ok=True) + record = {'attempt_id': 'p__a', 'case_id': 'a', 'ok': True, + 'artifact_path': 'answer.txt', 'metrics': metrics} + assertions = current_rule_outcomes(case, record) + reason = adjudication_reason(assertions) + row = {'attempt_id': 'p__a', 'agent_verdict': 'fail', 'final_verdict': 'pass', + 'adjudication': reason, 'adjudication_evidence': { + 'reason': reason, 'case_sha256': case_hash(case), + 'content_sha256': metrics['content_sha256'], + 'rule_ids': list(assertions), 'rule_assertions': assertions, + 'samples': [{'start': 0, 'end': len(text), 'text': text}]}} + return case, record, row + + +@pytest.mark.parametrize('defect', [None, 'unknown_rule', 'false_hit', 'false_result', + 'false_reason', 'empty_hit_claim', 'excerpt', 'verdict', 'historical_migration']) +def test_adjudication_rejects_semantic_falsehoods_not_only_missing_text(tmp_path, defect): + case, record, row = fixture(tmp_path) + proof = row['adjudication_evidence'] + review = {'attempts': [row]} + if defect == 'unknown_rule': + proof['rule_ids'].append('nonexistent-rule') + elif defect == 'false_hit': + proof['rule_assertions']['diagnostic_any']['hits'] = ['CAPTCHA'] + elif defect == 'false_result': + proof['rule_assertions']['required_facts']['passed'] = False + elif defect == 'false_reason': + proof['reason'] = row['adjudication'] = 'CAPTCHA 已命中；正文通过。' + elif defect == 'empty_hit_claim': + proof['reason'] = row['adjudication'] = '命中词：。因此原意见不成立。' + elif defect == 'excerpt': + proof['samples'][0]['text'] = 'Invented article' + elif defect == 'verdict': + row['final_verdict'] = 'fail' + row['agent_verdict'] = 'pass' + elif defect == 'historical_migration': + review['rule_binding_revision'] = {'new_independent_review': False} + if defect: + with pytest.raises(ValueError): + require_adjudication_evidence(tmp_path, {'cases': [case]}, [record], review) + else: + assert require_adjudication_evidence(tmp_path, {'cases': [case]}, [record], review) == 1 + assert '诊断词命中：无' in row['adjudication'] + + +@pytest.mark.parametrize('defect', [None, 'answer', 'excerpt', 'opinion', 'case', 'binding']) +def test_supplemental_review_binds_new_opinion_task_and_complete_answer(tmp_path, defect): + import copy + import json + from harness.web_fetch.adjudication_evidence import _require_supplemental_reviews + text = 'Complete document and sqlite3 test.db' + (tmp_path/'answer.txt').write_text(text) + case = {'id': 'a', 'required_facts': ['sqlite3 test.db']} + record = {'attempt_id': 'p__a', 'case_id': 'a', 'artifact_path': 'answer.txt', + 'metrics': {'content_sha256': hashlib.sha256(text.encode()).hexdigest()}} + row = {'attempt_id': 'p__a', 'agent_verdict': 'pass', 'agent_evidence': 'Required command is present.', + 'content_sha256': record['metrics']['content_sha256'], + 'review_evidence': {'case_sha256': case_hash(case), + 'samples': [{'start': 0, 'end': len(text), 'text': text}]}} + supplement = {'kind': 'supplemental_saved_output_review', 'results_sha256': 'a'*64, + 'provider_calls_added': 0, 'reviewed_attempts': 1, 'reviewer': {'agent': 'independent reviewer'}, + 'review_protocol': {'blind_initial_verdict': False}, 'attempts': [copy.deepcopy(row)]} + if defect == 'excerpt': + row['review_evidence']['samples'][0]['text'] = 'Invented answer' + supplement['attempts'] = [copy.deepcopy(row)] + path = tmp_path/'supplement.json' + path.write_text(json.dumps(supplement)) + binding = {'path': path.name, 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + review = {'attempts': [row], 'results_sha256': 'a'*64, 'supplemental_reviews': [binding]} + if defect == 'answer': (tmp_path/'answer.txt').write_text('Changed answer') + elif defect == 'opinion': row['agent_verdict'] = 'fail' + elif defect == 'case': case['required_facts'] = ['different command'] + elif defect == 'binding': binding['sha256'] = 'b'*64 + if defect: + with pytest.raises(ValueError): + _require_supplemental_reviews(tmp_path, {'cases': [case]}, [record], review) + else: + assert _require_supplemental_reviews(tmp_path, {'cases': [case]}, [record], review) == {'p__a'} + + +@pytest.mark.parametrize('defect', [None, 'positive_rule', 'answer', 'opinion', 'verdict', + 'policy_hit', 'dependent_reason', 'prior_binding', 'count', 'missing_receipt']) +def test_retention_requires_unchanged_decision_dependencies(tmp_path, defect): + import copy + import json + from harness.web_fetch.adjudication_evidence import require_retained_reviews + from harness.web_fetch.result_review import review_input_identity + text = 'Required complete article.' + if defect == 'policy_hit': text += ' CAPTCHA' + old_case = {'id': 'a', 'url': 'https://example.test/', + 'required_facts': [{'id': 'article', 'alternatives': ['Required complete article.']}], + 'forbidden_any': ['CAPTCHA']} + case = {k: v for k, v in old_case.items() if k != 'forbidden_any'} + case['diagnostic_any'] = ['CAPTCHA'] + (tmp_path/'answer.txt').write_text(text) + metrics = score_attempt(case, text.encode(), ok=True) + record = {'attempt_id': 'p__a', 'provider': 'p', 'case_id': 'a', 'ok': True, + 'artifact_path': 'answer.txt', 'metrics': metrics} + original = {'attempt_id': 'p__a', 'content_sha256': metrics['content_sha256'], + 'agent_verdict': 'pass', 'agent_evidence': 'Required article is complete.', + 'final_verdict': 'pass'} + if defect == 'dependent_reason': + original['adjudication_evidence'] = {'rule_ids': ['forbidden_any']} + def bind(name, value, *, jsonl=False): + path = tmp_path/name + path.write_text(json.dumps(value) + ('\n' if jsonl else '')) + return {'path': name, 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + old_record = copy.deepcopy(record) + assert review_input_identity(record) == review_input_identity(old_record) + result_binding = bind('old-results.jsonl', old_record, jsonl=True) + prior = {'results_sha256': result_binding['sha256'], 'attempts': [original]} + if defect == 'prior_binding': prior['results_sha256'] = 'x'*64 + receipt = {'kind': 'unchanged_review_dependency_retention', 'provider_calls_added': 0, + 'attempt_ids': ['p__a'], 'prior_results': result_binding, + 'prior_dataset': bind('old-dataset.json', {'cases': [old_case]}), + 'prior_review': bind('old-review.json', prior)} + row = {**copy.deepcopy(original), 'review_origin': 'retained'} + review = {'attempts': [row], 'retained_reviewed_attempts': 1, + 'fresh_reviewed_attempts': 0, 'retained_reviews': bind('retention.json', receipt)} + if defect == 'positive_rule': case['required_facts'] = ['A different article'] + elif defect == 'answer': (tmp_path/'answer.txt').write_text('Changed answer') + elif defect == 'opinion': row['agent_evidence'] = 'Rewritten opinion' + elif defect == 'verdict': row['final_verdict'] = 'fail' + elif defect == 'count': review['fresh_reviewed_attempts'] = 1 + elif defect == 'missing_receipt': review.pop('retained_reviews') + if defect: + with pytest.raises(ValueError): + require_retained_reviews(tmp_path, {'cases': [case]}, [record], review) + else: + assert require_retained_reviews(tmp_path, {'cases': [case]}, [record], review) == {'p__a'} + + +@pytest.mark.parametrize('defect', [None, 'opinion', 'reason', 'anomaly', 'unblinded', 'missing']) +def test_fresh_review_keeps_the_immutable_blind_initial_opinion(tmp_path, defect): + import copy + import json + from harness.web_fetch.adjudication_evidence import _require_supplemental_reviews + case, record, row = fixture(tmp_path) + text = (tmp_path/'answer.txt').read_text() + row.update(content_sha256=record['metrics']['content_sha256'], + agent_evidence='The required article is absent.', reviewer_anomalies=[], + review_evidence={'case_sha256': case_hash(case), + 'samples': [{'start': 0, 'end': len(text), 'text': text}]}) + initial = {'machine_judgments_withheld': defect != 'unblinded', 'attempts': [{ + 'attempt_id': 'p__a', 'verdict': 'fail', 'reason': row['agent_evidence'], 'anomalies': []}]} + if defect == 'missing': initial['attempts'] = [] + if defect == 'opinion': row['agent_verdict'] = 'pass' + elif defect == 'reason': row['agent_evidence'] = 'Rewritten initial opinion' + elif defect == 'anomaly': row['reviewer_anomalies'] = ['Invented anomaly'] + def bind(name, document): + path = tmp_path/name + path.write_text(json.dumps(document)) + return {'path': name, 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + supplement = {'kind': 'supplemental_saved_output_review', 'provider_calls_added': 0, + 'results_sha256': 'a'*64, 'reviewed_attempts': 1, 'reviewer': {'model': 'reviewer'}, + 'review_protocol': {'blind_initial_verdict': True}, + 'blind_initial_outputs': [bind('initial.json', initial)], 'attempts': [copy.deepcopy(row)]} + review = {'results_sha256': 'a'*64, 'attempts': [row], + 'supplemental_reviews': [bind('supplement.json', supplement)]} + if defect: + with pytest.raises(ValueError): + _require_supplemental_reviews(tmp_path, {'cases': [case]}, [record], review) + else: + assert _require_supplemental_reviews(tmp_path, {'cases': [case]}, [record], review) == {'p__a'} diff --git a/tests/test_anomaly_review.py b/tests/test_anomaly_review.py index 6edb97832..5dc6c7a27 100644 --- a/tests/test_anomaly_review.py +++ b/tests/test_anomaly_review.py @@ -44,7 +44,7 @@ def test_real_stable_structure_forbidden_shape_triggers(self): 'provider':'moli-webfetch','case_id':'dynamic-react-cart','ok':True, 'artifact_path':'answer.txt','metrics':{ 'stable_structure_pass':True, 'stable_content_pass':False, - 'stable_task_pass':False, 'forbidden_hits':['Add to cart'], + 'stable_task_pass':False, 'forbidden_requirement_hits':['Add to cart'], 'stable_task_errors':['forbidden_content']}} self.assertIn('positive-structure-only-forbidden-failed', detect_anomalies(root,[row])[row['attempt_id']]) @@ -96,7 +96,7 @@ def add(case, text='', *, ok=True, error=None, status=None, route='ordinary_mark metrics['stable_complete_record_count']=complete if forbidden: metrics.update(stable_structure_pass=True, - forbidden_hits=forbidden, + forbidden_requirement_hits=forbidden, stable_task_errors=['forbidden_content']) rows.append({'attempt_id':f'moli-webfetch__{case}__r001', 'provider':'moli-webfetch','case_id':case,'ok':ok, diff --git a/tests/test_browser_reference_scope.py b/tests/test_browser_reference_scope.py index 3824a2b02..c7d9f7dbc 100644 --- a/tests/test_browser_reference_scope.py +++ b/tests/test_browser_reference_scope.py @@ -347,7 +347,8 @@ def test_resource_observation_matches_every_declared_target_and_cdn_variants(): 'id': 'resource', 'url': 'https://example.test/page', 'track': 'image', 'task_kind': 'resource', 'expected_resources': [ - {'id': 'one', 'urls': ['https://images.unsplash.com/photo-one']}, + {'id': 'one', 'urls': ['https://images.unsplash.com/photo-one'], + 'aliases': ['https://images.unsplash.com/photo-one?w=1200']}, {'id': 'two', 'urls': [], 'url_patterns': [r'^https://cdn\.example\.test/two(?:\?|$)']}, ], 'resource_output_contract': {'unexpected': 'allow'}, diff --git a/tests/test_case_admission.py b/tests/test_case_admission.py new file mode 100644 index 000000000..1aae8d527 --- /dev/null +++ b/tests/test_case_admission.py @@ -0,0 +1,95 @@ +import json, threading, time +from pathlib import Path +from types import SimpleNamespace +from harness.web_fetch.admission import CaseAdmission +from lexbench_webfetch.models import EvalCase +from harness.web_fetch.models import ProviderResult +from harness.web_fetch.providers.base import Provider +from harness.web_fetch.runner import run_evaluation + +def method(name):return SimpleNamespace(name=name) +def case(host):return SimpleNamespace(id=host,url='https://'+host+'/') + +def test_same_site_waits_and_distinct_cloud_local_can_overlap(tmp_path): + local=CaseAdmission(tmp_path,group='local',site_cooldown=0) + cloud=CaseAdmission(tmp_path,group='cloud',site_cooldown=0) + a=local.acquire(method('a'),case('www.example.com'));seen=[] + t=threading.Thread(target=lambda:seen.append(cloud.acquire(method('b'),case('api.example.com'))));t.start();time.sleep(.15);assert not seen + local.release(a);t.join(1);assert seen;cloud.release(seen[0]) + a=local.acquire(method('a'),case('one.test'));b=cloud.acquire(method('b'),case('two.test'));state=json.loads((tmp_path/'state.json').read_text());assert len(state['active'])==2;local.release(a);cloud.release(b) + +def test_second_local_waits_and_parallel_timeout_reduces_to_serial(tmp_path): + local=CaseAdmission(tmp_path,group='local',site_cooldown=0);cloud=CaseAdmission(tmp_path,group='cloud',site_cooldown=0) + a=local.acquire(method('a'),case('one.test'));seen=[];t=threading.Thread(target=lambda:seen.append(local.acquire(method('b'),case('two.test'))));t.start();time.sleep(.15);assert not seen + c=cloud.acquire(method('c'),case('three.test'));cloud.release(c,record={'error_type':'timeout','attempt_id':'c'});assert not json.loads((tmp_path/'state.json').read_text())['parallel_enabled'];local.release(a);t.join(1);assert seen;local.release(seen[0]) + +def test_pending_call_keeps_site_and_resource_reserved(tmp_path): + local = CaseAdmission(tmp_path, group='local', site_cooldown=0) + cloud = CaseAdmission(tmp_path, group='cloud', site_cooldown=0) + token = local.acquire(method('a'), case('one.test')) + finish = threading.Event() + pending = threading.Thread(target=finish.wait, daemon=True) + pending.start() + local.release(token, pending=pending) + state = json.loads((tmp_path / 'state.json').read_text()) + assert state['active'][token['key']]['quarantined'] + admitted = [] + waiter = threading.Thread(target=lambda: admitted.append( + cloud.acquire(method('b'), case('one.test'))), daemon=True) + waiter.start() + try: + time.sleep(.15) + assert not admitted + finally: + finish.set() + pending.join(1) + waiter.join(2) + assert admitted, 'Finished background calls must release their reservation' + cloud.release(admitted[0]) + assert not json.loads((tmp_path / 'state.json').read_text())['active'] + +def test_admission_wait_excluded_from_answer_timeout_and_latency(tmp_path): + class Fixture(Provider): + def fetch(self,case,timeout):return ProviderResult(True,'Expected') + class Gate: + def describe(self):return {'fixture':True} + def acquire(self,provider,case):time.sleep(.3);return 'token' + def release(self,token,**kwargs):assert token=='token' + p=Fixture('fixture',{'type':'fixture'});c=EvalCase('case','Case','https://example.com','static',['Expected']) + start=time.monotonic();records,_=run_evaluation(output_dir=tmp_path/'run',dataset_meta={'id':'test'},dataset_path=Path('dataset.json'),cases=[c],providers=[p],provider_config_path=Path('providers.json'),runs=1,timeout=.1,delay=0,case_admission=Gate());assert time.monotonic()-start>=.3;assert records[0]['ok'];assert records[0]['elapsed_ms']<100;assert records[0].get('error_type') is None + + +def test_same_method_interval_is_not_replaced_by_site_cooldown(tmp_path): + gate = CaseAdmission(tmp_path, group='local', site_cooldown=.3) + a = gate.acquire(method('a'), case('one.test')) + gate.release(a) + begin = time.monotonic() + a = gate.acquire(method('a'), case('one.test')) + assert time.monotonic()-begin < .1 + gate.release(a) + begin = time.monotonic() + b = gate.acquire(method('b'), case('one.test')) + assert time.monotonic()-begin >= .2 + gate.release(b) + + +def test_failed_scoring_releases_admission_without_hiding_error(tmp_path, monkeypatch): + import pytest + from harness.web_fetch import runner + + class Fixture(Provider): + def fetch(self, case, timeout): + return ProviderResult(True, 'Expected') + + gate = CaseAdmission(tmp_path / 'admission', group='local', site_cooldown=0) + def broken_score(*args, **kwargs): + raise ValueError('scoring failed') + monkeypatch.setattr(runner, 'score_attempt', broken_score) + with pytest.raises(ValueError, match='scoring failed'): + run_evaluation(output_dir=tmp_path / 'run', dataset_meta={'id': 'test'}, + dataset_path=Path('dataset.json'), + cases=[EvalCase('case', 'Case', 'https://example.com', 'static', ['Expected'])], + providers=[Fixture('fixture', {'type': 'fixture'})], + provider_config_path=Path('providers.json'), runs=1, timeout=1, delay=0, + case_admission=gate) + assert not json.loads((gate.root / 'state.json').read_text())['active'] diff --git a/tests/test_code_preservation.py b/tests/test_code_preservation.py new file mode 100644 index 000000000..a122c58a8 --- /dev/null +++ b/tests/test_code_preservation.py @@ -0,0 +1,47 @@ +import re + +import pytest + +from lexbench_webfetch.scoring import score_attempt +from lexbench_webfetch.metrics import task_passed + + +def code_case(): + return {'id': 'example', 'url': 'https://example.test/', 'required_patterns': [ + {'id': 'include', 'match_scope': 'code_compact', + 'pattern': re.escape('#include')}, + {'id': 'loop', 'match_scope': 'code_compact', + 'pattern': re.escape('for(i=0;i\nfor(i=0; i
#include <stdio.h>\nfor(i=0; i<argc; i++){\nrc = sqlite3_open(argv[1], &db);
', True), + ('01 #include \n02 for(i=0; i ```c\n > 01 #include \n > 02 for(i=0; i 03 rc = sqlite3_open(argv[1], &db);\n > ```', True), + ('```c\n#include\nfor(i=0; isqlite3 *db;\nrc = sqlite3_open(argv[1], &db);\n```', False), + ('```c\n#include \nfor(i=0; iAPI names', False), +]) +def test_declared_code_fragments_preserve_tokens_not_just_api_names(answer, valid): + metrics = score_attempt(code_case(), answer.encode(), ok=True) + assert task_passed(True, metrics) is valid + + +@pytest.mark.parametrize('wrapper', ['markdown', 'html', 'tilde', 'long-fence']) +@pytest.mark.parametrize('split', [False, True]) +def test_required_code_fragment_must_fit_inside_one_complete_block(wrapper, split): + pattern = re.escape('sqlite3_open(argv[1],&db);') + case={'id':'blocks','url':'https://example.test/', 'required_patterns':[ + {'id':'open','match_scope':'code_compact','pattern':pattern}]} + fragments=['sqlite3_open(', 'argv[1], &db);'] if split else ['sqlite3_open(argv[1], &db);'] + if wrapper=='html': + answer='

unrelated prose

'.join('
'+f.replace('&','&')+'
' for f in fragments) + else: + fence='~~~' if wrapper=='tilde' else '````' if wrapper=='long-fence' else '```' + answer='\nunrelated prose\n'.join(fence+'c\n'+f+'\n'+fence for f in fragments) + metrics=score_attempt(case,answer.encode(),ok=True) + assert task_passed(True,metrics) is (not split) diff --git a/tests/test_cohort_exclusions.py b/tests/test_cohort_exclusions.py new file mode 100644 index 000000000..984f76360 --- /dev/null +++ b/tests/test_cohort_exclusions.py @@ -0,0 +1,35 @@ +import copy +import hashlib + +import pytest + +from harness.web_fetch.cohort_exclusions import apply_cohort_exclusions, require_cohort_exclusions +from lexbench_webfetch.metrics import task_measured, task_passed + + +def test_environment_exclusion_is_symmetric_and_preserves_calls(tmp_path): + proof = tmp_path / 'dns.json' + proof.write_text('{"address":"10.2.0.11"}') + dataset = {'cases': [{'id': 'a'}, {'id': 'b'}], 'comparison': {'excluded_cases': [ + {'case_id': 'a', 'reason': 'Private DNS address in local environment', + 'evidence_path': 'dns.json', 'evidence_sha256': hashlib.sha256(proof.read_bytes()).hexdigest()}]}} + records = [{'attempt_id': provider + '__' + case, 'case_id': case, 'provider': provider, + 'ok': provider == 'hosted', 'elapsed_ms': 7, 'artifact_path': 'original.txt', + 'metrics': {'content_pass': provider == 'hosted', 'content_sha256': 'a'*64}} + for provider in ('local', 'hosted') for case in ('a', 'b')] + before = copy.deepcopy(records) + apply_cohort_exclusions(tmp_path, dataset, records) + for old, new in zip(before, records): + assert {k: v for k, v in old.items() if k != 'metrics'} == {k: v for k, v in new.items() if k != 'metrics'} + if new['case_id'] == 'a': + assert not task_measured(new['metrics']) + assert not task_passed(new['ok'], new['metrics']) + else: + assert new == old + records[0]['metrics']['evaluation_excluded'] = False + with pytest.raises(ValueError, match='every method'): + require_cohort_exclusions(tmp_path, dataset, records) + records[0]['metrics']['evaluation_excluded'] = True + proof.write_text('changed') + with pytest.raises(ValueError, match='evidence changed'): + require_cohort_exclusions(tmp_path, dataset, records) diff --git a/tests/test_cowagent_budget.py b/tests/test_cowagent_budget.py index ebba04a18..24eacb7ad 100644 --- a/tests/test_cowagent_budget.py +++ b/tests/test_cowagent_budget.py @@ -37,7 +37,8 @@ def execute(self, arguments): continue assert answer.ok and answer.content == 'native answer' assert answer.metadata['effective_timeout_seconds'] == 30 - persist_result_artifacts(tmp_path, Path('artifacts/cow/' + suffix + '.txt'), answer) + persist_result_artifacts(tmp_path, Path('artifacts/cow/' + suffix + '.txt'), answer, + request_url='https://example.com/' + suffix) envelope = json.loads((tmp_path / answer.metadata['raw_response_path']).read_bytes()) assert envelope['raw_tool_response'] == { 'status': 'success', 'result': 'native answer', diff --git a/tests/test_crawl4ai_file_worker.py b/tests/test_crawl4ai_file_worker.py index 13fb9d8fe..98b951bde 100644 --- a/tests/test_crawl4ai_file_worker.py +++ b/tests/test_crawl4ai_file_worker.py @@ -162,6 +162,18 @@ def test_direct_pdf_selection_is_limited_to_document_body_tasks(self): ): self.assertFalse(worker.is_direct_pdf_document(request)) + def test_fixed_native_pdf_route_uses_declared_input_not_url_shape(self): + worker = self.load_worker(object) + request = {'url': 'https://example.test/no-extension', 'track': 'document', + 'task_kind': 'body', 'uniform_native_parameters': True, + 'input_media_kind': 'direct_pdf'} + self.assertTrue(worker.is_direct_pdf_document(request)) + request.update(url='https://example.test/looks-like.pdf', input_media_kind='html_page') + self.assertFalse(worker.is_direct_pdf_document(request)) + request.pop('input_media_kind') + with self.assertRaisesRegex(ValueError, 'declared input_media_kind'): + worker.is_direct_pdf_document(request) + def test_direct_pdf_uses_native_pdf_strategies_and_returns_markdown(self): calls = [] pdf_crawler_instance = object() diff --git a/tests/test_dynamic_100_moli_119.py b/tests/test_dynamic_100_moli_119.py index 65e92fc03..d728a2dbf 100644 --- a/tests/test_dynamic_100_moli_119.py +++ b/tests/test_dynamic_100_moli_119.py @@ -8,9 +8,7 @@ from harness.web_fetch import local_audit from harness.web_fetch.reference_inputs import input_tree from harness.web_fetch.report import local_audit_banner, provider_configuration -from lexbench_webfetch.scoring import score_attempt -from lexbench_webfetch.metrics import task_passed -from lexbench_webfetch.scoring_regressions import load_scoring_regressions +from harness.web_fetch.storage import data_home from lexbench_webfetch.stable_tasks import validate_acceptance_contract from tests.conftest import read_frozen_dataset_bytes @@ -41,7 +39,26 @@ def _dataset(): return json.loads(read_frozen_dataset_bytes(DATASET_PATH).decode("utf-8")) -def test_dynamic_100_dataset_matches_the_frozen_local_audit(tmp_path): +def test_current_scorer_successor_keeps_bound_corrections_executable(): + from lexbench_webfetch.scoring_regressions import require_scoring_regressions + revision = json.loads(REVISION_PATH.read_text()) + successor = revision['current_shared_scorer_authority'] + path = ROOT / successor['path'] + assert hashlib.sha256(path.read_bytes()).hexdigest() == successor['sha256'] + receipt = json.loads(path.read_text()) + assert receipt['provider_calls'] == 0 + assembly = json.loads((ROOT / 'evaluation/webfetch-seven-family-430/dataset-assembly.json').read_text()) + binding = assembly['case_corrections'] + path = ROOT / binding['path'] + assert hashlib.sha256(path.read_bytes()).hexdigest() == binding['sha256'] + corrections = json.loads(path.read_text()) + identifiers = set(receipt['correction_provenance']['regression_ids']) + identifiers.update(identifier for row in corrections['cases'] + for identifier in row.get('evidence', {}).get('regression_ids', [])) + require_scoring_regressions(sorted(identifiers)) + + +def test_dynamic_100_dataset_matches_the_frozen_local_audit(pytestconfig, tmp_path): dataset = _dataset() cases = dataset["cases"] receipt = json.loads(RECEIPT_PATH.read_text(encoding="utf-8")) @@ -319,7 +336,6 @@ def test_dynamic_100_dataset_matches_the_frozen_local_audit(tmp_path): restored.pop(leaf) historical = json.dumps(historical_dataset, ensure_ascii=False, indent=2) + '\n' assert hashlib.sha256(historical.encode()).hexdigest()==followup['predecessor_sha256'] - controls = load_scoring_regressions() for change in revision["dynamic_dataset"]["changes"]: case = next(case for case in cases if case["id"] == change["case_id"]) assert case[change["field"]] == change["value"] @@ -345,45 +361,29 @@ def test_dynamic_100_dataset_matches_the_frozen_local_audit(tmp_path): for row in revision['shared_scorer_followups']) assert set(parallel['regression_ids']) <= set(scorer_revision['regression_ids']) assert scorer_revision["main_baseline_sha256"] == compatibility["bindings"]["scoring_source_sha256"] - successor = revision["current_shared_scorer_authority"] - successor_path = ROOT / successor["path"] - assert hashlib.sha256(successor_path.read_bytes()).hexdigest() == successor["sha256"] - successor_receipt = json.loads(successor_path.read_text()) - assert successor_receipt["provider_calls"] == 0 - assert successor_receipt["scorer_binding"]["revised_scoring_source_sha256"] == ( - "8bc386c7c081b70701d635e4fc2ead8c22a7f9a6004ec42adcaf391f9d470f1b" - ) - assembly_definition = json.loads( - (ROOT / "evaluation/webfetch-seven-family-430/dataset-assembly.json").read_text()) - correction_binding = assembly_definition["case_corrections"] - correction_path = ROOT / correction_binding["path"] - assert hashlib.sha256(correction_path.read_bytes()).hexdigest() == correction_binding["sha256"] - current_corrections = json.loads(correction_path.read_text()) - current_scorer_bindings = { - row["evidence"]["scoring_source_sha256"] - for row in current_corrections["cases"] - if row.get("evidence", {}).get("scoring_source_sha256") - } - # These reference-correction receipts record their original scorer, not - # whatever unrelated scoring fixes are present in the current checkout. - assert current_scorer_bindings == { - "db507a6f0a90c8346aaa0fc7613505c8050c20427547cfccd94754548f1c8e9f"} - current_regression_ids = { - identifier - for row in current_corrections["cases"] - for identifier in row.get("evidence", {}).get("regression_ids", []) - } - assert current_regression_ids <= set(controls) - for identifier in current_regression_ids: - control = controls[identifier] - metrics = score_attempt(control['case'], control['artifact'].encode(), ok=control['ok'], - metadata=control.get('metadata'), title=control.get('title'), - final_url=control.get('final_url')) - assert task_passed(control['ok'], metrics, control['case'].get('task_kind','body')) is control['expected_task_pass'] - for key, expected in control.get('expected_metrics', {}).items(): - assert metrics.get(key) == expected - - assert set(successor_receipt["correction_provenance"]["regression_ids"]) <= set(controls) + # Validate the frozen audit against its actual source bytes, not a later + # implementation. Keep historical snapshots external and read-only. + scorer_files = ("scoring.py", "metrics.py", "stable_tasks.py", "dynamic_lists.py", + "list_verification.py", "reference_quality.py", "dynamic_entry_matching.py", + "response_input.py", "body_coverage.py", "structured_tables.py", + "ordered_tree.py", "composite_tasks.py") + snapshot = data_home() / "scoring-snapshots" / scorer_revision["revised_sha256"] + required = ["harness/web_fetch/" + name for name in scorer_files] + required += ["evaluation/scoring-regressions.json", "tests/test_composite_tasks.py", "tests/test_stable_tasks.py"] + missing = [name for name in required if not (snapshot / name).is_file()] + if missing: + message = "Requires frozen historical scorer inputs: " + str(snapshot) + if pytestconfig.getoption("--require-reference-inputs"): + pytest.fail(message) + pytest.skip(message) + assert not snapshot.is_symlink() + assert not any((snapshot / name).is_symlink() for name in required) + assert scorer_revision["revised_sha256"] == hashlib.sha256(b"\0".join( + (snapshot / "harness/web_fetch" / name).read_bytes() for name in scorer_files)).hexdigest() + assert scorer_revision["scoring_regressions_sha256"] == hashlib.sha256( + (snapshot / "evaluation/scoring-regressions.json").read_bytes()).hexdigest() + controls = {row['id']: row for row in json.loads( + (snapshot / 'evaluation/scoring-regressions.json').read_text())['cases']} assert set(scorer_revision["regression_ids"]) <= set(controls) modes = {} for identifier in scorer_revision["regression_ids"]: @@ -418,10 +418,28 @@ def test_dynamic_100_dataset_matches_the_frozen_local_audit(tmp_path): ] assert bindings["scoring_source_sha256"] == scorer_revision["main_baseline_sha256"] assert bindings["scoring_regressions_sha256"] != scorer_revision["scoring_regressions_sha256"] + assert bindings["composite_tests_sha256"] == hashlib.sha256( + (snapshot / "tests/test_composite_tasks.py").read_bytes() + ).hexdigest() + assert bindings["stable_tests_sha256"] == hashlib.sha256( + (snapshot / "tests/test_stable_tasks.py").read_bytes() + ).hexdigest() assert "Moli 1.1.9 local audit complete" in dataset["approval_status"] assert dataset["comparison"]["local_audit_receipt"] == RECEIPT_PATH.relative_to(ROOT).as_posix() +def historical_controls_path(): + revision = json.loads(REVISION_PATH.read_text()) + scorer = revision['shared_scorer'] + for followup in revision.get('shared_scorer_followups', []): + scorer = {**scorer, **followup} + path = data_home() / 'scoring-snapshots' / scorer['revised_sha256'] / 'evaluation/scoring-regressions.json' + if not path.is_file(): + pytest.skip('requires externally installed frozen historical scorer inputs') + assert hashlib.sha256(path.read_bytes()).hexdigest() == scorer['scoring_regressions_sha256'] + return path + + def require_historical_inputs(): try: input_tree(Path('evaluation/dataset.json'), 'sources') @@ -453,7 +471,7 @@ def test_historical_scoring_controls_allow_only_new_tail_entries(tmp_path, monke require_historical_inputs() dataset = _dataset() receipt = json.loads(RECEIPT_PATH.read_text(encoding="utf-8")) - registry = json.loads((definition_path('evaluation/scoring-regressions.json')).read_text()) + registry = json.loads(historical_controls_path().read_text()) original_file = local_audit._repository_file replacement = tmp_path / "scoring-regressions.json" @@ -527,10 +545,14 @@ def test_local_audit_banner_depends_on_dataset_publication_state(): assert "本地审核版" in local_audit_banner(_dataset()) -def test_dynamic_100_repository_evidence_and_scoring_revision_are_exact(): +def test_dynamic_100_repository_evidence_and_scoring_revision_are_exact(monkeypatch): require_historical_inputs() dataset = _dataset() receipt = json.loads(RECEIPT_PATH.read_text(encoding="utf-8")) + original = local_audit._repository_file + frozen = historical_controls_path() + monkeypatch.setattr(local_audit, '_repository_file', lambda relative: + frozen if relative == 'evaluation/scoring-regressions.json' else original(relative)) local_audit._require_repository_bindings(dataset, receipt) diff --git a/tests/test_failure_diagnosis.py b/tests/test_failure_diagnosis.py index ded672f49..53c9270e0 100644 --- a/tests/test_failure_diagnosis.py +++ b/tests/test_failure_diagnosis.py @@ -117,56 +117,33 @@ def assert_category(self, metrics, expected, **case_fields): self.assertEqual((record, case), before) return result - def test_blocked_page_with_enough_links_is_not_record_shortfall(self): + def test_prompt_diagnostics_do_not_replace_missing_target_reason(self): contract = self.contract(page_patterns=['Catalog'], stable_text_anchors=[ {'id': 'heading', 'alternatives': ['Browse products']}]) metrics = evaluate_stable_task('Access denied\n' + self.links, contract, self.base_url) self.assertTrue(metrics['stable_structure_pass']) - self.assertEqual(metrics['stable_unique_record_count'], 3) self.assertTrue(metrics['stable_blocked_page']) - self.assertIn('blocked_page', metrics['stable_task_errors']) - self.assertNotIn('too_few_complete_records', metrics['stable_task_errors']) - result = self.assert_category(metrics, '返回非目标内容', acceptance_contract=contract) - self.assertIn('blocked_page', result['evidence']) - self.assertNotIn('条目不足', result['evidence']) - # Both persisted representations are understood independently. - without_flag = deepcopy(metrics) - without_flag.pop('stable_blocked_page') - self.assert_category(without_flag, '返回非目标内容') - without_code = deepcopy(metrics) - without_code['stable_task_errors'].remove('blocked_page') - self.assert_category(without_code, '返回非目标内容') + self.assertNotIn('blocked_page', metrics['stable_task_errors']) + result = diagnose(self.record(metrics), {'task_kind': 'body', 'acceptance_contract': contract}, '') + self.assertIsNotNone(result) + self.assertNotIn('blocked_page', result['evidence']) - def test_forbidden_and_blocked_page_have_same_semantics(self): - metrics = evaluate_content('Access denied', [], forbidden_any=['Access denied']) - self.assert_category(metrics, '返回非目标内容') - stable = evaluate_content('Access denied\n' + self.links, [], - forbidden_any=['Access denied'], acceptance_contract=self.contract(), base_url=self.base_url) - self.assert_category(stable, '返回非目标内容') - self.assert_category({'stable_task_pass': False, 'stable_task_errors': ['forbidden_content']}, - '返回非目标内容') + def test_qualified_content_with_diagnostic_hint_has_no_failure(self): + facts = [{'id': 'target', 'alternatives': ['Required target content']}] + metrics = evaluate_content('Required target content\nAccess denied', [], required_facts=facts, + diagnostic_any=['Access denied']) + self.assertEqual(metrics['diagnostic_hits'], ['Access denied']) + self.assertIsNone(diagnose(self.record(metrics), {'task_kind': 'body'}, '')) + missing = evaluate_content('Access denied', [], required_facts=facts, diagnostic_any=['Access denied']) + self.assertFalse(missing['content_pass']) + self.assertIsNotNone(diagnose(self.record(missing), {'task_kind': 'body'}, '')) - def test_generic_blocked_page_precedes_resource_success(self): - blocked = evaluate_content('# Making sure you\'re not a bot!', []) - blocked.update(resource_pass=True, resource_results=[{'id': 'hero', 'pass': True}]) - result = diagnose(self.record(blocked), {'task_kind': 'resource'}, '') - self.assertEqual(result['category'], '返回非目标内容') - self.assertIn("Making sure you're not a bot", result['evidence']) - - prose = evaluate_content("The article explains why sites say you're not a bot.", []) - prose.update(resource_pass=True, resource_results=[{'id': 'hero', 'pass': True}]) - self.assertIsNone(diagnose(self.record(prose), {'task_kind': 'resource'}, '')) - - def test_chinese_gate_heading_precedes_resource_success_without_prose_fp(self): - blocked = evaluate_content('# 验证中心', []) - blocked.update(resource_pass=True, resource_results=[{'id': 'hero', 'pass': True}]) - result = diagnose(self.record(blocked), {'task_kind': 'resource'}, '') - self.assertEqual(result['category'], '返回非目标内容') - self.assertIn('验证中心', result['evidence']) - - prose = evaluate_content('本文介绍验证中心的设计与用户体验。', []) - prose.update(resource_pass=True, resource_results=[{'id': 'hero', 'pass': True}]) - self.assertIsNone(diagnose(self.record(prose), {'task_kind': 'resource'}, '')) + def test_prompt_diagnostics_do_not_veto_delivered_resource(self): + for heading in ("# Making sure you're not a bot!", '# 验证中心'): + metrics = evaluate_content(heading, []) + metrics.update(resource_pass=True, resource_results=[{'id': 'hero', 'pass': True}]) + self.assertTrue(metrics['blocked_page']) + self.assertIsNone(diagnose(self.record(metrics), {'task_kind': 'resource'}, '')) def test_too_many_records_reports_upper_bound_not_too_few(self): contract = self.contract(maximum_records=3) diff --git a/tests/test_failure_retry_policy.py b/tests/test_failure_retry_policy.py index 3069474c0..0cdf2f188 100644 --- a/tests/test_failure_retry_policy.py +++ b/tests/test_failure_retry_policy.py @@ -1,4 +1,5 @@ import json +from dataclasses import asdict import pytest @@ -17,15 +18,24 @@ def __init__(self, answers, *, native=False): def fetch(self, case, timeout): self.calls += 1 - return next(self.answers) + result = next(self.answers) + if self.config['uniform_native_parameters']: + import base64 + import hashlib + raw = json.dumps({'content': result.content}).encode() + result.metadata.update(_raw_response_b64=base64.b64encode(raw).decode(), + raw_response_sha256=hashlib.sha256(raw).hexdigest(), raw_response_bytes=len(raw), + native_answer_projection={'schema_version': 2, 'content_pointer': ['content'], 'content_format': 'text'}) + return result @pytest.fixture def run_task(tmp_path): def run(provider, **options): + case = EvalCase('one', 'One', 'https://example.com', 'static', ['Expected'], source_benchmark={'input_media_kind': 'html_page'}) rows, _ = run_evaluation( - output_dir=tmp_path, dataset_meta={'id': 'retry'}, dataset_path=tmp_path/'dataset.json', - cases=[EvalCase('one', 'One', 'https://example.com', 'static', ['Expected'])], + output_dir=tmp_path, dataset_meta={'id': 'retry', 'cases': [asdict(case)]}, dataset_path=tmp_path/'dataset.json', + cases=[case], providers=[provider], provider_config_path=tmp_path/'providers.json', runs=1, timeout=2, delay=0, **options, ) diff --git a/tests/test_fairness.py b/tests/test_fairness.py index d00ef0467..330a1beb7 100644 --- a/tests/test_fairness.py +++ b/tests/test_fairness.py @@ -287,7 +287,7 @@ def test_product_prefix_fields_follow_explicit_card_boundaries(self): self.assertNotIn('shipping', missing['stable_records'][0]['fields']) self.assertEqual(missing['stable_records'][1]['fields']['shipping'], 'Free shipping') - def test_dynamic_declared_exclusions_are_executable_not_prose_only(self): + def test_dynamic_prompt_hits_are_diagnostic_when_target_is_complete(self): cases = {case['id']: case for case in self.historical_dataset()['cases']} supplemental = { 'dynamic-react-cart': [ @@ -301,22 +301,23 @@ def test_dynamic_declared_exclusions_are_executable_not_prose_only(self): ], } for identifier, tokens in supplemental.items(): - case = cases[identifier] + case = dict(cases[identifier]) + case['diagnostic_any'] = case.pop('forbidden_any', []) target_only = case['ground_truth_main_content'] clean = score_attempt(case, target_only.encode(), ok=True) with self.subTest(case=identifier, variant='target-only'): self.assertTrue(task_passed(True, clean)) - self.assertEqual(clean['forbidden_hits'], []) + self.assertEqual(clean['diagnostic_hits'], []) for token in tokens: extended = score_attempt( case, f'{target_only}\n\n{token}'.encode(), ok=True) with self.subTest(case=identifier, supplemental=token): self.assertTrue(extended['stable_structure_pass']) if identifier == 'dynamic-react-cart' or token == 'Similar Products': - self.assertEqual(extended['forbidden_hits'], [token]) - self.assertFalse(task_passed(True, extended)) + self.assertEqual(extended['diagnostic_hits'], [token]) + self.assertTrue(task_passed(True, extended)) else: - self.assertEqual(extended['forbidden_hits'], []) + self.assertEqual(extended['diagnostic_hits'], []) self.assertTrue(task_passed(True, extended)) def test_end_to_end_processing_is_timed_and_late_answers_rejected(self): @@ -344,7 +345,7 @@ def test_historical_file_missing_is_unmeasured_not_failed(self): m = score_attempt(c,b'link only',ok=True) r = {'provider':'p','ok':True,'metrics':m,'elapsed_ms':20,'case_id':'x'} self.assertFalse(task_measured(m)) - self.assertEqual(result_label(r,c),'未测') + self.assertEqual(result_label(r,c),'无法判定') self.assertIn('data-result="unmeasured"',case_attrs(r,c)) self.assertIn('0/1', ranking([r],{'x':c},'media_file')) diff --git a/tests/test_file_report_reason.py b/tests/test_file_report_reason.py index b0a667098..dd2fb5415 100644 --- a/tests/test_file_report_reason.py +++ b/tests/test_file_report_reason.py @@ -17,3 +17,35 @@ def test_failed_and_unmeasured_remain_distinct(self): def test_late_bytes_do_not_get_success_explanation(self): row={'ok':True,'metrics':{'task_kind':'file','raw_file_pass':True,'deadline_overrun':True}} self.assertIn('超过',reason(row,{'task_kind':'file'})) + + def test_missing_page_identity_is_not_a_file_or_uncalled_result(self): + from harness.web_fetch.report import failure_category, result_label + row={'ok':True,'metrics':{'request_ok':True,'page_identity_required':True, + 'page_identity_status':'unavailable'}} + self.assertEqual(result_label(row,{}),'无法判定') + self.assertIn('最终网页身份',reason(row,{})) + self.assertNotIn('原文件',reason(row,{})) + self.assertIn('无法判定',failure_category(row,{})) + + def test_shared_evidence_lines_preserve_complete_exact_matches(self): + from harness.web_fetch.report import _exact_evidence_lines, _normalized_evidence_lines + output=' A & B \nUnrelated\nA & B\nA & B\n' + cached=_normalized_evidence_lines(output) + for expected in ['A & B','Unrelated','missing','']: + self.assertEqual(_exact_evidence_lines(expected,output), + _exact_evidence_lines(expected,output,normalized_lines=cached)) + self.assertEqual(_exact_evidence_lines('Unrelated',output,normalized_lines=cached),'Unrelated') + self.assertEqual(_exact_evidence_lines('missing',output,normalized_lines=cached),'') + + def test_unknown_result_is_visible_but_not_in_acceptance_denominator(self): + from harness.web_fetch.report import overall_ranking + cases={'a':{'id':'a','track':'web'},'b':{'id':'b','track':'web'}} + rows=[{'provider':'p','case_id':'a','ok':True,'elapsed_ms':1, + 'metrics':{'content_pass':True}}, + {'provider':'p','case_id':'b','ok':True,'elapsed_ms':1, + 'metrics':{'page_identity_required':True,'page_identity_status':'unavailable'}}] + rendered=overall_ranking(rows,cases) + self.assertIn('无法判定',rendered) + self.assertIn('1/1',rendered) + self.assertIn('可判定数',rendered) + self.assertNotIn('每题计一次通过或失败',rendered) diff --git a/tests/test_fixed_native_comparison_audit.py b/tests/test_fixed_native_comparison_audit.py index 12d688d6f..16ffc92dc 100644 --- a/tests/test_fixed_native_comparison_audit.py +++ b/tests/test_fixed_native_comparison_audit.py @@ -23,7 +23,15 @@ def __init__(self, name, config): "binary_sha256": config["expected_binary_sha256"], } + def native_cli_command(self, case): + from harness.web_fetch.providers.moli import MoliWebFetchProvider + return MoliWebFetchProvider.native_cli_command(self, case) + def fetch(self, case, timeout): + command = self.native_cli_command(case) + self.capture_native_request(case, transport='cli', interface='dynamic_native_markdown' if case.track=='dynamic' else 'ordinary_markdown', + parameters={'argv': ['', *command[1:-1], ''], + 'binary_sha256': self.config['expected_binary_sha256']}) diagnostic = (self.name + "/" + case.id + " debug\n").encode() metadata = { "route": "dynamic_native_markdown" if case.track == "dynamic" else "ordinary_markdown", @@ -67,6 +75,9 @@ def fetch(self, case, timeout): else "anchor" ) return ProviderResult(True, content, metadata={**metadata, + '_raw_stdout_b64': base64.b64encode(content.encode()).decode(), + 'raw_stdout_sha256': hashlib.sha256(content.encode()).hexdigest(), + 'native_answer_projection': {'schema_version': 2, 'source_format': 'bytes', 'content_format': 'utf8'}, "reported_version": self.config["expected_reported_version"], "binary_sha256": self.config["expected_binary_sha256"], }) @@ -82,7 +93,7 @@ def comparison_cohort(tmp_path: Path, *, call_order="case-alternating") -> dict[ source = { "id": "source", "tool": "web_fetch", - "cases": [asdict(case) for case in source_cases], + "cases": [{**asdict(case), "source_benchmark": {"input_media_kind": "html_page"}, "acceptance_contract": {"version": 1, "schema": "body_text", "minimum_records": 0, "stable_text_anchors": [{"id": "anchor", "alternatives": ["anchor"]}]}} for case in source_cases], } source_path = tmp_path / "source.json" source_path.write_text(json.dumps(source)) @@ -223,6 +234,7 @@ def convert_to_provider_batches(paths): batches = [] for provider in providers: batch = json.loads(json.dumps(original)) + batch["projection"] = {"kind": "complete-provider-case-subset-from-retained-run"} batch["providers"] = [ item for item in batch["providers"] if item["name"] == provider ] diff --git a/tests/test_fixed_native_policy.py b/tests/test_fixed_native_policy.py index 2bec5567a..af339c792 100644 --- a/tests/test_fixed_native_policy.py +++ b/tests/test_fixed_native_policy.py @@ -1,7 +1,9 @@ import json +from pathlib import Path from dataclasses import replace from unittest.mock import patch import pytest +from tests.conftest import read_frozen_dataset_bytes from lexbench_webfetch.models import EvalCase from harness.web_fetch.providers.base import build_provider from harness.web_fetch.providers.http_client import HttpResponse @@ -39,12 +41,15 @@ def test_fixed_native_requests_preserve_capabilities_without_product_timers(kind @pytest.mark.parametrize("kind", ["moli_webfetch", "firecrawl", "tavily_extract", "exa_contents", "cloudflare_markdown", "cowagent_builtin", "subprocess_jsonl"]) -def test_fixed_native_rejects_per_case_hints_before_call(kind): +def test_fixed_native_rejects_hints_and_unknown_outputs_before_call(kind): provider = build_provider(kind, {"type": kind, "uniform_native_parameters": True}) case = EvalCase("one", "one", "https://example.com/", "control", [], retrieval_hints={"completion": {"selector": ".answer"}}) with pytest.raises(ValueError, match="per-case"): provider.fetch(case, 45) + case = replace(case, retrieval_hints={}, task_kind="undeclared") + with pytest.raises(ValueError, match="declared native output"): + provider.fetch(case, 45) @pytest.mark.parametrize("track,task_kind", [ @@ -70,13 +75,13 @@ def test_fixed_native_rejects_case_specific_page_capabilities(): provider.check_native_call(case) -@pytest.mark.parametrize("track,task_kind", [("document", "resource"), ("image", "file"), +@pytest.mark.parametrize("track,task_kind", [("document", "resource"), ("image", "undeclared"), ("unknown", "body")]) def test_fixed_native_rejects_undeclared_task_routes(track, task_kind): provider = build_provider("fixture", {"type": "fixture", "uniform_native_parameters": True}) case = EvalCase("one", "one", "https://example.com/", "control", [], track=track, task_kind=task_kind) - with pytest.raises(ValueError, match="declared task route"): + with pytest.raises(ValueError, match="declared.*(?:route|output)"): provider.check_native_call(case) @@ -91,12 +96,93 @@ def test_runner_rejects_extra_attempts_before_preflight(tmp_path, runs, retries) assert not (tmp_path / "run").exists() +@pytest.mark.parametrize("kind,module", [("firecrawl", "firecrawl"), ("exa_contents", "exa"), + ("tavily_extract", "tavily"), ("cloudflare_markdown", "cloudflare")]) +def test_web_family_does_not_change_native_cloud_request(kind, module): + provider = build_provider(module, {"type": kind, "uniform_native_parameters": True, + "api_key": "fixture", "api_token": "fixture", "account_id": "fixture"}) + bodies = [] + for track in ("web", "dynamic", "challenge", "risk_control"): + case = EvalCase("one", "one", "https://example.com/", "control", [], track=track, + acceptance_contract={"version": 1, "schema": "body_text", "minimum_records": 0, + "stable_text_anchors": [{"id": "answer", "alternatives": ["Expected answer"]}]}) + with patch(f"harness.web_fetch.providers.{module}.request") as call: + call.return_value = HttpResponse(200, case.url, {}, b'{}') + provider.fetch(case, 45) + assert call.call_count == 1 + bodies.append(call.call_args) + assert all(body == bodies[0] for body in bodies) + + +def test_native_cloudflare_document_never_downloads_in_caller(): + provider = build_provider("cf", {"type": "cloudflare_markdown", "uniform_native_parameters": True}) + case = EvalCase("pdf", "pdf", "https://example.com/a.pdf", "control", [], track="document") + with patch("harness.web_fetch.providers.cloudflare._get_document", side_effect=AssertionError("extra download")), \ + patch("harness.web_fetch.providers.cloudflare._uses_direct_document_route", side_effect=AssertionError("native URL guessing")), \ + patch("harness.web_fetch.providers.cloudflare.request") as call: + call.return_value = HttpResponse(200, case.url, {}, b'{"success":true,"result":"native text"}') + assert provider.fetch(case, 45).content == "native text" + assert call.call_count == 1 + assert call.call_args.args[0].endswith("/markdown") + assert call.call_args.kwargs["json_body"]["url"] == case.url + + +def test_native_cloudflare_media_has_no_answer_dependent_selectors_or_timers(): + provider = build_provider("cf", {"type": "cloudflare_markdown", "uniform_native_parameters": True}) + bodies = [] + for expected in ([], [{"kind": "video-link", "urls": ["https://secret-answer.invalid/"]}]): + case = EvalCase("media", "media", "https://example.com/", "control", [], + track="video", task_kind="resource", expected_resources=expected) + with patch("harness.web_fetch.providers.cloudflare.request") as call: + call.return_value = HttpResponse(200, case.url, {}, b'{"success":true,"result":[]}') + provider.fetch(case, 45) + assert call.call_count == 1 + bodies.append(call.call_args.kwargs["json_body"]) + assert bodies[0] == bodies[1] + assert "actionTimeout" not in bodies[0] + assert "timeout" not in bodies[0]["gotoOptions"] + + +def test_historical_430_cases_pass_native_hint_and_output_guard(): + from harness.web_fetch.config import load_dataset + provider = build_provider("fixture", {"type": "fixture", "uniform_native_parameters": True}) + count = 0 + for filename in ("dataset.json", "dynamic-content-100-moli-webfetch/dataset.json", + "popular-sites/dataset.json", "risk-control/dataset.json"): + path = Path(__file__).resolve().parents[1] / "evaluation" / filename + read_frozen_dataset_bytes(path) + _, cases = load_dataset(path) + for case in cases: + if filename == "dataset.json" and case.track not in {"web", "document", "image", "video"}: + continue + provider.check_native_call(case) + count += 1 + assert count == 430 + + +def test_native_media_projection_keeps_all_returned_fields_without_link_guessing(): + payload = {'success': True, 'result': [ + {'selector': 'video', 'results': [{'attributes': [{'name': 'src', 'value': '/a.mp4'}]}]}, + {'selector': 'a[href]', 'results': [{'attributes': [ + {'name': 'href', 'value': '/opaque'}, {'name': 'href', 'value': '/about'}, + {'name': 'title', 'value': 'Unchanged native label'}]}]}]} + case = EvalCase('control', 'control', 'https://example.com/', 'control', [], + track='video', task_kind='resource') + provider = build_provider('cf', {'type': 'cloudflare_markdown', + 'uniform_native_parameters': True, 'account_id': 'fixture', 'api_token': 'fixture'}) + with patch('harness.web_fetch.providers.cloudflare.request', + return_value=HttpResponse(200, case.url, {}, json.dumps(payload).encode())) as call: + result = provider.fetch(case, 45) + assert call.call_count == 1 + assert result.metadata['videos'] == [ + 'https://example.com/a.mp4', 'https://example.com/about', 'https://example.com/opaque'] + assert json.loads(result.content) == payload['result'] def test_native_dataset_marker_reaches_real_runner_without_per_case_assistance(tmp_path): from harness.web_fetch.retrieval_hints import native_fetch_dataset from harness.web_fetch.runner import run_evaluation from harness.web_fetch.config import load_dataset source = {"id": "native-dynamic", "tool": "web_fetch", "cases": [{"id": "dynamic", "name": "Dynamic", - "url": "https://example.com/", "category": "dynamic", "track": "dynamic", "expected_any": ["fixture"]}]} + "url": "https://example.com/", "category": "dynamic", "track": "dynamic", "expected_any": ["fixture"], "source_benchmark": {"input_media_kind": "html_page"}, "acceptance_contract": {"version": 1, "schema": "body_text", "minimum_records": 0, "stable_text_anchors": [{"id": "fixture", "alternatives": ["fixture"]}]}}]} dataset = native_fetch_dataset(source) assert native_fetch_dataset(dataset) == dataset path = tmp_path / "dataset.json" diff --git a/tests/test_fixed_native_workers.py b/tests/test_fixed_native_workers.py index e6f39b506..5f9c0ab26 100644 --- a/tests/test_fixed_native_workers.py +++ b/tests/test_fixed_native_workers.py @@ -7,6 +7,10 @@ from types import SimpleNamespace from unittest.mock import patch +import pytest +from pydantic import BaseModel +from typing import Any + ROOT = Path(__file__).resolve().parents[1] REQUEST = {"url": "https://example.invalid/", "timeout_seconds": 45, "uniform_native_parameters": True, "track": "web", "task_kind": "body", @@ -28,7 +32,18 @@ def load(self): return [SimpleNamespace(page_content="all native text", metadata assert json.loads(output.getvalue())["content"] == "all native text" -def test_crawl4ai_retains_native_capabilities_and_page_timeout_default(): +@pytest.mark.parametrize("native_success", [True, False]) +def test_crawl4ai_retains_native_capabilities_and_page_timeout_default(native_success): + class NativeEnvelope(BaseModel): + success: bool + markdown: str + error_message: str | None + # Crawl4AI 0.9.4 exposes a deprecated descriptor in failure results. + fit_html: Any = property(lambda _: None) + + native_text = "all native text" if native_success else "" + native_error = None if native_success else "native navigation failed" + envelope = NativeEnvelope(success=native_success, markdown=native_text, error_message=native_error) seen = [] def run_config(**kwargs): seen.append(kwargs) @@ -39,9 +54,9 @@ async def __aenter__(self): return self async def __aexit__(self, *args): pass async def arun(self, **kwargs): assert kwargs["config"].page_timeout == 60000 - return SimpleNamespace(success=True, markdown=SimpleNamespace(raw_markdown="all native text"), + return SimpleNamespace(success=native_success, markdown=SimpleNamespace(raw_markdown=native_text), downloaded_files=[], status_code=200, url=REQUEST["url"], redirected_url=None, redirected_status_code=None, - error_message=None, media={}, links={}, model_dump=lambda **kw: {"native": "complete"}) + error_message=native_error, media={}, links={}, model_dump=envelope.model_dump) fake = {"crawl4ai": SimpleNamespace(AsyncWebCrawler=Crawler, BrowserConfig=lambda **kw: kw, CacheMode=SimpleNamespace(BYPASS="bypass"), CrawlerRunConfig=run_config), "crawl4ai.async_configs": SimpleNamespace(HTTPCrawlerConfig=object), @@ -58,8 +73,12 @@ async def arun(self, **kwargs): assert seen[0]["remove_overlay_elements"] is True assert not {"js_code", "wait_for"} & seen[0].keys() result = json.loads(output.getvalue()) - assert result["ok"], result - assert result["content"] == "all native text" + assert result["ok"] is native_success, result + assert result["content"] == native_text + assert result["error_message"] == native_error + assert result["raw_tool_response"]["success"] is native_success + assert result["raw_tool_response"]["markdown"] == native_text + assert result["raw_tool_response"]["error_message"] == native_error def test_typescript_workers_keep_native_timers(tmp_path): @@ -124,3 +143,31 @@ def test_typescript_workers_keep_native_timers(tmp_path): answer = json.loads(result.stdout) assert 'raw_tool_response' in answer, answer assert answer['raw_tool_response']['seen']['web-fetch-http']['timeoutMs'] == 30000 + + +def test_cowagent_worker_keeps_native_url_only_and_real_debug_logs(tmp_path): + import logging + calls = [] + class Tool: + def __init__(self, config): pass + def execute(self, arguments): + calls.append(arguments) + logging.getLogger("log").debug("native fixture diagnostic") + return SimpleNamespace(status="success", result="native answer", ext_data={}, display="native") + request = {**REQUEST, "log_level": "debug"} + module = SimpleNamespace(WebFetch=Tool, DEFAULT_TIMEOUT=30) + import importlib + real_import = importlib.import_module + def import_native(name, *args, **kwargs): + return module if name == "agent.tools.web_fetch.web_fetch" else real_import(name, *args, **kwargs) + with patch("importlib.import_module", side_effect=import_native), \ + patch("sys.argv", ["worker", str(tmp_path)]), \ + patch("sys.stdin", io.StringIO(json.dumps(request) + "\n")), \ + patch("sys.stdout", new_callable=io.StringIO) as output, \ + patch("sys.stderr", new_callable=io.StringIO) as logs: + worker = runpy.run_path(str(ROOT / "harness/web_fetch/workers/cowagent_worker.py")) + worker["main"]() + assert calls == [{"url": request["url"]}] + assert json.loads(output.getvalue())["content"] == "native answer" + assert "native fixture diagnostic" in logs.getvalue() + assert "native answer" not in logs.getvalue() diff --git a/tests/test_frozen_evidence_copy.py b/tests/test_frozen_evidence_copy.py new file mode 100644 index 000000000..627f3883b --- /dev/null +++ b/tests/test_frozen_evidence_copy.py @@ -0,0 +1,38 @@ +from unittest.mock import patch + +import pytest + +from harness.web_fetch.safety import copy_evidence_file + + +def test_frozen_copy_has_independent_identity_and_writes(tmp_path): + source, target = tmp_path/'source.txt', tmp_path/'target.txt' + source.write_text('saved measurement') + copy_evidence_file(source, target) + assert target.read_bytes() == source.read_bytes() + assert target.stat().st_ino != source.stat().st_ino + target.write_text('generated presentation') + assert source.read_text() == 'saved measurement' + + +def test_unsupported_clone_falls_back_to_an_independent_copy(tmp_path): + source, target = tmp_path/'source.txt', tmp_path/'target.txt' + source.write_bytes(b'saved bytes') + with patch('harness.web_fetch.safety.sys.platform', 'darwin'), \ + patch('ctypes.CDLL', side_effect=OSError('clone unavailable')): + copy_evidence_file(source, target) + assert target.read_bytes() == b'saved bytes' + assert target.stat().st_ino != source.stat().st_ino + + +@pytest.mark.parametrize('side', ['source', 'destination']) +def test_frozen_copy_rejects_symlinked_inputs_or_outputs(tmp_path, side): + real = tmp_path/'real.txt' + real.write_bytes(b'private original') + link = tmp_path/'link.txt' + link.symlink_to(real) + source = link if side == 'source' else real + target = link if side == 'destination' else tmp_path/'copy.txt' + with pytest.raises(ValueError, match='symlink'): + copy_evidence_file(source, target) + assert real.read_bytes() == b'private original' diff --git a/tests/test_interrupted_evidence.py b/tests/test_interrupted_evidence.py index bdf974abc..ade1e9d21 100644 --- a/tests/test_interrupted_evidence.py +++ b/tests/test_interrupted_evidence.py @@ -27,8 +27,7 @@ def setUp(self): config.write_text(json.dumps({"providers": {"fixture": {"type": "fixture"}}})) run_evaluation(output_dir=self.source, dataset_meta=dataset, dataset_path=path, cases=cases, providers=[FixtureProvider("fixture", {"type": "fixture", "fail_cases": ["fail"]})], - provider_config_path=config, runs=1, timeout=1, delay=0, - max_failure_retries=0) + provider_config_path=config, runs=1, timeout=1, delay=0, max_failure_retries=0) self.rows = [json.loads(line) for line in (self.source / "results.jsonl").read_text().split("\n") if line] self.rows = self.rows[:2] (self.source / "results.jsonl").write_text("".join(json.dumps(row) + "\n" for row in self.rows)) @@ -98,7 +97,7 @@ def test_adding_a_provider_preserves_the_declared_review_policy(self): output = root / provider run_evaluation(output_dir=output, dataset_meta=dataset, dataset_path=path, cases=[case], providers=[FixtureProvider(provider, {"type": "fixture"})], - provider_config_path=config, runs=1, timeout=1, delay=0) + provider_config_path=config, runs=1, timeout=1, delay=0, max_failure_retries=0) (output / "sources").mkdir() (output / "list-references").mkdir() runs.append(output) diff --git a/tests/test_local_audit.py b/tests/test_local_audit.py index babf0eb75..44134eb99 100644 --- a/tests/test_local_audit.py +++ b/tests/test_local_audit.py @@ -24,6 +24,21 @@ def test_frozen_scoring_controls_allow_additions_but_reject_edits(tmp_path, monk scoring_path.parent.mkdir(parents=True) scoring_path.write_bytes((original_root / bindings['scoring_revision_path']).read_bytes()) scoring = json.loads(scoring_path.read_text()) + # This boundary test owns a frozen synthetic registry. Historical measured + # registries are retained with their scorer, not selected by today's runtime. + controls = {'cases': [ + {'id': 'base', 'artifact': 'fixed'}, + {'id': 'positive', 'artifact': 'complete', 'root_cause': 'fixture', 'failure_mode': 'false-negative'}, + {'id': 'negative', 'artifact': 'missing', 'root_cause': 'fixture', 'failure_mode': 'false-positive'}]} + scoring['preexisting_control_ids'] = ['base'] + scoring['revision_controls'] = [{k: row[k] for k in ('id', 'root_cause', 'failure_mode')} + for row in controls['cases'][1:]] + scoring['bindings']['preexisting_controls_sha256'] = local_audit._canonical_sha256(controls['cases'][:1]) + scoring['bindings']['revision_controls_sha256'] = local_audit._canonical_sha256(controls['cases'][1:]) + scoring['root_causes'] = [{'id': 'fixture', 'false_positive_ids': ['negative'], 'false_negative_ids': ['positive']}] + scoring['scope'] = {'preexisting_controls': 1, 'revision_controls': 2, 'root_causes': 1, 'paired_error_directions': True} + _write(scoring_path, scoring) + bindings['scoring_revision_sha256'] = _digest(scoring_path) dataset = {'id': scoring['dataset_id'], 'cases': [{'id': 'fixture'}]} reference = root / 'evaluation/sources/fixture/reference.txt' reference.parent.mkdir(parents=True) @@ -42,8 +57,6 @@ def test_frozen_scoring_controls_allow_additions_but_reject_edits(tmp_path, monk }) bindings['source_manifest_sha256'] = _digest(source_manifest) relative = 'evaluation/scoring-regressions.json' - from lexbench_webfetch.inputs import definition_path - controls = json.loads(definition_path(relative).read_text()) controls['cases'].append({'id': 'new-independent-control'}) path = root / relative _write(path, controls) @@ -234,9 +247,9 @@ def test_new_local_matrix_is_complete_and_keeps_logs_without_claiming_review(tmp output = tmp_path / "report" build_report(evidence, output, local_audit=True, include_english=include_english) home = (output / "index.html").read_text() - assert "本地实测版" in home - assert "1 个案例 · 1 次调用，机器判分。" in home - assert "本轮未重新核验来源。" in home + assert "本地未认证诊断快照" in home + assert "查看调用链问题和日志留存范围" not in home + assert 'href="guide/index.html"' in home assert "5案例小范围" not in home guide = (output / "guide/index.html").read_text() assert "比例只对应本报告列出的案例与配置" in guide @@ -282,13 +295,102 @@ def test_new_local_matrix_is_complete_and_keeps_logs_without_claiming_review(tmp (output / "results.jsonl").write_text("".join(json.dumps(row) + "\n" for row in frozen_rows)) manifest = json.loads((output / "report-manifest.json").read_text()) manifest["results_sha256"] = _digest(output / "results.jsonl") + manifest["template"] = {"renderer": "historical-template"} _write(output / "report-manifest.json", manifest) binding = {"comparison_kind": "uniform_native_local_measurement", "result_count": 1, "case_count": 1, "providers": ["fixture"], "frozen_dataset_sha256": _digest(output / "dataset.json"), "results_sha256": manifest["results_sha256"], "measured_scoring_source_sha256": historical_scorer, - "measured_scoring_git_revision": "historical-revision"} + "measured_scoring_git_revision": "historical-revision", + "evidence_bindings": [{"path": name, "sha256": _digest(output / name)} + for name in ("local-run-contract.json", "execution.json")]} + _write(binding_path, binding) + for missing in ("local-run-contract.json", "execution.json"): + _write(binding_path, {**binding, "evidence_bindings": [ + item for item in binding["evidence_bindings"] if item["path"] != missing]}) + with pytest.raises(ValueError, match="input lacks a hash binding: " + missing): + build_report(output, tmp_path / (missing + "-unbound"), local_audit=True, + presentation_binding=binding_path) + _write(binding_path, binding) + with pytest.raises(ValueError, match="outside its source tree"): + build_report(output, output / "nested-report", local_audit=True, + presentation_binding=binding_path) + public_logs = output / "logs" + public_logs.mkdir(exist_ok=True) + attempt_id = frozen_rows[0]["attempt_id"] + debug = public_logs / (attempt_id + ".debug.txt") + response = public_logs / (attempt_id + ".response.txt") + debug.write_text("actual library event\n") + response.write_text('{"observed": "actual API response"}\n') + rebuilt = tmp_path / "frozen-bilingual-report" + before_results = (output / "results.jsonl").read_bytes() + with pytest.raises(ValueError, match="stale scoring"): + build_report(output, rebuilt, local_audit=True, include_english=True) + build_report(output, rebuilt, local_audit=True, include_english=True, + presentation_binding=binding_path) + assert (rebuilt / "results.jsonl").read_bytes() == before_results + assert (rebuilt / "dataset.json").read_bytes() == (output / "dataset.json").read_bytes() + assert (rebuilt / "en/index.html").is_file() + for prefix in ("", "en/"): + detail = (rebuilt / prefix / "attempts" / (attempt_id + ".html")).read_text() + assert attempt_id + ".debug.txt" in detail + assert attempt_id + ".response.html" in detail + from harness.web_fetch.report_i18n import untranslated_english_narrative, localize_html + from pathlib import PurePosixPath + english_detail = (rebuilt / "en/attempts" / (attempt_id + ".html")).read_text() + assert "View complete API response (credentials and local paths hidden)" in english_detail + for suffix in ("debug", "response"): + view = (rebuilt / "en/logs" / (attempt_id + "." + suffix + ".html")).read_text() + assert not untranslated_english_narrative(view, {"中文"}) + sample = ('

必需文字组 3（任一写法即可）

first\n或\nsecond
引文 first\n或\nsecond
' + '

正文没有抓全；低于目标的位置为：位置 1 77.5%。下方会把完整标准答案与实际返回并排展示。

' + '
原始返回：位置 1\n或\n真实正文
') + localized = localize_html(sample, PurePosixPath("attempts/example.html")) + assert "Required text group 3" in localized + assert "first\nor\nsecond" in localized + assert "引文 first\n或\nsecond" in localized + from harness.web_fetch.report import required_text_answer_panel, stable_answer_panel + alternatives = ["\n或\noriginal", "second"] + panels = [required_text_answer_panel({"required_facts": [{"alternatives": alternatives}]}), + stable_answer_panel({"stable_text_anchors": [{"alternatives": alternatives}], "minimum_records": 1})] + for panel in panels: + assert '<source>\n或\noriginal\n或\nsecond' in panel + english_panel = localize_html(panel, PurePosixPath("attempts/example.html")) + assert '<source>\n或\noriginal\nor\nsecond' in english_panel + assert "Region 1 77.5%" in localized + assert '
原始返回：位置 1\n或\n真实正文
' in localized + assert (rebuilt / "logs" / debug.name).read_bytes() == debug.read_bytes() + assert (rebuilt / "logs" / response.name).read_bytes() == response.read_bytes() + assert json.loads((rebuilt / "report-manifest.json").read_text())["presentation"]["frozen_scores"] + rebuilt_manifest = json.loads((rebuilt / "report-manifest.json").read_text()) + from harness.reporting.templates import resolve_template + assert rebuilt_manifest["template"] == resolve_template("webfetch") + assert rebuilt_manifest["generated_languages"] == ["zh", "en"] + assert rebuilt_manifest["presentation"]["authored_interpretation"] is False + from harness.web_fetch.report_check import check + for invalid_template in (None, {"renderer": "unregistered-renderer"}): + _write(rebuilt/"report-manifest.json", {**rebuilt_manifest, "template": invalid_template}) + with pytest.raises(ValueError, match="maintained template"): + check(rebuilt, authority_root=Path(__file__).resolve().parents[1], + presentation_binding=binding_path) + _write(rebuilt/"report-manifest.json", rebuilt_manifest) + reproduce = (rebuilt / "reproduce/index.html").read_text() + assert "--language zh --include-english" in reproduce + assert "--language bilingual" not in reproduce + assert "--presentation-binding" in reproduce + assert all(row["metrics"]["scoring_source_sha256"] == historical_scorer + for row in map(json.loads, (rebuilt / "results.jsonl").read_text().splitlines())) + chinese = tmp_path / "frozen-chinese-report" + build_report(rebuilt, chinese, local_audit=True, presentation_binding=binding_path) + assert list((chinese / "en").rglob("*.html")) == [chinese / "en/index.html"] + assert "not been generated" in (chinese / "en/index.html").read_text() + _write(binding_path, {**binding, "results_sha256": "0" * 64}) + with pytest.raises(ValueError, match="input binding changed"): + build_report(output, tmp_path / "invalid-rebuild", local_audit=True, + presentation_binding=binding_path) + _write(binding_path, binding) + binding.pop("measured_scoring_git_revision", None) _write(binding_path, binding) (output / "changes/index.html").write_text("Old release history") before = {path.relative_to(output).as_posix(): _digest(path) @@ -304,7 +406,10 @@ def interrupted_swap(path, target): refresh_local_presentation(output, binding_path) assert {path.relative_to(output).as_posix(): _digest(path) for path in output.rglob('*') if path.is_file()} == before - refresh_local_presentation(output, binding_path) + from harness.web_fetch.report_build import generate as current_renderer + with patch('harness.web_fetch.report_build.generate', wraps=current_renderer) as rendered: + refresh_local_presentation(output, binding_path) + assert rendered.call_args.kwargs['authored_interpretation'] is False allowed = {"index.html", "methodology/index.html", "adoption/index.html", "reproduce/index.html", "changes/index.html", "providers/fixture.html", "adoption/interfaces/fixture.json", "report-manifest.json"} @@ -317,6 +422,7 @@ def interrupted_swap(path, target): refreshed = json.loads((output / "report-manifest.json").read_text()) assert refreshed["certified"] is False assert refreshed["presentation_refresh"]["scores_changed"] is False + assert refreshed["presentation_refresh"]["scoring_git_revision"] is None assert "Old release history" not in (output / "changes/index.html").read_text() assert "2026-10-09-dynamic-1000-native-presentation" not in (output / "changes/index.html").read_text() assert "changes/index.html" in refreshed["presentation_refresh"]["changed_routes"] @@ -391,13 +497,14 @@ def test_local_call_chain_section_discloses_real_log_gaps_and_escapes_text(): dataset = {"comparison": {"call_chain_audit": { "summary": "原判定保留；评分问题未更正", "methods": [{"provider": "moli-webfetch", "calls": 5, "request_ok": 0, - "task_pass": 0, "logs": 5, "native_log_note": "2次原生日志缺失", + "task_pass": 0, "logs": 5, "native_log_note": "2次原生日志缺失；<内部过程未知>", "assessment": "<原生内部原因未知>"}], "findings": [{"text": "无新增调用"}]}}} html = local_call_chain_section(dataset) - assert "2次原生日志缺失" in html and "无新增调用" in html - assert "<原生内部原因未知>" in html - assert "原判定保留；评分问题未更正" in html + assert "2次原生日志缺失" in html + assert "<内部过程未知>" in html + assert "调用数" not in html and "题目通过" not in html + assert "调用方事件" in html and "产品与依赖库日志" in html def test_report_real_diagnostics_do_not_claim_missing_debug(tmp_path): @@ -550,3 +657,203 @@ def test_worker_protocol_response_is_not_a_runtime_log(): assert "native_tool_returned" in caller assert "raw_tool_response" not in native+caller assert runtime_log_text(response+progress+worker)==progress+worker + + +def test_local_offline_projection_keeps_native_url_and_call_facts(tmp_path): + import copy + original = {"attempt_id": "a", "ok": True, "elapsed_ms": 17, + "final_url": "https://example.test/input", "metrics": {"content_sha256": "body"}} + corrected = copy.deepcopy(original) + corrected.update(final_url="https://example.test/actual", scoring={"mode": "offline_rescore"}) + corrected["metrics"]["score"] = 1 + native = tmp_path / "native.json" + _write(native, {"data": {"url": corrected["final_url"]}}) + receipt = {"original_call_identities": {"a": local_audit.local_measurement_call_sha256(original)}, + "call_identities": {"a": local_audit.local_measurement_call_sha256(corrected)}, + "field_projections": {"a": {"final_url": {"path": "native.json", "sha256": _digest(native), + "json_pointer": ["data", "url"], "original_value": original["final_url"]}}}} + path = tmp_path / "local-evidence/scoring-correction.json" + _write(path, receipt) + local_audit._require_local_record_correction(tmp_path, original, corrected) + changed = copy.deepcopy(corrected) + changed["elapsed_ms"] = 1 + receipt["call_identities"]["a"] = local_audit.local_measurement_call_sha256(changed) + _write(path, receipt) + with pytest.raises(ValueError, match="measured call facts"): + local_audit._require_local_record_correction(tmp_path, original, changed) + _write(path, {**receipt, "call_identities": {"a": local_audit.local_measurement_call_sha256(corrected)}}) + _write(native, {"data": {"url": "https://example.test/wrong"}}) + with pytest.raises(ValueError, match="native response"): + local_audit._require_local_record_correction(tmp_path, original, corrected) + + +def test_local_review_retains_deferred_failure_and_rejects_wrong_coverage(tmp_path): + import copy + (tmp_path / "results.jsonl").write_text("saved results") + record = {"attempt_id": "a", "ok": False, "metrics": {"content_sha256": "body"}} + review = {"kind": "local_independent_review", "review_protocol": {"scope": "all"}, + "reviewed_attempts": 1, "expected_attempts": 1, + "results_sha256": _digest(tmp_path / "results.jsonl"), "unresolved": 1, + "attempts": [{"attempt_id": "a", "content_sha256": "body", "agent_verdict": "pass", + "current_verdict": "fail", "final_verdict": None, + "status": "deferred_unadjudicated"}]} + path = tmp_path / "review.json" + def bind(value): + _write(path, value) + return {"comparison": {"independent_review": {"path": "review.json", "sha256": _digest(path)}}} + assert local_audit._require_local_review(tmp_path, bind(review), [record])["unresolved"] == 1 + changed = copy.deepcopy(review) + changed["attempts"][0]["final_verdict"] = "pass" + with pytest.raises(ValueError, match="scored evidence"): + local_audit._require_local_review(tmp_path, bind(changed), [record]) + changed = copy.deepcopy(review) + changed["attempts"].append(changed["attempts"][0]) + with pytest.raises(ValueError, match="scope differs"): + local_audit._require_local_review(tmp_path, bind(changed), [record]) + changed = {**review, "unresolved": 0} + with pytest.raises(ValueError, match="unresolved count"): + local_audit._require_local_review(tmp_path, bind(changed), [record]) + + +def test_local_review_banner_distinguishes_reviewed_and_original_measurements(): + from harness.web_fetch.report import local_audit_banner + dataset = {"comparison": {"kind": "uniform_native_local_measurement", + "publication_state": "local_audit_only_not_for_release"}} + assert "未进行独立人工审核" in local_audit_banner(dataset) + dataset["comparison"]["independent_review"] = {"path": "review.json"} + text = local_audit_banner(dataset) + assert "已有复核记录" in text + assert "已完成全量独立复核" not in text + assert "未进行独立人工审核" not in text + + +def test_retrospective_page_review_preserves_call_and_checks_other_requirements(tmp_path): + import copy + from lexbench_webfetch.scoring import score_attempt + from lexbench_webfetch.metrics import task_passed + body = 'Expected complete body. A second distinctive sentence describing this page.' + case = {'id': 'synthetic', 'url': 'https://example.test/start', 'track': 'risk_control', + 'task_kind': 'body', 'page_identity_required': True, 'source_benchmark': {'risk_control': {'navigation_contract': { + 'requested_url': 'https://example.test/start', + 'observed_final_url': 'https://example.test/document'}}}, + 'acceptance_contract': {'version': 1, 'schema': 'body_text', 'minimum_records': 0, + 'stable_text_anchors': [{'id': 'body', 'alternatives': ['Expected complete body']}]}} + (tmp_path / 'answer.txt').write_text(body) + record = {'attempt_id': 'method__synthetic__r001', 'case_id': 'synthetic', + 'artifact_path': 'answer.txt', 'ok': True, 'final_url': None, + 'metrics': score_attempt(case, body.encode(), ok=True)} + captures = [{'url': 'https://example.test/document', 'text': body}] + annotation = {'attempt_id': record['attempt_id'], + 'case_sha256': local_audit._canonical_sha256(case), + 'call_sha256': local_audit.local_measurement_call_sha256(record), + 'content_sha256': record['metrics']['content_sha256'], + 'basis': 'retrospective_content_correspondence', 'reviewed_at': '2026-10-13', + 'capture_index': 0, 'capture_sha256': local_audit._canonical_sha256(captures[0]), + 'corresponding_url': captures[0]['url'], + 'matching_samples': ['Expected complete body.', + 'A second distinctive sentence describing this page.']} + before = copy.deepcopy(record) + scored = local_audit.score_reviewed_page_identity(tmp_path, case, record, annotation, captures) + assert record == before and record['final_url'] is None + assert task_passed(True, scored) + assert scored['page_identity_basis'] == 'retrospective_content_correspondence' + # Two substantial matches establish correspondence; additional true short + # labels do not invalidate them and cannot replace either long match. + with_short = {**annotation, 'matching_samples': annotation['matching_samples'] + ['body']} + assert task_passed(True, local_audit.score_reviewed_page_identity(tmp_path, case, record, with_short, captures)) + with pytest.raises(ValueError, match='substantive text'): + local_audit.score_reviewed_page_identity(tmp_path, case, record, + {**annotation, 'matching_samples': [annotation['matching_samples'][0], 'body']}, captures) + for samples in ([' ' * 20, ' ' * 21], ['.' * 20, '!' * 21], + ['Expected complete body.', '**Expected complete body.**']): + with pytest.raises(ValueError, match='substantive text'): + local_audit.score_reviewed_page_identity(tmp_path, case, record, + {**annotation, 'matching_samples': samples}, captures) + for field, value, message in ( + ('content_sha256', 'wrong', 'binding'), + ('capture_sha256', 'wrong', 'capture'), + ('matching_samples', ['Missing substantive text', 'Other missing substantive text'], 'matching'), + ): + bad = {**annotation, field: value} + with pytest.raises(ValueError, match=message): + local_audit.score_reviewed_page_identity(tmp_path, case, record, bad, captures) + wrong = [{'url': 'https://example.test/other', 'text': body}] + bad = {**annotation, 'corresponding_url': wrong[0]['url'], + 'capture_sha256': local_audit._canonical_sha256(wrong[0])} + with pytest.raises(ValueError, match='URL requirement'): + local_audit.score_reviewed_page_identity(tmp_path, case, record, bad, wrong) + # A reviewed page identity must never excuse missing required content. + missing = copy.deepcopy(case) + missing['acceptance_contract']['stable_text_anchors'].append( + {'id': 'missing', 'alternatives': ['A required fact that was not returned']}) + bad = {**annotation, 'case_sha256': local_audit._canonical_sha256(missing)} + with pytest.raises(ValueError, match='unavailable page identity'): + local_audit.score_reviewed_page_identity(tmp_path, missing, record, bad, captures) + + +def test_retrospective_page_review_requires_bound_receipt(tmp_path): + with pytest.raises(ValueError, match='lack a receipt'): + local_audit._require_page_identity_review(tmp_path, {}, [ + {'attempt_id': 'synthetic', 'metrics': { + 'page_identity_basis': 'retrospective_content_correspondence'}}]) + + +@pytest.mark.parametrize('defect', [None, 'target_changed', 'sample_changed']) +@pytest.mark.parametrize('implicit_identity', [False, True]) +def test_offline_hint_revision_reapplies_bound_identity_evidence(tmp_path, defect, implicit_identity): + import copy + from lexbench_webfetch.scoring import score_attempt + body = 'Expected complete body. A second distinctive sentence describing this page.' + old = {'id': 'synthetic', 'url': 'https://example.test/start', 'track': 'risk_control', + 'task_kind': 'body', 'forbidden_any': ['CAPTCHA'], + 'page_identity_required': True, 'source_benchmark': {'risk_control': {'navigation_contract': { + 'requested_url': 'https://example.test/start', 'observed_final_url': 'https://example.test/document'}}}, + 'acceptance_contract': {'version': 1, 'schema': 'body_text', 'minimum_records': 0, + 'stable_text_anchors': [{'id': 'body', 'alternatives': ['Expected complete body']}]}} + current = copy.deepcopy(old) + if implicit_identity: + old.pop('page_identity_required') + current['diagnostic_any'] = current.pop('forbidden_any') + (tmp_path/'answer.txt').write_text(body) + record = {'attempt_id': 'method__synthetic__r001', 'case_id': 'synthetic', + 'artifact_path': 'answer.txt', 'ok': True, 'final_url': None, + 'metrics': score_attempt(current, body.encode(), ok=True)} + captures = [{'url': 'https://example.test/document', 'text': body}] + def save(name, obj): + _write(tmp_path/name, obj) + return {'path': name, 'sha256': _digest(tmp_path/name)} + capture = save('captures.json', captures) + annotation = {'attempt_id': record['attempt_id'], 'case_sha256': local_audit._canonical_sha256(old), + 'call_sha256': local_audit.local_measurement_call_sha256(record), + 'content_sha256': record['metrics']['content_sha256'], 'basis': 'retrospective_content_correspondence', + 'reviewed_at': '2026-10-13', 'capture_index': 0, + 'capture_sha256': local_audit._canonical_sha256(captures[0]), 'corresponding_url': captures[0]['url'], + 'matching_samples': ['Expected complete body.', 'A second distinctive sentence describing this page.']} + if defect == 'sample_changed': annotation['matching_samples'] = [' '*20, ' '*21] + receipt = save('identity.json', {'kind': 'retrospective_page_identity_review', + 'reviewed_at': '2026-10-13', 'basis': 'retained human review', 'provider_calls_added': 0, + 'captures_path': capture['path'], 'captures_sha256': capture['sha256'], 'attempts': [annotation]}) + before = {'cases': [old], 'comparison': {'page_identity_review': receipt}} + dataset = {'cases': [current], 'comparison': {}} + prior_receipt = (tmp_path/'identity.json').read_bytes() + if defect == 'target_changed': current['task_prompt'] = 'A new target requirement' + if defect: + with pytest.raises(ValueError): + local_audit.carry_reviewed_page_identities(tmp_path, before, dataset, [record]) + else: + local_audit.carry_reviewed_page_identities(tmp_path, before, dataset, [record]) + assert local_audit.task_passed(record['ok'], record['metrics']) + assert record['final_url'] is None + local_audit._require_page_identity_review(tmp_path, dataset, [record]) + record['metrics']['scoring_source_sha256'] = 'historical-source' + sources = {record['attempt_id']: 'historical-source'} + local_audit._require_page_identity_review(tmp_path, dataset, [record], scoring_sources=sources) + with pytest.raises(ValueError, match='shared scorer'): + local_audit._require_page_identity_review(tmp_path, dataset, [record]) + with pytest.raises(ValueError, match='scoring source'): + local_audit._require_page_identity_review(tmp_path, dataset, [record], + scoring_sources={record['attempt_id']: 'other-source'}) + record['metrics']['content_chars'] += 1 + with pytest.raises(ValueError, match='shared scorer'): + local_audit._require_page_identity_review(tmp_path, dataset, [record], scoring_sources=sources) + assert (tmp_path/'identity.json').read_bytes() == prior_receipt diff --git a/tests/test_local_diagnostic_presentation.py b/tests/test_local_diagnostic_presentation.py new file mode 100644 index 000000000..1b50b83d7 --- /dev/null +++ b/tests/test_local_diagnostic_presentation.py @@ -0,0 +1,250 @@ +"""Uncertified local evidence must remain visibly diagnostic, even when complete.""" +import re +from pathlib import PurePosixPath + +import pytest +from bs4 import BeautifulSoup + +from harness.web_fetch.report import generate, local_audit_banner, overall_ranking, ranking +from tests.test_report_body_scope import case, record + + +def test_diagnostic_tables_do_not_assign_ranks_or_initial_score_order(): + c = case('body', kind='body', track='web') + high = record(c) + high['provider'] = 'z-high' + low = record(c, passed=False) + low['provider'] = 'a-low' + for rendered in (overall_ranking([high, low], {c['id']: c}, diagnostic=True), + ranking([high, low], {c['id']: c}, 'web', diagnostic=True)): + soup = BeautifulSoup(rendered, 'html.parser') + table = soup.select_one('table.ranking') + assert table.get('data-col') is None + assert table.select_one('th').get_text(strip=True).startswith('方法') + rows = table.select('tbody tr') + assert ['a-low', 'z-high'] == [row.select_one('td a')['href'].split('/')[-1].removesuffix('.html') for row in rows] + assert len(rows[0].select('td')) == len(table.select('th')) + assert '0)' in table.select_one('th button')['onclick'] + # Certified comparisons retain their existing score order and ordinal column. + certified = BeautifulSoup(overall_ranking([high, low], {c['id']: c}), 'html.parser') + assert certified.select_one('tbody tr td').get_text() == '1' + assert 'z-high' in certified.select_one('tbody tr').get_text() + + +def test_uncertified_status_cannot_be_suppressed(tmp_path): + dataset = {'comparison': {'kind': 'uniform_native_local_measurement', + 'publication_state': 'local_audit_only_not_for_release'}} + assert '不可用于排名' in local_audit_banner(dataset) + with pytest.raises(ValueError, match='cannot be hidden'): + generate(tmp_path / 'report', dataset, [], [], show_local_status_banners=False) + assert not (tmp_path / 'report').exists() + + +def test_diagnostic_status_is_localized(): + from harness.web_fetch.report_i18n import localize_html + banner = local_audit_banner({'comparison': {'kind': 'uniform_native_local_measurement', + 'publication_state': 'local_audit_only_not_for_release'}}) + english = localize_html(banner, PurePosixPath('index.html')) + assert 'uncertified' in english + assert not re.search('[\u3400-\u9fff]', english) + + +def test_table_header_gate_accepts_explicit_unranked_methods_only_locally(): + from harness.web_fetch.report_check import Tables, require_table_index_headers + for english, label in [(False, '方法 ↕'), (True, 'Method ↕')]: + parser = Tables() + parser.feed('
'+label+'
') + require_table_index_headers(parser, english=english, diagnostic=True) + with pytest.raises(ValueError, match='index header'): + require_table_index_headers(parser, english=english, diagnostic=False) + missing_flag = Tables() + missing_flag.feed('
'+label+'
') + with pytest.raises(ValueError, match='index header'): + require_table_index_headers(missing_flag, english=english, diagnostic=True) + + +def test_home_gate_requires_diagnostic_summary_and_rejects_winner_claims(): + from harness.web_fetch.report_check import require_home_comparison_layout + home = ('

任务判定汇总（非排名）

7 类任务概览

' + '

按任务类别查看诊断结果

判分完整性

') + require_home_comparison_layout(home, 7, diagnostic=True) + with pytest.raises(ValueError, match='winner disclosure'): + require_home_comparison_layout(home+'

本类最高：A

', 7, diagnostic=True) + with pytest.raises(ValueError, match='comparison must precede'): + require_home_comparison_layout(home, 7, diagnostic=False) + + +def test_release_check_rejects_hidden_status_or_ranked_diagnostic(tmp_path): + from harness.web_fetch.report_check import check_local_diagnostic_presentation + page = tmp_path / 'index.html' + prefix = '' + page.write_text(prefix + '

Complete results

') + with pytest.raises(ValueError, match='disclosure missing'): + check_local_diagnostic_presentation(tmp_path) + banner = '
Uncertified
' + page.write_text(prefix + banner + '
Rank
') + with pytest.raises(ValueError, match='assigns ranks'): + check_local_diagnostic_presentation(tmp_path) + page.write_text(prefix + banner + '
Method
') + check_local_diagnostic_presentation(tmp_path) + + +def test_frozen_download_summary_uses_current_scores_and_exclusions(tmp_path): + import copy + import json + from harness.web_fetch.report_build import write_frozen_result_summary + from harness.web_fetch.reliability import assess_reliability + c = case('body', kind='body', track='web') + included = record(c) + excluded = copy.deepcopy(included) + excluded['attempt_id'] = 'excluded' + excluded['provider'] = 'excluded-method' + excluded['metrics']['evaluation_excluded'] = True + rows = [included, excluded] + dataset = {'cases': [c], 'coverage_plan': {'web': [row['provider'] for row in rows]}} + saved = copy.deepcopy(rows) + (tmp_path / 'summary.json').write_text('{"stale":999}') + (tmp_path / 'reliability.json').write_text('{"stale":true}') + write_frozen_result_summary(tmp_path, dataset, rows) + summary = json.loads((tmp_path / 'summary.json').read_text())['web'] + assert summary[included['provider']]['task_pass'] == 1 + assert summary['excluded-method']['task_pass'] == 0 + assert summary['excluded-method']['task_evaluated_attempts'] == 0 + assert summary['excluded-method']['task_excluded_attempts'] == 1 + assert json.loads((tmp_path / 'reliability.json').read_text()) == assess_reliability(dataset, rows) + assert rows == saved + from harness.web_fetch.report_check import check_saved_result_summary + check_saved_result_summary(tmp_path, dataset, rows) + summary['excluded-method']['task_pass'] = 1 + (tmp_path / 'summary.json').write_text(json.dumps({'web': summary})) + with pytest.raises(ValueError, match='summary differs'): + check_saved_result_summary(tmp_path, dataset, rows) + + +def test_explicit_sample_ranking_reuses_existing_ranked_table_and_keeps_scope(tmp_path): + from harness.web_fetch.report import local_pass_rate_ranking + from harness.web_fetch.report_check import check_local_diagnostic_presentation + dataset = {'comparison': {'kind': 'uniform_native_local_measurement', + 'publication_state': 'local_audit_only_not_for_release', + 'presentation': {'ranking': 'saved_task_pass_rate'}}} + c = case('body', kind='body', track='web') + high, low = record(c), record(c, passed=False) + high['provider'], low['provider'] = 'z-high', 'a-low' + markup = overall_ranking([low, high], {c['id']: c}, diagnostic=not local_pass_rate_ranking(dataset)) + soup = BeautifulSoup(markup, 'html.parser') + assert soup.select_one('tbody tr td').get_text() == '1' + assert 'z-high' in soup.select_one('tbody tr').get_text() + banner = local_audit_banner(dataset) + assert 'data-ranking-scope="saved-task-pass-rate"' in banner + assert '未完成仓库认证流程' in BeautifulSoup(banner, 'html.parser').get_text() + assert 'hidden' not in banner + assert 'callout' not in banner + page = tmp_path/'index.html' + page.write_text(''+banner+markup) + check_local_diagnostic_presentation(tmp_path, ranked=True) + with pytest.raises(ValueError, match='assigns ranks'): + check_local_diagnostic_presentation(tmp_path) + page.write_text('
Uncertified
'+markup) + with pytest.raises(ValueError, match='scope missing'): + check_local_diagnostic_presentation(tmp_path, ranked=True) + + +def test_unreviewed_new_calls_are_explicit_and_cannot_hide_reviewed_rows(): + from harness.web_fetch.adjudication_evidence import require_current_review_state + c = case('body', kind='body', track='web') + r = record(c) + audit = {'attempts': [], 'unreviewed_attempt_ids': [r['attempt_id']], + 'reviewed_attempts': 0, 'accepted_unreviewed_passes': 1, 'unresolved': 0} + require_current_review_state([r], audit) + audit['accepted_unreviewed_passes'] = 0 + with pytest.raises(ValueError, match='coverage differs'): + require_current_review_state([r], audit) + + +@pytest.mark.parametrize('track', ['dynamic', 'document']) +def test_contract_prose_localizes_without_full_snapshot_or_fixed_field_counts(track): + from harness.web_fetch.report_components import task_pass_guide + from harness.web_fetch.report_i18n import localize_html + from harness.web_fetch.report_check import EnglishNarrative + cases = {'example': {'id': 'example', 'name': 'Example', 'track': track, + 'acceptance_contract': {'field_minimum_records': {'shipping': 2}}}} + english = localize_html(task_pass_guide(track, cases=cases), PurePosixPath('index.html')) + parser = EnglishNarrative({'Example'}) + parser.feed(english) + assert parser.untranslated == [] + assert 'shipping: at least 2 records' in english + + +@pytest.mark.parametrize('wrapper', [ + '', + '

Uncertified

', + '', + '

']) +def test_sample_ranking_rejects_invisible_scope_disclosure(tmp_path, wrapper): + from harness.web_fetch.report_check import check_local_diagnostic_presentation + attrs = 'data-evidence-state="local_measurement" data-ranking-scope="saved-task-pass-rate"' + (tmp_path/'index.html').write_text(''+wrapper.format(attrs=attrs)) + with pytest.raises(ValueError, match='disclosure missing'): + check_local_diagnostic_presentation(tmp_path, ranked=True) + + +def test_review_display_distinguishes_fresh_reviews_from_retained_history(tmp_path): + import hashlib + import json + from harness.web_fetch.report import current_local_review_findings, review_summary + audit = {'reviewed_attempts': 100, 'expected_attempts': 100, + 'fresh_reviewed_attempts': 10, 'retained_reviewed_attempts': 90, + 'retained_reviews': {'path': 'retention.json'}, + 'supplemental_reviews': [{'path': 'supplement.json'}], + 'attempts': [{'review_origin': 'fresh', 'agent_verdict': 'fail', 'final_verdict': 'pass'}]} + (tmp_path/'result-review.json').write_text(json.dumps(audit)) + summary = BeautifulSoup(review_summary(tmp_path, highlight=False), 'html.parser') + assert '本次重新检查 10 条' in summary.get_text() + assert '90 条沿用' in summary.get_text() + assert not summary.select('.callout') + receipt = tmp_path/'replacement.json' + receipt.write_text(json.dumps({'provider_calls_added': 1, 'inherited_results': 99})) + dataset = {'comparison': {'task_replacement': {'path': receipt.name, + 'sha256': hashlib.sha256(receipt.read_bytes()).hexdigest()}}} + c = case('body', kind='body', track='web') + dataset['cases'] = [c] + findings = current_local_review_findings(tmp_path, dataset, [record(c)], audit) + text = ' '.join(x['text'] for x in findings) + assert '重新检查 10 条' in text and '90 条沿用原始复核' in text + assert '初判与程序有 1 条分歧' in text + assert '复核前已知机器汇总' not in text + assert '新增调用中 10 条' not in text + + +@pytest.mark.parametrize('defect', [None, 'unbound', 'hash', 'scope', 'count', + 'option', 'unknown_option', 'missing_option', 'type']) +def test_displayed_native_options_require_actual_configuration_and_call_scope(tmp_path, defect): + import hashlib + import json + from harness.web_fetch.report import load_frozen_method_options + config = {'providers': {'fixture': {'max_bytes': 100, 'block_private_networks': True}}} + path = tmp_path/'config.json' + path.write_text(json.dumps(config)) + document = {'schema_version': 2, 'source_config': {'path': path.name, + 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()}, + 'case_ids': ['a'], 'result_count': 1, + 'scope_title': '已记录的选项', 'scope_title_en': 'Recorded options', + 'scope_description': '仅对应这个任务。', + 'scope_description_en': 'Only for this task.', + 'providers': {'fixture': {'max_bytes': 100, 'block_private_networks': True}}} + document['provider_config_sha256'] = document['source_config']['sha256'] + if defect == 'unbound': document['schema_version'] = 1 + elif defect == 'hash': document['source_config']['sha256'] = 'x'*64 + elif defect == 'scope': document['case_ids'] = ['other'] + elif defect == 'count': document['result_count'] = 2 + elif defect == 'option': document['providers']['fixture']['max_bytes'] = 200 + elif defect == 'unknown_option': document['providers']['fixture']['api_key'] = 'not-for-display' + elif defect == 'missing_option': document['providers']['fixture']['wait_until'] = None + elif defect == 'type': document['providers']['fixture']['block_private_networks'] = 1 + (tmp_path/'frozen-method-options.json').write_text(json.dumps(document)) + rows = [{'provider': 'fixture', 'case_id': 'a'}] + if defect: + with pytest.raises(ValueError): + load_frozen_method_options(tmp_path, rows) + else: + assert load_frozen_method_options(tmp_path, rows)['result_count'] == 1 diff --git a/tests/test_local_measurement_lineage.py b/tests/test_local_measurement_lineage.py new file mode 100644 index 000000000..203067952 --- /dev/null +++ b/tests/test_local_measurement_lineage.py @@ -0,0 +1,141 @@ +"""Frozen inputs and physical calls must survive report assembly unchanged.""" +import copy +import hashlib +import json + +import pytest + +from harness.web_fetch import local_audit +from harness.web_fetch.reliability import require_reliable + + +def bind(root, name, value, *, lines=False): + path = root / name + path.parent.mkdir(parents=True, exist_ok=True) + text = ''.join(json.dumps(row) + '\n' for row in value) if lines else json.dumps(value) + path.write_text(text) + return {'path': name, 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + + +def test_unknown_verdict_can_be_shown_locally_without_becoming_publishable(): + dataset = {'comparison': {'kind': 'uniform_native_local_measurement', + 'publication_state': 'local_audit_only_not_for_release'}, + 'coverage_plan': {'web': ['p']}, 'cases': [{'id': 'a', 'track': 'web'}]} + row = {'provider': 'p', 'case_id': 'a', 'metrics': {'page_identity_required': True, + 'page_identity_status': 'unavailable', 'request_ok': True}} + reliability = local_audit.require_complete_local_inventory(dataset, [row]) + assert reliability['present_calls'] == 1 and reliability['judged_calls'] == 0 + assert not reliability['complete'] + with pytest.raises(ValueError): + require_reliable(dataset, [row]) + for records in [[], [row, row], [{**row, 'case_id': 'other'}]]: + with pytest.raises(ValueError): + local_audit.require_complete_local_inventory(dataset, records) + dataset['comparison']['publication_state'] = 'certified' + with pytest.raises(ValueError): + local_audit.require_complete_local_inventory(dataset, [row]) + + +def test_multiple_frozen_sources_preserve_order_and_reject_changes(tmp_path, monkeypatch): + monkeypatch.setattr(local_audit, 'ROOT', tmp_path) + a = bind(tmp_path, 'first.json', {'cases': [{'id': 'a'}, {'id': 'b'}]}) + b = bind(tmp_path, 'second.json', {'cases': [{'id': 'c'}]}) + contract = {'dataset_sources': [a, b]} + assert local_audit.local_measurement_cases(contract) == [{'id': 'a'}, {'id': 'b'}, {'id': 'c'}] + for selection in [['b', 'a'], ['a', 'a'], ['missing']]: + changed = copy.deepcopy(contract) + changed['dataset_sources'][0]['case_ids'] = selection + with pytest.raises(ValueError): + local_audit.local_measurement_cases(changed) + changed = copy.deepcopy(contract) + changed['dataset_sources'].append(a) + with pytest.raises(ValueError): + local_audit.local_measurement_cases(changed) + bind(tmp_path, 'second.json', {'cases': [{'id': 'changed'}]}) + with pytest.raises(ValueError): + local_audit.local_measurement_cases(contract) + + +@pytest.fixture +def execution(tmp_path, monkeypatch): + monkeypatch.setattr(local_audit, 'local_measurement_cases', lambda _: [{'id': c} for c in 'abc']) + original_config = {'providers': {'p': {'command': ['{harness_root}/scripts/oss/p.py'], 'max_bytes': 20_000_000}, 'q': {'version': 'fixed'}}} + migrated_config = copy.deepcopy(original_config) + migrated_config['providers']['p']['command'] = ['{harness_root}/harness/web_fetch/workers/p.py'] + old_config = bind(tmp_path, 'original-config.json', original_config) + new_config = bind(tmp_path, 'migrated-config.json', migrated_config) + policy = {'runs': 1, 'timeout': 45, 'delay': .5, 'provider_concurrency': 1, 'max_failure_retries': 0} + original = {'kind': 'uniform_native_local_measurement', 'providers': ['p', 'q'], 'dataset_sources': [], 'scoring_source_sha256': 'd' * 64, 'runs': 1, 'timeout_seconds': 45, 'delay': .5, 'max_failure_retries': 0, 'runtime_source_sha256': 'a' * 64, 'provider_config_sha256': old_config['sha256']} + resumed = {'kind': 'post_merge_resume_and_runtime_correction', 'parameters': policy, 'case_ids_by_provider': {'p': ['b', 'c'], 'q': ['a', 'b', 'c']}, 'runtime_source_sha256': 'b' * 64, 'provider_config_sha256': new_config['sha256']} + plans = {'original': bind(tmp_path, 'original-plan.json', original), 'resumed': bind(tmp_path, 'resumed-plan.json', resumed)} + scheduling = {'max_parallel': 2, 'max_local': 1, 'max_cloud': 1} + admitted = {'kind': 'bounded_admission_continuation', 'predecessor_plan_sha256': plans['resumed']['sha256'], 'parameters': policy, 'providers': ['p', 'q'], 'case_ids_by_provider': {'p': ['c'], 'q': ['c']}, 'runtime_source_sha256': 'c' * 64, 'provider_config_sha256': new_config['sha256'], 'scheduling': scheduling} + plans['admitted'] = bind(tmp_path, 'admitted-plan.json', admitted) + for key in plans: + plans[key]['configuration'] = old_config if key == 'original' else new_config + contract = {**original, 'execution_plans': plans} + values = {'original': original, 'resumed': resumed, 'admitted': admitted} + batches, records = [], [] + for name in ['p', 'q']: + segments, method_rows = [], [] + chunks = [('original', 'a'), ('resumed', 'b'), ('admitted', 'c')] if name == 'p' else [('resumed', 'a'), ('resumed', 'b'), ('admitted', 'c')] + for index, (key, case) in enumerate(chunks): + row = {'attempt_id': f'{name}__{case}__r001', 'provider': name, 'case_id': case, 'run': 1, 'ok': True, 'elapsed_ms': 7, 'metrics': {}} + timeout = name == 'q' and case == 'a' + if timeout: + row.update(ok=False, error_type='timeout', elapsed_ms=45_000) + method_rows.append(row) + providers = [{'name': name, 'type': 'fixture'}] + manifest = {'providers': providers, 'parameters': policy, 'runtime_source_sha256': values[key]['runtime_source_sha256'], 'provider_config_sha256': values[key]['provider_config_sha256'], 'completed_at': 'complete', 'attempt_ids': [row['attempt_id']], 'attempt_count': 1, 'unexecuted': []} + if timeout: + manifest['unexecuted'] = [{'after_attempt_id': row['attempt_id'], 'reason': 'non_cancellable_call_still_running; remaining provider cases not executed'}] + if key == 'admitted': + manifest['scheduling'] = scheduling + stem = f'{name}-{index}' + segment = {'plan_id': key, 'manifest': bind(tmp_path, stem + '-manifest.json', manifest), 'records': bind(tmp_path, stem + '-rows.jsonl', [row], lines=True), 'providers': providers, 'parameters': policy, 'attempt_ids': [row['attempt_id']], 'attempt_count': 1} + if name == 'q' and case == 'b': + continuation = {'kind': 'resume_unexecuted_after_process_exit', 'method': name, 'resumed_plan_sha256': plans[key]['sha256'], 'parameters': policy, 'provider_config_sha256': values[key]['provider_config_sha256'], 'runtime_source_sha256': values[key]['runtime_source_sha256'], 'case_ids': ['b', 'c']} + segment['continuation_plan'] = bind(tmp_path, 'continuation.json', continuation) + segments.append(segment) + records.extend(method_rows) + batches.append({'providers': [{'name': name}], 'segments': segments, 'attempt_ids': [row['attempt_id'] for row in method_rows], 'attempt_count': 3}) + return tmp_path, contract, batches, records + + +def test_retained_calls_full_method_correction_and_three_runtime_phases(execution): + local_audit._require_local_execution_segments(*execution) + + +@pytest.mark.parametrize('change', ['omit_timeout', 'alter_latency', 'repeat_method', 'alter_parameters', 'wrong_runtime', 'alter_scheduling', 'alter_product_config']) +def test_segment_assembly_rejects_protocol_or_evidence_changes(execution, change): + root, contract, batches, records = copy.deepcopy(execution) + if change == 'omit_timeout': + batches[1]['segments'].pop(0) + elif change == 'alter_latency': + records[0]['elapsed_ms'] = 1 + elif change == 'repeat_method': + batches.append(copy.deepcopy(batches[0])) + elif change == 'alter_parameters': + batches[0]['segments'][0]['parameters']['timeout'] = 90 + elif change in {'wrong_runtime', 'alter_scheduling'}: + segment = batches[0]['segments'][-1] + manifest = json.loads((root / segment['manifest']['path']).read_text()) + if change == 'wrong_runtime': + manifest['runtime_source_sha256'] = 'wrong' + else: + manifest['scheduling'] = {'max_parallel': 10} + segment['manifest'] = bind(root, 'changed-manifest.json', manifest) + else: + config = json.loads((root / contract['execution_plans']['resumed']['configuration']['path']).read_text()) + config['providers']['p']['max_bytes'] = 1 + binding = bind(root, 'changed-config.json', config) + for key in ['resumed', 'admitted']: + plan = json.loads((root / contract['execution_plans'][key]['path']).read_text()) + plan['provider_config_sha256'] = binding['sha256'] + if key == 'admitted': + plan['predecessor_plan_sha256'] = contract['execution_plans']['resumed']['sha256'] + replacement = bind(root, f'changed-{key}-plan.json', plan) + replacement['configuration'] = binding + contract['execution_plans'][key] = replacement + with pytest.raises(ValueError, match='beyond worker paths' if change == 'alter_product_config' else None): + local_audit._require_local_execution_segments(root, contract, batches, records) diff --git a/tests/test_local_revision.py b/tests/test_local_revision.py new file mode 100644 index 000000000..798099b6a --- /dev/null +++ b/tests/test_local_revision.py @@ -0,0 +1,305 @@ +import hashlib +import json + +import pytest + +from harness.web_fetch.local_revision import require_local_revision, revise_local_snapshot +from lexbench_webfetch.scoring import score_attempt +from lexbench_webfetch.metrics import task_passed + + +def test_offline_revision_preserves_calls_and_rejects_tampering(tmp_path): + raw = b'0:00\n/\nLive\nWatch full video' + (tmp_path / 'answer.txt').write_bytes(raw) + case = {'id': 'a', 'url': 'https://example.test/', 'task_kind': 'structure', + 'content_score_applicable': False, + 'acceptance_contract': {'version': 1, 'schema': 'body_text', 'minimum_records': 0, + 'stable_text_anchors': [{'id': 'live', 'alternatives': ['Live']}]}} + original = {'attempt_id': 'p__a', 'provider': 'p', 'case_id': 'a', 'ok': True, + 'artifact_path': 'answer.txt', 'elapsed_ms': 7, + 'metrics': score_attempt(case, raw, ok=True)} + assert task_passed(True, original['metrics']) + (tmp_path / 'dataset.json').write_text(json.dumps({'cases': [case], 'comparison': {}})) + (tmp_path / 'results.jsonl').write_text(json.dumps(original) + '\n') + changed = {**case, 'acceptance_contract': {**case['acceptance_contract'], + 'text_exclusions': {'text_patterns': [r'^\s*\d+:\d{2}\s*/\s*Live\s*Watch full video\b']}}} + config = tmp_path / 'config.json' + config.write_text(json.dumps({'cases': [changed], 'rule_revision': True, 'authority': {'description': 'Retain ordinary page text requirement'}, + 'regression_ids': ['ordinary-text-player-control-only', 'ordinary-text-with-player-control']})) + invalid = json.loads(config.read_text()) + invalid['rule_revision'] = False + config.write_text(json.dumps(invalid)) + with pytest.raises(ValueError, match='explicit rule revision'): + revise_local_snapshot(tmp_path, config) + invalid['rule_revision'] = True + config.write_text(json.dumps(invalid)) + receipt = revise_local_snapshot(tmp_path, config) + assert receipt['provider_calls_added'] == 0 + assert receipt['rule_revision'] is True + assert receipt['task_verdict_changes'] == [{'attempt_id': 'p__a', 'before': True, 'after': False}] + assert (tmp_path / 'answer.txt').read_bytes() == raw + rows = [json.loads((tmp_path / 'results.jsonl').read_text())] + assert rows[0]['scoring']['scored_at'] + dataset = json.loads((tmp_path / 'dataset.json').read_text()) + path = next((tmp_path / 'local-evidence/offline-revisions').glob('*/receipt.json')) + binding = {'path': str(path.relative_to(tmp_path)), 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + require_local_revision(tmp_path, binding, dataset, rows) + # A self-consistent changed result hash does not authorize changing a call. + rows[0]['elapsed_ms'] = 8 + (tmp_path / 'results.jsonl').write_text(json.dumps(rows[0]) + '\n') + receipt['output_results_sha256'] = hashlib.sha256((tmp_path / 'results.jsonl').read_bytes()).hexdigest() + path.write_text(json.dumps(receipt)) + binding['sha256'] = hashlib.sha256(path.read_bytes()).hexdigest() + with pytest.raises(ValueError, match='measured call facts'): + require_local_revision(tmp_path, binding, dataset, rows) + + +def test_contract_revision_is_explicit_without_runtime_policy_switch(tmp_path): + raw = b'Required body\nCAPTCHA' + (tmp_path/'answer.txt').write_bytes(raw) + case = {'id': 'a', 'url': 'https://example.test/', 'task_kind': 'body', + 'forbidden_any': ['CAPTCHA'], 'required_facts': [ + {'id': 'body', 'alternatives': ['Required body']}]} + original = {'attempt_id': 'p__a', 'provider': 'p', 'case_id': 'a', 'ok': True, + 'artifact_path': 'answer.txt', 'metrics': score_attempt(case, raw, ok=True)} + assert not task_passed(True, original['metrics']) + (tmp_path/'dataset.json').write_text(json.dumps({'cases': [case], 'comparison': {}})) + (tmp_path/'results.jsonl').write_text(json.dumps(original)+'\n') + changed = {k: v for k, v in case.items() if k != 'forbidden_any'} + changed['diagnostic_any'] = ['CAPTCHA'] + config = tmp_path/'config.json' + config.write_text(json.dumps({'cases': [changed], 'rule_revision': True, + 'authority': {'description': 'Explicit task revision: error words are diagnostic observations'}, + 'regression_ids': ['declared-forbidden-token-fails', 'unforbidden-captcha-documentation-passes']})) + receipt = revise_local_snapshot(tmp_path, config) + assert receipt['rule_revision'] is True + assert 'acceptance_policy_version' not in receipt + rows = [json.loads((tmp_path/'results.jsonl').read_text())] + assert 'acceptance_policy_version' not in rows[0]['metrics'] + assert task_passed(True, rows[0]['metrics']) + + +def test_same_input_keeps_distinct_revision_receipts(tmp_path): + raw = b'Required article text. CAPTCHA' + (tmp_path/'answer.txt').write_bytes(raw) + case = {'id': 'a', 'url': 'https://example.test/', 'required_facts': [ + {'id': 'body', 'alternatives': ['Required article text']}], 'forbidden_any': ['CAPTCHA']} + row = {'attempt_id': 'p__a', 'provider': 'p', 'case_id': 'a', 'ok': True, + 'artifact_path': 'answer.txt', 'metrics': score_attempt(case, raw, ok=True)} + original_dataset = json.dumps({'cases': [case], 'comparison': {}}) + original_results = json.dumps(row)+'\n' + changed = {k: v for k, v in case.items() if k != 'forbidden_any'} + changed['diagnostic_any'] = ['CAPTCHA'] + config = tmp_path/'config.json' + preserved = None + for reason in ['First declared correction', 'Second explicitly distinct revision']: + (tmp_path/'dataset.json').write_text(original_dataset) + (tmp_path/'results.jsonl').write_text(original_results) + config.write_text(json.dumps({'cases': [changed], 'rule_revision': True, + 'authority': {'description': reason}, + 'regression_ids': ['declared-forbidden-token-fails', 'unforbidden-captcha-documentation-passes']})) + receipt = revise_local_snapshot(tmp_path, config) + path = tmp_path/receipt['input_dataset']['path'] + if preserved is None: + preserved = (path.parent/'receipt.json', (path.parent/'receipt.json').read_bytes()) + assert len(list((tmp_path/'local-evidence/offline-revisions').glob('*/receipt.json'))) == 2 + assert preserved[0].read_bytes() == preserved[1] + + +def _two_revision_chain(root): + raw = b'Required body CAPTCHA' + (root/'answer.txt').write_bytes(raw) + case = {'id': 'a', 'url': 'https://example.test/', 'forbidden_any': ['CAPTCHA'], + 'required_facts': [{'id': 'body', 'alternatives': ['Required body']}]} + row = {'attempt_id': 'p__a', 'provider': 'p', 'case_id': 'a', 'ok': True, + 'artifact_path': 'answer.txt', 'elapsed_ms': 7, + 'metrics': score_attempt(case, raw, ok=True)} + (root/'dataset.json').write_text(json.dumps({'cases': [case], 'comparison': {}})) + (root/'results.jsonl').write_text(json.dumps(row)+'\n') + changed = {k: v for k, v in case.items() if k != 'forbidden_any'} + changed['diagnostic_any'] = ['CAPTCHA'] + config = root/'config.json' + parent = None + receipts = [] + for index in range(2): + if index: + changed = {**changed, 'required_facts': changed['required_facts'] + [ + {'id': 'additional', 'alternatives': ['Additional required body']}]} + config.write_text(json.dumps({'cases': [changed], 'rule_revision': True, + 'authority': {'description': 'Declared correction ' + str(index)}, + 'regression_ids': ['declared-forbidden-token-fails', + 'unforbidden-captcha-documentation-passes'], + **({'predecessor_revision': parent} if parent else {})})) + receipt = revise_local_snapshot(root, config) + path = root/receipt['input_dataset']['path'] + path = path.parent/'receipt.json' + parent = {'path': str(path.relative_to(root)), + 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + receipts.append((parent, receipt)) + return receipts + + +def test_score_revision_chain_validates_both_steps_and_reaches_original_inputs(tmp_path): + from harness.web_fetch.local_revision import validate_revision_chain + (first_binding, first), (latest_binding, latest) = _two_revision_chain(tmp_path) + assert first['task_verdict_changes'] == [ + {'attempt_id': 'p__a', 'before': False, 'after': True}] + assert latest['task_verdict_changes'] == [ + {'attempt_id': 'p__a', 'before': True, 'after': False}] + original_bytes = (tmp_path/first_binding['path']).read_bytes() + assert validate_revision_chain(tmp_path, latest_binding)[1:] == ( + first['input_dataset'], first['input_results']) + assert (tmp_path/first_binding['path']).read_bytes() == original_bytes + + +@pytest.mark.parametrize('tamper, message', [ + ('authority', 'authority'), ('regression_ids', 'regression provenance'), + ('changed_case_ids', 'case scope'), ('task_verdict_changes', 'verdict transitions'), + ('scoring_source_sha256', 'scoring source'), + ('output_results_sha256', 'output binding'), ('measured_call', 'measured call facts'), +]) +def test_hash_connected_intermediate_revision_must_pass_full_validation(tmp_path, tamper, message): + from harness.web_fetch.local_revision import validate_revision_chain + (parent_binding, parent), (latest_binding, latest) = _two_revision_chain(tmp_path) + if tamper in {'authority', 'regression_ids'}: + parent.pop(tamper) + elif tamper in {'changed_case_ids', 'task_verdict_changes'}: + parent[tamper] = [] + elif tamper == 'measured_call': + old_path = tmp_path/parent['input_results']['path'] + old = json.loads(old_path.read_text()) + old['elapsed_ms'] = 99 + old_path.write_text(json.dumps(old)+'\n') + parent['input_results']['sha256'] = hashlib.sha256(old_path.read_bytes()).hexdigest() + else: + parent[tamper] = '0'*64 + parent_path = tmp_path/parent_binding['path'] + parent_path.write_text(json.dumps(parent)) + latest['predecessor_revision']['sha256'] = hashlib.sha256(parent_path.read_bytes()).hexdigest() + latest_path = tmp_path/latest_binding['path'] + latest_path.write_text(json.dumps(latest)) + latest_binding['sha256'] = hashlib.sha256(latest_path.read_bytes()).hexdigest() + with pytest.raises(ValueError, match=message): + validate_revision_chain(tmp_path, latest_binding)[1:] + + +def _native_answer_revision_fixture(root): + raw = b'Body fact' + (root/'answer.txt').write_bytes(raw) + native_content = 'Body fact | Saved to: /Users/person/private/page.txt' + native = (json.dumps({'content': native_content})+'\n').encode() + native_path = root/'private_artifacts/native.response.bin' + native_path.parent.mkdir() + native_path.write_bytes(native) + case = {'id': 'a', 'url': 'https://example.test/', + 'required_facts': [{'id': 'body', 'alternatives': ['Body fact']}]} + row = {'attempt_id': 'p__a', 'provider': 'p', 'case_id': 'a', 'ok': True, + 'url': case['url'], 'artifact_path': 'answer.txt', 'elapsed_ms': 7, + 'diagnosis': {'category': 'obsolete diagnosis', 'evidence': 'old answer'}, + 'failure_category': 'obsolete failure', + 'provider_metadata': {'raw_response_path': 'private_artifacts/native.response.bin', + 'raw_response_sha256': hashlib.sha256(native).hexdigest(), + 'raw_response_bytes': len(native)}, + 'metrics': score_attempt(case, raw, ok=True)} + (root/'dataset.json').write_text(json.dumps({'cases': [case], 'comparison': {}})) + (root/'results.jsonl').write_text(json.dumps(row)+'\n') + config = {'cases': [], 'authority': {'description': 'Restore declared native fields from the same call; preserve the former derived answer and measured time.'}, + 'regression_ids': ['native-json-resource-declared-image-field-preserved', + 'native-json-resource-untyped-url-echo-rejected'], + 'native_answer_corrections': [{'attempt_id': row['attempt_id'], + 'native_response': {'path': row['provider_metadata']['raw_response_path'], + 'sha256': row['provider_metadata']['raw_response_sha256']}, + 'projection': {'schema_version': 1, 'content_pointer': ['content'], + 'content_format': 'text'}}]} + config_path = root/'correction.json' + config_path.write_text(json.dumps(config)) + receipt = revise_local_snapshot(root, config_path) + receipt_path = root/receipt['input_dataset']['path'] + receipt_path = receipt_path.parent/'receipt.json' + binding = {'path': str(receipt_path.relative_to(root)), + 'sha256': hashlib.sha256(receipt_path.read_bytes()).hexdigest()} + return row, receipt, receipt_path, binding, native_content + + +def test_offline_native_correction_preserves_call_and_both_answers(tmp_path): + original, receipt, _, binding, native_content = _native_answer_revision_fixture(tmp_path) + current = json.loads((tmp_path/'results.jsonl').read_text()) + assert receipt['provider_calls_added'] == 0 + assert receipt['changed_case_ids'] == [] + assert receipt['task_verdict_changes'] == [] + assert current['elapsed_ms'] == original['elapsed_ms'] + assert 'diagnosis' not in current + assert 'failure_category' not in current + assert (tmp_path/'answer.txt').read_bytes() == b'Body fact' + assert current['artifact_path'] != original['artifact_path'] + assert (tmp_path/current['artifact_path']).read_text() == native_content + assert len(receipt['native_answer_corrections']) == 1 + require_local_revision(tmp_path, binding, json.loads((tmp_path/'dataset.json').read_text()), [current]) + + +@pytest.mark.parametrize('defect,message', [ + ('clock', 'physical call facts'), ('metadata', 'unrelated response metadata'), + ('answer', 'declared original fields'), ('source', 'original response'), + ('missing_proof', 'measured call facts'), +]) +def test_native_correction_rejects_self_consistent_but_false_projection(tmp_path, defect, message): + _, receipt, path, binding, _ = _native_answer_revision_fixture(tmp_path) + current = json.loads((tmp_path/'results.jsonl').read_text()) + proof = receipt['native_answer_corrections'][0] + if defect == 'clock': current['elapsed_ms'] += 1 + elif defect == 'metadata': current['provider_metadata']['injected'] = 'new answer data' + elif defect == 'answer': + artifact = tmp_path/current['artifact_path'] + artifact.write_text('Different answer not supplied by the call') + digest = hashlib.sha256(artifact.read_bytes()).hexdigest() + current['metrics']['content_sha256'] = digest + proof['corrected_artifact']['sha256'] = digest + elif defect == 'source': + source = tmp_path/proof['native_response']['path'] + source.write_text(json.dumps({'content': 'Different native response'})) + proof['native_response']['sha256'] = hashlib.sha256(source.read_bytes()).hexdigest() + else: receipt.pop('native_answer_corrections') + (tmp_path/'results.jsonl').write_text(json.dumps(current)+'\n') + receipt['output_results_sha256'] = hashlib.sha256((tmp_path/'results.jsonl').read_bytes()).hexdigest() + path.write_text(json.dumps(receipt)) + binding['sha256'] = hashlib.sha256(path.read_bytes()).hexdigest() + with pytest.raises(ValueError, match=message): + require_local_revision(tmp_path, binding, json.loads((tmp_path/'dataset.json').read_text()), [current]) + + +@pytest.mark.parametrize('field,changed', [('title', 'forged title'), ('final_url', 'https://forged.example.test/')]) +def test_native_context_revision_replays_native_fields_and_rejects_tampering(tmp_path, field, changed): + original, _, _, _, _ = _native_answer_revision_fixture(tmp_path) + current = json.loads((tmp_path/'results.jsonl').read_text()) + native_path = tmp_path/current['provider_metadata']['raw_response_path'] + # Prepare a new immutable native acquisition fixture for this transition. + payload = json.loads(native_path.read_bytes()) + payload.update(title='native title', final_url='https://returned.example.test/') + native = json.dumps(payload).encode() + second_native = tmp_path/'private_artifacts/context.response.bin' + second_native.write_bytes(native) + current['provider_metadata'].update(raw_response_path='private_artifacts/context.response.bin', + raw_response_sha256=hashlib.sha256(native).hexdigest(), raw_response_bytes=len(native)) + current.update(title=None, final_url=current['url']) + (tmp_path/'results.jsonl').write_text(json.dumps(current)+'\n') + config = {'cases': [], 'authority': {'description': 'Recover native title and redirect without new calls.'}, + 'regression_ids': ['native-title-must-match-original', 'native-returned-title-remains-answer', + 'native-final-url-must-match-original', 'native-returned-redirect-resolves-relative-image'], + 'native_answer_corrections': [{'attempt_id': current['attempt_id'], + 'native_response': {'path': 'private_artifacts/context.response.bin', 'sha256': hashlib.sha256(native).hexdigest()}, + 'projection': {'schema_version': 2, 'content_format': 'text', 'content_pointer': ['content'], + 'result_fields': {key: {'pointer': [key]} for key in ('title', 'final_url')}}}]} + config_path = tmp_path/'context-correction.json' + config_path.write_text(json.dumps(config)) + receipt = revise_local_snapshot(tmp_path, config_path) + fixed = json.loads((tmp_path/'results.jsonl').read_text()) + assert fixed['title'] == 'native title' + assert fixed['final_url'] == 'https://returned.example.test/' + assert fixed['elapsed_ms'] == original['elapsed_ms'] + assert receipt['provider_calls_added'] == 0 + from harness.web_fetch.local_revision import _require_native_answer_correction + proof = receipt['native_answer_corrections'][0] + _require_native_answer_correction(tmp_path, current, fixed, proof) + fixed[field] = changed + with pytest.raises(ValueError, match='context correction'): + _require_native_answer_correction(tmp_path, current, fixed, proof) diff --git a/tests/test_local_task_replacement.py b/tests/test_local_task_replacement.py new file mode 100644 index 000000000..ad73dc206 --- /dev/null +++ b/tests/test_local_task_replacement.py @@ -0,0 +1,140 @@ +"""A replacement must retain old calls and measure the full new method scope.""" +import copy +import hashlib +import json + +import pytest + +from harness.web_fetch.local_revision import require_local_task_replacement + + +def setup_replacement(tmp_path): + from datetime import datetime, timezone + from harness.web_fetch.config import load_dataset + from harness.web_fetch.providers import build_provider + from harness.web_fetch.runner import run_evaluation + before = {'cases': [{'id': 'old', 'track': 'web'}, {'id': 'keep', 'track': 'web'}], + 'coverage_plan': {'web': ['a', 'b']}} + added = {'id': 'new', 'name': 'New fixture', 'track': 'web', 'category': 'fixture', + 'url': 'https://example.invalid/', 'expected_any': ['fixture']} + dataset = {**before, 'cases': [added, before['cases'][1]]} + def row(p, c): + return {'attempt_id': p+'__'+c, 'provider': p, 'case_id': c, + 'run': 1, 'metrics': {'scoring_source_sha256': 'scorer-'+c}} + previous = [row(p, c) for p in ['a', 'b'] for c in ['old', 'keep']] + def save(name, data, lines=False): + raw = (''.join(json.dumps(r)+'\n' for r in data) if lines else json.dumps(data)).encode() + (tmp_path/name).write_bytes(raw) + return {'path': name, 'sha256': hashlib.sha256(raw).hexdigest()} + input_binding = save('frozen.json', {'tool': 'web_fetch', 'cases': [added]}) + providers = save('providers.json', {'providers': {p: {'type': 'fixture'} for p in ['a', 'b']}}) + network = save('network.json', {'dns_mode': 'system', 'hosts': [], 'usable_for_guarded_local_comparison': True}) + source = save('source.html', '

fixture

') + reference = save('reference.txt', 'fixture') + def capture(stage): + observation = save(stage+'.json', {'case_id': 'new', 'url': added['url'], + 'independent': True, 'status': 200, 'sha256': source['sha256'], + 'reference_sha256': reference['sha256'], 'extracted_sha256': reference['sha256'], + 'checked_at': datetime.now(timezone.utc).isoformat()}) + return {'evidence': observation, 'source': source, 'reference': reference, 'extracted': reference} + pre = capture('pre') + _, cases = load_dataset(tmp_path/'frozen.json') + calls, batches = [], [] + for method in ['a', 'b']: + rows, _ = run_evaluation(output_dir=tmp_path, + dataset_meta={'tool': 'web_fetch', 'cases': [added]}, dataset_path=tmp_path/'frozen.json', + cases=cases, providers=[build_provider(method, {'type': 'fixture'})], + provider_config_path=tmp_path/'providers.json', runs=1, timeout=45, delay=0, max_failure_retries=0) + batch = json.loads((tmp_path/'manifest.json').read_text()) + calls.extend(rows) + batches.append(batch) + post = capture('post') + contract = {'schema_version': 1, 'kind': 'uniform_native_local_measurement', + 'providers': ['a', 'b'], 'runs': 1, 'timeout_seconds': 45, 'max_failure_retries': 0, + 'dataset_storage': 'evidence', 'dataset_sources': [input_binding], + 'provider_config_sha256': providers['sha256'], + 'runtime_source_sha256': batches[0]['runtime_source_sha256'], + 'environment': batches[0]['environment'], 'parameters': batches[0]['parameters'], + 'runner_inputs': {'dataset': input_binding, 'providers': providers, 'network': network, + 'source_observations': {'new': {'pre': pre, 'post': post}}}} + receipt = {'kind': 'local_task_replacement', 'reason': 'private DNS address', + 'removed_case_id': 'old', 'added_case_id': 'new', 'provider_calls_added': 2} + receipt['baseline_dataset'] = save('before.json', before) + receipt['baseline_results'] = save('previous.jsonl', previous, True) + receipt['cohort_dataset'] = save('cohort.json', {'cases': [added]}) + receipt['cohort_results'] = save('calls.jsonl', calls, True) + receipt['execution_contract'] = save('execution-contract.json', contract) + receipt['cohort_execution'] = save('execution.json', {'batches': batches}) + binding = save('receipt.json', receipt) + records = [r for r in previous if r['case_id'] != 'old']+calls + return binding, dataset, records + + +def test_complete_replacement_preserves_distinct_scoring_sources(tmp_path): + binding, dataset, records = setup_replacement(tmp_path) + result = require_local_task_replacement(tmp_path, binding, dataset, records) + assert result == {r['attempt_id']: r['metrics']['scoring_source_sha256'] for r in records} + + +@pytest.mark.parametrize('defect', ['missing_method', 'changed_old_result', 'changed_case', + 'excluded_new_result', 'different_family', 'changed_call']) +def test_replacement_rejects_missing_or_altered_evidence(tmp_path, defect): + binding, dataset, records = setup_replacement(tmp_path) + dataset, records = copy.deepcopy(dataset), copy.deepcopy(records) + if defect == 'missing_method': records.pop() + elif defect == 'changed_old_result': records[0]['run'] = 2 + elif defect == 'changed_case': dataset['cases'][1]['url'] = 'https://wrong.example/' + elif defect == 'excluded_new_result': records[-1]['metrics']['evaluation_excluded'] = True + elif defect == 'different_family': dataset['cases'][0]['track'] = 'dynamic' + elif defect == 'changed_call': records[-1]['elapsed_ms'] = 1 + with pytest.raises(ValueError): + require_local_task_replacement(tmp_path, binding, dataset, records) + + +@pytest.mark.parametrize('defect', [None, 'missing_history', 'extra_call', 'old_task_counted']) +def test_replacement_execution_preserves_history_without_counting_old_task(tmp_path, defect): + from collections import Counter + from harness.web_fetch.report_check import require_replacement_execution + binding, dataset, records = setup_replacement(tmp_path) + physical = Counter({'a__old': 1, 'b__old': 1, 'a__keep': 1, 'b__keep': 1, + 'a__new__r001': 1, 'b__new__r001': 1}) + logical = Counter(r['attempt_id'] for r in records) + if defect == 'missing_history': + del physical['a__old'] + elif defect == 'extra_call': + physical['a__new__r001'] += 1 + elif defect == 'old_task_counted': + logical['a__old'] = 1 + if defect: + with pytest.raises(ValueError, match='inventory differs'): + require_replacement_execution(tmp_path, binding, physical, logical) + else: + require_replacement_execution(tmp_path, binding, physical, logical) + + +@pytest.mark.parametrize('defect', ['missing_contract', 'missing_inputs', 'environment', 'timeout', + 'config', 'dns', 'source', 'late_precheck', 'output']) +def test_replacement_requires_full_physical_execution_evidence(tmp_path, defect): + binding, dataset, records = setup_replacement(tmp_path) + receipt = json.loads((tmp_path/'receipt.json').read_text()) + if defect == 'missing_contract': + receipt.pop('execution_contract') + elif defect in {'missing_inputs', 'environment', 'timeout', 'config'}: + contract = json.loads((tmp_path/'execution-contract.json').read_text()) + if defect == 'missing_inputs': contract.pop('runner_inputs') + elif defect == 'environment': contract['environment']['python'] = 'wrong' + elif defect == 'timeout': contract['timeout_seconds'] = 44 + else: contract['provider_config_sha256'] = '0'*64 + (tmp_path/'execution-contract.json').write_text(json.dumps(contract)) + receipt['execution_contract']['sha256'] = hashlib.sha256((tmp_path/'execution-contract.json').read_bytes()).hexdigest() + elif defect == 'dns': (tmp_path/'network.json').write_text('{}') + elif defect == 'source': (tmp_path/'source.html').write_text('wrong page') + elif defect == 'late_precheck': + observation = json.loads((tmp_path/'pre.json').read_text()) + observation['checked_at'] = '2099-01-01T00:00:00+00:00' + (tmp_path/'pre.json').write_text(json.dumps(observation)) + else: (tmp_path/records[-1]['artifact_path']).write_text('changed output') + (tmp_path/'receipt.json').write_text(json.dumps(receipt)) + binding['sha256'] = hashlib.sha256((tmp_path/'receipt.json').read_bytes()).hexdigest() + with pytest.raises(ValueError): + require_local_task_replacement(tmp_path, binding, dataset, records) diff --git a/tests/test_moli_provider.py b/tests/test_moli_provider.py index 9a7f8ccf9..0f5c3e2d8 100644 --- a/tests/test_moli_provider.py +++ b/tests/test_moli_provider.py @@ -201,6 +201,34 @@ def case_with(self, **changes): values.update(changes) return SimpleNamespace(**values) + def test_native_families_never_use_caller_scripts_or_conversion(self): + provider = self.provider( + "assert '--eval' not in sys.argv and '--wait-script' not in sys.argv\n" + "assert '--trace-network' not in sys.argv\n" + "if sys.argv[-1].endswith(('.pdf', '.docx')):\n" + " assert '--dump' not in sys.argv\n" + "else:\n" + " assert sys.argv[sys.argv.index('--dump') + 1] in ('markdown', 'html')\n" + "print('native output')\n", uniform_native_parameters=True) + with patch("harness.web_fetch.document_text.extract_document_text", side_effect=AssertionError("caller conversion")): + for track, kind, url, media in [("challenge", "structure", "https://example.com/", "html_page"), + ("risk_control", "body", "https://example.com/", "html_page"), + ("document", "body", "https://example.com/a.docx", "direct_docx"), + ("document", "body", "https://example.com/a.pdf", "direct_pdf"), + ("document", "body", "https://example.com/viewer.html", "html_page"), + ("image", "resource", "https://example.com/", "html_page"), + ("video", "resource", "https://example.com/", "html_page"), + ("video", "body", "https://example.com/", "html_page")]: + result = provider.fetch(self.case_with(track=track, task_kind=kind, url=url, + source_benchmark={"input_media_kind": media}), 5) + self.assertTrue(result.ok, result.error_message) + if url.endswith((".pdf", ".docx")): + self.assertEqual(result.content, "") + self.assertEqual(result.files[0]["data"].strip(), b"native output") + else: + self.assertEqual(result.content.strip(), "native output") + self.assertEqual(result.metadata["dump"], "html" if kind == "resource" else "markdown") + def test_fixed_native_cli_omits_internal_timers(self): provider = self.provider( "import sys\n" @@ -397,6 +425,11 @@ def test_native_dynamic_uses_markdown_without_generated_wait_or_eval(self): self.assertEqual(result.metadata["route"], "dynamic_native_markdown") def test_native_wait_override_applies_to_every_page_in_a_family(self): + for fixed_native in (False, True): + with self.subTest(fixed_native=fixed_native): + self._assert_family_wait_override(fixed_native) + + def _assert_family_wait_override(self, fixed_native): provider = self.provider( "import sys\n" "url = sys.argv[-1]\n" @@ -406,6 +439,7 @@ def test_native_wait_override_applies_to_every_page_in_a_family(self): "assert '--eval' not in sys.argv and '--delay-ms' not in sys.argv\n" "print('native page body')\n", wait_until_by_track={"risk_control": "domstable"}, + uniform_native_parameters=fixed_native, ) for index in range(2): with self.subTest(index=index): @@ -1086,11 +1120,11 @@ def test_video_resource_page_uses_the_same_native_markdown_route(self): ) self.assertTrue(scored["resource_pass"]) - def test_fixed_native_resource_uses_product_timers_and_native_markdown(self): + def test_fixed_native_resource_uses_product_timers_and_native_html(self): provider = self.provider( "import sys\n" "assert not {'--timeout', '--http-timeout', '--http-connect-timeout', '--eval'} & set(sys.argv)\n" - 'assert sys.argv[sys.argv.index("--dump")+1] == "markdown"\n' + 'assert sys.argv[sys.argv.index("--dump")+1] == "html"\n' 'print("![photo](https://example.com/photo.jpg)")', uniform_native_parameters=True, ) diff --git a/tests/test_native_route_contract.py b/tests/test_native_route_contract.py new file mode 100644 index 000000000..46ff3d4fb --- /dev/null +++ b/tests/test_native_route_contract.py @@ -0,0 +1,718 @@ +"""Declared input types select native routes without URL or answer inspection.""" +from types import SimpleNamespace + +import pytest + +from harness.web_fetch.native_routes import INPUT_MEDIA_KINDS, declared_input_media_kind +from harness.web_fetch.providers.moli import _route_for_case + + +@pytest.mark.parametrize('media', sorted(INPUT_MEDIA_KINDS)) +def test_dataset_and_worker_use_the_same_explicit_input_type(media): + assert declared_input_media_kind({'source_benchmark': {'input_media_kind': media}}) == media + assert declared_input_media_kind({'input_media_kind': media}) == media + + +@pytest.mark.parametrize('value', [None, '', 'guess_from_url', [], {}]) +def test_invalid_or_missing_input_type_is_not_guessed(value): + with pytest.raises(ValueError, match='declared input_media_kind'): + declared_input_media_kind({'url': 'https://example.test/actual.pdf', + 'input_media_kind': value}) + + +def test_conflicting_input_declarations_are_rejected(): + with pytest.raises(ValueError, match='Conflicting'): + declared_input_media_kind({'input_media_kind': 'html_page', + 'source_benchmark': {'input_media_kind': 'direct_pdf'}}) + + +@pytest.mark.parametrize('media,expected', [ + ('direct_pdf', 'native_document_bytes'), ('direct_docx', 'native_document_bytes'), + ('html_page', 'ordinary_markdown'), ('direct_text', 'ordinary_markdown'), +]) +def test_moli_document_selection_cannot_read_url_or_reference(media, expected): + class DeclaredDocument: + track = 'document' + task_kind = 'body' + source_benchmark = {'input_media_kind': media} + @property + def url(self): + raise AssertionError('route selection inspected the URL') + @property + def expected_resources(self): + raise AssertionError('route selection inspected expected answers') + assert _route_for_case(DeclaredDocument(), native_default=True) == expected + + +def test_fixed_native_cohort_rejects_undeclared_media_before_preflight(tmp_path): + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.runner import run_evaluation + provider = build_provider('fixture', {'type': 'fixture', 'uniform_native_parameters': True}) + case = EvalCase('case', 'Case', 'https://example.test/a.pdf', 'control', []) + with pytest.raises(ValueError, match='declared input_media_kind'): + run_evaluation(output_dir=tmp_path/'run', dataset_meta={}, dataset_path=tmp_path/'absent.json', + cases=[case], providers=[provider], provider_config_path=tmp_path/'absent-providers.json', + runs=1, timeout=45, delay=0) + assert not (tmp_path/'run').exists() + + +def test_plain_provider_use_does_not_infer_optional_media(): + assert declared_input_media_kind(SimpleNamespace(source_benchmark={}), required=False) is None + + +def _route_http_fixture(tmp_path): + from dataclasses import asdict + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.native_routes import NativeRouteLedger + from harness.web_fetch.providers.base import build_provider + provider = build_provider('tavily', {'type': 'tavily_extract', 'api_key': 'fixture', + 'uniform_native_parameters': True}) + cases = [EvalCase(name, name, 'https://example.test/'+name, 'control', [], + source_benchmark={'input_media_kind': 'html_page'}) + for name in ('one', 'two')] + ledger = NativeRouteLedger(cases, [provider]) + binding = ledger.write(tmp_path, dataset_sha256='dataset', runtime_source_sha256='runtime') + manifest = {'native_routes_required': True, 'native_route_contract': binding, + 'dataset_sha256': 'dataset', 'runtime_source_sha256': 'runtime', + 'providers': [{'name': provider.name, 'fixed_native_parameters': True}]} + return provider, cases, ledger, manifest, {'cases': [asdict(c) for c in cases]} + + +def test_actual_native_http_requests_are_locked_before_transport(tmp_path): + import json + from datetime import datetime, timezone + from unittest.mock import patch + from harness.web_fetch.native_routes import verify_native_route_contract + from harness.web_fetch.providers.http_client import HttpResponse + provider, cases, ledger, manifest, dataset = _route_http_fixture(tmp_path) + records = [] + seen = [] + def transport(endpoint, **kwargs): + assert (tmp_path/'native-route-contract.json').exists() + seen.append(kwargs['json_body']) + return HttpResponse(200, endpoint, {}, json.dumps({'results': [ + {'url': kwargs['json_body']['urls'][0], 'raw_content': 'answer'}]}).encode()) + with patch('harness.web_fetch.providers.tavily.request', side_effect=transport): + for case in cases: + started = datetime.now(timezone.utc).isoformat() + result = provider.fetch(case, 45) + records.append({'provider': provider.name, 'case_id': case.id, + 'url': case.url, 'started_at': started, 'ok': result.ok, + 'provider_metadata': {'native_route_receipt': ledger.receipt(provider, case)}}) + assert len(seen) == 2 and seen[0]['urls'] != seen[1]['urls'] + assert len(ledger.observed) == 1 + assert all('fixture' not in str(row['provider_metadata']) for row in records) + assert verify_native_route_contract(tmp_path, manifest, dataset, records) + + +@pytest.mark.parametrize('defect', ['endpoint', 'parameters', 'input_url', 'configuration']) +def test_native_request_drift_is_rejected_before_second_transport(tmp_path, defect): + from unittest.mock import Mock + provider, cases, ledger, _, _ = _route_http_fixture(tmp_path) + callback = Mock(return_value=object()) + endpoint = 'https://api.tavily.com/extract' + kwargs = {'method': 'POST', 'json_body': {'urls': [cases[0].url], 'format': 'markdown', 'extract_depth': 'advanced', 'include_images': False}, + 'headers': {'Authorization': 'Bearer fixture'}, 'timeout': 45, 'max_bytes': 20000000} + provider.native_http_request(cases[0], callback, endpoint, **kwargs) + kwargs['json_body'] = {'urls': [cases[1].url], 'format': 'markdown', 'extract_depth': 'advanced', 'include_images': False} + if defect == 'endpoint': endpoint = 'https://api.tavily.com/other' + if defect == 'parameters': kwargs['json_body']['format'] = 'html' + if defect == 'input_url': kwargs['json_body']['urls'] = ['https://wrong.test/'] + if defect == 'configuration': provider.config['include_images'] = True + with pytest.raises(ValueError, match='Native'): + provider.native_http_request(cases[1], callback, endpoint, **kwargs) + assert callback.call_count == 1 + assert ledger.errors + + +@pytest.mark.parametrize('defect', ['missing_receipt', 'signature', 'media', 'freeze', 'scope']) +def test_release_check_rejects_changed_native_route_evidence(tmp_path, defect): + from copy import deepcopy + from datetime import datetime, timezone + from unittest.mock import Mock + from harness.web_fetch.native_routes import verify_native_route_contract + provider, cases, ledger, manifest, dataset = _route_http_fixture(tmp_path) + records = [] + for case in cases: + provider.native_http_request(case, Mock(), 'https://api.tavily.com/extract', + method='POST', json_body={'urls': [case.url], 'format': 'markdown', + 'extract_depth': 'advanced', 'include_images': False}, + headers={'Authorization': 'Bearer fixture'}, timeout=45, max_bytes=20000000) + records.append({'provider': provider.name, 'case_id': case.id, 'url': case.url, + 'started_at': datetime.now(timezone.utc).isoformat(), 'ok': True, + 'provider_metadata': {'native_route_receipt': ledger.receipt(provider, case)}}) + records = deepcopy(records) + if defect == 'missing_receipt': records[0]['provider_metadata'] = {} + if defect == 'signature': records[0]['provider_metadata']['native_route_receipt']['requests'][0]['sha256'] = '0'*64 + if defect == 'media': dataset['cases'][0]['source_benchmark']['input_media_kind'] = 'direct_pdf' + if defect == 'freeze': records[0]['started_at'] = '2000-01-01T00:00:00+00:00' + if defect == 'scope': records.pop() + with pytest.raises(ValueError, match='Native'): + verify_native_route_contract(tmp_path, manifest, dataset, records) + + +def test_runner_freezes_route_matrix_before_mocked_provider_calls(tmp_path): + import json + from dataclasses import asdict + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + from harness.web_fetch.runner import run_evaluation + cases = [EvalCase(label, label, 'https://example.test/'+label, 'control', [], + source_benchmark={'input_media_kind': 'html_page'}) for label in ('one', 'two')] + dataset = {'tool': 'web_fetch', 'id': 'fixture', 'cases': [asdict(c) for c in cases]} + source = tmp_path/'dataset.json' + source.write_text(json.dumps(dataset)) + provider = build_provider('tavily', {'type': 'tavily_extract', 'api_key': 'fixture', + 'uniform_native_parameters': True}) + output = tmp_path/'run' + def call(endpoint, **kwargs): + frozen = json.loads((output/'native-route-contract.json').read_text()) + assert frozen['pre_call_frozen'] and len(frozen['cases']) == 2 + target = kwargs['json_body']['urls'][0] + return HttpResponse(200, endpoint, {}, json.dumps({'results': [ + {'url': target, 'raw_content': 'complete answer'}]}).encode()) + with patch('harness.web_fetch.providers.tavily.request', side_effect=call) as transport: + records, _ = run_evaluation(output_dir=output, dataset_meta=dataset, dataset_path=source, + cases=cases, providers=[provider], provider_config_path=tmp_path/'providers.json', + runs=1, timeout=45, delay=0, reference_policy='frozen_existing') + assert transport.call_count == 2 + assert len(records) == 2 + manifest = json.loads((output/'manifest.json').read_text()) + assert manifest['native_routes_required'] and not manifest['native_route_errors'] + assert all(r['provider_metadata']['native_route_receipt']['requests'] for r in records) + + +def test_jsonl_worker_receives_declared_media_and_records_actual_packet(tmp_path): + import sys + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.native_routes import NativeRouteLedger + from harness.web_fetch.providers.base import build_provider + script = ('import sys,json\nfor line in sys.stdin:\n' + ' r=json.loads(line);print(json.dumps({"ok":True,"content":r["input_media_kind"]}),flush=True)\n') + provider = build_provider('worker', {'type': 'subprocess_jsonl', 'command': [sys.executable, '-u', '-c', script], + 'repo_path': str(tmp_path), 'uniform_native_parameters': True}) + case = EvalCase('case', 'Case', 'https://example.test/no-extension', 'control', [], track='document', + source_benchmark={'input_media_kind': 'direct_pdf'}) + ledger = NativeRouteLedger([case], [provider], timeout_seconds=5) + ledger.write(tmp_path, dataset_sha256='fixture', runtime_source_sha256='fixture') + try: + result = provider.fetch(case, 5) + assert result.ok and result.content == 'direct_pdf' + request = ledger.receipt(provider, case)['requests'][0]['signature']['parameters']['request'] + assert request['input_media_kind'] == 'direct_pdf' + assert request['url'] == '' + assert str(tmp_path) not in str(request) + finally: + provider.cancel() + + +@pytest.mark.parametrize('method,module', [ + ('tavily_extract', 'tavily'), ('exa_contents', 'exa'), + ('firecrawl', 'firecrawl'), ('cloudflare_markdown', 'cloudflare'), + ('cloudflare_content', 'cloudflare'), +]) +@pytest.mark.parametrize('track,kind,media', [ + ('web', 'body', 'html_page'), ('document', 'body', 'direct_pdf'), + ('image', 'resource', 'html_page'), ('video', 'resource', 'html_page'), + ('file', 'file', 'direct_text'), +]) +def test_each_http_adapter_matches_its_pre_call_contract(tmp_path, method, module, track, kind, media): + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.native_routes import NativeRouteLedger + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + config = {'type': method, 'uniform_native_parameters': True, + 'api_key': 'fixture', 'api_token': 'fixture', 'account_id': 'fixture', + 'include_resource_links': True, 'rich_links_limit': 12, 'include_images': True, + 'formats': ['markdown', 'links', 'images'], 'only_main_content': False, + 'browser': 'chrome'} + provider = build_provider('method', config) + case = EvalCase('case', 'Case', 'https://example.test/no-extension', 'control', [], + track=track, task_kind=kind, source_benchmark={'input_media_kind': media}) + ledger = NativeRouteLedger([case], [provider], timeout_seconds=17) + ledger.write(tmp_path, dataset_sha256='fixture', runtime_source_sha256='fixture') + # Deliberately empty native output: this test proves transport selection, + # not retrieval success, and cannot accidentally make a live request. + with patch('harness.web_fetch.providers.'+module+'.request', + return_value=HttpResponse(200, case.url, {}, b'{}')) as transport: + provider.fetch(case, 17) + assert transport.call_count == 1 + receipt = ledger.receipt(provider, case) + assert not ledger.errors and len(receipt['requests']) == 1 + cell = next(iter(ledger.cells.values())) + assert receipt['requests'][0]['signature']['parameters'] == cell['request_parameters'] + + +def test_custom_firecrawl_without_auth_still_matches_actual_request(tmp_path): + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.native_routes import NativeRouteLedger + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + provider = build_provider('firecrawl', {'type': 'firecrawl', 'base_url': 'https://fixture.test', + 'uniform_native_parameters': True}) + case = EvalCase('case', 'Case', 'https://example.test/', 'control', [], + source_benchmark={'input_media_kind': 'html_page'}) + ledger = NativeRouteLedger([case], [provider]) + ledger.write(tmp_path, dataset_sha256='fixture', runtime_source_sha256='fixture') + with patch('harness.web_fetch.providers.firecrawl.request', + return_value=HttpResponse(200, case.url, {}, b'{}')) as transport: + provider.fetch(case, 45) + assert transport.call_count == 1 and not ledger.errors + + +def test_contract_survives_aggregation_and_complete_declared_subset(tmp_path): + import json + from copy import deepcopy + from datetime import datetime, timezone + from unittest.mock import Mock + from harness.web_fetch.native_routes import ( + materialize_native_route_contracts, verify_native_execution_manifests, + ) + source, output = tmp_path/'source', tmp_path/'aggregate' + source.mkdir() + output.mkdir() + provider, cases, ledger, manifest, dataset = _route_http_fixture(source) + records = [] + for case in cases: + provider.native_http_request(case, Mock(), 'https://api.tavily.com/extract', + method='POST', json_body={'urls': [case.url], 'format': 'markdown', + 'extract_depth': 'advanced', 'include_images': False}, + headers={'Authorization': 'Bearer fixture'}, timeout=45, max_bytes=20000000) + records.append({'attempt_id': case.id, 'provider': provider.name, 'case_id': case.id, + 'url': case.url, 'started_at': datetime.now(timezone.utc).isoformat(), 'ok': True, + 'provider_metadata': {'native_route_receipt': ledger.receipt(provider, case)}}) + manifest['attempt_ids'] = [r['attempt_id'] for r in records] + original = (source/'native-route-contract.json').read_bytes() + materialize_native_route_contracts(output, source, [manifest]) + assert (output/manifest['native_route_contract']['path']).read_bytes() == original + assert verify_native_execution_manifests(output, [manifest], dataset, records) + subset = deepcopy(manifest) + subset['attempt_ids'] = [records[0]['attempt_id']] + with pytest.raises(ValueError, match='incomplete result scope'): + verify_native_execution_manifests(output, [subset], dataset, records) + subset['projection'] = {'kind': 'complete-provider-case-subset-from-retained-run'} + assert verify_native_execution_manifests(output, [subset], dataset, records) + # Even a self-consistent forged request hash cannot change frozen parameters. + from harness.web_fetch.native_routes import _digest + bad = deepcopy(records) + item = bad[0]['provider_metadata']['native_route_receipt']['requests'][0] + item['signature']['parameters']['body']['format'] = 'html' + item['sha256'] = _digest(item['signature']) + with pytest.raises(ValueError, match='frozen interface'): + verify_native_execution_manifests(output, [manifest], dataset, bad) + contract = output/manifest['native_route_contract']['path'] + doc = json.loads(contract.read_text()) + doc['pre_call_frozen'] = False + contract.write_text(json.dumps(doc)) + with pytest.raises(ValueError, match='hash changed'): + verify_native_execution_manifests(output, [manifest], dataset, records) + with pytest.raises(ValueError, match='bytes changed'): + materialize_native_route_contracts(tmp_path/'another', output, [deepcopy(manifest)]) + + +@pytest.mark.parametrize('media,kind', [('html_page', 'body'), ('direct_pdf', 'body'), + ('html_page', 'resource'), ('direct_text', 'file')]) +def test_moli_actual_cli_obeys_declared_route_without_network(tmp_path, media, kind): + import hashlib + import sys + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.native_routes import NativeRouteLedger + from harness.web_fetch.providers.base import build_provider + binary = tmp_path/'fake-moli' + binary.write_text('#!'+sys.executable+'\nimport sys\n' + 'print("moli 1.1.4" if sys.argv[1:]==["--version"] else "answer")\n') + binary.chmod(0o700) + provider = build_provider('moli', {'type': 'moli_webfetch', 'binary': str(binary), + 'expected_reported_version': 'moli 1.1.4', + 'expected_binary_sha256': hashlib.sha256(binary.read_bytes()).hexdigest(), + 'uniform_native_parameters': True, 'block_private_networks': False}) + assert provider.validate() == [] + case = EvalCase('case', 'Case', 'https://example.test/no-extension', 'control', [], + track='document' if kind=='body' else 'image' if kind=='resource' else 'file', + task_kind=kind, source_benchmark={'input_media_kind': media}) + ledger = NativeRouteLedger([case], [provider], timeout_seconds=5) + ledger.write(tmp_path, dataset_sha256='fixture', runtime_source_sha256='fixture') + provider.fetch(case, 5) + assert not ledger.errors and len(ledger.receipt(provider, case)['requests']) == 1 + + +def test_route_verifier_preserves_optional_dataset_defaults(tmp_path): + from datetime import datetime, timezone + from unittest.mock import Mock + from harness.web_fetch.native_routes import verify_native_route_contract + provider, cases, ledger, manifest, dataset = _route_http_fixture(tmp_path) + records = [] + for case in cases: + provider.native_http_request(case, Mock(), 'https://api.tavily.com/extract', + method='POST', json_body={'urls': [case.url], 'format': 'markdown', + 'extract_depth': 'advanced', 'include_images': False}, + headers={'Authorization': 'Bearer fixture'}, timeout=45, max_bytes=20000000) + records.append({'provider': provider.name, 'case_id': case.id, 'url': case.url, + 'started_at': datetime.now(timezone.utc).isoformat(), 'ok': True, + 'provider_metadata': {'native_route_receipt': ledger.receipt(provider, case)}}) + for case in dataset['cases']: + del case['track'] + del case['task_kind'] + assert verify_native_route_contract(tmp_path, manifest, dataset, records) + + +@pytest.mark.parametrize('stage', ['before_transport', 'after_transport']) +def test_failed_native_calls_require_an_explicit_transport_stage(tmp_path, stage): + from datetime import datetime, timezone + from unittest.mock import Mock + from copy import deepcopy + from harness.web_fetch.native_routes import verify_native_route_contract + provider, cases, ledger, manifest, dataset = _route_http_fixture(tmp_path) + records = [] + for case in cases: + if stage == 'after_transport': + provider.native_http_request(case, Mock(), *provider.native_http_arguments(case, 45)[:1], + **provider.native_http_arguments(case, 45)[1]) + records.append({'provider': provider.name, 'case_id': case.id, 'url': case.url, + 'started_at': datetime.now(timezone.utc).isoformat(), 'ok': False, + 'provider_metadata': {'native_route_receipt': ledger.receipt(provider, case, ok=False)}}) + assert verify_native_route_contract(tmp_path, manifest, dataset, records) + for defect in ('unknown_stage', 'missing_failure_stage', 'missing_signature', 'wrong_stage'): + bad = deepcopy(records) + receipt = bad[0]['provider_metadata']['native_route_receipt'] + if defect == 'unknown_stage': receipt.pop('transport_state') + if defect == 'missing_failure_stage': receipt.pop('failure_stage') + if defect == 'missing_signature': + receipt['requests'] = [] + receipt['transport_state'] = 'started' + if defect == 'wrong_stage': receipt['failure_stage'] = 'unknown' + with pytest.raises(ValueError, match='Native'): + verify_native_route_contract(tmp_path, manifest, dataset, bad) + + +def test_runner_persists_pre_transport_failure_without_inventing_request(tmp_path): + import json + from dataclasses import asdict + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.models import ProviderResult + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.runner import run_evaluation + case = EvalCase('case', 'Case', 'https://example.test/', 'control', [], + source_benchmark={'input_media_kind': 'html_page'}) + dataset = {'tool': 'web_fetch', 'id': 'fixture', 'cases': [asdict(case)]} + source = tmp_path/'dataset.json' + source.write_text(json.dumps(dataset)) + provider = build_provider('tavily', {'type': 'tavily_extract', 'api_key': 'fixture', + 'uniform_native_parameters': True}) + with patch.object(provider, 'fetch', return_value=ProviderResult(False, '', error_type='configuration')): + records, _ = run_evaluation(output_dir=tmp_path/'run', dataset_meta=dataset, dataset_path=source, + cases=[case], providers=[provider], provider_config_path=tmp_path/'providers.json', + runs=1, timeout=45, delay=0, reference_policy='frozen_existing') + receipt = records[0]['provider_metadata']['native_route_receipt'] + assert receipt['failure_stage'] == 'before_transport' + assert receipt['transport_state'] == 'not_started' and receipt['requests'] == [] + + +@pytest.mark.parametrize('method,module,payload,track,kind', [ + ('tavily_extract', 'tavily', {'results': [{'raw_content': 'answer', 'images': ['https://example.test/image.png']}]}, 'image', 'resource'), + ('tavily_extract', 'tavily', {'results': [{'raw_content': 'answer'}], 'images': ['https://example.test/image.png']}, 'image', 'resource'), + ('exa_contents', 'exa', {'results': [{'text': 'answer', 'extras': {'richLinks': [ + {'anchor': 'Article', 'url': 'https://example.test/article'}], 'imageLinks': ['https://example.test/image.png']}}]}, 'image', 'resource'), + ('firecrawl', 'firecrawl', {'success': True, 'data': {'markdown': 'answer', 'images': ['https://example.test/image.png'], + 'links': ['https://example.test/article'], 'videos': ['https://example.test/video.mp4'], 'video': 'https://example.test/legacy.mp4'}}, 'video', 'resource'), + ('firecrawl', 'firecrawl', {'success': True, 'data': {'rawBase64': 'b3JpZ2luYWwgZmlsZSBieXRlcw=='}}, 'file', 'file'), + ('cloudflare_markdown', 'cloudflare', {'success': True, 'result': 'answer'}, 'web', 'body'), + ('cloudflare_content', 'cloudflare', {'success': True, 'result': '
answer
'}, 'web', 'body'), + ('cloudflare_markdown', 'cloudflare', {'success': True, 'result': [{'selector': 'img', 'results': [ + {'attributes': [{'name': 'src', 'value': '/image.png'}, {'name': 'src', 'value': '/wrong.png'}]}]}]}, 'image', 'resource'), +]) +def test_http_native_answers_survive_persistence_and_reject_tampering(tmp_path, method, module, payload, track, kind): + from pathlib import Path + import json + from copy import deepcopy + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + from harness.web_fetch.native_routes import NativeRouteLedger + from harness.web_fetch.runner import persist_result_artifacts + from harness.web_fetch.call_boundary import prepare_answer + provider = build_provider('method', {'type': method, 'api_key': 'fixture', 'api_token': 'fixture', + 'account_id': 'fixture', 'uniform_native_parameters': True, 'include_images': True, + 'include_resource_links': True, 'rich_links_limit': 12, 'formats': ['markdown', 'images', 'links']}) + case = EvalCase('case', 'Case', 'https://example.test/', 'control', [], track=track, task_kind=kind, + source_benchmark={'input_media_kind': 'html_page'}) + ledger = NativeRouteLedger([case], [provider]) + ledger.write(tmp_path, dataset_sha256='fixture', runtime_source_sha256='fixture') + with patch('harness.web_fetch.providers.'+module+'.request', + return_value=HttpResponse(200, case.url, {}, json.dumps(payload).encode())): + result = prepare_answer(provider.fetch(case, 45), case) + assert result.ok + result.metadata['native_route_receipt'] = ledger.receipt(provider, case, ok=result.ok) + altered = deepcopy(result) + altered.metadata['images'] = ['https://example.test/forged.png'] + with pytest.raises(ValueError, match='native|projection'): + persist_result_artifacts(tmp_path, Path('artifacts/forged.txt'), altered, request_url=case.url) + persist_result_artifacts(tmp_path, Path('artifacts/answer.txt'), result, request_url=case.url) + + +def test_projection_cannot_read_request_echo_outside_declared_answer_fields(): + import json + from lexbench_webfetch.answer_projection import require_native_answer_projection + raw = json.dumps({'result': 'answer', 'request': {'images': ['https://example.test/forged.png']}}).encode() + metadata = {'images': ['https://example.test/forged.png'], + 'native_route_receipt': {'answer_fields': ['result']}, + 'native_answer_projection': {'schema_version': 1, 'content_pointer': ['result'], 'content_format': 'text', + 'answer_fields': {'images': {'pointer': ['request', 'images'], 'format': 'identity'}}}} + with pytest.raises(ValueError, match='frozen answer fields'): + require_native_answer_projection('answer', metadata, raw) + + +@pytest.mark.parametrize('field,forged', [('title', 'forged title'), ('final_url', 'https://forged.example.test/')]) +def test_native_title_and_address_are_checked_at_live_and_saved_boundaries(tmp_path, field, forged): + import base64 + import hashlib + import json + from copy import deepcopy + from pathlib import Path + from harness.web_fetch.models import ProviderResult + from harness.web_fetch.runner import persist_result_artifacts + from harness.web_fetch.native_routes import verify_saved_native_answer + raw = json.dumps({'content': 'answer', 'title': 'native title', + 'final_url': 'https://returned.example.test/'}).encode() + result = ProviderResult(True, 'answer', title='native title', final_url='https://returned.example.test/', + metadata={'_raw_response_b64': base64.b64encode(raw).decode(), + 'raw_response_sha256': hashlib.sha256(raw).hexdigest(), 'raw_response_bytes': len(raw), + 'native_answer_projection': {'schema_version': 2, 'content_pointer': ['content'], + 'content_format': 'text', 'result_fields': {name: {'pointer': [name]} for name in ('title', 'final_url')}}}) + altered = deepcopy(result) + setattr(altered, field, forged) + with pytest.raises(ValueError, match='title or final URL'): + persist_result_artifacts(tmp_path, Path('artifacts/forged.txt'), altered, request_url='https://requested.example.test/') + persist_result_artifacts(tmp_path, Path('artifacts/answer.txt'), result, request_url='https://requested.example.test/') + record = {'ok': True, 'artifact_path': 'artifacts/answer.txt', 'title': result.title, + 'url': 'https://requested.example.test/', 'final_url': result.final_url, + 'provider_metadata': result.metadata, 'metrics': {'content_sha256': hashlib.sha256(b'answer').hexdigest()}} + verify_saved_native_answer(tmp_path, record) + record[field] = forged + with pytest.raises(ValueError, match='title or final URL'): + verify_saved_native_answer(tmp_path, record) + + +def test_native_url_fallback_cannot_override_a_returned_redirect(): + import json + from lexbench_webfetch.answer_projection import require_native_answer_projection + request = 'https://requested.example.test/' + returned = 'https://returned.example.test/' + metadata = {'native_answer_projection': {'schema_version': 2, 'content_format': 'empty', + 'result_fields': {'final_url': {'pointer': ['url'], 'fallback': 'request_url'}}}} + require_native_answer_projection('', metadata, b'{}', request_url=request, final_url=request) + require_native_answer_projection('', metadata, json.dumps({'url': returned}).encode(), request_url=request, final_url=returned) + with pytest.raises(ValueError, match='title or final URL'): + require_native_answer_projection('', metadata, json.dumps({'url': returned}).encode(), request_url=request, final_url=request) + + +def test_cli_final_url_replays_only_same_call_main_document_log(): + from lexbench_webfetch.answer_projection import require_native_answer_projection + request = 'https://requested.example.test/' + returned = 'https://returned.example.test/' + log = f'fetch request complete method=GET url={request} final_url={returned} status=200'.encode() + metadata = {'native_answer_projection': {'schema_version': 2, 'source_format': 'bytes', + 'content_format': 'utf8', 'result_fields': {'final_url': {'format': 'fetch_log_final_url'}}}} + require_native_answer_projection('answer', metadata, b'answer', request_url=request, final_url=returned, diagnostic=log) + with pytest.raises(ValueError, match='title or final URL'): + require_native_answer_projection('answer', metadata, b'answer', request_url=request, final_url=request, diagnostic=log) + with pytest.raises(ValueError, match='diagnostic evidence'): + require_native_answer_projection('answer', metadata, b'answer', request_url=request, final_url=returned) + + +@pytest.mark.parametrize('method,module,payload', [ + ('cloudflare_markdown', 'cloudflare', {'success': False, 'meta': {'title': 'Origin unavailable', 'finalUrl': 'https://returned.example.test/'}}), + ('cloudflare_content', 'cloudflare', {'success': False, 'meta': {'finalUrl': 'https://returned.example.test/'}}), + ('firecrawl', 'firecrawl', {'success': False, 'data': {'metadata': {'url': 'https://returned.example.test/'}}}), +]) +def test_failed_native_envelope_retains_provable_result_context(tmp_path, method, module, payload): + import json + from pathlib import Path + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + from harness.web_fetch.runner import persist_result_artifacts + provider = build_provider('method', {'type': method, 'api_key': 'fixture', 'api_token': 'fixture', + 'account_id': 'fixture', 'uniform_native_parameters': True}) + case = EvalCase('case', 'Case', 'https://requested.example.test/', 'control', []) + with patch('harness.web_fetch.providers.'+module+'.request', + return_value=HttpResponse(502, 'https://api.example.test/', {}, json.dumps(payload).encode())): + result = provider.fetch(case, 45) + assert not result.ok + assert result.final_url == 'https://returned.example.test/' + persist_result_artifacts(tmp_path, Path('artifacts/answer.txt'), result, request_url=case.url) + + +def test_resource_adapter_uses_returned_address_before_resolving_relative_attributes(tmp_path): + import json + from pathlib import Path + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + from harness.web_fetch.runner import persist_result_artifacts + provider = build_provider('method', {'type': 'cloudflare_markdown', 'api_token': 'fixture', + 'account_id': 'fixture', 'uniform_native_parameters': True}) + case = EvalCase('case', 'Case', 'https://requested.example.test/', 'control', [], track='image', task_kind='resource') + payload = {'success': True, 'meta': {'finalUrl': 'https://returned.example.test/page'}, + 'result': [{'results': [{'attributes': [{'name': 'src', 'value': '/asset.png'}]}]}]} + with patch('harness.web_fetch.providers.cloudflare.request', + return_value=HttpResponse(200, 'https://api.example.test/', {}, json.dumps(payload).encode())): + result = provider.fetch(case, 45) + assert result.final_url == 'https://returned.example.test/page' + assert result.metadata['images'] == ['https://returned.example.test/asset.png'] + persist_result_artifacts(tmp_path, Path('artifacts/answer.txt'), result, request_url=case.url) + + + +def test_empty_same_call_diagnostic_is_retained_without_inventing_final_url(tmp_path): + import base64 + import hashlib + from pathlib import Path + from harness.web_fetch.models import ProviderResult + from harness.web_fetch.runner import persist_result_artifacts + from harness.web_fetch.native_routes import verify_saved_native_answer + result = ProviderResult(True, 'answer', metadata={ + '_raw_stdout_b64': base64.b64encode(b'answer').decode(), 'raw_stdout_sha256': hashlib.sha256(b'answer').hexdigest(), + '_diagnostic_log_b64': '', 'diagnostic_log_sha256': hashlib.sha256(b'').hexdigest(), 'diagnostic_log_bytes': 0, + 'native_answer_projection': {'schema_version': 2, 'source_format': 'bytes', 'content_format': 'utf8', + 'result_fields': {'final_url': {'format': 'fetch_log_final_url'}}}}) + persist_result_artifacts(tmp_path, Path('artifacts/answer.txt'), result, request_url='https://example.test/') + record = {'ok': True, 'url': 'https://example.test/', 'title': None, 'final_url': None, + 'artifact_path': 'artifacts/answer.txt', 'provider_metadata': result.metadata, + 'metrics': {'content_sha256': hashlib.sha256(b'answer').hexdigest()}} + verify_saved_native_answer(tmp_path, record) + assert (tmp_path/result.metadata['diagnostic_log_path']).read_bytes() == b'' + + +@pytest.mark.parametrize('result_fields', [{}, {'final_url': {'pointer': ['final_url']}}]) +@pytest.mark.parametrize('discarded_timeout', [False, True]) +def test_current_writer_rejects_schema_one_result_context(tmp_path, result_fields, discarded_timeout): + import base64 + import json + from pathlib import Path + from harness.web_fetch.models import ProviderResult + from harness.web_fetch.runner import persist_result_artifacts + from lexbench_webfetch.scoring import score_attempt + content = '' if discarded_timeout else 'answer' + raw = json.dumps({'content': content}).encode() + declaration = {'schema_version': 1, 'content_pointer': ['content'], + 'content_format': 'text', 'result_fields': result_fields} + metadata = {'native_answer_projection': declaration, + '_raw_response_b64': base64.b64encode(raw).decode()} + result = ProviderResult(not discarded_timeout, content, + error_type='timeout' if discarded_timeout else None, metadata=metadata) + artifact = Path('artifacts/answer.txt') + with pytest.raises(ValueError, match='result_fields require projection schema 2'): + persist_result_artifacts(tmp_path, artifact, result, request_url='https://example.test/') + assert not (tmp_path/artifact).exists() + with pytest.raises(ValueError, match='result_fields require projection schema 2'): + score_attempt({'url': 'https://example.test/'}, content.encode(), ok=result.ok, + metadata=metadata, native_response=raw) + + +def test_saved_schema_one_keeps_original_answer_scope(tmp_path): + import base64 + import hashlib + import json + from pathlib import Path + from harness.web_fetch.models import ProviderResult + from harness.web_fetch.runner import persist_result_artifacts + from harness.web_fetch.native_routes import verify_saved_native_answer + images = [{'src': 'https://example.test/image.png'}] + raw = json.dumps({'content': 'answer', 'images': images}).encode() + declaration = {'schema_version': 1, 'content_pointer': ['content'], 'content_format': 'text', + 'answer_fields': {'images': {'pointer': ['images']}}} + result = ProviderResult(True, 'answer', metadata={'images': images, + 'native_answer_projection': declaration, '_raw_response_b64': base64.b64encode(raw).decode(), + 'raw_response_sha256': hashlib.sha256(raw).hexdigest(), 'raw_response_bytes': len(raw)}) + persist_result_artifacts(tmp_path, Path('artifacts/answer.txt'), result) + record = {'ok': True, 'artifact_path': 'artifacts/answer.txt', + 'url': 'https://requested.example.test/', 'title': 'historical unaudited title', + 'final_url': 'https://historical.example.test/', 'provider_metadata': result.metadata, + 'metrics': {'content_sha256': hashlib.sha256(b'answer').hexdigest()}} + verify_saved_native_answer(tmp_path, record) + result.metadata['images'] = [{'src': 'https://forged.example.test/image.png'}] + with pytest.raises(ValueError, match='declared native field projection'): + verify_saved_native_answer(tmp_path, record) + result.metadata['images'] = images + declaration['result_fields'] = {} + with pytest.raises(ValueError, match='result_fields require projection schema 2'): + verify_saved_native_answer(tmp_path, record) + + +@pytest.mark.parametrize('retries', [0, 1, 3]) +def test_native_http_retries_keep_each_physical_route_receipt(tmp_path, retries): + import json + from copy import deepcopy + from dataclasses import asdict + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.native_routes import verify_native_route_contract + from harness.web_fetch.providers.base import build_provider + from harness.web_fetch.providers.http_client import HttpResponse + from harness.web_fetch.runner import run_evaluation + + case = EvalCase('one', 'One', 'https://example.test/one', 'control', ['Expected'], + source_benchmark={'input_media_kind': 'html_page'}) + dataset = {'id': 'retry', 'cases': [asdict(case)]} + source = tmp_path/'dataset.json' + source.write_text(json.dumps(dataset)) + provider = build_provider('tavily', {'type': 'tavily_extract', 'api_key': 'fixture', + 'uniform_native_parameters': True}) + calls = [] + + def transport(endpoint, **kwargs): + calls.append(kwargs) + content = 'Expected' if len(calls) == retries + 1 else 'incomplete' + return HttpResponse(200, endpoint, {}, json.dumps({'results': [ + {'url': case.url, 'raw_content': content}]}).encode()) + + output = tmp_path/'run' + with patch('harness.web_fetch.providers.tavily.request', side_effect=transport): + rows, _ = run_evaluation(output_dir=output, dataset_meta=dataset, dataset_path=source, + cases=[case], providers=[provider], provider_config_path=tmp_path/'providers.json', + runs=1, timeout=45, delay=0, reference_policy='frozen_existing', + max_failure_retries=retries) + assert len(calls) == retries + 1 + assert len(rows) == 1 and rows[0]['metrics']['content_pass'] + manifest = json.loads((output/'manifest.json').read_text()) + assert verify_native_route_contract(output, manifest, dataset, rows) + if not retries: + return + attempts = rows[0]['provider_metadata']['retry']['attempts'] + assert [a['native_route_receipt']['physical_attempt'] for a in attempts] == list(range(1, retries + 2)) + for defect in ('missing', 'signature', 'ordinal', 'final'): + broken = deepcopy(rows) + physical = broken[0]['provider_metadata']['retry']['attempts'][0] + if defect == 'missing': + physical.pop('native_route_receipt') + elif defect == 'signature': + physical['native_route_receipt']['requests'][0]['sha256'] = '0'*64 + elif defect == 'ordinal': + physical['native_route_receipt']['physical_attempt'] = 2 + else: + broken[0]['provider_metadata']['native_route_receipt']['physical_attempt'] = 99 + with pytest.raises(ValueError, match='Native'): + verify_native_route_contract(output, manifest, dataset, broken) + + +def test_native_retry_does_not_allow_two_requests_in_one_physical_call(tmp_path): + from unittest.mock import Mock + provider, cases, ledger, _, _ = _route_http_fixture(tmp_path) + case = cases[0] + ledger.begin_attempt(provider, case, 1) + endpoint, arguments = provider.native_http_arguments(case, 45) + transport = Mock() + provider.native_http_request(case, transport, endpoint, **arguments) + with pytest.raises(ValueError, match='one adapter invocation per physical attempt'): + provider.native_http_request(case, transport, endpoint, **arguments) + assert transport.call_count == 1 diff --git a/tests/test_network_preflight.py b/tests/test_network_preflight.py index f9a2b1592..9252791ae 100644 --- a/tests/test_network_preflight.py +++ b/tests/test_network_preflight.py @@ -4,6 +4,19 @@ class NetworkPreflightTests(unittest.TestCase): + def test_non_public_dns_is_not_a_fair_public_comparison(self): + for addresses,accepted in ((['43.162.217.19'],True),(['2606:4700:4700::1111'],True), + (['10.2.0.11'],False),(['127.0.0.1'],False),(['169.254.1.1'],False), + (['fd00::1'],False),(['43.162.217.19','10.2.0.11'],False)): + with self.subTest(addresses=addresses): + def resolve(host,port,**kwargs): + return [(socket.AF_INET6 if ':' in a else socket.AF_INET,socket.SOCK_STREAM,6,'',(a,port)) for a in addresses] + audit=inspect_system_dns(['https://public.example/page'],resolver=resolve) + self.assertEqual(audit['usable_for_guarded_local_comparison'],accepted) + if accepted:require_usable_public_dns(audit) + else: + with self.assertRaisesRegex(ValueError,'Do not disable'): + require_usable_public_dns(audit) def test_synthetic_dns_fails_before_calls_but_public_dns_passes(self): for address, accepted in [ ('198.19.93.254', False), @@ -59,6 +72,31 @@ def fetch(self, case, timeout): self.assertFalse(output.exists()) + def test_runner_rejects_non_public_dns_before_calls_and_artifacts(self): + from pathlib import Path + from tempfile import TemporaryDirectory + from unittest.mock import patch + from lexbench_webfetch.models import EvalCase + from harness.web_fetch.providers.base import Provider + from harness.web_fetch.runner import run_evaluation + calls = [] + class GuardedFixture(Provider): + def fetch(self, case, timeout): + calls.append(case.id) + raise AssertionError("No provider calls allowed in invalid DNS environment") + provider = GuardedFixture('guarded', {'type': 'openclaw_builtin'}) + fake = [(socket.AF_INET, socket.SOCK_STREAM, 6, '', ('10.2.0.11', 443))] + with TemporaryDirectory() as directory, patch('socket.getaddrinfo', return_value=fake): + output = Path(directory) / 'measurement' + with self.assertRaisesRegex(ValueError, 'non-public'): + run_evaluation(output_dir=output, dataset_meta={}, dataset_path=Path('fixture.json'), + cases=[EvalCase('control', 'control', 'https://example.com/', 'control', [])], + providers=[provider], provider_config_path=Path('fixture-providers.json'), + runs=1, timeout=45, delay=0) + self.assertEqual(calls, []) + self.assertFalse(output.exists()) + + def test_public_dns_mode_uses_real_resolver_and_records_mode(self): from types import SimpleNamespace from unittest.mock import patch diff --git a/tests/test_provider_evidence_boundary.py b/tests/test_provider_evidence_boundary.py index 9695fa9bf..508d34e71 100644 --- a/tests/test_provider_evidence_boundary.py +++ b/tests/test_provider_evidence_boundary.py @@ -165,15 +165,12 @@ class Fixture(Provider): def fetch(self, case, timeout): self.calls[case.id] = self.calls.get(case.id, 0) + 1 - private = b'{"synthetic_private_response":true}' - return ProviderResult( - self.calls[case.id] == 2, - 'Expected fixture' if self.calls[case.id] == 2 else '', - error_type=None if self.calls[case.id] == 2 else 'network', - error_message=( - None if self.calls[case.id] == 2 else 'synthetic transient failure' - ), - metadata={'_rendered_html': '
fixture
', + answer = 'Expected fixture' if self.calls[case.id] == 2 else 'Wrong' + private = json.dumps({'synthetic_private_response': True, 'content': answer}).encode() + return ProviderResult(True, 'Expected fixture' if self.calls[case.id] == 2 else 'Wrong', + metadata={'native_answer_projection': {'schema_version': 1, + 'content_pointer': ['content'], 'content_format': 'text'}, + '_rendered_html': '
fixture
', 'rendered_html_sha256': hashlib.sha256(b'
fixture
').hexdigest(), '_raw_stdout_b64': base64.b64encode(b'fixture stdout').decode(), 'raw_stdout_sha256': hashlib.sha256(b'fixture stdout').hexdigest(), diff --git a/tests/test_providers.py b/tests/test_providers.py index f5af70122..56c708eff 100644 --- a/tests/test_providers.py +++ b/tests/test_providers.py @@ -7,14 +7,15 @@ from pathlib import Path from unittest.mock import patch -from lexbench_webfetch.metrics import evaluate_resources +from lexbench_webfetch.metrics import evaluate_resources, task_passed, task_measured from lexbench_webfetch.models import EvalCase +from harness.web_fetch.models import ProviderResult from harness.web_fetch.providers.cloudflare import ( CloudflareContentProvider, CloudflareMarkdownProvider, _post_multipart, ) -from harness.web_fetch.providers.cowagent import CowAgentBuiltinProvider, _sanitize_public_content +from harness.web_fetch.providers.cowagent import CowAgentBuiltinProvider from harness.web_fetch.providers.exa import ExaContentsProvider from harness.web_fetch.providers.firecrawl import FirecrawlProvider from harness.web_fetch.providers.http import HttpProvider @@ -23,9 +24,36 @@ from harness.web_fetch.providers.tavily import TavilyExtractProvider from harness.web_fetch.publication_privacy import project_file from harness.web_fetch.runner import run_evaluation +from lexbench_webfetch.scoring import score_attempt class ProviderTest(unittest.TestCase): + @patch('harness.web_fetch.providers.firecrawl.request') + def test_native_final_url_controls_identity(self, request): + case=EvalCase(id='redirect-control',name='Redirect control',url='https://example.test/start', + category='static',expected_any=['Expected body'],track='risk_control',page_identity_required=True, + acceptance_contract={'version':1,'schema':'body_text','minimum_records':0, + 'stable_text_anchors':[{'id':'body','alternatives':['Expected body']}]}, + source_benchmark={'risk_control':{'navigation_contract':{ + 'requested_url':'https://example.test/start','observed_final_url':'https://example.test/final'}}}) + for native_url, passed, measured, identity in ( + ('https://example.test/final',True,True,'verified'), + ('https://example.test/other',False,True,'mismatch'), + (None,False,False,'unavailable'), + ): + with self.subTest(native_url=native_url): + metadata={'sourceURL':case.url,'statusCode':200} + if native_url is not None: metadata['url']=native_url + request.return_value=HttpResponse(200,'https://api.firecrawl.dev/v2/scrape',{}, + json.dumps({'success':True,'data':{'markdown':'Expected body','metadata':metadata}}).encode()) + result=FirecrawlProvider('firecrawl',{'api_key':'synthetic','uniform_native_parameters':True}).fetch(case,45) + self.assertEqual(result.final_url,native_url) + m=score_attempt(case,result.content.encode(),ok=result.ok,metadata=result.metadata, + status_code=result.status_code,final_url=result.final_url) + self.assertEqual(m['page_identity_status'],identity) + self.assertEqual(task_passed(result.ok,m),passed) + self.assertEqual(task_measured(m),measured) + @patch("harness.web_fetch.providers.firecrawl.request") def test_firecrawl_native_page_scope_is_explicit_and_preserves_answer(self, mock_request): returned = "Header\n\nMain text\n\nFooter" @@ -1115,9 +1143,9 @@ def test_cloudflare_video_resource_scrape_filters_ordinary_links( ).fetch(video, 45) self.assertEqual(result.metadata["videos"], [ - "https://example.com/watch/clip.webm" + "https://example.com/about", "https://example.com/watch/clip.webm" ]) - self.assertNotIn("/about", result.content) + self.assertEqual(json.loads(result.content), mock_request.return_value.json()['result']) @patch("harness.web_fetch.providers.cloudflare.request") def test_cloudflare_video_resource_keeps_packaged_video_downloads( @@ -1156,11 +1184,11 @@ def test_cloudflare_video_resource_keeps_packaged_video_downloads( self.assertTrue(result.ok) self.assertEqual(result.metadata["videos"], [ - "https://example.com/movies/movie.mp4.zip", + "https://example.com/", "https://example.com/movies/movie.mp4.zip", "https://example.com/movies/movie.webm.zip?mirror=1", + "https://example.com/movies/notes.zip", ]) - self.assertNotIn("notes.zip", result.content) - self.assertNotIn("\nhttps://example.com/\n", "\n" + result.content + "\n") + self.assertEqual(json.loads(result.content), mock_request.return_value.json()['result']) @patch("harness.web_fetch.providers.cloudflare.request") def test_cloudflare_video_link_task_keeps_page_href_without_media_suffix( @@ -1206,11 +1234,10 @@ def test_cloudflare_video_link_task_keeps_page_href_without_media_suffix( self.assertTrue(result.ok) self.assertEqual(result.metadata["videos"], [ + "https://example.com/about", "https://example.com/short/iframe.html", "https://example.com/video/BV1234567890", ]) - self.assertIn("https://example.com/video/BV1234567890", result.content) - self.assertNotIn("https://example.com/about", result.content) - self.assertNotIn("https://example.com/short/iframe.html", result.content) + self.assertEqual(json.loads(result.content), mock_request.return_value.json()['result']) self.assertEqual(mock_request.call_args.kwargs["json_body"]["elements"], [{"selector": "a[href]"}]) @@ -1284,7 +1311,7 @@ def test_cloudflare_failed_document_download_does_not_fallback_or_convert( @patch("harness.web_fetch.providers.cloudflare.request") def test_cloudflare_identity_uses_observed_destination_only(self, mock_request): provider = CloudflareMarkdownProvider("cf", {"api_token": "x", "account_id": "a"}) - for meta, expected in (({}, None), ({"finalUrl": "https://example.com/final"}, + for meta, expected in (({}, self.case.url), ({"finalUrl": "https://example.com/final"}, "https://example.com/final")): with self.subTest(meta=meta): body = json.dumps({"success": True, "result": "Body", "meta": meta}).encode() @@ -1450,15 +1477,18 @@ def test_cowagent_cannot_omit_declared_identity(self) -> None: self.assertIn("expected_repo_revision", errors) self.assertIn("required runtime paths", errors) - def test_cowagent_removes_workspace_path_but_keeps_document_text(self) -> None: - content = ( - "[Document: guide.pdf | Size: 10KB | Saved to: " - "/Users/person/.browser-eval/runs/work/tmp/guide.pdf]\n\nBody" - ) - self.assertEqual( - _sanitize_public_content(content), - "[Document: guide.pdf | Size: 10KB]\n\nBody", - ) + @patch('harness.web_fetch.providers.subprocess_jsonl.SubprocessJsonlProvider.fetch') + def test_cowagent_preserves_native_answer_and_error_before_scoring(self, fetch): + content = ('[Document: guide.pdf | Size: 10KB | Saved to: ' + '/Users/person/work/guide.pdf]\n\nBody') + native = ProviderResult(False, content, error_message=content, + metadata={'raw_response_sha256': 'native-envelope'}) + fetch.return_value = native + result = CowAgentBuiltinProvider('cowagent', {'repo_path': '.'}).fetch(self.case, 1) + self.assertIs(result, native) + self.assertEqual(result.content, content) + self.assertEqual(result.error_message, content) + self.assertEqual(result.metadata, {'raw_response_sha256': 'native-envelope'}) if __name__ == "__main__": diff --git a/tests/test_public_call_logs.py b/tests/test_public_call_logs.py index aaf755d16..e3df72116 100644 --- a/tests/test_public_call_logs.py +++ b/tests/test_public_call_logs.py @@ -65,10 +65,12 @@ def test_common_sensitive_fields_are_redacted_from_public_log(self): self.assertIn('view=full', serialized) def test_success_has_closed_complete_hash_bound_log(self): - raw = b'{"private":"Bearer never-publish"}' + raw = b'{"content":"Expected","final_url":"https://example.com/final#fragment","private":"Bearer never-publish"}' result = ProviderResult( True, "Expected", status_code=200, final_url="https://example.com/final#fragment", metadata={ + "native_answer_projection": {'schema_version': 2, 'content_format': 'text', + 'content_pointer': ['content'], 'result_fields': {'final_url': {'pointer': ['final_url']}}}, "route": "advanced_extract", "content_type": "text/markdown", "request_id": "request-123", "exit_code": 0, "request_method": "POST", diff --git a/tests/test_public_evidence_presentation.py b/tests/test_public_evidence_presentation.py new file mode 100644 index 000000000..128af5910 --- /dev/null +++ b/tests/test_public_evidence_presentation.py @@ -0,0 +1,35 @@ +import pytest +from harness.web_fetch.report_check import require_public_evidence_page + + +@pytest.fixture +def specimen(): + answer = {'original_sha256': 'a'*64, 'published_sha256': 'b'*64, + 'path': 'public-evidence/answer.txt', 'receipt': 'public-evidence/answer-receipt.json'} + native = {'path': 'public-evidence/native.json', 'receipt': 'public-evidence/native-receipt.json'} + return {'scored_answer': answer, 'native_response': native} + + +def page(views, prefix='../'): + answer = views['scored_answer'] + links = ''.join('Public download' + for item in views.values() for key,v in item.items() if key in {'path','receipt'}) + return '

'+answer['original_sha256']+' '+answer['published_sha256']+'

'+links+'' + + +@pytest.mark.parametrize('relative,prefix', [('attempts/p.html','../'),('en/attempts/p.html','../../')]) +def test_public_evidence_links_work_in_both_editions(specimen,relative,prefix): + require_public_evidence_page(page(specimen,prefix),relative,specimen) + + +@pytest.mark.parametrize('defect', ['original_hash','public_hash','native_link','receipt_link','escaped_markup']) +def test_release_rejects_missing_identity_or_download_evidence(specimen,defect): + html = page(specimen) + if defect == 'original_hash': html = html.replace('a'*64, 'wrong original') + if defect == 'public_hash': html = html.replace('b'*64, 'wrong public') + if defect == 'native_link': html = html.replace('../public-evidence/native.json','../unrelated.json') + if defect == 'receipt_link': html = html.replace('../public-evidence/answer-receipt.json','../unrelated.json') + if defect == 'escaped_markup': html = html.replace('Public download','<span data-report-en="Download">Download</span>',1) + # Payload text cannot stand in for a working reader-facing evidence link. + html += '
'+('a'*64)+' '+('b'*64)+' public-evidence/native.json
' + with pytest.raises(ValueError):require_public_evidence_page(html,'attempts/p.html',specimen) diff --git a/tests/test_public_evidence_views.py b/tests/test_public_evidence_views.py new file mode 100644 index 000000000..82247175b --- /dev/null +++ b/tests/test_public_evidence_views.py @@ -0,0 +1,426 @@ +import hashlib +import json + +import pytest + +from harness.web_fetch.publication_privacy import project_evidence_view +from lexbench_webfetch.answer_projection import project_native_answer + + +def test_native_envelope_view_redacts_decoded_paths_without_changing_other_fields(): + original = json.dumps({'content': 'Article\nSaved to: /Users/example/private/run/body.txt\nDone', + 'elapsed_ms': 1234.5, 'count': 7, 'success': True}, ensure_ascii=True).encode() + declaration = {'schema_version': 1, 'content_pointer': ['content'], 'content_format': 'text'} + scored, _ = project_native_answer(json.loads(original), declaration) + public, receipt = project_evidence_view('public-evidence/native/response.json', original, + role='native_response') + payload = json.loads(public) + assert payload['elapsed_ms'] == 1234.5 and payload['count'] == 7 and payload['success'] is True + assert payload['content'] == 'Article\nSaved to: [HOST_PATH_OMITTED]\nDone' + assert 'private/run' not in public.decode() + assert '/Users/example' in scored + assert receipt['original_sha256'] == hashlib.sha256(original).hexdigest() + assert receipt['published_sha256'] == hashlib.sha256(public).hexdigest() + assert receipt['original_sha256'] != receipt['published_sha256'] + assert receipt['provider_calls_added'] == 0 + + +def test_answer_and_envelope_public_views_preserve_native_field_projection(): + raw = json.dumps({'body': 'Answer\nSaved to: /Users/example/private/file.md\nEnd'}).encode() + declaration = {'schema_version': 1, 'content_pointer': ['body'], 'content_format': 'text'} + answer, _ = project_native_answer(json.loads(raw), declaration) + public_response, _ = project_evidence_view('public-evidence/response.json', raw, role='native_response') + public_answer, receipt = project_evidence_view('public-evidence/answer.txt', answer.encode(), role='scored_answer') + assert project_native_answer(json.loads(public_response), declaration)[0].encode() == public_answer + assert receipt['role'] == 'scored_answer' + assert receipt['original_sha256'] == hashlib.sha256(answer.encode()).hexdigest() + + +def test_view_handles_json_escaped_paths_and_retains_clean_original_bytes(): + raw = br'{"content":"failed at \u002fUsers\u002fexample\u002fprivate\u002frun.txt","count":4}' + public, _ = project_evidence_view('public-evidence/response.json', raw, role='native_response') + assert json.loads(public)['content'] == 'failed at [HOST_PATH_OMITTED]' + clean = b'{ "count": 4, "content": "Normal body" }\n' + assert project_evidence_view('public-evidence/response.json', clean, role='native_response')[0] == clean + + +def test_view_hides_explicit_temp_paths_and_credential_shapes(): + raw = b'Failed at /private/tmp/example/session.log\nkey: ghp_' + b'X'*32 + public, receipt = project_evidence_view('public-evidence/answer.txt', raw, role='scored_answer', + host_prefixes=['/private/tmp']) + assert b'/private/tmp' not in public and b'example/session' not in public + assert b'ghp_' not in public + assert receipt['redactions']['credential_shape'] == 1 + + +@pytest.mark.parametrize('name', ['private_artifacts/response.bin', '../response.txt', + '/public-evidence/response.txt', 'public-evidence/.env']) +def test_view_rejects_paths_that_could_confuse_original_and_public_evidence(name): + with pytest.raises(ValueError, match='separate safe'): + project_evidence_view(name, b'answer', role='scored_answer') + + +def test_view_does_not_accept_unknown_roles_or_binary_guessing(): + with pytest.raises(ValueError, match='role'): + project_evidence_view('public-evidence/a.txt', b'a', role='reference') + with pytest.raises(UnicodeDecodeError): + project_evidence_view('public-evidence/a.txt', b'\xff\xfe', role='native_response') + + +def test_redaction_cannot_merge_distinct_native_metadata_fields(): + raw = json.dumps({'/Users/one/a': 1, '/Users/two/b': 2}).encode() + with pytest.raises(ValueError, match='merge distinct'): + project_evidence_view('public-evidence/response.json', raw, role='native_response') + + +def test_saved_public_view_verifier_rejects_changed_bytes_or_false_identity(tmp_path): + from harness.web_fetch.publication_privacy import require_public_evidence_view + name = 'public-evidence/answer.txt' + public, receipt = project_evidence_view(name, b'Answer /Users/example/private/a.txt', role='scored_answer') + path = tmp_path/name + path.parent.mkdir() + path.write_bytes(public) + proof = tmp_path/'view-receipt.json' + def binding(): + proof.write_text(json.dumps(receipt)) + return {'path': proof.name, 'sha256': hashlib.sha256(proof.read_bytes()).hexdigest()} + assert require_public_evidence_view(tmp_path, binding()) == receipt + path.write_bytes(b'different') + with pytest.raises(ValueError, match='binding changed'): + require_public_evidence_view(tmp_path, binding()) + path.write_bytes(public) + receipt['original_sha256'] = receipt['published_sha256'] + with pytest.raises(ValueError, match='identity does not match'): + require_public_evidence_view(tmp_path, binding()) + + +def test_public_view_verifier_rejects_symlink_even_with_matching_bytes(tmp_path): + from harness.web_fetch.publication_privacy import require_public_evidence_view + public, receipt = project_evidence_view('public-evidence/a.txt', b'answer', role='scored_answer') + (tmp_path/'public-evidence').mkdir() + (tmp_path/'original.txt').write_bytes(public) + (tmp_path/'public-evidence/a.txt').symlink_to(tmp_path/'original.txt') + raw = json.dumps(receipt).encode() + (tmp_path/'receipt.json').write_bytes(raw) + with pytest.raises(ValueError): + require_public_evidence_view(tmp_path, {'path': 'receipt.json', 'sha256': hashlib.sha256(raw).hexdigest()}) + + +def _declared_view_snapshot(tmp_path): + original = b'Beginning\nSaved to: /Users/example/private/result.txt\nStable ending' + public_name = 'public-evidence/answer.txt' + public, receipt = project_evidence_view(public_name, original, role='scored_answer') + (tmp_path/'public-evidence').mkdir() + (tmp_path/public_name).write_bytes(public) + receipt_raw = json.dumps(receipt).encode() + (tmp_path/'receipt.json').write_bytes(receipt_raw) + original_binding = {'path': 'artifacts/method/case/run.txt', + 'sha256': hashlib.sha256(original).hexdigest()} + alias = tmp_path/original_binding['path'] + alias.parent.mkdir(parents=True, exist_ok=True) + alias.write_bytes(public) + index = {'kind': 'post_score_public_evidence_view_inventory', 'provider_calls_added': 0, + 'files': [{'original_path': original_binding['path'], + 'original_sha256': original_binding['sha256'], + 'view_receipt': {'path': 'receipt.json', 'sha256': hashlib.sha256(receipt_raw).hexdigest()}}]} + index_raw = json.dumps(index).encode() + (tmp_path/'public-views.json').write_bytes(index_raw) + profile = {'publication_evidence_views': {'path': 'public-views.json', + 'sha256': hashlib.sha256(index_raw).hexdigest()}} + (tmp_path/'presentation-binding.json').write_text(json.dumps(profile)) + return original_binding, original, public, index, profile + + +def test_bound_view_requires_explicit_profile_and_preserves_original_identity(tmp_path): + from harness.web_fetch.publication_privacy import read_evidence_binding, original_evidence_size + binding, original, public, _, _ = _declared_view_snapshot(tmp_path) + assert read_evidence_binding(tmp_path, binding).read_bytes() == public + assert original_evidence_size(tmp_path, binding) == len(original) + assert len(public) != len(original) + (tmp_path/'presentation-binding.json').unlink() + with pytest.raises(ValueError, match='binding changed'): + read_evidence_binding(tmp_path, binding) + + +@pytest.mark.parametrize('defect', ['inventory_hash', 'wrong_original', 'metadata_alias', 'duplicate']) +def test_bound_view_rejects_changed_inventory_and_unrelated_authority(tmp_path, defect): + from harness.web_fetch.publication_privacy import read_evidence_binding + binding, _, _, index, profile = _declared_view_snapshot(tmp_path) + if defect == 'inventory_hash': + (tmp_path/'public-views.json').write_text('{}') + else: + if defect == 'wrong_original': binding['sha256'] = '0'*64 + if defect == 'metadata_alias': + binding['path'] = 'dataset.json' + index['files'][0]['original_path'] = binding['path'] + if defect == 'duplicate': index['files'].append(index['files'][0].copy()) + raw = json.dumps(index).encode() + (tmp_path/'public-views.json').write_bytes(raw) + profile['publication_evidence_views']['sha256'] = hashlib.sha256(raw).hexdigest() + (tmp_path/'presentation-binding.json').write_text(json.dumps(profile)) + with pytest.raises(ValueError): + read_evidence_binding(tmp_path, binding) + + +def test_public_excerpt_keeps_original_coordinates_without_claiming_reconstruction(tmp_path): + from harness.web_fetch.publication_privacy import evidence_excerpt_matches + binding, original, public, _, _ = _declared_view_snapshot(tmp_path) + excerpt = 'Stable ending' + start = original.decode().index(excerpt) + sample = {'start': start, 'end': start+len(excerpt), 'text': excerpt} + assert evidence_excerpt_matches(tmp_path, binding, sample, public.decode()) + assert public.decode()[start:start+len(excerpt)] != excerpt + assert not evidence_excerpt_matches(tmp_path, binding, {**sample, 'text': 'invented evidence'}, public.decode()) + (tmp_path/'presentation-binding.json').unlink() + assert not evidence_excerpt_matches(tmp_path, binding, sample, public.decode()) + + +def test_native_custom_secret_fields_are_redacted_structurally(): + raw = json.dumps({'content': 'Stable answer', 'headers': {'authorization': 'Custom opaque-value'}, + 'meta': {'access_token': 'opaque-session', 'api_key': ''}, + 'token_count': 28}).encode() + public, receipt = project_evidence_view('public-evidence/response.json', raw, role='native_response') + payload = json.loads(public) + assert payload['headers']['authorization'] == '[REDACTED]' + assert payload['meta']['access_token'] == '[REDACTED]' + assert payload['meta']['api_key'] == '' + assert payload['content'] == 'Stable answer' and payload['token_count'] == 28 + assert receipt['redactions']['sensitive_values'] == 2 + + +def test_text_projection_keeps_native_json_looking_body_format(): + body = '{\n "path": "/Users/example/private/result.txt",\n "value": 3\n}' + raw = json.dumps({'content': body}).encode() + declaration = {'schema_version': 1, 'content_pointer': ['content'], 'content_format': 'text'} + public_native, _ = project_evidence_view('public-evidence/response.json', raw, role='native_response') + public_answer, receipt = project_evidence_view('public-evidence/answer.txt', body.encode(), + role='scored_answer', content_format='text') + assert project_native_answer(json.loads(public_native), declaration)[0].encode() == public_answer + assert receipt['encoding'] == 'utf-8-text' + assert b'\n "value": 3\n' in public_answer + + +def test_release_inventory_requires_complete_current_evidence(tmp_path): + from harness.web_fetch.publication_privacy import require_public_evidence_view_inventory + original, _, _, index, profile = _declared_view_snapshot(tmp_path) + records = [{'artifact_path': original['path'], 'metrics': {'content_sha256': original['sha256']}}] + assert require_public_evidence_view_inventory(tmp_path, profile['publication_evidence_views'], records) + records[0]['provider_metadata'] = {'raw_response_path': 'private_artifacts/response.bin', + 'raw_response_sha256': '1'*64} + with pytest.raises(ValueError, match='omits current evidence'): + require_public_evidence_view_inventory(tmp_path, profile['publication_evidence_views'], records) + + +def test_empty_native_token_is_not_a_secret_but_nonempty_value_is_masked(): + from harness.web_fetch.publication_privacy import publication_risk + empty = b'{"metadata":{"csrf-token":""},"content":"Stable body"}' + assert publication_risk(empty) is None + assert project_evidence_view('public-evidence/response.json', empty, role='native_response')[0] == empty + nonempty = b'{"metadata":{"csrf-token":"opaque-private-value"},"content":"Stable body"}' + assert publication_risk(nonempty) == 'sensitive field value' + public, _ = project_evidence_view('public-evidence/response.json', nonempty, role='native_response') + assert json.loads(public)['metadata']['csrf-token'] == '[REDACTED]' + assert publication_risk(public) is None + + +def test_serialized_header_name_has_no_value_but_real_header_is_sensitive(): + from harness.web_fetch.publication_privacy import publication_risk + raw = json.dumps({'header_names': ['Authorization:', 'Proxy-Authorization:'], 'count': 3}).encode() + assert publication_risk(raw) is None + assert project_evidence_view('public-evidence/response.json', raw, role='native_response')[0] == raw + private = json.dumps({'header': 'Authorization: Custom opaque-private-value'}).encode() + assert publication_risk(private) == 'sensitive field value' + public, _ = project_evidence_view('public-evidence/response.json', private, role='native_response') + assert b'opaque-private-value' not in public and publication_risk(public) is None + + +def test_json_escaped_sensitive_values_are_still_detected(): + from harness.web_fetch.publication_privacy import publication_risk + assert publication_risk(br'{"content":"failed at \u002fUsers\u002fexample\u002fprivate\u002fa"}') == 'machine-local user path' + assert publication_risk(br'{"\u0061pi_key":"private-value"}') == 'sensitive field value' + + +def test_public_call_log_view_support_does_not_weaken_acquisition_checks(tmp_path): + from tests.test_public_call_logs import run, ResultProvider + from harness.web_fetch.models import ProviderResult + from harness.web_fetch.artifacts import verify_record_public_call_logs + binding, original, public, index, profile = _declared_view_snapshot(tmp_path) + row = run(tmp_path/'run', ResultProvider(ProviderResult(True, original.decode()))) + root = tmp_path/'run' + # Install only the explicit public view proof into the simulated public root. + import shutil + shutil.copytree(tmp_path/'public-evidence', root/'public-evidence') + shutil.copy2(tmp_path/'receipt.json', root/'receipt.json') + index['files'][0]['original_path'] = row['artifact_path'] + raw = json.dumps(index).encode() + (root/'public-views.json').write_bytes(raw) + profile['publication_evidence_views']['sha256'] = hashlib.sha256(raw).hexdigest() + (root/'presentation-binding.json').write_text(json.dumps(profile)) + (root/row['artifact_path']).write_bytes(public) + with pytest.raises(ValueError, match='output binding changed'): + verify_record_public_call_logs(root, row) + verify_record_public_call_logs(root, row, allow_public_views=True) + (root/'presentation-binding.json').unlink() + with pytest.raises(ValueError): + verify_record_public_call_logs(root, row, allow_public_views=True) + + +def test_report_display_annotations_do_not_replace_original_score_hash(tmp_path): + from harness.web_fetch.publication_privacy import public_render_views + binding, _, _, _, _ = _declared_view_snapshot(tmp_path) + row = {'attempt_id': 'p__case', 'artifact_path': binding['path'], + 'metrics': {'content_sha256': binding['sha256']}} + before = json.dumps(row, sort_keys=True) + fields = public_render_views(tmp_path, [row])['p__case']['scored_answer'] + assert fields['original_sha256'] == row['metrics']['content_sha256'] + assert fields['published_sha256'] != fields['original_sha256'] + assert json.dumps(row, sort_keys=True) == before + + +@pytest.mark.parametrize('url', ['https://example.test/item?auth_key=opaque-private', + 'https://example.test/wiki/Function_(math)?token=opaque-private']) +def test_query_redaction_preserves_markdown_link_boundary(url): + from harness.web_fetch.publication_privacy import redact_url_queries + original = '[Stable title]('+url+')' + public, count = redact_url_queries(original) + assert count == 1 and 'opaque-private' not in public + assert public.startswith('[Stable title](') and public.endswith(')') + assert public.count('(') == original.count('(') + assert public.count(')') == original.count(')') + assert public == original.replace('opaque-private', '[REDACTED]') + assert redact_url_queries(public) == (public, 0) + + +def test_redacting_one_url_cannot_remove_adjacent_markdown_records(): + from harness.web_fetch.publication_privacy import redact_url_queries + from lexbench_webfetch.reference_quality import markdown_links + original = ('[Photo](https://cdn.test/image?auth_key=private-value)' + '[Item](https://example.test/pins/123)' + '[Other](https://example.test/pins/456)') + public, count = redact_url_queries(original) + assert count == 1 + assert public == original.replace('private-value', '[REDACTED]') + assert len(markdown_links(public)) == len(markdown_links(original)) + assert 'https://example.test/pins/123' in public and 'https://example.test/pins/456' in public + + +def test_unwrapped_url_token_with_parenthesis_is_fully_redacted(): + from harness.web_fetch.publication_privacy import redact_url_queries + public, count = redact_url_queries('https://example.test/api?token=private)continued') + assert count == 1 and 'continued' not in public and 'private' not in public + + +def test_public_review_excerpts_use_view_policy_without_changing_task_rules(): + from harness.web_fetch.publication_privacy import project_host_metadata_files + excerpt = 'Example /home/example/private/file.txt and stable body' + files = {'review.json': json.dumps({'attempts': [{'adjudication_evidence': { + 'samples': [{'start': 0, 'end': len(excerpt), 'text': excerpt}]}}]}).encode(), + 'dataset.json': json.dumps({'cases': [{'required_facts': [{'alternatives': [excerpt]}]}]}).encode()} + public, receipt = project_host_metadata_files(files, ['/Users/actual-user'], public_excerpt_views=True) + sample = json.loads(public['review.json'])['attempts'][0]['adjudication_evidence']['samples'][0] + assert sample['text'] == 'Example [HOST_PATH_OMITTED] and stable body' + assert sample['start'] == 0 and sample['end'] == len(excerpt) + assert public['dataset.json'] == files['dataset.json'] + assert receipt['files'][0]['path'] == 'review.json' + + +def test_query_timestamp_is_not_a_semicolonless_html_times_entity(): + from harness.web_fetch.publication_privacy import publication_risk + original = '[Image](https://example.test/photo?token=opaque-private×tamp=123456)' + assert publication_risk(original.encode()) == 'sensitive URL query value' + public, _ = project_evidence_view('public-evidence/answer.txt', original.encode(), + role='scored_answer', content_format='text') + assert b'×tamp=123456' in public and b'opaque-private' not in public + assert publication_risk(public) is None + assert publication_risk(b'https://example.test/?api_key=opaque-private') == 'sensitive URL query value' + + +@pytest.mark.parametrize('defect', ['different_display', 'original_leak', 'private_tree']) +def test_release_rejects_wrong_display_or_private_originals(tmp_path, defect): + from harness.web_fetch.publication_privacy import require_public_evidence_view_inventory + binding, original, _, _, profile = _declared_view_snapshot(tmp_path) + rows = [{'artifact_path': binding['path'], 'metrics': {'content_sha256': binding['sha256']}}] + if defect == 'different_display': (tmp_path/binding['path']).write_bytes(b'wrong answer') + if defect == 'original_leak': (tmp_path/binding['path']).write_bytes(original) + if defect == 'private_tree': (tmp_path/'private_artifacts').mkdir() + with pytest.raises(ValueError): + require_public_evidence_view_inventory(tmp_path, profile['publication_evidence_views'], rows) + + +@pytest.mark.parametrize('source,expected', [ + ('https://example.test/api?apiKey=opaque-private', + 'https://example.test/api?apiKey=[REDACTED]'), + ('https://example.test/api?x=1&apiKey=opaque-private&view=full', + 'https://example.test/api?x=1&apiKey=[REDACTED]&view=full'), + ('https://example.test/api?apiKey=opaque+private">safe', + 'https://example.test/api?apiKey=[REDACTED]">safe'), +]) +def test_encoded_query_delimiters_are_redacted_without_rewriting_html(source, expected): + from harness.web_fetch.publication_privacy import redact_url_queries, publication_risk + public, count = redact_url_queries(source) + assert public == expected and count == 1 + assert publication_risk(public.encode()) is None + assert redact_url_queries(public) == (public, 0) + + +def test_metadata_projection_supports_disk_backed_inputs_and_outputs(tmp_path): + from collections.abc import MutableMapping + from harness.web_fetch.publication_privacy import project_host_metadata_files + class Files(MutableMapping): + def __init__(self, root): self.root, self.names = root, set() + def __getitem__(self, key): return (self.root/key).read_bytes() + def __setitem__(self, key, value): + path = self.root/key + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(value) + self.names.add(key) + def __delitem__(self, key): (self.root/key).unlink(); self.names.remove(key) + def __iter__(self): return iter(sorted(self.names)) + def __len__(self): return len(self.names) + source, destination = Files(tmp_path/'original'), Files(tmp_path/'public') + source['child.json'] = b'{"file":"/Users/example/private/input.json"}' + source['parent.json'] = json.dumps({'child_sha256': hashlib.sha256(source['child.json']).hexdigest()}).encode() + public, receipt = project_host_metadata_files(source, ['/Users/example'], output_store=destination) + assert public is destination + assert json.loads(destination['child.json'])['file'] == '[HOST_PATH_OMITTED]' + assert json.loads(destination['parent.json'])['child_sha256'] == hashlib.sha256(destination['child.json']).hexdigest() + assert '/Users/example' in source['child.json'].decode() + assert len(receipt['files']) == 2 + + +def test_metadata_path_redaction_preserves_following_json_escaped_body(): + from harness.web_fetch.publication_privacy import project_host_metadata_files + packet = json.dumps({'content': 'Saved to /Users/example/private/a.txt\nStill visible'}) + original = json.dumps({'stdout': packet}).encode() + public, _ = project_host_metadata_files({'metadata.json': original}, ['/Users/example']) + inner = json.loads(json.loads(public['metadata.json'])['stdout']) + assert inner['content'] == 'Saved to [HOST_PATH_OMITTED]\nStill visible' + + +def test_bilingual_download_link_displays_text_instead_of_escaped_markup(): + from bs4 import BeautifulSoup + from pathlib import PurePosixPath + from harness.web_fetch.report import link + from harness.web_fetch.report_i18n import localize_html + raw = ''+link('../public-evidence/a.json', + '下载完整原生响应（公开副本）','Download complete native response (public view)')+'' + chinese = BeautifulSoup(raw, 'html.parser').select_one('a') + assert chinese.text == '下载完整原生响应（公开副本）' + english = BeautifulSoup(localize_html(raw, PurePosixPath('attempts/a.html')), 'html.parser').select_one('a') + assert english.text == 'Download complete native response (public view)' + assert english['href'] == '../../public-evidence/a.json' + + +def test_privacy_check_reuses_only_identical_bytes(): + from unittest.mock import patch + from harness.web_fetch import publication_privacy as owner + owner._cached_publication_risk.cache_clear() + safe = b'Unique unchanged evidence cache fixture' + unsafe = safe+b'\nAuthorization: Custom private-credential' + with patch.object(owner, '_scan_publication_risk', wraps=owner._scan_publication_risk) as scan: + assert owner.publication_risk(safe) is None + assert owner.publication_risk(safe) is None + assert scan.call_count == 1 + assert owner.publication_risk(unsafe) == 'sensitive field value' + assert scan.call_count == 2 + owner._cached_publication_risk.cache_clear() diff --git a/tests/test_public_report_language.py b/tests/test_public_report_language.py index 269a5fd0e..52ba8b160 100644 --- a/tests/test_public_report_language.py +++ b/tests/test_public_report_language.py @@ -13,6 +13,70 @@ ROOT = Path(__file__).resolve().parents[1] +def test_media_family_guides_preserve_numbers_and_translate_every_explanation(): + from collections import Counter + from pathlib import PurePosixPath + from harness.web_fetch.report_components import examples, task_contract_guide, task_pass_guide + from harness.web_fetch.report_i18n import localize_html + from harness.web_fetch.report_check import EnglishNarrative, PageShape + + source = task_pass_guide('web_media') + task_pass_guide('media_file') + task_contract_guide() + examples() + english = localize_html(source, PurePosixPath('methodology/index.html')) + zh_shape, en_shape = PageShape(), PageShape() + zh_shape.feed(source) + en_shape.feed(english) + assert Counter(zh_shape.numbers) == Counter(en_shape.numbers) + narrative = EnglishNarrative({'中文'}) + narrative.feed(english) + assert not narrative.untranslated + + +def test_registered_interfaces_and_pricing_have_complete_english(tmp_path): + from collections import Counter + from pathlib import PurePosixPath + from harness.web_fetch.report import table, cell, link, esc + from harness.web_fetch.report_interfaces import SPECS, render_interfaces + from harness.web_fetch.report_pricing import render_pricing + from harness.web_fetch.report_i18n import localize_html + from harness.web_fetch.report_check import EnglishNarrative, PageShape + + source = render_pricing(tmp_path, list(SPECS), table=table, cell=cell, link=link, esc=esc) + source += render_interfaces(tmp_path, []) + english = localize_html(source, PurePosixPath('adoption/index.html')) + zh_shape, en_shape = PageShape(), PageShape() + zh_shape.feed(source) + en_shape.feed(english) + assert Counter(zh_shape.numbers) == Counter(en_shape.numbers) + narrative = EnglishNarrative({'中文'}) + narrative.feed(english) + assert not narrative.untranslated + + +def test_scoped_english_refresh_preserves_unselected_pages_and_rejects_escaping_paths(tmp_path, monkeypatch): + from harness.web_fetch.report_i18n import write_english_report + (tmp_path/'index.html').write_text('

评测方法

') + (tmp_path/'en/attempts').mkdir(parents=True) + saved = tmp_path/'en/attempts/saved.html' + original = b'Exact previously generated result detail' + saved.write_bytes(original) + assert write_english_report(tmp_path, source_routes=['index.html']) == 1 + assert 'Methodology' in (tmp_path/'en/index.html').read_text() + assert saved.read_bytes() == original + monkeypatch.chdir(tmp_path) + assert write_english_report(Path('.'), source_routes=['index.html']) == 1 + for route in ('../outside.html', '/outside.html', 'en/index.html', 'results.jsonl'): + with pytest.raises(ValueError, match='relative Chinese HTML'): + write_english_report(tmp_path, source_routes=[route]) + assert saved.read_bytes() == original + (tmp_path/'alias').mkdir() + (tmp_path/'alias/index.html').write_text('') + (tmp_path/'outside').mkdir() + (tmp_path/'en/alias').symlink_to(tmp_path/'outside', target_is_directory=True) + with pytest.raises(ValueError, match='Symlinked evidence'): + write_english_report(tmp_path, source_routes=['alias/index.html']) + assert not (tmp_path/'outside/index.html').exists() + + @pytest.mark.parametrize("mode", ["--diagnostic-cohorts", "--webmainbench-full"]) @pytest.mark.parametrize("language", [None, "zh", "en"]) @pytest.mark.parametrize("saved", [False, True]) diff --git a/tests/test_publication_privacy.py b/tests/test_publication_privacy.py index 49b588e4a..481cbe901 100644 --- a/tests/test_publication_privacy.py +++ b/tests/test_publication_privacy.py @@ -575,3 +575,141 @@ def interrupted(*args): if __name__ == '__main__': unittest.main() + + +def test_host_metadata_projection_rebinds_dependencies_without_changing_measurements(): + import hashlib + import json + from harness.web_fetch.publication_privacy import project_host_metadata_files + config = json.dumps({'binary': '/Users/fixture/tool/bin', 'version': '1.2.3', 'environment': {'hostname': 'private-machine'}}).encode() + config_hash = hashlib.sha256(config).hexdigest() + result = json.dumps({'ok': True, 'duration': 1.25, 'content_sha256': 'a'*64, + 'provider_config_sha256': config_hash, + 'provider_metadata': {'stderr': 'at /Users/fixture/work/main.py:4'}}).encode()+b'\n' + files = {'config.json': config, 'results.jsonl': result, 'answer.txt': b'actual answer'} + original = files.copy() + public, receipt = project_host_metadata_files(files, ['/Users/fixture']) + assert files == original + assert public['answer.txt'] == files['answer.txt'] + row = json.loads(public['results.jsonl']) + assert row['provider_config_sha256'] == hashlib.sha256(public['config.json']).hexdigest() + assert (row['ok'], row['duration'], row['content_sha256']) == (True, 1.25, 'a'*64) + assert json.loads(public['config.json'])['version'] == '1.2.3' + assert 'hostname' not in json.loads(public['config.json'])['environment'] + assert b'/Users/fixture' not in b''.join(public.values()) + assert receipt['files'][0]['original_sha256'] == config_hash + assert receipt['files'][0]['published_sha256'] != config_hash + assert '/Users/fixture' not in json.dumps(receipt) + repeated, unchanged = project_host_metadata_files(public, ['/Users/fixture']) + assert repeated == public and unchanged['files'] == [] + + +def test_public_diagnostic_contexts_redact_queries_and_preserve_scores(): + from harness.web_fetch.publication_privacy import project_host_metadata_files + row = {'ok': True, 'elapsed_ms': 12.5, 'metrics': {'content_pass': True, + 'content_sha256': 'a' * 64, 'composite_group_results': [{'records': [ + {'context': '[Forum](https://example.test/thread?access_token=private-token-canary)'}]}]}} + raw = (json.dumps(row) + '\n').encode() + files = {'results.jsonl': raw, 'receipt.json': json.dumps({'results_sha256': hashlib.sha256(raw).hexdigest()}).encode()} + public, receipt = project_host_metadata_files(files, ['/Users/fixture'], public_excerpt_views=True) + projected = json.loads(public['results.jsonl']) + assert b'private-token-canary' not in public['results.jsonl'] + assert projected['ok'] == row['ok'] and projected['elapsed_ms'] == row['elapsed_ms'] + assert projected['metrics']['content_pass'] is True + assert projected['metrics']['content_sha256'] == 'a' * 64 + assert json.loads(public['receipt.json'])['results_sha256'] == hashlib.sha256(public['results.jsonl']).hexdigest() + assert files['results.jsonl'] == raw + assert any('.metrics.' in field and field.endswith('.context') for item in receipt['files'] for field in item['redacted_fields']) + unchanged, _ = project_host_metadata_files(files, ['/Users/fixture']) + assert unchanged == files + + +def test_host_metadata_projection_requires_an_explicit_non_root_prefix(): + import pytest + from harness.web_fetch.publication_privacy import project_host_metadata_files + for prefixes in ([], ['/'], ['relative/path']): + with pytest.raises(ValueError): + project_host_metadata_files({}, prefixes) + + +def test_public_metadata_receipt_checks_bytes_and_excludes_scored_payloads(tmp_path): + import hashlib + import json + import pytest + from harness.web_fetch.publication_privacy import project_host_metadata_files, require_host_metadata_projection + original={'config.json': b'{"binary":"/Users/fixture/tool"}'} + public,receipt=project_host_metadata_files(original,['/Users/fixture']) + (tmp_path/'config.json').write_bytes(public['config.json']) + def bind(): + path=tmp_path/'projection.json' + path.write_text(json.dumps(receipt)) + return {'path':path.name,'sha256':hashlib.sha256(path.read_bytes()).hexdigest()} + require_host_metadata_projection(tmp_path,bind()) + (tmp_path/'config.json').write_text('{"binary":"changed"}') + with pytest.raises(ValueError):require_host_metadata_projection(tmp_path,bind()) + (tmp_path/'config.json').write_bytes(public['config.json']) + receipt['files'][0]['path']='artifacts/answer.json' + with pytest.raises(ValueError,match='preserve scored payloads'): + require_host_metadata_projection(tmp_path,bind()) + + +def test_host_projection_does_not_match_another_user_with_the_same_prefix(): + from harness.web_fetch.publication_privacy import project_host_metadata_files + files={'config.json': b'{"binary":"/Users/fixture-other/tool"}'} + public,receipt=project_host_metadata_files(files,['/Users/fixture']) + assert public==files and receipt['files']==[] + + +def test_named_host_in_auxiliary_log_is_projected_with_bound_public_hash(tmp_path): + import hashlib + import json + from harness.web_fetch.publication_privacy import project_host_metadata_files, require_host_metadata_projection + name='logs/case.debug.txt' + raw=b'{"hostname":"fixture.local","message":"response completed"}' + inputs={name:raw,'answer.txt':b'unchanged scored text'} + public,receipt=project_host_metadata_files(inputs,['/Users/fixture'],host_names=['fixture.local']) + assert public['answer.txt']==inputs['answer.txt'] + assert b'fixture.local' not in public[name] + assert b'response completed' in public[name] + (tmp_path/'logs').mkdir();(tmp_path/name).write_bytes(public[name]) + path=tmp_path/'projection.json';path.write_text(json.dumps(receipt)) + require_host_metadata_projection(tmp_path,{'path':path.name,'sha256':hashlib.sha256(path.read_bytes()).hexdigest()}) + + +def test_host_projection_refuses_to_rewrite_score_bearing_or_response_payloads(): + import pytest + from harness.web_fetch.publication_privacy import project_host_metadata_files + for name in ('artifacts/answer.json','logs/case.response.txt','answer.txt'): + raw=b'{"host":"fixture.local"}' + with pytest.raises(ValueError,match='must not alter scored'): + project_host_metadata_files({name:raw},['/Users/fixture'],host_names=['fixture.local']) +def test_long_unterminated_escaped_string_preserves_content_without_rescanning_suffixes(): + from harness.web_fetch.publication_privacy import redact_url_queries + text = '"' + ('\\"fragment' * 20000) + assert redact_url_queries(text) == (text, 0) + + +def test_long_escaped_url_still_redacts_sensitive_query_value(): + from harness.web_fetch.publication_privacy import redact_url_queries + text = 'ordinary log ' * 500 + r'"https:\/\/example.com\/image?apiKey=private-value&width=300"' + public, count = redact_url_queries(text) + assert count == 1 + assert 'private-value' not in public + assert 'width=300' in public + + +def test_large_and_small_string_tokenization_have_identical_public_redaction_semantics(): + import random + from harness.web_fetch.publication_privacy import redact_url_queries + rng = random.Random(51) + prefix = ' ' * 4097 + fragments = [ + '"abc\\"def" tail "https://example.com/?token=private-value"', + '"unterminated\\"suffix', '"line\nfeed"', '"line\\\nfeed"', + '"line\\\rfeed"', '"\\u0068ttps://example.com/?token=private-value"', + '"nul\x00boundary"', '"backslash\\\\"', + ] + fragments += [''.join(rng.choice('ab"\\\r\n\t\x00') for _ in range(60)) for _ in range(64)] + for text in fragments: + expected, count = redact_url_queries(text) + assert redact_url_queries(prefix+text) == (prefix+expected, count) diff --git a/tests/test_reference_lifecycle.py b/tests/test_reference_lifecycle.py index f8a7d665b..cb0ac631e 100644 --- a/tests/test_reference_lifecycle.py +++ b/tests/test_reference_lifecycle.py @@ -273,6 +273,64 @@ def test_resource_source_browser_observation_must_satisfy_all_targets(self): with self.assertRaisesRegex(ReferenceLifecycleError, "does not cover every"): validate_resource_source_evidence(case, self.dataset_path) + def test_resource_equivalence_keeps_original_browser_observation_and_primary_target(self): + from harness.web_fetch.browser_observations import match_resource_targets + original = self.resource_case() + scope = resource_source_scope_identity(original) + candidates = [{'tag': 'a', 'attribute': 'href', 'url': original.expected_resources[0]['urls'][0]}] + browser = {'schema_version': 1, 'case_id': original.id, 'url': original.url, + 'status': 'success', 'independent': True, 'checked_at': self.now.isoformat(), + 'resource_scope_identity': scope, 'resource_candidates': candidates, + 'resource_candidates_sha256': sha256(json.dumps(candidates, ensure_ascii=False, + sort_keys=True, separators=(',', ':'))), + 'target_matches': match_resource_targets(original, candidates), + 'completion': {'satisfied': True}} + original = self.resource_case(evidence_updates={'state': 'needs_review', 'status': 403, + 'resources': []}, browser=browser) + browser_path = self.root/'sources/resource/browser-observation.json' + browser_bytes = browser_path.read_bytes() + aliases = [{**original.expected_resources[0], 'aliases': ['https://cdn.example.test/a.png?w=800']}] + revised = replace(original, expected_resources=aliases) + new_scope = resource_source_scope_identity(revised) + receipt = {'kind': 'offline_resource_equivalence_revision', 'provider_calls_added': 0, + 'new_source_observation': False, 'before_scope': scope, + 'after_scope_sha256': sha256(json.dumps(new_scope, ensure_ascii=False, + sort_keys=True, separators=(',', ':'))), + 'source_evidence_sha256': original.source_verification['evidence_sha256'], + 'browser_observation_sha256': sha256(browser_bytes), + 'authority': {'description': 'Same observed asset; a score-only display-size equivalent.'}, + 'regression_ids': ['resource-query-explicit-alias-preserved', + 'resource-query-different-resource-rejected']} + proof = self.root/'sources/resource/resource-equivalence-revision.json' + def bound_case(value): + raw = (json.dumps(value)+'\n').encode() + proof.write_bytes(raw) + return replace(revised, resource_output_contract={ + **revised.resource_output_contract, 'equivalence_revision': { + 'path': proof.name, 'sha256': sha256(raw)}}) + result = validate_resource_source_evidence(bound_case(receipt), self.dataset_path) + self.assertTrue(result['target_matches'][0]['satisfied']) + self.assertEqual(browser_path.read_bytes(), browser_bytes) + # Later score-only revisions get immutable proof names, preserving prior receipts. + prior_proof = proof + proof = proof.with_name('resource-equivalence-'+sha256((json.dumps(receipt)+'\n').encode())+'.json') + self.assertIsNotNone(validate_resource_source_evidence(bound_case(receipt), self.dataset_path)) + self.assertTrue(prior_proof.exists()) + # A new alias cannot silently rewrite or replace the captured scope. + for defect in ('primary', 'policy', 'source', 'observation', 'controls', 'fresh'): + bad = deepcopy(receipt) + if defect == 'primary': bad['before_scope']['expected_resources'][0]['urls'] = ['https://wrong.test/other.png'] + if defect == 'policy': bad['before_scope']['resource_output_contract'] = {'unexpected': 'reject'} + if defect == 'source': bad['source_evidence_sha256'] = '0'*64 + if defect == 'observation': bad['browser_observation_sha256'] = '0'*64 + if defect == 'controls': bad['regression_ids'] = [] + if defect == 'fresh': bad['new_source_observation'] = True + with self.subTest(defect=defect), self.assertRaises(ReferenceLifecycleError): + validate_resource_source_evidence(bound_case(bad), self.dataset_path) + without_receipt = replace(revised, resource_output_contract={}) + with self.assertRaisesRegex(ReferenceLifecycleError, 'scope/hash mismatch'): + validate_resource_source_evidence(without_receipt, self.dataset_path) + def test_per_case_rechecks_clock_and_immutable_scope(self): binding = self.checks()["cases"]["body"] with self.assertRaisesRegex(ReferenceLifecycleError, "30 minutes"): diff --git a/tests/test_report_article_cases.py b/tests/test_report_article_cases.py index 1a908611f..46ee030ec 100644 --- a/tests/test_report_article_cases.py +++ b/tests/test_report_article_cases.py @@ -41,12 +41,11 @@ def test_text_evidence_is_escaped_without_rewriting_its_content(self): self.assertIn('<script>', left) self.assertIn('<script>', right) self.assertNotIn('' - r={'ok':True,'status_code':200} - metrics=score_attempt(c,output.encode(),ok=True) - scoring_text=_body_scoring_text(r,output) - self.assertNotIn('hidden words',scoring_text) - rows,cards=body_region_rows(c,metrics,scoring_text) + metrics=score_attempt(c,b'abcdefghijkl',ok=True) + saved=copy.deepcopy(metrics) + with patch('lexbench_webfetch.body_coverage.evaluate_body_requirements', + side_effect=AssertionError('presentation must not rescore saved passages')): + rows,cards=body_region_rows(c,metrics,'abcdefghijkl') + self.assertEqual(metrics,saved) self.assertEqual(len(rows),1) self.assertIn('abcdefghijkl',''.join(cards)) @@ -289,7 +289,7 @@ def test_risk_control_failure_categories_are_exclusive_and_output_first(self): def failed(**metrics): r = record(c, passed=False) - r["metrics"].update(request_ok=True, content_chars=500, + r["metrics"].update(forbidden_requirement_hits=[], request_ok=True, content_chars=500, stable_task_pass=False) r["metrics"].update(metrics) return r @@ -298,6 +298,11 @@ def failed(**metrics): failure_category(failed(blocked_page=True, composite_acceptance_pass=False), c), "返回验证或拦截页面", ) + for track in ("risk_control", "web", "challenge", "popular"): + qualified_case = case("qualified", "body", track) + qualified = record(qualified_case, passed=True) + qualified["metrics"].update(forbidden_requirement_hits=[], blocked_page=True, stable_blocked_page=True) + self.assertEqual(failure_category(qualified, qualified_case), "—") self.assertEqual(failure_category(failed(content_chars=0), c), "完全空返回") self.assertEqual( failure_category(failed(content_chars=37, markdown_headings=0, markdown_links=0, @@ -1098,3 +1103,44 @@ def test_build_rejects_stale_scoring_implementation_before_render(self): if __name__=='__main__': unittest.main() + + +def test_body_display_shares_native_markup_and_entity_handling_with_scorer(): + from lexbench_webfetch.scoring import score_attempt + contract = {"version": 3, "unit": "region_compact_hexagram_recall", + "minimum_coverage": .8, "minimum_floor": .6, "minimum_region_passes": 1, + "regions": [{"id": "body", "text": "abcdefghijkl"}]} + definition = {"id": "markup", "url": "https://example.test", "body_requirements": contract} + for output in ("abcdef
ghijkl", "abcdefghijkl", "`
` abcdefghijkl", + "\\
abcdefghijkl", "
abcdef
", + 'abcdef source', + 'abcdef photo'): + record = {"ok": True, "title": "abcdefghijkl"} + metrics = score_attempt(definition, output.encode(), ok=True, title=record["title"]) + display = _body_scoring_text(record, output) + rows, cards = body_region_rows(definition, metrics, display) + assert len(rows) == 1 + assert "
Read

Saved content

' + safe = sanitize_article_html(raw) + soup = BeautifulSoup(safe, 'html.parser') + assert not soup.find(['style', 'script', 'iframe', 'form', 'input']) + assert not soup.img.get('src') and not soup.img.get('onerror') + assert not soup.a.get('href') and not soup.a.get('onclick') + assert soup.p['id'] == 'archived-rows' + assert 'Saved content' in safe + import base64 + png='data:image/png;base64,'+base64.b64encode(b'\x89PNG\r\n\x1a\n').decode() + preserved=BeautifulSoup(sanitize_article_html(''),'html.parser') + assert preserved.img['src']==png + rejected=BeautifulSoup(sanitize_article_html(''),'html.parser') + assert not rejected.img.get('src') + + +def test_english_chrome_preserves_raw_evidence_and_filter_keys(): + from pathlib import PurePosixPath + from harness.web_fetch.report_i18n import localize_html + raw = '
验证码 & 原始返回 26/26
' + source = '

任务规则未满足

' + raw + '' + english = localize_html(source, PurePosixPath('analysis/index.html')) + assert raw in english + assert 'data-failure-category="规则不满足"' in english + assert 'aria-label="Search"' in english + assert 'Task requirements not met' in english + + +def test_bound_report_translations_are_external_and_tamper_evident(tmp_path): + from harness.web_fetch.report_reading import archived_translations + source = tmp_path / "source" + source.mkdir() + translations = source / "translations.json" + translations.write_text(json.dumps({"text": {"核验说明": "Review notes"}})) + manifest = {"languages": {}, "translations": { + "path": "translations.json", "sha256": hashlib.sha256(translations.read_bytes()).hexdigest()}} + (source / "archived-reading.json").write_text(json.dumps(manifest)) + output = tmp_path / "output" + output.mkdir() + copy_archived_reading(source, output) + assert archived_translations(output) == {"核验说明": "Review notes"} + (output / "translations.json").write_text("changed") + with pytest.raises(ValueError, match="translations hash mismatch"): + archived_translations(output) + + +def test_article_translation_cannot_override_other_routes_or_remove_status(tmp_path): + from pathlib import PurePosixPath + from harness.web_fetch.report_i18n import localize_html, write_english_report + languages = {} + for language in ('zh', 'en'): + path = tmp_path / (language + '.txt') + path.write_text('

核验记录

') + languages[language] = {'path': path.name, 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()} + translation = tmp_path / 'translations.json' + translation.write_text(json.dumps({'text': {'核验记录': 'Archive-only label'}})) + (tmp_path / 'archived-reading.json').write_text(json.dumps({'languages': languages, + 'translations': {'path': translation.name, 'sha256': hashlib.sha256(translation.read_bytes()).hexdigest()}})) + (tmp_path / 'guide').mkdir() + banner = '
本地未认证诊断快照 · 不可用于排名
' + article = render_archived_reading(tmp_path) + (tmp_path / 'guide/index.html').write_text('
'+banner+article+'
') + root = '

核验记录

' + (tmp_path / 'index.html').write_text(root) + write_english_report(tmp_path) + assert (tmp_path / 'en/index.html').read_text() == localize_html(root, PurePosixPath('index.html')) + guide = (tmp_path / 'en/guide/index.html').read_text() + assert 'data-evidence-state="local_measurement"' in guide + assert 'Archive-only label' in guide + + +def test_model_translation_is_scoped_and_escaped(): + from pathlib import PurePosixPath + from harness.web_fetch.report import model_text + from harness.web_fetch.report_i18n import localize_html + raw='

核验记录

'+model_text('该结果的说明','Specific explanation ') + english=localize_html(raw,PurePosixPath('attempts/example.html')) + assert 'Specific explanation <script>bad</script>' in english + assert '该结果的说明' not in english + assert '' + for body,passed in ((cards,True),(hidden,False),(cards.replace('2026-09-02',''),False),(cards.replace('2026-09-02','&#50;026-09-02'),False),(cards.replace('
','