[chaliy]

Requested by Михайло · project thread

What changed

This PR replaces the stub markkit-eval CLI with a mira eval study, modeled on bashkit-eval. The study has one eval, markkit_extract:

• Samples: ten realistic HTML pages under crates/markkit-eval/data/pages/: news, docs, blog, commerce, recipe, wiki, forum, changelog, a data table and a 90s legacy layout. Each page wraps real content in the usual chrome (nav, cookie banners, ads, share buttons, comments, asides, footers). The pages are embedded with include_str!.
• Subject: markkit runs in-process (extract_html_sync) and records extract_us, input_bytes, output_bytes and size_ratio.
• mode axis: default (the builder defaults) and main_content (the opt-in main-content detector). Both modes run on the same pages and checks, so the report shows them side by side.
• Scorers: markkit_expectations passes only if every check passes, and its value is the weighted pass rate. There is also one scorer per metric: completeness, accuracy, formatting and noise. Checks are typed JSON (contains, not_contains, line, heading, link, order), with no regex.
• Target: only Target::cli("markkit"). The eval is offline, needs no key and gives the same result on every run. LLM evals (an LLM judge, and downstream Q&A across a model matrix) come in a follow-up; they need a model API key in the environment.
• Results: the first run is committed as crates/markkit-eval/results/markkit_extract-2026-10-27.{md,json}. Mira run folders are gitignored.

just eval and just eval-list run mira run and mira list with --study-bin markkit-eval. knowledge/operations/eval.md is rewritten to cover the design, dataset format, checks, mode axis, results and baseline.

Why

This is roadmap initiative 9 ("yes, use mira"). There was no way to measure extraction quality.

Before / After

Before: cargo run -p markkit-eval printed evaluation harness not yet implemented.

After: just eval, run on main at 5a7434c.

mode	pages passing	overall	completeness	accuracy	formatting	noise
default	0/10	73.0%	92.3%	100%	95.6%	34.3%
main_content	2/10	86.8%	88.3%	90.0%	88.1%	82.1%

What the eval found:

• An earlier run caught row-header tables (<tr><th>Weight</th><td>1.2 kg</td></tr>) dropping every value. IR v2 has since fixed this.
• New regression from fix(render): escape markdown output and allowlist URL schemes #23: 4.6 out of 5 stars now renders as 4\.6 out of 5 stars. 4.6 is not an ordered-list marker, so the backslash is just noise. Because of this, main_content on the product page dropped from 100% to 95%.
• main_content keeps only the accepted answer on a Q&A page and drops the question. Forum scores 81% in default and 62% in main_content.
• In both modes, <tfoot> and <caption> are dropped and [edit] links leak into headings.

Risk

• Low. The changes are limited to the eval crate (now publish = false), docs and knowledge. Library behavior is unchanged.
• The crate depends on mira-eval 0.5 (MIT) and on markkit with html only, so no reqwest or hyper. cargo deny check passes.
• mira run exits non-zero while any page fails. That is expected for a measurement, and CI does not run it.

Checklist

• Tests added or updated (23 unit tests): dataset validity, the pages table, each check type passing and failing, per-metric aggregation, scorer N/A, infra and failure handling, modes (unknown mode is an infra error, and main_content changes the output), and the subject.
• Backward compatibility considered. The stub's --provider and --dataset flags are removed; they never did anything.

Locally, fmt, workspace clippy -D warnings, cargo test --workspace, check_okf, okf-lint 0.1.1 and cargo deny check all pass, and the eval runs end to end with the real mira 0.5 CLI.

🤖 Generated with Claude Code