[chaliy]

Requested by Михайло · project thread

What changed

This PR adds a second mira eval, markkit_qa. It checks whether an LLM answers correctly when it reads markkit's output, and how many tokens that costs.

• Samples: 23 factual questions over the ten existing eval pages (data/qa-tasks.jsonl), each with a list of accepted answer substrings. A test checks that every question has an accepted answer literally present on its page.
• input axis: markdown (default markkit output), main_content (main-content mode) and html (the raw page, as a baseline).
• Scorer answer_correct: passes if the reply contains an accepted answer. Matching is case-insensitive with whitespace collapsed, so scoring is deterministic and needs no judge model.
• Targets: openrouter/meta/muse-spark-1.3-contributor and anthropic/claude-sonnet-5-5. Each target runs only when its API key is set (OPENROUTER_API_KEY, ANTHROPIC_API_KEY), so offline runs and CI skip every case.
• Provider: minimal single-turn clients (src/provider.rs) for the Anthropic Messages API and Chat Completions (OpenRouter). Keys stay inside the study process and are never logged. *_BASE_URL overrides the endpoints.
• just eval-llm runs the eval. The keys live in Doppler: doppler run -- just eval-llm.

Why

The maintainer asked for the LLM evals to use Muse 1.3 contributor through OpenRouter, with tokens from Doppler, and confirmed the Anthropic key is in Doppler as well. This is the downstream half of roadmap initiative 9: whether markkit output is better LLM context than raw HTML.

Before / After

Before, the only eval was markkit_extract, which runs offline.

After, an offline run with mira run --study-bin markkit-eval markkit_qa --dry-run and no keys gives 0 passed / 0 scored (0 failed, 0 n/a, 138 skipped).

End to end, I ran it against a local mock Anthropic endpoint (ANTHROPIC_BASE_URL=http://127.0.0.1:18733, --tag data_table):

  [PASS] markkit_qa/data_table_q1@anthropic/claude-sonnet-5-5[input=markdown]  (100%)
         ✓ answer_correct — answered "24.9"
  [FAIL] markkit_qa/data_table_q2@anthropic/claude-sonnet-5-5[input=markdown]  (0%)
         ✗ answer_correct — answered "24.9"; expected one of ["72.4"]
  [SKIP] markkit_qa/data_table_q1@openrouter/meta/muse-spark-1.3-contributor[input=markdown]

The mock always gives the same answer; the run only checks the wiring. There has been no run with real keys yet, because this cloud environment can't reach Doppler and blocks api.doppler.com and openrouter.ai.

Risk

• Low. All changes are in the eval crate. reqwest is added to markkit-eval only; cargo tree -p markkit still has no reqwest or hyper.
• Provider or HTTP failures are recorded as infra errors (N/A), not as wrong answers.

Checklist

• Tests added or updated (11 new): request and response shapes for both wire formats, malformed responses, error text truncation, unknown provider, dataset validity and answerability, answer matching, documents for each input, correct and wrong answers, and infra handling.
• Backward compatibility considered. markkit_extract is unchanged.

Locally, fmt, workspace clippy -D warnings, cargo test --workspace, check_okf, okf-lint and cargo deny check all pass.

---

Generated by Claude Code