Skip to content

chore(eval): record first keyed markkit_qa run and extract re-run - #59

Merged
chaliy merged 3 commits into
mainfrom
eval-results-2026-10-28
Oct 28, 2026
Merged

chaliy merged 3 commits into
mainfrom
eval-results-2026-10-28

Conversation

@chaliy

@chaliy chaliy commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Adds the first real-model eval results as references in crates/markkit-eval/results/: markkit_qa-2026-10-28.{md,json} and markkit_extract-2026-10-28.{md,json}. Also updates knowledge/operations/eval.md (a QA results table, the extract re-run, and where the Doppler keys live) and knowledge/log.md. No code changes.

Why

The LLM eval (#56) had never run against real models. These results are the reference baseline to compare against when the HTML extraction changes.

Before / After

markkit_qa with Claude Sonnet 5.5, 23 questions per input:

input correct mean input tokens
markdown 23/23 464
main_content 22/23 377
raw html 23/23 945

markkit markdown is as accurate as raw HTML for about half the input tokens. The one miss is a known gap: main-content mode drops the forum question body. Every OpenRouter Muse Spark call returned HTTP 402 because the key's daily credit limit is too low, so those cases are N/A. The error text in the JSON has the key's dashboard ID redacted.

markkit_extract re-run: main_content passes 3/10 pages (it was 2/10; product_page passes again after the escaping fix). Mean scores are 87.4% for main_content and 73.6% for default.

Risk

  • Low. Only results and docs change.

Checklist

  • Tests added or updated (not applicable, results only)
  • Backward compatibility considered

🤖 Generated with Claude Code

@chaliy
chaliy merged commit 7a4e62f into main Oct 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant