Skip to content

docs(evidence): seal the benchmark plan before any run - #16

Open
afflom wants to merge 1 commit into
mainfrom
docs/benchmark-plan
Open

afflom wants to merge 1 commit into
mainfrom
docs/benchmark-plan

Conversation

@afflom

@afflom afflom commented Oct 15, 2026

Copy link
Copy Markdown
Member

Item M10 of the completion plan #3, closes #12.

docs/evidence/benchmark-plan.md fixes, before any run it governs:

  • What is measured: per-image latency on one thread over the whole image set, and top-1 agreement (output codes where exposed).
  • Competitor field: onnxruntime 1.29.0 with default options, with ORT_ENABLE_ALL, and with the XNNPACK provider, plus a hand-specialized C implementation of the same integer arithmetic.
  • Procedure: one discarded pass, five timed passes, median with minimum and maximum, every implementation in the same session.
  • Recording: each run is one open ledger row naming the plan's SHA-256 (R2); a changed plan is a new file.

Competitor implementations and timing drivers stay outside this repository (R1) and are cited in model/authorities.toml by URL and SHA-256 before a run uses them; they are oracles and measurements, never authorities (C9). Further hardware is added as LexLean models each target (UOR-Foundation/LexLean#119); which targets come first is UOR-Foundation/LexLean#109.

No claim, register row or ledger row changes: the plan governs runs that have not happened yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Eo1UJoRXDrVSdkPw8UC35


Generated by Claude Code

The head-to-head so far compares the certified package with one onnxruntime
configuration. The plan fixes the competitor field, the image set, the
procedure, the agreement checks and how a run is recorded, before any run it
governs, so a result can only be an open ledger row against this file's
digest. Competitor implementations and timing drivers stay outside the
repository (R1) and are cited by URL and SHA-256.

Refs #12, #3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Eo1UJoRXDrVSdkPw8UC35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sealed benchmark plan against a fixed competitor field

2 participants