Skip to content

docs: publish an auditable Kimi agent run - #86

Open
acedatacloud-dev wants to merge 1 commit into
mainfrom
docs/kimi-agent-benchmark
Open

docs: publish an auditable Kimi agent run#86
acedatacloud-dev wants to merge 1 commit into
mainfrom
docs/kimi-agent-benchmark

Conversation

@acedatacloud-dev

Copy link
Copy Markdown
Member

Why

AceDataCloud needs original evidence content, not another generic product article. An August 4, 2026 Kimi Code CLI run already contains useful primary evidence: a complete multi-tool task, independent result checks, client/platform token reconciliation, historical billing arithmetic, a cache-pricing defect that was subsequently fixed, and a credential-hygiene finding.

The internal report is not a public, machine-verifiable artifact. This PR publishes a bounded case study and ledger while making its limitations explicit.

What changed

  • add an English Research page for one six-call kimi-k3 coding-agent run;
  • explain setup, task, execution sequence, independent correctness checks, mutation test, token reconciliation, historical billing, remediation, and credential safety;
  • commit a machine-readable JSON ledger with all six prompt/cache/completion counts, status codes, elapsed time, list Credits, and deducted Credits;
  • add a dependency-free verifier for call order, formulas, ratio, totals, HTTP status, cache total, task arithmetic, and report/data consistency;
  • add an English Research navigation group;
  • add path-scoped CI that runs the verifier and Docs navigation validation.

Arithmetic correction found during publication

The older internal narrative stated 113,032 cached tokens. Summing the six published rows gives 116,032. The public report, JSON, and verifier use the recomputed value. The exact ledger totals are:

  • historical list: 5.5824156 Credits
  • deducted: 5.13582235 Credits
  • recorded conversion: approximately $0.489007

This correction is one reason the machine-verifiable artifact is valuable.

Evidence and limitations

  • One task, one model, one client version, six calls—not a general quality leaderboard.
  • Historical rates are clearly labeled and must not be used as current pricing.
  • The original temporary CSV, generated code, session archive, and credential were deleted. The token/billing ledger is reproducible; the exact agent execution is not rerunnable from this repository.
  • No private platform logs, API keys, prompt-cache content, supplier details, or user data are published.
  • The original cache-pricing defect is described as fixed; the report does not present it as current behavior.

Validation

  • python3.12 scripts/verify_kimi_agent_benchmark.py — six calls, formulas, totals, cache sum, task arithmetic, and report strings verified
  • python3.12 scripts/validate_nav.py — schema OK, English page count increases by one
  • JSON parse — passed
  • supplier/internal routing scan — clean
  • git diff --check — clean

Scope boundaries

This PR publishes one research artifact. It does not change models, pricing, billing, APIs, SDKs, or the historical internal plan. It does not claim present-day price or service-level reliability.

Rollback

Revert this commit to remove the page, data, verifier, navigation entry, and path-scoped workflow.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant