Skip to content

feat: dynamic provider pricing with provenance and cache policy - #26

Open
daimonbot wants to merge 1 commit into
mainfrom
feat/live-model-pricing-20
Open

daimonbot wants to merge 1 commit into
mainfrom
feat/live-model-pricing-20

Conversation

@daimonbot

Copy link
Copy Markdown
Owner

What was asked

Issue #20: model prices must come from live provider price sources resolved at runtime, not from tables in the source, with provenance on every amount, a cache policy, deterministic historical results, and offline operation through a bundled snapshot or an explicit "unavailable". First and foremost: every session on Claude Opus 5.5 (claude-opus-5-5), and on any other model missing from the old table, showed a cost of $0.

Root cause of the $0

priceClaude() / priceCodex() looked the model up in a hardcoded table. A miss returned usd: 0 with confidence n/a, and the session aggregate then labelled the node partial, not n/a, even when no call was priced. From there the zero was indistinguishable from a real one: the React list and session views print $0.00 for 0, the CLI prints "$" + cost.toFixed(2), and the index's default min $0.5 filter hid those sessions entirely. claude-opus-5-5, claude-fable-5-1, claude-sonnet-5-5, claude-opus-4-8, gpt-5.6-terra, gpt-5.6-luna, gpt-6-sol and gpt-6-luna were all missing from the table.

What changed

src/pricing/ replaces src/prices.mjs:

  • sources.mjs: the two official price pages and a strict parser for each. A page that does not parse, or yields fewer than five priced rows, is a failed refresh, never a price. A rate a page does not publish stays null. It is never derived from a multiplier.
  • index.mjs: an append-only ledger of dated entries merged from the bundled snapshot.json and a local cache ($XDG_CACHE_HOME/agent-atlas/prices.json, override AGENT_ATLAS_PRICES_CACHE).
    • A refresh that sees a different rate appends an entry effective from the moment it was observed. Earlier calls keep their price, so a computed historical cost never moves.
    • A model's first entry applies back to its first use, because no page states when a price started.
    • A failed refresh appends nothing, and a model that disappears from a page keeps its entries.
  • snapshot.json: the last-known prices, generated by npm run prices:snapshot with the same parsers. Entries are only appended, so a release's snapshot diff is exactly the prices that changed.

Cache and refresh policy.

  • Each source is re-read at most once every 24 h, and a failed one is retried after 1 h.
  • serve refreshes in the index worker's background and never waits for it.
  • list / tree / export give a due refresh up to 2 s per source.
  • AGENT_ATLAS_PRICING=offline disables the network.
  • A down, slow or redesigned source never blocks discovery.

The README's "no runtime outbound network" line is updated to say this is now the one outbound request.

Provenance. Every node has cost.provenance, listing the ledger entries its own calls were priced with: vendor, model, currency, per-class rates, long-context rates, from, observedAt, and source kind and URL. Every call carries the price id of its entry. Nodes also get cost.ownConfidence next to the subtree cost.confidence.

Unknown is never $0.00.

  • A node with no priced call is n/a. One with some unpriced calls, or an unpublished token class, is partial and renders as a lower bound, ≥ $x.
  • An unpriced call's cost is null.
  • A parent's subtree is n/a only if nothing under it is priced.
  • All UI figures go through one helper, src/ui/money.ts: session list, summary tiles, per-model and per-agent breakdowns, and the session's flow, trace, agent table and call dialog. The terminal tree and CLI list use costLabel().
  • The index row carries costConfidence and unknownModels. The summary counts unpriced sessions. A session whose cost is not fully known is never hidden by the min $ filter.

Price corrections that fall out of using the official pages:

  • Sonnet 5 stays at $2/$10. The old code switched it to $3/$15 on 2026-09-01, but Anthropic cancelled that increase, so every Sonnet 5 call since September was over-priced by 50%.
  • Fable 5.1 cache reads are $0.25/MTok (0.025×) and Opus 5.5 cache reads are $0.20/MTok (0.05×). The old uniform 0.1× cache-read assumption does not hold for either.
  • gpt-5.5 publishes no cache-write rate, so the guessed 1.25× is gone.
  • OpenAI long-context rates apply above 272K input tokens per request.

One adjacent fix, needed for re-pricing to reach the index: the index worker's change signature looked summaries up by bare session id while the map is keyed provider:id. It always missed, so a session whose cost changed never registered as changed.

Price sources: choice and trade-offs

Vendor (harness) Source
Anthropic (Claude Code) https://platform.claude.com/docs/en/about-claude/pricing.md, the Model pricing table
OpenAI (Codex) https://developers.openai.com/api/docs/pricing.md, the Standard table, short and long context
  • Why these. They are the vendors' own published prices, with no API key, no account and no new dependency. Both vendors serve a Markdown rendition of the page at a stable .md URL, which is far more stable to parse than HTML.
  • Considered and not chosen.
    • Vendor model APIs: Anthropic's /v1/models exposes capabilities, not prices, and needs a key.
    • Third-party datasets (LiteLLM's model_prices_and_context_window.json, OpenRouter's /api/v1/models): they add a second party that must keep up with new models and can lag or diverge from list price.
  • Trade-off. A page redesign breaks the parser. That is contained: the refresh fails loudly per source, is recorded in the cache file, and pricing continues from cache and snapshot. The fix is a parser change plus a snapshot refresh.
  • Not modelled.
    • Batch/Flex/Fast (priority) tiers and data-residency uplifts. Transcripts record speed/service_tier, but every call on this host is standard, so priced tiers are left for later.
    • gpt-5.6-sol's rate is the published promotional one, through at least 2026-11-21. When OpenAI changes it, the next refresh records the new rate from that moment on.

Verification

  • npm run typecheck, npm run build and npm test pass. The Chromium browser test is skipped locally because the runtime is unavailable there; CI runs it.
  • New test/pricing.test.mjs:
    • pins the dated prices, retrieved 2026-09-29, for Opus 5.5, Opus 5, Fable 5.1, Fable 5, Sonnet 5.5, Sonnet 5 (before and after 2026-09-01), Haiku 4.5, Opus 4.8, GPT-6 Astra/Sol/Luna, GPT-5.6 Sol/Terra/Luna and GPT-5.5;
    • checks the parsers against page fixtures and rejects a redesigned page;
    • checks that offline mode never fetches, that failed or malformed sources change no price, and TTL behaviour;
    • checks that a live price change applies only from its observation time and survives a later failure;
    • checks that an unknown model builds and renders as n/a and never $0.00 through the whole stack (price, tree, index row, breakdowns, min-cost filter, renderers).
  • Real sessions on the development host (trees built per session):
    • 22 recent Claude sessions containing claude-opus-5-5. On main, 16 of them show $0.00. On this branch, all 21 that build are priced (for example $3.00, $9.54, $31.11, $89.05) and none is $0. The 22nd transcript cannot be read on either branch (see below).
    • Codex threads on gpt-6-astra, gpt-6-sol, gpt-6-luna, gpt-5.6-sol/terra/luna and gpt-5.5 are priced. codex-auto-review, which no official page lists, is n/a.

Found, not fixed here (out of scope)

  • On the development host's corpus, the server's index scan fails on main as well, with Cannot create a string longer than 0x1fffffe8 characters, because the Claude parser reads a >512 MB transcript as one string. This is the memory and indexing work tracked in perf: bounded memory footprint — don't hold every session in memory (4 GB for ~2k sessions today) #25.
  • node src/cli.mjs list|tree crashes on main (ERR_INVALID_ARG_TYPE). cli.mjs still calls claude.buildTree(path) instead of going through the provider registry. This PR only touches its cost rendering and the bounded price refresh.

Closes #20

Replace the hardcoded price tables in src/prices.mjs with src/pricing/:
an append-only ledger of dated price entries built from the vendors'
official pricing pages (Anthropic and OpenAI, read as the Markdown they
publish), a bundled last-known snapshot, and a local cache refreshed at
most daily in the background.

- Unknown models were priced at $0 and rendered as "$0.00" (e.g. every
  claude-opus-5-5 session). Prices now come from the ledger, and a
  model with no price anywhere is "n/a" end to end: never $0.00 in the
  UI, the terminal tree or the CLI, never hidden by the min-cost filter.
- A live price change applies from the moment it was observed, so an
  already computed historical cost never moves; a failed or malformed
  refresh changes nothing and never blocks session discovery.
- Every node carries cost.provenance and every call its price id.
- Corrects Sonnet 5 (stays at $2/$10 after 2026-09-01), Fable 5.1 and
  Opus 5.5 cache-read rates, and drops the guessed gpt-5.5 cache-write
  rate; adds OpenAI long-context rates.
- Fix the index worker's change signature, which looked summaries up by
  the wrong key and never saw a re-priced session.

Closes #20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: dynamic provider pricing with provenance and cache policy

1 participant