Skip to content

docs(metrics): the occupancy pair comes from the fullest turn by ratio, not the largest used - #16674

Merged
iamwhatever merged 1 commit into
mainfrom
fix/metrics-spec-occupancy-fullest-ratio
Oct 4, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/metrics-spec-occupancy-fullest-ratio

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The metrics spec describes the occupancy pair the Session Breakdown tree draws the way the code actually picks it, so a reader cannot "correct" working code into a wrong fill ratio.

docs/system-specs/modules/metrics.md says the occupancy pair
peak_context_used / context_window comes from the turn/completed
"that reported the largest occupancy". context_trace in
src/kiro_crew/dashboard/handlers/usage.py picks the turn with the highest
used / used_window ratio, compared by cross-multiplication.

While every turn in the window shares one context window the two readings name the
same turn. A model switch moves the window, and from that point they name different
turns: 90k used against a 100k window is 90% full, 200k used against a 1M window is
20% full. Read as the largest absolute used, the spec's sentence crowns the second
turn; the code crowns the first.

The same paragraph also used the bare word occupancy for a token count, while line
856 of the same document defines it as "a per-session ratio". That is what made the
sentence readable two ways.

Why it matters

Nothing on the dashboard is wrong today, because the code's choice is the correct one.
The cost falls on whoever reads the spec next.

website/src/pages/SessionBreakdownTree.tsx:100 divides the pair into a clamped 0..1
fill fraction, so the Session Breakdown tree is asking for the fullest turn. A reader
who takes the spec at face value and "corrects" the ratio comparison to an absolute one
breaks that fill ratio: the panel would draw a 20%-full turn as the session's peak and
hide the turn that was actually about to compact. A spec sentence that reads as a defect
report against working code is an invitation to introduce the defect.

Not a goal

  • No behaviour change. The ratio pick in context_trace is correct and is not touched.
  • No new test. The behaviour the spec now describes is already pinned by four tests
    (listed under Tests); a new one would assert what they assert.
  • No change to the context block's own "ordered by peak occupancy" wording. That
    surface ranks by peak_pct, a ratio, so it was already accurate.
  • No tooling to prevent the next spec/code drift of this kind. That is a separate
    change, noted at the bottom.

What changed (motivation -> approach -> change)

Observed symptom: the spec and context_trace name different turns as the peak once a
model switch moves the context window.

Root cause: the word occupancy carries two quantities in this one document. At line
856 it is "a per-session ratio"; in this paragraph it was a token count. The sentence
"the turn that reported the largest occupancy" is therefore true under one definition
and false under the other, and nothing in the sentence said which one applied.

The code is the correct half, so the change is in the document.

The sentence now says the pair comes from the FULLEST turn inside the window and names
the quantity: the highest used / used_window ratio. It then says why it is the ratio
and not the largest absolute used, with the 90k/100k versus 200k/1M case written out,
so the next reader meets the model-switch trap in the spec instead of discovering it in
the code. Two properties the implementation already has are stated alongside it: the
comparison is integer-exact by cross-multiplication, and a reading whose window the
provider never stated has no ratio and so is not a peak candidate at all.

The overloaded word is resolved in the same paragraph: "occupancy is tokens" becomes
"the occupancy pair is tokens". Occupancy keeps its ratio meaning throughout the
document, and the unit statement now applies to the named pair rather than to the
concept.

Backwards compatibility

Compatible: the diff is one prose paragraph in a specification document. No code, no
schema, no payload field and no test changes, so nothing that works today stops working.

Removes nothing: no path is hidden, no record or row deleted, no validator tightened, no
data migrated or pruned, and no key or public function removed or renamed. The sole
changed file is docs/system-specs/modules/metrics.md.

Tests

No new test. The behaviour the spec now describes is already locked in by
test/metrics/test_context_trace.py:

  • test_peak_is_the_fullest_ratio_not_the_largest_used_across_a_switch -- two turns on
    different windows, 90k/100k versus 200k/1M; requires the 90% turn's pair and a
    rendered ratio of 0.9. This is the exact case the spec sentence got wrong.
  • test_the_occupancy_pair_comes_from_one_turn -- the fullest turn on the small window,
    a later emptier turn on a large one; requires the peak's own window to travel with it
    rather than the newest configured size.
  • test_peak_keeps_the_later_turn_among_equally_full_readings -- the >= tie-break
    keeps the later of two equally-full turns across a switch.
  • test_a_reading_whose_window_is_unknown_is_not_a_peak_candidate -- a windowless 80k
    reading loses to the valid 40k/100k pair, so no peak is reported over a zero window.

Run locally from that file: 4 passed, 53 deselected.

Manual verification

N/A -- documentation-only diff with no runtime surface. Verified instead by reading the
implementation against the new wording and by the repository's own documentation gates,
all run locally on this diff:

gate exit
scripts/docs_lint.py --test 0 (self-test passed)
scripts/docs-lint.sh 0 (all documentation checks passed)
scripts/check_comment_history.py --test 0 (21 flagged, 15 clean probes)
scripts/check_comment_history.py on this diff 0
.github/scripts/take_away_check.py on this diff 0 (no take-away shape)
flake8 on handlers/usage.py 0
mypy on handlers/usage.py 0 (Success: no issues found in 1 source file)

Related Issues

Fixes #16671

Found by Opus review on #15367 (merged).

Pattern harvest

Rule candidate: review-prompt

Pattern: a specification sentence states a selection rule for a pair of values without
naming the quantity it ranks by, while the document defines that quantity two ways
elsewhere -- so the sentence is true under one reading and false under the other, and
the code is the only place the real rule is written down.

Worth considering separately, and not in this PR: the spec states the ratio rule in
prose and the code states it in a comment, with nothing tying the two together. A
docs-lint rule pinning a named invariant sentence to the function implementing it would
catch the next drift of this kind.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 4, 2026 05:59
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d5e340120bdfe7ba0bc739930dd8aa064ece1e4e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] d5e3401

Description read

[DESCRIPTION-READ] def84cca4b12dbb6927610a3ac93852a77475e444deed179c937949b359bb571

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d5e340120bdfe7ba0bc739930dd8aa064ece1e4e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d5e3401

Verdict parsed from the review's SHA-scoped output markers for commit d5e340120bdfe7ba0bc739930dd8aa064ece1e4e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d5e340120bdfe7ba0bc739930dd8aa064ece1e4e: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d5e340120bdfe7ba0bc739930dd8aa064ece1e4e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d5e3401

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d5e340120bdfe7ba0bc739930dd8aa064ece1e4e: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/metrics-spec-occupancy-fullest-ratio branch from 1fbee86 to 4d47927 Compare October 4, 2026 08:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 4, 2026
…o, not the largest used

The spec described peak_context_used / context_window as coming from the
turn/completed that reported the largest occupancy. context_trace picks the
highest used / used_window ratio instead, so once a model switch moves the
window the two name different turns: 90k of a 100k window is fuller than 200k
of a 1M one.

The code is the correct half -- SessionBreakdownTree divides the pair into a
fill fraction, so the panel wants the fullest turn. Name the ratio in the spec,
write out the model-switch case, and state the two properties the comparison
already has: integer-exact cross-multiplication, and a reading with no stated
window is not a candidate.

Also resolve the overloaded word: occupancy means a ratio at the context block,
so the unit statement now applies to the occupancy pair rather than to
occupancy itself.
@chenmingwei23
chenmingwei23 force-pushed the fix/metrics-spec-occupancy-fullest-ratio branch from 4d47927 to d5e3401 Compare October 4, 2026 09:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: docs-only. Checked the new wording against the code at this head: dashboard/handlers/usage.py:869-886 picks the occupancy peak by the highest used/used_window ratio via cross-multiplication (used * peak_window >= peak_used * row_used_window), and a row with no stated used_window (row_used_window <= 0) is never a candidate. The spec now says exactly that, and the 90k/100k vs 200k/1M example shows why ranking by absolute used was wrong. CI green.

@iamwhatever
iamwhatever merged commit d39178b into main Oct 4, 2026
106 of 107 checks passed
@iamwhatever
iamwhatever deleted the fix/metrics-spec-occupancy-fullest-ratio branch October 4, 2026 16:33
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants