Skip to content

feat(metrics): count work ledger goals, items, dispatches and reports - #22219

Merged
dodgeblaster merged 1 commit into
mainfrom
feat/21270-otel-work-ledger
Oct 19, 2026
Merged

dodgeblaster merged 1 commit into
mainfrom
feat/21270-otel-work-ledger

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: make the conductor work ledger measurable -- how many goals and items a conductor opens, how its items end and how long they take, how often work is handed to a worker, and what workers report back.

The work ledger has no instrument at all. Every write already lands as a work/recorded, crew/dispatch or crew/report entry in the crew log, but nothing counts them, so none of these questions can be answered today: how do items end, how long does an item stay open, how deep do conductor trees that dispatch actually go, how many reports come back blocked or waiting on a person.

Why it matters

The work ledger is how a conductor and its workers share state, and it is the mechanism every multi-session goal runs on. Without numbers there is no way to tell a healthy board from a stuck one in aggregate: whether items mostly get accepted or mostly abandoned, whether work sits open for minutes or for days, whether workers spend their reports blocked on a person. Those four shapes are what decide where the ledger needs work next, and each of them is currently invisible.

Not a goal

  • Where these numbers are sent, and which of them leave the machine. That is decided separately, by the export endpoint design, not here.
  • "Stale" work -- a worker that has not reported for hours. It is a derived read-time flag, not a write, so it has no commit point to count at.
  • Per-session, per-item or per-person attribution. No attribute here carries an id, a slot key, a title, a summary or any user text.
  • Counting decisions and verdicts. Neither carries a number this change asks for; an item's fail count is read off the close it leads to.
  • Changing any ledger behaviour. No route gains or loses a refusal, a field or a response shape.

What changed (motivation → approach → change)

Why counters at the write's commit point rather than a fold over the crew log. A counting fold over work/recorded would be the cheaper source, and it is how session usage is counted. It cannot serve here: the crew log's work fold keeps only the newest WORK_CLOSED_WINDOW closed items whole and forgets the rest, keeping just a total. An item's state, fails and open duration are therefore gone from the fold by the time any later reader looks at it. Those three values exist only at the moment the close commits, so that is where they are counted.

Where exactly the commit point is. Both write routes already treat one line as the point of no return: the work/recorded entry landing in the crew log (on_work_recorded returning True). Every refusal returns before it, and a cache commit whose record would not land is undone and also returns before it. Each counter sits immediately after that line, so a refused write counts nothing and a committed write counts exactly once. Nothing on the rebuild path is instrumented, so work_ledger_rebuild re-materialising a board from the crew log re-counts nothing.

What was built. A new src/kiro_crew/metrics/work.py declares the four instruments, their closed attribute enums and their bucket functions; dashboard/handlers/work_ledger.py calls it from the two routes through one small _count_recorded helper and one line in the report route.

Metric Attributes
kirocrew.work.created kind (goal / item), depth (0 / 1 / 2+)
kirocrew.work.item_closed state (accepted / rejected / abandoned), fails (0 / 1 / 2 / 3+), duration (le_10m / le_1h / le_4h / gt_4h)
kirocrew.work.dispatches target_kind (session / crew)
kirocrew.work.reports status (progress / done / blocked / question / failed), reason (approval / needs_human / none)

Four decisions inside that are worth naming, because each one is a place the obvious reading would be wrong:

  • Only the goal write that OPENS a board counts as created{kind=goal}. The committed header carries goal_version == 1 on exactly that write and higher on every reword, round bump, cap raise and explicit new_goal after it. Counting the later ones would make a counter named for boards report goal writes instead -- a conductor that rewords its goal five times would read as five conductors. What those later writes cost is visible in the items they create. A board purged and reopened on the same slot does count again, and should: the purge takes the header with it, so the reopened board's own first goal write carries goal_version == 1 from a fresh record. A rebuild is the other way round and also right -- it keeps the version the log holds, so a re-materialised board is not counted twice. Both directions are pinned by tests.
  • A duration is a bucketed counter, not a histogram, because that is this wave's stated rule. The metrics RFC this implements fixes one instrument rule for every metric in it: counter or gauge only, no new histogram, and a duration or size is a counter with a closed bucket attribute whose edges are stated in code and docs. This is not a claim that the repo cannot express a duration histogram -- it plainly can, and kirocrew.recovery.duration_secs is one, with provider bucket views behind it. It is that the wave deliberately adds no more of them, so that every number it ships survives a reader that keeps only counter series. The edges live in one place, DURATION_BUCKET_EDGE_SECS, and are restated beside the doc row. depth and fails are small integers that can in principle grow, so they are bucketed the same way rather than emitted raw.
  • A dispatch is counted on the bind, not on the crew/dispatch echo written just after it. That echo is best-effort and is skipped entirely for a board driven from a slot that belongs to no crew, so counting on it would silently drop every dispatch such a board made while the board plainly made one.
  • An unreadable or missing stamp yields duration=other, never the smallest bucket. An item whose stamps cannot be read was not open for under ten minutes; a plausible-looking reading is worse than no reading. A span that comes out NEGATIVE is different and does read as the smallest bucket: both stamps are the store's own, so a close stamped before its own create is the host's clock moving between two writes, not an unmeasurable item.

Two enum members are declared but never emitted by these routes, both of them values the RFC's own attribute table names: target_kind=crew, which the crew log's dispatch contract also names (CREW_TARGET_KINDS), and status=failed, which its report contract names (CREW_REPORT_STATUSES) while the ledger's worker half can commit only the other four. Declaring them keeps a second target kind or a failed writer landing in its own series instead of in other, and both are pinned against those contracts by tests so a drift reddens here rather than quietly reshaping real traffic.

There is no try/except around any of the five emits. emit_counter already swallows every failure of the emit itself after a debug log, so the only code a wrapper here would cover is the bucketing -- and the bucketing is written to be total: every input, including a type that is not a value at all, maps to a declared member of its own enum rather than raising. Two tests hold that property instead, so it is checked rather than wrapped.

Recording is an in-memory add on a thread the route already owns: no disk write, no network call, no log line, nothing new on the crew log's append thread. Telemetry consent gates it -- with telemetry off the provider hands back a no-op recorder and the meter is never touched.

Backwards compatibility

Compatible: no route gains or loses a refusal, a field or a response shape; the four counters are new series and no existing instrument changes name, type or attributes.

Removes nothing: the diff is three additive call sites, one new module and four new rows in the metrics doc. No path is hidden, no record or row is deleted, no validator is tightened, and no key or public function is removed or renamed.

Tests

test/test_work_ledger_metrics.py, 49 cases, two halves.

The bucket and enum functions directly, on their own edges and on every shape a stored record can hold that is not a value at all:

  • Each duration edge is inclusive of itself: 600 s is le_10m and 601 s is le_1h, the same at 3600 s and 14400 s.
  • depth 0 / 1 / 2 / 9, fails 0 / 1 / 2 / 3 / 4 / 99.
  • A negative, a bool, a numeric string, a float and an absent value each map to other. True is pinned specifically: it is an int subclass, so a bucket comparing magnitudes alone would report it as depth 1.
  • An absent, empty or unparseable stamp yields other; a close stamped before its create yields the smallest bucket; a stamp with no UTC offset still measures against one that has it.
  • Every bucket function answers only inside its own declared enum, across a sweep of in-range, out-of-range and non-numeric inputs; and every record_* function is total over ten hostile inputs, emitting a declared value for each attribute and never raising. That second test is the guarantee standing in place of a try/except around each emit: delete a _closed or _is_count guard and it raises.
  • CLOSED_STATES equals the ledger's own terminal states; REPORT_STATUSES equals CREW_REPORT_STATUSES; TARGET_KINDS equals CREW_TARGET_KINDS; every declared value is a short lower-case token (no id, path, name or free text).

The call sites through the routes, which is the only way to prove counted-once:

  • One goal that opens a board counts once at its depth; a reword, a round bump, a cap raise and an explicit new_goal all count nothing; a board purged and reopened on the same slot counts again.
  • Each create counts one item; a second-level board's creates count at depth 1; a stored header holding a depth past the cap counts in 2+ rather than falling into other.
  • A create refused past the goal's concurrency cap, a cap outside its range and a title past its cap each count nothing.
  • A close counts its state, fails bucket and duration bucket from the committed record; each terminal state, each fails edge (0 / 1 / 2 / 3 / 11 through verdicts) and each duration bucket (5 min / 42 min / 3 h / 9 h) is reached; closing an already-closed item counts no second close.
  • A bind counts one dispatch to a session; a second bind of the same item counts none; decide and verdict count nothing of their own.
  • Each report status counts once with its reason, including blocked with no reason given carrying the literal none; a done refused for missing evidence, a failed status the route refuses, and a report on a closed item each count nothing.
  • A write whose record did not land counts nothing (the route's own undo path).
  • A rebuild from the crew log, after a goal, create, bind, report and close, adds no count at all.
  • With telemetry off the real emit_counter runs and the meter is never touched; with consent the same five writes create their instruments.

Also run green: test_work_ledger_projection.py, test_work_ledger_projection_board_lifecycle.py, test_work_ledger_tools.py (159 cases), test/metrics/test_telemetry_column_taxonomy.py, scripts/docs_lint.py, the repository's own semgrep/ rule set over the changed files, and flake8 + isort + black.

Manual verification

N/A -- unit coverage sufficient: every assertion above drives the real routes against a real store and a real crew log, including the refusal, undo, purge and rebuild paths, so there is no surface a manual run would reach that the suite does not.

Related Issues

Refs #21270

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4e133e163d25e36e82eb8bbe3acd70cad8ba357a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4e133e163d25e36e82eb8bbe3acd70cad8ba357a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 4e133e1

Description read

[DESCRIPTION-READ] 134a5444ed4b93ae99e66bf88fdc8d40114e4c5be1d3c0b8e6c664581d559108

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 4e133e163d25e36e82eb8bbe3acd70cad8ba357a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

kirocrew.work.dispatches{target_kind} is an attribute that only ever has one value, and the "no histogram" rule rests on an RFC the base commit does not carry.

Not justified as shipped

  • Item 4 — one consumer, generalized: the only production caller is work_metrics.record_dispatch("session") (1 call site found by grepping record_dispatch(). The crew member is "declared because… a second target kind must land in its own series". That is a "so we can later" reason. Today the attribute carries no information.
    Subtraction: delete the target_kind parameter and attribute and TARGET_KINDS, and count bare dispatches. Do the same for status=failed in REPORT_STATUSES, which the description says "this route never emits".
    Clears when: a second production caller passes a different target kind, or the attribute is removed.
  • Item 2 — inherited: the duration buckets are justified only by "The metrics RFC this implements fixes one instrument rule… no new histogram" and by "rule 2 of the RFC requires". The base RFC status list has no metrics or telemetry RFC, and no doc on base contains "no new histogram" or "counter or gauge".
    Clears when: the RFC that states this rule is merged on base and linked.

What this change ships

Inventory (6 items) — 4 justified

Intent: make the work ledger's activity measurable in aggregate (how many goals and items open, how items end and how long they take, how often work is dispatched, what workers report back) — ADDITION.

  1. Each board opened and each item created is counted, by depth — justified
  2. Each item close is counted by outcome, fail count and a bucketed open duration — inherited: the cited metrics RFC is absent on base
  3. Goal rewords, round bumps and cap raises don't count as new boards — justified
  4. Each dispatch is counted, labelled with a target kind that is always session — one consumer, generalized
  5. Each committed worker report is counted by status and reason — justified
  6. The four metrics are documented in the metrics spec — justified

[FIRST-PRINCIPLES-REVIEWED] 4e133e1

Description read

[DESCRIPTION-READ] 2a444eeb723ecdeaedb836d2d4844ea02d488c7b4fe5e1460339e66655e6b7e6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4e133e163d25e36e82eb8bbe3acd70cad8ba357a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4e133e1

Verdict parsed from the review's SHA-scoped output markers for commit 4e133e163d25e36e82eb8bbe3acd70cad8ba357a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4e133e163d25e36e82eb8bbe3acd70cad8ba357a: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4e133e163d25e36e82eb8bbe3acd70cad8ba357a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4e133e1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4e133e163d25e36e82eb8bbe3acd70cad8ba357a: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/21270-otel-work-ledger branch from 014b4a4 to 90c16f2 Compare October 19, 2026 16:30
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/21270-otel-work-ledger branch from 90c16f2 to e8ad6dd Compare October 19, 2026 17:10
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Answering First Principles items 4 and 5 (target_kind=crew, status=failed) once, since the finding is advisory and the shape is deliberate.

Both values stay, and the reason is not symmetry with a sibling enum. Each one is a member of a crew-log entry contract that already exists on main, independent of this PR:

  • crew/dispatch declares target_kind as a CLOSED enum of session / crew (src/kiro_crew/crew_log/entry_types.py::CREW_TARGET_KINDS). The registry refuses a dispatch whose target names a third kind, so crew is not a hypothetical value -- it is one of exactly two the log will accept.
  • crew/report declares status over CREW_REPORT_STATUSES, which is its own spec's four (done / blocked / failed / progress) plus the work ledger's worker statuses. failed comes from the report contract, not from the ledger's worker half -- which is why the description says plainly that this route never emits it today.

What dropping them would cost is the thing the enum exists to prevent. _closed() maps any value outside the declared tuple to other. So on the day a second dispatch target or a failed writer appears, a narrowed enum would not fail loudly -- it would silently fold that traffic into other, where it is indistinguishable from a genuine mapping failure, and the series would under-report a real population while looking healthy. Declaring the value is what makes that traffic land in its own series the moment it exists.

The declaration is not an unchecked assertion either. Two tests pin it against the contracts rather than against this module's own text, so a change on either side reddens here:

  • test_the_target_kinds_match_the_crew_logs_own_dispatch_contract asserts TARGET_KINDS == CREW_TARGET_KINDS.
  • test_the_report_enums_match_the_crew_logs_own_report_contract asserts set(REPORT_STATUSES) == set(CREW_REPORT_STATUSES) and, separately, that failed is NOT in WORK_WORKER_STATUSES -- so the claim "this route cannot emit it" is itself a test, not a comment.

If either contract loses a member, the matching test fails and the enum here is narrowed with it. That is the condition under which these values should go away, and it is mechanised.

Items 3 and 7 were taken. Item 7 was correct: emit_counter never raises, so the five try/except wrappers covered only the bucketing, and the bucketing is total. The wrappers are deleted and test_every_record_function_is_total_over_hostile_input now holds that property instead -- ten hostile inputs through all five record_* functions, asserting a declared value for every attribute and no raise. Item 3's premise was right about the repo: kirocrew.recovery.duration_secs is a duration histogram with provider bucket views behind it. The description's justification has been corrected to cite the real reason, which is the metrics RFC's own instrument rule for this whole wave (counter or gauge only, no new histogram, a duration expressed as a counter with a closed bucket attribute whose edges are stated in code and docs) rather than a claim about backend support that the repo contradicts.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/21270-otel-work-ledger branch from e8ad6dd to 0fbad42 Compare October 19, 2026 17:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/21270-otel-work-ledger branch from 0fbad42 to f1a11a6 Compare October 19, 2026 17:45
Four counters behind telemetry consent, each emitted at the work ledger's
single commit point after its crew-log record landed:

- kirocrew.work.created{kind,depth}
- kirocrew.work.item_closed{state,fails,duration}
- kirocrew.work.dispatches{target_kind}
- kirocrew.work.reports{status,reason}

Durations and counts are bucketed counters rather than histograms, every
attribute value comes from a closed enum declared in metrics/work.py, and
no item id, title, summary or acceptance text reaches an attribute.
@chenmingwei23
chenmingwei23 force-pushed the feat/21270-otel-work-ledger branch from f1a11a6 to 4e133e1 Compare October 19, 2026 18:07
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 4e133e1. Approve on the code; merging waits on the design decision for #22180.

All four counters carry only closed attributes: kind, depth, state, fails, duration, target_kind, status and reason, with unknown values mapped to other. No item id, title, summary or acceptance text is emitted. Counts are taken from the committed record after the work/recorded entry lands, and goal is counted only on the write that opened the board (goal_version 1, which every later goal write increments). Letting an arbitrary string through _closed turns test_every_record_function_is_total_over_hostile_input red. The related work ledger tests pass on this head (345).

Follow-ups, not blocking:

  • Nothing pins that a ledger write survives a failing recorder. Making emit_counter re-raise leaves every test in test_work_ledger_metrics.py green. Since _count_recorded and record_report run on the committed path with no local guard, please add a test that patches the recorder to raise and asserts the record and report routes still answer 200, or wrap both in a try/except that logs at debug.
  • The dispatch and report counts are taken at different points than RFC 22180 section 6.3 names (bind instead of after the crew/dispatch write, and after work/recorded instead of after crew/report). The reasons are sound, but the RFC row should be updated to match.

@dodgeblaster
dodgeblaster merged commit beacdbc into main Oct 19, 2026
105 checks passed
@dodgeblaster
dodgeblaster deleted the feat/21270-otel-work-ledger branch October 19, 2026 21:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 19, 2026
@buluoray buluoray mentioned this pull request Oct 20, 2026
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants