Repository navigation
feat(metrics): count work ledger goals, items, dispatches and reports - #22219
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 4e133e1 Description read[DESCRIPTION-READ] 134a5444ed4b93ae99e66bf88fdc8d40114e4c5be1d3c0b8e6c664581d559108 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: make the work ledger's activity measurable in aggregate (how many goals and items open, how items end and how long they take, how often work is dispatched, what workers report back) — ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 4e133e1 Description read[DESCRIPTION-READ] 2a444eeb723ecdeaedb836d2d4844ea02d488c7b4fe5e1460339e66655e6b7e6 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
014b4a4 to
90c16f2
Compare
90c16f2 to
e8ad6dd
Compare
|
Answering First Principles items 4 and 5 ( Both values stay, and the reason is not symmetry with a sibling enum. Each one is a member of a crew-log entry contract that already exists on main, independent of this PR:
What dropping them would cost is the thing the enum exists to prevent. The declaration is not an unchecked assertion either. Two tests pin it against the contracts rather than against this module's own text, so a change on either side reddens here:
If either contract loses a member, the matching test fails and the enum here is narrowed with it. That is the condition under which these values should go away, and it is mechanised. Items 3 and 7 were taken. Item 7 was correct: |
e8ad6dd to
0fbad42
Compare
0fbad42 to
f1a11a6
Compare
Four counters behind telemetry consent, each emitted at the work ledger's
single commit point after its crew-log record landed:
- kirocrew.work.created{kind,depth}
- kirocrew.work.item_closed{state,fails,duration}
- kirocrew.work.dispatches{target_kind}
- kirocrew.work.reports{status,reason}
Durations and counts are bucketed counters rather than histograms, every
attribute value comes from a closed enum declared in metrics/work.py, and
no item id, title, summary or acceptance text reaches an attribute.
f1a11a6 to
4e133e1
Compare
buluoray
left a comment
There was a problem hiding this comment.
Reviewed head 4e133e1. Approve on the code; merging waits on the design decision for #22180.
All four counters carry only closed attributes: kind, depth, state, fails, duration, target_kind, status and reason, with unknown values mapped to other. No item id, title, summary or acceptance text is emitted. Counts are taken from the committed record after the work/recorded entry lands, and goal is counted only on the write that opened the board (goal_version 1, which every later goal write increments). Letting an arbitrary string through _closed turns test_every_record_function_is_total_over_hostile_input red. The related work ledger tests pass on this head (345).
Follow-ups, not blocking:
- Nothing pins that a ledger write survives a failing recorder. Making emit_counter re-raise leaves every test in test_work_ledger_metrics.py green. Since _count_recorded and record_report run on the committed path with no local guard, please add a test that patches the recorder to raise and asserts the record and report routes still answer 200, or wrap both in a try/except that logs at debug.
- The dispatch and report counts are taken at different points than RFC 22180 section 6.3 names (bind instead of after the crew/dispatch write, and after work/recorded instead of after crew/report). The reasons are sound, but the RFC row should be updated to match.
Problem / Motivation
Goal: make the conductor work ledger measurable -- how many goals and items a conductor opens, how its items end and how long they take, how often work is handed to a worker, and what workers report back.
The work ledger has no instrument at all. Every write already lands as a
work/recorded,crew/dispatchorcrew/reportentry in the crew log, but nothing counts them, so none of these questions can be answered today: how do items end, how long does an item stay open, how deep do conductor trees that dispatch actually go, how many reports come back blocked or waiting on a person.Why it matters
The work ledger is how a conductor and its workers share state, and it is the mechanism every multi-session goal runs on. Without numbers there is no way to tell a healthy board from a stuck one in aggregate: whether items mostly get accepted or mostly abandoned, whether work sits open for minutes or for days, whether workers spend their reports blocked on a person. Those four shapes are what decide where the ledger needs work next, and each of them is currently invisible.
Not a goal
What changed (motivation → approach → change)
Why counters at the write's commit point rather than a fold over the crew log. A counting fold over
work/recordedwould be the cheaper source, and it is how session usage is counted. It cannot serve here: the crew log'sworkfold keeps only the newestWORK_CLOSED_WINDOWclosed items whole and forgets the rest, keeping just a total. An item'sstate,failsand open duration are therefore gone from the fold by the time any later reader looks at it. Those three values exist only at the moment the close commits, so that is where they are counted.Where exactly the commit point is. Both write routes already treat one line as the point of no return: the
work/recordedentry landing in the crew log (on_work_recordedreturningTrue). Every refusal returns before it, and a cache commit whose record would not land is undone and also returns before it. Each counter sits immediately after that line, so a refused write counts nothing and a committed write counts exactly once. Nothing on the rebuild path is instrumented, sowork_ledger_rebuildre-materialising a board from the crew log re-counts nothing.What was built. A new
src/kiro_crew/metrics/work.pydeclares the four instruments, their closed attribute enums and their bucket functions;dashboard/handlers/work_ledger.pycalls it from the two routes through one small_count_recordedhelper and one line in the report route.kirocrew.work.createdkind(goal/item),depth(0/1/2+)kirocrew.work.item_closedstate(accepted/rejected/abandoned),fails(0/1/2/3+),duration(le_10m/le_1h/le_4h/gt_4h)kirocrew.work.dispatchestarget_kind(session/crew)kirocrew.work.reportsstatus(progress/done/blocked/question/failed),reason(approval/needs_human/none)Four decisions inside that are worth naming, because each one is a place the obvious reading would be wrong:
created{kind=goal}. The committed header carriesgoal_version == 1on exactly that write and higher on every reword, round bump, cap raise and explicitnew_goalafter it. Counting the later ones would make a counter named for boards report goal writes instead -- a conductor that rewords its goal five times would read as five conductors. What those later writes cost is visible in the items they create. A board purged and reopened on the same slot does count again, and should: the purge takes the header with it, so the reopened board's own first goal write carriesgoal_version == 1from a fresh record. A rebuild is the other way round and also right -- it keeps the version the log holds, so a re-materialised board is not counted twice. Both directions are pinned by tests.bucketattribute whose edges are stated in code and docs. This is not a claim that the repo cannot express a duration histogram -- it plainly can, andkirocrew.recovery.duration_secsis one, with provider bucket views behind it. It is that the wave deliberately adds no more of them, so that every number it ships survives a reader that keeps only counter series. The edges live in one place,DURATION_BUCKET_EDGE_SECS, and are restated beside the doc row.depthandfailsare small integers that can in principle grow, so they are bucketed the same way rather than emitted raw.bind, not on thecrew/dispatchecho written just after it. That echo is best-effort and is skipped entirely for a board driven from a slot that belongs to no crew, so counting on it would silently drop every dispatch such a board made while the board plainly made one.duration=other, never the smallest bucket. An item whose stamps cannot be read was not open for under ten minutes; a plausible-looking reading is worse than no reading. A span that comes out NEGATIVE is different and does read as the smallest bucket: both stamps are the store's own, so a close stamped before its own create is the host's clock moving between two writes, not an unmeasurable item.Two enum members are declared but never emitted by these routes, both of them values the RFC's own attribute table names:
target_kind=crew, which the crew log's dispatch contract also names (CREW_TARGET_KINDS), andstatus=failed, which its report contract names (CREW_REPORT_STATUSES) while the ledger's worker half can commit only the other four. Declaring them keeps a second target kind or afailedwriter landing in its own series instead of inother, and both are pinned against those contracts by tests so a drift reddens here rather than quietly reshaping real traffic.There is no try/except around any of the five emits.
emit_counteralready swallows every failure of the emit itself after a debug log, so the only code a wrapper here would cover is the bucketing -- and the bucketing is written to be total: every input, including a type that is not a value at all, maps to a declared member of its own enum rather than raising. Two tests hold that property instead, so it is checked rather than wrapped.Recording is an in-memory add on a thread the route already owns: no disk write, no network call, no log line, nothing new on the crew log's append thread. Telemetry consent gates it -- with telemetry off the provider hands back a no-op recorder and the meter is never touched.
Backwards compatibility
Compatible: no route gains or loses a refusal, a field or a response shape; the four counters are new series and no existing instrument changes name, type or attributes.
Removes nothing: the diff is three additive call sites, one new module and four new rows in the metrics doc. No path is hidden, no record or row is deleted, no validator is tightened, and no key or public function is removed or renamed.
Tests
test/test_work_ledger_metrics.py, 49 cases, two halves.The bucket and enum functions directly, on their own edges and on every shape a stored record can hold that is not a value at all:
le_10mand 601 s isle_1h, the same at 3600 s and 14400 s.depth0 / 1 / 2 / 9,fails0 / 1 / 2 / 3 / 4 / 99.bool, a numeric string, a float and an absent value each map toother.Trueis pinned specifically: it is anintsubclass, so a bucket comparing magnitudes alone would report it as depth 1.other; a close stamped before its create yields the smallest bucket; a stamp with no UTC offset still measures against one that has it.record_*function is total over ten hostile inputs, emitting a declared value for each attribute and never raising. That second test is the guarantee standing in place of a try/except around each emit: delete a_closedor_is_countguard and it raises.CLOSED_STATESequals the ledger's own terminal states;REPORT_STATUSESequalsCREW_REPORT_STATUSES;TARGET_KINDSequalsCREW_TARGET_KINDS; every declared value is a short lower-case token (no id, path, name or free text).The call sites through the routes, which is the only way to prove counted-once:
new_goalall count nothing; a board purged and reopened on the same slot counts again.2+rather than falling intoother.decideandverdictcount nothing of their own.blockedwith no reason given carrying the literalnone; adonerefused for missing evidence, afailedstatus the route refuses, and a report on a closed item each count nothing.emit_counterruns and the meter is never touched; with consent the same five writes create their instruments.Also run green:
test_work_ledger_projection.py,test_work_ledger_projection_board_lifecycle.py,test_work_ledger_tools.py(159 cases),test/metrics/test_telemetry_column_taxonomy.py,scripts/docs_lint.py, the repository's ownsemgrep/rule set over the changed files, and flake8 + isort + black.Manual verification
N/A -- unit coverage sufficient: every assertion above drives the real routes against a real store and a real crew log, including the refusal, undo, purge and rebuild paths, so there is no surface a manual run would reach that the suite does not.
Related Issues
Refs #21270