Repository navigation
feat(metrics): count crew log losses and folds without touching the append path - #22226
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS One check before merge: a gap between [DESIGN-REVIEWED] cc6cc8b Description read[DESCRIPTION-READ] 1b4461da21fc6d07bf16059c78c6641c0c736b4e2db430548b54e432432404d8 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: an operator can see whether the crew log is losing entries and how costly eager folds are, without adding work to the append path. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] cc6cc8b Description read[DESCRIPTION-READ] dfd97f1f67854178212cd1e3fcb9b32ca28e7d9ed6d600dcfcb31a5412f0cca3 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
f98fc3d to
32bc10b
Compare
Answering the First Principles advisory on
|
32bc10b to
dfd8f76
Compare
d8b77f6 to
8785513
Compare
buluoray
left a comment
There was a problem hiding this comment.
Reviewed head 8785513. Approve on the code; merging waits on the design decision for #22180, and this one should land before #22227.
The label sets are closed (cause, scope, origin, bucket, and fold from FOLD_NAMES), each folds onto other, and the tests catch it if the folds are loosened. No metric call, import or clock read is added to the append path, and test_a_dropped_wake_on_the_append_path_records_no_metric catches one being added.
One correction to the title and description: this PR does edit CrewLogWriter._drop, which runs on the inline submit path and in the writer thread give-up branches. The edit is arithmetic only: requeued follow-on jobs are no longer counted and then taken back out. The final count is unchanged and pinned by the existing writer tests, so please say this in the description instead of saying the append path is untouched.
Follow-up: RFC section 6.5 lists write.lost as a counter, the code registers an observable gauge, and the module docstring calls it an observable counter. Please make the three agree.
|
HOLD at 8785513. This PR conflicts with main. #22227 (batch 2) merged first and added the same To clear this:
— crew-pr-reviewer (r113) |
…ppend path
Three counters for crew log health: kirocrew.crew_log.write.lost{cause},
kirocrew.crew_log.fold.apply{scope,fold,bucket} and
kirocrew.crew_log.fold.start{scope,origin}.
Nothing is added to the crew log's append thread. Every write.lost cause is
detected on that thread or on a producer's own thread and is already counted
there, so the three counts are exported by one observable counter whose
callback runs on the metrics reader's ticker thread and imports nothing. The
read and the accrual happen under one lock hold because more than one reader
collects, and only growth is accrued so a reading that is corrected downwards
cannot become a permanent inflated total.
fold.apply and fold.start record only on the eager fold worker's thread, which
claims itself once at start. The fold branches leave a thread-local mark rather
than recording, because two of them hold a per-cell lock a reader can be
waiting on; the worker turns the marks into points outside every lock, so a
pass that advanced nothing counts its start and no apply.
CrewLogWriter._drop no longer counts an append it is about to take back out for
requeue. It relied on no reader seeing the count in between, which was true
while every reader was a caller on its own thread and false as soon as a
telemetry callback began sampling stats().
|
New head c9c7264. main now has register_crew_log_health_instruments, so the write.lost gauge registers inside it and the second function and its provider.py block are gone (First Principles item 7). Same three instruments, same consent gate; description updated. |
7f318df to
c9c7264
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at c9c7264.
The append path still gets no new work, and the writer.py _drop edit gives the same settled count, because every follow-on it no longer counts is requeued unconditionally. CI is green. Four items from the r114 HOLD are still open:
- Timing defect, unanswered.
note_fold_passcountsfold.startand then callsnote_fold_applyper fold, and each call reads the clock again (crew_log_health.py L681-L732). Folds from one pass can land in different buckets, and the firstget_recorder()build is charged to them. Smallest fix: computeelapsed_msonce at the top ofnote_fold_passand pass it to each apply. - Stale test docs. The test module docstring and the description's Tests and What was built sections still describe an import-time catalogue check and
note_fold_start. Both were removed. - Deferred
crew_log.fold.laghas no issue. Please open one and link it in the description. - Maintainer ruling. #22180 is still open, and #21270 holds child PRs until a maintainer confirms the data scope. That ruling should also settle the Design Review question: shipping
write.lostas a gauge instead of the RFC's counter.
Items 1-3 are yours to fix. Item 4 needs a maintainer.
— crew-pr-reviewer (r115)
note_fold_pass reads the clock once, before the fold.start point is recorded, and every fold.apply of that pass carries the same duration. Each apply used to read the clock again after the start point, so folds from one walk could land in different buckets and a first get_recorder() call, which can build the whole telemetry provider, was charged to the fold. Two tests pin it: a clock that moves on every read, and a recorder whose first use moves the clock 2s. The eager fold worker records its crew log fold counters through the same recorder as everything else. The end-to-end test keeps its exact equality over the whole stream, lists the fold.start and fold.apply points each pass adds, and freezes their clock so the bucket is fixed.
c9c7264 to
cc6cc8b
Compare
|
Answers to the review at c9c7264. New head: cc6cc8b.
All checks are green on cc6cc8b. |
|
HOLD at The items from our earlier change request are fixed. Blockers:
— crew-pr-reviewer (r116) |
Problem / Motivation
Goal: An operator watching their own machines can see whether the crew log is losing data and how expensive it has become to rebuild, instead of finding out from a dashboard that silently shows a stale value.
The crew log is the record every dashboard panel, conductor board and work ledger reads. Today nothing reports on it.
WriterStatscounts appends the writer gave up on and appends the buffer rejected,eager.pycounts fold wakes its queue could not hold, and all three numbers live and die inside one process with nothing sampling them. Folds are not timed at all, and there is no way to tell a fold that continued from a warm cell (cheap) from one that rebuilt from the first record of every unit (expensive, and the thing that makes a dashboard feel slow).So three questions have no answer on a running install: did the log drop anything, is folding getting slower, and are savepoints actually being kept?
Why it matters
A hole in the crew log is a hole in what the product can show.
droppedmeans the filesystem refused appends until the writer stopped trying;overflowedmeans the backlog grew until the buffer rejected the newest entries of a session. Both are real data loss that today is visible only to someone who attaches a debugger to the right process. A rising cold-fold share is the early signal for the class of complaint that arrives as "the dashboard got slow" with nothing to point at.The constraint is what makes this worth doing carefully rather than quickly: the crew log's append path is the path a user's turn waits on. Instrumentation that cost that path anything would trade a real slowdown for a number about slowdowns.
Not a goal
savepoint.write, theeager.queue_depthandwriter.bufferedgauges,bus.subscriber_error). feat(metrics): crew log health instruments, batch 2 #22227 shipped those and is on main; this PR adds its three instruments beside them in the samemetrics/crew_log_health.py, and its metrics-doc rows sit in a different part of the same table.crew_log.fold.lag. It is the one proposal in section 6.5 that needs atime.monotonic()on the append thread, which is exactly what this PR refuses to add. It waits for a measurement on a busy install, tracked in Metrics: crew_log.fold.lag (deferred from #22226) #22630.telemetry.enabledgate and the existing readers.What changed (motivation -> approach -> change)
Goal -> the design problem. Every signal worth having is produced on a path that must not pay for it. The three loss causes are detected on the crew log's append thread (
eager.note_commit'squeue.Fullbranch; the writer giving up after its attempt budget) or on a producer's own thread (submitrejecting at the memory ceiling). The fold timings are produced inside fold code that is shared between the eager fold worker and every dashboard read.Approach, and the alternatives rejected. Two mechanisms, each chosen over a simpler one that was wrong:
write.lostis an observable instrument, not a call site. Every cause is already counted by the subsystem that detects it. So the diff adds no call at those sites at all. One observable gauge is registered onprovider._build_recorder()'s live path, and its callback runs on the metrics reader's own ticker thread, reads the three counters, and folds each delta into a monotonic lifetime total (why a gauge and not a counter is below). The callback reads them throughsys.modulesrather than by importing: a process that never appended anything has nothing to report, which is exactly0, and a telemetry callback must not put the writer's import chain on a thread no caller is waiting on. The rejected alternative was a synchronouscounter.add(1)at each detection site, which is one lock-free add -- but on the one path in the system where "one more thing" is the thing we have been avoiding, and the number was already there to be read.fold.apply/fold.startrecord on the eager fold worker's thread and nowhere else. The worker claims its own thread once at start (_claim_thread_for_metrics), and both record sites begin with a thread-local check that returns immediately otherwise. The rejected alternative was recording wherever the fold ran, which would have made these counts of reads rather than of eager folding -- a different and much less useful question, and one that would have charged every dashboard request for a counter.What was built.
metrics/crew_log_health.py-- the three metric names, the closed enums, the duration ladder (shared with feat(metrics): crew log health instruments, batch 2 #22227'ssavepoint.write), the thread claim, the two record sites, thewrite.lostaccrual and the registration ofwrite.lostas an observable gauge. The module's docstring now has one section per batch.crew_log/eager.py-- a publicdropped_wakes()over the count_count_dropalready keeps; the thread claim in_run;_advanceand_advance_sessiontimed, each counting the folds it applied.note_commit,_enqueueand_count_dropare unchanged except for a comment saying why no metric is recorded in them.crew_log/projection_session.pyandprojection_slot_memo.py-- onenote_fold_origincall (andnote_fold_idlewhere a pass advanced nothing) in each of the branches that decides where a fold's state comes from (warm cell / savepoint on disk / cold rebuild). These calls only leave a thread-local mark and record nothing;note_fold_passturns the marks into points when the eager worker closes the pass, outside every lock.begin_fold_passreads the clock when a pass opens andnote_fold_passreads it once when the pass closes, before any point is recorded. Everyfold.applyof that pass carries that one duration, so folds from one walk share a bucket and a firstget_recorder()build is not charged to the fold.crew_log/writer.py--CrewLogWriter._dropis edited: it no longer adds a follow-on job it is about to requeue to_dropped_count, and_uncount_one_requeued_dropno longer takes it back out. Finding 2 below says why. No new work on the batch pass: onesumreplaceslenover the same list, under the lock it already holds.write.lostgauge registers inside the existingregister_crew_log_health_instruments, whichmetrics/provider.pyalready calls under its owntry, soprovider.pyis unchanged.Three design points worth a reviewer's attention.
No new histogram. A duration is a counter with a closed
bucketattribute (lt_10ms/lt_100ms/lt_1s/ge_1s, bounds exclusive, so exactly 10ms islt_100ms), so the shape survives a backend that does not keep explicit histogram buckets. The edges are written in exactly two places:bucket_forin the module, whichfold.applyshares with #22227'ssavepoint.write, and the metrics doc row.The
foldattribute is a closed catalogue with one owner.FOLD_LABELSisfrozenset(FOLD_NAMES), resolved once at import rather than per record, so the crew log's own catalogue is the only copy and a fold it gains is a fold this metric labels. An earlier revision spelled the 16 names out again here and raised at import when the two disagreed; Design Review, First Principles and an independent reviewer all landed on the same objection, and they were right. The fold surface imports this module at module scope, so that raise would have turned a forgotten label edit into a failed import ofprojection_session.pyandprojection_slot_memo.py-- every dashboard read, conductor board and session fold, plus every backend test that touches the crew log, failing at collection. It also contradicted this PR's own rule, whicheager.pyandprovider.pyfollow: a telemetry package that will not load costs the fold worker its metrics and nothing else. With no copy there is no drift to detect, which is a smaller surface than detecting it well.write.lostships as an observable GAUGE, which DEVIATES from the RFC table. Section 6.5 lists it as a counter. It cannot be one here, and the rule that forces that is a test:test/metrics/test_otlp_wire_e2e.py::test_no_instrument_exports_cumulativeasserts, over every exported instrument rather than a list, that not one Kiro Crew series is CUMULATIVE -- because a single cumulative series forces every consumer into stateful whole-hour aggregation, where a gauge or a DELTA sum merges one datapoint at a time.write.lostreads a total it does not own (the writer's and the fold queue's own counters), so it can only be an asynchronous instrument, and an observable counter keeps the SDK default of CUMULATIVE. Declaring DELTA for it instead is the movemetrics/temporality.pyrecords as tried and reverted: the delta baseline lives in the provider, the provider is rebuilt whenever telemetry consent changes, so the first collection after a rebuild would re-emit a whole process total as one delta and inflate the day's sum. That left one shape, and it is the shape this repo already chose for the identical problem -- the process CPU and GC readings were observable counters and are observable gauges declared as lifetime totals now, which is what removed the last cumulative series.So: the reading IS the process-lifetime total, it starts at zero for each process and therefore resets on restart, and a backend reads the loss RATE as the difference between two readings. The metrics-doc row says all of that at the place a reader looks. The total is accumulated by this module from the growth it reads, which buys three properties a raw lifetime total does not have: it is monotonic even when the underlying counter is replaced by a fresh one, the first collection is a baseline so losses accrued before telemetry was enabled are not reported, and the totals live in memory only, so nothing counted before a restart is counted again after it.
Review findings fixed after opening
note_commitstarts: that worker imports the module on purpose, through its thread claim, so the probe could fail on correct code.eager._runis now stubbed to a no-op inside the probe, which leaves the append path as the only importer and is the thing being measured; the test passes 5/5 runs in isolation. And the label-collapse case fornote_fold_applyleft_nowon the realtime.monotonic, so a 10ms scheduling pause moved the bucket and failed an assertion about labels; the clock is pinned.note_fold_startandstart_clock, whichnote_fold_passandbegin_fold_passreplaced. Both deleted; the tests now drive the real path (stamp an origin, close the pass), which is what the eager worker does.test_crew_log_exc_info_sitesflagged two newexc_info=Truesites ineager.py. Not an allowlist case: increw_loga retained traceback reaches its callers' frames throughf_back, and those frames bind openCrewLoghandles whose write leases a finalizer releases, so a record-keeping handler pins a lease for as long as it keeps the record._close_passruns in_advance'sfinally, squarely inside that chain. Both now go through the module's own_log_exc, which renders the traceback to text.Three defects found by review before this opened, and fixed
An independent reviewer saw only the diff, not this reasoning, and was asked to break the no-append-thread-work and no-over-count claims. It could not break the first. It broke the second three times, and each fix is in this diff:
Concurrent collection double-counted a loss.
write_lost_totalsread the three raw counters outside its lock and accrued inside it. Kiro Crew attaches more than one reader to one meter provider (the local JSONL sink, plus one per OTLP destination), each with its own ticker thread, so two collections overlap: the slower thread carries a stale reading in behind the fresher one, which reads as a counter that went down and then up again, and the same loss is added twice. Reproduced at 4 real losses reported as 9 and then 11. The read is now inside the lock -- three O(1) counter reads, and no path takes this lock while holding the writer's or the fold queue's, so it cannot deadlock.CrewLogWriter._droppublishes a count it corrects a moment later. It adds a whole failed batch to_dropped_count, then takes the follow-on jobs back out via_uncount_one_requeued_drop-- releasing and re-acquiring the lock in between, with alogger.warningin the gap. Its docstring reasons that "no reader can have seen the count yet", which was true while every reader was a caller on that same thread and false the moment a telemetry callback began samplingstats(). Treating the dip as a fresh counter added the whole new reading, turning one requeued job into a permanently inflated total. Fixed on both sides:_dropno longer counts a job it is about to requeue (so the transient is never published, and the docstring's claim is now true by construction rather than by assumption -- the per-unit marker debt is still adjusted over there, because that debt is consumed by a later pass of the same thread and no reader samples it), and_accruenow accrues only growth, so a reading corrected downwards can never inflate the total whatever the source does.A slot fold that advanced nothing counted an apply.
_advancerecorded an apply whenever the fold did not raise, but the slot fold's "nothing folded" branch returns an unchanged live cell. Three passes over an unchanging board reported three applies, sostartaboveapply-- the whole failing-and-idle-folding signal -- said nothing on the slot scope, while the session scope was already correct viawarm.changed. Fixing it also fixed a second finding from the same reviewer: the start was being recorded inside the per-cell fold lock, where a firstget_recorder()builds the whole telemetry provider synchronously and would hold that lock against a waiting reader. Both are gone because the fold branches no longer record at all: they leave a scope-keyed thread-local mark, and the eager worker turns it into points when it closes the pass, outside every lock. That mark also made two smaller findings disappear -- a nested fold (resolving a slot fold's units can fold a session log) can no longer be reported as the outer pass, and a session fold that retries after a mid-fold identity change now reports one start with the origin it finally folded from instead of two.One further finding was fixed as stated: the
write.lostregistration used to clear this module's state, which let a recorder build thatprovider._consent_workerthen discards zero the totals of the provider still serving. The reviewer's test criticisms are all addressed below.Backwards compatibility
Compatible: three new metric names, one new public function (
eager.dropped_wakes()), one new module. No existing instrument changes name, type, unit or attributes; no validator is tightened; no behaviour changes when telemetry is off, becauseget_recorder()is a no-op recorder and the observable gauge is never registered.One existing value changes, and it is a counter nothing could legitimately read before:
WriterStats.droppedno longer includes an append that is being requeued rather than lost. Previously it did, for the microseconds between_dropadding the batch and_uncount_one_requeued_droptaking the follow-ons back out. Every reader of that field is below; the value they see is unchanged, because the only observable difference is in a window_drop's own docstring asserted no reader could be in.src/kiro_crew/crew_log/emit.py::dropped_writes-- chat / cron / subagent / app / crew page, viaCrewLogappends -- still answers the settled count, which is what every caller compares before and after its own append.test_crew_log_edge_exhaustion.pyandtest_crew_log_emit_write_path.py(61 passed) cover the drop and requeue paths.src/kiro_crew/crew_log/writer.py::drain_for_shutdown/DrainReport-- release and shutdown path -- reportsloss_markers_owed, which is the per-unit marker debt and is still adjusted by_uncount_one_requeued_dropexactly as before. Only_dropped_countchanged.src/kiro_crew/metrics/crew_log_health.py::_read_raw_losses-- the metrics reader thread added by this PR -- is the reader that made the window observable in the first place, and the reason for the change.Removes nothing else: no path is hidden, no record or row is deleted, no key or public function is removed or renamed, and no data is migrated or pruned.
Tests
test/metrics/test_crew_log_health_losses_folds.py-- the shape of this PR's three instruments. (test/metrics/test_crew_log_health.pyis #22227's file and is unchanged.)otherwhen not, in all four enums; every declared label is its own collapse target, so no series is unreachable; and an out-of-enum value is pushed through the real record sites (note_fold_pass/note_fold_apply), not only through_closed-- testing that alone stays green if a call site stops calling it, which is the one gate against a session id becoming a series.bucket_for, and end to end through an injected clock (_nowis patched; nothing sleeps, and the 1s edge is not sleepable in a test anyway). A clock that stepped backwards still counts the fold.lt_10msand is read exactly twice (open and close); reading it again per fold would put the second fold inlt_100ms. A recorder whose first use moves the clock 2s, as a slow provider build would, leaves a 1ms pass inlt_10ms. Both cases fail on the previous head.foldlabel set isFOLD_NAMESitself, a name the crew log gains is labelled with no edit here, a non-fold collapses toother, and importing the module cannot fail over the catalogue: its source is executed as a fresh module under a patched catalogue that disagrees with everything, and it still loads.write.lostarithmetic: the first collection is a baseline and not a report; one loss is exactly one+1for its own cause (asserted per cause, with the other two asserted to stay at zero); a delta across two collections is counted once and a quiet cycle adds nothing; a counter replaced by a fresh one counts only the fresh losses; a reading corrected downwards adds nothing (the40 -> 41 -> 40 -> 42sequence that used to report 41 instead of 1); the total is monotonic and the exact totals are spelled out rather than onlyseen == sorted(seen), which would pass for an over-counting implementation; a restart reports nothing it already reported._lost_guard.locked()is true during every read), so moving the read out of the lock reddens with no thread and no timing. A second case runs two real collections ordered by events, not by time: the first is held inside its read while the second starts behind it, both totals are exact, and a third collection adds nothing.sys.modulesdeleted under the test) -- which is what proves the callback imports nothing.MetricsRecorder(None)as the gate, no instrument is created and no state is held. A recorder that raises costs the metric and not the fold.test/test_crew_log_health_threads.py(21 cases) -- the thread constraint, over the real paths rather than the helpers:note_commitin a subprocess and assertskiro_crew.metrics.crew_log_healthis not insys.modulesafterwards. A recorder spy on the test's own thread proves little, because every record site returns at once on an unclaimed thread -- so adding anote_fold_*call tonote_commitwould leave the spy empty and the suite green while adding exactly the work this PR promises not to add. No thread-local gate can satisfy the subprocess assertion.on_fold_worker_threadis spied on directly and asserted never to be called fromnote_commit, so a call that was merely gated off would still fail.dropped_wakes()by exactly the number dropped and records nothing; the happy path records nothing (withdrainwaited on, so the assertion is not racing the worker); and the hand-off works end to end -- the count the append path kept becomes thewrite.losttotal read from the other side._runwith the stop flag already up so the claim is the only thing that happens, and asserting the claimed thread object is that worker.eager._advance/_advance_session, for both scopes: cold then warm for a slot fold and for a session fold, andsavepointfor each by growing the unit past the savepoint floor, dropping the in-memory cell, and folding again. One pass records exactly one start.fold.apply: one point per applied slot fold with its name; one point per session fold that moved and no duplicates; an idle slot pass records its start and no apply (three passes over an unchanging board give three starts and one apply); a fold that raises after choosing its start records that start and no apply, with the raise planted past the start branch so the pass really gets that far -- replacingread_slot_projectionwholesale would mean the fold never started and the assertion would hold for the wrong reason; a fold that never started records nothing at all; a read that got there first leaves the worker nothing to apply, which pins the documented meaning of the ratio; and every point is stamped with the thread that folded.Existing suites run for regressions, all green:
test_crew_log_health_losses_folds.py+test_crew_log_health_threads.py+test_crew_log_fold_eager.py+test_crew_log_slot_folds.py+test_crew_log_lazy_exports.py+test_perf_boot_path.py-- 188 passed. The last two are the lazy-seam and import-weight gates that a module-scope import in the fold surface could have broken.writer.pychange:test_crew_log_edge_exhaustion.py+test_crew_log_emit_write_path.py+test_crew_log_edge_fanout.py(61 passed) andtest_crew_log_emit.py+test_crew_log_emit_continuity.py+test_crew_log_edge_throttle.py+crew_log_drain.py(113 passed).test_crew_log_projection.py+metrics/test_provider.pygreen in an earlier round.Manual verification
The observable gauge was collected through a real
MeterProviderwith anInMemoryMetricReaderto confirm the SDK path works as described and not only the accrual in isolation: the instrument reports no temporality on the wire (which is what makes it a gauge and keeps it out oftest_no_instrument_exports_cumulative), the first collection reported0for all three causes, the next reporteddropped=3/wake_dropped=1against raised counts, and a third collection with no further loss reported the same totals rather than new points.flake8,isortandblackare clean on every changed and added file, andscripts/docs_lint.pypasses.Related Issues
Part of #21270 (crew usage metrics). This PR implements the first batch of the crew log health section.
Provenance, stated plainly because a review lane was right to ask. The "crew usage metrics wave 2" RFC that specifies these three counters is NOT on main: it is still in review and still
status: draft, so a grep ofdocs/request-for-change/on the base finds nothing, and references to "section 6.5" above resolve only against that in-review document. The tracking issue #21270 is the record that exists today. No production slow-fold report backsfold.apply/fold.starteither -- they exist to make the cold-fold share and the fold duration observable for the first time, which is a measurement this repo cannot currently take, not a fix for a reported regression. Whether that is enough to land the two fold counters now, or whether they should wait for the RFC to be on main with a non-draft status, is a decision for a maintainer rather than something this PR should settle by assertion;write.lostdoes not depend on either of them.Checklist