Skip to content

feat(metrics): count crew log losses and folds without touching the append path - #22226

Merged
iamwhatever merged 2 commits into
mainfrom
feat/crew-log-health-batch1
Oct 21, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
feat/crew-log-health-batch1

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An operator watching their own machines can see whether the crew log is losing data and how expensive it has become to rebuild, instead of finding out from a dashboard that silently shows a stale value.

The crew log is the record every dashboard panel, conductor board and work ledger reads. Today nothing reports on it. WriterStats counts appends the writer gave up on and appends the buffer rejected, eager.py counts fold wakes its queue could not hold, and all three numbers live and die inside one process with nothing sampling them. Folds are not timed at all, and there is no way to tell a fold that continued from a warm cell (cheap) from one that rebuilt from the first record of every unit (expensive, and the thing that makes a dashboard feel slow).

So three questions have no answer on a running install: did the log drop anything, is folding getting slower, and are savepoints actually being kept?

Why it matters

A hole in the crew log is a hole in what the product can show. dropped means the filesystem refused appends until the writer stopped trying; overflowed means the backlog grew until the buffer rejected the newest entries of a session. Both are real data loss that today is visible only to someone who attaches a debugger to the right process. A rising cold-fold share is the early signal for the class of complaint that arrives as "the dashboard got slow" with nothing to point at.

The constraint is what makes this worth doing carefully rather than quickly: the crew log's append path is the path a user's turn waits on. Instrumentation that cost that path anything would trade a real slowdown for a number about slowdowns.

Not a goal

  • The second batch of RFC section 6.5 (savepoint.write, the eager.queue_depth and writer.buffered gauges, bus.subscriber_error). feat(metrics): crew log health instruments, batch 2 #22227 shipped those and is on main; this PR adds its three instruments beside them in the same metrics/crew_log_health.py, and its metrics-doc rows sit in a different part of the same table.
  • crew_log.fold.lag. It is the one proposal in section 6.5 that needs a time.monotonic() on the append thread, which is exactly what this PR refuses to add. It waits for a measurement on a busy install, tracked in Metrics: crew_log.fold.lag (deferred from #22226) #22630.
  • Where these numbers are sent, and which of them leave the machine. Unchanged by this PR: they ride the existing telemetry.enabled gate and the existing readers.
  • Per-session or per-slot attribution. No metric here carries a session id, a slot key, a unit id, a path or any user text.

What changed (motivation -> approach -> change)

Goal -> the design problem. Every signal worth having is produced on a path that must not pay for it. The three loss causes are detected on the crew log's append thread (eager.note_commit's queue.Full branch; the writer giving up after its attempt budget) or on a producer's own thread (submit rejecting at the memory ceiling). The fold timings are produced inside fold code that is shared between the eager fold worker and every dashboard read.

Approach, and the alternatives rejected. Two mechanisms, each chosen over a simpler one that was wrong:

  1. write.lost is an observable instrument, not a call site. Every cause is already counted by the subsystem that detects it. So the diff adds no call at those sites at all. One observable gauge is registered on provider._build_recorder()'s live path, and its callback runs on the metrics reader's own ticker thread, reads the three counters, and folds each delta into a monotonic lifetime total (why a gauge and not a counter is below). The callback reads them through sys.modules rather than by importing: a process that never appended anything has nothing to report, which is exactly 0, and a telemetry callback must not put the writer's import chain on a thread no caller is waiting on. The rejected alternative was a synchronous counter.add(1) at each detection site, which is one lock-free add -- but on the one path in the system where "one more thing" is the thing we have been avoiding, and the number was already there to be read.
  2. fold.apply / fold.start record on the eager fold worker's thread and nowhere else. The worker claims its own thread once at start (_claim_thread_for_metrics), and both record sites begin with a thread-local check that returns immediately otherwise. The rejected alternative was recording wherever the fold ran, which would have made these counts of reads rather than of eager folding -- a different and much less useful question, and one that would have charged every dashboard request for a counter.

What was built.

  • metrics/crew_log_health.py -- the three metric names, the closed enums, the duration ladder (shared with feat(metrics): crew log health instruments, batch 2 #22227's savepoint.write), the thread claim, the two record sites, the write.lost accrual and the registration of write.lost as an observable gauge. The module's docstring now has one section per batch.
  • crew_log/eager.py -- a public dropped_wakes() over the count _count_drop already keeps; the thread claim in _run; _advance and _advance_session timed, each counting the folds it applied. note_commit, _enqueue and _count_drop are unchanged except for a comment saying why no metric is recorded in them.
  • crew_log/projection_session.py and projection_slot_memo.py -- one note_fold_origin call (and note_fold_idle where a pass advanced nothing) in each of the branches that decides where a fold's state comes from (warm cell / savepoint on disk / cold rebuild). These calls only leave a thread-local mark and record nothing; note_fold_pass turns the marks into points when the eager worker closes the pass, outside every lock.
  • Timing: begin_fold_pass reads the clock when a pass opens and note_fold_pass reads it once when the pass closes, before any point is recorded. Every fold.apply of that pass carries that one duration, so folds from one walk share a bucket and a first get_recorder() build is not charged to the fold.
  • crew_log/writer.py -- CrewLogWriter._drop is edited: it no longer adds a follow-on job it is about to requeue to _dropped_count, and _uncount_one_requeued_drop no longer takes it back out. Finding 2 below says why. No new work on the batch pass: one sum replaces len over the same list, under the lock it already holds.
  • No new registration call: the write.lost gauge registers inside the existing register_crew_log_health_instruments, which metrics/provider.py already calls under its own try, so provider.py is unchanged.

Three design points worth a reviewer's attention.

No new histogram. A duration is a counter with a closed bucket attribute (lt_10ms / lt_100ms / lt_1s / ge_1s, bounds exclusive, so exactly 10ms is lt_100ms), so the shape survives a backend that does not keep explicit histogram buckets. The edges are written in exactly two places: bucket_for in the module, which fold.apply shares with #22227's savepoint.write, and the metrics doc row.

The fold attribute is a closed catalogue with one owner. FOLD_LABELS is frozenset(FOLD_NAMES), resolved once at import rather than per record, so the crew log's own catalogue is the only copy and a fold it gains is a fold this metric labels. An earlier revision spelled the 16 names out again here and raised at import when the two disagreed; Design Review, First Principles and an independent reviewer all landed on the same objection, and they were right. The fold surface imports this module at module scope, so that raise would have turned a forgotten label edit into a failed import of projection_session.py and projection_slot_memo.py -- every dashboard read, conductor board and session fold, plus every backend test that touches the crew log, failing at collection. It also contradicted this PR's own rule, which eager.py and provider.py follow: a telemetry package that will not load costs the fold worker its metrics and nothing else. With no copy there is no drift to detect, which is a smaller surface than detecting it well.

write.lost ships as an observable GAUGE, which DEVIATES from the RFC table. Section 6.5 lists it as a counter. It cannot be one here, and the rule that forces that is a test: test/metrics/test_otlp_wire_e2e.py::test_no_instrument_exports_cumulative asserts, over every exported instrument rather than a list, that not one Kiro Crew series is CUMULATIVE -- because a single cumulative series forces every consumer into stateful whole-hour aggregation, where a gauge or a DELTA sum merges one datapoint at a time.

write.lost reads a total it does not own (the writer's and the fold queue's own counters), so it can only be an asynchronous instrument, and an observable counter keeps the SDK default of CUMULATIVE. Declaring DELTA for it instead is the move metrics/temporality.py records as tried and reverted: the delta baseline lives in the provider, the provider is rebuilt whenever telemetry consent changes, so the first collection after a rebuild would re-emit a whole process total as one delta and inflate the day's sum. That left one shape, and it is the shape this repo already chose for the identical problem -- the process CPU and GC readings were observable counters and are observable gauges declared as lifetime totals now, which is what removed the last cumulative series.

So: the reading IS the process-lifetime total, it starts at zero for each process and therefore resets on restart, and a backend reads the loss RATE as the difference between two readings. The metrics-doc row says all of that at the place a reader looks. The total is accumulated by this module from the growth it reads, which buys three properties a raw lifetime total does not have: it is monotonic even when the underlying counter is replaced by a fresh one, the first collection is a baseline so losses accrued before telemetry was enabled are not reported, and the totals live in memory only, so nothing counted before a restart is counted again after it.

Review findings fixed after opening

  • GPT 6.1, 2 blocking, both real flakes in my own tests. The subprocess probe that asserts the append path never imports the metrics module raced the fold worker note_commit starts: that worker imports the module on purpose, through its thread claim, so the probe could fail on correct code. eager._run is now stubbed to a no-op inside the probe, which leaves the append path as the only importer and is the thing being measured; the test passes 5/5 runs in isolation. And the label-collapse case for note_fold_apply left _now on the real time.monotonic, so a 10ms scheduling pause moved the bucket and failed an assertion about labels; the clock is pinned.
  • Design Review and First Principles, both CONCERNS, both on the import-time catalogue check. Removed, as described above. First Principles also found two helpers with zero production callers after the pass-close rework -- note_fold_start and start_clock, which note_fold_pass and begin_fold_pass replaced. Both deleted; the tests now drive the real path (stamp an origin, close the pass), which is what the eager worker does.
  • Static Ratchet Gate, real. test_crew_log_exc_info_sites flagged two new exc_info=True sites in eager.py. Not an allowlist case: in crew_log a retained traceback reaches its callers' frames through f_back, and those frames bind open CrewLog handles whose write leases a finalizer releases, so a record-keeping handler pins a lease for as long as it keeps the record. _close_pass runs in _advance's finally, squarely inside that chain. Both now go through the module's own _log_exc, which renders the traceback to text.
  • Comment History Gate, real. Six comments narrated change history ("used to", "no longer") while explaining why the writer no longer counts a requeued job. All rewritten in the present tense.

Three defects found by review before this opened, and fixed

An independent reviewer saw only the diff, not this reasoning, and was asked to break the no-append-thread-work and no-over-count claims. It could not break the first. It broke the second three times, and each fix is in this diff:

  1. Concurrent collection double-counted a loss. write_lost_totals read the three raw counters outside its lock and accrued inside it. Kiro Crew attaches more than one reader to one meter provider (the local JSONL sink, plus one per OTLP destination), each with its own ticker thread, so two collections overlap: the slower thread carries a stale reading in behind the fresher one, which reads as a counter that went down and then up again, and the same loss is added twice. Reproduced at 4 real losses reported as 9 and then 11. The read is now inside the lock -- three O(1) counter reads, and no path takes this lock while holding the writer's or the fold queue's, so it cannot deadlock.

  2. CrewLogWriter._drop publishes a count it corrects a moment later. It adds a whole failed batch to _dropped_count, then takes the follow-on jobs back out via _uncount_one_requeued_drop -- releasing and re-acquiring the lock in between, with a logger.warning in the gap. Its docstring reasons that "no reader can have seen the count yet", which was true while every reader was a caller on that same thread and false the moment a telemetry callback began sampling stats(). Treating the dip as a fresh counter added the whole new reading, turning one requeued job into a permanently inflated total. Fixed on both sides: _drop no longer counts a job it is about to requeue (so the transient is never published, and the docstring's claim is now true by construction rather than by assumption -- the per-unit marker debt is still adjusted over there, because that debt is consumed by a later pass of the same thread and no reader samples it), and _accrue now accrues only growth, so a reading corrected downwards can never inflate the total whatever the source does.

  3. A slot fold that advanced nothing counted an apply. _advance recorded an apply whenever the fold did not raise, but the slot fold's "nothing folded" branch returns an unchanged live cell. Three passes over an unchanging board reported three applies, so start above apply -- the whole failing-and-idle-folding signal -- said nothing on the slot scope, while the session scope was already correct via warm.changed. Fixing it also fixed a second finding from the same reviewer: the start was being recorded inside the per-cell fold lock, where a first get_recorder() builds the whole telemetry provider synchronously and would hold that lock against a waiting reader. Both are gone because the fold branches no longer record at all: they leave a scope-keyed thread-local mark, and the eager worker turns it into points when it closes the pass, outside every lock. That mark also made two smaller findings disappear -- a nested fold (resolving a slot fold's units can fold a session log) can no longer be reported as the outer pass, and a session fold that retries after a mid-fold identity change now reports one start with the origin it finally folded from instead of two.

One further finding was fixed as stated: the write.lost registration used to clear this module's state, which let a recorder build that provider._consent_worker then discards zero the totals of the provider still serving. The reviewer's test criticisms are all addressed below.

Backwards compatibility

Compatible: three new metric names, one new public function (eager.dropped_wakes()), one new module. No existing instrument changes name, type, unit or attributes; no validator is tightened; no behaviour changes when telemetry is off, because get_recorder() is a no-op recorder and the observable gauge is never registered.

One existing value changes, and it is a counter nothing could legitimately read before: WriterStats.dropped no longer includes an append that is being requeued rather than lost. Previously it did, for the microseconds between _drop adding the batch and _uncount_one_requeued_drop taking the follow-ons back out. Every reader of that field is below; the value they see is unchanged, because the only observable difference is in a window _drop's own docstring asserted no reader could be in.

  • Reader: src/kiro_crew/crew_log/emit.py::dropped_writes -- chat / cron / subagent / app / crew page, via CrewLog appends -- still answers the settled count, which is what every caller compares before and after its own append. test_crew_log_edge_exhaustion.py and test_crew_log_emit_write_path.py (61 passed) cover the drop and requeue paths.
  • Reader: src/kiro_crew/crew_log/writer.py::drain_for_shutdown / DrainReport -- release and shutdown path -- reports loss_markers_owed, which is the per-unit marker debt and is still adjusted by _uncount_one_requeued_drop exactly as before. Only _dropped_count changed.
  • Reader: src/kiro_crew/metrics/crew_log_health.py::_read_raw_losses -- the metrics reader thread added by this PR -- is the reader that made the window observable in the first place, and the reason for the change.

Removes nothing else: no path is hidden, no record or row is deleted, no key or public function is removed or renamed, and no data is migrated or pruned.

Tests

test/metrics/test_crew_log_health_losses_folds.py -- the shape of this PR's three instruments. (test/metrics/test_crew_log_health.py is #22227's file and is unchanged.)

  • Every attribute value passes through when known and collapses to other when not, in all four enums; every declared label is its own collapse target, so no series is unreachable; and an out-of-enum value is pushed through the real record sites (note_fold_pass / note_fold_apply), not only through _closed -- testing that alone stays green if a call site stops calling it, which is the one gate against a session id becoming a series.
  • Each bucket edge, twice over: directly against bucket_for, and end to end through an injected clock (_now is patched; nothing sleeps, and the 1s edge is not sleepable in a test anyway). A clock that stepped backwards still counts the fold.
  • One pass, one clock reading. A clock that moves 9ms on every read gives both folds of one pass lt_10ms and is read exactly twice (open and close); reading it again per fold would put the second fold in lt_100ms. A recorder whose first use moves the clock 2s, as a slow provider build would, leaves a 1ms pass in lt_10ms. Both cases fail on the previous head.
  • The fold label set is FOLD_NAMES itself, a name the crew log gains is labelled with no edit here, a non-fold collapses to other, and importing the module cannot fail over the catalogue: its source is executed as a fresh module under a patched catalogue that disagrees with everything, and it still loads.
  • The pass marks: a stamped origin becomes one start plus its applies; a pass marked idle records its start and no apply; a pass that stamped nothing records nothing; opening a pass clears what the last one left; a nested session fold cannot be reported as the slot pass; a pass is taken once so a later one cannot repeat it; and all of it is a no-op off the fold worker's thread.
  • write.lost arithmetic: the first collection is a baseline and not a report; one loss is exactly one +1 for its own cause (asserted per cause, with the other two asserted to stay at zero); a delta across two collections is counted once and a quiet cycle adds nothing; a counter replaced by a fresh one counts only the fresh losses; a reading corrected downwards adds nothing (the 40 -> 41 -> 40 -> 42 sequence that used to report 41 instead of 1); the total is monotonic and the exact totals are spelled out rather than only seen == sorted(seen), which would pass for an over-counting implementation; a restart reports nothing it already reported.
  • The read happens inside the accrual lock, asserted directly (_lost_guard.locked() is true during every read), so moving the read out of the lock reddens with no thread and no timing. A second case runs two real collections ordered by events, not by time: the first is held inside its read while the second starts behind it, both totals are exact, and a third collection adds nothing.
  • Each cause reads the count its own subsystem keeps, and a subsystem that was never imported reports zero (sys.modules deleted under the test) -- which is what proves the callback imports nothing.
  • Telemetry off: with MetricsRecorder(None) as the gate, no instrument is created and no state is held. A recorder that raises costs the metric and not the fold.

test/test_crew_log_health_threads.py (21 cases) -- the thread constraint, over the real paths rather than the helpers:

  • The append path never reaches the metrics module at all. This runs note_commit in a subprocess and asserts kiro_crew.metrics.crew_log_health is not in sys.modules afterwards. A recorder spy on the test's own thread proves little, because every record site returns at once on an unclaimed thread -- so adding a note_fold_* call to note_commit would leave the spy empty and the suite green while adding exactly the work this PR promises not to add. No thread-local gate can satisfy the subprocess assertion.
  • The append path does not even consult the gate. on_fold_worker_thread is spied on directly and asserted never to be called from note_commit, so a call that was merely gated off would still fail.
  • A dropped wake moves dropped_wakes() by exactly the number dropped and records nothing; the happy path records nothing (with drain waited on, so the assertion is not racing the worker); and the hand-off works end to end -- the count the append path kept becomes the write.lost total read from the other side.
  • The eager worker claims its own thread, proven by running _run with the stop flag already up so the claim is the only thing that happens, and asserting the claimed thread object is that worker.
  • A thread that did not claim records nothing, and a dashboard read folds the same branches and records nothing.
  • Each origin is reached through its real branch via the real eager._advance / _advance_session, for both scopes: cold then warm for a slot fold and for a session fold, and savepoint for each by growing the unit past the savepoint floor, dropping the in-memory cell, and folding again. One pass records exactly one start.
  • fold.apply: one point per applied slot fold with its name; one point per session fold that moved and no duplicates; an idle slot pass records its start and no apply (three passes over an unchanging board give three starts and one apply); a fold that raises after choosing its start records that start and no apply, with the raise planted past the start branch so the pass really gets that far -- replacing read_slot_projection wholesale would mean the fold never started and the assertion would hold for the wrong reason; a fold that never started records nothing at all; a read that got there first leaves the worker nothing to apply, which pins the documented meaning of the ratio; and every point is stamped with the thread that folded.

Existing suites run for regressions, all green:

  • test_crew_log_health_losses_folds.py + test_crew_log_health_threads.py + test_crew_log_fold_eager.py + test_crew_log_slot_folds.py + test_crew_log_lazy_exports.py + test_perf_boot_path.py -- 188 passed. The last two are the lazy-seam and import-weight gates that a module-scope import in the fold surface could have broken.
  • For the writer.py change: test_crew_log_edge_exhaustion.py + test_crew_log_emit_write_path.py + test_crew_log_edge_fanout.py (61 passed) and test_crew_log_emit.py + test_crew_log_emit_continuity.py + test_crew_log_edge_throttle.py + crew_log_drain.py (113 passed).
  • test_crew_log_projection.py + metrics/test_provider.py green in an earlier round.

Manual verification

The observable gauge was collected through a real MeterProvider with an InMemoryMetricReader to confirm the SDK path works as described and not only the accrual in isolation: the instrument reports no temporality on the wire (which is what makes it a gauge and keeps it out of test_no_instrument_exports_cumulative), the first collection reported 0 for all three causes, the next reported dropped=3 / wake_dropped=1 against raised counts, and a third collection with no further loss reported the same totals rather than new points.

flake8, isort and black are clean on every changed and added file, and scripts/docs_lint.py passes.

Related Issues

Part of #21270 (crew usage metrics). This PR implements the first batch of the crew log health section.

Provenance, stated plainly because a review lane was right to ask. The "crew usage metrics wave 2" RFC that specifies these three counters is NOT on main: it is still in review and still status: draft, so a grep of docs/request-for-change/ on the base finds nothing, and references to "section 6.5" above resolve only against that in-review document. The tracking issue #21270 is the record that exists today. No production slow-fold report backs fold.apply / fold.start either -- they exist to make the cold-fold share and the fold duration observable for the first time, which is a measurement this repo cannot currently take, not a fix for a reported regression. Whether that is enough to land the two fold counters now, or whether they should wait for the RFC to be on main with a non-draft status, is a decision for a maintainer rather than something this PR should settle by assertion; write.lost does not depend on either of them.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

One check before merge: a gap between fold.start and fold.apply mixes refused or raised passes with idle passes and passes a read got to first, so it can't show failing folds alone.

[DESIGN-REVIEWED] cc6cc8b

Description read

[DESCRIPTION-READ] 1b4461da21fc6d07bf16059c78c6641c0c736b4e2db430548b54e432432404d8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

note_fold_apply is a second record path for fold.apply that no production code calls. Only tests reach it; the eager worker records through note_fold_pass.

Not justified as shipped

  • Item 6 — zero consumers: its docstring says "note_fold_pass does not call this: it reads the clock once for the whole pass and records through _count_apply directly". A grep for note_fold_apply across src/ finds 1 hit, the definition at metrics/crew_log_health.py:717, and 0 callers. The only callers are 6 sites in test/metrics/test_crew_log_health_losses_folds.py.
    Subtraction: delete note_fold_apply and point those tests at note_fold_pass, which already covers the _closed collapse and the bucket path.
    Clears when: the function is deleted, or a production caller is counted.

What this change ships

Inventory (6 items) — 5 justified

Intent: an operator can see whether the crew log is losing entries and how costly eager folds are, without adding work to the append path. This is an ADDITION.

  1. New crew_log.write.lost gauge: lifetime losses split by cause (dropped / overflowed / wake_dropped), read from counters that already exist — justified
  2. New crew_log.fold.apply counter: eager-worker folds counted by fold and duration bucket — justified
  3. New crew_log.fold.start counter: each eager pass labelled warm, savepoint or cold — justified
  4. The writer's dropped count no longer counts a requeued follow-on job and then subtracts it — justified
  5. New public eager.dropped_wakes() exposes the count the fold queue already keeps — justified
  6. New public note_fold_apply, a per-fold record path beside note_fold_pass — zero consumers: only tests call it

[FIRST-PRINCIPLES-REVIEWED] cc6cc8b

Description read

[DESCRIPTION-READ] dfd97f1f67854178212cd1e3fcb9b32ca28e7d9ed6d600dcfcb31a5412f0cca3

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cc6cc8b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] cc6cc8b

Verdict parsed from the review's SHA-scoped output markers for commit cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-health-batch1 branch 3 times, most recently from f98fc3d to 32bc10b Compare October 19, 2026 16:54
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Answering the First Principles advisory on fold.apply / fold.start provenance

The finding is right about the provenance and I have made the PR description say so plainly. The specification these two counters come from is in review, not on main: it is #22180 (docs/request-for-change/rfc-crew-usage-metrics.md), whose section 6.5 carries the crew log health table, and the tracking issue is #21270.

I am keeping both counters exactly as that table specifies rather than deferring them, for one reason: the table is where the decision about this metric set was made, and a PR that implements two of its three first-batch rows and silently drops the other two would make the implementation disagree with the record instead of following it. The same table is why this PR stops where it does -- savepoint.write, append.bytes, append.batch, the two gauges and bus.subscriber_error are its second batch and belong to a sibling PR, and fold.lag is deferred there explicitly because it is the one row that needs a time.monotonic() on the crew log's append thread, which this PR refuses to add.

On the substance of the harm: there is no production slow-fold report behind these two, and the description now says that outright. What they add is the ability to take the measurement at all -- the cold-fold share and the fold duration are not observable in this repo today, so "is folding getting slower, and is it rebuilding from scratch" currently has no answer to cite. That is the gap the row describes.

Whether that is enough to land them before the RFC is on main with a non-draft status is a maintainer's call, not something this PR should settle by assertion. write.lost is independent of both, as the finding notes, so it can be separated if a maintainer prefers that.

@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-health-batch1 branch from 32bc10b to dfd8f76 Compare October 19, 2026 17:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-health-batch1 branch 5 times, most recently from d8b77f6 to 8785513 Compare October 19, 2026 19:27
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
buluoray
buluoray previously approved these changes Oct 19, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 8785513. Approve on the code; merging waits on the design decision for #22180, and this one should land before #22227.

The label sets are closed (cause, scope, origin, bucket, and fold from FOLD_NAMES), each folds onto other, and the tests catch it if the folds are loosened. No metric call, import or clock read is added to the append path, and test_a_dropped_wake_on_the_append_path_records_no_metric catches one being added.

One correction to the title and description: this PR does edit CrewLogWriter._drop, which runs on the inline submit path and in the writer thread give-up branches. The edit is arithmetic only: requeued follow-on jobs are no longer counted and then taken back out. The final count is unchanged and pinned by the existing writer tests, so please say this in the description instead of saying the append path is untouched.

Follow-up: RFC section 6.5 lists write.lost as a counter, the code registers an observable gauge, and the module docstring calls it an observable counter. Please make the three agree.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 8785513.

This PR conflicts with main. #22227 (batch 2) merged first and added the same src/kiro_crew/metrics/crew_log_health.py and test/metrics/test_crew_log_health.py. A merge against current main gives add/add conflicts in both files and a content conflict in src/kiro_crew/metrics/provider.py. The green CI here was measured against the old base.

To clear this:

  • Rebase onto main and resolve the conflicts. The rebased head needs a fresh review.
  • Reply to First Principles items 8 (duplicate SCOPE_* constants in eager.py), 9 (release_fold_worker_thread has no production caller) and 10 (unreachable other fallbacks). Either fix each or rebut it.
  • Fix the description where it still calls write.lost an observable counter (the code registers a gauge), and say that CrewLogWriter._drop is edited, as buluoray asked.
  • A maintainer still has to settle the docs(rfc): crew usage metrics, wave 2 #22180 RFC question that buluoray's approval depends on.

— crew-pr-reviewer (r113)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
…ppend path

Three counters for crew log health: kirocrew.crew_log.write.lost{cause},
kirocrew.crew_log.fold.apply{scope,fold,bucket} and
kirocrew.crew_log.fold.start{scope,origin}.

Nothing is added to the crew log's append thread. Every write.lost cause is
detected on that thread or on a producer's own thread and is already counted
there, so the three counts are exported by one observable counter whose
callback runs on the metrics reader's ticker thread and imports nothing. The
read and the accrual happen under one lock hold because more than one reader
collects, and only growth is accrued so a reading that is corrected downwards
cannot become a permanent inflated total.

fold.apply and fold.start record only on the eager fold worker's thread, which
claims itself once at start. The fold branches leave a thread-local mark rather
than recording, because two of them hold a per-cell lock a reader can be
waiting on; the worker turns the marks into points outside every lock, so a
pass that advanced nothing counts its start and no apply.

CrewLogWriter._drop no longer counts an append it is about to take back out for
requeue. It relied on no reader seeing the count in between, which was true
while every reader was a caller on its own thread and false as soon as a
telemetry callback began sampling stats().
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

New head c9c7264. main now has register_crew_log_health_instruments, so the write.lost gauge registers inside it and the second function and its provider.py block are gone (First Principles item 7). Same three instruments, same consent gate; description updated.

@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-health-batch1 branch from 7f318df to c9c7264 Compare October 20, 2026 08:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES at c9c7264.

The append path still gets no new work, and the writer.py _drop edit gives the same settled count, because every follow-on it no longer counts is requeued unconditionally. CI is green. Four items from the r114 HOLD are still open:

  1. Timing defect, unanswered. note_fold_pass counts fold.start and then calls note_fold_apply per fold, and each call reads the clock again (crew_log_health.py L681-L732). Folds from one pass can land in different buckets, and the first get_recorder() build is charged to them. Smallest fix: compute elapsed_ms once at the top of note_fold_pass and pass it to each apply.
  2. Stale test docs. The test module docstring and the description's Tests and What was built sections still describe an import-time catalogue check and note_fold_start. Both were removed.
  3. Deferred crew_log.fold.lag has no issue. Please open one and link it in the description.
  4. Maintainer ruling. #22180 is still open, and #21270 holds child PRs until a maintainer confirms the data scope. That ruling should also settle the Design Review question: shipping write.lost as a gauge instead of the RFC's counter.

Items 1-3 are yours to fix. Item 4 needs a maintainer.

— crew-pr-reviewer (r115)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 20, 2026
note_fold_pass reads the clock once, before the fold.start point is
recorded, and every fold.apply of that pass carries the same duration. Each
apply used to read the clock again after the start point, so folds from one
walk could land in different buckets and a first get_recorder() call, which
can build the whole telemetry provider, was charged to the fold. Two tests
pin it: a clock that moves on every read, and a recorder whose first use
moves the clock 2s.

The eager fold worker records its crew log fold counters through the same
recorder as everything else. The end-to-end test keeps its exact equality
over the whole stream, lists the fold.start and fold.apply points each pass
adds, and freezes their clock so the bucket is fixed.
@chenmingwei23
chenmingwei23 force-pushed the feat/crew-log-health-batch1 branch from c9c7264 to cc6cc8b Compare October 21, 2026 01:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Answers to the review at c9c7264. New head: cc6cc8b.

  1. Timing defect: fixed in cc6cc8b. note_fold_pass now reads the clock once, before the fold.start point is recorded, and every fold.apply of that pass uses that one duration (_count_apply). Folds from one pass share a bucket, and a first get_recorder() build is not part of the duration. Two new tests in test/metrics/test_crew_log_health_losses_folds.py fail on c9c7264 and pass now: test_every_fold_of_one_pass_shares_one_clock_reading (a clock that moves 9ms per read; both folds lt_10ms, clock read exactly twice) and test_building_the_recorder_is_not_charged_to_the_fold (a recorder whose first use moves the clock 2s; a 1ms pass stays lt_10ms). The metrics doc row says the same.
  2. Stale test docs: fixed in cc6cc8b. The test module docstring and section header no longer describe an import-time catalogue check. In the description, the What was built section now names note_fold_origin / note_fold_idle and the timing, and the Tests section drops the import-time gate and note_fold_start.
  3. Deferred crew_log.fold.lag: tracked in Metrics: crew_log.fold.lag (deferred from #22226) #22630, linked on the fold.lag line under Not a goal.
  4. Maintainer ruling: waits on a maintainer.

All checks are green on cc6cc8b.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at cc6cc8bbeaef2c6bd095d6782fd7cc9230a5da9e.

The items from our earlier change request are fixed. note_fold_pass reads the clock once and every fold in the pass shares that duration (crew_log_health.py#L681-L714). The test docs are updated, and fold.lag is tracked in #22630. CI is green.

Blockers:

  1. The First Principles CONCERNS on this head has no reply. note_fold_apply (L717-L732) has no production caller; only tests use it. Delete it and point those tests at note_fold_pass, or reply with the caller it is meant for.
  2. The Design Review check on this head has no reply: the gap between fold.start and fold.apply mixes refused or raised passes with idle and read-first passes. Say in the spec row what that gap can and cannot show, or reply.
  3. A maintainer still needs to confirm the data scope, as OpenTelemetry: export per-module usage counts from the crew log #21270 requires. docs(rfc): crew usage metrics, wave 2 #22180 is still open.

— crew-pr-reviewer (r116)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 21, 2026 07:36
@iamwhatever
iamwhatever merged commit 2d2a08a into main Oct 21, 2026
118 of 124 checks passed
@iamwhatever
iamwhatever deleted the feat/crew-log-health-batch1 branch October 21, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants