Repository navigation
feat(metrics): crew log health instruments, batch 2 - #22227
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the test pinning [DESIGN-REVIEWED] b3e10ff Description read[DESCRIPTION-READ] 20eadfe5ca5a7babd42c707a1bf9a5bf95c5ad14c086f6916e8b1a36ed58e929 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Every design rule in Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: an operator can tell from metrics whether the crew log is backlogged, persisting savepoints, or has a failing subscriber. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] b3e10ff Description read[DESCRIPTION-READ] c6bc19904673417afc7ca365d5545e2abec8ac6f57996ffa0eb03114f7f12eee That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
4567581 to
ac25890
Compare
ac25890 to
63e68c2
Compare
63e68c2 to
9ab5a3b
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the careful write-up. The instruments are well bounded and the call sites are correct as far as I can trace them. I am leaving a comment rather than an approval, because three things need a maintainer before this can merge.
1. The design this implements is not agreed yet. The module docstring and the closed enums cite "RFC section 6.5" and "RFC rule 2". That RFC is #22180, which is still open with no review. The metric names, the attribute sets and the bucket banding on savepoint.write are product decisions, so a maintainer design sign-off on #22180 must come first. The First Principles lane makes the same point about bucket: the stated goal is "whether savepoints are landing", and the latency bands answer a different question. If the RFC lands with bucket, that concern goes away.
2. Merge order with #22226. Both PRs add metrics/crew_log_health.py and test/metrics/test_crew_log_health.py as new files, so the second to merge has an add/add conflict on both. test_the_four_shipped_names_are_exactly_these pins ALL_METRIC_NAMES to the four names here, so it must hold the union after the rebase. #22226 also edits crew_log/writer.py (the _dropped_count accounting). That is fine for this PR's AST checks, but it makes the "byte-identical to the base branch" claim untrue after the rebase (inline 2). Suggested order: #22180, then #22226, then this PR rebased on top. Both PRs also touch docs/system-specs/modules/metrics.md and metrics/provider.py with #22217, #22218, #22219, #22220 and #22223.
3. CI. The red Fast Gate on this head is the Harness Parity Gate failing on the ACP_BACKENDS_MEMBER_CAPABILITIES [claude] allowlist entry, and Coverage Gate fails behind it. This diff touches no backend file, so neither failure comes from this change. Main carries the same failure until #22178 merges. A rerun after that is needed for readiness: passed.
Agreed with the First Principles lane on one smaller point: _observations copies process_gauges._observations, and the reason given ("stays loadable when the SDK is absent") does not hold, because process_gauges also imports opentelemetry inside the function. Importing the existing helper removes the copy.
What I checked on head 9ab5a3b: the full diff; the callers of checkpoint.save and save_slot on main (the session and slot projection folds, not the write-behind), which answers the Design Review question; get_recorder and emit_counter on main for what the bus counter can reach on a write thread; the failing check logs; and the overlapping open PRs. I did not run the tests locally.
| calls: list = [] | ||
| try: | ||
| bus.subscribe(bus.TREE_ADVANCED, lambda e: (_ for _ in ()).throw(RuntimeError("x"))) | ||
| with patch.dict(sys.modules, {"kiro_crew.metrics.crew_log_health": None}): |
There was a problem hiding this comment.
🟡 This test stays green if the bus site is changed to an import, so it does not guard the property it names.
The test sets sys.modules["kiro_crew.metrics.crew_log_health"] = None and asserts that the counter was not called. Suppose _count_subscriber_error is changed to from kiro_crew.metrics import crew_log_health as health. With a None entry in sys.modules, that import raises ImportError. The bare except Exception: pass in _count_subscriber_error catches it, and calls stays empty. So the regression this test exists to catch, a write thread doing the first import of the metrics facade, passes it.
with patch.dict(sys.modules, {"kiro_crew.metrics.crew_log_health": None}):
with patch.object(clh, "bus_subscriber_error", calls.append):
bus.publish(bus.TREE_ADVANCED, object())Suggestion: pop the module instead of setting it to None, publish from a raising subscriber, then assert "kiro_crew.metrics.crew_log_health" not in sys.modules. Or add a parsed-source check that _count_subscriber_error contains no ast.Import or ast.ImportFrom node, like the check you already have for the two readers. If the import variant turns this test red, I will withdraw this.
| """The hard constraint: not one line of the write path is this change's. | ||
|
|
||
| Four assertions, because the claim has four parts: the two files that carry | ||
| the write path are byte-identical to the base branch; driving a real batch |
There was a problem hiding this comment.
🟡 The docstring and the PR description say the write path files are held "byte-identical to the base branch", but no test checks that.
The four tests in this class check imports, calls in _write_batch, global and augmented assignment in _write_then_sync, and a driven batch pass. None of them compares the files with the base. That is the right design, because #22226 edits crew_log/writer.py and a byte check would block it. But the claim is then not true after either PR merges, and a later reader can take it as a guarantee.
Suggestion: say what the tests hold, for example "the write path gains no metrics import, no clock read and no counter call", and remove "byte-identical" here and in the description.
9ab5a3b to
58136ed
Compare
|
Answering First Principles item 4 ( The lane is right that the support is not findable from this PR alone, and that is the gap worth closing rather than the attribute. The source is #22180, section 6.5, whose second-batch table specifies this metric as On the cost the item prices: the clock pair is on the fold and read path, roughly once per 256 entries, and never on a crew log write thread. Item 7 of the same verdict is fixed in Why |
58136ed to
3ce5d01
Compare
Add kirocrew.crew_log.savepoint.write, .eager.queue_depth, .writer.buffered and .bus.subscriber_error. append.bytes, append.batch and fold.lag are omitted: every source for them is work on the crew log write path, which this change leaves untouched byte for byte.
3ce5d01 to
b3e10ff
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the update. Rebased on current main, and two earlier points are resolved on head b3e10ff:
_observationsis now imported fromprocess_gaugesinsideregister_crew_log_health_instrumentsinstead of copied. The signatures match and the deferred import keeps the module loadable without the SDK.TestTheSavepointCounterIsOffTheWriteThreadnow asserts from the call graph that only the session and slot fold modules callsavepoints.save/savepoints.save_slot, and that noemit*.pymodule folds a session. I checked that both fold modules importcheckpoint as savepoints, so the qualified match does find them. That covers the Design Review question about the facade import on a write thread.
I am still leaving a comment rather than an approval:
1. The design is not agreed yet. This implements section 6.5 of #22180, which is still open with no review. The metric names, attributes and the bucket banding are product decisions, so a maintainer design sign-off on #22180 has to come first.
2. Merge order with #22226 is unchanged. Both PRs still add metrics/crew_log_health.py and test/metrics/test_crew_log_health.py as new files, so the second to merge gets an add/add conflict on both. #22226 also edits crew_log/writer.py, projection_session.py and projection_slot_memo.py. After the rebase, test_the_four_shipped_names_are_exactly_these must hold the union of names, and test_the_savepoint_writers_are_reached_from_only_the_fold_path should be rerun against those projection edits. Suggested order: #22180, then #22226, then this PR rebased on top.
3. The two test points from my last review still apply (inline). Neither the sys.modules[...] = None test nor the "byte-identical to the base branch" wording changed in this push.
4. CI. The rebase picks up the fix for the main-wide Harness Parity red. Checks on this head are still running and readiness is checking, so I have not verified that it turns green.
What I checked: the diff between the previous head 9ab5a3b and b3e10ff for the files this PR owns (only crew_log_health.py and the test file changed), process_gauges._observations on this head, the savepoint call sites in both fold modules, and the current state of #22180 and #22226. I did not run the tests locally.
| calls: list = [] | ||
| try: | ||
| bus.subscribe(bus.TREE_ADVANCED, lambda e: (_ for _ in ()).throw(RuntimeError("x"))) | ||
| with patch.dict(sys.modules, {"kiro_crew.metrics.crew_log_health": None}): |
There was a problem hiding this comment.
🟡 Still open from the last review: this test stays green if the bus site is changed to an import, so it does not guard the property it names.
With sys.modules["kiro_crew.metrics.crew_log_health"] = None, a future from kiro_crew.metrics import crew_log_health inside _count_subscriber_error raises ImportError. The bare except Exception: pass there swallows it and calls stays empty, so the regression this test exists for (a write thread doing the first import of the metrics facade) passes.
Suggestion: pop the module instead of setting it to None, publish from a raising subscriber, then assert "kiro_crew.metrics.crew_log_health" not in sys.modules. Or add a parsed-source check that _count_subscriber_error has no ast.Import or ast.ImportFrom node, like the one you already have for the two readers.
| """The hard constraint: not one line of the write path is this change's. | ||
|
|
||
| Four assertions, because the claim has four parts: the two files that carry | ||
| the write path are byte-identical to the base branch; driving a real batch |
There was a problem hiding this comment.
🟡 Still open from the last review: "byte-identical to the base branch" is not what these tests check.
The tests in this class check imports, calls in _write_batch, global and augmented assignment in _write_then_sync, and a driven batch. None compares the files with the base, which is the right design because #22226 edits crew_log/writer.py. The claim becomes untrue once either PR merges, and a later reader can take it as a guarantee.
Suggestion: state what the tests hold, for example "the write path gains no metrics import, no clock read and no counter call", and drop "byte-identical" here and in the PR description.
buluoray
left a comment
There was a problem hiding this comment.
Reviewed head b3e10ff. Approve on the code; merging waits on the design decision for #22180 and on a rebase onto #22226 after it merges.
The label sets are closed and fold onto other. The two gauges have no attributes. The write path files are unchanged, and the savepoint counter is reported after the lease is released.
Before merge (ordering):
- A trial merge onto #22226 conflicts in metrics/crew_log_health.py, its test file and metrics/provider.py. Both PRs define the same four bucket bands and their own fallback helpers; please keep one copy, and make test_the_four_shipped_names_are_exactly_these list all seven names.
- #22226 imports the crew_log_health module on the fold worker at startup and on every fold, whatever the telemetry setting. After both land, bus._count_subscriber_error can find the module in sys.modules before the recorder is built and run the first get_recorder() build on a crew log write thread. Please add a recorder-already-built check there, and re-run the call-graph tests on the merged tree.
Follow-ups, not blocking:
- test_a_raising_subscriber_on_that_thread_cannot_trigger_a_first_import still passes if the bus uses a plain module import instead of the sys.modules lookup, because the patched sys.modules entry makes that import raise and the bare except swallows it. Popping the module, or asserting it was not imported, would close this.
- Please drop "byte-identical to the base branch" from the test docstring and the PR description, since it stops being true once #22226 lands.
Problem / Motivation
Goal: an operator can see whether the crew log is keeping up -- how big its backlog is, whether savepoints are landing, and whether a fold subscriber is failing silently.
Main has no crew log metric at all. The crew log is the durable record every fold, every dashboard read and every resumed session is built from, so when a board looks stale there is nothing to look at: the in-process
WriterStatsand a handful of warning logs are the only evidence, and neither is sampled over time. "The log is slow" and "the log is stuck" are indistinguishable from outside the process, and a bus subscriber that raises on every published fold is invisible -- the publisher is told nothing by design, and a log line carries no rate and no total.Why it matters
The failure this closes is a silent one. A saturated write-behind, a savepoint store that refuses every payload, a backed-up fold queue and a subscriber raising on every fold all present identically to an operator: a dashboard whose numbers lag. Each needs a different response and today none can be told apart without attaching to the process. The two gauges separate a backlog in the write-behind from one in the fold queue; the savepoint counter says whether resume state is being persisted at all, which decides whether a restart costs a cold fold; the subscriber-error counter turns a silent riser into a number.
Not a goal
kirocrew.crew_log.append.bytes,kirocrew.crew_log.append.batchandkirocrew.crew_log.fold.lagare deliberately NOT shipped. Every available source for each is work on the crew log write path. The "What changed" section walks each one; the short version is that this PR adds nothing at all to that path rather than buying a byte count with an instruction on it.write.lost,fold.apply,fold.start). Separate PR.telemetry.enabled.What changed (motivation -> approach -> change)
The constraint that decided everything, including the omissions: recording a crew log metric must add no work to the write path. An append reaches disk on
CrewLogWriter._write_batchor on the write-behind's inline fast path, and both sit on the latency path of a turn that commits an entry.src/kiro_crew/crew_log/writer.pyandsrc/kiro_crew/crew_log/store_segment_writes.pyare therefore byte-identical to main in this diff, and four tests hold them that way.What ships. Four instruments:
kirocrew.crew_log.eager.queue_depthandkirocrew.crew_log.writer.buffered-- observable gauges read from live state when a reader collects, never up/down counters a crash can leave drifting. Both answer absent rather than zero when their subsystem does not exist in the process, because "no fold thread" and "an empty queue" are different facts. Neither reader imports anything: each resolves its source fromsys.modulesand answersNonewhen it is not loaded, so a collection every export interval cannot be the thing that pulls the storage layer into a process that has the crew log switched off. The backlog reader also reads the installed write-behind directly rather than through the accessor that would build one -- telemetry must not bring a subsystem into being.kirocrew.crew_log.savepoint.write{kind,outcome,bucket}-- one point per pass that actually offered a payload to the savepoint store, fromcheckpoint.saveandcheckpoint.save_slot. A pass that returned before writing records nothing: no write earned yet, no witness, the unit removed, the lease contended, an unserializable state. Those are refusals, and counting them as failures would read as a failing disk. For the same reasonoutcomecompares what landed against what was OFFERED, not against the whole bundle -- a session pass skips any fold whose boundary its witness does not certify, so holding the pass to those too would report a refusal as a failure. Whether the bundle may then claim a savepoint is a stricter question, answered separately insaveas before. Both points are taken in afinally, so a raise through the store counts asfailedrather than going unrecorded, and both are emitted afterrelease_lease: the firstget_recorder()of a process builds the SDK synchronously, and doing that under a held lease would lengthen the window in which a removal of that unit is refused.kirocrew.crew_log.bus.subscriber_error{kind}-- the two branches that swallow a subscriber's exception. Failure path only, and the count is what the log line cannot give: a WARNING per failed event carries neither a rate nor a total.The one honest caveat, stated rather than papered over.
bus.subscriber_erroris the only call here a crew log write thread can reach.tree_advancedis published from the session-tree projection, which asession/openedorsession/spawnedjob advances while the write-behind is running that job, so a subscriber of that kind runs on the write thread. Three things bound it: a subscriber that returns normally costs nothing at all; the raising path already formats and emits a WARNING, which costs more than the counter beside it; and the counter resolves the metrics module fromsys.modulesrather than importing it, so a write thread can never trigger the first load of the facade, its config read or its SDK build.fold_advancedis published only from the eager fold worker. Both bounds have tests.Why
append.bytesis not here. Three sources were considered and all three are either work on the write path or wrong:counter.add, or an integer field on the batch pass. This is work on exactly the thread the constraint protects, so it is out regardless of how small it is.store_segment_writes.py::_write_then_syncis the one place every byte passes through and has the exact length already encoded, which makes it the most accurate option -- and it is still on the write path. Out for the same reason._pending_total_bytes,_pending_bytes); recovering "written" from them needs a cumulative accepted total, which does not exist, so it would take a new field on the enqueue side -- itself on the turn path. And it would still be wrong three ways: the inline fast path runs a job without ever buffering it, so its bytes never enter the pending total at all; there is no cumulative dropped-byte figure to subtract, because a unit's loss debt is consumed when its marker is written; and the underlyingnbytesis the producer's size hint, which is a CHARACTER count carried by only the two body-bearing entry types -- so a non-ASCII message would be under-reported by roughly a third under a metric whose unit says bytes, andturn/started,session/*,work/recordedand the rest would contribute zero.append.batch(write + fsync time) andfold.lagare out for the first reason alone: nothing keeps per-batch timing or enqueue time today, and the only way to produce either is atime.monotonic()pair on that path.A metric that is absent costs a question nobody can answer yet. A metric that is present and wrong costs more, and an operator cannot tell which one they are reading. So these three wait for a source that is both exact and off the write path, and
TestOmissionsAreDeliberatekeeps them out until one exists -- a test rather than a comment, because "every source was on the write path" is a decision a later reader cannot reconstruct from the code that is here, and a plausible-looking source is exactly what would get one re-added.Vocabulary. Every attribute value comes from a closed enum with
otheras a member, so an unrecognised value maps into the enum rather than past it.savepoint.writeis at most 24 series andbus.subscriber_errorat most 3. No unit id, slot key, session id, fold name, path or exception text is ever an attribute.Backwards compatibility
Compatible: every instrument is additive and no existing behaviour changes.
_write_batch,submit,_write_then_syncandWriterStatsare untouched; the only edits to existing functions add a counter call on a branch that already existed, plus one new read-only helper (eager.queue_depth).Removes nothing: no field, key, record, function or path is deleted, renamed or narrowed.
Tests
test/metrics/test_crew_log_health.py, 58 cases, grouped by the guarantee each locks in.The write path gained nothing.
TestTheWritePathIsUntouchedasserts it four ways, because the claim has four parts. A real batch pass is driven with the entry point patched, counting every metrics call and every clock read strictly between its entry and its exit -- both must be zero -- and then driven again with a job that publishestree_advancedto a well-behaved subscriber, the one indirect route from that thread to this module._write_batchis read from the parsed source and no call it makes may name a clock, a counter, a recorder, a metric or a lock other than the write-behind's own, so a call added later on a branch no test drives still reddens._write_then_syncis read the same way and may contain noglobaland no augmented assignment, which is what keeps an accumulator from reappearing in the funnel. And both modules must import nothing fromkiro_crew.metricsat all, lazily or otherwise.The omissions stay omitted.
TestOmissionsAreDeliberatepins the shipped name list exactly, asserts no registered name containsappend.bytes,append.batchorfold.lag, and asserts the metrics doc promises none of them -- so re-adding one without a source off the write path fails here rather than sliding past review.Nothing a collection does reaches the crew log. Each reader answers
Nonewith its source absent fromsys.modules, and a parsed-source case asserts neither reader contains an import statement at all.Refusals are not failures. A fold skipped at another boundary counts as
written. A parsed-source case assertssave's outcome expression compares against what was offered and contains nolen(; another asserts both call sites sit inside afinallyand afterrelease_lease, since a point moved out of either stops being correct in a way no behavioural test over a store that never raises would notice.The rest of the acceptance list. Session vs slot kind, and both separable; a save that raises counts
failedand the raise propagates; each bucket edge, from an injected clock pair and never a sleep, plus a clock that went backwards and a NaN; one raising subscriber is one point, a second raise is a second point, a non-raising subscriber is none, an unknown event kind maps toother, and the publisher still learns nothing; an empty queue reports zero while an absent one reports nothing; the gauges equal live state at collection; a raising probe is a gap rather than an exporter failure; a telemetry failure never reaches the savepoint or the bus; with telemetry off nothing is recorded and no instrument is created; and a meter that blows up at registration is logged and swallowed.Also run: the four crew log suites this touches (bus, checkpoint, slot savepoint, write-behind) and the provider suite -- 257 cases with the new file, all passing -- plus
flake8,isort,black,mypyoversrc/kiro_crew/(3304 files) andscripts/docs_lint.py.Manual verification
N/A -- unit coverage sufficient. Both gauges are exercised through a real in-memory SDK pipeline (
InMemoryMetricReader+MeterProvider), and both counters are driven through their production call sites, so no path here is reachable only by an integration run.Related Issues
Part of the crew log health section of the usage-metrics work (second batch, minus
fold.lag).Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)