Repository navigation
feat: emit WakaTime cadence heartbeats at coding-tool dispatch - #13177
Conversation
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that WakaTime's keystroke timeout really defaults to two minutes, as What this change shipsInventory (6 items) — 4 justifiedIntent: let a developer's agent coding turns build up real coding time on their WakaTime dashboard, not only AI attribution (closes #11336). This is an ADDITION behind the existing default-off
[FIRST-PRINCIPLES-REVIEWED] d61eac3 Description read[DESCRIPTION-READ] f49866a9ceb75adacc73376331d3753fa7b16ae4f5d28852cd828fbeb39a2f1d That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/dashboard/chat_runner.py:9318 -- “Fires only after … an approved permission decision” contradicts the completed |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check a live WakaTime dashboard to confirm a multi-tool agent turn now adds duration. The unit tests stub the client, so they never test that premise. [DESIGN-REVIEWED] d61eac3 Description read[DESCRIPTION-READ] 6e5399fc3289650a65d25bd338f8f2a3dbea3d8f3aa4ee8ce226653de5ef7284 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
3e3e3c1 to
4c2aace
Compare
|
Both blocking findings from the GPT review of F1 (BLOCKING-FENCED, F2 (BLOCKING, Verified: |
|
🤖 Kiro Crew [operator: dodgeblaster#b5f0d847]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. |
The heartbeat producer sent one beat at end-of-turn, so a turn landed its AI attribution (tokens, lines, prompts, sessions) but almost no wall-clock duration: WakaTime derives duration from the gaps between heartbeats, and a single beat gives it none. A long agent turn therefore read as an instant against the editor plugins' own time. Emit an attribution-free cadence heartbeat on each coding-tool dispatch (the same is_coding_event classification the end-of-turn beat uses), so several beats span a turn and WakaTime accumulates duration the way an editor plugin does. The cadence beat carries no token or line-change fields — those ride the end-of-turn beat alone — so it can fire before a tool's permission decision without ever counting a denied tool as activity, and it never double-counts attribution however often it fires. Dispatch-time emission is throttled per turn to CADENCE_MIN_INTERVAL_SECS (120s), matching WakaTime's ~2-minute coalescing bucket so a burst of tool calls does not flood the API. The beat honors the same scope as the end-of-turn beat: interactive user turns only, never a restricted or app slot, and it re-checks the send opt-in inside the fire-and-forget task. Closes #11336.
4c2aace to
1c0c673
Compare
|
Pushed Opus BLOCK (restored-replay scope) fixed. The cadence scope gate now carries the same Advisory same-timestamp beat removed. The end-of-turn Merge conflict cleared. The branch was behind main; it is rebased onto the current tip, so the import collision in Verified on the rebased tree: 397 tests pass (including the updated cadence tests and the new restored-replay regression), black format and baseline, isort, mypy strict on both changed source files, and the sync-io-in-async gate all green. The sync-io gate passing confirms the cadence producer still does zero on-loop config I/O. GPT's last verdict was against the old SHA |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
e007468 to
fb6efb0
Compare
fb6efb0 to
a296ada
Compare
a296ada to
2d1c1c0
Compare
|
This one is ready for a maintainer review when someone has a moment. What it does: agent coding turns currently accrue WakaTime attribution but little coding-time duration, because cadence beats only fired on an explicit permission approval. This makes an auto-approved (pre-authorized) coding tool emit one throttled cadence beat at its completed result frame, so a turn of trusted tools accrues intra-turn duration like a prompted one. Closes #11336. State on head The sole remaining gate is this human approval. Two small open questions are flagged in the reply for your call, not blockers: whether |
A coding tool that runs under a pre-authorized trust decision never passes through the permission-request path, so the per-approval cadence beat never fires for it. Emit the throttled beat at the EVENT_TOOL_RESULT completed frame for a coding tool whose pending key is still present, which is exactly the auto-approved case: an approved tool's key was drained at the permission request and already produced its beat, so this adds no second beat for it, and a denied tool never reaches a completed frame with a pending key. The scope guard and the per-turn throttle are unchanged, so the denied-tool and restored/restricted/app guarantees still hold. Restore test_mcp_name_fallback_is_not_used_by_chat_runner as its own method; its body had been grafted onto the tail of the restored-queued test, losing its name and isolation. Await the scheduled send task under a bounded timeout in the async heartbeat tests instead of a fixed count of asyncio.sleep(0) yields. The producers schedule send_heartbeat with loop.create_task, and send_heartbeat awaits asyncio.to_thread for its destination read before it reaches the client, so a thread-pool round-trip is not deterministically outlasted by any fixed number of bare event-loop yields. A create_task spy captures the scheduled tasks and the test awaits them with a time limit, so the batch assertions run after the send has actually resolved. Cite WakaTime's own two-minute edit-heartbeat throttle convention and its duration-combining algorithm on CADENCE_MIN_INTERVAL_SECS, so the 120s window is grounded in the provider's documented default rather than an assumed one.
2d1c1c0 to
d61eac3
Compare
|
Addressed the GPT 6.1 blocking finding (F1, test determinism) on The flagged tests polled for a scheduled send with a fixed count of Fix: a Verified on On the two advisory First Principles CONCERNS, since they ride the same revision: Item 4 (the 120s / WakaTime-timeout premise): Item 3 ( |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR gives WakaTime intra-turn timestamps so agent coding turns accrue duration, not just attribution. It adds an attribution-free cadence heartbeat, throttled to one per 120 s per turn, fired after an approved coding call and at the completed result frame of an auto-approved one. The end-of-turn attribution beat is unchanged. I read heartbeats.py, both call sites in chat_runner.py and the gating around _wt_pending_coding. The security-class concerns from dodgeblaster's 2026-10-03 note are fixed on head d61eac3. A denied call's key is discarded at EVENT_PERMISSION_REQUEST, so it can reach neither the approval beat nor the result-frame beat. The scope guard now carries not _turn_provenance_restored beside the user-actor, restricted-slot and app-slot checks, so a restart replay, cron or injected turn emits nothing. The cadence beat carries only the existing project label, adds no new egress field, and send_heartbeat re-checks the default-off wakatime.send_heartbeats opt-in off the loop before sending. The GPT determinism finding was answered by waiting on the captured send tasks instead of counted yields. Checks are green and both lanes PASS. Small nit: the _wt_maybe_cadence and note_coding_cadence docstrings still say the beat fires only on an approved decision. The needs-author-decision label was set while those blockers were open and looks stale now; a maintainer should remove it. — crew-pr-reviewer (deep re-check)
Why no screenshot: backend-only change — the WakaTime heartbeat producer (
wakatime/heartbeats.py) plus its post-gating call sites indashboard/chat_runner.py(the per-approval path and the completed-result frame). No frontend paths are touched and nothing renders.Problem / Motivation
Goal: make a developer's agent coding turns accrue real coding-time duration on their WakaTime dashboard, not just AI attribution.
WakaTime AI-coding heartbeats from Kiro Crew (added in #9450) carry correct attribution — AI lines, tokens, prompts, sessions, spend — but accrue almost no coding-time duration. WakaTime derives duration from the cadence between heartbeats, and #9450 emits one heartbeat at end of turn, so a coding turn has no intra-turn cadence for WakaTime to interpolate across. Observed on a live dashboard: agent turns populate the AI-coding panel and per-project AI stats while wall-clock duration is driven only by the editor plugins.
Why it matters
A developer who works through the agent sees their agent coding time land in WakaTime as attribution but close to zero duration, so the AI-vs-human time breakdown still undercounts the agent's share and billable-hours totals stay driven by the editor alone. Duration accrual is what makes the shipped send-side integration reflect real time spent, which was the point of feeding agent work into the AI-coding lane.
Not a goal
What changed (motivation → approach → change)
Goal: give WakaTime enough intra-turn cadence to accrue duration, without disturbing the attribution the end-of-turn beat already gets right.
Approach: emit an attribution-free cadence heartbeat at the points where a coding tool is known to have executed, in addition to (not instead of) the end-of-turn attribution beat. Keeping the two separate is what makes the change safe: the cadence beat carries no token or line-change fields, so it can never double-count attribution however often it fires, and it is only ever scheduled past the gating decision, so a denied tool is never counted as activity. Only the end-of-turn beat carries the deltas. The alternative (emitting at dispatch time, before the permission decision) was rejected because it could schedule a beat for a tool that is then denied.
Change: a new
note_coding_cadenceproducer builds the field-freeai codingbeat and schedules it fire-and-forget. A single_wt_maybe_cadence()helper in_run_chatowns the per-turn throttle (CADENCE_MIN_INTERVAL_SECS, 120s) and the scope guard (interactive user turns only, never restored, restricted, or an app slot), and re-checks thewakatime.enabled+wakatime.send_heartbeatsopt-in inside the send task. It is called from two post-gating points: the per-approval path (_wt_note_approved) for prompted coding tools, and theEVENT_TOOL_RESULTcompleted frame for coding tools that executed without a decision. The pending-key discriminator keeps each coding tool to one beat: an approved tool's key is drained atEVENT_PERMISSION_REQUEST, so it is gone by result time and does not re-beat; an auto-approved tool's key is still pending at its result, so it beats once there. The 120s throttle matches WakaTime's coalescing bucket so a burst of tool calls produces one beat rather than flooding the API.Since first review (commit e007468)
The First Principles and Design reviews raised the same advisory CONCERNS: the cadence beat fired only from the per-approval path, so a turn made entirely of auto-approved / pre-authorized coding tools accrued no intra-turn duration, which is exactly the harm #11336 names for operators who trust their write tools. The second commit closes that gap. A throttled cadence beat now also fires in the
EVENT_TOOL_RESULTarm when a coding tool reaches its completed frame, where the tool has demonstrably executed and the decision point is already past. The discriminator keeps every coding tool to exactly one cadence opportunity: an approved tool's pending key is drained atEVENT_PERMISSION_REQUESTand is gone by result time (no re-beat), while an auto-approved tool's key is still pending so it beats once here and nowhere else. A denied tool never reachestool_finalwith a pending key, so it still emits nothing, and the_wt_maybe_cadencescope guard (user-only, not restored/restricted/app, 120s throttle) is unchanged. The security-class concern from the earlier sha (emission before the permission decision) does not apply to this emission point, which is strictly past the tool result. Two review-flagged test defects are also fixed: the throttle is now pinned bytest_many_auto_approved_coding_results_emit_one_throttled_cadence(five completing coding calls, one beat), andtest_mcp_name_fallback_is_not_used_by_chat_runneris restored as its own method after its body had been grafted onto the tail oftest_restored_queued_turn_emits_no_cadence_heartbeat, which had lost its name and pin.Backwards compatibility
Compatible: adds one new opt-in-gated producer and one throttled call site; changes no config key, signature, or default, and sends nothing unless the existing
wakatime.send_heartbeatsflag (default off) is set.Removes nothing: no path, record, validator, key, or public function is removed, renamed, or tightened; the change is purely additive behind an existing default-off flag.
Tests
test/test_wakatime_heartbeats.py:test_cadence_heartbeat_carries_no_attributionlocks in that a cadence beat carries noai_input_tokens/ai_output_tokens/ai_line_changes;test_send_flag_off_schedules_nothingandtest_cadence_send_drops_when_opt_in_is_offlock in that it is a no-op when the send flag is off;test_cadence_schedules_a_task_without_reading_configlocks in the fire-and-forget schedule.test/test_chat_runner_coverage.py:test_approved_coding_call_emits_one_cadence_heartbeatlocks in that an approved coding call emits one cadence beat;test_auto_approved_coding_call_emits_one_cadence_heartbeatlocks in that an auto-approved call (no permission decision) emits one beat at its completed result frame;test_many_auto_approved_coding_results_emit_one_throttled_cadencedrives five completing coding calls in one turn and pins the 120s throttle collapsing them to a single beat.test/test_chat_runner_coverage.py:test_denied_coding_call_emits_no_cadence_heartbeat,test_restored_queued_turn_emits_no_cadence_heartbeatandtest_injected_turn_emits_no_cadence_heartbeatlock in that a denied, restored, or non-user (cron/injected) turn emits none.Manual verification
N/A — unit coverage is sufficient. The producer is exercised through the stubbed
WakaTimeClientand the dispatch path through a scripted_run_chatturn; a live WakaTime call is covered by the client's own tests from the first backend PR (#8169).Related Issues
Closes #11336.
Checklist
feat: ...)heartbeats.pymodule docstring now describes cadence + end-of-turn emission