Skip to content

feat: send WakaTime AI-coding heartbeats for agent coding turns - #9450

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
Behordeun:feat/wakatime-heartbeat
Sep 23, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
Behordeun:feat/wakatime-heartbeat

Conversation

@Behordeun

@Behordeun Behordeun commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Kiro Crew can already read a user's WakaTime coding stats (the productivity view and CSV/JSON export shipped earlier), but it never sends anything back. A developer's Kiro Crew coding time is invisible in their WakaTime dashboard: the agent edits files and runs commands for them, yet none of that activity shows up alongside the time logged from their editor. WakaTime now models AI-assisted coding as a first-class category with its own AI Percent, AI Prompts, and AI Sessions views, and Kiro Crew contributes nothing to them.

Why it matters

Developers who track their time in WakaTime lose the portion of it that happens through Kiro Crew. For anyone billing hours or measuring where their coding time goes, that is a real gap: agent-driven work is a growing share of a day and currently reads as zero. Feeding it into the AI-coding lane makes the picture complete and lets the existing AI-vs-human breakdown reflect reality.

What changed (motivation → approach → change)

Goal: turn a completed agent coding turn into a WakaTime heartbeat, using the client's existing bulk-send method (which had no producer).

Approach: a small in-process producer, hooked at end-of-turn where the usage telemetry already fires. A turn counts as coding activity only when it dispatched a shell command (as resolved by the whole-frame shell classifier — a raw execute tool kind alone does not count, since codex-acp labels read-only MCP calls with it) or a file-write tool; the tool-NAME fallback applies only to native tools, so an MCP server exposing a tool named write is never misread as coding. Each coding tool call's id is held until its permission decision: a denial through Kiro Crew's permission gate drains it without counting, an approval counts it, and an id still held when the turn ends counts then — so a tool Kiro Crew's gate denied never logs time, while a harness that never sends permission events still has its executed tools counted at end of turn. The held set is bounded; ids beyond the cap are dropped and reported in one end-of-turn log line. Heartbeats are produced only from ordinary interactive sessions: a restricted (incognito/temporary) session and an app-owned slot never emit one, and a turn injected by a cron job, an automation loop, or a sub-agent does not count. The project a heartbeat is attributed to is captured at turn start, so a project switch landing mid-turn cannot relabel the work. Each qualifying turn schedules ONE fire-and-forget send task; the destination (base URL and API key) and the opt-in are resolved together inside that task at send time, so nothing is buffered across a credential rotation. The send never awaits on the turn's hot path and a failure drops the row rather than retrying — WakaTime already dedupes by two-minute buckets.

The heartbeat is mapped to WakaTime's ai coding category and carries the turn's real signals: input and output token counts read from the turn's usage, a total line-change count computed from the file-change snapshots the turn already keeps (a truncated snapshot is skipped, so an oversized file cannot inflate the count or stall the turn), and the project directory basename as the entity. No session identifier is sent. Editor and operating system are resolved by WakaTime from the request User-Agent rather than a body field, so the client sends a plugin-style User-Agent carrying Kiro Crew's real package version on every WakaTime API request, replacing the default that would log as an unknown OS.

Sending is a separate opt-in from reading. A new wakatime.send_heartbeats config flag defaults off, and the producer is gated on both it and the existing wakatime.enabled, so a user who has not opted in pays nothing and sends nothing. The token and line counts come from the turn that just ran, so they are genuine since-the-last-heartbeat deltas; a line count is only sent when it can be computed, never guessed. The project entity is the directory basename only, never an absolute path.

Where each piece lives:

Area Change
src/kiro_crew/wakatime/heartbeats.py (new) the heartbeat producer: coding-turn detection with an MCP-aware name fallback, a bounded redacted-then-truncated entity label, and one per-turn fire-and-forget send with send-time destination resolution
src/kiro_crew/wakatime/client.py plugin-style User-Agent with the real package version on every WakaTime API request (set in the shared request headers, so stats reads carry it too)
src/kiro_crew/dashboard/chat_runner.py end-of-turn hook: holds each coding tool id until its permission decision or end of turn, collects token/line signals, skips truncated file snapshots, and skips restricted and app-owned sessions
src/kiro_crew/config/sections.py, src/kiro_crew/config/loader.py the wakatime.send_heartbeats opt-in flag (its help text names every exclusion, including incognito/temporary sessions) and its loader passthrough
config-baseline.json baseline entry for the new flag
src/kiro_crew/security_posture.py WakaTime heartbeat send listed as a posture-visible outbound sink
test/test_wakatime_heartbeats.py (new) producer coverage

Tests

New test/test_wakatime_heartbeats.py covers the producer end to end with a stub client (no network):

  • The integration disabled, or the send flag off, sends nothing.
  • An enabled, opted-in coding turn sends one ai coding heartbeat carrying the token counts, line-change count, and project basename — and no session identifier.
  • Zero-valued AI fields are dropped rather than sent as zero, so per-session averages are not diluted.
  • The line-change helper counts added and removed lines, returns a floor of zero when an after-snapshot is missing, and is zero for malformed input.
  • The entity is the project basename, never the full path, is redacted BEFORE it is truncated to its bound (a credential straddling the cut cannot leak a fragment), and falls back to a stable label when there is no project.
  • The send task resolves the destination and re-checks the opt-in at send time, swallows a send failure without raising, never runs on the turn's hot path, and is a no-op when no API key resolves.
  • Only shell commands (via the whole-frame shell signal) and file-write tools count as coding activity; read-only tools do not, including an event carrying a raw execute kind with the shell signal false. An MCP-served tool named write does not count; a native fs_write with an empty tool kind still does; an MCP tool with edit kind still counts on the kind evidence.

test/test_chat_runner_coverage.py locks the accounting: a coding id drains uncounted on a Kiro Crew-mediated denial and a denied-only turn emits no heartbeat, an approval counts it, an id that never receives a permission decision still counts at end of turn, ids past the per-turn bound are dropped with exactly one end-of-turn summary line, a truncated file snapshot contributes nothing to the line count, injected (cron/automation) turns emit nothing, and the heartbeat project is the one captured at turn start.

The existing WakaTime client tests continue to pass with the added User-Agent.

Manual verification

Verified against the live WakaTime dashboard: with the flag enabled, agent coding turns land in the account's AI-coding activity, and the AI Percent and AI Prompts panels populate for the project rather than showing the activity as unknown-OS human time. The token counts and line-change counts were confirmed to flow into the AI lane using the field shapes documented in the WakaTime heartbeats API. Portal screenshots are personal-account data and are not committed.

Screenshots / video

Why no screenshot: backend-only change — a heartbeat producer and config flag with no dashboard UI.

Related Issues

Part of #8094

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Behordeun
Behordeun requested review from a team and dwu96 and a lite review from Copilot September 8, 2026 10:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 8c7d5fd36dd4e656df3d3fb158a341e5a3bf70e5 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

One end-of-turn heartbeat per turn undercounts duration: a long agent turn collapses to a single timestamp, so billed coding time stays near zero.

Watch

  • WakaTime credits time from gaps between heartbeats inside its keystroke timeout; the design's one heartbeat at STOP_REASON_END_TURN (timestamped time.time() at construction) means a 30-minute turn reads as an instant, and turns spaced past the timeout accrue ~nothing — the description's own harm ("billing hours… agent-driven work reads as zero") is only partly removed. The manual verification ("AI Percent and AI Prompts panels populate") confirms activity landed, not that durations accrue.
    Clears when: verified duration totals on the live dashboard for a multi-minute turn, or heartbeats emitted per approved coding tool call (WakaTime's two-minute dedupe, which the description already cites, bounds the volume).

Suggestions

  • _wt_collect runs a full KiroCrewConfig.load() on every interactive coding turn even with the feature off, and re-reads after-snapshots that _flush_file_changes (chat_runner.py:2346) reads again moments later — gate on the already-loaded config and reuse the flush's before/after pairs to keep turn-end cost flat for non-opted-in users.

[DESIGN-REVIEWED] 8c7d5fd

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 8c7d5fd36dd4e656df3d3fb158a341e5a3bf70e5 via the fork AI-review pipeline; updated in place on each push.

Review details

Two advisory findings; nothing blocks the merge.

FINDING — src/kiro_crew/wakatime/heartbeats.py:114 — difflib.SequenceMatcher(a=before_lines, b=after_lines, autojunk=False) explicitly disables the popularity heuristic that is the only bound on find_longest_match's for j in b2j[a[i]] inner loop, and nothing caps line count: an edit to an existing ~200 KB file of short, largely repeated lines (a generated CSV/JSONL/log fixture) is untruncated, so both sides reach ~100k lines and the awaited await asyncio.to_thread(_wt_collect) in the END_TURN arm pins the turn for minutes on a GIL-holding pure-Python loop that cancellation cannot interrupt — unlike the repo's existing make_unified_diff, which keeps the default autojunk → Fix: skip a pair whose before_lines/after_lines exceed a line-count bound, or drop autojunk=False.

FINDING — src/kiro_crew/config/sections.py:7887 — "Enable the WakaTime integration (send coding-activity heartbeats and read back stats)" is now false: _wt_collect requires _cfg.wakatime.enabled and _cfg.wakatime.send_heartbeats, so an operator who sets enabled alone reading that sentence gets no heartbeat and no error → Fix: amend the enabled help (and its config-baseline.json:6485 copy) to say sending is a separate opt-in.

[OPUS-REVIEWED] 8c7d5fd

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 8c7d5fd36dd4e656df3d3fb158a341e5a3bf70e5 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_wt_collect re-implements the per-turn snapshot resolution _flush_file_changes already runs — a second spelling that will silently diverge.

Not justified as shipped

  • Item 10 — duplicate of src/kiro_crew/dashboard/chat_runner.py:2346: the dedupe-by-path / _safe_read_snapshot / truncation-flag block in _wt_collect re-spells the resolution _flush_file_changes performs on the same slot._file_changes every turn (grep _safe_read_snapshot + truncated in chat_runner.py: 2 resolution sites after this diff). The copies already differ on unreadable and truncated entries, which is how spellings diverge.
    Subtraction: delete _wt_collect's resolution loop; compute the line count from the resolved entries _flush_file_changes attaches (skipping its truncated-marked rows), moving the note after that flush.
    Clears when: the duplicate resolution block is deleted in favor of the flushed entries, or the heartbeat count is shown to need content that resolution cannot supply (it redacts in place before attaching).

What this change ships

Inventory (10 items) — 9 justified

Intent: make agent coding time appear in the user's WakaTime AI-coding dashboard — an ADDITION.

  1. Opted-in agent coding turns now appear in WakaTime's AI-coding views — justified
  2. New wakatime.send_heartbeats config key, default off — justified
  3. Every WakaTime request (stats reads too) now sends a Kiro Crew User-Agent — justified
  4. A coding tool denied at the permission gate never logs time — justified
  5. Incognito/temporary and app-owned sessions never emit a heartbeat — justified
  6. Cron, automation, and subagent turns never emit one — justified
  7. Heartbeat project is the one captured at turn start — justified
  8. One end-of-turn warning when the pending-id cap sheds keys — justified
  9. Heartbeat send registered as a posture-visible outbound redaction sink — justified
  10. Line-change count re-resolves the turn's file snapshots itself — duplicate of chat_runner.py:2346 (_flush_file_changes)

[FIRST-PRINCIPLES-REVIEWED] 8c7d5fd

@Behordeun
Behordeun force-pushed the feat/wakatime-heartbeat branch from be3df69 to b034735 Compare September 8, 2026 10:29
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 8, 2026
@Behordeun

Copy link
Copy Markdown
Contributor Author

FINDING (chat_runner.py end-of-turn heartbeat block) — fixed in b03473510d4011e5b496232ed5e315a9910b552d.

The finding holds: the end-of-turn block ran the _safe_read_snapshot loop over slot._file_changes before the send opt-in was consulted, so a default install (flag off) paid discarded synchronous file reads on the coding path.

Fix: added heartbeats.heartbeats_enabled(config), which returns true only when both wakatime.enabled and wakatime.send_heartbeats are set. The chat_runner block now calls it and early-outs before entering the _safe_read_snapshot loop, so a default install does zero disk or config work on the coding path. When the flag is on, the reads happen once here and feed the line-change delta.

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 8c7d5fd36dd4e656df3d3fb158a341e5a3bf70e5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8c7d5fd

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@Behordeun
Behordeun force-pushed the feat/wakatime-heartbeat branch from b034735 to f0e6bd3 Compare September 8, 2026 10:51
@Behordeun

Copy link
Copy Markdown
Contributor Author

BLOCKING (feature-map-correctness, loader omits send_heartbeats) — fixed in f0e6bd3e114a35b802ba5d84794a25a14454423c.

Correct and the most important one: the loader constructed WakaTimeConfig(enabled=..., api_base_url=...) without the new field, so the opt-in stayed False regardless of config and the feature was silently inert.

Fix: src/kiro_crew/config/loader.py now passes send_heartbeats=bool(wakatime_data.get("send_heartbeats", False)). Verified by the config round-trip test that the field survives load. The config-baseline snapshot already carried the field.

@Behordeun

Copy link
Copy Markdown
Contributor Author

BLOCKING (no-blocking-call-on-event-loop, synchronous snapshot read) — rebutted (disproportionate) at f0e6bd3e114a35b802ba5d84794a25a14454423c.

The adjudication in this same review DOWNGRADED this finding, and the code now narrows it further.

The identical synchronous _safe_read_snapshot already runs unconditionally on every turn-exit path in _flush_file_changes, so this block adds no new class of blocking call; it mirrors the module's established pattern for a rare, self-correcting read. It is now also gated behind heartbeats_enabled(), so it only runs when the user has opted in, and never on a default install. Offloading a common fast read to a per-file thread hop would diverge from the module's own convention for a latency blip that self-corrects the instant the read returns.

@Behordeun

Copy link
Copy Markdown
Contributor Author

BLOCKING (residual/crash-data-loss-corruption, buffered heartbeat can remain unsent) — fixed in f0e6bd3e114a35b802ba5d84794a25a14454423c.

Holds: a final coding turn inside the flush interval left its heartbeat buffered with no timer armed, so it was lost on exit.

Fix: _maybe_schedule_flush now arms a one-shot loop.call_later for the remaining interval when a heartbeat lands inside the window, instead of returning. _fire_delayed_flush drains the buffer when it fires. One timer at a time, tracked by _delayed_flush_armed. Covered by test_delayed_flush_sends_a_buffered_row_within_the_interval.

@Behordeun

Copy link
Copy Markdown
Contributor Author

BLOCKING (residual/crash-data-loss-corruption, repeated edits inflate line-change telemetry) — fixed in f0e6bd3e114a35b802ba5d84794a25a14454423c.

Holds: the block iterated slot._file_changes without deduping by path, so a file edited twice diffed each stale before-snapshot against the same final content and double-counted.

Fix: the loop now dedups by path keeping the first before-snapshot (a _wt_seen set), matching _flush_file_changes. A multi-edit file contributes one before/after pair, so the line-change count is not inflated.

@Behordeun

Copy link
Copy Markdown
Contributor Author

BLOCKING (residual/crash-data-loss-corruption, rejected tools counted as coding activity) — fixed in f0e6bd3e114a35b802ba5d84794a25a14454423c.

Holds: _turn_coding_activity was set on EVENT_TOOL_CALL, a pre-execution notification, so a hook-denied write/shell request flipped the flag and emitted a spurious heartbeat.

Fix: EVENT_TOOL_CALL now only records the coding tool_call_id as pending (_wt_pending_coding). The flag is promoted to true at EVENT_TOOL_RESULT (confirmed execution), and the pending id is discarded in the TOOL_DENY branch, so a denied tool can never promote to activity even if a result event follows.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 8, 2026
@Behordeun

Copy link
Copy Markdown
Contributor Author

FINDING (top-level-imports, in-function heartbeat import) — rebutted (disproportionate) at f0e6bd3e114a35b802ba5d84794a25a14454423c.

The in-function import is deliberate and the top-level alternative would break a stronger constraint.

kiro_crew.wakatime.heartbeats imports service.build_client at module top, which imports wakatime/client.py, which imports aiohttp at module top. Importing heartbeats at the top of dashboard/chat_runner.py would therefore pull aiohttp onto the gateway boot path, which is exactly the boot-path import that PR 2 removed for build_client. The top-level-imports rule is blocking: false; the boot-path constraint it would violate here is the load-bearing one, so the lazy import stays. The imports are grouped and named, so traceability is preserved.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 8, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

BLOCKING (MCP write misclassification) — fixed span=03a5a6d3c12e
mechanism: MCP-identity gate on the tool-name fallback in is_coding_event

Fixed in 6c8bec8: is_coding_event now takes mcp_server_name and consults the CODING_TOOL_NAMES fallback only when it is empty (native tool), so an MCP server exposing a tool named write can never route into the coding lane; is_shell and tool_kind=="edit" classification are unchanged for all calls.
Retrospective note: this span (path+lane) is at its 3rd occurrence — three distinct defects in heartbeats.py (r2 TOCTOU, r4 unbounded label, r6 name fallback); the round-6 retrospective's accepted First-Principles subtraction (see the target=first-principles record) removes the buffer apparatus that hosted the first of them.
Tests: MCP-served write is not coding; native fs_write with empty kind still is; MCP tool with edit kind still counts on kind evidence.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

FINDING (sections.py Slack-scope wording) — fixed span=1b88ed28a3ad

Fixed in 6c8bec8 with the review's exact wording: the send_heartbeats help text now says "Covers dashboard chat and dashboard-linked Slack threads"; config-baseline.json regenerated to match.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

NOT JUSTIFIED (buffer/timer/fingerprint apparatus) — fixed by subtraction span=d8e90a425f8b
mechanism: subtraction — _buffer, _maybe_schedule_flush, _fire_delayed_flush, _flush, _fingerprint_destination, _resolve_destination, _destination_fingerprint, _MAX_BUFFERED and the overflow accounting are deleted; note_coding_activity now schedules one fire-and-forget send_heartbeats([hb]) per turn with the destination and opt-in resolved together inside the send task

Accepted in 6c8bec8 per the round-6 retrospective: the buffer, 120s timer, and destination fingerprint existed only to serve batching; emission is gated to interactive user turns, WakaTime dedupes by two-minute buckets, and no measured request-volume cost justified retaining it.
This also retires the r2-added single-snapshot TOCTOU mechanism (spans retired: the r2 gpt fix under 03a5a6d3c12e) — nothing is buffered across a credential rotation, so the misroute class it guarded is unreachable by construction.
The _MAX_ENTITY_CHARS redact-then-truncate bound is kept: its rationale (never POST an unbounded or credential-shaped label) is independent of the buffer.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

NOT JUSTIFIED (_wt_collect duplicates the resolution loop) — rebutted span=366df3b9d275

The finding's own second clear condition holds: _wt_collect runs in the turn's EVENT_COMPLETE arm, before _flush_file_changes runs at turn exit (verified by ordering in round 3 of this PR: _wt_collect at the completion arm precedes the flush at turn teardown), so the resolved after-snapshots are not yet available on slot._file_changes when the count is taken.
The enqueue-before-exit ordering therefore precludes reusing the flush loop's resolution; the second _safe_read_snapshot pass is forced, and the count it produces is a valid floor either way.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

WATCH (turn-end-only cadence undercounts long turns) — accepted-and-deferred span=f890aead5299

Deferred to #13023 (label deferred-finding, assignee NicholasRBowers, Due: 2026-10-14): during-turn per-approved-tool heartbeats are exactly that issue's per-tool duration accrual, a stated round-0 non-goal.
Round-0 intent is the addition — agent coding stops reading as zero on the dashboard — which the turn-end heartbeat satisfies; duration-gap fidelity for multi-minute turns is the deferred scope.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

SUGGESTIONS (fold the duplicated count blocks) — fixed span=ac45e24f7120

Fixed in 6c8bec8: the per-arm if id in _wt_pending_coding: _wt_counted_coding.add(id) blocks are replaced by one _wt_note_approved(event) helper called from every approval arm, so a future arm cannot silently omit the count; done as part of the same rewrite that bounds the tool-id retention (span db37e66c9e96).

@Behordeun

Copy link
Copy Markdown
Contributor Author

F1 (BLOCKING, chat_runner.py heartbeat tool-ID tracking is unbounded) — fixed in 8fe0a63f6.

The pending-coding set now routes every retained id through the turn's shared bounded-admission helper _tcid_identity_key, so the stored value is a fixed 16-char digest instead of the raw backend id, and admission is capped at _MAX_TCID_SOURCES (the same bound the turn already applies to _tcid_first_source). Past the cap the key is dropped and the overflow is logged once; the set is fixed-width and cannot grow with the number of distinct backend tool ids in a turn. The EVENT_PERMISSION_REQUEST drain uses the same digest key, so admission and drain agree by construction and the approve/deny/auto-approve accounting is unchanged.

Covered by test_many_undecided_pending_coding_ids_stay_capped, which floods one turn with _MAX_TCID_SOURCES + 50 undecided pending coding calls and asserts the overflow warning fires and nothing is miscounted.

F2 (BLOCKING-FENCED, heartbeats.py MCP tools named write corrupt coding statistics) — already fixed at head.

is_coding_event gates the tool-NAME fallback behind not mcp_server_name, so an MCP-served call (which carries its server in mcp_server_name) never reaches is_coding_tool; only native tools use the name path. Pinned by test_mcp_tool_name_fallback_requires_native_identity and test_mcp_name_fallback_is_not_used_by_chat_runner.

FINDING (sections.py help text overstates Slack coverage) — already fixed at head. The send_heartbeats help now reads "dashboard chat and dashboard-linked Slack threads" and explicitly excludes the other messaging channels, the task runner, subagents, apps, and the CLI.

@Behordeun

Copy link
Copy Markdown
Contributor Author

All three findings from the GPT review of 2ee78b8f9 are fixed in 279a009a3.

F1 (BLOCKING-FENCED, chat_runner.py:11683 — statusless calls counted before permission) — fixed. A coding tool_call now retains the turn's shared bounded key whether or not the frame carried tool_pending, instead of the old else arm setting the turn bit immediately. Every retained key is gated at its permission decision (a denial discards it and never counts); a key still held at end-of-turn reached no gating denial, so it counts then. This makes a kiro statusless shell tool_call followed by a denial correctly record nothing, while a statusless call with no permission request still counts. Covered by test_statusless_coding_call_demotes_when_permission_is_denied and test_statusless_coding_call_without_permission_stays_counted.

F2 (BLOCKING-FENCED, chat_runner.py:14548 — truncated snapshots corrupt line-change telemetry) — fixed. _wt_collect now skips a file-change when either side is truncated — the before-side truncated flag on the change entry (from _snapshot_write_target) or _Snapshot.truncated on the after-side read — rather than diffing partial content and sending a false line-change total. A file past the 200k-char cap now contributes nothing instead of a fabricated count.

F3 (chat_runner.py:11677 — overflow logged per dropped call; adjudication DOWNGRADED to advisory) — fixed anyway. The per-call logger.warning is replaced by a turn-scoped _wt_pending_dropped counter and one bounded summary emitted in the end-turn arm, so exceeding the cap logs a single line per turn instead of one per additional event. Pinned by test_many_undecided_pending_coding_ids_stay_capped.

Verified: test_wakatime_heartbeats.py + test_chat_runner_coverage.py (333 pass), plus black/isort/flake8/mypy --platform linux and Codacy pylint clean on the changed files (remaining pylint hits are pre-existing lines untouched by this change).

@NicholasRBowers

Copy link
Copy Markdown
Contributor

self-added: yes
mechanism: coding-key retention until permission decision, with end-of-turn promotion

  • span=db37e66c9e96 — statusless coding tool counted before its permission decision (chat_runner.py:11683)

fixed in 279a009 (the author's push), carried on aa666c5: the premature else: _wt_coded_this_turn = True arm is gone.
Every coding key is now retained until its permission decision; a denial drains it uncounted; a key still held at STOP_REASON_END_TURN counts then.
The fenced scenario (statusless shell dispatch, then user deny) now emits nothing; locked by the denial-drain tests in test_chat_runner_coverage.py.
Residual, stated: a harness-internal refusal with no KiroCrew permission event counts at end of turn — an over-count of an attempted coding turn, and the PR body scopes the never-logs claim to KiroCrew-mediated denials.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

self-added: yes

  • span=db37e66c9e96 — truncated file snapshot fed to line-count telemetry (chat_runner.py:14548)

fixed in 279a009, carried on aa666c5: _wt_collect now skips a change entry when either side is truncated (if _fc.get("truncated") or _after.truncated: continue).
A truncated pair contributes zero (floor semantics) instead of a fabricated diff over a cut snapshot.
This also bounds the difflib inputs to the snapshot cap on both sides. Locked by the truncated-snapshot test in test_chat_runner_coverage.py.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

self-added: yes

  • span=db37e66c9e96 — per-drop WARNING spam on pending-set overflow (chat_runner.py:11677)

fixed in 279a009, carried on aa666c5: overflow drops are tallied into _wt_pending_dropped and announced in ONE end-of-turn summary line.
A duplicate re-announce of an already-held key is guarded from being mis-tallied as a drop.
The rewritten test asserts exactly one shed line per turn.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

mechanism: end-of-turn promotion of undecided coding keys; dead tool_pending plumbing removed

  • span=f906bca7bc7d — cross-harness pending semantics (a harness that never sends a permission event silently undercounts)

fixed in 279a009, carried on aa666c5: a held coding key now counts at end of turn regardless of harness, so the pi-style pending tool_call → tool_result flow (no permission event) produces a heartbeat.
test_statusless_coding_call_without_permission_stays_counted is exactly this replay case, and it is harness-independent because the tool_pending flag is no longer consulted — aa666c5 deletes that now-dead plumbing entirely (field, both set sites, provider forward, fixtures).
Residual, stated plainly: the failure mode moved from silent undercount to a narrow over-count on a harness-internal refusal (tool dispatched, refused inside the harness, no KiroCrew permission event) — acceptable telemetry semantics for an attempted coding turn.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=5d34daa2804d — count on terminal tool_result status instead of permission events

rebutted as disproportional: end-of-turn promotion (in 279a009) already achieves the harness independence this Suggestion wanted — STOP_REASON_END_TURN always fires — without depending on terminal-status fidelity.
Terminal-result counting would introduce a real undercount class: _dispatch.py's result builder documents rawOutput as unstructured passthrough, and a content-bearing result without status yields tool_final=False, so such a tool would never count.
Net cost: a new undercount class plus a rewrite of the accounting tests, to save ~12 lines of decision plumbing. Smaller-wins does not favor it.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=fe63d67e3f12 — batch-shaped module API with exactly one single-row producer

fixed in aa666c5: the module-level producer is now send_heartbeat(hb) taking one heartbeat; the if not heartbeats guard is deleted.
The client method keeps its batch signature (send_heartbeats([hb])) because the WakaTime bulk endpoint is genuinely batch-shaped; the singular wrapper documents that the batch exists at the API boundary only.
Locked by the updated tests in test_wakatime_heartbeats.py.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=e48125549479 — live kiro-cli sends statusless tool_call then session/request_permission for the same toolCallId (chat_runner.py:11682)

fixed in 279a009, carried on aa666c5: the statusless/pending distinction no longer decides counting — every coding key is retained until its permission decision, so the kiro/session.jsonl flow (statusless tool_call, then permission) counts exactly once, on the approval arm.
The tool_pending field this hinged on is deleted in aa666c5 as dead code (its only reader was removed by the author's fix).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=2958b0fb8002 — difflib.SequenceMatcher(autojunk=False) on snapshots bounded only by _MAX_SNAPSHOT (heartbeats.py:114)

fixed in 279a009, carried on aa666c5: truncated snapshots are skipped before diffing, so both difflib inputs are hard-bounded by the snapshot cap and an oversized file contributes zero instead of a worst-case quadratic diff.
The per-turn path count remains bounded by the turn's own file-change snapshot set, which the runner already caps.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=bd4b0fed318d — help text omits the restricted-session exclusion (sections.py:7903)

fixed in aa666c5: the send_heartbeats help now states that incognito and temporary sessions never send a heartbeat (they keep no record of the chat, and a heartbeat is an external record of it).
config-baseline.json regenerated with scripts/generate_config_baseline.py; the diff is that one string, and test_config_baseline.py pins it.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=26a67c893b0c — User-Agent sends literal wakatime/unset while the docstring promises a version (client.py:53)

fixed in aa666c5: the leading slot now carries Kiro Crew's real package version (wakatime/<ver> (<os>-<release>) kirocrew/<ver>), verified by importing _user_agent().
The docstring now states why both slots carry the Kiro Crew version (no bundled wakatime-cli), so the header and its description cannot drift again.

@Behordeun

Copy link
Copy Markdown
Contributor Author

GPT F1 (security-FENCED, heartbeats.py:234 — endpoint and key not resolved atomically) — rebutted as out of scope; not introduced by this PR.

The finding is accurate as a description of the seam, but the property it asks for is neither new to this change nor closable here in proportion:

  • The split pre-exists this PR. service.py build_client (the stats-read path that shipped earlier) resolves the pair the same unsynchronized way — resolve_api_key() then resolve_base_url(cfg) at service.py:56 and :59. The heartbeat producer reuses those same two resolvers (heartbeats.py:233), so it inherits the existing seam rather than adding one. A git grep on the base branch shows the pattern already present before this PR.

  • The trigger is a genuine edge, and the pipeline's own adjudication FLAGged it as such. Reaching a stale-key/new-endpoint pairing requires all of: the non-default double opt-in (wakatime.enabled and wakatime.send_heartbeats, the latter default off); a custom api_base_url (otherwise the endpoint is the public default and does not rotate); an operator mid-rotation who commits the config.json write before the vault write; and a heartbeat — at most one per turn — landing in the sub-millisecond gap between the two reads at heartbeats.py:233-234.

  • The exposure is bounded and self-correcting. The key that could leak is the pre-rotation key the rotation is retiring; the mismatched send 401s, is swallowed, and the row is dropped (heartbeats.py:241-242); the next turn re-reads both stores and self-corrects.

  • The proposed remedy is disproportionate and does not close the feared shape. A versioned cross-store commit protocol between config.json and the SecretVault exists nowhere in the tree; adding it for this path alone buys nothing while the identical pre-existing split in build_client remains, so the whole seam would have to convert. And an api_base_url pointed at the wrong host receives whatever key is live whether or not the pair is version-stamped — the control there is the config-write gate, not atomicity.

This is a security-class fence, so clearance is a human override recorded by a repository writer, not an agent action. Recommendation to the maintainer: clear it as a pre-existing, out-of-scope seam — the honest scope of this PR is the heartbeat producer, and converting the config/vault resolution seam to a versioned commit protocol is separate work that should stand on its own if it is wanted.

Kiro Crew could read a user's WakaTime stats but never sent anything back,
so agent coding time was invisible in their WakaTime dashboard. This adds
the send side: a completed agent turn that edited files or ran a command
becomes a WakaTime heartbeat.

A small in-process producer hooks at end-of-turn, where the usage
telemetry already fires. A turn counts as coding activity only when it
dispatched a filesystem-write or shell tool. Each qualifying turn enqueues
one heartbeat on a bounded buffer, and a time-gated flush drains it,
chunks to the API's 25-per-request cap, and sends. The send is
fire-and-forget: it never awaits on the turn's hot path, and a failure
drops the batch rather than retrying, since WakaTime dedupes by two-minute
buckets.

The heartbeat uses WakaTime's ai coding category and carries the turn's
real signals: input and output token counts from the turn's usage, a
total line-change count from the file-change snapshots the turn already keeps,
and the project directory basename as the entity. No session identifier
is sent. Editor
and OS are resolved by WakaTime from the request User-Agent, so the client
now sends a plugin-style User-Agent identifying Kiro Crew.

Sending is a separate opt-in from reading. A new wakatime.send_heartbeats
config flag defaults off, and the producer is gated on both it and
wakatime.enabled, so a user who has not opted in sends nothing and pays
nothing. The token and line counts come from the turn that just ran, so
they are genuine since-the-last-heartbeat deltas; a line count is only
sent when it can be computed. The project entity is the directory
basename, never an absolute path.

Part of #8094

Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io>

A tool awaiting permission retains only the turn's shared bounded key
(the 16-char `_tcid_identity_key` digest, never the raw backend id), and
admission is capped at `_MAX_TCID_SOURCES` with the overflow logged once,
so the pending-coding set stays fixed-width regardless of how many
distinct tool ids a turn produces. The permission-request drain uses the
same digest key, so admission and drain agree by construction.
@NicholasRBowers

Copy link
Copy Markdown
Contributor

self-added: yes
mechanism: pending-key admission now reuses the turn's shared _tcid_first_source/_tcid_collapsed tracking

  • span=db37e66c9e96 — independently capped IDs bypass ambiguity tracking (chat_runner.py:11684)

fixed in 8c7d5fd36dd4a34cf36a0da84e79e97c4a3560ff: the add site admits a key only when it is in _tcid_first_source AND not in _tcid_collapsed — the same present-and-not-collapsed pattern the existing reader uses; the independent len() < _MAX_TCID_SOURCES cap is gone (the first-source map already enforces that bound).
A collapsed id is withheld as ambiguous (fail closed: no heartbeat), so an unrelated approval under the shared digest can no longer consume a denied coding call's key.
A key omitted by the shared cap still tallies into _wt_pending_dropped for the one end-of-turn summary.
Mutation-verified regression: test_collapsed_coding_id_cannot_be_counted_by_unrelated_approval fails on the prior code, passes on 8c7d5fd36dd4a34cf36a0da84e79e97c4a3560ff.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

self-added: yes

  • span=db37e66c9e96 — restored turns are treated as interactive (chat_runner.py:14536)

fixed in 8c7d5fd36dd4a34cf36a0da84e79e97c4a3560ff: the emit gate now carries and not _turn_provenance_restored, the same guard the adjacent spending arm at chat_runner.py:11002 pairs with this actor test.
A queue row restored after a gateway restart loses its actor stamp by construction and falls back to "user"; the provenance flag is the surviving signal, so the replayed turn emits nothing.
Mutation-verified regression: test_restored_queued_turn_does_not_emit_heartbeat fails on the prior code, passes on 8c7d5fd36dd4a34cf36a0da84e79e97c4a3560ff.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=70edebd5f431 — a lone heartbeat credits near-zero duration for a single long agentic turn

accepted-and-deferred: real, and already tracked — issue #13023 (deferred-finding label, assignee NicholasRBowers, Due: 2026-10-14) owns duration crediting design (turn-start heartbeat vs duration field), decided before this PR as out of its scope.
This PR deliberately ships presence-correct heartbeats first; interactive chains already accumulate correctly, and the deferral names the single-long-turn case explicitly.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=2a7395f9a819 — fold _wt_collect's snapshot reads into _flush_file_changes' own resolution pass

rebutted as disproportional: the duplicate reads happen once per turn, off the event loop, bounded by the turn's snapshot set — a constant-factor saving.
Merging telemetry collection into the flush path would couple the heartbeat's per-turn read to a display-flush routine with different timing and error semantics, a restructure of shipped code for a micro-optimization; the retrospective ruling on this PR favors the smaller shape.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=e48125549479 — permission decision drains the key at decision START, so a same-call re-prompt loses it (chat_runner.py:12760)

rebutted as disproportional: the failure direction is a MISSED heartbeat (undercount), never a false one — the key is drained, _wt_deciding_coding recomputes False on the repeat frame, and nothing counts.
The design's bias is fail-closed telemetry (floor semantics throughout); guarding the rare same-call re-prompt frame would need re-admission state on an uncompleted decision, new plumbing for a telemetry miss in an edge flow.
The recorded tradeoff covers this class: ambiguity or interrupted decisions withhold, they never fabricate.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • span=2958b0fb8002 — difflib.SequenceMatcher(autojunk=False) sizing on snapshots (heartbeats.py:114)

rebutted: both inputs are hard-bounded — a truncated snapshot on either side is skipped before diffing, so each side is ≤ _MAX_SNAPSHOT chars — and the whole collection runs in one asyncio.to_thread off the event loop, once per turn.
The path count is the turn's own file-change snapshot set, which the runner already bounds; the worst case is bounded background CPU, not an event-loop stall or unbounded work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants