Skip to content

fix(channels): bound the observe history file by observe_max_entries - #6557

Open
leonlaiyc wants to merge 2 commits into
kirodotdev:mainfrom
leonlaiyc:fix/observe-history-file-cap
Open

leonlaiyc wants to merge 2 commits into
kirodotdev:mainfrom
leonlaiyc:fix/observe-history-file-cap

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An observed channel's history file stays bounded by observe_max_entries, so a busy channel can no longer fill the disk.

A channel with observe mode on writes every message to a per-channel history file (channel_history.py) that nothing ever shrinks.

Why it matters

A busy observed channel grows its history file without bound until the disk fills; the gateway then fails on unrelated writes.

Not a goal

  • Changing the file's format: it stays an append-per-message log, as on main; this PR bounds it rather than replacing it with a periodic rewrite-only file.
  • Owner-only permissions on Windows: there the file keeps the history directory's inherited ACL, as on main.
  • Recovering messages lost to a forced exit, for example by backfilling from the messaging platform.

What changed (motivation → approach → change)

An observed channel's history is now a pair of files, <channel>.jsonl (live) and <channel>.jsonl.1 (older), and once any inherited oversized file has been replaced, the pair never holds more than about twice observe_max_entries records. Once the live file holds the cap, the next append renames it over .1 and starts a fresh file. That is the same rotation jsonl_util.rotate_jsonl_at gives the repo's other append-only logs, counted in records instead of bytes.

Piece What it does
Count-triggered rotation The disk lane counts records in the live file and renames it aside at the cap; nothing is ever re-serialized, so every record on disk is exactly what an append wrote
Load Reads .1 then the live file into a deque bounded to the cap through jsonl_util.bounded_raw_records with a per-record byte cap derived from the field caps (an oversized line is skipped with one warning and still counts toward the bound), skips malformed, undecodable, ts-less and non-finite lines, drops expired entries from memory, never writes a record, and logs only the record counts it read, naming any generation it could not read
TTL on disk Each load (gateway start or observe re-enable) removes a whole generation with no record inside observe_ttl_hours, and moves a live file whose oldest record has expired aside when no .1 exists, so an expired record leaves the disk once its generation's newest record is past the TTL at a load, or a rotation replaces that generation; nothing is rewritten, and a generation still holding a fresh record is never replaced
Restart bound The load seeds the record count from the live file (a live file that fails to read is counted as full, so the next append rotates first when no .1 exists); whenever a rotation would replace history whose read failed (an unreadable .1, however full the live file is, or an unreadable live file beside the only readable .1), that channel's disk appends pause with one warning, while the in-memory window keeps every message; a paused channel re-reads its generations on a schedule (first after 60 s, backing off to an hour) and resumes as soon as every one reads, and a load also clears it; an inherited single file over the cap is renamed aside whole, never trimmed, and only into an empty .1 slot
Rotation failure When the live file is full and the rename fails, including when the history folder is missing, the disk append is skipped (the entry stays in memory) until a rename succeeds, so the file never passes the cap
Failed writes After a write or its close fails, the lane re-reads the live file through the same guarded open a load uses and takes its real line count and torn end, so a failed write makes rotation fire neither early nor late; if that read fails too, the channel's disk appends wait (entries stay in memory) and re-measure before any later write or rotation
Torn tail A last line left unterminated by a crash is skipped on load, and the next append starts on a fresh line
Bounded fields Every channel's entries, observe or not, in memory and on disk, cap text at 40,000 characters (Slack's own message limit, so a real message is unchanged) and ids at 256
Off-loop IO Every file operation runs on one single-worker lane (executors.py); appends are admission-bounded at 64 queued jobs
Path safety The history folder is trusted once and re-verified on every operation; opens and renames happen inside the pinned parent and never follow a link, a junction or a FIFO, and refuse a hard link, at either generation; such an entry at .1 is left in place, and pauses that channel's disk appends until it is removed

The rename replaces a rewrite because a rewrite rebuilds the file from an in-memory window that can be stale or partial, which is how a live limit change could delete records on disk. A rename moves records without recomputing them, so the window and the file need no reconciliation. Rotation needs no cross-process lock: one gateway runs per home (gateway_lock's flock), and inside it every history file operation runs on the single-worker lane. Turning observe off removes both files, each independently, so a failure on one never leaves the other; inside the data home each append on POSIX tightens the file to best-effort owner-only 0o600 and the folder is created 0700 (owner_only_files); a history folder configured elsewhere keeps the umask default, as main's #17629 decided for every configurable-location writer (on Windows the file keeps the history directory's inherited ACL). The Windows append opens through main's platform_compat.open_create_no_reparse, so platform_compat.py is unchanged. The sensitive-path screen matches the canonical root plus the lexical leaf (is_sensitive_resolved_path) and never resolves the leaf, so a planted UNC reparse point is not followed before the no-follow open refuses it. docs/system-specs/modules/channel-history.md and docs/system-specs/modules/slack-gateway.md describe the pair, and .github/black-baseline.txt tracks the formatting baseline.

Backwards compatibility

Compatible: existing history files load unchanged; one already over the cap is renamed to <channel>.jsonl.1 on first load and replaced at the channel's next rotation. A saturated lane (64 queued appends on a stalled disk) drops the disk write for that message while memory keeps it.

Tests

test/test_channel_history.py locks in: a busy channel rotates without a rewrite; the pair stays within twice the cap at every drain; a cap raised while a load is queued keeps every record within it; an inherited oversized file is rotated aside, not rewritten; a load never writes a record; fully expired generations are removed on load while a generation with one fresh record is kept whole, a slow channel's live file with an expired oldest record is moved aside, and a TTL raised during the read is honoured; a live file that fails to read still rotates before the next append, and pauses disk appends instead of replacing a readable .1; a paused channel resumes once a scheduled re-read finds every generation readable, and backs off while one stays unreadable; an unreadable .1 is never replaced, at load or by later appends, however full the live file is; an oversized line is skipped with bounded reads while a maximum-size record round-trips; the load log names an unreadable generation instead of counting it; a generation holding no parsable record is removed and logged for that reason; a history folder missing at rotation never resets the count; failed and partly landed writes never let the live file pass the cap; zero-byte write failures on a full disk never rotate a short live file over a full .1, and a live file that cannot be measured blocks rotation until it can; the record count survives a restart; a persistent rotation failure never grows the live file past the cap and appends resume once rotation works; torn tails never swallow the next append or make rotation fire early; field caps; link, hard link, FIFO and swapped-parent refusal at both generations, with a link or hard link at .1 left in place; observe off removes both files; and an append still lands when the owner-only tightening is refused. test/test_channel_history_containment_428.py keeps .1 beside the live file, proves the screen never hands the leaf to a resolver (fails on the old is_sensitive_path call) and still refuses a root inside ~/.ssh. test/test_owner_only_files.py's two history tests now drain the disk lane before they look at the files.

Manual verification

N/A -- unit coverage sufficient: every mechanism has a direct pytest exercising the real file system in a temp dir.

Screenshots / video

N/A -- backend-only change, no rendered surface.

Related Issues

no linked issue: the cap gap was found in review of observe mode, no tracked issue exists.

Pattern harvest

Not generalizable: the bounded-file cap pattern is specific to the observe history lane; no repo-wide rule candidate.

Checklist

  • Tests pass locally
  • Docs updated (docs/system-specs/modules/channel-history.md)

@leonlaiyc
leonlaiyc requested review from a team and chenmingwei23 August 28, 2026 15:22
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Aug 28, 2026
@github-actions

github-actions Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 3eb462169743f2f8d085daf5ee51253b3fc06559 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

About 1,300 lines of hardened file-store machinery are packed into the in-memory window class to fix what is roughly a 40-line count-and-rename bug.

Watch

  • Scope and proportion. The goal is "stays bounded by observe_max_entries". The diff also does four other things: it moves all IO onto a new lane (set_observe on the loop now returns before the load), it pins and identity-checks the history folder and refuses links, FIFOs and hard links, it adds a pause/backoff scheduler for unreadable generations, and it adds field caps for every channel. Most of the pause/retry and measurement states (_live_unknown, _disk_paused, _pause_retry, _rotate_failed) exist only because of that hardening, not because of the bound. Every later change to observe history now has to reason through all of those states. A smaller fix was available: count lines at append time and os.replace to .1 at the cap, on the existing open path.
    Clears when: the bound ships alone, with the off-loop lane and the path hardening split into their own PRs, or the description says why the bound cannot be correct without them.
  • Architectural fit. channel_history.py goes from 367 to about 1,530 lines, close to the 1,600-line split point. It now holds two responsibilities: the in-memory per-channel window and a crash-safe, rotated, path-pinned JSONL store. AGENTS.md says "A new responsibility gets a new module".
    Clears when: the disk generation store (pin, open, rotate, measure, unlink, pause) moves to a sibling module that ChannelHistory calls.
  • Security change mixed into a disk-bound fix. The path screen changes from is_sensitive_path to is_sensitive_resolved_path, which no longer resolves the leaf. That changes the sensitive-path gate's subject in a PR about file size.
    Clears when: it moves to its own PR, or a security-module owner signs off on it here.

Suggestions

  • Drop the 0-byte-write, unknown-count and paused-retry branches, and treat an unreadable .1 as "rotate anyway". The file is described as an ephemeral context cache, so losing an unreadable cache generation costs less than keeping this state machine.

[DESIGN-REVIEWED] 3eb4621

Description read

[DESCRIPTION-READ] d76e63ca89f56f28452eabeb66604f191a1ecf5dfe7dbc1fbf0cde5159faa597

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🔴 BLOCK

Premise-level review of 3eb462169743f2f8d085daf5ee51253b3fc06559 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: BLOCK

The fix needs only jsonl_util.rotate_jsonl_at before each append; ~1,400 lines rebuild it, plus an executor, a pause state machine and path hardening.

Not justified as shipped

  • Item 1 — duplicate of src/kiro_crew/jsonl_util.py:569 rotate_jsonl_at: it already does "Rotate path aside to <name>.1… total disk use stays bounded at about twice the cap." That is the Goal's harm ("can no longer fill the disk"), and a byte cap bounds disk use more directly than a record count.
    Subtraction: replace _rotate_generation, _live_records/_live_torn/_live_unknown and the measure paths with one rotate_jsonl_at(path, <bytes>) call in the existing append.
    Clears when: the bespoke rotation is replaced, or the PR shows a disk-fill case that rotate_jsonl_at leaves open.
  • Item 3 — oversized: the base appended inline and nothing reports a loop stall from it.
    Subtraction: delete channel_history_executor, _submit_append, the async _load_observe/_apply_observe_load and _observe_generation.
    Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 4 — oversized: the diff calls this file "only a cache", yet adds a pause with 60 s → 1 h backoff to protect an unreadable .1.
    Subtraction: delete _disk_paused, _pause_retry, _PauseRetry and _resume_paused_appends.
    Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 5 — oversized: dir pinning, identity capture and st_nlink refusal are hardening the Goal never asked for.
    Subtraction: keep the base's _observe_path checks.
    Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 7 — rides along: field caps now apply to non-observe, memory-only channels too.

What this change ships

Inventory (9 items) — 3 justified

Intent: keep an observed channel's history file from growing until the disk fills. FIX.

  1. History becomes a live + .1 pair, rotated at the record cap — duplicate of jsonl_util.rotate_jsonl_at
  2. A load removes or rotates whole expired generations instead of rewriting — justified
  3. All history disk IO moves to a new single-worker lane; appends drop at 64 queued, loads become asynchronous — oversized
  4. Disk appends pause, with a backoff retry, when a generation can't be read — oversized
  5. Pinned-parent, no-follow, single-link opens; the sensitive-path screen changes — oversized
  6. Load skips lines that are malformed, oversized, ts-less or have non-finite timestamps — rides along
  7. Text capped at 40,000 chars and ids at 256 on every channel — rides along
  8. A live limit change resizes the observe buffers immediately — rides along
  9. Observe-off removes both files on the lane — justified

Blockers

Second spelling of the existing rotation helper
The description says it is "the same rotation jsonl_util.rotate_jsonl_at gives the repo's other append-only logs, counted in records instead of bytes." The reasons it gives for not reusing the helper are overshoot and parent-swap safety. Overshoot needs concurrent writers, and the diff says this file has exactly one. Parent-swap safety is item 5, which is outside the Goal.
Subtraction: call rotate_jsonl_at from the existing append.
Clears when: the bespoke rotation is replaced by rotate_jsonl_at, or the PR names a disk-fill case that rotate_jsonl_at leaves open.

[FIRST-PRINCIPLES-REVIEWED] 3eb4621

Description read

[DESCRIPTION-READ] 6f675b04927fd825ffc3870136538f803c840331f7e03d7eb4894b80bb35edd0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 3eb462169743f2f8d085daf5ee51253b3fc06559 via the fork AI-review pipeline; updated in place on each push.

Adjudication upheld 1 of 1 blocking finding(s); those must be resolved before merging.

BLOCKING -- test/test_channel_history.py:2633 -- Snapshot check fails on scheduler delays

assert elapsed < 1.0, f"the root snapshot waited {elapsed:.1f}s on the preparer"
A scheduler pause exceeding one second between the clock reads -> the snapshot test includes that pause in elapsed -> correct non-blocking code fails the assertion.
Anchor: tests-are-deterministic (D7)
Fix: Hold _history_root_lock directly while asserting the snapshot is None; remove the preparer thread and elapsed-time assertion.

[BLOCK-MERGE] 3eb4621
[GPT-REVIEWED] 3eb4621

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1: I'm upholding it. The new test (diff line 4193, file test/test_channel_history.py:2633) asserts elapsed < 1.0, an upper bound on measured wall-clock time. That is exactly item 5 (D7) of the tests-are-deterministic rule in .review-base-rules/AUTOSDE.yaml:367, which carries blocking: true (line 368) and lists elapsed < 0.5 as its example at line 413. None of the rule's exceptions at lines 415–419 cover it: it isn't a lower bound, isn't relative to a span the test measured itself, and isn't an existing complexity guard.
The rule's flag decides the verdict without weighing. The fix is cheap anyway: the snapshot is None assertion already proves the snapshot didn't wait, so the timing line can simply be removed.

[ADJUDICATION] 3eb462169743f2f8d085daf5ee51253b3fc06559 total=1 uphold=1 downgrade=0
UPHOLD F1 test/test_channel_history.py:2633 reason=autosde-blocking-rule
[GPT-ADJUDICATED] 3eb462169743f2f8d085daf5ee51253b3fc06559

@github-actions

github-actions Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 3eb462169743f2f8d085daf5ee51253b3fc06559 via the fork AI-review pipeline; updated in place on each push.

Review details

Advisory only. Nothing blocks the merge, but CI's line-cap lint will fail on the test file.

FINDING — test/test_channel_history.py:1 — this PR grows the file from 366 to 3,622 lines, and .github/file-line-cap-baseline.txt has no row for it, so scripts/check_file_line_cap.py fails the Backend Lint job for this touched file (rule keep-modules-small) → Fix: move the new rotation, pause, persistence and containment tests into new sibling modules (for example test_channel_history_rotation.py and test_channel_history_pause.py) so each file stays under 1,600 lines.

[OPUS-REVIEWED] 3eb4621

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 28, 2026
@bolichen97
bolichen97 enabled auto-merge August 29, 2026 23:59
@bolichen97
bolichen97 disabled auto-merge September 3, 2026 21:25
@bolichen97
bolichen97 enabled auto-merge (squash) September 3, 2026 21:25
@chenmingwei23 chenmingwei23 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 4, 2026
@NicholasRBowers NicholasRBowers added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 4, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: All CI shards and 3 of 4 AI review lanes are green; the sole blocker is one GPT blocking finding at channel_history.py:264 — the new size-triggered compaction runs synchronously (atomic_write) on the async gateway path, which can stall the event loop on a large observe history. Plan: offload the compaction write off the event loop (matching the codebase's existing pattern for async callers of synchronous writes), then rebase and drive to green.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@NicholasRBowers
NicholasRBowers force-pushed the fix/observe-history-file-cap branch from 3182b69 to d862511 Compare September 6, 2026 23:50
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

  • Disposition: fixed — span=c27922ab5eb3 — BLOCKING src/kiro_crew/channel_history.py:264: synchronous compaction blocks the gateway event loop

self-added: no
mechanism: single-worker disk lane (channel_history_executor) through which every observe-file mutation runs in submission order

Fixed in d862511. _compact now serializes the deque on the calling thread (its only mutator) and hands only the finished bytes to a dedicated single-worker executor lane — channel_history_executor() in executors.py, the repo's established named-pool pattern. The blocking atomic_write never runs on the event-loop thread.
The lane also orders every observe-file mutation (per-message append, compaction rewrite, unlink), so submission order is execution order: a compaction snapshot cannot erase an append submitted after it, and a queued write cannot resurrect a file unset_observe removed. This ruling covers the class: any finding requiring an observe-history disk write to run inline on the loop thread, or requiring ordering between the append log and the compaction rewrite, is answered by the lane.
Four red-before-green tests in test/test_channel_history.py (TestObserveCompactionOffLoop) pin both properties: append- and load-triggered compaction write off the loop thread; an append submitted after the compaction snapshot survives in the file; unset_observe's unlink wins over queued writes.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

Drive-to-green round 1 pushed: d862511 (rebase onto current main + fix for the GPT blocking finding).

What changed on top of Leon's original diff, and why:

  1. Rebased onto current main — the branch was cut 2026-08-28; the fork GPT lane needs the base to carry .github/review-prompts/gpt-block-adjudication.md, and a fresh head is what re-fires the fork workflow_run review lanes. The original diff carried over clean (patch-id identical through the rebase).
  2. Moved compaction's atomic_write off the event-loop thread (the GPT BLOCKING finding at channel_history.py:264): _compact serializes the deque on the calling thread and hands only the finished bytes to a new single-worker channel_history_executor() (executors.py, following its existing named-pool pattern — pool global, lazy double-checked creation, __all__, shutdown wiring).
  3. Routed every observe-file mutation through that one-worker lane (ChannelHistory._run_on_disk_lane): per-message append, compaction rewrite, and unlink. One worker makes submission order execution order, which closes the follow-on data-loss race a fire-and-forget offload would open (an append landing between a compaction's snapshot and its rename would be erased by the older snapshot) and keeps unset_observe's unlink from being overwritten by a queued write. Off the loop (tests, sync startup) jobs run inline, so existing read-after-write tests stay deterministic.
  4. Four red-before-green tests (TestObserveCompactionOffLoop in test/test_channel_history.py): both compaction triggers write off the loop thread (fail pre-fix with "compaction ran its atomic_write on the event-loop thread"), an append submitted after the compaction snapshot survives in the file, and unset_observe's unlink wins over queued writes.
  5. One accepted residual, stated in the code: a hard crash can lose mutations still queued on the lane — the same window any buffered writer has; the queue is normally empty (a write is microseconds, messages arrive seconds apart), and the file is an ephemeral TTL/cap-bounded context cache. The alternative (inline writes on the loop) is the stall the finding blocked.
  6. PR body reconciled: event-loop-safety paragraph, round-1 tests/gates, corrected the now-stale "a failed append does not count" line (the counter now counts submissions; a failed append at worst compacts one message early), removed the CLA placeholder section.

Leon's authorship and commit are preserved (fix amended into the single commit, Co-authored-by: Kiro Crew trailer added). Local gates: isort/flake8/mypy clean, full suite at pre-existing-failure baseline parity, 66/66 in the two touched test files.

@NicholasRBowers
NicholasRBowers force-pushed the fix/observe-history-file-cap branch from d862511 to 3472909 Compare September 7, 2026 00:39
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

  • Disposition: fixed — span=abe2e87d8478 — FINDING src/kiro_crew/channel_history.py:258: function-local imports violate top-level-imports

self-added: yes
mechanism: none added this round (imports hoisted, test patch point moved to the use-site binding)

Fixed in 3472909. on_event_loop, channel_history_executor, and atomic_write are now module-scope imports in channel_history.py; no import cycle exists (executors.py imports stdlib only, atomic_write.py imports only platform_compat). The pre-existing function-local from kiro_crew.hooks import is_sensitive_path in _observe_path predates this PR and is untouched.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

  • Disposition: rebutted (not a defect) — span=c27922ab5eb3 — BLOCKING src/kiro_crew/channel_history.py:263: queued history is lost on shutdown

self-added: yes
mechanism: none added this round (ruling on the round-1 disk lane)

Not a defect for this file's contract. The observe history file is an ephemeral, TTL- and cap-bounded context cache (channel_history.py:1-9) backed by the messaging platform's own durable history: on restart _load_observe rebuilds from the file and the rolling window self-repopulates from the platform. The loss window is a hard os._exit landing inside the microsecond-wide submit-to-execute gap of a normally-empty single-worker queue — at most the few writes queued at that instant, never a corrupt or partial file (every write is a complete atomic_write; every append is a whole line).
The demanded remedy — draining the history lane before every gateway hard exit — contradicts the deliberate wait=False fast-exit contract those os._exit sites exist to guarantee (gateway.py:11133, :11365) and would add a cross-module ordering guard on multiple rare exit paths to protect cache data the platform already holds durably. The lane's wait=False, cancel_futures=True shutdown mirrors every sibling pool in executors.py. This ruling covers the class: findings that require queued observe-history cache writes to survive a hard process exit are answered here.
This lane's own Opus adjudicator reached the same conclusion on this head and FLAGGED F1 as a machine-flagged likely edge case (harm rung LOW, "a human would plausibly accept this residual"); the security fence keeps it blocking pending a human override.

For the maintainer, after independent verification: /ai-review override gpt 347290930175a2a55008c67e10bd48706aa26bcd: Accepted residual — hard-exit loss of queued writes to the ephemeral observe context cache is bounded to a microsecond window, self-heals from the platform's durable history, and draining the lane would break the deliberate wait=False fast-exit contract.

@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

Round 2 pushed: 3472909 (rebase onto current main + the GPT advisory fix).

  • Hoisted the round-1 function-local imports (on_event_loop, channel_history_executor, atomic_write) to module scope in channel_history.py per the top-level-imports finding; no import cycle (verified: executors.py imports stdlib only, atomic_write.py only platform_compat). Test patch point moved to the use-site binding accordingly.
  • The security-fenced blocking finding F1 (queued cache writes lost on a hard os._exit) is rebutted as not-a-defect in its disposition comment — this lane's own Opus adjudicator flagged it as a likely edge case (harm LOW). Per the fence's rules it stays blocking until a repository writer records an /ai-review override; a pre-drafted override line is in the disposition for independent verification.
  • Gates on this head: isort/flake8/mypy clean, 120/120 across every ChannelHistory-touching test module.

Leon's authorship and the single-commit shape are preserved.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@NicholasRBowers
NicholasRBowers force-pushed the fix/observe-history-file-cap branch from 3472909 to 357c1ba Compare September 7, 2026 01:18
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]

  • Disposition: fixed — span=abe2e87d8478 — FINDING src/kiro_crew/channel_history.py:386: docstring says "fsync" but atomic_write defaults fsync=False

self-added: yes
mechanism: none added this round (one docstring word)

Fixed in 357c1ba. The _compact docstring now reads "temp file, write, rename" — verified atomic_write's signature defaults fsync: bool = False (atomic_write.py:820).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=c893e283e42a an unreadable live file is rotated over a readable .1

Fixed in 116cd81. When the load finds the live file unreadable and .1 readable, it no
longer seeds the count at the cap. It pauses that channel's disk appends until the channel's next load (a lane-owned
_disk_paused set checked first in the append job), logs one warning naming both files, and renames or unlinks nothing,
so the only loadable generation stays on disk. An unreadable live file beside an absent or unreadable .1 keeps the
seed-at-cap path, since nothing readable can be replaced there. The docstring and the spec state both cases. Regression:
test_an_unreadable_live_file_is_never_rotated_over_a_readable_older_generation (fails on d9062ef, passes now).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=cd2677aa4d47 a generation with no parsable record is logged as expired

Fixed in 116cd81. The load splits the two grounds: a generation whose every record is past
the TTL logs "every record is past the TTL", and one holding no parsable record logs "holds no parsable record".
_unlink_generation takes the reason from its caller. Which files are removed is unchanged. Regression:
test_a_generation_holding_no_parsable_record_is_removed_for_that_reason (fails on d9062ef, passes now).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • rebutted — span=abe2e87d8478 a TTL increase during a load still filters the window with the pre-read cutoff

Same finding and same code as on e917651; the reasoning in #6557 (comment)
still holds. The in-memory window is built once per load against the TTL in force when the read runs, and
set_observe_limits (channel_history.py:199) never reloads from disk, so a TTL raised mid-read behaves exactly
like one raised a second after the load: the records it now covers appear at the next load. Every disk decision
uses the cutoff taken again after both reads (:942), so none of those records is unlinked or rotated away in the
meantime. Main filters the window the same way and then compacts, which deletes them outright.

@NicholasRBowers

NicholasRBowers commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Rebased onto main 7610731. No code changed: the PR's delta on channel_history.py, platform_compat.py, the tests and the specs is byte-identical to 116cd81, which GPT, Opus, Design and First Principles all cleared.

Checks on b2d815a: 951 scoped (including the ratchet) and 9,165 related tests pass; black, isort, flake8, mypy, comment-history, sync-IO, loop-bound-locks and docs-lint all pass.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=c27922ab5eb3 Unreadable older history is overwritten

Fixed in b0d6bc8. When a load finds .1 present but unreadable, it pauses that channel's disk appends until the next load, whatever the live file holds, with one warning naming both files; it renames and unlinks nothing. A load rotates only into an empty .1 slot, so neither the load path (an oversized or aged live file) nor the append path (a live file filling to the cap) can replace the generation whose read failed. A link, FIFO or directory planted at .1 falls in the same case and is left in place: the load cannot tell it from unreadable history without following it. Regressions: test_oversized_live_file_is_not_rotated_over_an_unreadable_older_generation and test_appends_never_rotate_a_filling_live_file_over_an_unreadable_older_generation (both fail on b2d815a).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=af56eab32990 Shared test isolation is removed mid-test

Fixed in b0d6bc8. The test no longer calls monkeypatch.undo(); its patch is scoped with monkeypatch.context(), so it is lifted at the end of that block and every fixture-level guard stays in force for the rest of the test. That was the only undo() in the file.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=f0ac3bb037eb spec overstates the pair bound and the retention window

Fixed in b0d6bc8, all three clauses in channel-history.md: the pair bound of about twice observe_max_entries is now stated as holding once any inherited oversized generation has been replaced (line 79, and the same qualifier where a lowered cap is described); loads are described as the only source of TTL-driven removals, since an append rotation can also replace an expired .1 (line 98); and retention now reads that an expired record leaves disk once its generation's newest record is past the TTL at a load, or a rotation replaces that generation, instead of within the TTL plus two loads (line 118). The PR description uses the same wording.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=cd2677aa4d47 the load log reports a fabricated record count for an unreadable generation

Fixed in b0d6bc8. The cap seed for an unreadable live file stays the lane's rotation seed only. The load log now sums the line counts of the generations it actually read and names each generation that exists but could not be read, e.g. "(0 records counted;

/C1.jsonl could not be read, and the records of an unreadable generation are not counted)". It no longer adds the cap, or a zero for an unreadable .1, to the total. Rotation and pause behaviour are unchanged. Regressions: test_load_log_names_an_unreadable_live_file_instead_of_claiming_a_full_generation, test_load_log_counts_only_the_readable_older_generation_beside_an_unreadable_live_file and test_load_log_names_an_unreadable_older_generation_instead_of_counting_it_as_empty (all fail on b2d815a).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=c27922ab5eb3 Failed writes can discard a full readable generation

Fixed in bec4ebc. A failed write no longer adds its payload to the live file's record count. After the write or the close-time flush fails, the disk lane re-reads the live file through the same pinned-parent, no-follow open a load uses and takes its real line count and torn end, so zero bytes landing on a full disk leaves the count where the file is and cannot walk it to the cap. If that re-read fails too, the count is unknown: later appends re-measure first, keep their entries in memory and skip the disk write while the file still cannot be measured, and never rotate on an assumed count; a load also reseeds it. A completed write still advances the count by its payload. Regressions: test_zero_byte_write_failures_do_not_replace_a_full_older_generation and test_unknown_live_count_blocks_rotation_until_remeasurement_succeeds (both fail on b0d6bc8).

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • rebutted — span=415f8e4ac87b a stateless size check would remove the disk-fill harm with less state

Kept. observe_max_entries is the retention contract, not only a disk guard: after a restart the load serves the channel's last observe_max_entries messages, and the pair never holds fewer than the newest full generation (channel-history.md). A byte trigger cannot keep that promise, because one record runs from about 100 bytes to about 160 KB (40,000 UTF-8 characters). A byte cap loose enough to always hold cap records is cap × 160 KB (32 MB per generation at the default cap of 200), which bounds nothing useful; any tighter byte cap drops the restart window below observe_max_entries for a channel with long messages.

Most of the state named does not come from counting. _live_torn exists because a crash can leave a partial last line that any appender must step over. _disk_paused exists because any rotation renames over .1, and an unreadable .1 must not be replaced. _rotate_failed exists because any rotation can fail, and the file must not grow past its bound meanwhile. Only _live_records and _live_unknown are count-specific, and each has a direct test. jsonl_util.rotate_jsonl_at is byte-triggered, so it has the same window gap; that reason is recorded in 5862439758.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • rebutted — span=22910de24103 the off-loop disk lane rides along on the bound fix

Kept: the lane is what makes the bound safe to run where it runs, and it was added on this PR as the fix for a blocking finding. channel_history.push runs inside async def _route_message (slack/events.py:2447 on main), and set_observe, which runs the load, inside the gateway's async startup path (slack/gateway.py:3669). The bound adds disk work to both: a load reads and parses up to two generations of cap records and may unlink or rename a generation; an append renames at the cap and, after a failed write, re-reads the live file. Done inline, that blocks the gateway's event loop on disk IO, which is the round-1 GPT BLOCKING finding answered in 5563152489 (2026-09-06). The single-worker lane follows executors.py's existing named-pool pattern.

The 64-slot admission bound keeps a stalled disk from queuing unbounded jobs in memory. A dropped disk write is logged and leaves the message in the in-memory window, which is what the live channel reads.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=c27922ab5eb3 Torn-tail append rotates retained history early

Fixed in ee66994. An append now counts as exactly one line. Every path that marks the live file torn has already counted its unterminated last line (the load's scan, the re-measure after a failed write, and the cap seed for an unreadable live file), so the leading newline only terminates a counted line and the record adds one. The newline is still written, so the torn tail stays skipped on load. Regression: test_torn_tail_at_cap_minus_one_rotates_after_one_complete_append (fails on bec4ebc with "torn live tail rotated one record early"). The same push makes the post-failure re-measure read in fixed 64 KiB chunks instead of whole lines, which the tree's unbounded-read ratchet (test_jsonl_util.py::TestNoUnboundedHandleIteration) flagged on bec4ebc.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=cd2677aa4d47 a hard link at the history leaf is not refused; a torn tail is double-counted

Both advisory findings from the Opus verdict on bec4ebc (not published because of a GitHub API rate limit; read from the run's withheld-verdict artifact) are fixed in ee66994.

Hard link: the append and the guarded read now judge the opened descriptor as a regular file with exactly one link (st_nlink != 1 refused), the shape of platform_log_append._regular_single_link. A hard link at the live leaf is refused before any byte is written and before the owner-only chmod; one at .1 is treated as unreadable, so the load reads nothing through it and pauses appends until the next load. Renames and unlinks act on the directory entry only, so the outside file is never touched. Regressions: test_append_refuses_a_hard_link_at_the_live_leaf_and_leaves_its_target_alone and test_hard_link_at_the_older_generation_is_refused_and_left_in_place (both fail on bec4ebc).

Torn double count: the same fix as GPT's span c27922ab5eb3 in this push; an append counts as one line.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=c27922ab5eb3 unbounded JSONL line can exhaust gateway memory

Fixed in 3f2c871. The load no longer iterates the handle line by line. It takes the line count and torn end from the same fixed-chunk counter the post-failure re-measure uses, and reads records through jsonl_util.bounded_raw_records with a per-record cap derived from this module's field caps (6 × (text + 3 ids) + 1 KiB = 245,632 bytes, so a legitimate record always fits). An over-cap line is skipped whole with one WARNING naming the file, still counts toward the disk bound, and never occupies more than a bounded read. Regressions: test_load_skips_oversized_record_with_bounded_reads (fails on ee66994: "history load materialized a 1965057-byte read; record cap is 245632") and test_maximum_sized_persisted_record_round_trips.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=4c4af006c8eb the private rotation drops jsonl_util's cross-process lock

Fixed in 3f2c871 by recording the exemption, per the item's own clear condition. Verified first that the gateway is the history files' only writer: kirocrew gateway holds GatewayLock (cli.py:3314-3327, gateway_lock.py's exclusive flock on <home>/gateway.lock), so one gateway runs per home; the only ChannelHistory is built in the gateway (slack/gateway.py:3657-3661); every append, rotation, read and unlink runs on channel_history_executor() (max_workers=1); the only out-of-process opener, kirocrew security audit's scan_history, only reads. channel-history.md now has a "Why no cross-process rotation lock" paragraph saying so, including that a future out-of-process writer must take the same lock, and _rotate_generation's docstring points to it.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • fixed — span=172ee235bde0 a paused channel stays paused until the next load

Fixed in 3f2c871. _rotate_failed and _live_unknown already retry on every later append; the one state that held until the next load was _disk_paused. A paused channel now re-reads its generations on a schedule: the first append after 60 s re-measures both generations through the guarded read (no rename, no unlink), backing off by doubling to one hour. As soon as every existing generation reads, the pause lifts, the count and torn state are reseeded as a load would, one INFO line says disk appends resumed, and the append continues. While anything stays unreadable nothing is written or rotated. The load warnings state the schedule. Regressions: test_a_paused_channel_resumes_disk_appends_once_a_scheduled_re_read_succeeds (fails on ee66994: the pause never lifts) and test_a_paused_channel_backs_off_while_a_generation_stays_unreadable.

@NicholasRBowers

Copy link
Copy Markdown
Contributor
  • rebutted — span=7fd749b3af0a a byte cap would delete most of the per-channel state

Kept, for the reason in 5883051824: observe_max_entries is the restart-window contract, and a byte cap cannot keep it. A record runs from about 100 bytes to the new per-record cap of 245,632 bytes, so a byte cap loose enough to always hold cap records bounds nothing useful, and any tighter one drops the restart window below observe_max_entries for a channel with long messages. The state is also mostly trigger-independent: _live_torn (step over a crash's partial line), _disk_paused (never rename over an unreadable .1) and _rotate_failed (never grow past the bound while a rename fails) would all exist under a byte trigger; only _live_records and _live_unknown come from counting.

Copy link
Copy Markdown
Contributor

PR triage (automated, 2026-10-05): REBASE needed. No duplicate found.

  • Relationship: DISTINCT. No other open PR touches src/kiro_crew/channel_history.py or test/test_channel_history*.py. fix(knowledge): store bare source-language canonical names with English aliases #12551 (open) also edits channel-history.md and executors.py, but it only changes spec prose and unrelated pools. It does not cap the observe file. No merged PR since the merge base b5ca357ba8 implements the cap: on origin/main af1eb612da, _append_to_disk still appends with no limit.
  • Conflicts with origin/main (git merge-tree):
  • Breaking change: none found. Existing history files still load, and an oversized one is renamed to .1.
  • Outstanding blockers, already raised (not repeated here):
    • The maintainer's CHANGES_REQUESTED review on 6bbb0f151.
    • The GPT BLOCK-MERGE on channel_history.py:511: is_sensitive_path(str(path)) resolves the lexical leaf on Windows. That behavior was already on main, where _observe_path calls .resolve(), so this PR does not introduce it, but the block is still open.

@iamwhatever

Copy link
Copy Markdown
Contributor
  • fixed — channel_history.py:511 sensitive-path screen resolves a planted Windows reparse point

Fixed in 3eb4621. _observe_path now screens with security.paths.is_sensitive_resolved_path(str(path)): path is the canonical root captured at first trust (root.resolve() in _prepare_history_root) joined to one lexical leaf (channel_id holds no separator), matched lexically with no mc-pathres resolution, so the leaf is never realpathed. A link at the leaf is then refused by the no-follow open (O_NOFOLLOW openat on POSIX, platform_compat.open_create_no_reparse on Windows).
Every _observe_path caller in src passes history_root and runs on the mc-chan-hist lane, off the event loop, which is the caller shape that entry point is for.
Regressions: test_observe_path_screens_without_resolving_the_leaf (fails on the old is_sensitive_path call: "the leaf was handed to the bounded resolver") and test_observe_path_still_refuses_a_sensitive_root (a root under ~/.ssh is still refused).
self-added: no
mechanism: lexical leaf screen under a canonical root

@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased on main fb56ea27b as 3eb4621 (one commit). Status of @bolichen97's CHANGES_REQUESTED review on 6bbb0f151, point by point:

  • Owning spec contradicts the code — fixed earlier on this PR: docs/system-specs/modules/channel-history.md documents the live/.1 pair, the rotation, load, TTL and pause rules, and slack-gateway.md's bounded-executor bullet names the mc-chan-hist lane.
  • Process-global drain gate tripped by other suites — gone: the rewrite-based design and its _channel_history_draining gate were replaced by count-triggered rotation; no drain gate exists in executors.py or channel_history.py.
  • atomic_write gated on a hand-rolled probe — gone with the rewrite: nothing in the lane calls atomic_write; generations move by os.rename inside the pinned parent.
  • Stale black-baseline note — the body no longer claims it; the diff drops channel_history.py from .github/black-baseline.txt and the file is black-clean.
  • Nits — _run_on_disk_lane no longer referenced; open_append_no_reparse and its copied CreateFileW flags are removed: the Windows append now opens through main's platform_compat.open_create_no_reparse, so this PR no longer touches platform_compat.py or test_platform_compat.py.

Also in this push, to fit main as it is now:

Local: scripts/local-gate.py --base origin/main backend related set 4610 passed, 18 skipped; test_channel_history*.py + test_owner_only_files.py 232 passed, 3 repeat runs; black/isort/flake8/mypy clean on the changed files. Re-review requested on the new head.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 3eb462169743f2f8d085daf5ee51253b3fc06559 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@iamwhatever

Copy link
Copy Markdown
Contributor
  • needs-a-decision — span=blocker "second spelling of the existing rotation helper" and items 3/4/5 (lane, pause state machine, path hardening)

The First Principles BLOCK and the Design CONCERNS (scope, architectural fit) both ask to shrink this PR from ~1,300 lines of store machinery to a count-and-rename (or jsonl_util.rotate_jsonl_at) in main's existing inline append. That discards most of the author's design, including the off-loop lane an earlier GPT round demanded and the record-count cap the author defended against a byte cap.
@bolichen97, which shape should land? (a) Shrink: count records at append, os.replace to .1 at observe_max_entries, load reads .1 then live, observe-off removes both, about 60 lines plus tests; the lane, pause/backoff, pinning and field caps are dropped or go to follow-up PRs. (b) Keep the current design: a maintainer amends the Goal or overrides the First Principles lane, and the test file is then split under the 2,000-line cap.
Either way the GPT 6.1 D7 finding (elapsed < 1.0 at test_channel_history.py:2633) is a one-line removal and will be fixed in the next push.

bolichen97
bolichen97 previously approved these changes Oct 25, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[operator: bolichen97] My earlier change request is addressed on 3eb4621. Approving to clear it; other open gates on this PR still apply.

Nick Bowers and others added 2 commits November 1, 2026 10:55
…entries

Rewrites the observed channel's on-disk history to the newest observe_max_entries whenever appends double the cap, publishing only complete views; suppression is OSError-only; cap/TTL are read apply-time on the loop thread; load tightens legacy file modes and marks any non-clean disk line for republish.
…unded-read probe

kirodotdev#26774 made jsonl_util frame records from fixed 1 MiB reads, so one
read in the history load can now be _FRAME_CHUNK bytes even though the
record cap is ~240 KB. The probe still asserted every read stayed under
the record cap. Bound it by the larger of the two, and size the oversized
line at four times that bound, so materialising the whole line still
fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (c1f8bdce61). The new head is 2e52544456.

  • Conflict: src/kiro_crew/channel_history.py, against fix(history): skip JSON that is not an object in two history readers #26044 (skip non-object JSON lines). This PR's loader already rejects a non-dict row, with its own warning text, so I kept this PR's side in all three hunks. The other main changes to the file merged cleanly.
  • New commit: test-only. After perf: keep the transcript page index valid across saves and restarts #26774, jsonl_util frames records from fixed 1 MiB reads (_FRAME_CHUNK). That broke test_load_skips_oversized_record_with_bounded_reads, which asserted no single read exceeds the ~240 KB record cap. The bound is now max(record_cap + 2, _FRAME_CHUNK), and the oversized line is four times that, so reading the whole line still fails. The source is unchanged from the approved head.
  • Local checks (Windows, Python 3.13): test_channel_history.py, test_history_non_object_json.py and test_owner_only_files.py give 124 passed / 113 skipped. The 30 other test files that import channel_history give 1705 passed. black is clean.
  • Not attributable to this rebase: test_channel_history_containment_428.py::test_observe_path_still_refuses_a_sensitive_root also fails on this Windows host at the old head 3eb4621697.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

drive-to-green PR claimed by drive-to-green pipeline fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants