Skip to content

feat(doctor): add a Temp directories row (tmpfs/no swap/low headroom) - #22672

Merged
bolichen97 merged 1 commit into
mainfrom
feat/doctor-temp-dirs-17126
Oct 30, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/doctor-temp-dirs-17126

Conversation

@dodgeblaster

@dodgeblaster dodgeblaster commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: kirocrew doctor tells the operator what kind of filesystem their gateway's temp directory is, and warns before it fails work silently.

The gateway, crons and CLI commands all write scratch to one temp directory (TMPDIR). The existing "Runtime tmpfs" row warns when that directory runs low on space or inodes, but nothing shows the operator what it is: whether it is RAM-backed, whether swap can absorb a spike, or whether a tmpfiles.d rule deletes files there by age.

Why it matters

The failure modes are silent and hard to diagnose. A RAM-backed temp dir with no swap throttles under memory pressure with oom_kill at 0 (#6788); an inode-exhausted temp dir returns ENOSPC while 90% of its bytes are free; and systemd-tmpfiles can delete a run's work after 10 days. One doctor line turns each of these into one obvious message with a named workaround.

Not a goal

  • It does not create, clean, chmod or otherwise touch the temp dir — it is a read-only diagnostic.
  • It does not fail the doctor's exit code: the temp dir's filesystem and ageing policy are host configuration the operator owns, so the row is advisory.
  • It does not change the existing "Runtime tmpfs" row or tempfile resolution.

What changed (motivation → approach → change)

Operators had no line naming what filesystem backs the temp directory their tools write scratch to, so a tmpfs-with-no-swap stall or a systemd-tmpfiles age-sweep that deleted work looked like an unexplained failure.

The approach adds only what the doctor does not already show, and reuses existing code for the rest. The directory is tempfile.gettempdir() — with no TMPDIR set, the system default (/tmp), which a gateway installed without a TMPDIR override also uses. The filesystem type comes from the repo's existing mount-table resolver, taskq/store.py's _linux_mount_type (no second parser is added); swap comes from the existing _swap_total_kib(). One best-effort probe is added in doctor_checks/resources.py: _tmpfiles_ages() asks systemd-tmpfiles --cat-config (binary resolved through the trusted-bin pin, 5s cap) whether a rule ages the dir, returning unknown rather than guessing on an x/X exclusion that covers the dir or a glob that could reach it.

The new _doctor_temp_directories row in doctor_checks/resources.py prints the path, filesystem and ageing, and warns — without touching the exit code — only when the dir is tmpfs with no swap: a RAM-backed temp dir with no release valve throttles under memory pressure. It deliberately does NOT re-measure space or inodes (the existing "Runtime tmpfs" row already warns on those for this exact directory, since it measures /tmp and the temp roots the process resolves, tempfile.gettempdir() included) and does NOT re-print the swap total (the Memory Pressure section already shows it). It names the directory it actually resolved rather than claiming to read the gateway service's TMPDIR: this probe reads the CLI process's own environment and mount namespace, so a service given a divergent TMPDIR is not visible to it. _linux_mount_type's longest-mount tie-break changes from > to >= so that, among stacked mounts at the same point, the visible topmost one wins (a tmpfs over an ext4 /tmp reads as tmpfs) — a one-character correctness fix that also benefits its existing taskq caller. Every probe degrades to an informational line on failure, and the whole row reports the path plus "not applicable" on macOS and Windows. cli_doctor.py wires the row into the report orchestration and forwards the helper through its facade.

Backwards compatibility

Compatible: the change adds a new read-only doctor row and one probe helper, and makes _linux_mount_type's mount-point tie-break prefer the topmost of equal-depth stacked mounts (> to >=) — a correctness fix for an over-mounted path that changes no result where mounts are not stacked. No existing row, threshold or tempfile resolution changes.

Removes nothing: the row is purely additive and no path, record, key or capability is removed or tightened.

Tests

New tests live in test/test_cli_doctor.py; the doctor facade ratchet updated is test/test_cli_doctor_refactor_facade.py and the report ratchet test/test_cli_doctor_refactor_report.py; the spawn-audit allowlist in test/spawn_audit_helpers.py gains the new probe. The reused resolver taskq/store.py::_linux_mount_type keeps its existing coverage in test/test_taskq_store.py.

  • TestTempDirectories — the row passes on a healthy disk-backed dir; warns (and never adds an issue) only on tmpfs-with-no-swap; stays quiet on tmpfs-with-swap and on a disk-backed dir regardless of swap; never prints space/inodes (the Runtime tmpfs row's job) and never re-prints swap; reports ageing, unreadable-mount-table and unknown-tmpfiles informationally; prints path + "not applicable" and runs no Linux probe off Linux; and degrades cleanly when gettempdir() raises.
  • TestTmpfilesAges — an age rule on the temp dir or a parent is True; a rule on a subdir or elsewhere, or with no age field, is False; an x/X exclusion covering the dir, or a glob that could reach it, is None (unknown); an exclusion strictly below the dir does not suppress a parent ageing rule; comments/blanks are ignored; an absent tool, non-zero exit or probe failure is None.
  • test_spawn_audit.py — the systemd-tmpfiles --cat-config probe is added to the benign-spawn allowlist as a fixed-argv, read-only, trusted-bin-resolved system query, alongside the existing doctor probes.
  • The doctor facade/report/section ratchets are updated to carry the new row and the forwarded helper.

Manual verification

Ran the row against the real host: it resolved the temp directory in use, reported xfs on its device and no ageing rule, fired no warning, and left the issues list empty — confirming the advisory, exit-code-neutral contract end to end.

Screenshots / video

N/A — backend-only diagnostic row with no frontend or user-visible UI surface.

Related Issues

Closes #17126

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A, the row is self-describing
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d01d6516051c340beadc6baa896f481a0fe9dfce touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d01d6516051c340beadc6baa896f481a0fe9dfce — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the row is read-only and advisory, reuses the existing mount resolver and swap reader, and leaves the exit code alone.

Notes

  • Adding about 217 lines takes doctor_checks/resources.py to 1,618 lines, over the 1,600 split threshold in AGENTS.md. The temp-directory row and _tmpfiles_ages are a new, separate responsibility, so they belong in a sibling module such as doctor_checks/temp_dirs.py, still re-exported through the cli_doctor facade.
  • The > to >= tie-break in _linux_mount_type also changes what detect_network_filesystem returns for the taskq WAL decision on stacked mounts. That is a fix for a second caller in a doctor PR. The description mentions it, but no test pins the stacked-mount case.

[DESIGN-REVIEWED] d01d651

Description read

[DESCRIPTION-READ] a18b80b626aa6054806be060e98a28a89667112c245594409eab55f72ad4861b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d01d6516051c340beadc6baa896f481a0fe9dfce — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The new row is read-only and advisory, it reuses the existing mount resolver and swap reader, and it does not re-measure the space and inode figures the Runtime tmpfs row already owns.

Notes

  • Item 5 — rides along: the _linux_mount_type change from > to >= also changes what taskq/store.py:315 gets back, and no test in this PR fails on base for it ("keeps its existing coverage"). The row does not need it to work: it prints a filesystem type either way, and only the stacked-mount case reads differently. Subtraction: defer the tie-break to its own fix together with a stacked-mount test.
  • Item 6 — one consumer, generalized: _doctor_temp_directories(issues) takes issues only to throw it away (del issues # ... keeps the call-site signature uniform). Subtraction: drop the parameter and call _doctor_temp_directories().
  • Item 1 — framing: the title promises "(tmpfs/no swap/low headroom)", but the diff says it "deliberately does NOT re-measure space/inodes". Subtraction: drop "low headroom" from the title.

What this change ships

Inventory (6 items) — 3 justified

Intent: the operator can see what filesystem backs their temp dir and gets a warning before tmpfs with no swap, or an age-based sweep, silently breaks work. ADDITION.

  1. A new "Temp directories" doctor section showing the path and filesystem — justified
  2. A warning when the temp dir is tmpfs and the host has no swap, without failing the doctor exit code — justified
  3. An ageing line saying whether a systemd-tmpfiles rule deletes files there by age — justified
  4. The doctor now runs systemd-tmpfiles --cat-config (spawn allowlist entry added) — justified
  5. The taskq mount-type resolver now picks the topmost of stacked mounts — rides along, no failing-on-base test
  6. An unused issues parameter on the new row — one consumer, generalized

[FIRST-PRINCIPLES-REVIEWED] d01d651

Description read

[DESCRIPTION-READ] 13484d45d5eb90b651362fa0ee6028ffbcf995376bd9cb8fd49f6fe5e7c40219

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d01d6516051c340beadc6baa896f481a0fe9dfce and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d01d651

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d01d6516051c340beadc6baa896f481a0fe9dfce false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d01d6516051c340beadc6baa896f481a0fe9dfce — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • src/kiro_crew/doctor_checks/resources.py is now 1,618 lines, past the 1,600-line working target in keep-modules-small. That rule is non-blocking. Moving the Temp directories row into its own doctor_checks/ module would keep the file under the target.

[OPUS-REVIEWED] d01d651

Verdict parsed from the review's SHA-scoped output markers for commit d01d6516051c340beadc6baa896f481a0fe9dfce.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d01d6516051c340beadc6baa896f481a0fe9dfce false-positive: <reason, 20 to 500 characters>

@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from 423acdd to c9b9e1f Compare October 21, 2026 05:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from c9b9e1f to e7b0f8f Compare October 21, 2026 05:28
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Verified: _mount_source_candidate_roots() (sandbox_mount_sweep.py:260) appends tempfile.gettempdir(), so _doctor_runtime_tmpfs already warns on space/inodes for this exact dir with the same thresholds. Confirmed legitimate.
Fixed in e7b0f8f: deleted the space/inodes print, low_space/low_inodes, and their reasons branches from _doctor_temp_directories. The row now shows only filesystem type/source, swap, and tmpfiles ageing — none of which the Runtime tmpfs row shows — and warns solely on tmpfs-with-no-swap.

  • Item 3: the space/inodes check duplicates the Runtime tmpfs row — removed the duplicate measurement entirely; the row no longer reads statvfs.

@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from e7b0f8f to 5212113 Compare October 21, 2026 05:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from 5212113 to 88424d4 Compare October 21, 2026 05:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from 88424d4 to 3a365be Compare October 21, 2026 06:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from 3a365be to 50d1188 Compare October 21, 2026 06:56
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Verified: taskq/store.py::_linux_mount_type already does the longest-mount-prefix fs-type lookup. Took the subtraction exactly as directed.
Fixed in 50d1188: deleted _mountinfo_fs, _PROC_MOUNTINFO and their facade exports; the row now reads the type through _linux_mount_type (via a function-local import, so the family binds no project module at module scope) and drops the "on " part. The over-mount tie-break is now the one-character >= inside _linux_mount_type, which also corrects its existing taskq caller. Dropped "low headroom" from the title and the call-site comment (Item 6).

  • Mount-table lookup reimplemented — removed; the row reuses _linux_mount_type and no second parser ships.

@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from 50d1188 to 0ea8a5f Compare October 21, 2026 07:09
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from 0ea8a5f to 38a2c22 Compare October 21, 2026 07:19
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch 2 times, most recently from aec8293 to d4c9483 Compare October 21, 2026 07:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

F1 (:826, UPHELD) verified: a quoted path containing a space (d "/tmp/crew work" 10d) shifted every whitespace-split field, so the age read as - and the rule was skipped -> false "no ageing rule". Fixed in d4c9483: a line carrying a quote or backslash is now unreadable at the line level and returns unknown for the whole probe, before any positional field is trusted. Test added.
F2 (:862 glob exclusion no-age) and F3 (:874 duplicate first-entry precedence) were Opus-DOWNGRADED as disproportionate; per the conductor's frozen scope this reader matches only plain literal paths and reports unknown for anything else — it does not reimplement systemd's exclusion-precedence or override-ordering semantics. The row is advisory only.

  • span=24b55a5c658f — F1 fixed (line-level quote/backslash -> unknown); F2/F3 downgraded, left to the literal-only invariant.

@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from d4c9483 to c7dcbaa Compare October 21, 2026 07:46
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Verified: the line-level quote/backslash check was too broad — it bailed on a quote ANYWHERE, including stock systemd's f+! /run/nologin ... "System is booting up." whose quote is only in the argument field, so the whole probe returned unknown on every systemd host even when /tmp is plainly aged.
Fixed in c7dcbaa: the check now looks only at the six STRUCTURAL fields (type, path, mode, uid, gid, age) via line.split(None, 6)[:6]; a quote in the argument field is exempt. A quoted path with a space still lands in a structural field -> unknown (F1 preserved). Test added for the nologin case still yielding True.

  • span=199b4e7bfae0 — fixed: quote/backslash check scoped to structural fields; argument-field quotes no longer blind the probe.

@dodgeblaster

Copy link
Copy Markdown
Contributor Author

The red Coverage Gate on this head is not this PR's. It is the repo-wide regression tracked in #22697: on a backend-only diff the frontend-lint job is skipped, which poisons lint-gate and skips every test shard, so frontend-test=skipped and the gate fails closed. Triangulated: PRs #22691 and #22689 (backend-only, sharing none of this diff) fail Coverage Gate with the byte-identical frontend-test=skipped — failing closed line this head shows. Every other check on this head is green (Backend Tests all shards, Build Desktop, E2E, Backend Lint) and all four AI-review lanes pass. The Coverage Gate clears once #22697 lands and this PR is rebased onto it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@dodgeblaster
dodgeblaster force-pushed the feat/doctor-temp-dirs-17126 branch from c7dcbaa to d01d651 Compare October 29, 2026 22:49
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

Rebased: main moved c7dcbaa's base (098bc0a) -> a8fe017. #25423 (merged) reworked the Runtime tmpfs row to measure /tmp and the process temp roots; this PR's diff is unchanged except one block-comment line and the body's motivation, which now describe that row accurately. Same 7 files; CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 29, 2026
@dodgeblaster
dodgeblaster marked this pull request as ready for review October 29, 2026 23:41
@dodgeblaster
dodgeblaster requested review from a team and KiroCrewCaptain as code owners October 29, 2026 23:41
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 29, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to ship. The new Temp directories row is read-only and advisory, never touches the exit code, and reuses the existing mount resolver and swap reader.

Head reviewed: d01d651. Readiness is passed, every check on this head is green, and all AI lanes pass with no BLOCK. Linked issue is tier T2.

What I verified at this head:

  • Read every hunk in all 7 files.
  • Ran the new row on a real Linux host. It printed the path, the filesystem type and the ageing line, and added no issue. _tmpfiles_ages returned True for /tmp and /var/tmp on that host's stock tmpfiles config.
  • Ran TestTempDirectories, TestTmpfilesAges and test/test_taskq_store.py: 29 passed.
  • Security: the only new spawn is systemd-tmpfiles --cat-config, resolved through trusted_system_bin, fixed argv, 5s timeout, stdout read only. No auth, trust boundary, sandbox, approval or secret change.

Two non-blocking notes:

  1. taskq/store.py line 248: no test pins the > to >= tie-break. I reverted it to > locally and test/test_cli_doctor.py plus test/test_taskq_store.py still passed (208 passed). A small test with two /proc/mounts lines at the same mount point (ext4 then tmpfs on /tmp) that expects tmpfs would pin it for both callers. A follow-up is fine.
  2. The PR title still says "low headroom", but the row deliberately does not measure space or inodes. Dropping that word from the title would make the squash commit match the change.

@bolichen97
bolichen97 merged commit 3282902 into main Oct 30, 2026
97 of 98 checks passed
@bolichen97
bolichen97 deleted the feat/doctor-temp-dirs-17126 branch October 30, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

doctor: add a "Temp directories" row (tmpfs /tmp, no swap, low space/inodes)

2 participants