Repository navigation
fix(sandbox): re-apply the agent slice ceilings after a re-login - #22342
Conversation
Sensitive Change ReviewSensitive files changed:
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: each slice ceiling's state is now stamped with the login session it was recorded in, which fixes the cause rather than the symptom, and it reuses the existing probe and spawn-path seams with no new surface. [DESIGN-REVIEWED] a8e2053 Description read[DESCRIPTION-READ] aac6be063b6e756b93f00995431112c03719425d3a27bc7dcd366fb9d078046c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix comes from #16723's three cases and works at the cause: state tied to a login session was being cached for the life of the process. Two paths now re-apply the hard ceiling, and one could go. Notes
What this change shipsInventory (7 items) — 7 justifiedIntent: keep the agent slice's aggregate memory and task ceilings in force after a user logs out and back in, without restarting the gateway. FIX.
[FIRST-PRINCIPLES-REVIEWED] a8e2053 Description read[DESCRIPTION-READ] ec02774dfe38c7258b1599ce43d3fa7e56aa075c322d32c28f5e135ab55e5da1 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
31016c7 to
504eaa7
Compare
|
|
|
|
|
|
|
504eaa7 to
ec26a87
Compare
|
|
|
|
|
|
|
|
3879a5e to
0e9e258
Compare
Sensitive change review
|
0e9e258 to
dcec4c3
Compare
Sensitive change review
|
|
HOLD on dcec4c3, reserved for human review. This PR changes when the agent slice's aggregate memory and task ceilings are applied, re-armed and disarmed after a re-login ( Left for the human reviewer: the Design item about no real logout/re-login on a non-lingering host is answered as needs-a-decision on unit coverage only. A maintainer either runs the Manual verification steps and posts the — crew-pr-reviewer (r128) |
dcec4c3 to
a8e2053
Compare
Sensitive change review
|
bolichen97
left a comment
There was a problem hiding this comment.
This PR adds an enforcement epoch to _probe_cgroup_scope. It goes up on the first available answer after an unavailable one, or under a new session fingerprint. The MemoryHigh applied value and kill switch are stamped with that epoch and dropped when it changes. The hard ceiling is applied from the spawn reconcile worker only when this login session never got it, once per epoch, and the sampler self-heal keeps the applied-then-vanished case. That covers all three cases in #16723: a clean re-login, a failure during logout, and a gateway started while logged out. #11576 stays open, as the body says.
Duplicate check: no other PR targets #16723, and main has no epoch logic in sandbox_slice.py. #22636 and #24207 touch the same file for other reasons. That only affects merge order, not correctness.
Verified: the epoch is incremented under the probe lock. cgroup_scope_argv probes before it schedules the reconcile. The spawn path and the sampler gate on opposite values of _SLICE_LIMITS_APPLIED. The test kill switch carries no stamp, so it stays set. The new test file drives each case. All checks are green, and every AI lane passes.
No blocking findings. A maintainer still needs to decide on the open manual re-login verification on a non-lingering host.
AI-assisted review (Kiro), posted for bolichen.
Problem / Motivation
Goal: After a user logs out and back in, a running gateway puts the agent slice's aggregate memory and task ceilings back, without a restart.
On a Linux host without lingering (for example a Cloud Desktop after the last session closes), logind stops the user manager and the slice's
--runtimeproperties vanish with it. Since the scope probe follows the login session, spawns are bounded again after re-login, but the slice-levelMemoryHighstays off until the gateway restarts, with no log line. Three cases from #16723 and its follow-up comment: a clean re-login, an apply that fails while the manager is going away, and a gateway that started while logged out (it never gets the sliceMemoryMax/TasksMaxat all).Why it matters
The slice ceiling is what throttles the sum of all agents at 75% of RAM before the kernel OOM-kills. Without it, many concurrent agents can push a swapless host into memory pressure and take the gateway down with them. Users on Cloud Desktops report periodic memory/CPU-pressure crashes of the gateway; this is one silent way the protection goes missing.
Not a goal
What changed (motivation → approach → change)
The two slice reconcilers cached their outcome for the life of the process: an applied MemoryHigh value made every later reconcile a no-op, a failure disarmed it for good, and the hard ceiling was only applied at startup, so a gateway that started logged out never got it. That was safe while the probe answer never changed inside one process; it changes now, per login session.
The probe in
src/kiro_crew/sandbox_cgroup.pynow counts an enforcement epoch: it goes up each time the probe finds enforcement after none, or under a new session fingerprint. The MemoryHigh reconciler insrc/kiro_crew/sandbox_slice.pystamps what it recorded with the epoch, and state from an older epoch is discarded before the next reconcile. For the hard ceiling, the pressure sampler already re-applies one that was applied and then vanished; what nothing covered is a session that never got it, so the spawn path's reconcile worker applies it only then, once per epoch, which keeps a brokensystemctlfrom being retried on every spawn. A kill switch set outside a reconcile carries no stamp and stays set, so the test suite's disarm still holds. MemoryHigh is not detected by readingmemory.highin the sampler instead, because that read cannot tell a failed apply from a deliberate disarm, and the slice cgroup does not exist after a re-login until the first scope does, so the read would only answer after spawns already started without the ceiling.src/kiro_crew/sandbox.pyonly re-exports the new names through its facade, anddocs/architecture/resource-protection.mdnow says when each slice ceiling is applied again.Backwards compatibility
Compatible: no config, API or stored data changes; the ceilings are applied with the same
systemctl --user set-property --runtimecalls, only also after a re-login.Removes nothing: no capability, key or record is taken away; a failed apply now disarms for one login session instead of for the whole process.
Tests
test/test_slice_ceiling_reapply_after_relogin.py(new): MemoryHigh re-applied after a re-login; a failure disarms only that session; an outside kill switch survives a re-login; the spawn path schedules a reconcile after a failure in an older session; slice limits applied for a gateway that started logged out, once per session; an applied slice ceiling left to the sampler's self-heal after a re-login; a failed apply not retried in the same session; the probe's epoch rules.test/conftest.py: the autouse MemoryHigh disarm also resets and restores the two epoch stamps, so a stamp left by one test cannot re-arm the kill switch in a later one.test/test_sandbox_refactor_facade.py: the new names are listed with their owners.Determinism: the new file passes in file order and in reversed order, alone and alongside
TestAgentSliceMemoryHigh. Without the fix, the issue's repro (apply, probe flips off then on under a new session, apply) makes oneset-propertycall instead of two.Manual verification
Not done on a real host: it needs the last login session of a non-lingering systemd host to end under a running gateway, which on a dev or CI host ends the session running the check. Steps for a maintainer: start the gateway from an SSH session on a host without
loginctl enable-linger, run one agent, close every session, log in again, run one agent, then readsystemctl --user show kirocrew-agents.slice -p MemoryHigh -p MemoryMax(both set) and the gateway log lineagent slice kirocrew-agents.slice: MemoryHigh=... appliedafter the re-login. Unit coverage drives the real_probe_cgroup_scopeflip and fakes onlysubprocess.run.Related Issues
Closes #16723
Pattern harvest
Rule candidate: review-prompt
Pattern: a process-lifetime cache of state owned by an external resource that can be recreated while the process runs (here a login-scoped
--runtimeproperty).Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)