Skip to content

fix(sandbox): re-apply the agent slice ceilings after a re-login - #22342

Merged
bolichen97 merged 1 commit into
mainfrom
fix/slice-memhigh-reapply-16723
Oct 29, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/slice-memhigh-reapply-16723

Conversation

@JLargent42

@JLargent42 JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: After a user logs out and back in, a running gateway puts the agent slice's aggregate memory and task ceilings back, without a restart.

On a Linux host without lingering (for example a Cloud Desktop after the last session closes), logind stops the user manager and the slice's --runtime properties vanish with it. Since the scope probe follows the login session, spawns are bounded again after re-login, but the slice-level MemoryHigh stays off until the gateway restarts, with no log line. Three cases from #16723 and its follow-up comment: a clean re-login, an apply that fails while the manager is going away, and a gateway that started while logged out (it never gets the slice MemoryMax/TasksMax at all).

Why it matters

The slice ceiling is what throttles the sum of all agents at 75% of RAM before the kernel OOM-kills. Without it, many concurrent agents can push a swapless host into memory pressure and take the gateway down with them. Users on Cloud Desktops report periodic memory/CPU-pressure crashes of the gateway; this is one silent way the protection goes missing.

Not a goal

What changed (motivation → approach → change)

The two slice reconcilers cached their outcome for the life of the process: an applied MemoryHigh value made every later reconcile a no-op, a failure disarmed it for good, and the hard ceiling was only applied at startup, so a gateway that started logged out never got it. That was safe while the probe answer never changed inside one process; it changes now, per login session.

The probe in src/kiro_crew/sandbox_cgroup.py now counts an enforcement epoch: it goes up each time the probe finds enforcement after none, or under a new session fingerprint. The MemoryHigh reconciler in src/kiro_crew/sandbox_slice.py stamps what it recorded with the epoch, and state from an older epoch is discarded before the next reconcile. For the hard ceiling, the pressure sampler already re-applies one that was applied and then vanished; what nothing covered is a session that never got it, so the spawn path's reconcile worker applies it only then, once per epoch, which keeps a broken systemctl from being retried on every spawn. A kill switch set outside a reconcile carries no stamp and stays set, so the test suite's disarm still holds. MemoryHigh is not detected by reading memory.high in the sampler instead, because that read cannot tell a failed apply from a deliberate disarm, and the slice cgroup does not exist after a re-login until the first scope does, so the read would only answer after spawns already started without the ceiling. src/kiro_crew/sandbox.py only re-exports the new names through its facade, and docs/architecture/resource-protection.md now says when each slice ceiling is applied again.

Backwards compatibility

Compatible: no config, API or stored data changes; the ceilings are applied with the same systemctl --user set-property --runtime calls, only also after a re-login.

Removes nothing: no capability, key or record is taken away; a failed apply now disarms for one login session instead of for the whole process.

Tests

  • test/test_slice_ceiling_reapply_after_relogin.py (new): MemoryHigh re-applied after a re-login; a failure disarms only that session; an outside kill switch survives a re-login; the spawn path schedules a reconcile after a failure in an older session; slice limits applied for a gateway that started logged out, once per session; an applied slice ceiling left to the sampler's self-heal after a re-login; a failed apply not retried in the same session; the probe's epoch rules.
  • test/conftest.py: the autouse MemoryHigh disarm also resets and restores the two epoch stamps, so a stamp left by one test cannot re-arm the kill switch in a later one.
  • test/test_sandbox_refactor_facade.py: the new names are listed with their owners.

Determinism: the new file passes in file order and in reversed order, alone and alongside TestAgentSliceMemoryHigh. Without the fix, the issue's repro (apply, probe flips off then on under a new session, apply) makes one set-property call instead of two.

Manual verification

Not done on a real host: it needs the last login session of a non-lingering systemd host to end under a running gateway, which on a dev or CI host ends the session running the check. Steps for a maintainer: start the gateway from an SSH session on a host without loginctl enable-linger, run one agent, close every session, log in again, run one agent, then read systemctl --user show kirocrew-agents.slice -p MemoryHigh -p MemoryMax (both set) and the gateway log line agent slice kirocrew-agents.slice: MemoryHigh=... applied after the re-login. Unit coverage drives the real _probe_cgroup_scope flip and fakes only subprocess.run.

Related Issues

Closes #16723

Pattern harvest

Rule candidate: review-prompt
Pattern: a process-lifetime cache of state owned by an external resource that can be recreated while the process runs (here a login-scoped --runtime property).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/sandbox.py

  • src/kiro_crew/sandbox_cgroup.py

  • src/kiro_crew/sandbox_slice.py

  • @JLargent42 posted a complete reasoning template for a8e2053.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a8e2053f7f6a1de045d51403231675ffd5a3e0cd touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a8e2053f7f6a1de045d51403231675ffd5a3e0cd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: each slice ceiling's state is now stamped with the login session it was recorded in, which fixes the cause rather than the symptom, and it reuses the existing probe and spawn-path seams with no new surface.

[DESIGN-REVIEWED] a8e2053

Description read

[DESCRIPTION-READ] aac6be063b6e756b93f00995431112c03719425d3a27bc7dcd366fb9d078046c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a8e2053f7f6a1de045d51403231675ffd5a3e0cd — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix comes from #16723's three cases and works at the cause: state tied to a login session was being cached for the life of the process. Two paths now re-apply the hard ceiling, and one could go.

Notes

  • Item 3 — oversized: after a re-login, two triggers now re-apply MemoryMax/TasksMax. One is the sampler's existing self-heal (if _SLICE_LIMITS_APPLIED: ... memory.max == "max", sandbox_slice.py:937). The other is the new per-epoch spawn path. The description itself says "the slice cgroup does not exist after a re-login until the first scope does", so both fire at the first spawn. Subtraction: make _reapply_slice_limits_for_new_session cover the applied-then-vanished case as well, and delete the self-heal block at sandbox_slice.py:930-943. That leaves one re-apply path, keyed by epoch.

What this change ships

Inventory (7 items) — 7 justified

Intent: keep the agent slice's aggregate memory and task ceilings in force after a user logs out and back in, without restarting the gateway. FIX.

  1. The first spawn after a re-login re-applies MemoryHigh to the new user manager — justified
  2. A failed MemoryHigh apply now disarms it for one login session, not for the whole process — justified
  3. A gateway that started while the user was logged out gets the slice MemoryMax/TasksMax at the first spawn after login, once per session — justified
  4. The probe counts an enforcement epoch per session, read through a new accessor — justified
  5. The SECURITY warning for a failed apply now fires once per login session, not once per process — justified
  6. The resource-protection doc now says when each slice ceiling is applied again — justified
  7. The test conftest disarm now resets and restores the epoch stamps — justified

[FIRST-PRINCIPLES-REVIEWED] a8e2053

Description read

[DESCRIPTION-READ] ec02774dfe38c7258b1599ce43d3fa7e56aa075c322d32c28f5e135ab55e5da1

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a8e2053f7f6a1de045d51403231675ffd5a3e0cd — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a8e2053

Verdict parsed from the review's SHA-scoped output markers for commit a8e2053f7f6a1de045d51403231675ffd5a3e0cd.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a8e2053f7f6a1de045d51403231675ffd5a3e0cd false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a8e2053f7f6a1de045d51403231675ffd5a3e0cd and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a8e2053

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a8e2053f7f6a1de045d51403231675ffd5a3e0cd false-positive: <reason, 20 to 500 characters>

@JLargent42
JLargent42 force-pushed the fix/slice-memhigh-reapply-16723 branch from 31016c7 to 504eaa7 Compare October 20, 2026 01:21
@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=6bcfe8a8f674: each path has a gap the other covers: fixed in 504eaa7

The sampler now covers every vanished-after-applied case for the hard ceiling (including a manager restart under the same fingerprint), and the epoch covers only the never-applied case (gateway started logged out), which the sampler cannot see because it gates on _SLICE_LIMITS_APPLIED. For MemoryHigh, the fingerprint includes the bus sockets' inode identity, so a user-manager restart that recreates its sockets does change it.

@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=06b1e2622bc2: duplicate of check_agents_slice_pressure, re-login half: fixed in 504eaa7

The shrink is made: the spawn-path re-apply after a re-login is removed for a ceiling that was applied, leaving that to the check_agents_slice_pressure self-heal; the contradicting docstring claim is gone and the PR description names the self-heal. What remains is the started-logged-out case, where the self-heal never fires because _SLICE_LIMITS_APPLIED was never True (test_slice_limits_follow_a_gateway_that_started_logged_out).

@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=be7420c24c9e: sampler memory.high check; epoch only for started-logged-out: fixed in 504eaa7 (second half), rebutted (first half)

Taken: the epoch is now used for the hard ceiling only in the started-logged-out case. Not taken: a sampler memory.high check for MemoryHigh, for the reasons in the span a801d2f026a9 reply; MemoryHigh stays on the per-spawn reconcile with epoch-scoped state.

@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=b26562ac794b: the two paths race on the same globals: fixed in 504eaa7

The spawn path now acts only while _SLICE_LIMITS_APPLIED is False and the sampler only while it is True, and ensure_agents_slice_limits keeps its original early return. A concurrent overlap can at most repeat one idempotent set-property --runtime call.

@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=542c5291cac9: Two re-apply mechanisms for one property: fixed in 504eaa7

Legitimate. The spawn-path re-apply of the hard ceiling now runs only when the ceiling was never applied in this session (_reapply_slice_limits_for_new_session returns when _SLICE_LIMITS_APPLIED is set). A ceiling that was applied and then vanished stays the sampler self-heal's job alone, so each path owns a disjoint state.

@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=2c14b6c48eb8: maintenance trap: fixed in 504eaa7

Removed by making the two hard-ceiling paths act on disjoint states; both docstrings (ensure_agents_slice_limits, _reapply_slice_limits_for_new_session) name the other path and its scope, and test_an_applied_slice_ceiling_is_left_to_the_sampler_self_heal pins the split.

@JLargent42

JLargent42 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=a801d2f026a9: detect memory.high == max in the sampler: rebutted

That would add a second, sampler-driven MemoryHigh path beside the per-spawn reconcile that already owns MemoryHigh, which is the duplication the other items object to. It also cannot tell a failure-disarm from the deliberate kill switch, and the slice cgroup is absent until the first scope after a re-login exists. The epoch keeps MemoryHigh inside its one existing mechanism and re-applies on the first spawn of the new session.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@JLargent42
JLargent42 force-pushed the fix/slice-memhigh-reapply-16723 branch from 504eaa7 to ec26a87 Compare October 20, 2026 01:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=ea2be3fec1bc: set-property --runtime on the freshly recreated slice is accepted: rebutted

The PR issues no new systemd call: it reuses the exact systemctl --user set-property --runtime commands main already sends to a fresh user manager at every gateway start (ensure_agents_slice_limits, _ensure_agent_slice_memory_high on the first spawn) and from the sampler self-heal after a logout. What changes is only that the call is made again after a re-login. If systemd refused it, the existing failure path applies: one SECURITY warning and a disarm, now for that login session only.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=115737a63530: the session fingerprint actually changes on a re-login: rebutted

The epoch does not depend on it alone: at the logout the probe already answers unavailable (the runtime directory and bus are gone), and the first available answer after that bumps the epoch whatever the fingerprint is (not cached[0] in _probe_cgroup_scope). The fingerprint is the second trigger, for a re-login that falls inside the 60 s cache; it stats the runtime directory, the bus sockets and the user slice's controllers file, all recreated by logind, and test_ceiling_returns_when_the_manager_comes_back (shipped with the probe change) recreates a real listening AF_UNIX socket to exercise exactly that flip. A real logout on the CI or dev host would end the session running the test, so it is not run here; that check is left to the maintainer.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=9f1937279ebc: resource-protection.md says the hard cap is applied only at startup: fixed in ec26a87

The doc now says the hard cap is applied at startup or on the first spawn of a login session that started after the gateway, that the sampler re-applies a vanished one, and that MemoryHigh is re-applied on the first spawn after a re-login and disarms only for one session on failure.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=94c6154afb52: the probe recomputes before the next spawn's reconcile reads the epoch: rebutted

This is ordered in code, not timing: cgroup_scope_argv calls _probe_cgroup_scope() as its first statement and returns early when it is unavailable; only after that does it schedule _reconcile_slice_memory_high_off_thread(), whose worker reads _cgroup_scope_epoch(). So the reconcile always sees the epoch this spawn's probe wrote. test_the_spawn_path_schedules_a_reconcile_after_a_failure_in_an_older_session drives that worker path.

@JLargent42

Copy link
Copy Markdown
Contributor Author

Answer to the PASS ask (the two hard-ceiling paths must never both run for one session): they key on opposite values of _SLICE_LIMITS_APPLIED. The spawn path returns when it is True; the sampler self-heal acts only when it is True and sets it False just before its own ensure_agents_slice_limits() call. ensure_agents_slice_limits returns early when it is already True, so even an interleaving issues at most one extra idempotent set-property --runtime. Pinned by test_an_applied_slice_ceiling_is_left_to_the_sampler_self_heal and test_slice_limits_follow_a_gateway_that_started_logged_out.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=c8ddef2e0717: set-property --runtime works from the reconcile worker against the new manager: rebutted

The reconcile worker already sends this exact call on main, against whatever user manager is live, on the first spawn of every gateway; the sampler self-heal sends the hard-ceiling call against a re-created manager after a logout. This PR adds no new call shape or caller thread, only one more occasion. A refusal takes the existing failure path: one SECURITY warning, disarmed for that session only.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=fa671d08df1d: a real logind re-login bumps the epoch: rebutted

Either trigger suffices, and the first does not depend on the fingerprint: logind removes the runtime directory and bus at the last logout, so the probe answers unavailable, and the first available answer after that bumps the epoch (not cached[0] in _probe_cgroup_scope). The second, a new fingerprint, covers a re-login inside the 60 s cache; it stats the runtime directory, bus sockets and slice controllers file that logind recreates. test_ceiling_returns_when_the_manager_comes_back exercises the flip with a real listening AF_UNIX socket removed and recreated.

@JLargent42

Copy link
Copy Markdown
Contributor Author
  • span=c984c584fff2: the epoch duplicates the sampler's cgroup-state check: rebutted

The two now detect different things. For the hard ceiling the sampler's memory.max == max read stays the only detector of a vanished ceiling; the epoch only marks a session the ceiling was never applied in, which a cgroup read cannot see (no applied value to compare). For MemoryHigh, reading memory.high cannot tell a failed apply under the new manager from a deliberate disarm, and after a re-login the slice cgroup does not exist until the first scope does, so it would answer only after spawns started unthrottled. The description's What changed now says this.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@JLargent42
JLargent42 force-pushed the fix/slice-memhigh-reapply-16723 branch from 3879a5e to 0e9e258 Compare October 26, 2026 18:03
@JLargent42

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 0e9e258

  • What rule changed:
    The agent-slice resource ceilings (MemoryHigh on kirocrew-agents.slice, and
    MemoryMax/MemorySwapMax=0/TasksMax from ensure_agents_slice_limits())
    are systemctl --user set-property --runtime properties that vanish with the
    login session. Previously they were applied once at gateway startup and the
    applied/disabled state was process-lifetime. This adds an enforcement-epoch
    counter (_CGROUP_SCOPE_EPOCH, bumped by _probe_cgroup_scope whenever it
    newly finds enforcement available — first success, recovery from unavailable,
    or a changed session fingerprint) and stamps the slice state with the epoch it
    was recorded in. Stale-epoch state is discarded so the next agent spawn
    re-applies the ceiling to the new user manager.

  • Before -> after (what was allowed/blocked, now):
    Before: after a user re-login (or a gateway that started while the user was
    logged out), the runtime ceiling was gone and nothing re-applied it at the
    slice level — agent spawns proceeded with the AGGREGATE slice ceiling NOT
    enforced (per-scope MemoryMax still applied). A failed apply disarmed the
    reconciler for the whole process.
    After: the first agent spawn in a new login epoch re-applies MemoryHigh;
    a gateway that never applied the hard ceiling gets it from the reconcile
    worker via _reapply_slice_limits_for_new_session(); a prior failure disarms
    only until the next login epoch. No permission, path, or allow/deny rule is
    loosened — this strictly re-arms an existing protection.

  • Worst case if wrong (breaks commands / leaks secret / locks user out):
    No secret path, no auth rule, no command allow/deny list is touched, so it
    cannot leak a secret or lock a user out. The realistic worst case is a logic
    bug in epoch tracking causing either (a) a redundant set-property call per
    spawn (a cheap idempotent no-op string-compare in steady state), or (b) the
    ceiling still not re-applying — i.e. the pre-existing behavior, no worse than
    today. _ensure_agent_slice_memory_high never raises, so an agent spawn can
    never fail because of this code; failures degrade to a loud SECURITY warning
    and uncontained-at-slice-level (per-scope MemoryMax still bounds each scope).

  • Evidence checked (which sandbox result proves it):
    New test test/test_slice_ceiling_reapply_after_relogin.py drives the real
    functions against a fake subprocess.run (the host user manager is never
    contacted) and proves: MemoryHigh re-applies after a relogin
    (test_memory_high_is_applied_again_after_a_relogin); a failure seen at
    logout disarms only that session and recovers next login
    (test_a_failure_seen_at_logout_disarms_only_that_session); a kill switch set
    outside a reconcile survives a relogin
    (test_a_kill_switch_set_outside_a_reconcile_survives_a_relogin); the hard
    ceiling follows a gateway that started logged out and runs once per session
    (test_slice_limits_follow_a_gateway_that_started_logged_out,
    test_a_failed_slice_limits_apply_is_not_retried_in_the_same_session); an
    already-applied ceiling is left to the pressure-sampler self-heal, so the two
    re-apply paths never double-act
    (test_an_applied_slice_ceiling_is_left_to_the_sampler_self_heal); and the
    probe counts exactly one epoch per session it finds enforcement in
    (test_the_probe_counts_a_new_epoch_per_session_it_finds_enforcement_in).

  • No regression:

    • Tools still work (which tools/commands were run, result):
      _ensure_agent_slice_memory_high is unchanged in steady state (same-epoch
      reconcile is still a string compare and no-ops); set-property argv is
      the same. Agent spawns are never blocked or made to raise.
    • Backward compatible (old config / old data / old callers still work, how checked): no config keys, serialized state, or on-disk artifacts are
      introduced — the new state is in-process module globals only. Callers of
      ensure_agents_slice_limits() keep the same signature/return. The new
      _reapply_slice_limits_for_new_session() is additive and only invoked from
      the reconcile worker.
    • Existing tests (which suites passed on this head): rebased onto current main
      (resolving two additive name-list conflicts with fix(sandbox): give each agent runtime scope a finite MemoryHigh #24205 in src/kiro_crew/sandbox.py
      and test/test_sandbox_refactor_facade.py), then ran locally with pytest -n 2:
      the slice/cgroup/facade/conftest suites (766 passed, 17 skipped) and the test files
      touched by fix(sandbox): give each agent runtime scope a finite MemoryHigh #24205 including test/test_scope_memory_high.py (724 passed, 17 skipped).
      Black (py310), isort (full CI scope), flake8 and mypy on the changed sandbox modules
      are clean; this head also re-sorts one forwarded name in src/kiro_crew/sandbox.py.
    • How to undo:
      Revert commit 0e9e258. It is a single
      self-contained commit touching sandbox epoch-tracking plus its tests and
      doc; reverting restores the prior process-lifetime apply with no migration.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026
@JLargent42
JLargent42 force-pushed the fix/slice-memhigh-reapply-16723 branch from 0e9e258 to dcec4c3 Compare October 27, 2026 02:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@JLargent42

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: dcec4c3

  • What rule changed:
    The agent-slice resource ceilings (MemoryHigh on kirocrew-agents.slice, and
    MemoryMax/MemorySwapMax=0/TasksMax from ensure_agents_slice_limits())
    are systemctl --user set-property --runtime properties that vanish with the
    login session. Without this change they are applied once at gateway startup and
    the applied/disabled state is process-lifetime. This adds an enforcement-epoch
    counter (_CGROUP_SCOPE_EPOCH, bumped by _probe_cgroup_scope whenever it
    newly finds enforcement available — first success, recovery from unavailable,
    or a changed session fingerprint) and stamps the slice state with the epoch it
    was recorded in. Stale-epoch state is discarded so the next agent spawn
    re-applies the ceiling to the new user manager.

  • Before -> after (what was allowed/blocked, now):
    Before: after a user re-login (or a gateway that started while the user was
    logged out), the runtime ceiling was gone and nothing re-applied it at the
    slice level — agent spawns proceeded with the AGGREGATE slice ceiling NOT
    enforced (per-scope MemoryMax still applied). A failed apply disarmed the
    reconciler for the whole process.
    After: the first agent spawn in a new login epoch re-applies MemoryHigh;
    a gateway that never applied the hard ceiling gets it from the reconcile
    worker via _reapply_slice_limits_for_new_session(); a prior failure disarms
    only until the next login epoch. No permission, path, or allow/deny rule is
    loosened — this strictly re-arms an existing protection.

  • Worst case if wrong (breaks commands / leaks secret / locks user out):
    No secret path, no auth rule, no command allow/deny list is touched, so it
    cannot leak a secret or lock a user out. The realistic worst case is a logic
    bug in epoch tracking causing either (a) a redundant set-property call per
    spawn (a cheap idempotent no-op string-compare in steady state), or (b) the
    ceiling still not re-applying — i.e. the pre-existing behavior, no worse than
    today. _ensure_agent_slice_memory_high never raises, so an agent spawn can
    never fail because of this code; failures degrade to a loud warning and
    uncontained-at-slice-level (per-scope MemoryMax still bounds each scope).

  • Evidence checked (which sandbox result proves it):
    New test test/test_slice_ceiling_reapply_after_relogin.py drives the real
    functions against a fake subprocess.run (the host user manager is never
    contacted) and proves: MemoryHigh re-applies after a relogin
    (test_memory_high_is_applied_again_after_a_relogin); a failure seen at
    logout disarms only that session and recovers next login
    (test_a_failure_seen_at_logout_disarms_only_that_session); a kill switch set
    outside a reconcile survives a relogin
    (test_a_kill_switch_set_outside_a_reconcile_survives_a_relogin); the hard
    ceiling follows a gateway that started logged out and runs once per session
    (test_slice_limits_follow_a_gateway_that_started_logged_out,
    test_a_failed_slice_limits_apply_is_not_retried_in_the_same_session); an
    already-applied ceiling is left to the pressure-sampler self-heal, so the two
    re-apply paths never double-act
    (test_an_applied_slice_ceiling_is_left_to_the_sampler_self_heal); and the
    probe counts exactly one epoch per session it finds enforcement in
    (test_the_probe_counts_a_new_epoch_per_session_it_finds_enforcement_in).

  • No regression:

    • Tools still work (which tools/commands were run, result): _ensure_agent_slice_memory_high is unchanged in steady state (same-epoch
      reconcile is still a string compare and no-ops); set-property argv is
      the same. Agent spawns are never blocked or made to raise.
    • Backward compatible (old config / old data / old callers still work, how checked): no config keys, serialized state, or on-disk artifacts are
      introduced — the new state is in-process module globals only. Callers of
      ensure_agents_slice_limits() keep the same signature/return. The new
      _reapply_slice_limits_for_new_session() is additive and only invoked from
      the reconcile worker.
    • Existing tests (which suites passed on this head): rebased cleanly onto current main
      96c4fef5 (no conflicts; includes the crew_log lease fix), then ran locally with
      pytest -n 2 across the slice re-login, sandbox facade, agents-slice admission,
      cgroup-scope probe/ratchet, slice identity/encoding, scope MemoryHigh, resource-limits
      schema, signing-key preflight and subslice-notice suites: 784 passed, 17 skipped, 1
      timed out on a host memory stall (a whole-tree source scan) and passed on a serial
      re-run in 0.63s. Black (py310), isort (full CI scope), flake8, mypy on the changed
      sandbox modules and the comment-history gate are clean.
    • How to undo:
      Revert commit dcec4c3. It is a single
      self-contained commit touching sandbox epoch-tracking plus its tests and
      doc; reverting restores the prior process-lifetime apply with no migration.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD on dcec4c3, reserved for human review.

This PR changes when the agent slice's aggregate memory and task ceilings are applied, re-armed and disarmed after a re-login (sandbox_cgroup.py, sandbox_slice.py, sandbox.py). Sandbox enforcement changes go to a human, so I am not approving it. The head is a rebase of 8525187 (r123) with an identical changed-line set, and I found no definite defect in the epoch bump, the stale-state reset or the once-per-epoch hard-ceiling apply. All AI lanes pass, the sensitive-change template is posted for this head, and CI is green.

Left for the human reviewer: the Design item about no real logout/re-login on a non-lingering host is answered as needs-a-decision on unit coverage only. A maintainer either runs the Manual verification steps and posts the systemctl --user show kirocrew-agents.slice output, or accepts unit coverage. Auto-merge is armed, so it lands on the first approval.

— crew-pr-reviewer (r128)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 28, 2026
@JLargent42
JLargent42 force-pushed the fix/slice-memhigh-reapply-16723 branch from dcec4c3 to a8e2053 Compare October 28, 2026 18:23
@JLargent42

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: a8e2053

  • What rule changed:
    The agent-slice resource ceilings (MemoryHigh on kirocrew-agents.slice, and
    MemoryMax/MemorySwapMax=0/TasksMax from ensure_agents_slice_limits())
    are systemctl --user set-property --runtime properties that vanish with the
    login session. Without this change they are applied once at gateway startup and
    the applied/disabled state is process-lifetime. This adds an enforcement-epoch
    counter (_CGROUP_SCOPE_EPOCH, bumped by _probe_cgroup_scope whenever it
    newly finds enforcement available — first success, recovery from unavailable,
    or a changed session fingerprint) and stamps the slice state with the epoch it
    was recorded in. Stale-epoch state is discarded so the next agent spawn
    re-applies the ceiling to the new user manager.

  • Before -> after (what was allowed/blocked, now):
    Before: after a user re-login (or a gateway that started while the user was
    logged out), the runtime ceiling was gone and nothing re-applied it at the
    slice level — agent spawns proceeded with the AGGREGATE slice ceiling NOT
    enforced (per-scope MemoryMax still applied). A failed apply disarmed the
    reconciler for the whole process.
    After: the first agent spawn in a new login epoch re-applies MemoryHigh;
    a gateway that never applied the hard ceiling gets it from the reconcile
    worker via _reapply_slice_limits_for_new_session(); a prior failure disarms
    only until the next login epoch. No permission, path, or allow/deny rule is
    loosened — this strictly re-arms an existing protection.

  • Worst case if wrong (breaks commands / leaks secret / locks user out):
    No secret path, no auth rule, no command allow/deny list is touched, so it
    cannot leak a secret or lock a user out. The realistic worst case is a logic
    bug in epoch tracking causing either (a) a redundant set-property call per
    spawn (a cheap idempotent no-op string-compare in steady state), or (b) the
    ceiling still not re-applying — i.e. the pre-existing behavior, no worse than
    today. _ensure_agent_slice_memory_high never raises, so an agent spawn can
    never fail because of this code; failures degrade to a loud warning and
    uncontained-at-slice-level (per-scope MemoryMax still bounds each scope).

  • Evidence checked (which sandbox result proves it):
    New test test/test_slice_ceiling_reapply_after_relogin.py drives the real
    functions against a fake subprocess.run (the host user manager is never
    contacted) and proves: MemoryHigh re-applies after a relogin
    (test_memory_high_is_applied_again_after_a_relogin); a failure seen at
    logout disarms only that session and recovers next login
    (test_a_failure_seen_at_logout_disarms_only_that_session); a kill switch set
    outside a reconcile survives a relogin
    (test_a_kill_switch_set_outside_a_reconcile_survives_a_relogin); the hard
    ceiling follows a gateway that started logged out and runs once per session
    (test_slice_limits_follow_a_gateway_that_started_logged_out,
    test_a_failed_slice_limits_apply_is_not_retried_in_the_same_session); an
    already-applied ceiling is left to the pressure-sampler self-heal, so the two
    re-apply paths never double-act
    (test_an_applied_slice_ceiling_is_left_to_the_sampler_self_heal); and the
    probe counts exactly one epoch per session it finds enforcement in
    (test_the_probe_counts_a_new_epoch_per_session_it_finds_enforcement_in).

  • No regression:

    • Tools still work (which tools/commands were run, result): _ensure_agent_slice_memory_high is unchanged in steady state (same-epoch
      reconcile is still a string compare and no-ops); set-property argv is
      the same. Agent spawns are never blocked or made to raise.
    • Backward compatible (old config / old data / old callers still work, how checked): no config keys, serialized state, or on-disk artifacts are
      introduced — the new state is in-process module globals only. Callers of
      ensure_agents_slice_limits() keep the same signature/return. The new
      _reapply_slice_limits_for_new_session() is additive and only invoked from
      the reconcile worker.
    • Existing tests (which suites passed on this head): rebased onto current main
      2808e947, resolving two additive name-list conflicts (the facade forwarded-name
      list in sandbox.py and the _MOVED table in the facade test) by keeping both
      sides. Ran locally with pytest -n 2 across the slice re-login, sandbox facade,
      agents-slice reclaim and admission, resource-limits schema, scope MemoryHigh and
      cgroup-scope probe/ratchet suites: 759 passed, 17 skipped, 0 failed. Black (py310),
      isort (full CI scope), flake8, mypy on the changed sandbox modules and the
      comment-history gate are clean.
    • How to undo:
      Revert commit a8e2053. It is a single
      self-contained commit touching sandbox epoch-tracking plus its tests and
      doc; reverting restores the prior process-lifetime apply with no migration.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 28, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds an enforcement epoch to _probe_cgroup_scope. It goes up on the first available answer after an unavailable one, or under a new session fingerprint. The MemoryHigh applied value and kill switch are stamped with that epoch and dropped when it changes. The hard ceiling is applied from the spawn reconcile worker only when this login session never got it, once per epoch, and the sampler self-heal keeps the applied-then-vanished case. That covers all three cases in #16723: a clean re-login, a failure during logout, and a gateway started while logged out. #11576 stays open, as the body says.

Duplicate check: no other PR targets #16723, and main has no epoch logic in sandbox_slice.py. #22636 and #24207 touch the same file for other reasons. That only affects merge order, not correctness.

Verified: the epoch is incremented under the probe lock. cgroup_scope_argv probes before it schedules the reconcile. The spawn path and the sampler gate on opposite values of _SLICE_LIMITS_APPLIED. The test kill switch carries no stamp, so it stays set. The new test file drives each case. All checks are green, and every AI lane passes.

No blocking findings. A maintainer still needs to decide on the open manual re-login verification on a non-lingering host.

AI-assisted review (Kiro), posted for bolichen.

@bolichen97
bolichen97 merged commit 2859f88 into main Oct 29, 2026
178 of 186 checks passed
@bolichen97
bolichen97 deleted the fix/slice-memhigh-reapply-16723 branch October 29, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sandbox: agents-slice MemoryHigh is never re-applied after a logout/re-login (regression from #15622)

3 participants