Repository navigation
fix(slack): bound the per-thread override maps together - #18985
bolichen97 merged 2 commits into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The PR caps a cache of roughly 125 bytes per thread, but the real problem is that in-memory overrides can say something the persisted metadata doesn't, and it stays unfixed under a 292-line concurrency layer. Watch
Suggestions
[DESIGN-REVIEWED] b5ce892 Description read[DESCRIPTION-READ] 7595160b7d550d9ded4e1b20e112775dcaff0679ccb14b6965997023c1227f16 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The per-value length refusals and the setter-path write counter guard against harms that, by the description's own words and the diff's comments, are already covered. Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: stop the Slack gateway's per-thread override cache from growing for the life of the process. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] b5ce892 Description read[DESCRIPTION-READ] 7cad650013f3ae0f6214bd8e39137184c7b7cff36669c7c881c735d477c687a7 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
Thanks, the finding holds. Fixed in a second commit, 278bddc, on top of 1094535. What was wrong. The change.
Red. Green. On 278bddc the test file passes, 14 tests, 3 of 3 runs. The new test checks that each container stays within |
278bddc to
d42d0f6
Compare
|
Thanks, both findings hold. Fixed by amending the second commit, now d42d0f6; both commits are rebased onto main 843abe3, and the first, now 9195b87, has the same patch. F1 ( F2 ( Green: the test file passes 19 tests, 3 of 3; the 17 Slack handler test files from the description: 1,093 passed. |
d42d0f6 to
97a10d4
Compare
|
Thanks, both findings hold. Fixed by amending the second commit, now 97a10d4. The first, 9195b87, is unchanged, and both are still on main 843abe3. F1 ( The class behind F1. Each instance so far has been one class: a hydration read that began before a command's metadata write publishes the old value after it, because eviction takes the thread out of the hydration guard that on main stops such a read. So hydration now publishes nothing it read before a newer F2 ( Green on 97a10d4: the file's 28 tests pass 3 of 3, and the second commit's fifteen tests fail on the first commit 9195b87, 3 of 3. |
The Slack handler kept one entry per thread, for the life of the process, in the hydration guard (_hydrated_sessions) and the !ta / !project maps (_thread_agents, _thread_projects). Only the reset commands removed an override, and nothing removed a guard entry, so a gateway that answered many threads never gave the memory back. The three containers are now held to THREAD_SLOTS threads together, in the least-recently-used shape the thread-replies watermark already uses, and THREAD_SLOTS is that watermark's own bound, not a second literal. An evicted thread loses its guard entry and both overrides at once, so its next message hydrates again and reads the overrides back from the conversation metadata. An override a person set with !ta or !project in this process is pinned and never evicted while it is set, because its metadata write is best-effort and hydration prefers a recorded execution template over the agent field. Pinned threads count inside THREAD_SLOTS, at most MAX_PINNED_THREADS (THREAD_SLOTS - 1) of them, so one slot always stays evictable and hydrating a new thread never evicts that thread itself. A !ta or !project that would pin one more thread past that is refused with a reply that says so, and sets nothing. Each agent name and project path the maps keep is bounded by OVERRIDE_VALUE_MAX_CHARS (4096, Linux's PATH_MAX), whichever path writes it. Hydration keeps a value it reads back from the conversation metadata only when it is text within the limit, and skips any other with one warning that names its length or type. !ta and !project refuse a longer value before they pin or write anything, with a reply that names its length and the limit: a project agent's name is whatever its spec declares, and a Windows long path can pass 4096. So the maps keep at most THREAD_SLOTS threads x 2 values x 4096 characters.
`!ta` and `!project` await agent discovery or a directory check before they pin. Threads hydrating meanwhile can evict the command's own thread, and pin() then admitted it without evicting anything, so the pins and the unpinned threads together held THREAD_SLOTS + 1 threads, and the evicted thread had no hydration guard, so its next message hydrated over the override the command had just set. pin() now takes the shared containers, as note_hydrated does. It evicts the oldest unpinned threads from the guard and both override maps until the pins and the rest fit THREAD_SLOTS, through the same loop note_hydrated uses (_evict_over_bound), before the command publishes its override. It does not mark the thread hydrated: only a hydration whose read the store reports as readable does. Hydration reads the line once through thread_override_slots.read_back (get_metadata_status), and a line it cannot read leaves the thread unhydrated, so its next message reads it again; a line that stays unreadable is read on each message. Hydration never publishes over a kind a command pinned instead (thread_override_slots.kind_is_pinned), so a pinned thread's next message cannot hydrate over its override. An evicted thread also lost its other override: a project or agent restored from the conversation metadata. Both commands now run _hydrate_thread_overrides just before pin(), with no await between them, so an evicted thread reads both overrides back, keeping any value set meanwhile. If that hydration gives up, the thread stays unhydrated, and its next message reads the other override back. The refusals this bound adds now write a SEL command record, as the other `!project` refusals do: agent_denied_too_long (with the length, never the name), agent_denied_capacity and project_denied_capacity. A reset could be undone by eviction. Hydration prefers a recorded execution template over the `agent` field `!ta off` clears, and reads the old value back when the clear's metadata write fails. So after the clear, `!ta off` and `!project off` read back what hydration would restore (thread_override_slots.read_back). They read the line through get_metadata_status, because get_metadata answers a line the store could not read with empty metadata, the same as one with nothing set. A thread that would get a value back, or whose line cannot be read back, keeps a pin with nothing set, so it is never evicted in this process, as on main. Any other thread releases its pin. When every pin is taken, the reset applies and the reply says the recorded value can come back. Either way the command first hydrates the thread again and clears its value again, with no await after the hydration. A thread evicted while the command awaited its write can be hydrated by its own next message from a read made before the write landed; the release branch then replied that the reset was done while the old value stayed live. Eviction also lets such a read publish later: one still in flight after the command replied, for a thread evicted again by then, found it outside the guard and published the old value. Hydration now publishes nothing it read before a newer `!ta` or `!project` metadata write. The commands count their writes in one process-wide integer (thread_override_slots.note_command_write), each as soon as its write returns; hydration compares the count before and after its read and reads again when it moved, at most HYDRATION_READ_ATTEMPTS (3) times. If every read overlapped a write, it publishes nothing and leaves the thread unhydrated, so the thread's next message hydrates it. unpin() put any thread back in the evictable order, so `!ta off` on a thread evicted while the command awaited, one that held no pin, tracked it again without evicting anything: THREAD_SLOTS + 1 threads. It now re-tracks a thread only when it releases that thread's last pin, and otherwise only touches a tracked one.
97a10d4 to
b5ce892
Compare
|
span=25bf41b83f59 (GPT 6.1 F1, span=9c152b799da0 (GPT 6.1 F2, Line references in Not a goal moved with later revisions: the pin refusal it cites as |
bolichen97
left a comment
There was a problem hiding this comment.
Approved at head b5ce892: PR Readiness is green, no open change requests, and the review lanes passed. (AI-assisted review at bolichen97's request.)
Problem / Motivation
Goal: the Slack handler's per-thread override state stops growing with every thread the bot has ever heard from, each value it keeps is bounded, and a thread that is evicted gets its
!taand!projectchoices back on its next message.src/kiro_crew/slack/handler.pykeeps three module-level containers for per-thread state:_hydrated_sessions(the guard that keeps_hydrate_thread_overridesfrom re-reading a thread's metadata on every message),_thread_agents(the!tachoice) and_thread_projects(the!projectchoice). Every inbound message for a new thread adds its session key to the guard, and to the two maps when the thread's conversation metadata names an agent or a project (src/kiro_crew/slack/handler_runtime/inbound.py,_hydrate_thread_overrides). Only!ta offand!project offremove an entry from the two maps, and nothing removes a key from the guard. The entries outlive the thread's session: the idle sweep ends the session, and the three entries stay for the life of the process.Why it matters
A gateway that answers many threads holds one entry per thread forever, about 125 bytes in the guard and more for each override. A busy workspace adds thousands of threads a week, and a long-running gateway never gives the memory back until it restarts.
Not a goal
_pending_approvalsand_linked_approvalsare not touched; they are in-flight registries with their own removal.!taand!projectpersist or to the hydration order of the inbound paths.!taor!projectchoice in more than 4,095 threads in one process. Pinned threads count inside the bound, at mostMAX_PINNED_THREADS=THREAD_SLOTS - 1of them, so once 4,095 threads are pinned, a command that would pin one more thread is refused with a reply that names the limit and says how to free a slot (!ta offor!project offin a thread), and it sets nothing (thread_override_slots.py:140-158,handler_runtime/commands.py:724,:842). A restart also frees them, and hydration then reads the persisted choices back.!taand!projectrefuse one with a reply that names its length and the limit, and hydration skips one with a warning. On Linux and macOS no directory path that long can be opened, so in practice this refuses a Windows long path, or a project agent whose spec declares a name that long.What changed
read_back(slack/thread_override_slots.py:258), which reads it throughget_metadata_status, becauseget_metadataanswers a line the store could not read with empty metadata, the same as a thread with nothing set, and an unreadable read returns before the thread is marked (slack/handler_runtime/inbound.py:253).pin(slack/thread_override_slots.py:194) no longer adds the thread to the hydration guard, and hydration never publishes over a kind a command pinned (kind_is_pinned,slack/thread_override_slots.py:105, checked atslack/handler_runtime/inbound.py:274-279). So a pinned thread whose hydration gave up, or could not read the line, stays out of the guard, and its next message reads its other override back. The cost: a thread whose metadata line stays unreadable is read again on each of its messages. A reset releases its pin only after a read that the store reports as readable:!ta offand!project offread back through the sameread_back, and an unreadable read-back keeps the pin (slack/handler_runtime/commands.py:711,:856). Tests: seven new cases fail on the previous head 97a10d4 and pass here, 3 of 3; the file, 35 passed, 3 of 3; the 17 neighbour files, 1,109 passed.src/kiro_crew/slack/thread_override_slots.py: the three containers are held toTHREAD_SLOTSthreads together, in the least-recently-used shape already used for the thread-replies watermark (_last_turninsrc/kiro_crew/slack/thread_replies.py).THREAD_SLOTSis that watermark's own bound,_WATERMARK_SLOTS(4096), imported rather than written a second time, so the two cannot drift (thread_override_slots.py:46). Past the bound, the oldest thread loses its guard entry and both override entries at once. Its next message finds no guard entry, so_hydrate_thread_overridesreads the overrides back from the conversation metadata as it does for a thread it has never seen._hydrate_thread_overridesrecords a thread throughnote_hydratedinstead of_hydrated_sessions.add, and a repeat message for a tracked thread moves it to the fresh end (touch)._get_agent_for_session, the reader the background paths use (monitor ticks, autonudge and subagent notices insrc/kiro_crew/slack/gateway.py), also touches the thread, so a thread with background activity stays resident.!ta <name>or!project <path>in this process is pinned and never evicted while it is set (pinin_bang_thread_agentand_bang_project;!ta offand!project offrelease it, unless hydration would read the cleared value back, below). Hydration cannot be relied on to read these choices back: the command's metadata write is skipped when there is no conversation log and only logged at debug when it fails, and hydration prefers a recorded execution template over theagentfield. An override that came from the metadata is evictable, because the metadata restores it. Pinned threads count insideTHREAD_SLOTS: each hydration evicts the oldest unpinned threads until pinned and unpinned together fit (note_hydrated,slack/thread_override_slots.py:176), andpinrefuses a new thread onceMAX_PINNED_THREADS(THREAD_SLOTS - 1,slack/thread_override_slots.py:51) are pinned (slack/thread_override_slots.py:214). That keeps one slot evictable, so hydrating a new thread never evicts that same thread and then writes its rows outside every count. Both commands callpinbefore they write the override, so a refused command writes no row (slack/handler_runtime/commands.py:765andslack/handler_runtime/commands.py:926). Between a message's hydration and the pin, other threads hydrating can evict the command's own thread, its other override included: the inbound path awaits after it hydrates (for exampleadmit_inbound_callback,slack/transport_dispatch.py:350), the dispatcher,_handle_slash_command, awaits the deprecation notice for!ta(slack/handler_runtime/commands.py:95), and!taawaits its agent lookup (slack/handler_runtime/commands.py:733). So each command runs_hydrate_thread_overridesagain just before it pins, with no await in between (slack/handler_runtime/commands.py:762andslack/handler_runtime/commands.py:923). For a thread still in the guard it returns at once. An evicted thread reads both overrides back from the conversation metadata, and a value set while that read ran is kept, the rule every hydration applies (slack/handler_runtime/inbound.py:251-279). Thenpincalls_evict_over_bound(slack/thread_override_slots.py:218), the loopnote_hydratedalso calls (slack/thread_override_slots.py:191), which evicts the oldest unpinned threads from the guard and both maps until the pins and the rest fitTHREAD_SLOTS(_evict_over_bound,slack/thread_override_slots.py:161).pindoes not put the thread in the hydration guard: only a hydration that read the metadata does. If the hydration just before the pin gave up, the thread stays out of the guard and its next message reads its other override back, and hydration never publishes over the kind the command pinned (kind_is_pinned), which keeps that message, and any later one, from hydrating over the override. That runs before the command writes its row (_thread_agents,slack/handler_runtime/commands.py:779;_thread_projects,slack/handler_runtime/commands.py:940). The test reset intest/conftest.pynow also clears the module's_orderand_pinned, so an existing!tatest no longer leaves a pin for the rest of its worker.agentfield!ta offclears (messaging/session_resume.py:99-104), and reads the old value back when the clear's metadata write failed. So once it has cleared the override, and after the write,!ta offreads back what hydration would restore throughread_back(slack/thread_override_slots.py:258, called atslack/handler_runtime/commands.py:701;!project off,:848). It reads the metadata line once throughget_metadata_status, becauseget_metadataanswers a line the store could not read with empty metadata, the same as a thread with nothing set, and hydration's reader,_read_thread_overrides, resolves that line. Then, on either branch, the thread hydrates again and its value is cleared again, with no await after the hydration (:708-709,:853-854). A thread evicted while the command awaited its write can meanwhile be hydrated by its own next message from a read made before the write landed, and the value that read published is cleared before the command replies; a read of that kind that is still in flight when the command replies is refused by the hydration rule below. A thread that would get a value back, or whose metadata line cannot be read back, is then pinned with nothing set (:712,:857), so it is never evicted in this process, the same as on origin/main, where nothing evicts it before a restart, and no hydration publishes that kind for it again. Any other thread releases its pin (:716,:861). When every pin is taken, the reset still applies and the reply says it lasts until the thread is evicted (unheld_reset_text,slack/thread_override_slots.py:230).unpinreturns a thread to the evictable order only when it releases that thread's last pin, and otherwise only touches a tracked thread (slack/thread_override_slots.py:287-292). So a reset of a thread evicted while its command awaited, one that held no pin, does not track that thread again outside the bound.!taor!projectmetadata write. Eviction is what makes this needed: on origin/main a thread stays in the hydration guard for the life of the process, so a read that began before a command's write finds the guard set and publishes nothing. Here, a read still in flight after!ta offreplied could find its thread evicted again by then, and publish the old value. The commands count their metadata writes in one process-wide integer,_command_writes(slack/thread_override_slots.py:73), so there is nothing to bound. Each command bumps it as soon as its write's worker returns or raises, before it awaits anything else (note_command_write;!ta offslack/handler_runtime/commands.py:696,!ta <name>:789,!project off:844,!project <path>:949)._hydrate_thread_overridestakes the count before each read and compares it after; when it moved, it reads again, at mostHYDRATION_READ_ATTEMPTS(3) times (slack/thread_override_slots.py:78,slack/handler_runtime/inbound.py:244-263). If every read overlapped a write, it publishes nothing, does not mark the thread hydrated, and logs a warning, so the thread's next message hydrates it and the message whose hydration gave up runs with the defaults (slack/handler_runtime/inbound.py:252-259). The count is process-wide, so a write in another thread also causes a read again; giving up needs a write to land during each of three consecutive reads. If that happens to the hydration a command runs just before its pin, the thread stays unhydrated after the pin, so its next message reads its other override back. There is no bump before the write. A read can only be stale if it began before the write landed, so the count it took precedes the bump: if it is back after the bump, it reads again, and if it is back before, it publishes before the command's own hydrate-and-clear above removes the value. A bump before the write would not help: a read begun after it but before the write landed takes the bumped count and is just as stale. The!ta <name>and!project <path>threads are already pinned when they write, so hydration never publishes over that kind and a stale read cannot reach it; their bump keeps the rule from depending on the pin.OVERRIDE_VALUE_MAX_CHARS(4096,thread_override_slots.py:61), so the bound holds whichever path wrote it. With the count bound, the maps keep at most 4096 threads × 2 values × 4096 characters._read_thread_overridesreturns:_hydrate_thread_overrideswrites it athandler_runtime/inbound.py:277and:279, and eviction now sends every evicted thread back through it. That read-back of the conversation metadata is bounded by nothing: origin/main keepsmeta["project"]after onlyis_sensitive_path(inbound.py:203at 843abe3), and keeps theagentfield, or whatsession_agent_from_metadatareturns, as it is (:195). Each value is now kept only when it is text within the limit, throughbounded_override(thread_override_slots.py:126, called atinbound.py:198and:202). A longer or non-text value is skipped with one warning that names its length or type, never its text, and the thread uses the default for that kind. A list-valued project, which made main's hydration raise fromis_sensitive_path(executed:TypeError: expected str, bytes or os.PathLike object, not list), is skipped the same way.setter_refusal,thread_override_slots.py:110).!tachecks the name it resolved (handler_runtime/commands.py:744): a project agent's name is whatever its spec file declares, and nothing else limits it (_resolve_agent_namereturnsproject_agent_name(spec),slack/handler.py:808).!projectchecks the realpath it would keep (slack/handler_runtime/commands.py:884), before the sensitive-path and directory checks, so no audit record carries an over-limit path; it recordsproject_denied_too_longwith the length instead.!projectrefusals do:agent_denied_too_longwith the name's length, never the name (slack/handler_runtime/commands.py:754), and, whenpinrefuses,agent_denied_capacity(slack/handler_runtime/commands.py:774) orproject_denied_capacity(slack/handler_runtime/commands.py:935). A capacity record names the agent or path, which already passed the per-value limit, asagent_switchandproject_denied_invaliddo.handle_messageandmaybe_handle_keyword_commandinhandler.py, the two sites insrc/kiro_crew/slack/transport_dispatch.py, and the!ta,!projectand!agentcommands inhandler_runtime/commands.py) runs after that turn's_hydrate_thread_overrides, so an evicted entry is rehydrated before it is read. The one reader outside the inbound path,_get_agent_for_session, falls back to the default agent for a thread that was evicted before its background turn, the same fallback it takes for every thread after a gateway restart today.test/test_slack_thread_override_slots.py, thirty-five tests (one of them run for three refusals, one for both resets, one for a reset whose write lands or fails) that drive the real_hydrate_thread_overrideswith a fake conversation log as the one seam: with every pinpinadmits taken (4,095), 50 new threads hydrate and every thread with an override row is still counted, and the newest keeps its agent; the bound holds; an evicted thread restores both overrides on its next message; a recently active thread is kept over an idle one; one thread hydrated many times adds one entry; a command-set override survives any number of other threads and is released by its reset; 100 pinned threads count inside the bound; through the real_bang_thread_agent, with every slot but two pinned, a thread that two other threads' hydration evicts while!taawaits its agent lookup is pinned with every container still withinTHREAD_SLOTSand its hydration guard back; and, through the real_bang_thread_agentwith only the agent lookup replaced,!tain a new thread when every slot is pinned is refused out loud and sets nothing; a 16 MiB agent name and project path in the metadata are not kept, each logged once by length; a value of exactly 4096 characters is kept and one of 4097 is not; a project that is not text is not kept while the agent beside it is; through the real_bang_thread_agentand_resolve_agent_name,!tanaming a project agent whose spec declares a 4,097-character name is refused out loud, pins nothing and sets nothing, while one declaring 4,096 characters is set; and through the real_bang_project, a directory path one character over the limit is refused out loud with aproject_denied_too_longaudit record that carries no path, while one at the limit is set (no directory path over 4,096 characters can be opened on Linux, so that test lowers the limit to a length real directories undertmp_pathstraddle); through the real_bang_thread_agent, a thread whose project came from its conversation metadata, evicted by two other threads while!taawaits its agent lookup, keeps that project, and!projectthen reports it; through the real_bang_project, a thread evicted after its message hydrated keeps the agent its metadata names; and each refusal the bound adds (!tawith a name one character over the limit,!taand!projectwith every pin taken) writes one SEL command record with its outcome, the over-limit one with the name's length and never the name; through the real_bang_thread_agent,!ta offon a thread whose metadata records an execution template stays reset after 4,096 other threads hydrate and the thread's next message hydrates again, with its project kept; through the real_bang_project,!project offwhose metadata write fails stays cleared the same way; and!ta offfor a thread evicted before it ran, with every pin but one taken, leaves the pinned and evictable threads withinTHREAD_SLOTS; and, with every pin but one taken and the command's metadata write held open,!ta offand!project offstay reset when another thread's message evicts the thread and the thread's own next message hydrates it again from a read made before the write landed, whether that hydration publishes before the command reads the metadata back or after the command has replied;!ta offand!project offstay reset when such a read is still in flight after the command replied and the thread is evicted again before it publishes; and, through the real_bang_thread_agentrun for another thread during each read, a hydration whose every read a command's metadata write overlaps publishes nothing and leaves the thread unhydrated, and the thread's next message hydrates it; through the real_bang_thread_agent, an evicted thread whose hydration before the pin gives up (a!ta offin another thread writes metadata during each of its reads) is left unhydrated by the pin, so its next message reads its project back, and that hydration does not publish the recorded template over the agent!taset; and!ta offand!project offwhose write fails and whose read-back meets a line the store cannot read keep their pin, and eviction after the store recovers does not bring the value back.Backwards compatibility
Compatible: a gateway that sees up to 4096 threads behaves as before, entry for entry, and so do
!taand!projectuntil 4,095 threads have a choice set in one process. An override value of up to 4096 characters is restored from the metadata as before. A reset made while the store cannot read the thread's metadata line keeps its pin, as a reset that hydration would undo does, so it holds a slot until a later reset reads the line, or until a restart. A thread whose metadata line the store cannot read is not marked hydrated, so each of its messages reads the line again until a read succeeds; on main its first message marks it hydrated with no overrides.Removes nothing: no setting, field, command or event is removed.
Reader: nothing visible changes in Slack. An evicted thread's next message costs one metadata read, as its first message did.
Tests
Red on origin/main 843abe3 (a throwaway detached worktree with only the new test file copied in), 3 of 3 serial runs, each:
What each failure pins: the first three, the guard set and both maps grow with every thread, and nothing evicts the oldest or keeps the recently active one; the next four (
the bound does not exist), the pin and refusal rules; the next three, a 16 MiB value, a value one character over the limit and a list-valued project are kept from the metadata (the last makes main's hydration raise); the last two,!takeeps an over-limit project agent name, and!projecthas no limit to check against. The passing test is the control: one thread hydrated many times adds one entry. On origin/main 7bb9d57, the base of this revision, the whole file (35 tests) gives 34 failed, 1 passed, 3 of 3; the pass is the same control.Each guard also has a test that fails without it. This file, run against earlier versions of this change that lacked one or more guards, 3 of 3 serial runs each:
4196 threads kept with 100 pinned: pins are outside the bound; the refusal test stops atpin, which answers nothing in that version; the five per-value and setter tests fail as on maina 16 MiB agent name was kept; the 4,097-character value kept; the list-valued project raisingTypeError;an over-limit agent name was kept; the!projectsetter test stops at the missing limit constant (AttributeError)an over-limit agent name was kept;an over-limit project path was kepta thread past the bound got an override row;4146 threads keep override rows with 4096 slots and 4096 pinned (50 untracked by the bound)The last row is the reason for
MAX_PINNED_THREADS: with every slot pinned, hydration evicted each new thread it had just added and still wrote that thread's rows.The new file's autouse
_homefixture pinsKIROCREW_HOMEandKIROCREW_WORKSPACEthrough the isolation floor's own_floor_monkeypatch, so a test'smonkeypatch.undo()cannot lift the pins.The red block above is the first commit's 13 tests on main. The second commit's fifteen tests, run against the first commit 9195b87 (a throwaway detached worktree with only this test file copied in), 3 of 3 serial runs, each:
What each failure pins: the first,
pinadmitted a thread evicted during its command without evicting another, so the containers heldTHREAD_SLOTS + 1threads; the next two, a thread evicted before its pin lost the other override its conversation metadata names,!taits project and!projectits agent; the next three, the refusals the bound adds wrote no SEL record; the next two, eviction undid a reset that hydration reads back,!ta offon a thread that records an execution template and!project offwhose write failed; the ninth,unpintracked a thread that!ta offreached after it was evicted, with no pin to release, outside the bound; the next three,!ta offand!project offreplied that the reset was done while the value that a hydration published from a read made before the write landed stayed live; the next two, a read made before the write and still in flight when the command replied published the old value after the thread was evicted again; the last, hydration published a read that a command's metadata write overlapped. The five tests after the first also fail, with the same messages, 3 of 3, on this branch's previous head 278bddc, which had the first fix but not the other two. The nine after the refusal records fail, with the same messages, 3 of 3, on the previous head d42d0f6 (9 failed, 19 passed). Each of the two mechanisms behind the last six has its own failing test, 3 of 3 serial runs each. On this branch without the write count (no bump, no read again) the last three fail (3 failed, 25 passed). With only the hydration before the release taken out, the two race tests whose hydration publishes before the command reads the metadata back fail (2 failed, 26 passed); the one whose read publishes after the reply passes there, because the count already covers it. Adding a bump before the write to that variant changes nothing (2 failed, 26 passed). The two!tarace tests drive the real_bang_thread_agent: the agent lookup it awaits runs in a worker thread, which hydrates two other threads on the loop before it answers, so the command's own thread is evicted beforepinadmits it. The three reset race tests hold the command's metadata write on a gate in its worker thread, hydrate another thread and then this one on the loop, and only then open the gate; in the second, this thread's read waits on a second gate until the command has replied. The two later-eviction tests do the same as the second and, after the reply, hydrate a third thread before they open the read's gate. The give-up test's fake log runs a real!ta offfor another thread on the loop, throughrun_coroutine_threadsafe, from inside each read of the thread under test. No test sleeps: each waits on events or futures, within a 10-second bound.The seven newest cases, run against the previous head 97a10d4 (rebased onto 7bb9d57 as 645315d969, same patch-id; a throwaway detached worktree with only this test file copied in), 3 of 3 serial runs, each:
The last four are this revision's. In the two
while_its_metadata_cannot_be_readcases the store cannot read the thread's line while!ta <name>or!ta offruns on an evicted thread and then recovers: the thread's next message must read its project back. With hydration reading throughget_metadataagain, both fail witha hydration that could not read the metadata marked the thread hydrated (True) .... The twopromises_nothingcases take a reset that cannot be held or read back: when the clear's write lands, the old agent never comes back; when it fails on a thread evicted before the reset, it comes back with the thread's next message, with no eviction. With the reply's former wording, that the value "comes back if it is evicted", the first fails withthe reply ties the agent's return to an eviction, but here it never came back. With an unreadable hydration read marked hydrated again (one change, on this branch), the twowhile_its_metadata_cannot_be_readcases and the secondpromises_nothingcase fail, 3 failed, 32 passed, 3 of 3.Each guard has its own failing case, 3 of 3 serial runs each, on this branch with one change: with
pinmarking the thread hydrated again, the first case fails as above (1 failed, 2 passed); without hydration's per-kind skip, it fails at the agent,the thread's next message hydrated the recorded template over the agent !ta set: 'reviewer'(1 failed, 2 passed); with the resets reading back through_read_thread_overridesandget_metadataagain, both reset cases fail as above (2 failed, 1 passed). The first case runs a real!ta offfor another thread on the loop, throughrun_coroutine_threadsafe, from inside each of the three reads; the store in the other two raises on a write and answers a read as the real conversation log answers an unreadable line,{}fromget_metadataand({}, False)fromget_metadata_status.Green on the branch (on origin/main 7bb9d57): 35 passed, in three serial runs. The control (one thread hydrated many times adds one entry) passes on both sides.
test/test_slack_thread_override_slots.pyfirst, then the 16 Slack handler test files that importthread_override_slots,handler_runtime.inboundorhandler_runtime.commands(includingtest_thread_override_persistence.pyand the existing!ta offand!project offtests), run serially: 1,109 passed. Their conversation-log fakes now answerget_metadata_statusas the real log does, with the same metadata reported readable. Three golden transcripts intest/test_slack_handler_composition_contract.pywere re-recorded. Their fake now answersget_metadata_status, as the real log does (history.py:3913), so the naming turn's record read (messaging/auto_title.py:281, reached fromslack/handler.py:2913and:3243) reads the fake's metadata. On the old fake that read raised AttributeError and took its unknown-record branch, which the real log never does. The transcripts gain only those reads: two inlinked_dashboard_session, two inpinned_answer, one inturn_with_history. mypy, flake8, isort, black,scripts/check_black_formatting.py,scripts/check_subprocess_encoding.py,scripts/check_agent_sdk_boundary.py,scripts/docs-lint.shand the comment-history check withCOMMENT_HISTORY_BASE_REF=origin/mainexit 0.The Not a goal section's line numbers are the first revision's. On this branch the two
pincalls are atslack/handler_runtime/commands.py:765and:926, andpin's refusal check is atslack/thread_override_slots.py:214.Manual verification
Not run against a live Slack workspace.
Related Issues
None found for the growth. #8715 (open) is about races in the agent-override persistence and does not change with this PR. Open PRs that edit
handler_runtime/inbound.pyorhandler_runtime/commands.py, checked by hunk: #15273 edits the import block andmaybe_route_linked_threadininbound.py; #14089 edits_bang_stopand #11398 edits_handle_slash_commandincommands.py. None of them touches_hydrate_thread_overrides,_bang_thread_agentor_bang_project, where this change lives.Pattern harvest
Rule candidate: review-prompt
Pattern: a module-level map keyed by a conversation identifier with no eviction path, or a guard set that gates a re-read but is not evicted together with the state it guards.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)