feat: dl --refresh-stale recreates the stale workspaces that are idle

CHANGELOG.md

@@ -27,6 +27,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   line to print, so it says that the recreate ends the agents and that `claude --resume`
   in the new container picks a conversation up again. No failure here refuses the
   recreate. See docs/cli.md, "A recreate starts the agents it ends again".
+- **`dl --refresh-stale` recreates the stale workspaces that are idle** (#673). It runs
+  the recreate above, with no attach, on each workspace `dl --ls` calls stale, so it can
+  run after the daily pull. It skips a workspace, and says why, when another `dl` is
+  launching it, a build runs in it, an agent (`claude`, `codex` or `gemini`) runs a
+  command in it or is not idle, or it holds an agent session `dl` cannot start again
+  (outside herdr, any agent session). It prints its plan first and a count last, checks
+  each workspace again just before its recreate, and exits 1 when a recreate failed.
+  See docs/cli.md, "`--refresh-stale`: recreate the stale workspaces that are
+  idle".
 
 ## [0.60.1] - 2026-10-16
 

README.md

@@ -316,6 +316,7 @@ instead. [docs/cli.md](docs/cli.md) has the full `--rm` contract, including whic
 | `dl --prune` | Remove the clone directories no workspace opens any more, the agent git worktrees inside the clones it keeps, and the volumes and launch locks of workspaces devpod has forgotten |
 | `dl --reconcile` | Re-point workspaces whose recorded source folder went missing. Deletes nothing |
 | `dl --purge` | Remove devlaunch's own workspaces and caches |
+| `dl --refresh-stale` | Recreate each workspace that runs an older image than its reference, when its agents are idle and nothing builds in it. Starts the agents again and skips the rest, saying why. See [docs/cli.md](docs/cli.md) |
 | `dl --install` | Install shell completions, and the `dl-herdr-shell` name a herdr pane opens through |
 | `dl --herdr-setup` | Install and configure the Herdr pane shell |
 | `dl --herdr-env set KEY=VALUE` | Save a variable for new host shells in this Herdr workspace |

docs/cli.md

@@ -930,11 +930,19 @@ to bring each agent back (devlaunch#673):
 2. After the `up` succeeds, `dl` types each pane's line back into that pane with
    `herdr pane run`. herdr refuses a pane until the old session in it has exited and its
    shell is back, so `dl` asks again every half second, for up to ten seconds per pane.
-3. Then `dl` attaches to the new container as a recreate always did.
+3. `dl` reports each line it typed to herdr again, as that pane's saved line, and reads
+   herdr's saved session to check that herdr kept it. The `dl` that the line runs saves
+   no line of its own, and herdr drops the old line when the old agent ends. Without
+   this step, the next recreate finds no line and ends the agent. herdr refuses the
+   report until it sees the new agent, so `dl` asks again every second, for up to
+   twenty seconds. All panes are asked at the same time.
+4. Then `dl` attaches to the new container as a recreate always did.
 
 Each line is the one herdr saved for the pane, for example
 `dl <ws> -- ... claude ... --remote-control=<ws> --resume <uuid>`. It holds the session
-the pane is in now, because the container hook reports a new id after `/clear`.
+the pane is in now, because the container hook reports a new id after `/clear`. An
+agent started again from a line has no container hook, so after a `/clear` in it the
+saved line still names the session it was started with.
 
 Nothing here can refuse the recreate. Each failure is one line on stderr, and the
 recreate goes ahead:
@@ -948,10 +956,82 @@ recreate goes ahead:
   fails, gets the line printed, to run there by hand.
 - The pane that `dl` itself runs in is never typed into, because its foreground is that
   `dl`. Its line is printed instead.
+- A pane whose line herdr did not keep gets its agent back, and `dl` says that the next
+  recreate ends it.
+
+#### A workspace on an image that devpod built
+
+With `"features"` (or anything else that makes devpod build), the container does not
+run the reference itself but an image devpod built on it, tagged
+`<prefix>:devpod-<hash>`. devpod names that tag by a hash of the config, not of the
+reference, and uses a tag it finds. So a recreate alone builds nothing, and the
+workspace stays on the older image.
+
+When the workspace is stale (as `dl --ls` reports it), and its container runs a
+`:devpod-` tag that differs from the reference, `dl <ws> recreate` first removes that
+tag with `docker image rm --force <tag>`. devpod then builds the image again from the
+newer reference. `--force` is necessary because the old container still uses the
+image; with a tag, it removes only the tag, and the old container keeps its image until
+the `up` replaces it. `dl` removes no tag that devpod did not make. If the removal
+fails, `dl` says so and the recreate goes ahead on the older image.
 
 `reset` and `code` do none of this. A `reset` drops the volumes, and `code` keeps the
 container when nothing changed.
 
+### `--refresh-stale`: recreate the stale workspaces that are idle
+
+`dl --ls` names each workspace whose container runs an older image than its reference
+now names. `dl --refresh-stale` recreates those workspaces, but only the ones where a
+recreate ends nothing that is in progress, and starts their agents again as
+`dl <ws> recreate` does (devlaunch#673). It asks no question and attaches to nothing,
+so it can run after the daily pull:
+
+```bash
+docker pull ghcr.io/owner/image:latest && dl --refresh-stale
+```
+
+It prints one line for each stale workspace before it starts: `Refreshing <ws>`, or
+`Skipping <ws>` with the reason. Then it takes each workspace it did not skip in turn.
+It checks the workspace again just before its recreate, because the recreates before
+it can take minutes. If the workspace is now to be skipped, it prints
+`Skipping <ws> after all` with the reason and leaves it alone. Otherwise it recreates
+it, with the recreate's own lines. It ends with a count. A recreate that fails does not
+stop the others. The exit status is 1 when a recreate failed, and 0 otherwise. A skip
+is not a failure.
+
+`dl` skips a stale workspace when any of these is true, and names the first that is:
+
+- Another `dl` holds the workspace's launch lock.
+- The container is stopped. A recreate would also start it, which a refresh does not
+  do. `dl <ws> recreate` starts it on the new image.
+- `dl` cannot read the processes in the container. It reads them with one
+  `docker exec` as root.
+- A build runs in the container: a process named `bazel`, `bazelisk`, `cargo`, `rustc`,
+  `cc1`, `cc1plus`, `make`, `gmake`, `ninja`, `cmake`, `colcon`, `gcc`, `g++`,
+  `clang`, `clang++`, `ld`, `ld.lld`, `ld.gold` or `collect2`. `pixi` is not on the
+  list: an open `pixi shell` is not a build, and a build that `pixi run` starts runs one
+  of the programs above.
+- An agent process has a shell as its child. An agent is any of the agents `dl`
+  knows by name: `claude`, `codex` and `gemini`, run directly or by `node`. The agents
+  run their tool calls and background tasks through a shell, and Claude starts its MCP
+  servers directly, so a shell under an agent is work in progress and an MCP server
+  is not.
+- An agent runs in the container and `dl` is not in a herdr pane, so it cannot see the
+  sessions to start them again. A container with no agent process needs no herdr.
+- herdr does not answer, a pane holds a live agent and no saved line, or an agent is
+  not `idle` or `done` (it is `working`, or `blocked` on a permission prompt).
+- More agent sessions run in the container than herdr's panes hold, counting agents
+  of every kind on both sides. A session that was started from a terminal outside
+  herdr has no line to start it again.
+
+A workspace on an image that devpod built is refreshed as `dl <ws> recreate` does it,
+by removing devpod's tag first. If the removal fails, the refresh does not recreate
+that workspace, because it would stay stale and be recreated again on every run. It
+counts as a failed recreate.
+
+The second check is taken a moment before the recreate. An agent that starts work in
+that moment ends with the recreate, the same as with `dl <ws> recreate`.
+
 ## `kill`: the workspace that will not answer
 
 `dl <ws> stop` asks devpod to stop a workspace, and it is the right thing to type

docs/development.md

@@ -69,7 +69,7 @@ once under the `api` section and once under the module that owns them.
 
 **What it still does not reach, and it is not one type.** A type `api` never re-exports but a
 promised signature hands back is reachable from outside and classified as binary surface. Counted
-on the checked-in files rather than guessed at, that is **40 types owning close to six hundred rows** in
+on the checked-in files rather than guessed at, that is **41 types owning close to six hundred rows** in
 `public-api.rest.txt`, and the command that lists them needs no toolchain:
 
 **A falling count is not automatically a win, and it is worth knowing which kind you are looking

docs/workspace-tools.md

@@ -1478,6 +1478,10 @@ pane's saved directory. `dl` is what tells it the line, for an agent `aid` start
 dl <workspace id> -- CLAUDE_CODE_DISABLE_TERMINAL_TITLE=1 IS_SANDBOX=1 claude --dangerously-skip-permissions --remote-control=<workspace id> --resume <session id>
 ```
 
+The line starts with the `dl` that was run, by its file name, so a pane started with the
+dev build `dl-next` (or `aid-next`, or `dla-next`) comes back under `dl-next` and not under
+the released `dl`.
+
 So after a herdr server restart, or a reboot, every such pane comes back into its own
 conversation with no `aid resume` and no pickers. The workspace starts if it is stopped.
 Measured against herdr 0.9.2: about 20 seconds from the server start to the claude

rust/devlaunch-core/completions/dl.bash

@@ -73,7 +73,7 @@ _dl_completion() {
     # The retired spellings (--stop, --autorm) are absent by rule rather than by
     # hand: the grammar marks them `hide = true`, and the test drops every hidden
     # flag, so a spelling this build only still answers for is never offered.
-    local global_opts="--ls --install --refresh --prune --reconcile --purge --herdr-shell --herdr-setup --herdr-env --herdr-workspace --rm --devcontainer --claude-profile --claude-profiles --help -h --version"
+    local global_opts="--ls --install --refresh --refresh-stale --prune --reconcile --purge --herdr-shell --herdr-setup --herdr-env --herdr-workspace --rm --devcontainer --claude-profile --claude-profiles --help -h --version"
     if [[ "$cmd" == aid ]]; then
         global_opts="--claude --codex --gemini --model --effort --devcontainer --claude-profile --help -h --version"
     fi

rust/devlaunch-core/public-api.api.txt

@@ -36,6 +36,10 @@ impl core::marker::StructuralPartialEq for devlaunch_core::flows::lifecycle::Ins
 pub enum devlaunch_core::api::LaunchNotice
 pub devlaunch_core::api::LaunchNotice::AgentSessionInThisPane
 pub devlaunch_core::api::LaunchNotice::AgentSessionInThisPane::line: alloc::string::String
+pub devlaunch_core::api::LaunchNotice::AgentSessionLineNotKept
+pub devlaunch_core::api::LaunchNotice::AgentSessionLineNotKept::pane_id: alloc::string::String
+pub devlaunch_core::api::LaunchNotice::AgentSessionLineUnchecked
+pub devlaunch_core::api::LaunchNotice::AgentSessionLineUnchecked::pane_id: alloc::string::String
 pub devlaunch_core::api::LaunchNotice::AgentSessionNotRestarted
 pub devlaunch_core::api::LaunchNotice::AgentSessionNotRestarted::line: alloc::string::String
 pub devlaunch_core::api::LaunchNotice::AgentSessionNotRestarted::pane_id: alloc::string::String
@@ -72,6 +76,10 @@ pub devlaunch_core::api::LaunchNotice::CodexStageMissing
 pub devlaunch_core::api::LaunchNotice::CodexStageMissing::workspace_id: alloc::string::String
 pub devlaunch_core::api::LaunchNotice::CreateNeverFinished
 pub devlaunch_core::api::LaunchNotice::CreateNeverFinished::workspace_id: alloc::string::String
+pub devlaunch_core::api::LaunchNotice::DerivedImageKept(devlaunch_core::flows::launch::DerivedImageKept)
+pub devlaunch_core::api::LaunchNotice::DerivedImageRemoved
+pub devlaunch_core::api::LaunchNotice::DerivedImageRemoved::image: alloc::string::String
+pub devlaunch_core::api::LaunchNotice::DerivedImageRemoved::workspace_id: alloc::string::String
 pub devlaunch_core::api::LaunchNotice::DevcontainerIgnoredRunning
 pub devlaunch_core::api::LaunchNotice::DevcontainerIgnoredRunning::spec: alloc::string::String
 pub devlaunch_core::api::LaunchNotice::DevcontainerIgnoredRunning::workspace_id: alloc::string::String
@@ -142,6 +150,7 @@ pub devlaunch_core::api::LaunchVerb::Attach::command: core::option::Option<devla
 pub devlaunch_core::api::LaunchVerb::Code
 pub devlaunch_core::api::LaunchVerb::Dotfiles
 pub devlaunch_core::api::LaunchVerb::Recreate
+pub devlaunch_core::api::LaunchVerb::RecreateUnattended
 pub devlaunch_core::api::LaunchVerb::Reset
 pub devlaunch_core::api::LaunchVerb::Restart
 pub devlaunch_core::api::LaunchVerb::Up
@@ -611,6 +620,10 @@ impl core::marker::StructuralPartialEq for devlaunch_core::flows::launch::ColdRe
 pub enum devlaunch_core::flows::launch::LaunchNotice
 pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionInThisPane
 pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionInThisPane::line: alloc::string::String
+pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionLineNotKept
+pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionLineNotKept::pane_id: alloc::string::String
+pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionLineUnchecked
+pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionLineUnchecked::pane_id: alloc::string::String
 pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionNotRestarted
 pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionNotRestarted::line: alloc::string::String
 pub devlaunch_core::flows::launch::LaunchNotice::AgentSessionNotRestarted::pane_id: alloc::string::String
@@ -647,6 +660,10 @@ pub devlaunch_core::flows::launch::LaunchNotice::CodexStageMissing
 pub devlaunch_core::flows::launch::LaunchNotice::CodexStageMissing::workspace_id: alloc::string::String
 pub devlaunch_core::flows::launch::LaunchNotice::CreateNeverFinished
 pub devlaunch_core::flows::launch::LaunchNotice::CreateNeverFinished::workspace_id: alloc::string::String
+pub devlaunch_core::flows::launch::LaunchNotice::DerivedImageKept(devlaunch_core::flows::launch::DerivedImageKept)
+pub devlaunch_core::flows::launch::LaunchNotice::DerivedImageRemoved
+pub devlaunch_core::flows::launch::LaunchNotice::DerivedImageRemoved::image: alloc::string::String
+pub devlaunch_core::flows::launch::LaunchNotice::DerivedImageRemoved::workspace_id: alloc::string::String
 pub devlaunch_core::flows::launch::LaunchNotice::DevcontainerIgnoredRunning
 pub devlaunch_core::flows::launch::LaunchNotice::DevcontainerIgnoredRunning::spec: alloc::string::String
 pub devlaunch_core::flows::launch::LaunchNotice::DevcontainerIgnoredRunning::workspace_id: alloc::string::String
@@ -717,6 +734,7 @@ pub devlaunch_core::flows::launch::LaunchVerb::Attach::command: core::option::Op
 pub devlaunch_core::flows::launch::LaunchVerb::Code
 pub devlaunch_core::flows::launch::LaunchVerb::Dotfiles
 pub devlaunch_core::flows::launch::LaunchVerb::Recreate
+pub devlaunch_core::flows::launch::LaunchVerb::RecreateUnattended
 pub devlaunch_core::flows::launch::LaunchVerb::Reset
 pub devlaunch_core::flows::launch::LaunchVerb::Restart
 pub devlaunch_core::flows::launch::LaunchVerb::Up