Skip to content

feat: dl --refresh-stale recreates the stale workspaces that are idle - #676

Merged
blooop merged 39 commits into
mainfrom
feat/refresh-stale
Oct 16, 2026
Merged

blooop merged 39 commits into
mainfrom
feat/refresh-stale

Conversation

@blooop

@blooop blooop commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Part of #673 — A workspace keeps its old image after a pull, and a recreate ends its agent. This PR is part 3 of 3: dl --refresh-stale recreates each stale workspace whose agents are idle. It can run after the daily pull.

What this changes

  • A new flag, dl --refresh-stale. It takes no workspace, no -y and no --force. It is in the README command table, dl.bash global_opts, the --help examples and docs/cli.md.
  • A new module, flows/refresh_stale.rs, makes the plan. It changes nothing. For each stale workspace from part 1, it gives "refresh", or "skip" with the first rule that fails.
  • A new verb, LaunchVerb::RecreateUnattended, runs the recreate path of part 2 with no attach. Part 2 then starts the agents again in their herdr panes.
  • StaleImage::container() gives the container id that part 1 already reads.
  • The words for each plan line and each skip are in rust/dl/src/render.rs.

Where this fits

dl --refresh-stale
  stale = stale_images(every workspace)                # part 1
  plan  = for each stale workspace: judge(it)          # refresh_stale::plan
  print every plan line                                # "Refreshing <ws>: ..." or "Skipping <ws>: <reason>. ..."
  for each workspace to refresh:
    judge(it) again                                    # an agent can start work during an earlier recreate
    if it is now a skip: print "Skipping <ws> after all"
    else: recreate with no attach                      # part 2 collects and restarts the agents
  print the count; exit 1 if a recreate failed

judge(ws) skips when:
  another dl holds the launch lock
  the container is stopped                             # docker exec: "container <id> is not running"
  the processes cannot be read                         # one docker exec as root, /proc/*/stat and cmdline
  a build runs                                         # bazel, cargo, rustc, cc1plus, make, ninja, cmake, gcc, clang, ld, ...
  an agent process has a shell child                   # a tool call or a background task; MCP servers are not shells
  an agent runs and dl is outside herdr                # dl cannot start it again
  herdr does not answer, a pane has no saved line,
    or an agent is not idle or done
  the container runs more agents than herdr's panes hold

"Agent" means each agent that dl knows by name: claude, codex and gemini, run directly or by node. herdr's done means "idle, and nobody has looked at it yet", so idle and done both let a refresh go ahead.

How I checked it

All tests run on FakeRunner, or on the real binary with a fake docker on PATH. This devcontainer has no herdr and no docker daemon.

  • Core: 22 tests on the plan. They cover each skip rule and each build name, zombie processes, an agent run by node, a process list that ends early, a stat line that does not parse, herdr that does not answer, and several workspaces with different verdicts. I broke three rules on purpose, and five tests failed.
  • End to end (rust/dl/tests/read_side.rs): nothing stale, the plan printed before any recreate, exit 1 with the loop going on after a failed recreate, and a stopped workspace skipped as "it is stopped".
  • The review fixes each have a test that failed first:
    • A working codex agent no longer passes when no Claude process runs.
    • A Claude outside herdr no longer passes because an idle codex pane makes the count match.
    • A workspace is judged again just before its recreate, so a turn that started during an earlier recreate is not ended.
  • cargo test --workspace, clippy with -D warnings, cargo fmt --check: clean.
  • pytest test/ against the debug build: 850 passed.
  • prek run --from-ref origin/main: passed.

I did not run:

  • A real refresh. Done on 2026-10-15 on a real host: see "Found on a real host" below.
  • scripts/public-api-snapshots.sh. The tools are not in the container. I copied the diff from the public-api CI job into the snapshot files.

Merge danger

Door: two-way. The flag is new, and nothing else calls the new verb. The promised API (public-api.api.txt) gains LaunchVerb::RecreateUnattended, so a caller with an exhaustive match on LaunchVerb needs an arm for it.

Blast radius: medium. A false "refresh" ends a working agent. Each rule therefore skips when it is in doubt. These limits remain:

  • An agent can start work between the second check and the recreate. That agent ends, the same as with dl <ws> recreate. docs/cli.md says this.
  • The agent check needs the process name, or one of the first two cmdline words, to end in an agent name. A native install that starts a versioned binary by its own path is not seen as an agent. If it runs outside herdr, a refresh ends it.
  • The shell-child rule needs Claude to start a new shell for each command. If Claude keeps one shell for the whole session, every Claude workspace is skipped. That is safe, but no refresh then happens.

Found on a real host

A test on a real host (docker 29 with the containerd image store, devpod v0.26.3, herdr 0.9.5) found seven defects that the unit tests missed, because the fakes used the same wrong shapes. Details are in #677. Commits 34ec914, 3fa0ece and c94b5e7 fix six of them:

  1. devpod writes ContainerDetails.ID, not "Id". The stale check flagged nothing, so --ls and --refresh-stale never acted. Fixed in 34ec914; the fakes now say "ID".
  2. herdr pane ids are Crockford base32 (w3:pAX is pane 349). saved_argv parsed them as decimal, so a recreate ended every agent. Fixed in 34ec914; the tests use base32 ids.
  3. A restarted agent lost its saved line, so the second recreate ended the agent. The recreate now reports to herdr the line it typed, and retries on resume_not_accepted. Fixed in c94b5e7.
  4. The saved line starts with dl, so a resumed session runs whichever dl is on PATH. Not fixed: it matters only when a dev build runs beside the released one. The host test put a dl shim on PATH in the agent pane.
  5. A recreate reused devpod's derived <prefix>:devpod-<hash> image when devcontainer.json has "features", so the workspace stayed stale. dl now runs docker image rm --force <tag> before up, which only untags. Fixed in c94b5e7.
  6. With the containerd image store, a moved tag hides the old image from docker image inspect. Fixed in 3fa0ece: a container made from the reference is stale when its image id differs.
  7. Created was compared as text, so +01:00 and Z sorted wrong. Fixed in 3fa0ece; it is parsed as a timestamp now.

Host check on the tested tree (all pass):

  • dl --ls printed proj runs an older image than dl673-test:latest now names.
  • The first dl proj recreate printed Removed the tag proj-736a5:devpod-1c27…3834 that devpod built it under, so the recreate builds the image again from the newer one. docker events show this order: untag, new image built, old container stopped, new container started. docker gave no error. /stale-marker is in the new container, and --ls no longer says stale.
  • When a second container holds the tag, docker image rm --force <tag> prints only Untagged: and exits 0. Both containers keep running.
  • Two recreates in a row with an idle agent each printed Started the agent in pane wG:p4P again. herdr's session.json kept --resume <id> both times. After the second recreate, the agent remembered the word from before the first.
  • dl --refresh-stale run twice, each time after a new stale build. Both runs printed Refreshing proj… and Refreshed 1 stale workspace, skipped 0, and 0 failed. and exited 0. With nothing stale, it printed No workspace runs an older image than its reference names. and exited 0.
  • Finding 6 without the workaround: --ls still named the workspace stale after the old base left the store.

Unit tests on that host: 2232 pass. Four "pushed by URL" flows::lifecycle tests fail, and they fail the same way on f5551ce without these commits. CI's rust job passes them.

Stack

  1. #674 — feat: dl --ls says which workspaces run an older image
  2. #675 — feat: dl recreate starts the agents it ends again
  3. #676 — feat: dl --refresh-stale recreates the stale workspaces that are idle ← this PR

🤖 Generated with Claude Code

Summary by Sourcery

Add an unattended stale-workspace refresh workflow that safely recreates eligible workspaces, restores their agents, and reports skipped or failed workspaces.

New Features:

  • Add the global dl --refresh-stale command to recreate stale workspaces whose agents are idle and whose containers have no active work.
  • Support unattended recreates that restart agent sessions without attaching to the workspace.

Bug Fixes:

  • Ensure stale-image detection handles devpod records, moved image tags, differing timestamp offsets, and containerd image-store behavior correctly.
  • Preserve restarted agent session lines in herdr so subsequent recreates can resume them reliably.
  • Prevent unattended refreshes from recreating workspaces when devpod's derived image tag cannot be removed.

Enhancements:

  • Add conservative workspace safety checks that skip refreshes during launches, builds, active agent work, unreadable process states, or unresumable agent sessions.
  • Recheck each workspace immediately before recreating it and continue processing other workspaces after failures.
  • Improve herdr pane ID handling and report refreshed agent sessions back to herdr.

Documentation:

  • Document the new refresh command, safety rules, output, exit behavior, and derived-image handling in the README, CLI documentation, and changelog.

Tests:

  • Add unit and end-to-end coverage for refresh planning, safety skip reasons, stale-image edge cases, unattended recreates, agent restoration, and failure continuation.

Follow-up fixes

From the review in #677. One commit per finding.

  • A. 89f047d: the resume line names the dl that was run (argv[0]'s file name, aid-next maps to dl-next, falls back to dl), so a dl-next pane resumes under dl-next. Not current_exe(), because of pixi global's trampoline. A path is cut to its file name because herdr refuses a path.
  • B. c5c9b6c: a report thread that panics counts as a line not kept.
  • C. b92f32e: a line herdr took but dl could not read back gives the new notice AgentSessionLineUnchecked, not silence.
  • D. 12c183a: clients::docker returns ExecFailure::NotRunning; refresh_stale matches on it, not on stderr text.
  • E. d3d6d96: one /proc/<pid>/stat parser (domain::proc_stat, last-) rule), with a test for a comm that holds ) . The two Process records stay apart because each holds a field the other flow never reads.
  • F. 4a425e6: LaunchVerb::is_recreate() / is_unattended() replace the matches! checks and the unattended / warn_unseen bools.
  • G. 960ffce: report_seq() and read_session_file() replace the copied closures.
  • H. d5fdd48: one herdr::Locator for the resume report and the pane scan.
  • I. ada9df4: collect matches on AgentReading directly.
  • J. skipped: production reads AgentStatus / PaneAgent::status (refresh_stale::sessions_skip gives Skip::AgentBusy from it).
  • K. 94feec5: HeldSession::shell_line().
  • L. 6bfd3f2: restart_agent_sessions and hand_back_agent_sessions replace the up: bool.
  • M. 51f6fba: the notices name each agent's own resume words (herdr::AGENT_RESUMES), not claude --resume for every agent. A Python test diffs that table against aid's.
  • N. b6e22f2: agent_reading is pub(crate).
  • O. d38afa7: agent_sessions::Manager is now HerdrPanes.
  • P. ff8015b: the second look before each recreate, the skip that wins, and the tally move into core refresh_stale::run, with unit tests there.
  • Q. de3d64b: one DerivedImageKept struct for the notice and the refusal.
  • 6685496: snapshot row order fix for P.

Public API. The promised API (public-api.api.txt) changes: LaunchNotice gains AgentSessionLineUnchecked { pane_id }, and LaunchNotice::DerivedImageKept now holds the struct flows::launch::DerivedImageKept in place of three fields. A caller that builds or destructures that variant needs a change. DerivedImageKept is not re-exported from api, so the residual count goes from 40 to 41 types. In public-api.rest.txt: HeldSession::shell_line, herdr_resume_line(s), refresh_stale::{run, Refresher, Tally} and the DerivedImageKept struct are new; LaunchRefusal::DerivedImageKept holds the struct; agent_reading is gone; Manager rows are renamed to HerdrPanes.

blooop added 14 commits October 7, 2026 15:08
flows::refresh_stale::plan reads, per stale workspace, its launch lock,
one docker exec of its process table, and the Claude sessions herdr's
panes hold, and says Refresh or Skip with the reason. A build process,
a shell under a Claude process, a busy or unresumable agent, a Claude
session no pane holds, and a herdr that cannot be seen are each a skip.

StaleImage now carries the container it was judged by, which is the
container the plan reads.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
LaunchVerb::RecreateUnattended is recreate's path, agent sessions
included, ending at Launched::Ready instead of a session. It is what
dl --refresh-stale runs for each workspace, with nobody to attach.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
… seen

dl --refresh-stale recreates outside herdr only a container it read no
Claude process in, so the warning that the recreate ends the agents it
cannot see was false there. A recreate typed by hand still warns.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
It prints one line per stale workspace (refresh, or skip and why), then
runs the unattended recreate for each it may refresh, says each launch's
notices as they happen, and ends with a count. A recreate that fails is
counted and the loop goes on; the exit status is 1 when any failed.
Named in dl.bash's global options and in the help's examples.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…review/refresh-stale

# Conflicts:
#	CHANGELOG.md
#	rust/devlaunch-core/src/flows/launch.rs
…sed a Claude outside herdr beside a held codex

The rule knew only Claude processes. A codex running a command, or any
codex at all outside herdr, left no Claude process to find, so the
workspace was refreshed and the codex ended. And the count compared
Claude processes against herdr's panes of every agent kind, so one held
codex covered a Claude started outside herdr, which then ended with no
line to start it again.

An agent is now any agent dl knows by name (herdr::agent_named), found
the way Claude was: by comm, or argv[0] or argv[1] for node. The shell
rule and the count apply to all of them, so both sides of the count are
agents of any kind.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…rdr gives

Seven cases the plan had no test for: a listing that ends halfway through a
process, a stat line that does not parse, a held agent herdr does not answer
for, a saved line in a pane herdr says holds no agent, a launch lock that
cannot be asked, a Claude started under a Claude, and three stale workspaces
whose verdicts must not leak into each other. Each was checked to fail with
its branch broken.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
dl --refresh-stale judged every stale workspace once, printed the plan,
and then recreated them in turn. A recreate can take minutes, so a
workspace that was idle at plan time could have a build or a working
agent by the time its turn came, and the recreate ended it.

Each workspace the plan would refresh is now judged again just before
its recreate. A skip then wins: it is counted as skipped and printed as
"Skipping <ws> after all, judged again before its recreate: <why>".
The plan is still printed whole, first.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
render_refresh_stale had no test. Four now run the binary against the
read-side world with a docker fake that answers the process read:

- nothing stale: the "nothing" line, exit 0, no docker exec.
- one quiet and one with cc1plus: the whole plan is on stdout before
  the first devpod up, one recreate, refreshed 1 skipped 1 failed 0.
- the first recreate fails: the second still runs, 1 failed, exit 1.
- a stopped container: a Skipping line, no up, exit 0.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
pixi was on the build list, so a pixi shell that stays open in a pane
kept that workspace stale for as long as it ran. A build that pixi run
starts runs cargo, a compiler or another program the list already
names, so the list loses nothing by dropping pixi.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…a docker fault

A stopped stale workspace was skipped as "could not read the processes
in its container (docker exec failed ...)". Skipping is right, because
a recreate would also start it, but the line said docker was broken.
docker exec into a stopped container says "container <id> is not
running", so the plan now skips that as Skip::Stopped, and dl says the
workspace is stopped and that `dl <ws> recreate` starts it on the new
image. (The Skip::Stopped variant and its match arm went in with the
previous commit, which touched the same file.)

Also: herdr's `done` is an idle agent nobody has looked at, so with the
part 2 fix a done agent is held and a refresh may go ahead. The test
that pinned the old skip now expects a refresh, and docs/cli.md says
why pixi is not on the build list.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
@sourcery-ai

sourcery-ai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Sorry @blooop, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 23 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@sourcery-ai

sourcery-ai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Reviewer's Guide

Introduces dl --refresh-stale, which identifies stale workspaces, conservatively refreshes only those whose containers and agent sessions appear idle and recoverable, rechecks eligibility immediately before each unattended recreate, restarts held agents without attaching, and reports skips and failures.

Sequence diagram for unattended stale workspace refresh

sequenceDiagram
    participant User
    participant CLI as dl CLI
    participant Stale as stale_images
    participant Planner as refresh_stale::plan
    participant Docker as Docker container
    participant Herdr as herdr
    participant Launch as Launch

    User->>CLI: --refresh-stale
    CLI->>Stale: stale_images(workspaces)
    Stale-->>CLI: stale images and container ids
    CLI->>Planner: plan(stale workspaces)
    Planner->>Docker: exec_as_root(read process table)
    Docker-->>Planner: processes or unreadable
    Planner->>Herdr: collect and read agent sessions
    Herdr-->>Planner: pane states and saved lines
    Planner-->>CLI: Refresh or Skip verdicts
    CLI->>User: Print complete plan
    loop Each planned refresh
        CLI->>Planner: plan(workspace) again
        Planner-->>CLI: Refresh or Skip verdict
        alt Still eligible
            CLI->>Launch: render_launch(RecreateUnattended)
            Launch->>Launch: Recreate container
            Launch->>Herdr: Restart held agent panes
            Launch-->>CLI: Success or failure
        else No longer eligible
            CLI->>User: Skipping after all
        end
    end
    CLI->>User: Print tally and exit status
Loading

Flow diagram for stale workspace eligibility checks

flowchart TD
    A[Stale workspace] --> B{Launch lock free?}
    B -- No --> S1[Skip: another dl is launching it]
    B -- Yes --> C[Read processes with docker exec as root]
    C --> D{Container stopped?}
    D -- Yes --> S2[Skip: it is stopped]
    D -- No --> E{Processes readable?}
    E -- No --> S3[Skip: processes unreadable]
    E -- Yes --> F{Build process present?}
    F -- Yes --> S4[Skip: build is running]
    F -- No --> G{Agent has shell child?}
    G -- Yes --> S5[Skip: agent is running a command]
    G -- No --> H{Agent sessions present?}
    H -- No --> R[Refresh]
    H -- Yes --> I{Sessions visible and recoverable?}
    I -- No --> S6[Skip: sessions unavailable or unresumable]
    I -- Yes --> J{Every agent idle or done?}
    J -- No --> S7[Skip: agent is busy or unanswered]
    J -- Yes --> K{Running agents <= herdr panes?}
    K -- No --> S8[Skip: sessions outside panes]
    K -- Yes --> R
Loading

File-Level Changes

Change Details Files
Adds the global --refresh-stale command and documents its unattended behavior.
  • Adds CLI parsing, validation, help text, completion, README, changelog, and detailed CLI documentation.
  • Restricts the command to global use without a workspace, -y, or --force.
  • Defines plan, skip, recreate, output, and exit-status behavior for stale workspaces.
rust/dl/src/cli.rs
rust/devlaunch-core/completions/dl.bash
README.md
docs/cli.md
CHANGELOG.md
Implements conservative stale-workspace eligibility checks before recreating containers.
  • Adds a refresh planning flow that evaluates stale workspaces in rule order and reports the first failure.
  • Reads container process tables via a root Docker exec, detecting stopped/unreadable containers, builds, agent shell children, and zombie processes.
  • Validates herdr visibility, resumable saved commands, agent idle/done state, and matching agent-session counts.
  • Carries the stale container ID through StaleImage for process inspection.
rust/devlaunch-core/src/flows/refresh_stale.rs
rust/devlaunch-core/src/flows/refresh_stale/tests.rs
rust/devlaunch-core/src/flows/stale_images.rs
rust/devlaunch-core/src/flows/mod.rs
Extends the launch flow with an unattended recreate that restarts held agents without attaching a terminal.
  • Adds LaunchVerb::RecreateUnattended using the existing recreate path and agent-session holding/restart behavior.
  • Suppresses unseen-agent warnings for the unattended mode because eligibility checks already exclude unsafe outside-herdr cases.
  • Adds coverage confirming agents restart in herdr panes and no SSH session is opened.
rust/devlaunch-core/src/flows/launch.rs
Integrates planning, revalidation, execution, rendering, and failure aggregation into the CLI command.
  • Prints the complete initial plan before any recreate begins.
  • Rejudges each planned refresh immediately before execution, skips workspaces that became busy, and continues after individual recreate failures.
  • Reports refreshed/skipped/failed counts and returns exit status 1 if any recreate fails.
  • Adds human-readable reasons for every skip and end-to-end coverage for ordering, stopped containers, rejudging, and continued execution.
rust/dl/src/commands.rs
rust/dl/src/render.rs
rust/dl/tests/read_side.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.97406% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.58%. Comparing base (6e8f127) to head (4857788).

Files with missing lines Patch % Lines
rust/dl/src/render.rs 89.36% 20 Missing ⚠️
rust/devlaunch-core/src/flows/launch.rs 98.00% 9 Missing ⚠️
rust/devlaunch-core/src/clients/docker.rs 93.54% 4 Missing ⚠️
rust/devlaunch-core/src/flows/agent_sessions.rs 96.89% 4 Missing ⚠️
rust/devlaunch-core/src/flows/refresh_stale.rs 98.30% 3 Missing ⚠️
rust/dl/src/commands.rs 96.87% 2 Missing ⚠️
Additional details and impacted files
Flag Coverage Δ
python 42.98% <ø> (ø)
rust 94.79% <96.97%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
shipped code (rust) 94.79% <96.97%> (+0.05%) ⬆️
harness and tooling (python) 42.98% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copied from the public-api CI job's diff. public-api.api.txt gains
LaunchVerb::RecreateUnattended, a change to the promised API: a caller
that matches LaunchVerb exhaustively needs an arm for it.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…shape

Found on a real host (#677, findings 1 and 2). devpod writes
ContainerDetails.ID, not "Id", so the stale check flagged nothing and
--ls and --refresh-stale never acted. herdr pane ids end in Crockford
base32 (w3:pAX is pane 349), not decimal, so saved_argv failed for every
real pane and a recreate ended the agent. The test fakes used the same
wrong shapes; they now use the real ones.
…imes as text

Found on a real host (#677, findings 6 and 7). With docker's containerd
image store, a local image whose only tag moved is gone from
docker image inspect while a container still runs it. A container made
from the reference itself is now stale when its image id differs from
the reference's id. Created is parsed as a timestamp, so a local build's
+01:00 and a pull's Z compare correctly.
Found on a real host (#677, findings 3 and 5).

Finding 5: with "features" in devcontainer.json, devpod builds a
<prefix>:devpod-<hash> image whose hash covers the config, not the base,
so up --recreate reused it and the workspace stayed stale. Before up,
dl now runs docker image rm --force on that tag, which only untags while
the old container runs. If that fails, recreate goes ahead with a notice
and --refresh-stale skips the workspace.

Finding 3: a line typed back into a pane got no DEVLAUNCH_HERDR_RESUME,
so herdr kept only {"cwd"} and the second recreate ended the agent.
recreate now reports the line it typed to herdr, retries while herdr
says resume_not_accepted, and warns when herdr did not keep it.

@blooop blooop left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during review.

wf-review of #676 at c94b5e7e. Fixed point: merge-base with feat/recreate-keeps-agent, 6b158582. Range: git diff 6b158582...HEAD, which has 18 commits, 20 files, +2628/−37. Two separate subagents ran the two axes in fresh context. Neither saw the code being written. CI was green when the review ran, except for review and gate.

Standards

Sources: AGENTS.md (CLAUDE.md holds the same text) and docs/development.md, then the Fowler baseline. I found no fixed-sleep retry in library code: wait is injected, and the threads are std::thread::scoped and joined.

  1. rust/dl/src/render.rs:3677-3692: misplaced doc comment. derived_image_kept sits between the doc block of launch_refusal and the fn launch_refusal. Rustdoc now joins that block to the new helper's lines, so launch_refusal loses its doc. Move the helper above the comment. Non-blocking. Fix it before merge.
  2. devlaunch-core/src/flows/refresh_stale.rs:263-327 against agent_worktrees/agent_lock.rs:681,763-790: Duplicated Code. This is a second /proc/<pid>/stat parser and a second struct Process. The two parsers differ: one splits after the last ), the other on ") ". The repo rule says one source, or a diff test. Non-blocking.
  3. refresh_stale.rs:182: the flows layer classifies docker stderr. why.contains("is not running") matches the text of a formatted String. A typed "not running" error from clients::docker keeps a change to the message from turning Stopped into ProcessesUnread. Non-blocking.
  4. flows/launch.rs:5217-5234: Repeated Switches and bool flags. matches!(verb, Recreate | RecreateUnattended) appears twice, and two more matches! become the bare bools unattended and warn_unseen. Add is_recreate() and is_unattended() to LaunchVerb, beside rebuild(). Non-blocking.
  5. launch.rs:2998: a panic is swallowed. report.join().ok().flatten() treats a panicked report thread as "line kept", so the user sees AgentSessionRestarted and not AgentSessionLineNotKept. Map Err to Some(pane_id). Non-blocking.
  6. launch.rs:2975, :3022: Duplicated Code. The seq and read closures are copied from beside_resume_report. Non-blocking.
  7. dl/src/commands.rs:1286-1371: Divergent Change. The binary holds part of the refresh policy: it judges again before each recreate, lets a skip win, and counts the tallies. The core module says it "decides and does nothing". Only read_side.rs end-to-end tests cover that half. Non-blocking.
  8. Minor: Data Clump and Primitive Obsession. LaunchNotice::DerivedImageKept and LaunchRefusal::DerivedImageKept repeat {workspace_id, image, why}. Planned.workspace_id is a String. Both follow the existing StaleImages convention. Non-blocking.

Axis verdict: Comment. Nothing is blocking.

Spec

Spec: #673, the body of #676, and the decision comments on #677 (6084475333, 6128046327).

The spec items below match the code:

  • Stale and idle only. #673: "applies 2 to each stale workspace whose Claude sessions are all idle … and that runs no build; it skips busy ones and reports them". plan looks only at stale workspaces (refresh_stale.rs:137). It checks builds (:185), an agent with a shell child (:199), idle or done state (:240), and agents that no herdr pane holds (:253). Each skip line gives its reason. The code judges each workspace again just before its recreate (commands.rs:1331).
  • Exit codes. A failed recreate counts as a failure, and a skip does not (commands.rs:1366). Ending::Refused gives exit 1.
  • Finding 3, option (a). "the recreate reports the same line to herdr … retries on resume_not_accepted (1 s, up to 20 tries), one thread per pane". The code does this at launch.rs:2957, agent_sessions.rs:286, and session_manager.rs:480-481. The notice AgentSessionLineNotKept is at launch.rs:5380.
  • Finding 5, options (a) and (b) as built. "runs docker image rm --force <tag> … If that fails: recreate goes ahead with notice DerivedImageKept; --refresh-stale refuses that workspace". The code does this at docker.rs:98-101 and launch.rs:5270-5308, and stale_images.rs:201-203 checks that the tag differs from the reference.
  • Finding 4 ("The typed line's argv[0] is dl") is not fixed, and the PR says so. I do not count it against the PR.

Findings:

  1. "reads session.json back. When herdr does not keep the line, dl prints a new notice AgentSessionLineNotKept". report_again treats ResumeReported::Taken as Kept (agent_sessions.rs:306). report_resume also returns Taken when it cannot read session.json. In that case dl never confirmed the line, the notice does not show, and the next recreate can end the agent. Non-blocking.
  2. Option (b): "--refresh-stale skips such a workspace and says that a recreate cannot move it". The plan still prints "Refreshing " for that workspace. The refusal comes later, when the untag fails (render.rs:3714). The result is correct, but the plan line for that workspace is wrong. Non-blocking.
  3. #673 "Done when": "dl --refresh-stale refreshes idle stale workspaces and leaves busy ones alone". The PR body said "I did not run: A real refresh." Non-blocking. Note from the run owner: the host run on 2026-10-15 at c94b5e7e's tree did run a real refresh. Two refreshes and two recreates with an idle agent passed, and the run found no stale workspace left behind. The evidence is in the "Found on a real host" section of the PR body.

Axis verdict: Approve. Nothing is blocking.

Verdict

Comment. Neither axis has a blocking finding. Standards 1, the misplaced doc comment, should be fixed before merge. The other findings can be follow-ups.

Found in the wf-review of #676 (Standards 1). The helper sat between
launch_refusal's doc block and the fn, so rustdoc joined the two and
launch_refusal lost its doc.
Found in the wf-review of #675 (Standards 1). agent_sessions and
docs/cli.md treat done as an idle agent nobody has looked at; the enum
doc said the agent had ended, which invites dropping done panes.
The saved resume line herdr types back always began with a plain `dl`, so a
pane started with the dev build came back under the released one. The line now
names the dl this process was started as: argv[0]'s file name, with aid's name
mapped to its dl, falling back to `dl`. Not current_exe(), which resolves pixi
global's trampoline to a path that moves on upgrade. One function,
herdr::restore_program, feeds both the reported line and the hook's prefix.
report_typed_lines_again joined each report thread with .ok().flatten(),
so a thread that panicked counted as kept and the user saw no notice that
the next recreate would end the agent. A panic confirms nothing, so the
pane now counts as not kept. The per-pane threads move into
lines_not_kept, which takes the report as a closure so a test can make
it panic.
report_again mapped ResumeReported::Taken to LineKept::Kept, and
report_resume returns Taken when it cannot read session.json back, so a
line nobody confirmed went by as a restart that went well. It now maps
to LineKept::Unchecked, and the launch says so with a new notice,
AgentSessionLineUnchecked: herdr took the line and dl could not check
that it kept it. That is not the same as herdr refusing it, so it does
not reuse the not-kept wording.
blooop added 16 commits October 16, 2026 13:25
…aunch

beside_resume_report and report_typed_lines_again each built the same
wall-clock seq closure and the same read closure, and the agent-session
collect built the read a third time. They now share report_seq and
read_session_file.
restart and the not-restarted notice each joined HeldSession::line into
a shell line by hand. HeldSession::shell_line now does it once.
refresh_stale told a stopped container from a docker fault by searching
the error words in the flows layer, far from the docker call that wrote
them. exec_as_root now reads docker's stderr itself and returns
ExecFailure::NotRunning, and refresh_stale matches on that variant.
agent_lock and refresh_stale each parsed the stat line their own way.
The two agree on every real line today, but two copies are two chances
to drift. domain::proc_stat now holds the one parser, which ends the
command name at the last ')', and both flows read through it. Test
fixtures in refresh_stale now print whole stat lines, as the kernel does.
…attended

run_rebuild asked matches!(verb, Recreate | RecreateUnattended) twice and
passed drop_derived_image and hold_agent_sessions a bare bool each, one
named unattended and one warn_unseen that was its negation. The verb now
answers is_recreate() and is_unattended() itself and the two helpers take
the verb, so the call sites read as what they ask.
restart_agent_sessions took up: bool and did one of two unrelated things
on it: start each agent in its pane, or hand each line to the user. The
caller now picks by the up outcome, and each half has its own name.
…n claude --resume

The notices for agents a recreate ends and cannot restart named
'claude --resume' as the way back, and render.rs picked claude's words
with agent == "claude". A codex or gemini pane got either no resume or
claude's. Core now keeps each known agent's resume words beside
AGENT_NAMES (herdr::resume_line / resume_lines, exported for dl), a
Python test diffs them against aid's AGENTS table, and the notices name
the agent's own resume, or every known agent's when dl cannot see which
one ran.
Only collect and refresh_stale call it, both inside devlaunch-core, so it
leaves the public API. Drops its public-api.rest.txt row.
It folded AgentReading into an Option<Option<String>> and then matched that,
so a reader had to decode what None and Some(None) meant. The match now names
each reading, and every case is spelled out.
agent_sessions::Manager::resolve was a copy of ResumeReport::resolve: the
provisioning gate, session.json beside the socket, and the PATH fallback for
the binary. herdr::Locator now holds that rule once; the pane id is optional
in it because the scan needs none and the report requires one.
Manager sat beside flows::session_manager and said nothing about what it
holds: the herdr whose panes a recreate scans and types lines into. The
public-api.rest.txt rows move with the name; their order is unchanged.
The binary judged each workspace again before its recreate, let a skip
found then win, and counted the tally. That is refresh policy, so it now
lives in refresh_stale::run, which takes the recreate through a Refresher
and returns a Tally. Core tests cover a workspace that turns busy after
the plan and the tally. The binary keeps the wiring and the lines.
LaunchNotice::DerivedImageKept and LaunchRefusal::DerivedImageKept both
spelled out workspace_id, image and why. They now hold one struct, so the
two cannot drift apart.
The rows for refresh_stale's Refresher trait were added by hand above the
BUILD_PROGRAMS row, where cargo public-api renders them below it. CI's
public-api job diffs the order too.
main made dla a second name for aid (#679), so restore_program maps dla and
dla-next the way it maps aid and aid-next.
Base automatically changed from feat/recreate-keeps-agent to main October 16, 2026 15:07
@blooop
blooop merged commit 8b20165 into main Oct 16, 2026
15 checks passed
@blooop
blooop deleted the feat/refresh-stale branch October 16, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant