Repository navigation
Handover: validate the #673 stack (#674–#676) on a real host #677
Description
A handover for the agent that continues this work on another machine.
Goal
Finish validating the stack for
#673 — A workspace keeps its old image after a pull, and a recreate ends its agent
on a real host (docker + devpod + herdr), and turn the defects found so far into failing tests.
Report only. Do not push, comment on GitHub, or merge unless the user says so.
Stack (bottom first), all on blooop/devlaunch:
- #674 — feat: dl --ls says which workspaces run an older image,
feat/stale-image-ls - #675 — feat: dl recreate starts the agents it ends again,
feat/recreate-keeps-agent - #676 — feat: dl --refresh-stale recreates the stale workspaces that are idle,
feat/refresh-stale(holds all three; was atf5551ced)
Use a machine with no real workspaces or agents you care about — --refresh-stale and recreate
end every process in a container. Still use a scratch DEVPOD_HOME and XDG_CACHE_HOME.
State: what the first run (2026-10-08, another host) found
Two one-line defects that make the stack do nothing on a real host. The unit tests miss both because
their fakes use the same wrong shape.
rust/devlaunch-core/src/flows/stale_images.rs:216readsContainerDetails["Id"]; devpod writes
ContainerDetails.ID. Result: part 1 flags nothing, so--lsand--refresh-stalenever act.
Fakes with"Id": tests instale_images.rs,rust/dl/tests/read_side.rs:426,965,
rust/devlaunch-core/src/flows/lifecycle/tests.rs:94.rust/devlaunch-core/src/clients/herdr.rs:2101saved_argvparses the pane-id suffix as decimal.
herdr pane ids are Crockford base32 (w3:pAX= 10×32+29 = 349, matching
public_pane_numbers: {"226": 349}in herdr'ssession.json). Every real id fails, so a recreate says
"herdr saved no line to start it again" and ends the agent. Tests usew1:p9, which hides it.
With both fixed, these passed: --ls matched a by-hand check on 29 real workspaces (a compose devcontainer whose service
names a plain local tag included: declared is null, createdFrom is that tag, and that works); recreate with
the agent done restarted it with the same session id and --remote-control=<ws>; /clear + recreate
brought back the new id (needs DEVLAUNCH_HERDR=1); recreate outside herdr warned and did not hang;
--refresh-stale refreshed an idle agent and skipped correctly for: agent running a command, a make
process, a stopped workspace, an agent started outside herdr. No false "Refreshing" was seen.
Other open findings to confirm and turn into tests:
- A restarted agent loses its saved line. The typed line is
dl <ws> -- … claude … --resume <id>; a
--resumelaunch gets noDEVLAUNCH_HERDR_RESUME, so the container hook exits early and herdr ends up
with{"cwd": …}only. The second recreate drops the agent; the second--refresh-staleskips with
"no saved line". Seen with and withoutDEVLAUNCH_HERDR=1. - The typed line's argv[0] is
dl, so the releaseddlruns the resumed session, notdl-next. - With
"features"in devcontainer.json,recreatereuses the cached<ws>-xxxx:devpod-<hash>image, so
the container stays on the old base and stays stale;--refresh-stalewould recreate it every run. - With docker's containerd image store, a locally built image whose only tag moved is gone from
docker image inspectwhile a container still runs it, so part 1 calls the workspace current.
Workaround used in testing:docker tag <ref> <ref-prev-N>before each rebuild. created.get(..19)comparesCreatedas text; local builds print+01:00, pulls printZ.- Without a persistent
~/.claude, the resumed agent hits Claude's Bypass Permissions prompt and
--resumefinds no conversation. Bind a scratch dir to/home/vscode/.claude(a compose setup that binds the
host's~/.claudeis fine). Side effect seen: with that bind, dl stopped sendingCLAUDE_CODE_OAUTH_TOKEN, so the agent
needed/login— not diagnosed.
The first run's full report stays on the first host; everything needed to continue is in this issue.
The WIP fix (not pushed; apply the diff below)
Apply on top of feat/refresh-stale:
--- a/rust/devlaunch-core/src/clients/herdr.rs
+++ b/rust/devlaunch-core/src/clients/herdr.rs
@@ -2100,7 +2100,9 @@ pub(crate) enum Saved {
pub(crate) fn saved_argv(session_json: &str, pane_id: &str) -> Option<Option<Vec<String>>> {
let (workspace_id, public) = pane_id.split_once(":p")?;
- let public: u64 = public.parse().ok()?;
+ let public: u64 = public.chars().try_fold(0u64, |n, c| {
+ "0123456789ABCDEFGHJKMNPQRSTVWXYZ".find(c.to_ascii_uppercase()).map(|d| n * 32 + d as u64)
+ })?;
let session: serde_json::Value = serde_json::from_str(session_json).ok()?;
--- a/rust/devlaunch-core/src/flows/stale_images.rs
+++ b/rust/devlaunch-core/src/flows/stale_images.rs
@@ -213,7 +213,7 @@ fn created(devpod_home: Option<&DevpodHome>, workspace_id: &str) -> Option<Creat
let created_from = text(&details["Config"]["Image"])?;
Some(Created {
- container: text(&details["Id"])?,
+ container: text(&details["ID"])?,Check the base32 alphabet against herdr's source before trusting it; it fit every id seen (p2G=80, pA1=321, pAX=349).
Runner
git clone --branch feat/refresh-stale https://github.com/blooop/devlaunch ~/dl673/src
cd ~/dl673/src && git apply <the diff above> # or edit the two lines by hand
./dev.sh # installs dl-next / aid-next; needs cargo
dl-next --version # "dl <version>-dev"
cd rust && cargo test -p devlaunch-core # unit tests
Load the herdr skill; run inside a herdr pane (HERDR_ENV=1). Gotchas from the first run:
herdr tab create --env K=Vreaches only that tab's root pane; a split pane needs the exports again.- Recreate attaches a shell at the end. Run it under
script -qfc "dl-next <ws> recreate" log; piping it to
teehangs. pane wait-output --match 'RC='matches the typed command line; use--regex '(?m)^RC=\d'.herdr agent prompt … --timeoutneeds--wait.- Claude Code blocks a standalone
sleep 120; ask fortimeout 120 tail -f /dev/null; echo finished. devcontainers/base:ubuntuhas multi-call coreutils, socp /bin/sleep /tmp/makefails. Use
printf '#!/bin/sh\nsleep 300\n' > /tmp/make; chmod +x /tmp/make; /tmp/make &in the container.- A new workspace stops
aid-nextat the folder-trust prompt and the Bypass Permissions prompt; answer them
withherdr agent send-keys <pane> down enter/enter. - After each restart, start a fresh agent (
DEVLAUNCH_HERDR=1 aid-next <ws> 'Say hi.'), or finding 3 makes
the next row skip for "no saved line".
Test project (scratch env exported):
export DEVPOD_HOME=~/dl673/devpod XDG_CACHE_HOME=~/dl673/cache
devpod provider add docker && devpod provider use docker
docker pull mcr.microsoft.com/devcontainers/base:ubuntu
docker tag mcr.microsoft.com/devcontainers/base:ubuntu dl673-test:latest
mkdir -p ~/dl673/proj/.devcontainer ~/dl673/claude-home && cd ~/dl673/proj && git init -q
echo '{"image":"dl673-test:latest","mounts":["source='$HOME'/dl673/claude-home,target=/home/vscode/.claude,type=bind"]}' > .devcontainer/devcontainer.json
git add -A && git commit -qm init && dl-next ~/dl673/proj
# make it stale, keeping the old image inspectable (finding 6):
docker tag dl673-test:latest dl673-test:prev-1
printf 'FROM dl673-test:latest\nRUN touch /stale-marker\n' | docker build -t dl673-test:latest -
Next steps
- Run the one row not done:
--refresh-stalewith nothing stale → expect "No workspace runs an older
image…", exit 0. - Re-run step 4 with an
idleagent (onlydonewas tested with the fixes) and ask the agent what you
said before, to prove the conversation came back. - Write failing tests for findings 1 and 2 (real shapes:
"ID", ids likew3:pAX), then fix. Use the
tddskill. - Confirm findings 3, 5 and 6 and decide with the user whether each blocks the stack.
- Report back with a ledger; push nothing without the user's word.
- Clean up:
dl-next proj rm --forceunder the scratch env,docker rmieverydl673-test:*and
proj-*:devpod-*image, close your herdr tab,rm -rf ~/dl673(it may hold a.credentials.json).
🤖 Generated with Claude Code
Activity
blooop commentedon Oct 9, 2026
Update: the tests for defects 1 and 2 are written. This diff replaces the one in the issue body; it holds both fixes and the tests, on top of
feat/refresh-staleatf5551ced.a_pane_id_is_read_as_herdrs_base32_numberand fivestale_imagestests, whose fakes now say"ID").cargo test --locked -p devlaunch-core -p dlpasses except fourflows::lifecycle::tests(commits_pushed_by_url_do_not_stop_a_plain_rmand three others). Those four fail the same way without this diff, and CIrustpasses them atf5551ced, so they depend on the first host. Re-check them on the new machine.Apply with
git applyfrom the repo root:Steps left (from the issue body): row 6 (nothing stale), step 4 with an
idleagent, and a decision on findings 3, 5 and 6.🤖 Generated with Claude Code
blooop commentedon Oct 9, 2026
Update 2: findings 6 and 7 are fixed with tests, and findings 3 and 5 need a decision.
Fixed, on top of the previous comment's diff (
stale_images.rs):docker image inspectcannot find the running image, a container made from the reference itself is stale if its image id differs from the reference's id. A derived image is not judged without its layers.Createdis parsed as ajiff::Timestamp, not cut to 19 characters, so...Zand...+01:00compare correctly.a_container_made_from_the_reference_is_stale_after_its_image_left_the_storeandcreated_times_compare_across_utc_offsetsfail. Green: all 15stale_imagestests pass.cargo clippy -p devlaunch-core --all-targets -D warningsis clean. The same four host-onlylifecycletests fail, as before.Apply after the previous comment's diff:
(
cargo fmtalso reflowed theherdr.rslines from the previous diff. Runcargo fmtafter you apply.)Finding 3, decision needed: a restarted agent loses its saved line.
Cause: only
aidgives dl a line to report (dl/src/lib.rs:573passesNone;aid/src/main.rs:251is the onlyrun_resumablecaller). A typed-backdl ws -- … claude --resume <id>reports nothing and gets noDEVLAUNCH_HERDR_RESUME. This is deliberate:dl/src/lib.rs:579-581says adlthat reads a line out of a command tail "would be guessing at another program's flags". Also, withDEVLAUNCH_HERDR=1, the old agent'sSessionEndhook runspane release-agent(herdr.rs:1592), which can clear the saved line.Options:
recreatereports the exact line it typed into the pane again, afterRestarted::Typed, and retries onresume_not_accepted. No guess, because the line is the one herdr held. Also check the order against the old agent's release.dlbuilds anAgentResumefrom a tail that ends inclaude … --resume <id>(at theNonearm oflaunch.rs:2888). This reverses the recorded decision. Nearest tests:an_agent_launched_in_a_pane_tells_herdr_how_to_start_it_againanda_consented_pane_hands_the_container_the_line_to_resume_by_idinlaunch.rs.Finding 5, decision needed: a recreate keeps a devpod-derived image.
devpod v0.26.3 has no flag that forces the derived image to rebuild (
uphas--recreateand--resetonly;buildhas no force flag). The derived tag is a hash of the config, not of the base image, so devpod reuses it every time.Options:
up --recreate, when the workspace is stale andcreatedFrom≠ the declared image, rundocker rmi <createdFrom>without-f. If docker refuses (another container uses it), fall back to (b).--refresh-staleskips such a workspace and says that a recreate cannot move it, so it does not recreate it on every run.🤖 Generated with Claude Code
blooop commentedon Oct 12, 2026
Update 3: findings 3 and 5 fixed (local commit
435a42da, on top of2713b4c0)The user chose the recommended options.
Finding 3. After
restarttypes a line into a pane, the recreate reports the same line to herdr (pane report-agent-session, throughsession_manager::report_resume). It retries onresume_not_accepted(1 s, up to 20 tries), one thread per pane, and readssession.jsonback. When herdr does not keep the line, dl prints a new noticeAgentSessionLineNotKept. Limit: an agent restarted from a line has no container hook, so after/clearin it the saved line still has the old id.Finding 5.
StaleImagenow carriesderived_image(): theContainerDetails.Config.Imagetag, when it is<prefix>:devpod-<hash>and differs from the reference. Beforeupinrecreateand in the unattended recreate, dl runsdocker image rm --force <tag>.--forceis required because the old container still uses the image; with a tag it only untags. If that fails:recreategoes ahead with noticeDerivedImageKept;--refresh-stalerefuses that workspace (LaunchRefusal::DerivedImageKept), so it is not recreated on every run.Tests: 3 new in
agent_sessions, 2 inlaunch(finding 3; both fail without the wiring), 3 instale_images, 3 inlaunch(finding 5; all fail without the untag step). Full suite green except the 5 host-only "pushed by URL" lifecycle tests, which also fail on the base. Clippy-D warningsand fmt clean.public-api.*.txtedited by hand (nocargo public-apion that host) — let CI'spublic-apijob confirm.To confirm on the other machine:
docker image rm --force <tag>with the old container running: only untags, the container keeps running, anddevpod up --recreatethen builds a new:devpod-image on the new base. After it,dl-next --lssays current.idleagent: the second recreate still brings the agent back (herdr'ssession.jsonholds the line after the first).Apply with
git amon top of the earlier diffs:patch
blooop commentedon Oct 15, 2026
Takeover prompt for a real host (2026-10-15)
The agent that reads this owns the rest of #673: test, fix, push, CI and review. The merge stays with the user.
Take over the devlaunch #673 work fully on this machine: test the fixes on a real host, fix what fails, and bring the stack to a reviewable, green state. The user has handed the whole task to you; the previous agent will do no more. Do not merge: the merge stays with the user.
Context: #677 is the handover. Read its body and its three comments first. The stack is PR #674 → #675 → #676 on blooop/devlaunch; branch feat/refresh-stale at f5551ce holds all three. The comments on #677 hold three patches (```diff blocks) with fixes found on a real host. Nobody has run fixes 3 and 5 against real docker, devpod and herdr yet. That is your job.
Safety (read before you start)
recreateand--refresh-staleend every process in a container.~/dl673/dlnthat exports DEVPOD_HOME=/dl673/devpod and XDG_CACHE_HOME=/dl673/cache, refuses to run if either is not under ~/dl673, then execs dl-next "$@". Use it for every call.1. Build
The three patches applied in that order, then cargo fmt, give the tested tree (verified 2026-10-15). Expected unit-test result: all pass, except possibly 4–5 "pushed by URL" lifecycle tests, which fail on some hosts without these patches too. Say which fail.
2. Test project (the derived-image case, finding 5)
Run inside a herdr pane (HERDR_ENV=1). Load the herdr skill.
Record ContainerDetails.Config.Image from the workspace_result.json under ~/dl673/devpod (expect
<prefix>:devpod-<hash>), and its image id.3. Finding 5: a recreate rebuilds devpod's image
printf 'FROM dl673-test:latest\nRUN touch /stale-marker\n' | docker build -t dl673-test:latest -~/dl673/dln --lsmust name the workspace as stale.script -qfc "~/dl673/dln proj recreate" ~/dl673/recreate5.log(piping to tee hangs). Expect the line "proj runs an older image. Removed the tag that devpod built it under…".:devpod-image (new image id);/stale-markerexists in the new container;~/dl673/dln --lsno longer says stale.docker run -d --name dl673-hold <tag> sleep 600), and confirm whetherdocker image rm --force <tag>still only untags. Report what docker says. Remove dl673-hold after.4. Finding 3: a restarted agent keeps its line
DEVLAUNCH_HERDR=1 aid-next proj 'Remember the word pelican. Reply ok.'Answer the folder-trust and Bypass prompts withherdr agent send-keys <pane> down enter/enter. If it says "Not logged in", ask the user to run /login there (known issue, finding 10). Wait until herdr reports it idle. Record the session id (agent_resume.argv in herdr's session.json beside $HERDR_SOCKET_PATH).--resume <id>.pane wait-outputmatches the typed command; use--regex '(?m)^RC=\d'.herdr agent prompt --timeoutneeds--wait. Claude Code refuses a baresleep 120; usetimeout 120 tail -f /dev/null.5. --refresh-stale
~/dl673/dln --refresh-stale; echo RC=$?→ "Refreshing proj", agent back, exit 0. Then run it again after one more stale rebuild: it must refresh again (the line was kept), not skip with "no saved line".~/dl673/dln --refresh-stale; echo RC=$?→ "No workspace runs an older image…", exit 0.6. Clean up
~/dl673/dln proj rm --force;DEVPOD_HOME=~/dl673/devpod devpod listmust print [].docker rmievery dl673-test:* and proj-:devpod- image you made. Close your herdr tabs.rm -r ~/dl673(it can hold a .credentials.json). Remove ~/.local/bin/dl-next, ~/.local/bin/aid-next and ~/.local/share/devlaunch-dev.7. Fix what failed
For each check in steps 3–5 that failed: write a failing test first (the real shapes: devpod writes ContainerDetails.ID, herdr pane ids are base32 like w3:pAX), fix it, and run the test suite again. Run the host check again for that step.
8. Push and bring the stack to green
gh pr checks 676 --repo blooop/devlaunch --interval 60or slower (the GitHub token quota is shared). Fix any failure. Thepublic-apijob checks rust/devlaunch-core/public-api.*.txt, which were edited by hand: if it fails, run scripts/public-api-snapshots.sh and commit the result.gatejob also needsreview: "Nothing reviewed this pull request. Post a wf-review report and re-run this job, or label 'no-external-review'". Do the review properly (a wf-review report as the repo's docs describe it). Do not add the label to skip it.9. Report
Post one comment on #677 with: the results of steps 1, 3, 4 and 5 (the exact dl output lines, exit codes, pass/fail per check), what you fixed, the commits you pushed, and the CI state of #676. Do not name the machine or company. End with a ledger: ✅ done (with evidence), ❌ failed (with the output), ⛔ blocked (on what). When the stack is green and reviewed, close #677 with a link to that comment.
fix: dl read devpod's container id and herdr's pane ids in the wrong …
fix: the stale check missed images that left the store and compared t…
fix: a recreate kept devpod's derived image and lost the agent's line
blooop commentedon Oct 15, 2026
Report: the real-host test of the #673 stack (2026-10-15)
I followed the takeover prompt. The host has docker 29.8.0 with the containerd image store, devpod v0.26.3 and herdr 0.9.5. Every
dl-nextcall went through the~/dl673/dlnwrapper, which sets a scratchDEVPOD_HOMEandXDG_CACHE_HOME.Step 1: build — PASS
The three patches applied cleanly to
f5551cedin the given order, andcargo fmtran after them.dl-next --versionprinteddl 0.59.4-dev.cargo test -p devlaunch-core --lib: 2232 passed, 4 failed. The 4 failures areflows::lifecycle::teststests:a_host_that_prunes_on_fetch_does_not_turn_a_merged_branch_into_unpushed_worka_site_pushed_by_url_does_not_stop_a_plain_rmcommits_pushed_by_url_do_not_stop_a_plain_rma_remote_that_cannot_be_reached_leaves_the_refusal_standingThe same 4 fail on the unpatched
f5551ced, and they also fail withGIT_CONFIG_GLOBAL=/dev/null. CI'srustjob passes them, so this host causes the failures.Step 2: test project — PASS
The workspace came up on
proj-736a5:devpod-1c27a15a38d2a6944d1f7f3d59e73834, imagesha256:b4bedc152c49….Because of finding 6, I ran
docker tag dl673-test:latest dl673-test:prev-Nbefore each stale rebuild.Step 3: finding 5, a recreate rebuilds devpod's image — PASS
--lsprintedproj runs an older image than dl673-test:latest now names. Run 'dl proj recreate' to move it.and exited 0.proj runs an older image. Removed the tag proj-736a5:devpod-1c27…3834 that devpod built it under, so the recreate builds the image again from the newer one.and exited 0.b4bedc15, the new image474d8f8abuilt and tagged, the old container stopped and destroyed, the new container started. docker gave no error during the untag./stale-markeris in the new container.--lsno longer says stale.dl673-holdrunning on the tag,docker image rm --force <tag>printed onlyUntagged: proj-736a5:devpod-1c27…3834and exited 0. Both containers kept running. The image stayed inspectable by id, withRepoTags [].Step 4: finding 3, a restarted agent keeps its line — PASS
ok, and the session id was170467e0-…. herdr'sagent_resume.argvheld--resume 170467e0-….The recreate ends the agent in pane wG:p4P of proj; dl starts it again, resumed, once the new container is up.and thenStarted the agent in pane wG:p4P again.It did not print "herdr did not keep the line".session.jsonstill held--resume 170467e0-….Pelican.Finding 4 is confirmed. The saved line starts with plain
dl, so a restart runs the releaseddl. For the test, adlshim on PATH in that pane only sent the call todl-next.Step 5: --refresh-stale — PASS
Refreshing proj: it runs an older image than dl673-test:latest now names.,Started the agent in pane wG:p4P again.andRefreshed 1 stale workspace, skipped 0, and 0 failed., and exited 0. The second run refreshed again; it did not skip with "no saved line". Onlyprojwas refreshed.No workspace runs an older image than its reference names.and exited 0.A check without the finding 6 workaround: after the old base left the store,
--lsstill named the workspace stale.Other things seen (not fixed)
[?1047l[<u[?2004l…) appear in dl output before the hostname line and before "Started the agent…".--lsshows UTC, not local time.upprintedproj: the hostname setup stage exited 1.Step 7: fixes
No check in steps 3 to 5 failed, so step 7 had nothing to fix. The reviews found two doc defects, and I fixed both.
Commits pushed to
feat/refresh-stale34ec9140fix: dl read devpod's container id and herdr's pane ids in the wrong shape (findings 1, 2)3fa0eceffix: the stale check missed images that left the store and compared times as text (findings 6, 7)c94b5e7efix: a recreate kept devpod's derived image and lost the agent's line (findings 3, 5)fc6dc67dfix: the derived-image helper took launch_refusal's doc comment (review of feat: dl --refresh-stale recreates the stale workspaces that are idle #676)8b5fb9bbdocs: herdr's done status means idle, not ended (review of feat: dl <ws> recreate starts the agents it ends again #675)The tree at
c94b5e7eis byte-identical to the tree tested on the host.CI and review
8b5fb9bb: all checks pass, includingpublic-api,reviewandgate. The description has a new section, "Found on a real host". The wf-review report on #676 has the verdict Comment, with no blocking findings.donepanes.Nothing is merged; the merge is for the maintainer.
Ledger
f5551cedtoo.--refresh-stale, 5.1 and 5.2 pass. Each run exited 0.[], and every test image, the herdr tab,~/dl673(copied credentials included),dl-next,aid-nextanddevlaunch-devare removed.dl, so a resume runs whicheverdlis on PATH. This matters only with a dev build installed beside the released one./proc/<pid>/statparser, the flows layer reading docker stderr, a swallowed thread panic atlaunch.rs:2998, andTakencounted asKeptwhensession.jsoncannot be read.🤖 Generated with Claude Code
blooop commentedon Oct 15, 2026
The stack is green and reviewed. Report: #677 (comment)