Repository navigation
feat(trainer): pod resource samples + vanish post-mortem — prove restart vs leak - #459
Merged
Merged
Conversation
…art vs leak Three legs died with the pod dark 6-15 min and back with its run dir gone (steps 14400-16450): container recreated by the provider, or a leak of ours? Every detached poll now reads the pod's resources (MemAvailable, cgroup memory use/limit, free disk, kernel uptime, pid 1 start); a vanish runs a read-only post-mortem after the log fetch and before teardown; the samples prove CONTAINER RESTARTED / HOST REBOOTED. Logged every 30 min and on reconnect. The failure text the fault classifier bills on is unchanged — the evidence rides only into the saved failed-leg log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BMT4D7eZUKUF2gocYLVFmj
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owner-requested 2026-11-13 ("Are u sure it's not a bug on our end" → "do that"). Trainer-side only: no worker-image change, no consensus field, no billing change. Takes effect at the next trainer restart (owner's go, quiet window).
Why
Three legs (5G6jNi6YAWfD Nov 5, 5GMzsPiUhy6g + 5Ctyc5w8gwif tonight) died the same way: ssh dark 6-15 min, pod back, worker gone, run dir and logs gone (empty tails), at steps 15600 / 16450 / 14400 of 23841. Empty tails mean the filesystem was reset — a recreated container — but the step clustering also fits a resource leak of ours killing the container. Nothing on record could tell the two apart, and the pod is torn down seconds after.
What
RES:line: hostMemAvailable, the container cgroup's memory use and limit (v2 or v1), free disk in the workdir, kernel uptime, pid 1's start (ticks after boot). Token stays the last line;parse_resourcesnever raises on missing fields.pod resources (step N): mem_avail=… cg_mem=…/… disk_free=… uptime=… pid1_start=…) and on every reconnect after a dark spell.resource_verdict: kernel uptime went backwards ⇒HOST REBOOTED; pid-1 start moved ⇒CONTAINER RESTARTED. Added to the quarantine reason and the reconnect line.GONE, after the log fetch and before teardown, one read-only ssh: resources,uptime -s, pid-1 start + comm, whether the run dir survived (+ls), the run root listing, kernel OOM/Xid lines (dmesg/journalctl -kwhen readable),free -m,df -h. Best-effort; a failure is recorded, never changes the leg's outcome.faults.classify_*) reads is byte-identical to before — a dmesg "Killed process" line must never move infra-vs-miner billing. The verdict, post-mortem and last hour of samples go only into_train_work/_failed_legs/<hotkey>-<ts>.log(tail-kept, so they survive the size cap).Tests
tests/unit/test_pod_vanish_evidence.py: the real poll command parses on Linux; missing fields; restart/reboot verdicts; a scripted dark→GONE leg runs the post-mortem after the fetch, keeps the classified text exactly the old shape, carries verdict + samples inproc.postmortem; post-mortem failure is harmless; a clean exit makes no extra ssh call; the evidence file keeps it.test_miner_cost_protectionspoll-output test updated for theRES:line.Checked on the box (
python -P, bash only): the poll line parses (mem_avail=2.7G … disk_free=94.4G uptime=10549190s pid1_start=39), the post-mortem prints boot, pid 1, run dir, OOM lines. ruff clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01BMT4D7eZUKUF2gocYLVFmj