clonevm_exec_test hangs after 'second stage': post-exec rendezvous never completes (1 boot in 30, aarch64 cortex-a72)

[ryanbreen]

Summary

One boot in 30 on the aarch64 cortex-a72 service-sequence profile stalls
inside clonevm_exec_test: it prints CLONEVM_EXEC_TEST: second stage and
never reaches CLONEVM_EXEC_TEST: post-exec rendezvous complete. init is
blocked in waitpid on it, so the whole service sequence stops there — bsshd,
bwm, telnetd and bounce are never spawned and the gate times out at 45s
with UNATTRIBUTED.

The guest is not dead: heartbeats continue, [net-rx-counters] samples keep
advancing, and the strand census keeps sampling to the wall clock. It is one
process wedged in a rendezvous, not a hang.

Serial

Preserved in-repo:
docs/planning/green-program/sockets/serials/aarch64-clonevm-second-stage-stall-20260829.txt

Green boots on the same run reach:

CLONEVM_EXEC_TEST: second stage
CLONEVM_EXEC_TEST: post-exec rendezvous complete
CLONEVM_EXEC_TEST: post-exec futex keys derived
CLONEVM_EXEC_TEST: PASS

The failing boot stops after second stage. Its cloned thread exits normally
([syscall] exit(0) pid=99 name=thread-99); the process itself never exits.

The strand census on the stalled boot reports live scheduling irregularity that
may or may not be related, recorded here rather than interpreted:

[SCHED_STRAND_ORACLE:aarch64:...:stranded=0:...:nonprogress=1:
 queued_on_nondispatching_cpu=1:worst_queued_nondispatch_ms=1599:
 worst_cpu_scheduler_silence_ms=1642:worst_silence_cpu=0]

Rate, and what it is not

tree	profile	boots	occurrences
fix/568-poll-wedge	cortex-a72	30	1
fix/568-poll-wedge	max	10	0
main @ 6d0d2ac5	cortex-a72	30	0

1/30 against 0/30 does not discriminate, so this is not attributed to the
branch and not attributed to main. It is filed rather than absorbed
because R52 makes an unexplained boot failure gate-failing, and it now has a
name and a serial instead of a shrug.

It is not the branch's poll work: the #568 oracle PASSED on this very boot
([POLL_TCP_ORACLE:PASS:stages=3:idle_ms=128:late_ms=82:park_ms=81:attempts=1],
[init] poll_tcp_oracle exited pid=95 code=0) several service steps earlier, and
kernel/src/task/scheduler.rs on that branch is byte-identical to main.

Where to start

The marker that never arrives is the post-exec rendezvous, which is futex-based.
The adjacent history is #602/#584 (clonevm rendezvous and FUTEX_WAIT lost
wake, closed by #604) and #586 (starved wake-loss misreport). This may be a
surviving instance of that family at a much lower rate, or a distinct
exec-side ordering problem. Reproducing it will need a soak: at 1/30 a 200-boot
cortex-a72 battery is the smallest sample that says anything.

Found while re-doing #568's evidence (green program, sockets).

[ryanbreen]

Round-3 aarch64 observation: did not recur in 20 boots

run-aarch64-service-sequence-gate.sh --boots 10 --profile both --rebuild at
fix/568-poll-wedge @ efe08b17 (the round-3 merge candidate, which reinstates
the Scheduler::schedule() reorder — shared code, hence the re-run):

• 20/20 GREEN, UNATTRIBUTED=0, both profiles PASSED.
• CLONEVM_EXEC_TEST: second stage reached and completed on every boot; the
  post-exec rendezvous stall this issue describes occurred 0/20, against
  1/30 in round 2 on the same host and profile.
• Every gate-failing bucket zero: 575, 576, 626, resume-PC refusals,
  per-CPU stack aliens, CPU identity splits, ret-stage refusals.
• Reported-not-gated: ctx596_divergence 4 marker lines across 4/20 boots;
  lr_nontext 122 lines across 20/20.

0/20 against 1/30 does not discriminate, so this is a non-recurrence, not
evidence of a fix — nothing in this round touches the clonevm rendezvous. Read
it only as: at a ~3% prior, twenty more boots produced no new specimen and no
new information about the cause.

One adjacent thing worth recording here, because it is the x86 sibling of the
same test: clonevm_exec_test is also being saved blocked in a kernel wait and
never restored on x86, on pristine main as well as on branches. That shape
now has its own issue, #695, and its own detector
(scripts/x86-strand-census.sh). It may or may not share a cause with this one;
nothing measured so far establishes that either way.

[ryanbreen]

Status at the #568 landing

Unchanged and still open. Recording where it stands so it is not lost behind the
#568 work that surfaced it.

• Rate at the round-3 bytes: 1 boot in 30 on the aarch64 cortex-a72
  service-sequence gate. The 50-boot confirmation run of that gate at the merge
  candidate was 50/50 GREEN with UNATTRIBUTED=0, so this did not recur there.
• Serial preserved in-repo:
  docs/planning/green-program/sockets/serials/aarch64-clonevm-second-stage-stall-20260829.txt.
• Independent of sys_poll wedges guest on connected TCP socket #568's defect and of its fix: the stall is in the post-exec
  rendezvous after second stage, not in the blocking-poll path, and the
  poll_tcp_oracle marker is present and PASS on the affected boot.
• Not allowlisted anywhere. Under the standing rule a recurrence scores
  UNATTRIBUTED and is gate-failing, which is the right behaviour and should stay.

Possibly adjacent: #695, clonevm_exec_test saved blocked in a kernel wait and
never restored on pristine x86 main, named by the new
scripts/x86-strand-census.sh. Same program, different architecture, both
stalling around the same rendezvous. Worth checking whether they are one defect
before either is diagnosed separately.

[ryanbreen]

2:1 triage sweep, 2026-08-29 — leave-with-evidence

Read-only verification pass over this issue at main 782ab96f. The sweep's verdict, verbatim:

#690 — clonevm_exec_test hangs after 'second stage', aarch64, 1/30

Verified current — issue's last comment (#568 landing status) already reflects today's main:
50/50 GREEN on the SS gate confirmation run at the merge candidate, UNATTRIBUTED=0. Independent
of #568's fix. Possibly related to x86's #695 (post-exec rendezvous). Correctly left open.
Verdict: LEAVE-WITH-EVIDENCE — no new evidence to add this pass; restate that main is still
at the byte-for-byte state the last comment describes (782ab96 is the #568-landing merge base,
confirmed by git log -1 main).

[ryanbreen]

New occurrence: green-arc-4 (TTY-aarch64) confirmation slot, run-aarch64-service-sequence-gate.sh --profile both --boots 25 --rebuild at feat/green-tty @ 63e31a2, cortex-a72 profile, boot 19/25, timeout at 45s (24/25 GREEN otherwise). Matches this issue's signature exactly: last line CLONEVM_EXEC_TEST: second stage, no post-exec rendezvous complete, init blocked in waitpid, strand-oracle shape nonprogress=1:queued_on_nondispatching_cpu=1. Rate 1/25 here vs 1/30 in the original filing -- same rate class.

This occurrence was initially misfiled as a new issue (#706, now closed as a duplicate of this one -- its RCA sketch pointed at TOMBSTONE_CENSUS, which is just the periodic census ticking after the real stall, not the hang point).

Serial preserved at docs/planning/green-program/tty/serials/aarch64-clonevm-second-stage-stall-boot19-20260830.txt.

The service-sequence gate now pre-registers this signature as its own attributed bucket (690, still gate-failing -- attribution is not a tolerance, same pattern as #641) instead of UNATTRIBUTED, so future recurrences report under this issue: docker/qemu/run-aarch64-service-sequence-gate.sh @ 42fb4d8.