Skip to content

fix(e2b): real command exit codes, pod-IP wait after resume, warm-pool status - #653

Merged
xiaods merged 4 commits into
mainfrom
fix/e2b-exit-code-and-pod-ip
Oct 19, 2026
Merged

xiaods merged 4 commits into
mainfrom
fix/e2b-exit-code-and-pod-ip

Conversation

@xiaods

@xiaods xiaods commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Three defects found by running the official E2B SDK examples (docs.e2b.dev/client) against a live cluster. The reproducer is tests/e2b_official_aligned.py.

1. commands.run always reported exit_code=-1 (or 0)

Stdout/stderr were correct, exit codes were not — the blocker for real agent workflows.

Root cause: sandboxd/src/main.zig:setupSignals() installed SIGCHLD as SIG_IGN + SA_NOCLDWAIT. Both make the kernel auto-reap children, so wait4 on any child returns -ECHILD with the status word untouched, and every exec silently fell back to its hardcoded value (-1 streaming, 0 buffered — so cmd; echo $? lied to every caller).

Fix: install a real (non-SIG_IGN) handler, which keeps children waitable. exec.zig/pty.zig/venv.zig already wait, so they now recover the true status. background.zig relied on auto-reap for its children, so it drains them with a detached reaper (its completion signal is unchanged — the BG_WRAPPER EXIT trap still writes the exit_code file). Signal deaths now report 128+signo like a shell instead of a bare -1.

Verified on Linux with PID-1 semantics (unshare -r --pid --fork --mount-proc):

case old new
/exec exit 42 exit_code:0 exit_code:42
/exec/stream exit 7 {"exit":-1} {"exit":7}
/exec/stream python3 40+2 {"exit":-1} stdout 42, {"exit":0}
background exit 5 5 5 (unchanged)
zombies 0 0 (reaper holds parity)

2. First call after a resume: "session has no pod IP"

wakeForTraffic auto-resumed a paused sandbox but returned the pre-resume session view, whose PodIp is empty — the old pod is gone and the new one has no address yet. Callers address sandboxd by pod IP, so E2B watch_dir and every other envd call failed. Fix: poll for the new pod IP (bounded, 30s) and return the fresh view. The same wait covers explicit resume(), where the phase is already running while the pod is still being created.

3. WarmPool .status was never written

The CRD has carried readyCount/pendingCount since KIP-30 but reconcileSinglePool never populated them, so kubectl get sandboxwarmpools showed an empty READY column. Now written every pass.

Tests

Every new test is a regression test — each fails without its fix and passes with it:

  • pkg/sandbox/e2b/gateway_podip_test.go — TestWakeForTrafficWaitsForPodIPAfterResume, TestWakeForTrafficRunningWithoutPodIPWaits
  • pkg/sandboxmatrix/controller_test.go — TestReconcileSinglePool_PopulatesStatus, TestReconcileSinglePool_ReportsPendingWhenFilling

No change to any pre-existing failure (the 12 sandboxd test failures on macOS — PTY tests plus a 1 MiB readAllFromFdTrunc off-by-one — are identical before and after; they are Linux-side and tracked separately).

…l status

Three defects found by running the official E2B SDK examples against a live
cluster (tests/e2b_official_aligned.py, aligned with docs.e2b.dev/client).

1. sandboxd: commands.run always reported exit_code -1 (streaming) or 0
   (buffered) even though stdout/stderr were correct.

   Root cause: main.setupSignals installed SIGCHLD as SIG_IGN plus
   SA_NOCLDWAIT. Both make the kernel auto-reap children, so wait4 on any
   child returns -ECHILD and leaves the status word untouched — every exec's
   exit code silently collapsed to its fallback.

   Fix: install a real (non-SIG_IGN) SIGCHLD handler, which keeps children
   waitable. exec.zig, pty.zig and venv.zig already wait, so they now recover
   the true status. background.zig relied on auto-reap for its children, so it
   now drains them with a detached reaper thread; its completion signal is
   unchanged (the BG_WRAPPER EXIT trap still writes the exit_code file).

   The wait sites also distinguish a real exit from a signal death, reporting
   128+signo the way a shell does instead of the bare -1 that hid "killed".

   Verified on Linux with PID-1 semantics (sandboxd run as PID 1 via
   `unshare -r --pid --fork --mount-proc`):

     /exec         exit 42    old: exit_code:0    new: exit_code:42
     /exec/stream  exit 7     old: {"exit":-1}    new: {"exit":7}
     /exec/stream  python 40+2                    new: stdout 42, {"exit":0}
     background    exit 5     old: 5              new: 5 (unchanged)
     zombies       old: 0                        new: 0 (reaper holds parity)

2. e2b: the first call after a resume failed with "session has no pod IP"
   (E2B watch_dir, files.stat, and any envd traffic on a paused sandbox).

   wakeForTraffic auto-resumed but returned the pre-resume session view, whose
   PodIp is empty because the old pod is gone and the new one has not been
   assigned an address yet. Callers address sandboxd by pod IP, so the request
   failed immediately.

   Fix: poll for the new pod IP (bounded, 30s) and return the fresh view. The
   same wait covers the explicit-resume case, where the phase is already
   running while the pod is still being created.

3. sandbox-matrix: WarmPool .status was never written.

   The CRD has carried readyCount/pendingCount since KIP-30 but
   reconcileSinglePool never populated them, so `kubectl get
   sandboxwarmpools` reported an empty READY column and operators had no
   in-cluster signal for whether the pool was serving.

   Fix: write readyCount (running + Ready condition) and pendingCount (the gap
   the pass is closing) each pass.

All new tests are regression tests: each fails without its fix and passes with
it. No change to any pre-existing failure.
@xiaods xiaods mentioned this pull request Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

977 tests  +4   950 ✅ +4   6m 49s ⏱️ +4s
120 suites ±0    27 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 09d70cf. ± Comparison against base commit 1435252.

♻️ This comment has been updated with latest results.

…us change

Code-quality follow-up on the e2b PR, flagged by DeepSource (Go) and
SonarCloud's quality gate. No behaviour change beyond removing one invented
field and one latent panic.

- Drop the duplicate PodReady helper. pkg/sandboxmatrix/grpc already exposes
  sandboxgrpc.PodReadyCondition with identical semantics, so the local copy
  was pure duplication (SonarCloud's duplication metric).

- Stop writing status["size"]. SandboxWarmPoolStatus declares only readyCount
  and pendingCount, so writing an undeclared field produced data the CRD
  schema drops on read, duplicating spec.size for nothing.

- Remove a latent panic. The status write indexed
  pool.Object["spec"].(map[string]interface{}) with an unchecked assertion,
  which panics on any pool whose spec is absent or not an object. Read it
  through the two-value form like the field above it.

- Rename the loop variable to index the slice rather than copying each pod,
  since PodReadyCondition takes a pointer.

- gofmt / zig fmt the touched files.

Pre-existing findings are left alone: pkg/sandboxmatrix/api/v1alpha1/types.go
and five sandboxd Zig files are already unformatted on main, and the 12
sandboxd macOS test failures are unrelated.
SonarCloud S1117 on gateway_podip_test.go lines 120 and 155: both declared a
variable solely to compare it against the empty string in the same condition.
The condition is the only use, so the temporary is pure noise — use the
expression directly.

The neighbouring `got` is kept: it is reported in the failure message.
SonarCloud's Quality Gate failed on 5.6% duplicated lines in new code, all of
it in controller_test.go: the two WarmPool status tests repeated the same
arrange/act/observe block — create the pool CR, seed the pods, reconcile, read
the status back — twice, which buried the one assertion each test actually
makes.

Extract warmPoolStatusAfterReconcile so the sequence exists once and each test
states only its own expectation. Same coverage, same A/B: both tests still fail
without the status write and pass with it.
@sonarqubecloud

Copy link
Copy Markdown

@xiaods
xiaods merged commit 721c0c5 into main Oct 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant