Skip to content

feat: carry a failing command's stderr snippet in the failed:<step> tag#65

Merged
kurok merged 1 commit into
mainfrom
phone-home-failure-detail
Jul 6, 2026
Merged

feat: carry a failing command's stderr snippet in the failed:<step> tag#65
kurok merged 1 commit into
mainfrom
phone-home-failure-detail

Conversation

@kurok

@kurok kurok commented Jul 6, 2026

Copy link
Copy Markdown

Summary

The bootstrap phone-home tag (ec2-github-runner:bootstrap) only ever named
the step that failed (failed:<step>) — zero detail on why. This has cost
real debugging time (a #63-class bug, and an unexplained warm-restart-only
failure at the configuring step where the real config.sh error was
invisible without live SSH/SSM).

  • gh_runner_phone_home_failed() (new, in PHONE_HOME_HELPERS) tees each
    shell's stderr into a scratch file and, on failure, appends a sanitized,
    single-line, ~150-char tail of it to the tag:
    failed:<step>failed:<step>:<detail>.
  • The scratch file is truncated on every successful step transition, so a
    step's detail only reflects that step's own commands (not noise left by an
    earlier || true-tolerated one, e.g. mount).
  • The value is hard-capped at 256 chars (the EC2 tag-value limit) regardless
    of how much garbage a failing command emits, and control bytes/newlines are
    sanitized to a single printable line.
  • Applied everywhere the phone-home pattern is used: buildUserData()'s
    outer + inner shells, buildReusableUserData()'s outer + inner shells, and
    its registerScript (per-boot re-registration for reuse: stop).
  • src/wait.js splits on the first : after the failed: prefix to
    recover step vs. detail (step names never contain :, so this is
    unambiguous even when the detail itself has colons, e.g. a
    config.sh: error: ... line). Fully backward compatible with the old
    no-detail failed:<step> shape — error.bootstrapDetail is simply absent.

Before / after

# before
failed:configuring

# after
failed:configuring:config.sh failed: Not configured. Run config.sh to configure a runner.

A note on the tee race

tee's copy into the scratch file runs via a background process
substitution, so reading the file immediately in the ERR trap can race and
see it still empty. gh_runner_phone_home_failed() closes this shell's end
of that pipe (fd 2 — the script is terminating anyway) and polls (bounded,
~200ms max) for the tee process to exit via kill -0, which only happens
once it has drained everything written to the pipe. Verified empirically
against both bash 3.2 and bash 5.3 — deliberately not a plain wait on the
tee PID, since that needs bash >= 4.4 and could hang indefinitely if tee's
write end ever blocked.

Test plan

  • tests/phone-home-detail.test.js (new): extracts the actual lines
    buildUserData() / buildReusableUserData() produce (outer root
    shells, both inner heredocs, and registerScript), splices in a
    fabricated failing command, and runs the result as a genuine bash
    script file against a fake curl (IMDS) + fake aws CLI — proving
    the real generated code truncates/sanitizes correctly, including one
    case with ~3.6KB of multi-line, control-character/ANSI-laden stderr,
    and a pathological ~50KB newline-filled case that must stay under 256
    chars with no embedded newlines. Also covers the backward-compatible
    no-detail case.
  • tests/wait.test.js: detail surfaced in the thrown error, core.error,
    and error.bootstrapDetail; a detail containing colons splits
    correctly; backward compat with old-style failed:<step> (no
    bootstrapDetail, unchanged message).
  • npm test — 237 passed, 18 suites.
  • npm run lint — clean.
  • npm run package rebuilt twice — byte-identical (sha256sum match),
    diff scoped to the src/aws.js / src/wait.js changes only.

The phone-home failure tag only ever named the step that failed, giving
no clue why. gh_runner_phone_home_failed() now tees each shell's stderr
into a scratch file and appends a sanitized, single-line, ~150-char tail
of it to the tag as failed:<step>:<detail>, capped well under the 256-char
EC2 tag-value limit. Applies to buildUserData(), buildReusableUserData(),
and its registerScript. src/wait.js splits on the first ':' after the
failed: prefix to recover step vs. detail, staying backward compatible
with the old no-detail shape.

Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
@kurok kurok merged commit 453ef2e into main Jul 6, 2026
5 checks passed
@kurok kurok deleted the phone-home-failure-detail branch July 6, 2026 18:06
kurok added a commit to namecheap/terraform-provider-namecheap that referenced this pull request Jul 6, 2026
The warm-restart path (unlike the already-verified cold-launch path)
still fails at the configuring step with zero detail beyond the step
name. Since buildReuseUserData() regenerates the identical bootstrap
script buildUserData() uses (already confirmed sed-fix-correct), the
warm-restart-specific failure cause is otherwise invisible without live
instance access. namecheap/ec2-github-runner#65 adds a captured stderr
snippet to the failed:<step> phone-home tag (now readable via this
repo's existing debug:true diagnostic flag), merged as 453ef2e, v4.0.0
moved to match. Re-pinning this PR's 4 usages so the next warm-restart
run surfaces the real config.sh error instead of another opaque
'bootstrap failed during configuring'.

Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant