feat: warm pools — reuse stopped instances (reuse: stop)#59
Merged
Conversation
…ob starts Every job paid the full cold-start tax (launch + boot + yum + ~100 MB runner download + registration, ~2-4 min). Stopped instances restart in seconds with disk intact. New reuse: stop lifecycle: - Start: find a stopped pool instance (strict: signature + reuse-pool-tag + instance type + arch) → StartInstances + re-register; empty pool → cold launch that joins the pool. Warm reuse applies for count 1; batches cold-launch. - Re-registration without leaking the token: the reuse bootstrap installs a per-boot systemd service that reads a fresh registration token/label from the instance's IMDS user-data each boot; warm start rewrites that user-data (ModifyInstanceAttribute) with a fresh token, so the token never lives in a readable tag (IMDS is instance-local). - Stop: reuse:stop stops instead of terminating; recycles (terminates) once the instance has served reuse-max-cycles jobs (default 20; cycle counter tag). - Reaper: cleanup mode now drains stopped pool instances older than reaper-stopped-max-age (default 1440 min) — decision matrix + listing extended to stopped state. - Signature tags gain pool + cycles. reuse:terminate default is byte-identical to today (regression). Security boundary documented: reuse carries disk state between jobs — unsafe for public/untrusted-PR repos. Tests: findStoppedPoolInstance strict matching + near-miss safety, warmStart ordering, stop/cycle helpers, reuse user-data re-registration hook + default- variant regression, reaper stopped-drain matrix, config validation (224 tests total). README "Warm pools" section + security warning; action.yml inputs. Note: the real-infra warm-start timing (>=60% faster) is a manual checklist item — can't provision EC2 in CI. Orchestration + user-data generation are unit-tested. Closes #48 Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
This was referenced Jul 2, 2026
kurok
added a commit
that referenced
this pull request
Jul 2, 2026
Adds CHANGELOG.md documenting the 4.0.0 capability wave (features #50-#59) and its breaking changes (ec2:CreateTags now always required; cleanup-on-start-failure defaults to terminate; DeleteOnTermination forced). Updates all README/docs/example usage snippets from @V3 to @v4 and adds a Changelog pointer. Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #48 — roadmap (#49) Phase 2, the final and most design-sensitive item. Depends on the reaper (#42, drain semantics) and bootstrap diagnostics (#41, re-registration attribution).
Problem
Every job pays the full cold-start tax (~2–4 min: launch + boot +
yum+ ~100 MB runner download + registration). Stopped instances restart in seconds with disk intact and cost only EBS while stopped.Design
reuse: stop(defaultterminate), set on both start and stop:findStoppedPoolInstance(strict — signature +reuse-pool-tag+ instance type + arch, near-misses skipped) →StartInstances+ re-register; empty pool → cold launch that joins the pool. Warm reuse applies forcount: 1; batches cold-launch.ModifyInstanceAttributewith a fresh token — so the token is never in a readable tag (IMDS is instance-local, IMDSv2-gated).reuse-max-cyclesjobs (default 20; a cycle-counter tag).cleanupreaper now terminates stopped pool instances older thanreaper-stopped-max-age(default 24h) — decision matrix + listing extended tostoppedstate.max-lifetime-minutesstill bounds wall-clock age.Safety
terminateis unaffected.reuse: terminatedefault → user-data + behavior byte-identical to today (regression test).Tests / build
findStoppedPoolInstance(match + 4-way near-miss safety + empty),warmStartInstanceordering (modify→tag→start),stop/cycle helpers, reuse user-data re-registration hook (systemd + IMDS-read + token embed) + default-variant regression, reaper stopped-drain matrix, config validation. 224 tests pass (was 203). Lint clean, dist rebuilt (verify-dist green). All three bash layers of the reuse user-data passbash -n.Not automated (per issue)
The real-infra warm-start timing (≥60% faster, numbers recorded) is a manual checklist item — EC2 can't be provisioned in CI. Orchestration + user-data generation are unit-tested; the re-registration mechanism warrants the design review the epic flagged.
Out of scope (stated)
Always-on pre-warmed pools; cross-repo pools; AMI-baking pipelines.