Skip to content

fix(e2e): skip container dev without a container runtime and fail fast on startup errors - #2413

Open
notgitika wants to merge 1 commit into
refactorfrom
fix/e2e-container-dev-runtime-skip
Open

notgitika wants to merge 1 commit into
refactorfrom
fix/e2e-container-dev-runtime-skip

Conversation

@notgitika

Copy link
Copy Markdown
Contributor

Description

Every ci.yml push run on refactor since #2339 landed fails exactly one e2e test on both Linux and macOS: local invocation > 'py_strands_container' runs locally (37/38 pass, including deploy + cloud invoke of the same template). Latest example: run 36069970196.

Root cause. agentcore dev runs build: Container runtimes through ContainerDevRunner, which needs docker, podman, or finch. Neither e2e runner has a usable daemon:

  • Linux runs on the agentcore-e2e CodeBuild project, which has privilegedMode: false, so the docker CLI in the image has no daemon.
  • macOS runs on the GitHub-hosted macos-26-arm64 image, which ships no docker/podman/finch (and cannot run one).

The supervisor prints [py_strands_container] Agent 'py_strands_container' failed to start: 'container runtime' was not found on your PATH. ... on stderr, but dev keeps serving the other eight agents and never exits. The test only watched for an is running on port N line and for the process exiting, so it retried until its 162s deadline. #2398 made the full dev output visible in the error; this PR acts on it.

Change.

  • e2eTest/helpers/container.ts: probe docker|podman|finch info (a reachable daemon, unlike --version, which also passes on the CodeBuild image).
  • templates.test.ts: tag each template with its build; the runs locally case uses test.for and context.skip(...) for Container templates when no daemon is reachable. Deploy, cloud invoke, and remove for the container template are unchanged and still run.
  • templates.test.ts + helpers/retry.ts: record Agent '<name>' failed to start: <reason> lines from dev output and throw a NonRetryableError carrying that reason, so a real startup failure surfaces in seconds with the actual message instead of a timeout.
  • e2eTest/README.md: note the container runtime requirement.

Follow-up (not in this PR). To exercise container dev in CI on Linux, enable privileged mode on the agentcore-e2e CodeBuild project. macOS will keep skipping.

Related Issue

No tracking issue; follows up on #2394 and #2398.

Documentation PR

Not applicable (e2e README updated here).

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe):

Testing

Ran the local-invocation slice of the e2e suite (-t "can be added|runs locally") against the built CLI on macOS, with a private vitest config that omits the CloudFormation cleanup setup file, in two host configurations:

  1. No container runtime (only a finch stub that exits 1 on PATH): the container case is reported as ↓ 'py_strands_container' runs locally 1ms [no container runtime (docker, podman, or finch) is available on this host], and dev's own output shows the failed to start line. All seven Python runtimes ran and were invoked locally. (The two TypeScript templates failed at npm install in the add step on my machine, before this code runs; re-running npm install in the scaffolded directory succeeded, so that was a transient local issue.)
  2. Runtime visible to the test but unusable by the CLI (docker stub where only info exits 0): the case fails in 504ms with Runtime 'py_strands_container' failed to start: 'container runtime' was not found on your PATH. Install Docker (...), Podman (...), or Finch (...). instead of after 162s with no reason.
  • I ran bun test (3563 pass, 0 fail)
  • I ran the relevant end-to-end tests with bun run test:e2e, or explained why they are not applicable (local-invocation slice as described above; the full suite will run on this branch's push CI)
  • I ran bun run typecheck
  • I ran bun run lint:check
  • I ran bun run format:check
  • I ran bun run build
  • If I modified src/assets/, I updated affected snapshots with bun test <test-file> --update-snapshots and committed them (not applicable)

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published (fix(e2e): retain local dev output on failure #2398 is merged)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

@github-actions github-actions Bot added the size/s PR size: S label Sep 25, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 25, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 25, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Looks good

Solid, well-scoped test-infra change:

  • detectContainerTool probes docker/podman/finch with info (correctly gates on a reachable daemon, unlike --version) in the same order as src/core/dev/container.ts.
  • The failure-line regex matches the exact format emitted by Supervisor (Agent '<name>' failed to start: ...), so unrecoverable startup errors now fail fast via NonRetryableError instead of waiting out the invoke timeout.
  • Switching to test.for is required to get the skip context; the container skip is scoped to the "runs locally" case only, so the "can be added to a project" case still exercises the container template on all hosts.

No changes required.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 25, 2026
…t on startup errors

The 'py_strands_container' runs locally case has failed on every push to
refactor since #2339 landed: neither e2e runner has a usable container
daemon (the CodeBuild project runs without privileged mode, and the
macos-*-arm64 GitHub images ship no docker/podman/finch). `agentcore dev`
reports "failed to start" for that runtime but keeps serving the others, so
the test waited out its full retry deadline with no useful message.

- probe docker/podman/finch with `<tool> info` and skip local invocation of
  Container-build templates when no daemon is reachable
- record "Agent '<name>' failed to start" lines from dev output and stop
  retrying immediately with that message instead of waiting for the deadline
@codecov-commenter

codecov-commenter commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.25%. Comparing base (f035e3f) to head (c0d8c1a).

Additional details and impacted files
@@            Coverage Diff            @@
##           refactor    #2413   +/-   ##
=========================================
  Coverage     97.25%   97.25%           
=========================================
  Files           612      612           
  Lines         41019    41019           
=========================================
  Hits          39893    39893           
  Misses         1126     1126           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@notgitika
notgitika force-pushed the fix/e2e-container-dev-runtime-skip branch from 698b828 to c0d8c1a Compare September 25, 2026 19:18
@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Sep 25, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 25, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 25, 2026
@Hweinstock

Hweinstock commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

I believe the CI does have docker based on the latest error showing docker logs:
https://github.com/aws/agentcore-cli/actions/runs/36168595503/job/108182391001, but maybe its not usable?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants