Skip to content

ci: cache tool downloads of the container tests in a local proxy - #7670

Merged
viceice merged 4 commits into
mainfrom
viceice/ci/download-proxy
Oct 8, 2026
Merged

viceice merged 4 commits into
mainfrom
viceice/ci/download-proxy

Conversation

@viceice

@viceice viceice commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Changes

The container tests now download tools through a caching proxy on the runner, so stages and bake retries in a job share their downloads:

  • New local action .github/actions/download-proxy: starts nginx as a caching reverse proxy and sets CONTAINERBASE_CDN=http://host.docker.internal:8099, which the builds already pass through docker-bake.hcl. The CLI rewrites https://<host>/<path> to <cdn>/<host>/<path>, the proxy forwards it upstream and caches the response; redirects are followed inside the proxy, so GitHub release assets are cached too.
  • Like the apt proxy (squid-deb-proxy) it only accepts the runner and the docker networks, and only fetches from a fixed list of download hosts (allowed-hosts.map), checked for every redirect hop as well; TLS is verified.
  • Used in distro, base-arm64 and lang (after the base image build, so its cache key stays the same as in base); base and the release job don't use it. On failure, a step prints the proxy's rejected and failed requests, and every run adds the cache hits and misses to the job summary.
  • npm, pip and gem keep going direct (their CDN flags stay unset), and the node test stage with URL_REPLACE_* unsets the CDN to keep testing the url replacements.

Context

Please select one of the following:

  • This closes an existing Issue, Closes: #
  • This doesn't close an Issue, but I accept the risk that this PR may be closed if maintainers disagree with its opening or implementation

Related: #972 (uses the same CONTAINERBASE_CDN rewrite, and follows redirects inside the proxy) and #7 (the proxy log lists the hosts the test builds download from).

AI assistance disclosure

Did you use AI tools to create any part of this pull request?

Please select one option and, if yes, briefly describe how AI was used (e.g., code, tests, docs) and which tool(s) you used.

  • No — I did not use AI for this contribution.
  • Yes — minimal assistance (e.g., IDE autocomplete, small code completions, grammar fixes).
  • Yes — substantive assistance (AI-generated non‑trivial portions of code, tests, or documentation).
  • Yes — other (please describe):

Written by Claude Opus 5.5 in Claude Code.

Use of AI in replying to PR comments

Who answers review comments:

  • @viceice will read and reply directly. Name the account.
  • An agent will draft replies and @username will read them before they are posted. Name the account.
  • An agent will draft replies and reply autonomously. This is heavily discouraged, and we prefer that there are humans in the loop
  • Nobody has explicitly committed to replying.

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified tests

The nginx config was tested locally in docker: GitHub redirect chains, encoded paths (kustomize's %2F), dl.k8s.io → cdn.dl.k8s.io, cache hits, 403 for other hosts, and an install-tool node through CONTAINERBASE_CDN. The first full CI run passed with the proxy in all docker test jobs on both architectures, so no download host is missing.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Build and test workflows now route selected downloads through a caching proxy.
    • Workflow summaries include download cache statistics, and failed runs provide additional proxy diagnostics.
    • Proxy availability is checked before tests run, helping identify download issues during CI.

Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 52 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: containerbase/coderabbit/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2a7cc6ed-6b95-4cd5-a795-cea18f054a56
📥 Commits

Reviewing files that changed from the base of the PR and between 1ac7df5 and 5648e41.

📒 Files selected for processing (3)
  • .github/actions/download-proxy/nginx.conf
  • .github/actions/download-proxy/stats.sh
  • .github/actions/download-proxy/test.sh

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: containerbase/coderabbit/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f2860dfe-3354-4569-a260-fa9b97ba83ca
📥 Commits

Reviewing files that changed from the base of the PR and between d2989fd and 1ac7df5.

📒 Files selected for processing (3)
  • .github/actions/download-proxy/nginx.conf
  • .github/actions/download-proxy/stats.sh
  • .github/workflows/build.yml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Adds an Nginx download proxy and integrates it into the distro, base-arm64, and lang build jobs. The jobs collect cache statistics and report proxy errors after failures. The Node.js testp stage clears CONTAINERBASE_CDN.

Changes

Download proxy

Layer / File(s) Summary
Proxy setup, routing, and caching
.github/actions/download-proxy/action.yml, .github/actions/download-proxy/nginx.conf
The action sets up Nginx, exports the proxy URL, and checks the Node.js distribution index. Nginx restricts client access and upstream hosts, verifies TLS, follows specified redirects, and caches responses.
Build job integration and proxy reporting
.github/workflows/build.yml, .github/actions/download-proxy/errors.sh, .github/actions/download-proxy/stats.sh, test/node/Dockerfile
The distro, base-arm64, and lang jobs invoke the action, collect cache statistics, and report proxy errors after failures. The lang job starts the proxy after the base-image build and before its architecture-specific test. The Node.js testp stage clears CONTAINERBASE_CDN.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant BuildJob
  participant DownloadProxyAction
  participant Nginx
  participant NodeDistribution
  BuildJob->>DownloadProxyAction: invoke local action
  DownloadProxyAction->>Nginx: configure and start proxy
  DownloadProxyAction->>Nginx: check Node.js distribution index
  Nginx->>NodeDistribution: forward allowlisted request
  NodeDistribution-->>Nginx: return response
  Nginx-->>DownloadProxyAction: return check response
Loading

Merge Risk: ⚪ Minimal · up to 1ac7d

The proxy is wired into the intended container tests, with no established issue requiring a fix before merge.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding a local proxy to cache tool downloads for container tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@viceice
viceice marked this pull request as ready for review October 8, 2026 15:36
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Cache container-test tool downloads through a runner-local proxy

✨ Enhancement ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Share tool downloads across container-test stages and bake retries using a runner-local cache.
• Restrict upstream hosts and redirects while verifying TLS and logging failed requests.
• Preserve base-image cache reuse and disable the CDN where URL replacements are tested.
Diagram

graph TD
  CI["CI test jobs"] --> Bake["Buildx bake"] --> CLI["CLI URL rewrite"] --> Proxy["Local nginx proxy"] --> Hosts["Allowed hosts"] --> Upstream["Download hosts"]
  Proxy --> Cache[("Runner disk cache")]
  Upstream -- "redirect hop" --> Hosts
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Shared remote caching proxy
  • ➕ Could reuse downloads across runners and jobs.
  • ➖ Requires operating shared infrastructure and controlling access to it.
2. BuildKit cache mounts
  • ➕ Would avoid a runner-level network service.
  • ➖ Would require changing download steps and may not share downloads across independent build stages or retries as broadly.

Recommendation: The job-local proxy is a good fit for reuse within a job because it uses the existing CDN rewrite and bake arguments without changing individual tool installers. Review should focus on redirect handling, allowlist coverage, and nginx behavior on both runner architectures.

Files changed (6) +198 / -0

Tests (1) +2 / -0
DockerfileKeep Node URL-replacement tests off the CDN +2/-0

Keep Node URL-replacement tests off the CDN

• Clears CONTAINERBASE_CDN in the pnpm and Yarn replacement-test stage so CDN rewriting cannot bypass the URL replacements under test.

test/node/Dockerfile

Other (5) +196 / -0
action.ymlStart and check the runner-local download proxy +29/-0

Start and check the runner-local download proxy

• Adds a composite action that installs nginx if needed, loads the proxy configuration, starts it, and checks a Node.js download endpoint. It exports the proxy URL as CONTAINERBASE_CDN for later steps.

.github/actions/download-proxy/action.yml

allowed-hosts.mapAllowlist tool download and redirect hosts +29/-0

Allowlist tool download and redirect hosts

• Defines the hosts nginx may fetch from, including download sources and redirect destinations. The list applies to both initial requests and redirect hops.

.github/actions/download-proxy/allowed-hosts.map

errors.shSurface rejected and failed proxy requests +11/-0

Surface rejected and failed proxy requests

• Adds a diagnostic script that prints the nginx error log and access-log entries with 403 or 5xx responses when a CI job fails.

.github/actions/download-proxy/errors.sh

nginx.confConfigure restricted, caching HTTPS reverse proxy +104/-0

Configure restricted, caching HTTPS reverse proxy

• Reconstructs upstream URLs from CDN request paths, restricts clients and upstream hosts, verifies TLS, and follows allowlisted redirects. It caches successful responses on runner disk for reuse within the job.

.github/actions/download-proxy/nginx.conf

build.ymlEnable proxy for selected container-test jobs +23/-0

Enable proxy for selected container-test jobs

• Starts the proxy in distro, base-arm64, and language test jobs and prints proxy diagnostics on failure. The language job starts it after building the base image to preserve that image's existing cache key.

.github/workflows/build.yml

@qodo-code-review

qodo-code-review Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (5) 📎 Requirement gaps (1)

Grey Divider


Action required

1. Cached builds hide download hosts 📎 Requirement gap ⭐ New
Description
stats.sh prints Download proxy hosts from the current access log without warning that cached
build steps generate no proxy requests. When Bake reuses an image layer, its download hosts are
absent from the table, so someone using the summary to plan network allowlisting may miss hosts
needed for a clean build.
Code

.github/actions/download-proxy/stats.sh[R35-39]

+
+  echo ""
+  echo "### Download proxy hosts"
+  echo ""
+  echo "| Host | Requests | Cache hits | Redirects to |"
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new host table does not explain that cached build steps can omit hosts needed by a clean build.

## Fix Focus Areas
- .github/actions/download-proxy/stats.sh[35-40]

## Recommended Fix
Add a visible note beside the host table stating that it reflects only requests observed by the proxy in this job and that cached build steps can leave required hosts out. Keep the note in both the job log and summary.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Remediation recommended

2. Redirect summaries omit some hosts 🐞 Bug ⭐ New
Description
stats.sh deduplicates redirect targets by searching for the new hostname as a substring of the
accumulated list. If one requested host redirects to both cdn.dl.k8s.io and dl.k8s.io in that
order, the second target is treated as already present and omitted from the host summary.
Code

.github/actions/download-proxy/stats.sh[R54-55]

+      if (final != "" && final != host && index(targets[host], final) == 0) {
+        targets[host] = targets[host] (targets[host] == "" ? "" : ", ") final
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Substring matching can omit a distinct redirect host from the download-host summary.

## Fix Focus Areas
- .github/actions/download-proxy/stats.sh[52-60]

## Recommended Fix
Track redirect targets by both requested host and complete target hostname, then assemble the displayed list from those distinct pairs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View medium (3)
3. A second local test stops the first 🐞 Bug ⭐ New
Description
test.sh always uses the container name containerbase-cdn-test, and its exit trap removes that
name even when docker run fails. If two local test runs overlap, the second run’s name collision
exits through the trap and forcibly removes the first run’s container.
Code

.github/actions/download-proxy/test.sh[R18-20]

+cleanup() {
+  docker rm -f "${name}" > /dev/null 2>&1 || true
+  rm -rf "${logs}"
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A failed test startup can remove a container belonging to another concurrent test run.

## Fix Focus Areas
- .github/actions/download-proxy/test.sh[10-22]
- .github/actions/download-proxy/test.sh[27-31]

## Recommended Fix
Give each run a unique container name or ID, and arm container cleanup only after that run has successfully created its container.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


4. Cache hits are overstated by some URLs 🐞 Bug ⭐ New
Description
stats.sh searches each entire access-log line for cache=[A-Z]* instead of reading nginx’s final
cache-status field. When a request URL contains a query parameter such as cache=HIT, that request
contributes a false hit as well as its actual cache status to the job summary.
Code

.github/actions/download-proxy/stats.sh[31]

+  read_log | grep -o 'cache=[A-Z]*' | sort | uniq -c | while read -r count status; do
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The cache summary matches `cache=` text anywhere in a request, including query parameters, and can count one request twice.

## Fix Focus Areas
- .github/actions/download-proxy/stats.sh[29-34]
- .github/actions/download-proxy/nginx.conf[13-16]

## Recommended Fix
Parse the final `cache=` field emitted by nginx, then aggregate one status per access-log record.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


5. Relative redirects stop tool downloads 🐞 Bug
Description
$redirect_host is populated only when an upstream Location is an absolute HTTPS URL with a path,
so @redirect rejects relative locations as disallowed. If an allowed download host responds with a
valid redirect such as /release/file, the proxy returns 403 instead of resolving that path against
the upstream host.
Code

.github/actions/download-proxy/nginx.conf[R43-44]

+    default '';
+    ~^https://(?<location_host>[^/:]+)(:443)?/ $location_host;
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The proxy rejects valid relative upstream redirects because its redirect allowlist only recognizes absolute HTTPS locations.
## Fix Focus Areas
- .github/actions/download-proxy/nginx.conf[40-51]
- .github/actions/download-proxy/nginx.conf[95-101]
## Recommended Fix
Resolve relative Location values against the current upstream URL, then validate the resulting HTTPS host against the allowlist before proxying each hop. Preserve validation for absolute redirects.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



View low (1)
Informational
6. Cached downloads lose redirect host logs 🐞 Bug ⭐ New
Description
location / resets $final_host to the requested host, while only @redirect updates it to the
redirect target. A cache hit for a previously redirected download never enters @redirect, so its
access-log record incorrectly reports that the requested host was the final host.
Code

.github/actions/download-proxy/nginx.conf[91]

+      set $final_host $upstream_host;
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Cache hits for redirected downloads log the original host as the final host because redirect handling is skipped.

## Fix Focus Areas
- .github/actions/download-proxy/nginx.conf[13-16]
- .github/actions/download-proxy/nginx.conf[75-101]

## Recommended Fix
Persist the final redirect host with the cached response and use it when logging cache hits, or mark the final host unknown on hits rather than reporting the original host as the redirect destination.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Resolved findings
1. Job summaries hide download hosts ✓ Resolved
Description
stats.sh aggregates proxy requests by cache status and writes only those counts to the job output
and summary. When builds succeed or responses come from cache, users see no contacted host names, so
restricted-network operators still cannot determine which external hosts must be allowed.
Code

.github/actions/download-proxy/stats.sh[24]

+} | tee -a "${GITHUB_STEP_SUMMARY}"
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The download proxy statistics output reports only cache-status counts and does not expose the external hosts contacted during successful or cached downloads. This prevents restricted-network users from identifying the hosts they need to allow.

## Fix Focus Areas
- .github/actions/download-proxy/stats.sh[3-4]
- .github/actions/download-proxy/stats.sh[24-24]

## Recommended Fix
Extend the proxy log summary to extract and display the contacted upstream host for every request, including requests served from cache and hosts reached through redirects, while retaining the cache hit and miss counts.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can keep summaries lean with Findings visible per group, which tucks the rest behind a View link

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@viceice

viceice commented Oct 8, 2026

Copy link
Copy Markdown
Member Author

Re the relative redirect finding: that's intended. A redirect is only followed when its Location is an absolute https:// url to a host in allowed-hosts.map, so every hop is checked against the list; anything else gets a 403 and shows up in the "Download proxy errors" step. None of the allowed hosts redirects relatively (the full CI run passed), so I'd rather keep the stricter rule than resolve relative paths.

(Comment by Claude Opus 5.5 in Claude Code on behalf of @viceice.)

Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
@viceice
viceice added this pull request to the merge queue Oct 8, 2026
@viceice
viceice removed this pull request from the merge queue due to a manual request Oct 8, 2026
Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
Comment thread .github/actions/download-proxy/stats.sh Outdated
Co-Authored-By: Claude Opus 5.5 <michael.kriese+claude-code@mend.io>
@viceice
viceice enabled auto-merge October 8, 2026 16:47
Comment on lines +35 to +39

echo ""
echo "### Download proxy hosts"
echo ""
echo "| Host | Requests | Cache hits | Redirects to |"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Cached builds hide download hosts 📎 Requirement gap ◔ Observability

stats.sh prints Download proxy hosts from the current access log without warning that cached
build steps generate no proxy requests. When Bake reuses an image layer, its download hosts are
absent from the table, so someone using the summary to plan network allowlisting may miss hosts
needed for a clean build.
Agent Prompt
## Issue description
The new host table does not explain that cached build steps can omit hosts needed by a clean build.

## Fix Focus Areas
- .github/actions/download-proxy/stats.sh[35-40]

## Recommended Fix
Add a visible note beside the host table stating that it reflects only requests observed by the proxy in this job and that cached build steps can leave required hosts out. Keep the note in both the job log and summary.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

@viceice
viceice added this pull request to the merge queue Oct 8, 2026
Merged via the queue into main with commit a2ed5b9 Oct 8, 2026
58 checks passed
@viceice
viceice deleted the viceice/ci/download-proxy branch October 8, 2026 17:22
@viceice

viceice commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Re the "Cached builds hide download hosts" finding: fixed in #7693, which adds a note above the host table that it only lists the requests the proxy saw in that job, so hosts of build steps reused from the cache can be missing.

(Comment by Claude Opus 5.5 in Claude Code on behalf of @viceice.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant