Repository navigation
CD: build arm64 natively instead of under QEMU (~12 min per cold run) #67
Description
Follow-up to #64, which added linux/arm64 to the CD image build so images run natively on the arm64 (Oracle Ampere) deploy host. It works, but the arm64 half is built under QEMU emulation on an amd64 runner, which is slow enough to be worth fixing.
Measured cost
| Run | Platforms | Duration |
|---|---|---|
| 34763820630 (pre-#64) | amd64 | 9m42s |
| 34849446439 (post-#64, cold arm64 cache) | amd64 + arm64 | 21m50s |
So the arm64 half costs roughly +12 min on a cold cache.
Root cause
It is not the UniDic download. That layer (Dockerfile:43, ~700 MB) is network-bound, so emulation barely affects it.
The dominant cost is uv sync at Dockerfile:24. As the comment at Dockerfile:9-13 notes, pyopenjtalk ships source-only on PyPI with no wheels for any arch, so it compiles its bundled OpenJTalk/HTS-engine C++ from sdist during the build. That is pure CPU-bound C++ compilation, and under QEMU it runs roughly an order of magnitude slower than native.
Why this is not urgent
Layer caching is already correct — every expensive step sits before COPY . . (Dockerfile:45) and is gated by uv.lock / download_unidic.sh, so an app-code-only commit reuses all of it. GHA cache usage is currently 3.64 GB / 10 GB across 41 entries, so LRU eviction is not yet a factor either.
The thing that will actually bite: GitHub evicts caches not accessed for 7 days. CD only runs on pushes to main, so any gap longer than a week means the next run pays the full cold-cache emulation cost again.
Proposal
This repo is public, so GitHub's native arm64 runners (ubuntu-24.04-arm) are free. Build each arch natively and merge the manifest, eliminating QEMU entirely.
Sketch:
- Matrix job over
{ubuntu-latest → linux/amd64, ubuntu-24.04-arm → linux/arm64}; each builds and pushes by digest (outputs: type=image,push-by-digest=true,name-canonical=true) with its own cache scope. - A
mergejob runsdocker buildx imagetools createover the per-arch digests to produce the tagged multi-arch index, applying thedocker/metadata-actiontags.
Complication worth planning for
The current smoke test depends on the single-platform shape: it builds amd64 with load: true (buildx cannot load a multi-platform result), runs the container on the runner, then pushes both arches in a second build that reuses the cache. A matrix split changes that flow — but it also improves it, since each arch can then smoke-test its own image natively instead of only amd64 being exercised. The merge job should only publish tags once both arch smoke tests pass, so a broken arm64 image can never take the dev/stable tag.
Also note provenance: true / sbom: true currently attach attestations (the two unknown/unknown entries in the index); the merge step needs to preserve these.
Not blocking
#64 is deployed and correct — ghcr.io/sessatakuma/api-tools:dev is now an OCI image index with both linux/amd64 and linux/arm64. This issue is purely a CI wall-clock optimization.
Activity
torridfish commentedon Sep 14, 2026
Absorbing #68: arm64 must also be smoke-tested (closing #68 as a duplicate of this issue)
#68 reported that CD publishes the arm64 image without ever running it. That is the same work item as this issue — #68's preferred fix ("native
ubuntu-24.04-armrunner") is this issue's proposal, and the "Complication worth planning for" section above is exactly #68's problem. Folding it in here so one PR closes both.The verification gap (from #68)
Since #64, CD publishes a
linux/amd64,linux/arm64manifest, but.github/workflows/cd.ymlonly ever executes the amd64 half:Build amd64 image for smoke test(L80-95) buildslinux/amd64withload: true— buildx cannotloada multi-platform result into the runner daemon.Smoke-test production lifecycle offlineruns that amd64 image.Push smoke-tested image (amd64 + arm64)(L117-126) builds the arm64 half under QEMU and pushes it without ever running it.So the image that actually runs on the OCI Ampere deploy host ships unverified. An arch-specific wheel resolving differently, or OpenJTalk / fugashi misbehaving on aarch64, goes undetected until deploy.
Why the native-runner plan fixes both at once
The matrix split proposed above is not just a wall-clock optimisation — it is also what makes per-arch smoke testing possible, since each arch then has a single-platform image it can
loadand run natively on its own runner. QEMU-baseddocker run --platform linux/arm64(#68's option 2) would need the wait loops widened a lot and makes CD slower, so it is strictly worse on both axes.Acceptance criteria
ubuntu-latest/ubuntu-24.04-arm), no QEMU.mergejob publishes thedev/stabletags only after both arch smoke tests pass, so a broken arm64 image can never take a tag.provenance/sbomattestations survive theimagetools createmerge.