Skip to content

ci: reproducible musl static builds for clone (x86_64 + aarch64 + arm32) - #36

Open
sysretq0 wants to merge 29 commits into
unixshells:masterfrom
sysretq0:ci/static-builds
Open

sysretq0 wants to merge 29 commits into
unixshells:masterfrom
sysretq0:ci/static-builds

Conversation

@sysretq0

Copy link
Copy Markdown

What

Adds .github/workflows/build-static.yml: reproducible musl static builds of clone, clone-init, clone-agent across x86_64-unknown-linux-musl, aarch64-unknown-linux-musl, and arm-unknown-linux-musleabihf (cross-rs toolchain containers, toolchain never installed on dev hosts).

Why static musl

Single-binary deliverable that runs identically on bare-metal thin-Dom0 hosts (aarch64 vendor-kernel EL2 host, e.g. MT6789/G99 phone booting the 5.10 vendor kernel) and on x86_64 dev boxes, with no dependency on the host's shared-library stack.

Notes

  • arm-unknown-linux-musleabihf is continue-on-error: true: the current VMM core is x86-only (TSC/CPUID/kvm-clock heavy). The aarch64 target may need the same port before it turns green — artifacts upload as soon as any target compiles.
  • fail-fast: false so all matrix targets report independently.
  • Artifact upload includes file/ldd static-link sanity checks per binary.

… / arm32

Toolchain lives entirely on GitHub Actions runners: rustup targets +
cross-rs containers per target triple, so no dev host needs a Rust stack
installed locally. Artifacts: clone + clone-init + clone-agent, verified
static via file/ldd, uploaded as branch artifacts.

x86_64 + aarch64 marked required; arm32 marked experimental until the
VMM core grows an arm32 boot path (TSC/cpuid-heavy code today).

Change-Id: I2d1db61739f8bdac08387f0fc3c60fc4533df794
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
@github-actions github-actions Bot added the area:ci CI workflows, hooks, lint config label Nov 10, 2026
- src/sys.rs: single ABI choke-point. ioctl() takes the raw u64 request:
  musl types libc::Ioctl as c_int (fixed 3-arg ABI), glibc as c_ulong.
  renameat2 falls back to SYS_renameat2 syscall on musl (wrapper only
  exists in glibc builds of the libc crate).
- all 34 ioctl sites: request exprs as u64, third-arg coerced to
  *mut c_void (value-args via ioctl_value, pointers as *const _ as *mut).
- vCPU thread handles carried as usize across threads/channels/VmHandle;
  musl types pthread_t as *mut c_void which is !Send. pthread_kill casts
  back to pthread_t at the libc boundary.
- clone-init enable_iopl: cfg-gated to x86 — libc doesn't declare iopl
  on other arches (fixes aarch64 E0425).

Change-Id: I7e955a2a7c539119f3296dc7b0c6b0879413f563
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
@github-actions github-actions Bot added area:vmm VMM core: boot, control, main loop area:virtio virtio devices (block/net/vsock/fs/balloon) area:net Networking, NAT, bridge, iptables labels Nov 10, 2026
… no port I/O

aarch64 build was clean for the VMM core; only the guest init's x86
'in al, dx'/'out dx, al' serial asm blocked compile. Early console on
arm64 goes through /dev/console instead (same as the non-linux stub).

Change-Id: Iedac2e7b2abef6b19b324a4cf8cf8689490f7443
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
cross's user-mode emulation breaks build scripts against host glibc
(gettid GLIBC_2.30 relocation error). Build scripts are host binaries;
the final binaries link self-contained against the musl libc shipped
in the rustup target via rust-lld.

Change-Id: I3208f8c6d0cf6a008f5fe2b3504ec50736367bef
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…st port)

Arch-dispatched boot so the same binary serves x86_64 hosts and the
arm64 thin-Dom0 (MT6789/G99, 5.10 vendor kernel):

- src/boot/arm64.rs: arm64 Image loader (magic @0x38, text_offset u32@0x04,
  load at 0x40000000+offset), initrd at 0x48000000, DTB at 0x47000000.
  Token-faithful FDT v17 builder (no libfdt) ported from dom0/kvmhost.c:
  memory/distance-map/chosen/cpus/timer/psci/uartclk/pl011/gic-v3/
  virtio-mmio nodes with the real MmioBus (base, irq) table.
- vmm: x86-only irqchip+PIT+GSI routing cfg-gated; aarch64 creates the
  vGICv3 device AFTER vCPUs (dist@0x09000000, redist@0x080b0000,
  explicit CTRL_INIT) via raw KVM ioctls — same proven sequence as kvmhost.
- vcpu: aarch64 branch does KVM_ARM_VCPU_INIT (generic_v8 + PSCI_0_2) and
  sets BSP PC/x0 via KVM_SET_ONE_REG (ids 0x6030000000100040 /
  0x6030000000100000, confirmed against this system's uapi headers).
  PL011@0x0A000000 MMIO exits forward guest console bytes to host stdout.
- snapshot/fork/migration capture+restore paths stay x86-only (bail on
  other arches); fresh boot is the /lokiinit path.
- kernel loader dispatch: load_kernel_for_arch()

Change-Id: I521221e0ec0fb5cc1cf557329c6ddc3e87f2b92b
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
- arm64.rs finish(): compute lengths first, then write header + extend
  slices — clears E0382 (moved self.st) and E0499
- vcpu.rs: remove stray cfg attr above check_pause (E0599 x2); PL011
  MmioRead/MmioWrite arms fold into the handled/inline flow instead of
  early Ok-return (run_loop returns Result<()>, so Ok(ExitAction) was
  E0308); Debug arm cfg-gated (kvm_debug_exit_arch has no pc/dr6 on
  arm64); cmos_read un-gated (pure data lookup, arch-neutral)
- vmm/mod.rs: kvmclock_ctrl call in fork_boot cfg-gated x86 (method
  doesn't exist on aarch64 VcpuFd)
- jailer.rs: SYS_arch_prctl x86-only; SYS_epoll_wait/SYS_poll gated to
  arches that still have them (aarch64 uses epoll_pwait/ppoll)
- arm64.rs: cap DTB RAM span below the 3GB MMIO hole

Change-Id: I40fcdd3a82d5491a4b66262c51b994e64dc964d5
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…l list restructure

- arm64.rs prop(): hoist name_off to a local — self.put32(self.name_off(..))
  is two nested &mut self borrows (like vec.push(vec.pop())), E0499.
  This was the root of the whole E0433/E0423 arm64 cascade.
- vcpu.rs: associated fn call needs Self::set_arm64_core_reg
- jailer.rs: #[cfg] on array elements is E0658 (attributes on expressions
  are experimental); rebuild list as base_allowed() + arch_specific()
  with cfg'd pushes (stable on statements). Fixes x86_64 job too.

Change-Id: I207446bbdc76fd3e522f2a82eb3977ef74f91f2b
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
E0423 'cannot find value arm64 in module crate::boot' + note 'exists in
another namespace': let arm64 = crate::boot::arm64 tries to bind a module
in the value namespace. Replace the three bogus bindings with a
cfg(aarch64) use crate::boot::arm64; import in vcpu.rs / vmm/mod.rs.

Change-Id: I6bff6c1f2aad09d0668dbd48d0f1a0b433b23d7d
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…I injection

- GuestMem gains a GPA base: arm64 registers the single KVM slot at
  GPA 0x40000000 (kvmhost layout — Image at RAM_BASE+0x80000, DTB
  advertises memory@40000000); all GPA→HVA translation subtracts base.
- boot(): aarch64 allocates a plain contiguous slab (no x86 MMIO-hole
  logic) and registers slot 0 at RAM_BASE; x86_64 branch unchanged.
- arm64 loader: text_offset is u64@0x08 (deprecated, 0 on modern
  kernels — 0x04 is code1, not text_offset), validate image_size
  u64@0x10 against the RAM span budget, mirroring kvmhost load_image().
- IRQ injection: arm64 KVM_IRQ_LINE takes a composite word
  type<<24 | vcpu<<16 | num with KVM_ARM_IRQ_TYPE_SPI=1 (kvmhost
  vblk_inject). New set_spi_line() helper used by all virtio/serial
  injection sites; x86 keeps the raw line number.

Change-Id: I506afd4e7642d4e4d50d441f0161f4f8a0a12e62
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
- set_spi_line was inside impl Vm but is called from spawned closures
  and free fns — move it next to setup_gsi_routing at module scope.
- create_guest_memory_with_hole's GuestMem literal needs the new base
  field (E0063).

Change-Id: If183ef1cd421fa81f55cf4fbdacbdc9bb094c244
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
… placeholders

Change-Id: I448d7cc4df0f1e81fadd1d4b9a79587b2d894b26
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
kvm_vcpu_init is { u32 target; u32 features[7] } = 32 bytes — not the
132-byte struct assumed. The _IOC_SIZE field must match the kernel's
encoding or the ioctl switch never matches:

- KVM_ARM_VCPU_INIT: 0x4084AEEE -> 0x4020AEEE, KvmVcpuInit.features
  [u32;32] -> [u32;7]. This was the root cause of the EINVAL on
  KVM_ARM_VCPU_INIT (verified kvmhost_smoke passes with target=5 on the
  same kernel, so the call itself was bad).
- KVM_CREATE_DEVICE: 0xC010AEE0 -> 0xC00CAEE0 (kvm_create_device = 3
  u32s = 12 bytes; would have broken vGICv3 creation next).
- KVM_SET_DEVICE_ATTR (0x4018AEE1) and KVM_SET_ONE_REG (0x4010AEAC)
  re-verified correct against /usr/include/linux/kvm.h.

Change-Id: Ib43561001dde5f70e1d067e9931fe744fe410d14
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I0fa21b987deea27955ca2068b9703c93985a2f2e
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
The ioctl number was hallucinated as 0x…AEEE (confused with other
KVM ioctls); the real uapi value is _IOW(KVMIO=0xAE, nr=0xae, 32) =
0x4020AEAE. Verified against /usr/include/linux/kvm.h via a compiled
probe and against the proven kvmhost.c binary (strace-decoded as
KVM_ARM_VCPU_INIT). Confirmed the full boot pipeline on the device
with a ctypes replication: 2 vCPUs VCPU_INIT(target=5, PSCI_0_2),
vGICv3 create + DIST/REDIST attrs + CTRL_INIT all succeed.

Change-Id: I248d76acbdc92aebb05a7205376391b96027eb29
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
… FR=0x90

Two device-proven fixes:
- finish() allocated the DTB buffer with extend_from_slice() appends, so
  the struct/strings landed AFTER the region the header points at and
  off_dt_struct=56 read as zeros -> guest spun silently before earlycon
  (0 KVM_RUN exits in 25s). Now written in place at the header offsets.
- PL011 Flag Register (offset 0x18) must read TXFE|RXFE (0x90): the
  kernel console path polls FR for "TX FIFO empty" SET; returning 0
  stalled the console after the first store. DR writes forward only the
  low byte of the store (32-bit MMIO stores carry the char in byte 0).

Verified end-to-end with a ctypes KVM replica on the device: 2 vCPUs +
vGICv3 + LOG_DIRTY_PAGES, dtc-validated guest DTB -> mainline 7.3.3
arm64 kernel boots through smp/GIC/timer init to the expected no-rootfs
panic and a clean PSCI SYSTEM_OFF exit.

Change-Id: I85d3b2fd4083855ad63761010aa91083389e2b36
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…CI SystemEvent

- gicreg[7]: redistributor size is ncpus * 0x20000 (each vCPU owns a
  2-page GICR frame). With 0x20000 hardcoded the kernel crashed in
  gic_iterate_rdists as soon as num_cpus > 1 (kvmhost.c is proven with
  0x20000 only because it runs a single vCPU). Verified: python ctypes
  replica with 2 vCPUs boots through the whole boot to the expected
  no-rootfs panic once the DTB declares 0x40000.
- VcpuExit::SystemEvent on aarch64 is the guest's PSCI SYSTEM_OFF/
  RESET (panic=-1 or clean poweroff) — log it and stop the vCPU
  cleanly instead of 'unhandled exit'.

Change-Id: I8e558d25eb9e9a4a87047f2539816508f3b8db96
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…ON works

Without this bit, kvm_psci_vcpu_on() treats a never-entered secondary
vCPU as 'already on' (arch.power_off == false) and returns
PSCI_RET_ALREADY_ON, which the guest maps to -EINVAL: 'psci: failed to
boot CPU1 (-22)', SMP stays at 1 CPU. QEMU/kvmtool set this flag for
secondaries; we didn't. Verified on-device with a ctypes replica:
'smp: Brought up 1 node, 2 CPUs', CPU1 boots with its own GICR at
0x080d0000.

Change-Id: I15173027bbb3ebb63e1bf74db5273e2d6fdb2780
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…irtio chain

embed_kernel_modules keyed the module tree off uname -r, i.e. the HOST
kernel — on a phone Dom0 that is the Android kernel, so nothing was
embedded and --rootfs could never mount anything on Debian arm64 guest
kernels where CONFIG_EXT4_FS/VIRTIO_MMIO/VIRTIO_BLK are =m.

- Pass the guest kernel version (parsed from the --kernel filename,
  vmlinuz-<version>) through generate_initrd().
- Embed the ext4 dependency chain (crc32c-cryptoapi, crc16, jbd2,
  mbcache, ext4) plus virtio_mmio and virtio_blk in dependency order.
  Module chain verified with 'modprobe --set-version <guest>
  --show-depends ext4' on the device.
- clone-init: insmod them before waiting for /dev/vda.

Change-Id: I26d2a81aa01f2f8d392f028229949b6b14667a70
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…pecs

The GIC node inherited kvmhost.c's #interrupt-cells=<1>, so every
3-cell interrupt spec (PPIs 4x, PL011, virtio-mmio devices) parsed as
garbage: 'arch_timer: No interrupt available, giving up' and
'request_threaded_irq: -EINVAL' in virtio_blk probe
(vm_find_vqs -> init_vq). kvmhost only ever ran earlycon so the
breakage was invisible there. Flags stay unchanged: 0x108 = level-low
(PPIs), 4 = level-high (SPIs).

Change-Id: I0dba18c807434d953f3eaca8f9fd18a966249b5f
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…arm64 SIGSEGV)

On arm64 the single KVM memory slot lives at GPA 0x40000000, but the
virtio layer assumed guest RAM starts at GPA 0: gpa_to_ptr computed
host_ptr + gpa, so the first virtio device whose DRIVER_OK path reads
guest queues (virtio-vsock prepare_activate) dereferenced 0x40000000
past the mmap → SIGSEGV in MmioTransport::handle_status_write.

Plumb GuestMem::base() (0 on x86, RAM_BASE on arm64) through
MmioBus → MmioTransport → Virtqueue and via a new VirtioDevice::
set_mem_base() to devices that do their own translation:
- queue.rs gpa_to_offset(): checked_sub(mem_base) first
- vsock.rs gpa_to_ptr(): subtract base + bounds-check (null on bad GPA;
  prepare_activate now skips out-of-range queues instead of crashing)
- net.rs gpa_to_ptr()/gpa_to_hva(): subtract base
- balloon.rs process_inflate(): PFNs are relative to mem_base
  (previously used raw PFN addresses + hardcoded x86 3-4GB hole)
- vmm/mod.rs: mmio_bus.set_mem_base(guest_memory.base()) at both
  set_guest_memory_with_hole call sites (fresh VM and restore)

Change-Id: I8f1f1a72636280eea30844d12eb81fe1538a3717
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I3ebaeb2406309a2887e56747e094e46c59b87f92
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…_TYPE_CPU

The vCPU run loop's MMIO-write path (virtio QUEUE_NOTIFY kicks) called
vm_fd.set_irq_line(irq, ...) directly. On aarch64 that ioctl word
encodes irqtype=0 (KVM_ARM_IRQ_TYPE_CPU) + line number — a CPU/PPI line,
not the SPI the DTB promises (interrupts=<0 5 4> etc.). The guest
virtio driver therefore never received its interrupt: virtio_blk
probed, /dev/vda appeared, but the first real block request (ext4
mount) never completed — console stalls after '[vda] ... logical
blocks' with no EXT4 mount.

Route through vmm::set_spi_line() (KVM_ARM_IRQ_TYPE_SPI, same as the
vsock/balloon paths and dom0/kvmhost.c's proven vblk_inject). On x86
set_spi_line degrades to the raw call, so behavior is unchanged.

Change-Id: Id501c415b9ab013ac978a14d3d402c87292984d7
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…SPI number

strace on-device showed the block-device kick DID reach clone, but
ioctl(4, KVM_IRQ_LINE, ...) = -1 EINVAL: the kernel validates the
KVM_ARM_IRQ_TYPE_SPI irqid as an INTID (>= 32, kvmtool does spi_irq+32,
QEMU does irq+GIC_INTERNAL). Passing the bare SPI number (5/6/7) failed
the range check, so every virtio completion IRQ was silently dropped
and the first ext4 mount read hung forever after '[vda] ... logical
blocks'.

Change-Id: Ia958745005e6f1b08de237816136a9ab8a4919b9
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I5fa27e86d46d8535637b631f4c513812d30c2a06
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I510db7ed12cb461b7a8f2e485886e1572ba210aa
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…ection (temp)

Change-Id: Ib7624e5c038ec26c70095851770d01ce02544e8a
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: Ie9d6656c213b1cb7807bc850abd92dbd4f73a417
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: Ie400c173fbde16da5614362058c0fc003feb4bc1
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: Ifbc5f3edc5f3f5e14876eceed651e9f84daf767d
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
@realrasengan

Copy link
Copy Markdown
Contributor

Thanks for this, and for the work on getting clone running on an arm64 host. I read through the whole diff. I can't merge it as is, but there's a piece of it I'd take.

The main problem is scope. The title and description say this is a CI workflow, but it's +1468 lines across 22 files and includes a full arm64 boot path (FDT builder, vGICv3, PL011). Can you close this and split it into two PRs?

Split 1: musl portability + CI (I'd merge this)

  • src/sys.rs shim, the pthread_t -> usize change, the x86-only cfg gates
  • build-static.yml, x86_64 only for now
  • Fix src/main.rs: mod sys; was inserted between #[cfg(target_os = "linux")] and mod vmm;, so vmm lost its cfg and non-Linux builds break
  • Workflow: pin third-party actions by SHA, and don't trigger on push to every branch plus every PR (each dependabot branch would build twice)

Split 2: arm64 port (open as a draft)

Debug code that needs to come out first. The last six commits are marked debug: ... (temp) and are still in the diff:

  • src/virtio/mmio.rs: tracing::info! on every QUEUE_NOTIFY and on every virtio-mmio write. These aren't arch-gated, so they hit the x86 hot path too.
  • src/vmm/mod.rs: the VGIC_FD static, dbg_store_vgic_fd, dbg_vgic_read_reg, dbg_vgic_dump_spi, and the x86 no-op stub for it.
  • src/vmm/vcpu.rs: dbg_vgic_dump_spi(irq + 32) runs after every virtio kick, which is four ioctls and an info log per kick.
  • KVM_GET_DEVICE_ATTR only exists for that debug path and the number is wrong anyway (should be 0x4018AEE2, _IOW, nr 0xe2).

Bugs I hit while reading:

  • Balloon: guest_addr = mem_base + pfn * PAGE_SIZE, but the guest's PFNs are already absolute, and the host offset never subtracts the base. Small guests skip every page, and guests over 1GB can madvise(MADV_DONTNEED) the wrong page.
  • vhost-net: the mem table still has guest_phys_addr: 0.
  • PL011: every write in the UART window goes to stdout, including control register writes. Only DR should.

Other things:

  • Snapshot/template/fork bail on arm64, and the migration receiver creates no interrupt controller. Fork is the point of clone, so the description should say plainly what works and what doesn't.
  • References to dom0/kvmhost.c, /lokiinit and the "loki thin-dom0 vm" model string are from your own tree, not this repo. Please drop or generalize them.
  • No tests. The FDT builder and the GPA translation with a non-zero base are both easy to unit test.
  • Squash the history. 29 commits of compile fixes is hard to review.

Once it's split, I'll review the first one quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI workflows, hooks, lint config area:net Networking, NAT, bridge, iptables area:virtio virtio devices (block/net/vsock/fs/balloon) area:vmm VMM core: boot, control, main loop

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants