Repository navigation
Conversation
… / arm32 Toolchain lives entirely on GitHub Actions runners: rustup targets + cross-rs containers per target triple, so no dev host needs a Rust stack installed locally. Artifacts: clone + clone-init + clone-agent, verified static via file/ldd, uploaded as branch artifacts. x86_64 + aarch64 marked required; arm32 marked experimental until the VMM core grows an arm32 boot path (TSC/cpuid-heavy code today). Change-Id: I2d1db61739f8bdac08387f0fc3c60fc4533df794 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
- src/sys.rs: single ABI choke-point. ioctl() takes the raw u64 request: musl types libc::Ioctl as c_int (fixed 3-arg ABI), glibc as c_ulong. renameat2 falls back to SYS_renameat2 syscall on musl (wrapper only exists in glibc builds of the libc crate). - all 34 ioctl sites: request exprs as u64, third-arg coerced to *mut c_void (value-args via ioctl_value, pointers as *const _ as *mut). - vCPU thread handles carried as usize across threads/channels/VmHandle; musl types pthread_t as *mut c_void which is !Send. pthread_kill casts back to pthread_t at the libc boundary. - clone-init enable_iopl: cfg-gated to x86 — libc doesn't declare iopl on other arches (fixes aarch64 E0425). Change-Id: I7e955a2a7c539119f3296dc7b0c6b0879413f563 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
… no port I/O aarch64 build was clean for the VMM core; only the guest init's x86 'in al, dx'/'out dx, al' serial asm blocked compile. Early console on arm64 goes through /dev/console instead (same as the non-linux stub). Change-Id: Iedac2e7b2abef6b19b324a4cf8cf8689490f7443 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
cross's user-mode emulation breaks build scripts against host glibc (gettid GLIBC_2.30 relocation error). Build scripts are host binaries; the final binaries link self-contained against the musl libc shipped in the rustup target via rust-lld. Change-Id: I3208f8c6d0cf6a008f5fe2b3504ec50736367bef Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…st port) Arch-dispatched boot so the same binary serves x86_64 hosts and the arm64 thin-Dom0 (MT6789/G99, 5.10 vendor kernel): - src/boot/arm64.rs: arm64 Image loader (magic @0x38, text_offset u32@0x04, load at 0x40000000+offset), initrd at 0x48000000, DTB at 0x47000000. Token-faithful FDT v17 builder (no libfdt) ported from dom0/kvmhost.c: memory/distance-map/chosen/cpus/timer/psci/uartclk/pl011/gic-v3/ virtio-mmio nodes with the real MmioBus (base, irq) table. - vmm: x86-only irqchip+PIT+GSI routing cfg-gated; aarch64 creates the vGICv3 device AFTER vCPUs (dist@0x09000000, redist@0x080b0000, explicit CTRL_INIT) via raw KVM ioctls — same proven sequence as kvmhost. - vcpu: aarch64 branch does KVM_ARM_VCPU_INIT (generic_v8 + PSCI_0_2) and sets BSP PC/x0 via KVM_SET_ONE_REG (ids 0x6030000000100040 / 0x6030000000100000, confirmed against this system's uapi headers). PL011@0x0A000000 MMIO exits forward guest console bytes to host stdout. - snapshot/fork/migration capture+restore paths stay x86-only (bail on other arches); fresh boot is the /lokiinit path. - kernel loader dispatch: load_kernel_for_arch() Change-Id: I521221e0ec0fb5cc1cf557329c6ddc3e87f2b92b Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
- arm64.rs finish(): compute lengths first, then write header + extend slices — clears E0382 (moved self.st) and E0499 - vcpu.rs: remove stray cfg attr above check_pause (E0599 x2); PL011 MmioRead/MmioWrite arms fold into the handled/inline flow instead of early Ok-return (run_loop returns Result<()>, so Ok(ExitAction) was E0308); Debug arm cfg-gated (kvm_debug_exit_arch has no pc/dr6 on arm64); cmos_read un-gated (pure data lookup, arch-neutral) - vmm/mod.rs: kvmclock_ctrl call in fork_boot cfg-gated x86 (method doesn't exist on aarch64 VcpuFd) - jailer.rs: SYS_arch_prctl x86-only; SYS_epoll_wait/SYS_poll gated to arches that still have them (aarch64 uses epoll_pwait/ppoll) - arm64.rs: cap DTB RAM span below the 3GB MMIO hole Change-Id: I40fcdd3a82d5491a4b66262c51b994e64dc964d5 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…l list restructure - arm64.rs prop(): hoist name_off to a local — self.put32(self.name_off(..)) is two nested &mut self borrows (like vec.push(vec.pop())), E0499. This was the root of the whole E0433/E0423 arm64 cascade. - vcpu.rs: associated fn call needs Self::set_arm64_core_reg - jailer.rs: #[cfg] on array elements is E0658 (attributes on expressions are experimental); rebuild list as base_allowed() + arch_specific() with cfg'd pushes (stable on statements). Fixes x86_64 job too. Change-Id: I207446bbdc76fd3e522f2a82eb3977ef74f91f2b Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
E0423 'cannot find value arm64 in module crate::boot' + note 'exists in another namespace': let arm64 = crate::boot::arm64 tries to bind a module in the value namespace. Replace the three bogus bindings with a cfg(aarch64) use crate::boot::arm64; import in vcpu.rs / vmm/mod.rs. Change-Id: I6bff6c1f2aad09d0668dbd48d0f1a0b433b23d7d Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…I injection - GuestMem gains a GPA base: arm64 registers the single KVM slot at GPA 0x40000000 (kvmhost layout — Image at RAM_BASE+0x80000, DTB advertises memory@40000000); all GPA→HVA translation subtracts base. - boot(): aarch64 allocates a plain contiguous slab (no x86 MMIO-hole logic) and registers slot 0 at RAM_BASE; x86_64 branch unchanged. - arm64 loader: text_offset is u64@0x08 (deprecated, 0 on modern kernels — 0x04 is code1, not text_offset), validate image_size u64@0x10 against the RAM span budget, mirroring kvmhost load_image(). - IRQ injection: arm64 KVM_IRQ_LINE takes a composite word type<<24 | vcpu<<16 | num with KVM_ARM_IRQ_TYPE_SPI=1 (kvmhost vblk_inject). New set_spi_line() helper used by all virtio/serial injection sites; x86 keeps the raw line number. Change-Id: I506afd4e7642d4e4d50d441f0161f4f8a0a12e62 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
- set_spi_line was inside impl Vm but is called from spawned closures and free fns — move it next to setup_gsi_routing at module scope. - create_guest_memory_with_hole's GuestMem literal needs the new base field (E0063). Change-Id: If183ef1cd421fa81f55cf4fbdacbdc9bb094c244 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
… placeholders Change-Id: I448d7cc4df0f1e81fadd1d4b9a79587b2d894b26 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
kvm_vcpu_init is { u32 target; u32 features[7] } = 32 bytes — not the
132-byte struct assumed. The _IOC_SIZE field must match the kernel's
encoding or the ioctl switch never matches:
- KVM_ARM_VCPU_INIT: 0x4084AEEE -> 0x4020AEEE, KvmVcpuInit.features
[u32;32] -> [u32;7]. This was the root cause of the EINVAL on
KVM_ARM_VCPU_INIT (verified kvmhost_smoke passes with target=5 on the
same kernel, so the call itself was bad).
- KVM_CREATE_DEVICE: 0xC010AEE0 -> 0xC00CAEE0 (kvm_create_device = 3
u32s = 12 bytes; would have broken vGICv3 creation next).
- KVM_SET_DEVICE_ATTR (0x4018AEE1) and KVM_SET_ONE_REG (0x4010AEAC)
re-verified correct against /usr/include/linux/kvm.h.
Change-Id: Ib43561001dde5f70e1d067e9931fe744fe410d14
Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I0fa21b987deea27955ca2068b9703c93985a2f2e Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
The ioctl number was hallucinated as 0x…AEEE (confused with other KVM ioctls); the real uapi value is _IOW(KVMIO=0xAE, nr=0xae, 32) = 0x4020AEAE. Verified against /usr/include/linux/kvm.h via a compiled probe and against the proven kvmhost.c binary (strace-decoded as KVM_ARM_VCPU_INIT). Confirmed the full boot pipeline on the device with a ctypes replication: 2 vCPUs VCPU_INIT(target=5, PSCI_0_2), vGICv3 create + DIST/REDIST attrs + CTRL_INIT all succeed. Change-Id: I248d76acbdc92aebb05a7205376391b96027eb29 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
… FR=0x90 Two device-proven fixes: - finish() allocated the DTB buffer with extend_from_slice() appends, so the struct/strings landed AFTER the region the header points at and off_dt_struct=56 read as zeros -> guest spun silently before earlycon (0 KVM_RUN exits in 25s). Now written in place at the header offsets. - PL011 Flag Register (offset 0x18) must read TXFE|RXFE (0x90): the kernel console path polls FR for "TX FIFO empty" SET; returning 0 stalled the console after the first store. DR writes forward only the low byte of the store (32-bit MMIO stores carry the char in byte 0). Verified end-to-end with a ctypes KVM replica on the device: 2 vCPUs + vGICv3 + LOG_DIRTY_PAGES, dtc-validated guest DTB -> mainline 7.3.3 arm64 kernel boots through smp/GIC/timer init to the expected no-rootfs panic and a clean PSCI SYSTEM_OFF exit. Change-Id: I85d3b2fd4083855ad63761010aa91083389e2b36 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…CI SystemEvent - gicreg[7]: redistributor size is ncpus * 0x20000 (each vCPU owns a 2-page GICR frame). With 0x20000 hardcoded the kernel crashed in gic_iterate_rdists as soon as num_cpus > 1 (kvmhost.c is proven with 0x20000 only because it runs a single vCPU). Verified: python ctypes replica with 2 vCPUs boots through the whole boot to the expected no-rootfs panic once the DTB declares 0x40000. - VcpuExit::SystemEvent on aarch64 is the guest's PSCI SYSTEM_OFF/ RESET (panic=-1 or clean poweroff) — log it and stop the vCPU cleanly instead of 'unhandled exit'. Change-Id: I8e558d25eb9e9a4a87047f2539816508f3b8db96 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…ON works Without this bit, kvm_psci_vcpu_on() treats a never-entered secondary vCPU as 'already on' (arch.power_off == false) and returns PSCI_RET_ALREADY_ON, which the guest maps to -EINVAL: 'psci: failed to boot CPU1 (-22)', SMP stays at 1 CPU. QEMU/kvmtool set this flag for secondaries; we didn't. Verified on-device with a ctypes replica: 'smp: Brought up 1 node, 2 CPUs', CPU1 boots with its own GICR at 0x080d0000. Change-Id: I15173027bbb3ebb63e1bf74db5273e2d6fdb2780 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…irtio chain embed_kernel_modules keyed the module tree off uname -r, i.e. the HOST kernel — on a phone Dom0 that is the Android kernel, so nothing was embedded and --rootfs could never mount anything on Debian arm64 guest kernels where CONFIG_EXT4_FS/VIRTIO_MMIO/VIRTIO_BLK are =m. - Pass the guest kernel version (parsed from the --kernel filename, vmlinuz-<version>) through generate_initrd(). - Embed the ext4 dependency chain (crc32c-cryptoapi, crc16, jbd2, mbcache, ext4) plus virtio_mmio and virtio_blk in dependency order. Module chain verified with 'modprobe --set-version <guest> --show-depends ext4' on the device. - clone-init: insmod them before waiting for /dev/vda. Change-Id: I26d2a81aa01f2f8d392f028229949b6b14667a70 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…pecs The GIC node inherited kvmhost.c's #interrupt-cells=<1>, so every 3-cell interrupt spec (PPIs 4x, PL011, virtio-mmio devices) parsed as garbage: 'arch_timer: No interrupt available, giving up' and 'request_threaded_irq: -EINVAL' in virtio_blk probe (vm_find_vqs -> init_vq). kvmhost only ever ran earlycon so the breakage was invisible there. Flags stay unchanged: 0x108 = level-low (PPIs), 4 = level-high (SPIs). Change-Id: I0dba18c807434d953f3eaca8f9fd18a966249b5f Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…arm64 SIGSEGV) On arm64 the single KVM memory slot lives at GPA 0x40000000, but the virtio layer assumed guest RAM starts at GPA 0: gpa_to_ptr computed host_ptr + gpa, so the first virtio device whose DRIVER_OK path reads guest queues (virtio-vsock prepare_activate) dereferenced 0x40000000 past the mmap → SIGSEGV in MmioTransport::handle_status_write. Plumb GuestMem::base() (0 on x86, RAM_BASE on arm64) through MmioBus → MmioTransport → Virtqueue and via a new VirtioDevice:: set_mem_base() to devices that do their own translation: - queue.rs gpa_to_offset(): checked_sub(mem_base) first - vsock.rs gpa_to_ptr(): subtract base + bounds-check (null on bad GPA; prepare_activate now skips out-of-range queues instead of crashing) - net.rs gpa_to_ptr()/gpa_to_hva(): subtract base - balloon.rs process_inflate(): PFNs are relative to mem_base (previously used raw PFN addresses + hardcoded x86 3-4GB hole) - vmm/mod.rs: mmio_bus.set_mem_base(guest_memory.base()) at both set_guest_memory_with_hole call sites (fresh VM and restore) Change-Id: I8f1f1a72636280eea30844d12eb81fe1538a3717 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I3ebaeb2406309a2887e56747e094e46c59b87f92 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…_TYPE_CPU The vCPU run loop's MMIO-write path (virtio QUEUE_NOTIFY kicks) called vm_fd.set_irq_line(irq, ...) directly. On aarch64 that ioctl word encodes irqtype=0 (KVM_ARM_IRQ_TYPE_CPU) + line number — a CPU/PPI line, not the SPI the DTB promises (interrupts=<0 5 4> etc.). The guest virtio driver therefore never received its interrupt: virtio_blk probed, /dev/vda appeared, but the first real block request (ext4 mount) never completed — console stalls after '[vda] ... logical blocks' with no EXT4 mount. Route through vmm::set_spi_line() (KVM_ARM_IRQ_TYPE_SPI, same as the vsock/balloon paths and dom0/kvmhost.c's proven vblk_inject). On x86 set_spi_line degrades to the raw call, so behavior is unchanged. Change-Id: Id501c415b9ab013ac978a14d3d402c87292984d7 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…SPI number strace on-device showed the block-device kick DID reach clone, but ioctl(4, KVM_IRQ_LINE, ...) = -1 EINVAL: the kernel validates the KVM_ARM_IRQ_TYPE_SPI irqid as an INTID (>= 32, kvmtool does spi_irq+32, QEMU does irq+GIC_INTERNAL). Passing the bare SPI number (5/6/7) failed the range check, so every virtio completion IRQ was silently dropped and the first ext4 mount read hung forever after '[vda] ... logical blocks'. Change-Id: Ia958745005e6f1b08de237816136a9ab8a4919b9 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I5fa27e86d46d8535637b631f4c513812d30c2a06 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: I510db7ed12cb461b7a8f2e485886e1572ba210aa Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
…ection (temp) Change-Id: Ib7624e5c038ec26c70095851770d01ce02544e8a Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: Ie9d6656c213b1cb7807bc850abd92dbd4f73a417 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: Ie400c173fbde16da5614362058c0fc003feb4bc1 Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Change-Id: Ifbc5f3edc5f3f5e14876eceed651e9f84daf767d Signed-off-by: sysretq0 <sysretq0@users.noreply.github.com>
Contributor
|
Thanks for this, and for the work on getting clone running on an arm64 host. I read through the whole diff. I can't merge it as is, but there's a piece of it I'd take. The main problem is scope. The title and description say this is a CI workflow, but it's +1468 lines across 22 files and includes a full arm64 boot path (FDT builder, vGICv3, PL011). Can you close this and split it into two PRs? Split 1: musl portability + CI (I'd merge this)
Split 2: arm64 port (open as a draft) Debug code that needs to come out first. The last six commits are marked
Bugs I hit while reading:
Other things:
Once it's split, I'll review the first one quickly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.github/workflows/build-static.yml: reproducible musl static builds ofclone,clone-init,clone-agentacrossx86_64-unknown-linux-musl,aarch64-unknown-linux-musl, andarm-unknown-linux-musleabihf(cross-rs toolchain containers, toolchain never installed on dev hosts).Why static musl
Single-binary deliverable that runs identically on bare-metal thin-Dom0 hosts (aarch64 vendor-kernel EL2 host, e.g. MT6789/G99 phone booting the 5.10 vendor kernel) and on x86_64 dev boxes, with no dependency on the host's shared-library stack.
Notes
arm-unknown-linux-musleabihfiscontinue-on-error: true: the current VMM core is x86-only (TSC/CPUID/kvm-clock heavy). The aarch64 target may need the same port before it turns green — artifacts upload as soon as any target compiles.fail-fast: falseso all matrix targets report independently.file/lddstatic-link sanity checks per binary.