Cosim-GPU

YAML-driven QEMU + gem5 co-simulation for AMD MI300X

English · 简体中文

Getting started · Topology and hardware support · Architecture

Define QEMU and Device Model settings in YAML, then launch a session with the Rust cosim CLI. The Host runs Host-side QEMU and separate gem5 or QEMU Device Model processes. QEMU uses KVM or TCG to run the vHost OS and presents each vGPU/vGPGPU through the vfio-user interface backed by libvfio-user.

Architecture in motion

Describe one machine or a group of machines in YAML. Each vHost has its own QEMU process, GPU Device Models and shared memory pool on the Host.

Single-node multi-GPU 1 vHost · 8 MI300X 96 GiB pool gpu-8.yaml gives each GPU its own gem5 process.

Multi-node multi-GPU 4 vHosts · 32 MI300X 4 × 96 GiB pools cluster-4x8.yaml uses count: 4 to create four independent machines in one local session.

QEMU ↔ gem5: control messages and shared memory

Register and Doorbell accesses travel over a vfio-user Unix socket. QEMU and gem5 map the same pool file for vHost RAM and VRAM; file descriptors and mmap give both processes access to the shared data. The animation shows an 8 GiB RAM region and a 16 GiB VRAM region in one 24 GiB pool.

Explore multi-machine topologies and the shared memory model.

Architecture

flowchart LR
    YAML[Topology YAML] --> CLI[Rust cosim launcher]

    subgraph Host OS
        CLI --> H[Host-side QEMU]
        CLI --> G[gem5 MI300X Device Model]
        CLI --> Q[QEMU Device Model]
        H <-->|vfio-user / libvfio-user| G
        H <-->|vfio-user| Q
        H --- P[Machine memory pool]
        G --- P
        Q --- P
    end
    H -->|KVM or TCG| V[vHost OS · Linux · ROCm/HIP]

Loading

The launcher prepares shared memory and disk overlays, starts Device Models before Host-side QEMU, and cleans up the session. Device Models implement the simulated vGPU/vGPGPU behavior and timing.

Quick start

just dev-submodules
just dev-image        # Non-Linux Hosts, or set COSIM_ENV=container

# Build the toolchain and x86_64 vHost image on the Host with KVM
just build-all
just run-topology resources/machines/single-gpu.yaml

The first image build uses the selected vHost recipe under resources/images/recipes/ and stores its verified disk and kernel under build/images/. Inspect the resolved recipe and launch commands without building or starting processes:

just run-topology resources/machines/single-gpu.yaml --dry-run

Sample topologies

• resources/machines/single-gpu.yaml — start a single-GPU vHost.
• resources/machines/mixed-models.yaml — combine gem5 and QEMU Device Models.
• resources/machines/gpu-8.yaml — inspect an eight-GPU machine with a 96 GiB memory pool.
• resources/machines/host-only-aarch64.yaml — check an ARM64 vHost without a Device Model.

Example: one machine with gem5 and QEMU devices

version: 1
imports:
  - vhosts/qemu.yaml
  - devices/gpu/mi300x.yaml
  - devices/pci/qemu-edu.yaml
machines:
  - id: m0
    vhost: {use: vhost-x86-rocm, memory: 32G}
    devices:
      - {use: mi300x, count: 2}
      - {use: qemu-edu}

The selected vHost profile includes its image recipe and boot inputs. Build outputs are stored under build/.

The QEMU Device Model example serves the real edu PCI model through x-vfio-user-server from the QEMU built by just build-qemu.

Host and build environment

Launcher and configuration tools

Use Rust 1.95+, Cargo and just for the launcher. QEMU and gem5 builds, topology runs and Rust checks use scripts/host/dev.sh. On non-Linux Hosts or with COSIM_ENV=container, they use a temporary container from scripts/host/Dockerfile.dev. vHost image and HIP builds, plus just run-launch and just test-gpu, use their own Docker workflows.

Linux Host sessions

• glibc and setpriv from util-linux.
• QEMU for the Host architecture and qemu-img.
• x86_64 Hosts use qemu-system-x86_64; ARM64 Hosts use qemu-system-aarch64.
• COSIM_HOST_ARCH selects the build architecture and defaults to the current Host. gem5 builds into build/gem5/x86_64/VEGA_X86 on x86_64 and build/gem5/aarch64/VEGA_X86 on ARM64.

MI300X and QEMU Device Models

• MI300X co-simulation needs gem5 VEGA_X86, QEMU with vfio-user support, vHost image inputs, and KVM for the Ubuntu vHost.
• QEMU device servers need a build with --enable-vfio-user-server.
• The supplied Ubuntu/ROCm vHost image builder targets x86_64.
• vHost image builds and simulator artifacts need sufficient disk space. The memory-pool filesystem must accommodate the complete machine layout.

Repository map

• Rust workspace: crates/cosim-topology/, crates/cosim-runtime/ and crates/cosim-cli/.
• Simulation models: third_party/gem5/, third_party/qemu/ and configs/sst/.
• GPU workloads and vHost firmware inputs: third_party/gem5-resources/.
• Machine configurations: resources/vhosts/, resources/devices/, resources/machines/ and resources/clusters/.
• Example programs: samples/ is reserved for programs run after a vHost boots.
• Project tools: scripts/, justfile and .agents/.
• vHost image recipes: resources/images/recipes/ with Packer/QEMU metadata, Ansible playbooks and OS-specific files.
• Shared vHost scripts: resources/scripts/ for scripts used by multiple vHost recipes.
• Host and CI scripts: scripts/host/ for Host tools called by just, and scripts/ci/ for workflow-only helpers and tests.

More documentation

• Documentation home
• Reference
• Contributing

The top-level project uses the Apache 2.0 license. Submodules retain their respective licenses.