mini-run

A minimal container runtime CLI in Go — the core primitives real runtimes (runc) use, without image management, overlayfs, a daemon, or networking:

• Namespace isolation — CLONE_NEWPID (container sees itself as PID 1), CLONE_NEWNS (isolated mount table), CLONE_NEWUTS (custom hostname), CLONE_NEWIPC (isolated IPC), via syscall.SysProcAttr.Cloneflags.
• Filesystem jailing via pivot_root — not chroot: the old root is fully unmounted and detached, preventing container breakout.
• cgroups v2 resource limits — per-container memory ceiling and CPU quota under /sys/fs/cgroup/mini-run/<id>/.
• Interactive PTY passthrough — stdio is wired straight through so /bin/sh, top, etc. work interactively.

It is a stateless CLI, not a daemon: sudo mini-run run <rootfs> <cmd> [args...] against a rootfs directory you've already unpacked yourself (e.g. Alpine's minirootfs tarball, extracted).

Repository Layout

.
├── cmd/mini-run/             CLI entrypoint
│   ├── main.go               arg dispatch: run | child (hidden) | -h
│   ├── run.go                `run` subcommand: flags -> container.RunOptions
│   └── child.go              hidden `child` subcommand -> container.RunChild
├── internal/
│   ├── container/
│   │   ├── types.go          RunOptions
│   │   ├── run_linux.go      parent-side orchestration (namespaces, cgroups, wait)
│   │   ├── child_linux.go    PID-1 child logic: hostname, pivot_root, exec
│   │   ├── rootfs_linux.go   pivot_root + synthetic /dev implementation
│   │   ├── id.go             random 12-hex-char container ID
│   │   ├── id_test.go
│   │   └── *_other.go        non-Linux build-tag stubs
│   └── cgroups/
│       ├── cgroups_linux.go  cgroups v2 Setup / AddProcess / Cleanup
│       ├── limits.go         ParseMemory / ParseCPUs (pure, cross-platform)
│       ├── limits_test.go
│       └── cgroups_other.go  non-Linux build-tag stub
├── test/integration/
│   └── mini_run_test.go      black-box, root+Linux-gated integration suite
│                              (automates the Verification checklist below)
├── scripts/
│   ├── fetch-rootfs.sh       downloads + extracts Alpine's minirootfs
│   ├── bench.sh              startup-latency benchmark harness (hyperfine)
│   └── plot_bench.py         renders the benchmark bar chart (matplotlib)
├── docs/
│   └── benchmark.png         chart embedded in this README
├── lima.yaml                 disposable Ubuntu VM config for macOS dev/test
└── go.mod                    zero external dependencies

Architecture

mini-run has no long-running daemon and no supervisor process tree: the parent CLI invocation sets up resource limits, re-execs itself into new namespaces, and then gets out of the way — the containerized command becomes a real PID 1, not a child of some Go supervisor loop.

sequenceDiagram
    participant Host as Host shell
    participant Parent as mini-run run (parent)
    participant CG as cgroups v2 (/sys/fs/cgroup/mini-run/<id>)
    participant Child as mini-run child (becomes PID 1)
    participant Target as exec'd command

    Host->>Parent: sudo mini-run run --memory 128m ./rootfs /bin/sh
    Parent->>Parent: NewID(), resolve rootfs, ParseLimits()
    Parent->>CG: mkdir <id>/, write memory.max / cpu.max
    Parent->>Child: re-exec self as "child" with Cloneflags<br/>NEWNS|NEWUTS|NEWIPC|NEWPID, Pdeathsig=SIGKILL
    Note over Child: new process, born as PID 1 inside<br/>its own PID/mount/UTS/IPC namespaces
    Parent->>CG: write cgroup.procs (attach child's real host PID)
    Child->>Child: Sethostname("mini-run-&lt;id[:8]&gt;")
    Child->>Child: pivotRoot(rootfs) — see filesystem diagram below
    Child->>Target: syscall.Exec(resolved cmd) — replaces its own process image
    Note over Target: target command IS the container's PID 1 now;<br/>stdio was wired straight through from the host shell
    Target-->>Parent: process exits
    Parent->>CG: rmdir &lt;id&gt;/ (Cleanup(), retries up to ~500ms)
    Parent-->>Host: propagate the container's exit code

Loading

Two design choices worth calling out:

• syscall.Exec, not a subprocess — the child process replaces its own image with the target command instead of spawning it as a child. This means the container's PID 1 is /bin/sh (or whatever was requested), with correct init/signal/reaping semantics, rather than a Go process that has to babysit a subprocess.
• cgroup limits are created before the container exists — memory.max and cpu.max don't require a PID to be written, so the cgroup is fully configured first and the container's host PID is attached to it right after the namespaced process starts, closing the window where an unconstrained process could run free.

Filesystem jailing: pivot_root, not chroot

chroot only changes what path resolution treats as /; the old root filesystem is still mounted and reachable through tricks like relative .. traversal via an open file descriptor. pivot_root swaps the entire mount tree's root and lets the old root be fully unmounted and deleted, so there is nothing left to break out into:

Before pivot_root                      After pivot_root
(inside the new mount namespace)       (inside the new mount namespace)
----------------------------------     ----------------------------------
/                                      /                                   (== former rootfs,
├── proc/, sys/, dev/, ...             ├── bin/, etc/, usr/, ...           now the real root)
├── home/.../container-runtime/        ├── dev/                            (fresh tmpfs: null, zero,
│   └── rootfs/  <- bind-mounted       │                                   full, random, urandom,
│       onto itself so pivot_root(2)   │                                   tty, fd -> /proc/self/fd)
│       accepts it as the new root     ├── proc/                           (freshly mounted here,
└── ...rest of the host filesystem...  │                                   reflects the new PID ns)
                                       └── .old_root                       unmounted (MNT_DETACH) and
                                                                           os.RemoveAll'd -- gone

The sequence, from rootfs_linux.go: make all mounts private+recursive (so pivot_root doesn't EINVAL on distros with shared mount propagation) → bind-mount rootfs onto itself (a mount-point requirement of pivot_root(2)) → build a minimal synthetic /dev via tmpfs + hand-rolled mknod (never bind-mount the host's real /dev) → pivot_root → chdir("/") → mount a fresh /proc (only valid after the pivot, since it must reflect the new PID namespace) → unmount and recursively delete .old_root.

Requirements

Namespaces, pivot_root, and cgroups v2 are Linux kernel features and do not exist on macOS/Windows. Building the CLI works anywhere (Go's build tags isolate the Linux-only code), but actually running a container requires Linux, root, and a cgroups v2 unified hierarchy mounted at /sys/fs/cgroup.

Dev/test environment (macOS)

A Lima config is included for a disposable Ubuntu VM:

brew install lima
limactl start --name mini-run ./lima.yaml
limactl shell mini-run
cd container-runtime
go build -o mini-run ./cmd/mini-run
./scripts/fetch-rootfs.sh
sudo ./mini-run run --memory 128m --cpus 0.5 ./rootfs /bin/sh

(Ubuntu/systemd is used rather than Alpine/OpenRC because systemd keeps the root cgroup free of directly-attached processes — otherwise enabling the cpu/memory controllers on /sys/fs/cgroup fails with EBUSY.)

If you already have a Linux box, skip Lima and just go build + sudo ./mini-run run ... there directly.

Usage

sudo mini-run run [--memory 512m] [--cpus 0.5] <rootfs-path> <cmd> [args...]

• --memory — e.g. 512m, 1.5g (default: unlimited)
• --cpus — fraction of one core, e.g. 0.5, 2 (default: unlimited)

Flags must precede <rootfs-path>; everything after the command is passed through untouched (so sudo mini-run run ./rootfs /bin/sh -c "ls -la" works as expected).

Verification

Run inside the Lima VM (or any Linux box):

1. PID namespace: echo $$ inside the container prints 1.
2. UTS namespace: hostname prints a generated mini-run-xxxxxxxx name, distinct from the host's.
3. Mount namespace: mount inside the container shows only /, /proc, and the synthetic /dev tmpfs — compare against a second limactl shell mini-run window on the host.
4. Process visibility: ps aux inside the container shows only the container's own processes, not the host's (/proc reflects the new PID namespace).
5. Breakout prevention: ls /.old_root inside the container fails with "No such file or directory"; ls / shows only the rootfs's own directories, never host paths like /home or the project directory.
6. cgroup limits applied: from the host, cat /sys/fs/cgroup/mini-run/<id>/memory.max and .../cpu.max match what was requested; cgroup.procs shows the real host-namespace PID (vs. 1 seen inside the container).
7. Memory limit enforced: inside the container, run a memory bomb (sh -c 'a=AAAA; while true; do a="$a$a"; done') — it gets OOM-killed by the cgroup, without affecting the host VM.
8. CPU limit enforced: yes > /dev/null inside the container while watching host-side top — usage caps near the configured fraction of one core.
9. Cleanup: after exit (or Ctrl-C mid-session), /sys/fs/cgroup/mini-run/<id> is gone.
10. Exit code propagation: sudo ./mini-run run ./rootfs /bin/sh -c "exit 42"; echo $? prints 42.

The portable pieces (flag parsing, ID generation, memory/CPU string parsing) build and vet directly on macOS without Lima:

go build ./... && go vet ./...

Testing

Unit tests cover the pure, cross-platform logic (ID generation, memory/CPU parsing) and run anywhere, no Lima or root needed:

go test ./...

Everything else — namespace isolation, pivot_root, cgroup enforcement, exit-code propagation, cleanup — is covered by an integration suite that automates the 10 verification steps above. It's black-box (drives the built binary as a real user would) and requires Linux, root, and a fetched rootfs, so it's gated behind the integration build tag and skips itself with a clear message if either is missing. Run it inside the Lima VM:

go build -o mini-run ./cmd/mini-run
./scripts/fetch-rootfs.sh
sudo go test -tags integration ./test/integration/... -v

Benchmarking

scripts/bench.sh measures container startup latency and compares it against a staircase of baselines to show what each isolation layer costs: raw exec, chroot alone, chroot + namespaces (via unshare), and mini-run itself (+ cgroups v2 setup/cleanup and pivot_root). It uses hyperfine for statistically sound timing and, if matplotlib is available, renders bench-results.png (via scripts/plot_bench.py) as a labeled bar chart of the four variants. Run inside the Lima VM (both hyperfine and matplotlib are preinstalled by lima.yaml's provisioning script):

sudo ./scripts/bench.sh

Results

Measured on an aarch64 Lima VM (2 vCPUs, 4GiB RAM, Ubuntu 24.04) running on a macOS host — absolute numbers will vary by machine, but the shape of the staircase (what each layer adds) is the interesting part:

Command	Mean	Min	Max	Slowdown vs. raw exec
a: raw exec	1.1 µs	0.0 µs	1027.1 µs	1×
b: chroot only	253.4 µs	166.1 µs	5011.2 µs	~238×
c: chroot + namespaces	519.8 µs	452.4 µs	1132.3 µs	~488×
d: mini-run	3.36 ms	2.16 ms	17.98 ms	~3160×

Reading the staircase:

• (a) raw exec — true, no isolation at all. Hyperfine's own warning ("command took less than 5 ms") is the tell that this number is really measuring hyperfine's shell-invocation floor, not a meaningful process cost — treat it as "approximately zero," the baseline everything else is relative to, not a precise figure.
• (a) → (b) chroot only, +252 µs — this jump is not the cost of the chroot(2) syscall itself (that's sub-microsecond). It's the cost of spawning the external chroot binary as its own process before it execs /bin/true — i.e. this measures "one extra fork+exec plus a chroot() and chdir() call," which is representative of what a shell-script-based jail would actually cost, not a syscall microbenchmark in isolation.
• (b) → (c) + namespaces via unshare, +266 µs — on top of another process spawn (the unshare binary itself), the kernel now allocates and populates a new PID, mount, UTS, and IPC namespace (CLONE_NEWPID| CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC). Namespace creation has real kernel-side cost — the mount namespace in particular requires copying the process's mount table — which is why this step costs meaningfully more than the chroot-only step even though both are dominated by process-spawn overhead.
• (c) → (d) mini-run, +2.84 ms — this is where mini-run's extra work over the unshare+chroot equivalent shows up, and it's the sum of several things the manual baseline doesn't do at all:
  1. Two full process launches, not one clone(2) — the parent (mini-run run) re-execs itself as mini-run child via os/exec.Command rather than calling clone(2) directly in-process. That's two separate ~2.8MB Go binary startups (runtime init, GC init, etc.) chained together, versus unshare's single clone()+execve().
  2. cgroups v2 setup and teardown — mkdir, two subtree_control writes (root and mini-run/), memory.max/cpu.max writes, a cgroup.procs attach, and on the way out, an rmdir that retries up to 10×50ms if kernel accounting hasn't caught up yet. Each is a synchronous sysfs write with kernel-side validation — unshare/chroot do none of this.
  3. The full pivot_root sequence — making mounts private+recursive, a recursive bind-mount of the rootfs onto itself, a tmpfs mount plus six mknod calls for /dev, pivot_root(2) itself, and finally unmounting (MNT_DETACH) and recursively deleting the old root. chroot is a single syscall by comparison; this is roughly a dozen.
  4. A fresh /proc mount after the pivot (namespace-aware, so it can't happen before).
  • The wide variance (±1.86 ms, up to 18 ms) is consistent with this: it's a long chain of syscalls in a nested-virtualization guest (Lima VM on top of macOS), plus the cgroup cleanup retry loop occasionally kicking in if a previous container's accounting hadn't settled yet.

The takeaway: mini-run's overhead over hand-rolled namespaces is almost entirely cgroups v2 bookkeeping and the pivot_root//dev setup, not the namespaces themselves — namespace creation (b→c) is cheap relative to what a real container runtime does around it (c→d). This roughly tracks how runc and similar runtimes spend their startup budget too.

References

Primitives this project implements:

• namespaces(7), pid_namespaces(7), mount_namespaces(7) — Linux manual pages for the isolation primitives behind CLONE_NEWPID / CLONE_NEWNS / CLONE_NEWUTS / CLONE_NEWIPC.
• pivot_root(2) — why it's used instead of chroot(2) for filesystem jailing.
• cgroups v2 (kernel docs) — the unified hierarchy, subtree_control, memory.max, cpu.max.
• OCI Runtime Specification — the spec real container runtimes (runc, crun) implement; mini-run deliberately implements a subset of the same underlying primitives without the spec's config/bundle format.
• runc — the reference OCI runtime implementation; mini-run's pivot_root sequence follows the same shape (private+recursive remount, self bind-mount, pivot, detach).

Tooling used in this repo:

• Lima — the Linux VM used for macOS development (lima.yaml).
• hyperfine — the benchmarking tool used by scripts/bench.sh.
• Alpine Linux minirootfs — the rootfs fetched by scripts/fetch-rootfs.sh.

License

This project is licensed under the MIT License. See LICENSE for details.