OpenRC breaks cgroups resource accounting with cgroups v2 for an Incus/LXC container

[pva]

When OpenRC runs as PID 1 inside an Incus/LXC container with cgroup v2 delegated controllers, it leaves PID 1 and early processes such as agetty in the cgroup namespace root (in /sys/fs/cgroup/cgroup.procs). This cgroup is not the root cgroup as defined in man 7 cgroups, and thus the cgroups v2 "no internal processes" rule that is mentioned on the same man page applies.

As a result, the "container-root-cgroup" contains member processes and cannot enable domain controllers in cgroup.subtree_control. This breaks the delegation of CPU, memory, I/O, and pids controllers to child cgroups.

Consequences in more details

The cgroups service effectively starts “successfully”, but the controllers are not enabled.

During the boot process, errors from init.d/cgroups.in are clearly visible when it tries to write to cgroup.subtree_control:

uz-app3 ~ # lxc-start -F uz-app3-docker
INIT: version 3.14 booting

   OpenRC 0.62.6 is starting up Gentoo Linux (x86_64) [LXC]

 * /proc is already mounted
 * Mounting /run ... [ ok ]
 * /run/openrc: creating directory
 * /run/lock: creating directory
 * /run/lock: correcting owner
/etc/init.d/cgroups: line 92: echo: write error: Device or resource busy
/etc/init.d/cgroups: line 92: echo: write error: Device or resource busy
/etc/init.d/cgroups: line 92: echo: write error: Device or resource busy

The failing operation is effectively (see init.d/cgroups.in line 80):

echo "+${x}" > cgroup.subtree_control

Because the root cgroup contains member processes, the kernel rejects this with Device or resource busy.

OpenRC resource limits do not work correctly under this hierarchy.

Nested runtimes such as Docker can still create cgroup directories and start containers, but Docker resource accounting is incomplete or empty: docker stats reports zeros for CPU, memory, pids and block I/O.

The same problem and possible workaround were reported here:
https://discuss.linuxcontainers.org/t/enabling-root-cgroup-subtree-control-with-alpine-linux-containers/26515

[navi-desu]

note that gentoo's default init isn't openrc-init but sysvinit

we could potentially move pid1 to a cgroup (probably doing so in init.d/cgroup, though it'd be a bit messy), but we can't guarantee everything will be moved if pid1 isn't openrc-init

[acrix]

As far as I understand, /etc/init.d/cgroups shouldn't start in a container.

description="Mount the control groups."

: "${cgroup_opts:="nodev,noexec,nosuid"}"

depend()
{
	keyword -docker -podman -prefix -systemd-nspawn -vserver -wsl
	after sysfs
}

In container root cgroup2 already mounted by lxc, before init run.
In alpine linux container template /etc/init.d/cgroups disabled by default.

[acrix]

As solution /etc/init.d/cgroups can start in any environment but must check if cgroup2 already mounted and skip mount , just populate cgroup.subtree_control, but init process must move self in subtree ( if pid 1 move to subtree, is then all subprocess moved also?). Or container supervisor must set cgroup.subtree_control before run init.

[pva]

@navi-desu On the affected server PID 1 is actually sysvinit, not openrc-init. I was able to move it by hand into a child cgroup, together with the other processes from the cgroup v2 root, and after that cgroup.subtree_control could be populated and Docker resource accounting started working.

Do you have a particular sysvinit setup in mind where moving those processes to a child cgroup would not be valid?

@acrix I do not think OpenRC should generally assume that the cgroups service is irrelevant in containers.

There is some history here: cgroups mounting was split out of sysfs exactly to make it possible to handle cgroups in LXC/LXD containers. The Linux Containers discussion mentions that LXD developers expected OpenRC to mount cgroups inside the container, and the linked OpenRC fix says this was necessary for LXC/LXD:
https://discuss.linuxcontainers.org/t/running-lxd-an-openrc-container-on-a-openrc-system-trouble-with-cgroups/843

In modern Incus/LXC containers the cgroup2 filesystem may already be mounted by the container manager, but that does not make the OpenRC cgroups service useless. In this case the important part is enabling delegated controllers in
cgroup.subtree_control.

Configuring cgroup.subtree_control from the container supervisor could work for some setups, but it is a bad fit for our use case. I need the container to run as a full OS that can start Docker/containerd and manage resource accounting from inside the container. If OpenRC's cgroups service is enabled and it attempts to set up cgroup v2 controllers, it should make the delegated hierarchy valid before writing cgroup.subtree_control.

[pva]

Proposed fix:

Before OpenRC enables cgroup v2 subtree controllers, move all PIDs out of the cgroup v2 root into a dedicated rc.init cgroup.

I was initially unsure whether this should be done only in containers or also on regular hosts. I think it is better to do it unconditionally for cgroup v2, because:

1. As far as I understand, there is no fully reliable way for OpenRC to detect that the current cgroup namespace root is a delegated container root rather than the real host root.
2. systemd also keeps init in a separate init.scope, which serves the same basic purpose: the cgroup root is kept as a parent cgroup, not as a place for running processes.

So the patch makes this part of the normal cgroup v2 setup rather than a container-specific workaround.

I have not tested the patch in the affected Incus setup yet, and that will take some time, but the manual workaround using the same idea fixed the issue there.

The fix is applied to both unified and hybrid cgroup modes. It only affects the cgroup v2 hierarchy.

[acrix]

I think it is better to do it unconditionally for cgroup v2

good decision

there is no fully reliable way for OpenRC to detect

sure, but i think it good idea check if cgroup2 already mounted before mount, description of cgroups service "Mount the control groups.", better description "Enable the control groups."