Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions quest/m0/ci/queue-concurrency.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,19 @@ host load during the trial.
The trial started early (decided 2026-11-05, maintainer, /quest-plan): the
second queue runner `nixos-moq-ci2` landed in
[kixelated/nixos#24](https://github.com/kixelated/nixos/pull/24), and ruleset
2420853 went to `max_entries_to_build: 2` at 2026-11-05T15:49:30Z. A scheduled
task sets it back to 1 at 2026-11-07T08:00Z. Measure the setting-2 window
against a comparable setting-1 window before it, such as
2026-11-03 06:47Z to 2026-11-04 06:47Z, and check the ruleset reads 1 again
before recording the result.
2420853 went to `max_entries_to_build: 2` at 2026-11-05T15:49:30Z.

It ran out of disk the same day, so the maintainer ended it early
(decided 2026-11-05, maintainer, /quest-complete; ruleset back to 1 at
2026-11-05T18:02:15Z, scheduled revert cancelled). At 17:12Z

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merge-group build did not fail at 17:12Z: the job started 17:12:49Z, hit error: couldn't create a temp dir: No space left on device at 17:16:35Z, and concluded 17:16:42Z. Since this record uses second-level timestamps everywhere else, 17:12Z reads as the failure time and will mislead anyone reconstructing the incident.

`nixos-moq-ci2` failed a merge-group build with `No space left on device`
under `/var/cache/github-runner-work/moq/moq`
([job](https://github.com/moq-dev/moq/actions/runs/42072368255/job/127403437474)),
dropping [moq-dev/moq#7716](https://github.com/moq-dev/moq/pull/7716). Every
CI runner on `nixos` shares one XFS project quota (project 42000, 667 GiB
hard; 578 GiB used afterwards), and each moq queue runner keeps its own
Cargo targets and mbx caches inside it (`modules/ci-runners.nix`: MBX target
Comment on lines +48 to +49

@pullfrog pullfrog Bot Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says each queue runner keeps its own mbx caches, while the Method paragraph above (line 31-33) still warns that "the shared mbx cache" could erase the gain and tells the retry to watch for it. In this failing job mbx state lived in /var/cache/github-runner-home/.cache/mbx on moq-ci2, and .github/actions/rust-cache/action.yml notes runners on one host may share a home but never a workspace, so the per-runner wording here looks right and the shared-cache premise is stale. Mark that earlier sentence superseded (or say which holds) so the record and the retry guidance agree.

25 GiB, GC 50-80 GiB). Before a retry, size the per-runner caches or the
quota in kixelated/nixos for two concurrent moq builds, and show the quota's
headroom with both runners warm. The 15:49Z-18:02Z window is too short and
too disturbed to compare.
Loading