Repository navigation
quest: the queue concurrency trial ran out of disk #7970
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,8 +35,19 @@ host load during the trial. | |
| The trial started early (decided 2026-11-05, maintainer, /quest-plan): the | ||
| second queue runner `nixos-moq-ci2` landed in | ||
| [kixelated/nixos#24](https://github.com/kixelated/nixos/pull/24), and ruleset | ||
| 2420853 went to `max_entries_to_build: 2` at 2026-11-05T15:49:30Z. A scheduled | ||
| task sets it back to 1 at 2026-11-07T08:00Z. Measure the setting-2 window | ||
| against a comparable setting-1 window before it, such as | ||
| 2026-11-03 06:47Z to 2026-11-04 06:47Z, and check the ruleset reads 1 again | ||
| before recording the result. | ||
| 2420853 went to `max_entries_to_build: 2` at 2026-11-05T15:49:30Z. | ||
|
|
||
| It ran out of disk the same day, so the maintainer ended it early | ||
| (decided 2026-11-05, maintainer, /quest-complete; ruleset back to 1 at | ||
| 2026-11-05T18:02:15Z, scheduled revert cancelled). At 17:12Z | ||
| `nixos-moq-ci2` failed a merge-group build with `No space left on device` | ||
| under `/var/cache/github-runner-work/moq/moq` | ||
| ([job](https://github.com/moq-dev/moq/actions/runs/42072368255/job/127403437474)), | ||
| dropping [moq-dev/moq#7716](https://github.com/moq-dev/moq/pull/7716). Every | ||
| CI runner on `nixos` shares one XFS project quota (project 42000, 667 GiB | ||
| hard; 578 GiB used afterwards), and each moq queue runner keeps its own | ||
| Cargo targets and mbx caches inside it (`modules/ci-runners.nix`: MBX target | ||
|
Comment on lines
+48
to
+49
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This says each queue runner keeps its own mbx caches, while the Method paragraph above (line 31-33) still warns that "the shared mbx cache" could erase the gain and tells the retry to watch for it. In this failing job mbx state lived in |
||
| 25 GiB, GC 50-80 GiB). Before a retry, size the per-runner caches or the | ||
| quota in kixelated/nixos for two concurrent moq builds, and show the quota's | ||
| headroom with both runners warm. The 15:49Z-18:02Z window is too short and | ||
| too disturbed to compare. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The merge-group build did not fail at 17:12Z: the job started 17:12:49Z, hit
error: couldn't create a temp dir: No space left on deviceat 17:16:35Z, and concluded 17:16:42Z. Since this record uses second-level timestamps everywhere else, 17:12Z reads as the failure time and will mislead anyone reconstructing the incident.