tools: queue heavy QA jobs behind the shared local lock

.github/workflows/qa-resource.yml

@@ -0,0 +1,43 @@
+name: QA resource queue
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "tools/qa-resource/**"
+      - ".github/workflows/qa-resource.yml"
+  pull_request:
+    paths:
+      - "tools/qa-resource/**"
+      - ".github/workflows/qa-resource.yml"
+
+# Pull request code runs here, so the token can only read and no secrets are used.
+permissions:
+  contents: read
+
+# A newer commit cancels a pull request's older run. Each push to main gets a group of its own,
+# because a shared group would replace a pending push before it was tested.
+concurrency:
+  group: qa-resource-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  qa-resource-tests:
+    name: QA resource queue tests
+    # The queue reads Darwin-only process data (sysctl KERN_PROCARGS2 and kinfo_proc) and
+    # uses kqueue, so its tests need macOS. They need no Xcode, simulator or build setup.
+    runs-on: macos-26
+    # The 74 tests took about 55 s locally. The limit leaves room for runner setup.
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      # The suite was developed and passed on Python 3.14.
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+      - name: Run the queue tests
+        env:
+          PYTHONDONTWRITEBYTECODE: "1"
+        run: python -m unittest discover -s tools/qa-resource -p 'test_*.py' -v

tools/qa-resource/README.md

@@ -0,0 +1,70 @@
+# Run heavy QA jobs
+
+Use this queue for package builds and tests, Blender renders, and simulator runs. Run quick unit checks directly; they never need to wait for this queue. Python 3 on macOS is the only dependency.
+
+From the worktree, submit a package job and wait for its result:
+
+```sh
+python3 tools/qa-resource/qa_resource.py submit --job-id package-001 --kind package --cwd "$PWD" --admission-deadline 1200 --timeout 600 -- swift test --package-path ios/HouseScanKit --jobs 1
+python3 tools/qa-resource/qa_resource.py wait --job-id package-001 --max-wait 1800
+python3 tools/qa-resource/qa_resource.py status --job-id package-001
+python3 tools/qa-resource/qa_resource.py status
+python3 tools/qa-resource/qa_resource.py cancel --job-id package-001
+```
+
+Use `run` instead of `submit` to stay in the foreground. Use `--kind render` for renders. For simulator work, use `--kind simulator --device UDID`; other kinds reject `--device`. Pick a new job id for every attempt. Existing job directories are never overwritten.
+
+All subcommands accept `--state-dir DIR` and `--lock FILE`, before or after the subcommand. Defaults are `~/.codex/qa-resource` and the existing shared `~/.codex/local-ios-qa.lock`. Do not delete or replace the shared lock file.
+
+Read `~/.codex/qa-resource/jobs/ID/job.json` for status, reasons, timestamps, ownership, and cleanup errors. `output.log` contains child output. `runner.log` contains timestamped runner events, also echoed by `run`. `submit` prints one JSON receipt with `jobId`, `jobDir`, `resultFile`, and runner `pid`. `wait` uses the job's kernel liveness lock, not polling. Its default maximum wait is 86400 seconds.
+
+Nothing pushes a completion event to T3 or a manager session. A parent learns the result in one of two ways. It can block in `wait`, running it again with the same id if its shell tool's time limit is shorter than the job. Or it can start `run` as a background command and let its host report the exit. Either way, `job.json` holds the result.
+
+`cancel` never signals a PID. Each runner reads a per-job FIFO, `cancel.fifo` in the job directory, and cancels itself when a byte arrives. The command that creates the job opens the read end before the runner starts and hands it to the runner, so a cancel sent during runner startup waits in the pipe and takes effect once the runner listens. When neither holds the read end, `cancel` cannot open the FIFO, reports that nothing was sent, and returns 4, so a reused runner PID can never receive the request. Jobs started by an older runner have no FIFO, and `cancel` refuses them. If `submit` is interrupted after starting the runner, it cancels that runner, which is still its own unreaped child, and prints the receipt with `cancelRequested: true` before returning 130, so `wait` can follow the job to `cancelled`. If the interruption arrives while the receipt is being written, stdout was blocked, usually by a pipe nobody reads. `submit` then still cancels the runner and returns 130, but prints no second receipt, because writing again would block again and leave it unstoppable. The caller supplied the job ID, so `wait ID` and `job.json` still report the outcome.
+
+Read `job.json`, not just the exit code. `run` and `wait` return the child's code for success or failure, 124 for child timeout, 130 for cancellation, 75 for admission timeout, and 125 for sensing or cleanup errors. `wait` returns 3 when its own deadline expires without affecting the job, 4 for a runner that died without a terminal result, and 2 for an unknown job. Child exit codes can overlap these values. If a child dies from signal N, `run` and `wait` return 128+N. `job.json` keeps the negative child return code in `exitCode` and N in `signal`.
+
+Admission requires memory pressure level 1 or 2, at least 35% memory free, and at least 5 GiB free on the HOME volume. These are operational limits copied from `qa-slot.py`, not calibrated measurements. Unreadable resources fail admission. Heavy build and render processes block admission, except a verified agent-device `testCommand` service, meaning an `xcodebuild test-without-building` with no other action whose every `-only-testing` selector names `testCommand`. That service blocks a simulator job unless every one of its `-destination` values names a different device by `id=`, because a name, an OS or an implicit destination may resolve to the job's device. The named simulator must exist and be Shutdown; other booted devices do not block. Package and render jobs never call `simctl`.
+
+Heavy-process exit waits use kqueue. Memory recovery, disk space, and device state have no stdlib kernel notification here, so they are resampled every 30 seconds. The global lock stays held during resource waits. The admission deadline covers queueing and resource checks. Simulator boot and bootstatus each have a fixed 300-second limit, an operational limit rather than a measured startup time. Cancellation waits for the boot result and ownership record. If boot times out, the runner reports the device state but never shuts it down without proof of ownership. The child timeout starts separately.
+
+## Inside a machine-wide command lock
+
+Some machines also make every heavy command take an outer lock, such as `/usr/bin/lockf -k <lock> <command>`. Take that lock first and run the queue's `run` inside it:
+
+```sh
+/usr/bin/lockf -k <lock> python3 tools/qa-resource/qa_resource.py run --job-id JOB --kind package --cwd "$PWD" --admission-deadline 1800 --timeout 3600 -- <command>
+```
+
+`lockf` holds its lock only until its own command exits. `run` stays in the foreground until cleanup ends, so both locks cover the whole job. `submit` returns as soon as the detached runner starts, so under `lockf` the job runs without the outer lock. Don't put `lockf` inside the queue (`... -- lockf <lock> <command>`) either. That reverses the lock order, so a job waits on another lane that holds the outer lock while it waits for the queue. A simulator job would also boot its device before it holds the outer lock. Cancel with `cancel --job-id`, not by killing `lockf`.
+
+## Escaped descendants
+
+Each payload gets a random `QA_RESOURCE_OWNER_<token>=<jobId>` environment entry. At admission and cleanup, the runner reads process memory with Darwin `KERN_PROCARGS2`. An exact entry or live-parent ancestry proves ownership at discovery. A `ps` snapshot only nominates descendants, because its PIDs may have been reused since. Ancestry counts only when the kernel's current parent chain, read from the pinned process up to the payload, holds at every link, and each parent started no later than its child. Otherwise the process is not treated as owned. The runner resets an inherited `SIGCHLD` disposition to the default while it owns the payload, so the payload stays an unreaped zombie that keeps its exit status and pins its PID and process-group id. The parser checks every NUL-separated string after the executable because empty `argv[0]` is indistinguishable from padding. Admission accepts only owner headers with a 32-character lowercase hexadecimal token.
+
+On cancellation, timeout, and normal exit, the runner signals only its pinned child group: SIGTERM, then SIGKILL after the grace period. The group leader stays unreaped until every check finishes, so the group id cannot be reused while the runner signals it. Escaped processes are discovered before that SIGTERM, so their proof is recorded before their parents die, and they are reported but never signalled. A PID recheck followed by `kill` cannot rule out the PID being reused in between, and macOS has no signal bound to a process identity, so the runner does not try.
+
+An escaped process that is still alive makes cleanup incomplete. The job ends `cleanup_failed` (exit 125) with the process listed in `cleanupError`, and the runner keeps the shared lock through the cleanup hold before releasing it. A person decides whether to stop the process.
+
+`escapedProcesses` in `job.json` and `runner.log` records `pid`, `start_us`, `uid`, human-readable `start`, executable at discovery, proof, and outcome. `exited` means the process ended on its own before cleanup finished; `retained` means it was still alive.
+
+Confirmation requires two consecutive complete empty passes at least 0.1 seconds apart, after the last observed owned process. Incomplete reads or expired inspection budgets never count as empty. Discovery errors do not skip signaling the pinned group, but the job reports `cleanup_failed` and enters the cleanup hold.
+
+Admission waits for live same-user owner entries, including orphans from dead runners or expired holds, and for any `retained` process recorded by an earlier job while that exact execution (PID, start time and uid) is still alive. The second check covers escapees proven only by ancestry, which carry no token. Admission never signals these blockers. The runner and its ancestors are excluded as a defense. The payload does not inherit the global lock descriptor. `run` and `submit` reject an inherited owner entry before creating a job, returning 2 with `nested qa-resource jobs are not supported: this process belongs to job X`.
+
+## What the tool cannot guarantee
+
+The token is a cooperative marker in mutable process memory, not an authenticated exec record. A process can overwrite its environment strings, exec without the token, or copy the token. Copying it onto an unrelated same-uid process makes cleanup report that process as retained and admission wait for it; the runner never signals it.
+
+Invisible descendants are not reported. Darwin omits the environment of `cs_restricted` processes without an error, other-uid environments are unreadable, and already-reparented tokenless helpers have no ancestry after natural exit. An unreadable environment without ancestry is not ownership proof. If ancestry finds a descendant under another uid, it is reported like any other escapee. `succeeded` means no observable owned process remained, not that every descendant has exited.
+
+A chain whose processes each live shorter than one scan interval can evade two-pass confirmation.
+
+The queue cannot identify a foreign shared-lock holder or attribute a heavy process to a project. FIFO ordering applies only to this tool's tickets; `qa-slot.py` and `package-run.py` still race for the shared flock. Cancellation stops only the job's own child group, reports proven escaped descendants without signalling them, and shuts down only the device the job booted. The runner handles SIGTERM, SIGINT, and SIGHUP; later signals cannot interrupt cleanup. `run` and `submit` hold those signals from job creation until their handlers are installed, so an interruption at any point after the job exists is recorded as `cancelled`. If the job directory or its first record cannot be created, or a later setup failure cannot be recorded, the directory is removed and the id stays reusable. A stale job's recorded group and booted-device ownership are reported, never cleaned. Inspect those manually before reusing a device.
+
+
+Boot ownership does not identify a simulator session; if another lane shuts down and re-boots the same UDID mid-job, final shutdown affects their session.
+
+A stuck helper is killed at its subprocess timeout, but `subprocess.run` then reaps it without a deadline, so that failure can exceed the stated limits.
+
+If group or escaped-process cleanup fails, the runner holds the shared lock for up to 300 more seconds, an operational limit rather than a measured recovery time. It releases the lock after two complete empty passes or when the hold expires, records retained PID identities and the hold outcome in `cleanupError`, and keeps status `cleanup_failed`. Same-user token-bearing survivors still block later admission.