Replace hosted CI with a local validation check - #586
Merged
Merged
Conversation
No GitHub-hosted GA image has Xcode 27 / iOS 27.0 yet, so Validate had been manual-only. Move Lint, CI_iOS Build and CI_iOS Tests onto the badbundle self-hosted runner ([self-hosted, macOS, studio]) and turn the pull_request and push triggers back on. - Skip the self-hosted jobs for pull requests from forks, since the repo is public. - Select Xcode with DEVELOPER_DIR instead of `sudo xcode-select`; the runner has no passwordless sudo. - Keep Release Config on GitHub-hosted macos-26 (it only needs Ruby). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The self-hosted runner runs one job at a time, in a fresh macOS VM per job, so the 14 per-suite test shards only added overhead: on the first run, about 85% of each shard's time was repeated setup (checkout, downloading and unpacking the test products, booting the simulator), and the tests themselves were 308s of 2050s. - Merge CI_iOS Build and the CI_iOS Tests matrix into one CI_iOS Build & Test job that builds for testing and then runs the whole iOSAllTests plan (the same 14 suites the matrix listed), with parallel testing still off. The CI_iOS check job keeps its name. - Correct the header: the fork `if:` guard is not a security boundary, since a fork PR can edit this file; the approval requirement for external contributors is what protects the runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every change is authored and tested on the developer's Mac, so running CI as well is overhead. Instead, `make validate` validates the exact commit locally and posts the result to GitHub as the "Validate (local)" status, which main will require before a PR can merge. - scripts/validate.sh checks the commit out into a separate worktree with its own DerivedData, then runs lint, the Fastlane config check (skipped and noted if the pinned Ruby isn't installed), and a build and full run of the iOSAllTests plan on a dedicated, freshly erased iPhone 18 Pro Max / iOS 27.0 simulator with Xcode 27.0. Test diagnostics collection is off, so a failure doesn't hang for 10 minutes. - .githooks/pre-push posts stored results for commits validated before they were pushed. It never blocks a push. - Remove the Validate workflow, and document the process in README.md, Vault/README.md and the AGENTS.md files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bradleymackey
changed the title
bradleymackey
added a commit
that referenced
this pull request
Sep 25, 2026
…r rbenv (#587) ## Summary This fixes two leftovers from #586: a flaky test that could fail validation, and the Fastlane check that validation was skipping. - **Flaky test:** the "Cancellation is always checked before returning" tests in `Task+RaceTests.swift` created a task and then cancelled it. The task could run `Task.race` to completion before `cancel()` was called. When it did, the race wasn't cancelled and correctly threw `.noTasksScheduled` instead of `CancellationError`. Both `FirstValue` and `FirstResolved` have this race. The tests now wait for the cancellation, using the existing `TaskCancellationWaiter`, before starting the race. `Task.race` itself is unchanged. - **Fastlane check:** Ruby 4.0.5 from `.ruby-version` is now installed on the validating Mac through rbenv. `validate.sh` now also puts rbenv's shims on `PATH` itself. Before this, shells that hadn't run `rbenv init` (non-interactive ones, like an agent's) found the system Ruby 2.6 and still skipped the check. ## Testing - **Before the fix:** running `FoundationExtensionsTests/TaskRaceTests` on macOS with `-test-iterations 500 -run-tests-until-failure`, `FirstResolved` failed with `.noTasksScheduled`. - **After the fix:** 3,000 repetitions all passed. - `bundle exec fastlane lanes` succeeds on Ruby 4.0.5, and `Gemfile.lock` is unchanged. - `make validate` passed on c34aa3c in 3m36s. This time the Fastlane config step ran instead of being skipped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every change to Vault is authored and tested on the developer's Mac, so running CI on top of that is overhead. This PR removes the Validate workflow and replaces it with
make validate, which validates the exact commit locally and posts the result to GitHub as the Validate (local) commit status. A new ruleset onmainrequires that status, with an admin bypass for emergencies, so a green check is still needed to merge.scripts/validate.shchecks the commit out into a separate worktree with its own DerivedData, so uncommitted changes and your usual build folders can't affect it. With Xcode 27.0 it then runs:make lint;.ruby-versionisn't installed;iOSAllTestsplan on a throwaway iPhone 18 Pro Max / iOS 27.0 simulator, created for the run and deleted afterwards.The simulator has to be named exactly "iPhone 18 Pro Max", because the snapshot tests check the device name. The script therefore addresses it by UDID. Test diagnostics collection is off, so a failing run reports straight away instead of hanging for up to 10 minutes. If the run is interrupted, the status is set to
error..githooks/pre-pushposts the stored result for any commit validated before it was pushed, so you can validate before or after pushing. It never blocks a push. Enable it once per clone withgit config core.hooksPath .githooks.Docs:
README.mdgains a Validation section, andVault/README.mdlistsmake validate. TheAGENTS.mdfiles tell agents to validate after every commit on a PR branch, and never to post the status by hand or work around a failing test.Removed:
.github/workflows/validate-all.yml.The check is self-attested: it records that the commit passed on the machine that posted it, not on independent CI.
This PR started as moving Validate onto a self-hosted Mac runner, then onto ephemeral macOS VMs with the 14 test shards merged into one job. In the end, local validation replaced the workflow entirely, so the net diff removes it.
Test plan
make validateon this PR's head commit: lint, build and all tests passed in 3m27s, with no test-host restarts. The Fastlane check was skipped because Ruby 4.0.5 isn't installed on this Mac.mainruleset requiresValidate (local), and this PR shows as mergeable with the check present.🤖 Generated with Claude Code