Repository navigation
ci: fork-safe required checks + keyless coverage + drop needless max-parallel - #70
Conversation
…ation) Fork PRs never receive repository secrets, so integration-tests (which needs VOYAGE_API_KEY) always fails on them. Because the legacy required-check bridge job "tests" depended on integration-tests, the required Tests (3.x) contexts were red on every fork PR, blocking merge permanently. - tests bridge now needs only the fork-safe jobs (unit-tests-no-local-deps, local-model-tests), so the required Tests (3.x) contexts can go green on forks. - integration-tests skips on fork PRs (runs on push to main and same-repo PRs, where secrets exist); integration coverage runs post-merge on main. Branch protection should keep the required set to the fork-safe checks and the Tests (3.x) bridge; Integration Tests (3.x) should not be required. See #69.
This job sets no VOYAGE_API_KEY and installs no local extras, so all integration/local tests skip and it makes zero API calls. The TPM rate-limit rationale (copied from integration-tests) does not apply, and max-parallel: 1 serialized the 6-version matrix ~6x for no reason. Run it in parallel; keep serialization only on integration-tests. Closes #68.
… by review) Drop the fork-skip if: condition. Policy: integration always runs; on fork PRs it is red (no VOYAGE_API_KEY) and that is handled by review + post-merge revert, not by skipping. For same-repo PRs and push to main it runs for real. It must not be a required check. The Tests bridge already excludes it, so fork PRs still go green on the required checks.
jasonqiu95
left a comment
There was a problem hiding this comment.
Automated review of the CI changes. Three points on the required-check bridge; all are on the workflow semantics, none block the fork-safe intent conceptually.
One extra note (not inline, pre-existing): local-model-tests is a required bridge dependency but is not hermetic — it downloads voyage-4-nano from HuggingFace with no -m filter, so a Hub outage/anonymous-throttle can red a required check on fork PRs. Same job-level cascade as the matrix note below.
Address review on #70: - Required checks lost coverage of the keyless mock/offline suite: test_unit.py, test_video_utils.py and test_import_order.py ran only inside integration-tests (bare pytest), which is non-required. They now run in unit-tests-no-local-deps (no API key), which feeds the required Tests bridge. (pytest -m "not integration" was rejected: it is not keyless -- the live-API client/multimodal tests carry no integration marker and fail without a key.) - Matrix-mismatch footgun: needs is job-level, so a failing 3.13/3.14 leg of a bridge feeder would fail the whole job and skip the bridge, leaving the required Tests (3.9-3.12) contexts unreported and the PR blocked. Feeders are now the required set (3.9-3.12); 3.13/3.14 run in separate advisory jobs that are red on failure (visible) but are not bridge dependencies, so they never block merges.
jasonqiu95
left a comment
There was a problem hiding this comment.
Re-reviewed after c9f78a7. All three prior points are addressed correctly:
- Coverage gap — fixed. And your pushback is right:
contextualized_embedcalls_require_api_key()before validation (client.py:129), andAuthenticationErroris aVoyageError, not aValueError— sotest_client.py's validation tests would raiseAuthenticationErrorwithout a key and can't be keyless. No residual gap there; the genuinely-keyless files are the right set. - Matrix/bridge decoupling — fixed. Feeders and bridge are both 3.9–3.12; 3.13/3.14 moved to non-required
*-nextjobs. Correct. - Stale #68 ref — fixed.
Two new points that the fix commit itself introduced (one is a real flakiness risk on a required check):
…action Address review round 2 on #70: - test_video_utils.py::test_optimize_video_e2e_example_video calls _get_client_config() -> hf_hub_download() with no offline fallback, so it hits the HuggingFace Hub. Including it in the required, keyless job made a required Tests (3.x) context depend on HF availability (flaky on outages/rate limits), breaking the job's zero-network invariant. Removed from the fork-safe jobs; it still runs in integration-tests. Required keyless coverage stays via test_unit.py (mocks the HF download) and test_import_order.py. - The four test jobs duplicated their entire install/run step block. Extracted the shared setup into a local composite action (.github/actions/setup-and-test) taking python-version, extras, pytest-args and install-openblas. Kept the four separate jobs and the needs topology intact (no continue-on-error/matrix.include collapse, which would re-couple non-required versions to the required gate).
…ency
Proactive hardening:
- Cache ~/.cache/huggingface in the jobs that load the local model
(local-model-tests[-next], integration-tests). huggingface_hub falls back to
a populated cache when the Hub is unreachable, so an HF outage/rate-limit no
longer fails the required local-model gate. Keeps the deliberate 'download
errors must surface' behavior (no skip-masking) while removing routine flake.
- Composite action passes extras/pytest-args via env instead of interpolating
${{ inputs.* }} directly into run: scripts (avoids the script-injection
pattern actionlint flags).
- Add a concurrency group so superseded PR runs are cancelled (integration is a
~2h serial matrix). Not cancelled on main, so each merged commit is still
validated post-merge.
|
Self-review hardening (pre-empting likely comments):
Consciously deferred (happy to take if wanted): single-source |
jasonqiu95
left a comment
There was a problem hiding this comment.
Re-reviewed after 43b9d90 + 6d57695. Both round-2 findings are resolved, and I checked the new surface for regressions:
1. HF network in a required check — resolved (and verified deeper than the fix comment claims). test_video_utils.py is out of the fork-safe jobs. Note that test_unit.py's TestVideoUtilsIntegration stays in the required job and still calls optimize_video / _compute_basic_usage_for_path / Video.from_file with model="voyage-multimodal-3.5" unmocked — so it's not "mocked" as the reply says. But it's genuinely safe: _get_video_token_config swallows HF errors → None (video_utils.py:308-312), both optimize_video and _compute_basic_usage_for_path have video_config is None fallbacks, and no test in that class asserts on the config-derived estimated_tokens — they only assert ffmpeg-probe values (num_pixels, num_frames, .optimized). So an HF outage degrades gracefully instead of reding the required check. Confirmed resolved.
2. Job duplication — resolved. The composite action (.github/actions/setup-and-test) collapses the four jobs to ~5 lines each, keeps the needs/job topology intact, and passes inputs via env: (not ${{ }} interpolation) — good injection hygiene. POETRY_HOME is hardcoded in the action (correct — composite actions don't inherit workflow env) and the only workflow-level $POETRY_HOME consumer is the coverage step, which does inherit it. No mismatch.
New additions checked, all correct:
concurrency:github.refisrefs/pull/<N>/merge(unique per PR), andcancel-in-progressis gated topull_requestso the post-mergemainintegration gate is never cancelled.- HF
actions/cache: fixed key across parallel legs → save races are non-fatal warnings; warm cache letshuggingface_hubfall back on an outage, a strict improvement over the prior always-live behavior. (Residual: a cold cache + HF outage can still red the requiredlocal-model-testsfeeder, but that's pre-existing, not introduced here.) PYTEST_ARGSword-splitting is safe for all three arg sets (no spaces/globs);SC2086disable is appropriate. Checkout precedes the local action in every job; all actions SHA-pinned.
No new correctness issues. LGTM from my side.
Fixes #69 and closes #68 — CI-workflow-only changes.
#69 — fork PRs can never pass required checks
GitHub does not pass repository secrets to
pull_requestworkflows from a fork, sointegration-tests(needsVOYAGE_API_KEY) is always red on fork PRs. The legacy required-check bridgetestsdepended on it, so the requiredTests (3.x)contexts were red on every fork PR — permanentlyBLOCKED.testsbridgeneedsonly fork-safe, keyless jobs → requiredTests (3.x)can go green on forks.integration-testsruns everywhere (no skip); red on fork PRs by design, gated by review (external: merge + revert if the post-mergemainrun fails; same-repo: don't merge on red). Not a required check (confirmed in the merge UI).Keyless required coverage
The keyless mock suite (
test_unit.py,test_import_order.py) previously ran only inside the non-requiredintegration-tests, so a same-repo PR breakingtest_unit.pywould pass every required check. Now covered by the required feederunit-tests-no-local-deps(no API key).pytest -m "not integration"was rejected — it is not keyless (the live-API client/multimodal tests carry nointegrationmarker; 147 fail without a key).test_video_utils.pyis deliberately NOT in the required jobs: its e2e leg calls_get_client_config()→hf_hub_download()with no offline fallback, so it needs the HuggingFace Hub and would make a required check flaky on HF outages. It runs inintegration-tests.Matrix isolation
needsis job-level, so a failing 3.13/3.14 leg of a bridge feeder would fail the whole job and skip the bridge → required contexts unreported → blocked. Bridge feeders are the required set 3.9-3.12; 3.13/3.14 run in separate advisory jobs (*-next): red on failure (visible) but not bridge dependencies, so they never block merges (rerun turns them green).Dedup
The four test jobs shared an identical install/run block. Extracted into a local composite action (
.github/actions/setup-and-test:python-version/extras/pytest-args/install-openblas). Four separate jobs and theneedstopology kept intact (nomatrix.include+continue-on-errorcollapse, which would re-couple non-required versions to the required gate).#68 — needless
max-parallel: 1onunit-tests-no-local-depsZero API calls (no key, no local extras) → the TPM rationale copied from
integration-testsdoesn't apply. Removed; serialization stays only onintegration-tests.Admin follow-up
None. The
Testsbridge keeps emitting the legacy required contexts;Integration Tests (3.x)is not required. The bridge can be retired later if protection is switched to require the real job names.Related: #67 (the fork PR that surfaced this).