Skip to content

ci: skip the suite on a documentation-only PR, without blocking it - #107

Merged
MarketDataApp merged 1 commit into
mainfrom
skip-tests-on-docs-only
Sep 17, 2026
Merged

MarketDataApp merged 1 commit into
mainfrom
skip-tests-on-docs-only

Conversation

@MarketDataApp

Copy link
Copy Markdown
Owner

Why

The docs sync bot (marketdata-docs-sync) opens pull requests here that touch
nothing but docs/. Every one runs the full suite to test markdown. On the live
sync PR in this repo today, that is the whole matrix — and in sdk-go and
sdk-csharp it spends live API quota.

The review bot already ignores these PRs: marketdata-docs-sync[bot] is in its
default ignore_authors, and docs/** is in ignore_paths. Tests were the half
with no exemption.

The constraint, and the trap

A required status check can never be conditional — the list is static. And a
matrix job that GitHub skips does not expand its name
: it reports one check with
the template unevaluated and zero per-combination contexts, so the pull request
waits forever for checks that never arrive. Measured on
MarketDataApp/sdk-php#95, where a docs-only PR reported
P${{ matrix.php }} - ${{ matrix.stability }} - ${{ matrix.os }} and no P8.x.

How

  1. changes — always runs, diffs against the merge base, sets code=true or
    code=false.
  2. The suite gates on it. A job that already declares needs: inherits the skip
    through its dependency instead, because a second needs: key is silently
    discarded by YAML.
  3. Tests passed — a new aggregate with if: always(). This is the context
    branch protection should require.

The path list is inverted on purpose

It lists what documentation looks like (^docs/, \.md$) and runs the suite
for everything else. Enumerating code paths instead would skip the suite the first
time somebody adds a source directory nobody remembered to list.

Two classes of skip, which is the part that matters

Tests passed does not treat all skips alike:

  • strict jobs apply to every change, so the only acceptable skip is the one
    this gate asked for. A failed changes job leaves CODE empty, which reads as
    a skip nobody asked for, and fails. A green over untested code is the failure
    mode being designed out.
  • lenient jobs carry conditions that predate this gate — the live suite does
    not run on an ordinary push, nor for dependabot, which cannot read the token.
    Branch protection already treats those skips as passing, so this does too.

A failure never passes in either class.

Each gate reads != 'false', not == 'true', so a broken gate runs the suite.

⚠️ Do not merge before the protection change

This commit only adds a check. Branch protection on main must move to
Tests passed, or a docs-only pull request blocks. Merged with it, the existing
contexts still run and still have to pass whenever code changes — Tests passed
fails if any of them fail.

Verification

This branch touches a workflow file, which is not documentation, so its own suite
must run. A docs-only pull request on top must show the suite skipped and
Tests passed green. Both results go in this thread before merge, the way
sdk-php #94 was proven in both directions.

Version impact

none. CI configuration. No source change, no release.

The docs sync bot (marketdata-docs-sync) opens pull requests here that touch
nothing but docs/. Every one of them runs the full suite to test markdown.

A required status check can never be conditional: the list is static. And a
matrix job that GitHub SKIPS does not expand its name, so it reports no
per-combination context at all and the pull request waits forever for checks that
never arrive. Measured on MarketDataApp/sdk-php#95.

So `Tests passed` becomes the single context branch protection requires. A
`changes` job always runs, diffs against the merge base, and the suite gates on
it.

The path list is inverted on purpose: it says what DOCUMENTATION looks like and
runs the suite for everything else. Enumerating code paths instead would skip the
suite the first time somebody adds a source directory nobody listed.

`Tests passed` separates two classes of skip. A job that applies to every change
may only skip when the gate asked for it, so a failed gate cannot produce a green
over untested code. A job with a condition of its own -- the live suite does not
run on an ordinary push, nor for dependabot, which cannot read the token -- may
skip for its own reasons, which branch protection already tolerates. A FAILURE
never passes in either class.
@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1ac2d64) to head (87c18cd).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #107   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           69        69           
  Lines         2187      2187           
  Branches       272       272           
=========================================
  Hits          2187      2187           

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ac2d64...87c18cd. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MarketDataApp
MarketDataApp merged commit 43eb5fc into main Sep 17, 2026
10 checks passed
@MarketDataApp
MarketDataApp deleted the skip-tests-on-docs-only branch September 17, 2026 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant