Skip to content

ci: run the integration tests on every pull request - #90

Merged
MarketDataDev03 merged 5 commits into
mainfrom
ci/integration-tests-on-pull-requests
Oct 6, 2026
Merged

MarketDataDev03 merged 5 commits into
mainfrom
ci/integration-tests-on-pull-requests

Conversation

@MarketDataDev03

Copy link
Copy Markdown
Collaborator

Run the integration tests on every pull request

The integration tests (src/integrationTest/**, against the live API) ran on every push to
main and when a maintainer commented /integrationtest. They did not run on pull requests.

The other SDKs run their live suite on every pull request (sdk-py, sdk-csharp, sdk-go, sdk-php),
as the SDK requirements ask (§13). Here, a pull request that added integration tests stayed
blocked by the code review until somebody ran them by hand: #89 was left with
"Proof: Run /integrationtest with an API token and provide the output".

What changes

  1. A new integration job in pull-request.yml (Integration tests (live API, JDK 17)).

    • Same steps, variables and command as the job in main.yml, on JDK 17 only. The full
      {17, 21, 25} matrix stays in main.yml and behind /integrationtestfull.
    • It is part of the Tests workflow on purpose: the code review starts when Tests
      completes, so it finds the live run already finished.
    • It runs when the pull request changes code. A documentation-only pull request spends no
      API quota.
    • It does not run for pull requests from dependabot or from forks, which cannot read the
      API token. There it shows as "skipped", not as a green run that tested nothing.
    • When it runs and MARKETDATA_TOKEN is empty, it fails (§17.3).
    • It runs in parallel with verify (needs: changes only), so the wait is the longer of
      the two jobs and not their sum.
    • timeout-minutes: 15.
  2. Tests passed counts the new job.

    • success passes.
    • skipped passes only for the three reasons above: documentation only, dependabot, or a
      fork.
    • Anything else fails, including the job refusing to run without its token.
  3. The on-demand workflow stays (/integrationtest, /integrationtestfull), now as a
    complement:

    • the full JDK matrix;
    • pull requests from forks and from dependabot, where a maintainer decides whether to run
      their code against the live API;
    • running the suite again without a new push.

    Its check, Integration tests pass, only exists after somebody comments a command. Branch
    protection must not require it; the comments in the workflow now say so.

  4. Documentation: README.md, CLAUDE.md and the comments in the three workflows describe
    the new behaviour.

main.yml still runs the full matrix on every push to main. The build, the tests and the SDK
code are untouched, and so is CHANGELOG.md: this is a CI change.

Security

The repository is public. The API token is never within reach of code from a fork: the trigger is
pull_request (not pull_request_target), which does not pass secrets to forks, and the job is
skipped for them as well.

Proof

This pull request changes a workflow, so it counts as code and the new job runs on it: see
Integration tests (live API, JDK 17) in this pull request's checks, and Tests passed, which
now reads its result.

Cost

Each pull request that changes code makes the suite's live API calls once per push, on one JDK.
Before, that happened once per merge to main, on three.

The live suite only ran on pushes to main and when a maintainer commented
/integrationtest. Add an `integration` job to the Tests workflow so it runs
on JDK 17 for every pull request that changes code, as the other SDKs do
(SDK requirements 13).

It skips, as a job, for documentation-only pull requests and for pull
requests from dependabot or a fork, which cannot read MARKETDATA_TOKEN.
When it runs and the secret is empty it fails (SDK requirements 17.3).
Tests passed is the check branch protection requires. It now needs the
`integration` job too: success passes, and a skip passes only when the
documentation gate asked for it or the pull request is from dependabot or a
fork. Any other result fails.
Pull requests now run the live suite themselves, so the comments that said
they do not were wrong. The on-demand workflow stays for the full JDK
matrix, for fork and dependabot pull requests, and for re-runs without a
push. Its "Integration tests pass" check only exists after a command is
commented, so it must not be a required check.

Comments only; no behaviour changes.
@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.27%. Comparing base (d744f98) to head (f72fd04).

Additional details and impacted files
@@            Coverage Diff            @@
##               main      #90   +/-   ##
=========================================
  Coverage     98.27%   98.27%           
  Complexity     1043     1043           
=========================================
  Files           123      123           
  Lines          2606     2606           
  Branches        301      301           
=========================================
  Hits           2561     2561           
  Partials         45       45           

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d744f98...f72fd04. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@marketdata-code-review

marketdata-code-review Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

✅ Ready — Code review

This adds an automatic live integration-test job for eligible pull requests and makes the existing aggregate check enforce its result while retaining documented skip cases. Version impact: none; only CI configuration and documentation change, so no release or changelog entry is needed.

Rating patch 5/6 (solid) · proof 5/6 (reproducible) · overall 5/6

Since last review

  • Resolved: a9b712d5
  • 0 still open, 0 new

Reviewed f72fd04 · revision 2 · codex (gpt-5.6-sol) · claude unavailable · rules: default, sdk

`github.actor` is whoever triggered the run. A maintainer who reopens a
dependabot pull request becomes the actor, so the `integration` job ran as
on an ordinary pull request: with the real token, against a freshly bumped
dependency, without anybody having decided to.

Test `github.event.pull_request.user.login` instead, in the job's `if:` and
in the negated expression `Tests passed` uses, so a dependabot pull request
never runs the live suite on its own. A maintainer runs it with
/integrationtest.
@MarketDataDev03

Copy link
Copy Markdown
Collaborator Author

@marketdata-code-review waive a9b712d5

Why this finding is waived

The finding is accurate about the job: it runs the pull request's code with MARKETDATA_TOKEN
available. We accept that, for these reasons:

  • Who can reach the token does not change. Only people with write access can push a branch
    to this repository. Pull requests from forks and from dependabot do not receive the token, and
    the job is skipped for them.
  • The same people could already do this. /integrationtest runs the pull request's code
    with this token and asks only for write access. The verify job also runs pull request code
    with a secret (CODECOV_TOKEN).
  • It is how the other SDKs work. sdk-py, sdk-csharp, sdk-go and sdk-php run their live suite
    on every pull request with the token, as the SDK requirements ask (§13).

What does change, and we accept: code written by the SDK sync pipeline now runs with the
token without a person triggering it first. Those pull requests come from a branch of this
repository, pass the pipeline's internal review before they are opened, and are reviewed by a
person before merge.

What we did not choose: a protected environment with required approval would close the
finding, but it brings back a manual step on every pull request, which is what this change
removes. A separate least-privileged token for pull requests remains an option if the scope of
MARKETDATA_TOKEN is ever a concern.

@MarketDataDev03
MarketDataDev03 merged commit bf30c72 into main Oct 6, 2026
11 checks passed
@MarketDataDev03
MarketDataDev03 deleted the ci/integration-tests-on-pull-requests branch October 6, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant