ci: make `/evals` compare a PR with its base by default by calebevans-ab · Pull Request #1402 · airbytehq/PyAirbyte

Caleb Evans (calebevans-ab) · 2026-10-17T00:24:04Z

Summary

TL;DR: Plain /evals now compares the PR with its base branch, running only the scenarios for the MCP tools the PR changed, and posts a before/after table as the PR comment. /evals full=true compares on the full suite. /evals compare=<target> compares with a release, commit, branch or another PR instead (e.g. compare=main, compare=1.0.0-rc.33, compare=#1399). /evals compare=false runs only the PR and posts a full results comment (headline, what ran, per-harness counts, one row per failing trial) instead of one line.

evals-command.yml gets a compare input that defaults to true and accepts true, false or a target matching ^#?[A-Za-z0-9][A-Za-z0-9_./-]*$. Anything else fails. A target is dispatched as -f compare=true -f compare_to=<target> to cloud-mcp-evals evals.yml (airbytehq/cloud-mcp-evals#65, which needs to merge first).
In compare mode, the report step downloads the comparison-<run_id> artifact and posts that Markdown via body-path, plus a link to the full run.
With compare=false, the report step downloads the comment-<run_id> artifact (rendered by cloud-mcp-evals Feat: Add BigQuery cache support; Fix: IPython rendering bug when using a TTY #65) and posts it as-is. If the artifact isn't there, it falls back to the old one-line summary.
full=, ref= and compare= are removed from the free-form arguments. The rest still goes to run_args.
Also carries ci: pass /evals CLI arguments through to Cloud MCP evals #1401's run_args passthrough onto main. ci: pass /evals CLI arguments through to Cloud MCP evals #1401 merged into release-candidate/v1, but slash commands run from main, so neither change works until it's here.
PR welcome texts describe the compare default, full=true, compare=<target> and compare=false, and say to write options as --repeats 3, not --repeats=3 (the dispatcher treats key=value tokens as workflow inputs).

Verification: actionlint passes. The mode-parsing step was checked with true, false, main, #1399 and 1.0.0-rc.33, with full=true and without it, and bad;rm is rejected. The compare and non-compare comment bodies were checked against real comparison output. The non-compare comment path was simulated locally with a stubbed gh: with the comment-<run_id> artifact it posts comment.md unchanged; without it, it falls back to the one-line summary built from run 38980034242's evidence. No real /evals comment has run yet, because the slash command dispatches cloud-mcp-evals from main, where #65 isn't merged.

Link to Devin session: https://app.devin.ai/sessions/db7acb18b78e45c48bbd1d265dc3e254
Requested by: Caleb Evans (@calebevans-ab)

(cherry picked from commit 30775e8)

devin-ai-integration · 2026-10-17T00:24:08Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

octavia-bot · 2026-10-17T00:24:14Z

Note

📝 PR Converted to Draft

More info...

Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy.

As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page.

To skip draft status in future PRs, please include [ready] in your PR title or add the skip-draft-status label when creating your PR.

chatgpt-codex-connector · 2026-10-17T00:24:20Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-18T05:40:03.423646Z	`cc84ee9`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review" or "Codex (@codex) security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-10-17T00:24:31Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This PyAirbyte Version

You can test this version of PyAirbyte using the following:

# Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1792195132-evals-compare' pyairbyte --help

# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1792195132-evals-compare'

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/fix-pr - Fixes most formatting and linting issues
/uv-lock - Updates uv.lock file
/test-pr - Runs tests with the updated PyAirbyte
/prerelease - Builds and publishes a prerelease version to PyPI
/evals - Runs the Cloud MCP evals that use the MCP tools this PR changed (/evals full=true runs the full suite)

📚 Show Repo Guidance

Helpful Resources

Community Support

Questions? Join the #pyairbyte channel in our Slack workspace.

📝 Edit this welcome message.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fb9dd6e42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

github-code-quality · 2026-10-17T00:28:49Z

Code coverage overview

Languages: Python

Python / code-coverage/pytest-no-creds

main d9f652f	devin/1792195132-eva... cc84ee9	+/-
65%	77%	+12%

View coverage of the 10 most impacted files.

File	main d9f652f	devin/1792195132-eva... cc84ee9	+/-
`airbyte/mcp/cloud.py`	52%	71%	+19%
`airbyte/_util/api_util.py`	36%	56%	+20%
`airbyte/mcp/_otel.py`	0%	89%	+89%
`airbyte/mcp/int..._registry_ui.py`	0%	92%	+92%
`airbyte/cloud/models.py`	0%	95%	+95%
`airbyte/mcp/_sso_auth.py`	0%	96%	+96%
`airbyte/mcp/int...nc_status_ui.py`	0%	97%	+97%
`airbyte/_direct...nnector_docs.py`	0%	97%	+97%
`airbyte/mcp/agents.py`	0%	98%	+98%
`airbyte/_direct...ctors/models.py`	0%	99%	+99%

Python / code-coverage/pytest

main d9f652f	devin/1792195132-eva... cc84ee9	+/-
71%	80%	+9%

View coverage of the 10 most impacted files.

File	main d9f652f	devin/1792195132-eva... cc84ee9	+/-
`airbyte/mcp/cloud.py`	52%	71%	+19%
`airbyte/mcp/_otel.py`	0%	89%	+89%
`airbyte/mcp/int..._registry_ui.py`	0%	92%	+92%
`airbyte/_direct...ors/api_util.py`	0%	92%	+92%
`airbyte/mcp/_sso_auth.py`	0%	96%	+96%
`airbyte/cloud/models.py`	0%	96%	+96%
`airbyte/mcp/int...nc_status_ui.py`	0%	97%	+97%
`airbyte/_direct...nnector_docs.py`	0%	97%	+97%
`airbyte/mcp/agents.py`	0%	98%	+98%
`airbyte/_direct...ctors/models.py`	0%	99%	+99%

Python / code-coverage/pytest-unit

main	devin/1792195132-eva... cc84ee9	+/-
—	73%	—

Line coverage data for the main branch is not yet available.

View coverage of the 10 most covered files.

File	main	devin/1792195132-eva... cc84ee9	+/-
`airbyte/mcp/agents.py`	—	98%	—
`airbyte/mcp/_sso_auth.py`	—	96%	—
`airbyte/cloud/models.py`	—	95%	—
`airbyte/cloud/client.py`	—	91%	—
`airbyte/mcp/_otel.py`	—	89%	—
`airbyte/exceptions.py`	—	84%	—
`airbyte/progress.py`	—	80%	—
`airbyte/mcp/cloud.py`	—	71%	—
`airbyte/shared/...ql_processor.py`	—	68%	—
`airbyte/_util/api_util.py`	—	56%	—

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

_{Updated October 18, 2026 04:32 UTC}

Caleb Evans (calebevans-ab) · 2026-10-17T19:02:07Z

Codex (@codex) review

chatgpt-codex-connector · 2026-10-17T19:04:33Z

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: e08ba7d1ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Caleb Evans (calebevans-ab) · 2026-10-18T05:37:40Z

Codex (@codex) review

chatgpt-codex-connector · 2026-10-18T05:40:01Z

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: cc84ee9b20

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Alexandre Girard (girarda) and others added 3 commits October 17, 2026 00:02

ci: pass /evals CLI arguments through to Cloud MCP evals (#1401)

fbc9419

(cherry picked from commit 30775e8)

feat(ci): add eval comparison mode

9bf7958

ci: skip the evidence download in compare mode

2fb9dd6

devin-ai-integration Bot assigned Caleb Evans (calebevans-ab) Oct 17, 2026

octavia-bot Bot marked this pull request as draft October 17, 2026 00:24

chatgpt-codex-connector Bot reviewed Oct 17, 2026

View reviewed changes

Comment thread .github/workflows/evals-command.yml Outdated

Comment thread .github/pr-welcome-community.md Outdated

devin-ai-integration Bot added 2 commits October 17, 2026 06:44

docs(ci): clarify eval repeat argument syntax

5d37ac9

feat(ci): compare evals against arbitrary targets

e08ba7d

devin-ai-integration Bot changed the title ~~ci: add /evals compare=true to compare a PR's evals with its base~~ ci: add /evals compare=... to compare a PR's evals with its base or any target Oct 17, 2026

Caleb Evans (calebevans-ab) marked this pull request as ready for review October 17, 2026 19:05

devin-ai-integration Bot added 2 commits October 17, 2026 21:37

feat(ci): paste the evals results comment in /evals

b4196ee

feat(ci): compare with the base by default in /evals

cc84ee9

devin-ai-integration Bot changed the title ~~ci: add /evals compare=... to compare a PR's evals with its base or any target~~ ci: make /evals compare a PR with its base by default Oct 18, 2026

Caleb Evans (calebevans-ab) requested a review from Alexandre Girard (girarda) October 18, 2026 06:31

Conversation

Caleb Evans (calebevans-ab) commented Oct 17, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

devin-ai-integration Bot commented Oct 17, 2026

Uh oh!

octavia-bot Bot commented Oct 17, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Oct 17, 2026

👋 Greetings, Airbyte Team Member!

Testing This PyAirbyte Version

PR Slash Commands

Helpful Resources

Community Support

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

github-code-quality Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code coverage overview

Python / code-coverage/pytest-no-creds

Python / code-coverage/pytest

Python / code-coverage/pytest-unit

Uh oh!

Caleb Evans (calebevans-ab) commented Oct 17, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 17, 2026

Uh oh!

Caleb Evans (calebevans-ab) commented Oct 18, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 18, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Caleb Evans (calebevans-ab) commented Oct 17, 2026 •

edited by devin-ai-integration Bot

Loading

chatgpt-codex-connector Bot commented Oct 17, 2026 •

edited

Loading

github-code-quality Bot commented Oct 17, 2026 •

edited

Loading