Skip to content

ci: pass /evals CLI arguments through to Cloud MCP evals #1401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Caleb Evans (calebevans-ab) merged 2 commits into from
Oct 17, 2026
Merged

ci: pass /evals CLI arguments through to Cloud MCP evals #1401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fb0afd4
feat(pyairbyte): pass eval run args to workflow
devin-ai-integration[bot] Oct 16, 2026
7a2df0b
ci: only send run_args when the comment has some
devin-ai-integration[bot] Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/pr-welcome-community.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ As needed or by request, Airbyte Maintainers can execute the following slash com
- `/uv-lock` - Updates uv.lock file
- `/test-pr` - Runs tests with the updated PyAirbyte
- `/prerelease` - Builds and publishes a prerelease version to PyPI
- `/evals` - Runs the Cloud MCP evals that use the MCP tools this PR changed (`/evals full=true` runs the full suite)
- `/evals` - Runs the Cloud MCP evals that use the MCP tools this PR changed (`/evals full=true` runs the full suite). Add any `cloud_mcp_evals run` arguments after it, e.g. `/evals full=true --repeats 3 scenarios/regressions`.

</details>

Expand Down
2 changes: 1 addition & 1 deletion .github/pr-welcome-internal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Airbyte Maintainers can execute the following slash commands on your PR:
- `/uv-lock` - Updates uv.lock file
- `/test-pr` - Runs tests with the updated PyAirbyte
- `/prerelease` - Builds and publishes a prerelease version to PyPI
- `/evals` - Runs the Cloud MCP evals that use the MCP tools this PR changed (`/evals full=true` runs the full suite)
- `/evals` - Runs the Cloud MCP evals that use the MCP tools this PR changed (`/evals full=true` runs the full suite). Add any `cloud_mcp_evals run` arguments after it, e.g. `/evals full=true --repeats 3 scenarios/regressions`.

</details>

Expand Down
40 changes: 37 additions & 3 deletions .github/workflows/evals-command.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,37 @@ jobs:
fi
echo "short-sha=${SHA:0:7}" >> "$GITHUB_OUTPUT"

- name: Read run arguments
id: args
if: inputs.comment-id != ''
env:
COMMENT_ID: ${{ inputs.comment-id }}
GH_TOKEN: ${{ github.token }}
BASE_LABEL: ${{ steps.mode.outputs.label }}
run: |
set -euo pipefail
COMMENT_BODY="$(gh api "repos/$GITHUB_REPOSITORY/issues/comments/$COMMENT_ID" --jq .body)"
export COMMENT_BODY
RUN_ARGS="$(python3 - <<'PY'
import os
import re

first_line = os.environ["COMMENT_BODY"].split("\n", 1)[0].rstrip("\r")
first_line = re.sub(r"^\s*/evals(?:\s+|$)", "", first_line, count=1)
first_line = re.sub(r"(?<!\S)(?:full|ref)=[^\s]+", "", first_line)

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Handle equals-bearing CLI tokens before workflow dispatch

With the checked dispatch-type: workflow setup in .github/workflows/slash_command_dispatch.yml, a token such as codex-sdk=gpt-6.1-sol or --option=value is parsed as a named workflow input before this step can read or sanitize the comment. Because evals-command.yml does not declare arbitrary input names, GitHub rejects the dispatch and the eval workflow never starts. The action's workflow-dispatch documentation confirms that unexpected named inputs produce validation errors. This breaks the advertised pass-through for common CLI syntax, including the harness form described by this change; the arguments need to be encoded or captured before slash-command-dispatch validates named inputs, or dispatched through a mechanism that preserves the raw command.

Useful? React with 👍 / 👎.

print(first_line.strip())
PY
)"
delimiter="$(python3 -c 'import secrets; print(secrets.token_hex(16))')"
{
printf 'run-args<<%s\n' "$delimiter"
printf '%s\n' "$RUN_ARGS"
printf '%s\n' "$delimiter"
} >> "$GITHUB_OUTPUT"
label="$BASE_LABEL"
[[ -n "$RUN_ARGS" ]] && printf -v label "%s, \`%s\`" "$BASE_LABEL" "$RUN_ARGS"
printf 'label=%s\n' "$label" | tee -a "$GITHUB_OUTPUT"

- name: Append comment with job run link
id: first-comment-action
uses: peter-evans/create-or-update-comment@e8674b075228eee787fea43ef493e45ece1004c9 # v5.0.0
Expand All @@ -67,7 +98,7 @@ jobs:
issue-number: ${{ inputs.pr }}
body: |

> Cloud MCP evals queued for `${{ steps.mode.outputs.short-sha }}` (${{ steps.mode.outputs.label }}). [Check job output.](${{ steps.vars.outputs.run-url }})
> Cloud MCP evals queued for `${{ steps.mode.outputs.short-sha }}` (${{ steps.args.outputs.label || steps.mode.outputs.label }}). [Check job output.](${{ steps.vars.outputs.run-url }})

# The token lasts one hour; a full evals run takes about 20 minutes.
- name: Authenticate as GitHub App
Expand All @@ -85,11 +116,14 @@ jobs:
env:
GH_TOKEN: ${{ steps.get-app-token.outputs.token }}
CHANGED_ONLY: ${{ steps.mode.outputs.changed-only }}
RUN_ARGS: ${{ steps.args.outputs.run-args }}
run: |
set -euo pipefail
since=$(date -u +%Y-%m-%dT%H:%M:%SZ)
extra=()
[[ -n "$RUN_ARGS" ]] && extra=(-f run_args="$RUN_ARGS")
gh workflow run evals.yml -R "$EVALS_REPO" --ref main \
-f pyairbyte="#$PR" -f changed_tools_only="$CHANGED_ONLY"
-f pyairbyte="#$PR" -f changed_tools_only="$CHANGED_ONLY" "${extra[@]}"
title="PyAirbyte evals: #$PR"
for _ in $(seq 18); do
sleep 10
Expand Down Expand Up @@ -119,7 +153,7 @@ jobs:
GH_TOKEN: ${{ steps.get-app-token.outputs.token }}
RUN_ID: ${{ steps.dispatch.outputs.run-id }}
RUN_URL: ${{ steps.dispatch.outputs.run-url }}
LABEL: ${{ steps.mode.outputs.label }}
LABEL: ${{ steps.args.outputs.label || steps.mode.outputs.label }}
CHANGED_ONLY: ${{ steps.mode.outputs.changed-only }}
run: |
set -euo pipefail
Expand Down
Loading