Repository navigation
ci: pass /evals CLI arguments through to Cloud MCP evals - #1401
Caleb Evans (calebevans-ab) merged 2 commits into
Conversation
Co-Authored-By: alexandre@airbyte.io <alexandre@airbyte.io>
Co-Authored-By: alexandre@airbyte.io <alexandre@airbyte.io>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
Note 📝 PR Converted to Draft More info...Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy. As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page. To skip draft status in future PRs, please include |
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1792193149-evals-run-args' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1792193149-evals-run-args'PR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful ResourcesCommunity SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
Code coverage overviewLanguages: Python Python / code-coverage/pytest-no-creds
View coverage of the 10 most impacted files.
Python / code-coverage/pytest
View coverage of the 10 most impacted files.
Python / code-coverage/pytest-unit
Line coverage data for the release-candidate/v1 branch is not yet available. View coverage of the 10 most covered files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
30775e8
into
release-candidate/v1
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a2df0b142
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
|
||
| first_line = os.environ["COMMENT_BODY"].split("\n", 1)[0].rstrip("\r") | ||
| first_line = re.sub(r"^\s*/evals(?:\s+|$)", "", first_line, count=1) | ||
| first_line = re.sub(r"(?<!\S)(?:full|ref)=[^\s]+", "", first_line) |
There was a problem hiding this comment.
With the checked dispatch-type: workflow setup in .github/workflows/slash_command_dispatch.yml, a token such as codex-sdk=gpt-6.1-sol or --option=value is parsed as a named workflow input before this step can read or sanitize the comment. Because evals-command.yml does not declare arbitrary input names, GitHub rejects the dispatch and the eval workflow never starts. The action's workflow-dispatch documentation confirms that unexpected named inputs produce validation errors. This breaks the advertised pass-through for common CLI syntax, including the harness form described by this change; the arguments need to be encoded or captured before slash-command-dispatch validates named inputs, or dispatched through a mechanism that preserves the raw command.
Useful? React with 👍 / 👎.
2 participants
Summary
Text after
/evalson the comment's first line is now forwarded to the cloud-mcp-evals workflow as its newrun_argsinput (https://github.com/airbytehq/cloud-mcp-evals/pull/62). With that, anypython -m cloud_mcp_evals runargument works from the PR:Why
evals-command.ymlreads the comment itself: withdispatch-type: workflow,peter-evans/slash-command-dispatchforwards onlykey=valuenamed args and silently drops the rest. So the new "Read run arguments" step fetches the comment bycomment-id, keeps its first line, strips/evalsand the dispatcher's ownfull=…/ref=…tokens, and passes the remainder through. Splitting and validation happen in cloud-mcp-evals (shlex, never evaluated by a shell).full=truebehaves exactly as before. Without it, the changed-tools filter still applies, now to whatever scenarios were given.run_argsis only sent when non-empty, so plain/evalskeeps working before or after cloud-mcp-evals#62 merges. Using args needs #62 merged first, because GitHub rejects undeclared dispatch inputs.The queued comment shows the args, e.g. "(changed tools only,
--repeats 5)".Verification:
actionlint, plus local checks of the parsing snippet on/evals,/evals full=true,/evals full=true --repeats 5 scenarios/smoke,/evals --model 'a b' --issue PLAT-1, leading whitespace with a trailing CR, andref=removal. The end-to-end slash command can only be exercised after merge, sinceslash_command_dispatchtargets the default-branch workflow.Link to Devin session: https://app.devin.ai/sessions/d7c7d8e7ea9a4d49a9180b295cdcf06b
Open in Devin Desktop: https://app.devin.ai/desktop/session/d7c7d8e7ea9a4d49a9180b295cdcf06b?variant=devin
Requested by: Alexandre Girard (@girarda)