Add native Ruby eval support - #139
Open
Doug Guthrie (dpguthrie) wants to merge 2 commits into
Open
Doug Guthrie (dpguthrie) wants to merge 2 commits into
Doug Guthrie (dpguthrie) wants to merge 2 commits into
Conversation
Braintrust eval reportSay Hi Bot Python (abhijeet@braintrustdata.com-1789502231)
|
Braintrust eval reportSay Hi Bot Python (abhijeet@braintrustdata.com-1789502234)
|
Braintrust eval reportConsole logging (HEAD-1789502242)
My Evaluation (HEAD-1789502242)
|
Braintrust eval report
|
Braintrust eval report
|
Doug Guthrie (dpguthrie)
force-pushed
the
feat/ruby-runtime-support
branch
from
September 15, 2026 19:33
07f4d3f to
713583a
Compare
Doug Guthrie (dpguthrie)
force-pushed
the
feat/ruby-runtime-support
branch
from
September 15, 2026 19:52
575abfb to
11c56b9
Compare
Braintrust eval reporteval-action-ruby-ci (uppercase-35016210959-1)
|
Doug Guthrie (dpguthrie)
force-pushed
the
feat/ruby-runtime-support
branch
from
September 15, 2026 19:55
11c56b9 to
cbce1e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ruby evals cannot currently use the action's PR diff report. This adds
runtime: rubywith direct Ruby and Bundler execution, using one explicit entrypoint that can run and report multiple evals.The release includes:
examples/rubyworkflow, deterministic eval, server-backed reporting helper, and helper teststerminate_on_failureinputThe example keeps reporting application-owned for this first release. An optional follow-up in
braintrust-sdk-rubycan own the server-backed summary method and JSONL reporter, centralizing flush/readiness retries, endpoint/auth selection, and serialization so users no longer copy the helper. This action change does not depend on that SDK work.Validation:
pnpm --dir eval run all(format, lint, 16 tests, package)ruby examples/ruby/test/braintrust_report_test.rb(8 tests, 25 assertions)ruby -c examples/ruby/evals/run.rbruby -c examples/ruby/scripts/braintrust_report.rbaction.ymlandexamples/ruby/ruby.ymlgit diff --checkbundle exec ruby evals/run.rbproduced a linkedeval-action-ruby-cireport with a 100%Exact_matchscore and server metricsBefore publishing a release tag, complete the remaining live synthetic acceptance for an explicit comparison baseline, improvement/regression counts, execution failures, and push behavior.
Closes #86