Repository navigation
Leaderboard Submission: Opus 5.5 (high) + Jitto Build - #10
Conversation
|
Hi maintainers, this submission is ready for review. All 13 source jobs are public on the Hub under jamlabs. One thing to flag: every leaderboard workflow run on this PR ended in startup_failure before any job started. That covers Static Analysis (pull_request_target) and Leaderboard Recheck, Trajectory Analysis and Apply Disqualifications (issue_comment). The maintainer-triggered Publish base image run on 2026-08-19 did the same. GitHub reports 'a workflow file issue' with no jobs run. That pattern usually points to an org or repo Actions policy, for example third-party actions such as astral-sh/setup-uv or marocchino/sticky-pull-request-comment not being on the allow-list, rather than to this PR. Happy to rerun or adjust anything once CI can start. Thanks! |
sfc-gh-dhuang
left a comment
There was a problem hiding this comment.
Thanks!
We also plan a new dataset revision with several verifier fixes (including the dbt-fix-division-by-zero DuckDB reference you flagged, fixed in #12), so we'll decide how v1.0 results are labelled
Leaderboard submission for Jitto Build (JamLabs), with Claude Opus 5.5 at high effort.
snowflake-labs/data-eng-bench@sha256:de31878d…(v1.0), unmodified. Default timeouts, no agent, verifier or resource overrides.This PR also adds the display-name entries for
jitto-buildandanthropic/claude-opus-5-5.We noticed that the reference solution for
dbt-fix-division-by-zerofails on DuckDB, even under Harbor's oracle agent. Happy to share details in a separate issue.