Skip to content

Leaderboard Submission: Opus 5.5 (high) + Jitto Build - #10

Merged
sfc-gh-dhuang merged 2 commits into
Snowflake-Labs:masterfrom
charliefrantowski:submission/2026-09-29-anthropic-claude-opus-5-5-high-jitto-build
Nov 12, 2026
Merged

sfc-gh-dhuang merged 2 commits into
Snowflake-Labs:masterfrom
charliefrantowski:submission/2026-09-29-anthropic-claude-opus-5-5-high-jitto-build

Conversation

@charliefrantowski

Copy link
Copy Markdown
Contributor

Leaderboard submission for Jitto Build (JamLabs), with Claude Opus 5.5 at high effort.

  • Dataset: snowflake-labs/data-eng-bench@sha256:de31878d… (v1.0), unmodified. Default timeouts, no agent, verifier or resource overrides.
  • Coverage: all 103 tasks × 3 trials = 309 trials, 0 errored.
  • Our local result: 210/309 rewarded (67.96%). CI re-derives the figure from the 13 source jobs.
  • Agent: Claude Code 2.1.280, with WebSearch and WebFetch disabled per the leaderboard rules.
  • Trials: public on the Hub. Local file paths and runtime session identifiers are replaced with placeholders; every step, tool call and output is kept.

This PR also adds the display-name entries for jitto-build and anthropic/claude-opus-5-5.

We noticed that the reference solution for dbt-fix-division-by-zero fails on DuckDB, even under Harbor's oracle agent. Happy to share details in a separate issue.

@snowflake-security-bot snowflake-security-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

@charliefrantowski

charliefrantowski commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Hi maintainers, this submission is ready for review. All 13 source jobs are public on the Hub under jamlabs.

One thing to flag: every leaderboard workflow run on this PR ended in startup_failure before any job started. That covers Static Analysis (pull_request_target) and Leaderboard Recheck, Trajectory Analysis and Apply Disqualifications (issue_comment). The maintainer-triggered Publish base image run on 2026-08-19 did the same. GitHub reports 'a workflow file issue' with no jobs run. That pattern usually points to an org or repo Actions policy, for example third-party actions such as astral-sh/setup-uv or marocchino/sticky-pull-request-comment not being on the allow-list, rather than to this PR. Happy to rerun or adjust anything once CI can start. Thanks!

@sfc-gh-dhuang sfc-gh-dhuang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

We also plan a new dataset revision with several verifier fixes (including the dbt-fix-division-by-zero DuckDB reference you flagged, fixed in #12), so we'll decide how v1.0 results are labelled

@sfc-gh-dhuang
sfc-gh-dhuang merged commit 74d4bcd into Snowflake-Labs:master Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants