Repository navigation
slayer-evals: capability benchmark — do agents reach for SLayer's DSL? - #1
Merged
ZmeiGorynych merged 11 commits intoOct 6, 2026
Conversation
Project scaffold (pyproject, lock), test helpers, fake SDK client and fake agents, fixtures captured from slayer mcp 1.0.2, synthetic run directories for the report, and tests covering every scenario of the six delta specs.
…port and CLI Runner may import report (design.md node table and the import-law test updated, approved). Test-only fixes: typing narrowings, split composite asserts, atomic attempt counter in the fake agent.
…pilot
Columns match by values when names differ (mapping recorded on the verdict), ISO months normalize,
the CLI's {"result": ...} envelope and SLayer's SQL preamble are unwrapped, the system prompt asks
for JSON results, SLayer launches under env -i, and transcripts end with a session line.
Unordered truth rows are sorted and snapshots compare floats within 1e-9 (parallel sums differ in the last bits).
…lity predicates and hack rules Kind lists for refusal tasks, the system prompt names the submit tool exactly.
…nds saved measures)
…xit, path-safe names, strict selectors, clean store rebuild
…tive failure breakdown
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A benchmark that measures whether an agent talking to SLayer through its MCP server answers analytics questions with one well-formed SLayer query, or pulls rows and does the work by hand. Misses point at the MCP surface (tool descriptions, help, errors) to fix in
slayer.What's in it
queryreturned the truth itself, with no combining and no post-processing. Columns match by name, then by values, so agents can name them freely. Five informational flags (used Python, raw SQL, model edits, SLayer errors, several queries) are counted but don't affect passing.env -i. Two profiles:slayerandslayer+python(Python in a sandboxed subprocess). Any agent can be plugged in through one adapter class.slayer-evals build | truth | run | report. Each trial runs in its own process on its own copy of the database and store. Runs supportrepeat/until-passmodes, explicit--subscription-auth/--api-key-auth, and concurrency. A run directory holds the results, normalized traces, the full session transcripts and a markdown report.Baseline (Opus 5.5, one trial per task and profile)
slayerslayer+pythonThe answers are almost always right. Most misses involve rolling windows (Q11), ordering by a measure that isn't shown (Q13), deep composition (Q7) and fan-out-safe cross-model aggregates (Q6). See
results/baseline/report.md.Building the tasks turned up two SLayer bugs: DEV-2062 and DEV-2063.
Closes DEV-2055