[ZmeiGorynych]

Turns the benchmark from "do agents reach for SLayer's DSL?" into a three-way comparison with an agent that queries the database directly, and adds the tasks that make that comparison interesting: questions combining several capabilities, and traps where the straightforward SQL silently returns wrong numbers.

Linear: DEV-2064

What changes

A third flavor, sql+python. No SLayer at all. The agent gets a sql tool, plus submit_answer and the same Python sandbox (which is not given the database). The sql tool:

• runs exactly one DuckDB statement per call on a fresh read-only connection to the trial's database copy;
• has external access disabled and the configuration locked, so ATTACH, COPY, INSTALL/LOAD, file reads and re-enabling access all fail;
• has a 60 s timeout that interrupts the statement before the next call can start;
• returns {columns, rows, truncated}, capped at 1,000 rows.

Runs now default to all three flavors.

$ slayer-evals run --subscription-auth --env-file .env.agents --tasks t-chasm
→ one trial each in slayer, slayer+python and sql+python

Grading across flavors.

• single_query now counts any successful SLayer query or sql call whose own result is the answer.
• Refusal tasks in sql+python pass on an empty answer whose message names the problem.
• Tasks that rely on a saved SLayer definition (uses_saved, e.g. monthly_rev) auto-fail in sql+python without starting an agent (end reason auto_fail).
• New compare option null_as_zero.
• The flag slayer_errors is now query_errors.

The report leads with a suite × flavor table and says plainly that correct is the cross-flavor comparison, since one SQL statement can express nearly anything. A pitfall × flavor table follows, then the per-row tables, where a multi-row task counts under every row it covers.

Task format.

• row becomes covers: rows plus pitfall kinds, from which the suite is derived (capability, combo, trap); files live under tasks/<suite>/.
• Every task now carries its slayer_query, and traps carry the naive_sql that falls into them.
• tests/test_task_proofs.py runs every reference query through SLayer and every naive statement on the built database. It fails when a reference query doesn't answer its task, when a trap's naive SQL happens to be right, or when a covered row's DSL marker is missing from the reference query (new arc42 principle 7).

Data and tasks.

• Five new tables: products, order_items (lines sum to the order amount to the cent), campaigns, campaign_members (a bridge), and cities (keyed on city and region; Oslo is in two regions).
• The original tables are unchanged, pinned by committed hashes.
• 17 traps covering 15 pitfalls: fan-out, chasm, bridge, non-unique key, outer-join filter, NOT IN with NULL, filtered totals, distinct re-aggregation, missing months, filtering before a window, row windows over gaps, timestamp bounds, average of averages, and more.
• 11 combo tasks, three of them covering three rows.
• Prompt deny-list entries now match whole words and include pitfall hints (join, null, not in, …); q3, q6, q10, q12 and q25 were reworded.

System prompt. It now asks every agent to "submit exact values without rounding, and use null as the label of a group whose key is unknown". Without it the strict grader marked the raw-SQL agent wrong on answers that only differed in rounding or a made-up "Unknown" label.

Baseline (Opus 5.5, one trial per task × flavor)

Suite	slayer correct / passed	slayer+python	sql+python
capability (24)	23 / 19	23 / 16	20 / 17 (2 auto-failed)
combo (11)	11 / 4	11 / 4	10 / 6 (1 auto-failed)
trap (17)	17 / 10	16 / 8	16 / 15

The raw-SQL agent fell into none of the traps. On this set SLayer's lead is not correctness; its agents still often assemble a correct answer from several queries. docs/baseline-failures.md breaks it all down.

The sql+python trials were rerun after the system-prompt sentence was added; the SLayer flavors ran just before it. q12-net-revenue was rerun in all flavors after its prompt fix.

Notes for review

• SLayer is pinned to git commit 1653a91 (main, the DEV-2062 fix for a saved ratio measure plus a transform over it), which combo/c-aov-change needs. Switch to the release once it's out. That version requires direction= on rank, so the reference queries use it.
• TrialResult and task files changed shape (row → covers), so old result files no longer load. The baseline and test fixtures are regenerated.
• The OpenSpec change is in openspec/changes/dev-2064-…/.