feat: add scenario checks and command-based CLI - #247

Merged

joshblack merged 69 commits into

mainfrom

feat/add-support-for-checks

Sep 16, 2026

Commits on Sep 10, 2026

feat: add --scenario flag
joshblack
committed
a9bc4e5
feat: add scaffold for yargs-based cli
joshblack
committed
3887d23

Commits on Sep 11, 2026

feat: update to new cli structure, remove shared environment context from signatures
joshblack
committed
63fa673

Commits on Sep 14, 2026

refactor: update how work flows through entities
joshblack
committed
859e75f
refactor: use save stage in run trial
joshblack
committed
fedf794
feat: update output type
joshblack
committed
9844c9d
feat: add debug logs to plan
joshblack
committed
d95dd47

Commits on Sep 15, 2026

feat: add benchmark plan command
joshblack
committed
e9289e7
feat: add plan create and plan run options for benchmarks
joshblack
committed
e06b175
feat: add support for merging shard results
joshblack
committed
246ec45
refactor: clean up files and exports
joshblack
committed
3863d31
refactor: update benchmark entrypoint and rename to full-path names
joshblack
committed
58d23a2
refactor: move experiment into new cli and folder setup
joshblack
committed
e17c01b
refactor: move scenario into folder
joshblack
committed
1ebe118
refactor: move report into model-specific behavior while retaining shape
joshblack
committed
0681a38
chore: file formatting
joshblack
committed
e523ac7
test: remove outdated test
joshblack
committed
6127b05
feat: add initial check config and scenario
joshblack
committed
3525564
refactor: move testPath setup in favor of checks
joshblack
committed
3ac88e7
feat: add support for checks in scenarios and for running scenarios in cli
joshblack
committed
55b58df
test: add support for eslint checks to 000-checks
joshblack
committed
455401d
refactor: use output file instead of stdout for eslint
joshblack
committed
86064f5
refactor: infer result type and transform it in schema
joshblack
committed
9aac634
refactor: use type from vitest instead of local one
joshblack
committed
addcd7b
refactor: clean up eslint errors
joshblack
committed
2660387
refactor: normalize checks config to return arrays
joshblack
committed
80e8f5a
refactor: normalie check run output and make sure browser example works
joshblack
committed
29ddb44
chore: fix eslint error
joshblack
committed
3d595ea
feat: add support for chencks in benchmark and experiment comparisons
joshblack
committed
85ba1a2
refactor: move to checks-based approach
joshblack
committed
122ec91
feat: add support for copilot and container concurrency options
joshblack
committed
9d567dd
refactor: update report output in cli
joshblack
committed
8875c6c
refactor(website): update website with new agent-eval APIs
joshblack
committed
ef6c1fa
feat: add support for capability filtering to website
joshblack
committed
808be65
chore: clean up log convention
joshblack
committed
25eea91
fix: address ssr hydration mismatch errors
joshblack
committed
d924d00
refactor: update local scripts to new cli format
joshblack
committed
81de8b5
docs: add changesets
joshblack
committed
f0fd188
refactor: offload run assets and load dynamically
joshblack
committed
5e23ecb
refactor: update website for LCP
joshblack
committed
04cf614
refactor: add description to cli
joshblack
committed
aac1287
docs: update readme and add docs
joshblack
committed
c45cdef
docs: update readme
joshblack
committed
75a7605
test: remove outdated test
joshblack
committed
f2b9699
docs: update readme files
joshblack
committed
de62bce
ci: update workflows to work with new shard format
joshblack
committed
a89d400
Modify readdir to include hidden files in listing

joshblack
and
Copilot
authored
44960dc
Refactor filepath check to use relative path

joshblack
and
Copilot
authored
a8d0c80
Change fs.stat to fs.lstat for filepath

joshblack
and
Copilot
authored
3fbe912
Remove test file existence check

joshblack
and
Copilot
authored
915262a
Ensure output directory is created for benchmark results

joshblack
and
Copilot
authored
feb9604
Change file path mapping to use relative paths

joshblack
and
Copilot
authored
770880a
test: remove obsolete scenario test-file fixture

joshblack
and
Copilot
committed
c03a711
fix: withhold configured check files from implementation agents

joshblack
and
Copilot
committed
719a440
fix: contain benchmark shard artifact paths

joshblack
and
Copilot
committed
2adbb35
fix: contain experiment shard artifact paths

joshblack
and
Copilot
committed
7b5332f
fix: resolve benchmark plans by manifest ID

joshblack
and
Copilot
committed
ee11956
refactor!: remove the public plan package entry point

joshblack
and
Copilot
committed
2f46614
fix: hash treatment and capability names as UTF-8

joshblack
and
Copilot
committed
966aef7
fix: reject duplicate benchmark plan trial IDs

joshblack
and
Copilot
committed
1be491d
fix: contain check and judge reference real paths

joshblack
and
Copilot
committed
373c747
fix: publish scenario-relative reference paths in trial details

joshblack
and
Copilot
committed
b2b4c16

Commits on Sep 16, 2026

fix: contain walkthrough media paths within artifact roots
joshblack
committed
12195a0
fix: preserve benchmark shards until merge succeeds
joshblack
committed
b0e20a9
fix: validate benchmark shard trial IDs
joshblack
committed
bb9ace9
fix: match complete benchmark shard filenames
joshblack
committed
4b1ad45
fix: apply capability setup to benchmark trials
joshblack
committed
0b433ae
fix: validate benchmark plan capability scenario pairs
joshblack
committed
46b04aa
fix: validate model variants in benchmark plans
joshblack
committed
105f86f