feat: update project to support new experiment and benchmark structure - #176

Merged

joshblack merged 52 commits into

mainfrom

feat/add-benchmark-support

Sep 4, 2026

Commits on Sep 4, 2026

feat: add benchmark folder
joshblack
committed
c4c1036
feat: add benchmark scenarios
joshblack
committed
5de02fb
feat: add run to benchmark
joshblack
committed
ebc72b2
test: add host for local testing
joshblack
committed
742ebe4
refactor: update scenario implementation
joshblack
committed
73c5a55
refactor: update model implementation
joshblack
committed
de0d665
test: add test file for benchmarks
joshblack
committed
797a9c4
feat: add benchmark functionality
joshblack
committed
ab2f28e
feat: add trial concept for individual treatment runs
joshblack
committed
0a0c7fa
feat: add plan concept for ordering trials and storing order across shards
joshblack
committed
f1a7eda
refactor: update benchmarks, scenario, treatment, and run
joshblack
committed
249e755
refactor: add virtual and system sandboxes
joshblack
committed
ff1dc6b
feat: add sandbox interface for testing
joshblack
committed
dc9807c
refactor: remove host arg from runTrial
joshblack
committed
4a6ea53
refactor: mock virtual sandbox instead of adding custom implementation
joshblack
committed
cc2cc26
refactor: change defineScenario to defineConfig
joshblack
committed
e05cb5c
feat: build through experiments run flow
joshblack
committed
4abe367
feat: support running trials
joshblack
committed
18cfd65
feat: improve copilot cli message parsing and run trial behavior
joshblack
committed
46a22c4
refactor: move run into trial and plan
joshblack
committed
68ad168
refactor: add support for testing trial runs
joshblack
committed
8506898
test: fill out trial tests
joshblack
committed
c8eb054
feat: update output format and serialize, deserialize
joshblack
committed
daa9b02
feat: restore table emit behavior
joshblack
committed
dc21648
test: add tests and changeset
joshblack
committed
dc35b3b
refactor: update to new config
joshblack
committed
e3784de
chore: update design system benchmark
joshblack
committed
07c41a4
fix: make config const generic so the type does not have to be explicit
joshblack
committed
e6c44b6
refactor: update tests and benchmark summaries
joshblack
committed
d09dff1
ci: add workflows for benchmark
joshblack
committed
02f360f
feat: switch up output format, change website
joshblack
committed
d3b45e1
refactor: remove baseline results
joshblack
committed
2f0e34a
feat: move towards benchmark trends
joshblack
committed
db929c3
refactor: clean up benchmark models and sandbox cleanup
joshblack
committed
cb59e18
refactor: move benchmark scenarios to stacked change

joshblack
and
Copilot
committed
aa64c16
chore: log sandbox commands

joshblack
and
Copilot
committed
2e98455
chore: align benchmark support lockfile

joshblack
and
Copilot
committed
b469cc3
fix: accept sub-agent user messages

joshblack
and
Copilot
committed
02edbcd
fix: remove stale package exports

joshblack
and
Copilot
committed
3bc0e0c
fix: exclude walkthrough skill from artifacts

joshblack
and
Copilot
committed
8a88259
fix: collect benchmark output token metrics
joshblack
committed
f6be5ce
fix: restore benchmark overview interactions
joshblack
committed
24dfd15
fix: preserve scenario path and browser tests

joshblack
and
Copilot
committed
3897fc6
refactor: remove cli package entrypoint

joshblack
and
Copilot
committed
5ca8506
docs: shorten benchmark changesets

joshblack
and
Copilot
committed
031cf72
docs: update benchmark API documentation

joshblack
and
Copilot
committed
1c3bc63
refactor: remove baseline experiment
joshblack
committed
3f36743
test: consolidate experiment coverage
joshblack
committed
f57a0f5
fix: resolve benchmark support integration
joshblack
committed
ac56793
refactor: derive artifact directory from output
joshblack
committed
47d2719
fix: address benchmark review feedback

joshblack
and
Copilot
committed
e58491f
fix: address remaining benchmark review feedback
joshblack
committed
76ed0da