Conversation
The most basic question a compare can answer is whether adding a skill changes anything at all, which wants a baseline arm with no skill. Config load rejected `"baselineSkills": []`, so the workaround was a placeholder skill file. That placeholder contaminates the baseline it is standing in for: its text lands in the system prompt under inline delivery, and its description is visible in the sandbox registry under install delivery. An explicit empty array now loads. Omitting or misspelling the key still fails in normalizeSkills, so a typo cannot silently become a no-skill baseline. The proposed arm still requires at least one skill for a real compare, but not under single-arm loading, where proposed is a mirror of baseline that no run exercises: without that, measure on a no-skill scenario failed with an error naming a proposed arm the user never wrote. Closes #38 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Nitpick review — approve
The change adds small, well-scoped logic to the engine's config and comparison paths, backed by substantial test coverage including dedicated config tests. The diff is clean with no correctness, security, or performance concerns; the change is safe to merge.
0 inline comment(s).
Verdict was
approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #38.
Problem
comparerejected"baselineSkills": []withcompare requires at least one baseline skill, so the most basic question a compare can answer, whether adding a skill changes anything at all, needed a placeholder skill file whose body said "No additional guidance". That placeholder contaminates the baseline arm it stands in for: under inline delivery its text is in the system prompt, and under install delivery it registers a skill whose description the model can see.Change
An explicit empty array now loads. Under inline delivery the baseline system prompt is the agent file alone; under install delivery that arm's sandbox registry stays empty.
Two guards were deliberately kept or narrowed rather than dropped:
baselineSkillsstill throws innormalizeSkills, so a typo cannot silently become a no-skill baseline. Only an explicit[]is accepted.proposedSkillsstill requires at least one skill for a real compare, but not under single-arm loading.measuremirrors baseline into a proposed arm that no run exercises, so once an empty baseline became legal,measureon a no-skill scenario failed withcompare requires at least one proposed skill, naming an arm the user never wrote.validateCompareConfignow takes the load options and skips that check whensingleArmis set.Also: the install-delivery progress line printed a dangling
installed skills:with nothing after it for an empty arm, and now prints(none).Tests
140 pass, 0 fail,
tsc --noEmitclean.[]loads with an empty baseline arm, omitted key still throws, emptyproposedSkillsstill throwsskillsset does not backfill an explicit[], pinning theArray.isArraybranch that returns before the shared-skills fallbackinstallSkills([], dir)installs nothing and creates no.claude/skillsdirectoryDocs
README gains a note that
baselineSkillsmay be[]and why that beats a placeholder file. SPEC's scenario-format list now says the baseline may be empty.🤖 Generated with Claude Code