Skip to content

feat: allow an empty baseline skill list - #44

Open
queso wants to merge 1 commit into
mainfrom
feat/empty-baseline-skills
Open

queso wants to merge 1 commit into
mainfrom
feat/empty-baseline-skills

Conversation

@queso

@queso queso commented Sep 22, 2026

Copy link
Copy Markdown
Owner

Closes #38.

Problem

compare rejected "baselineSkills": [] with compare requires at least one baseline skill, so the most basic question a compare can answer, whether adding a skill changes anything at all, needed a placeholder skill file whose body said "No additional guidance". That placeholder contaminates the baseline arm it stands in for: under inline delivery its text is in the system prompt, and under install delivery it registers a skill whose description the model can see.

Change

An explicit empty array now loads. Under inline delivery the baseline system prompt is the agent file alone; under install delivery that arm's sandbox registry stays empty.

Two guards were deliberately kept or narrowed rather than dropped:

  • Omitting or misspelling baselineSkills still throws in normalizeSkills, so a typo cannot silently become a no-skill baseline. Only an explicit [] is accepted.
  • proposedSkills still requires at least one skill for a real compare, but not under single-arm loading. measure mirrors baseline into a proposed arm that no run exercises, so once an empty baseline became legal, measure on a no-skill scenario failed with compare requires at least one proposed skill, naming an arm the user never wrote. validateCompareConfig now takes the load options and skips that check when singleArm is set.

Also: the install-delivery progress line printed a dangling installed skills: with nothing after it for an empty arm, and now prints (none).

Tests

140 pass, 0 fail, tsc --noEmit clean.

  • explicit [] loads with an empty baseline arm, omitted key still throws, empty proposedSkills still throws
  • a top-level shared skills set does not backfill an explicit [], pinning the Array.isArray branch that returns before the shared-skills fallback
  • single-arm loading accepts an empty baseline and its mirrored empty proposed arm; the same scenario as a real compare still throws
  • inline delivery with an empty baseline sends the agent body verbatim, with no skill section
  • installSkills([], dir) installs nothing and creates no .claude/skills directory

Docs

README gains a note that baselineSkills may be [] and why that beats a placeholder file. SPEC's scenario-format list now says the baseline may be empty.

🤖 Generated with Claude Code

The most basic question a compare can answer is whether adding a skill
changes anything at all, which wants a baseline arm with no skill. Config
load rejected `"baselineSkills": []`, so the workaround was a placeholder
skill file. That placeholder contaminates the baseline it is standing in
for: its text lands in the system prompt under inline delivery, and its
description is visible in the sandbox registry under install delivery.

An explicit empty array now loads. Omitting or misspelling the key still
fails in normalizeSkills, so a typo cannot silently become a no-skill
baseline. The proposed arm still requires at least one skill for a real
compare, but not under single-arm loading, where proposed is a mirror of
baseline that no run exercises: without that, measure on a no-skill
scenario failed with an error naming a proposed arm the user never wrote.

Closes #38

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick review — approve

The change adds small, well-scoped logic to the engine's config and comparison paths, backed by substantial test coverage including dedicated config tests. The diff is clean with no correctness, security, or performance concerns; the change is safe to merge.

0 inline comment(s).

Verdict was approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compare requires a baseline skill, so a "no skill" baseline needs a placeholder

1 participant