Skip to content

Phase A: constraint benchmark and solve rate reporting - #37

Merged
barrybecker4 merged 6 commits into
masterfrom
claude/constraint-benchmark
Oct 25, 2026
Merged

barrybecker4 merged 6 commits into
masterfrom
claude/constraint-benchmark

Conversation

@barrybecker4

Copy link
Copy Markdown
Member

Phase A of the new strategies plan: a constraint benchmark

This branch includes the plan (#36), so merging this makes #36 unnecessary.

What it adds

  • Three constraint problems in scala-test/.../benchmark, each with a known solution and fitness 0 as the goal:
    • NQueensProblem: a permutation, with an allocation free swap delta in evaluateMove.
    • GraphColoringProblem: integer variables, built around a hidden coloring so a solution exists.
    • AllIntervalProblem: a permutation with many plateaus; the check that a strategy is not tuned to queens.
  • Solve rate reporting: ./gradlew benchmark -Pargs="--solve-rate=true" runs them with 20 seeds and 2 s each, stops a run at its first solution, and writes docs/solve-rate-results.md (solved runs of 20, median time and evaluations of the solved ones, median broken constraints of the rest). The ordinary benchmark is unchanged.
  • A strategy that rejects a problem when it starts (global annealing and global hill climbing cannot sample 200 integer parameters) is skipped rather than ending the benchmark.
  • Tests for each problem (known solutions, delta equals full evaluation for every swap), the limiter, the report, the options and the run.

Baseline (two processors, 20 seeds, 2 s)

Problem Annealing Genetic search Hill climbing, ILS
queens-100 / 400 20/20 0/20 0/20
coloring-200 0/20 0/20 0/20
all-interval-14 1/20 10/20 0 to 5/20

So the benchmark separates the strategies: annealing is already good on swap problems, nothing handles integer variables, and genetic search wins only the all-interval series. Full table in docs/solve-rate-results.md. Notes on those results and other problems worth adding (3-SAT, Costas arrays) are in the plan.

Testing

562 tests pass in my manual build; the 6 failures are the same known ones as before. I could not run Gradle here.


Generated by Claude Code

@barrybecker4
barrybecker4 merged commit 560aae1 into master Oct 25, 2026
1 check passed
@barrybecker4
barrybecker4 deleted the claude/constraint-benchmark branch October 25, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants