From 5cbc1e38a01fe1d425215031ded2b7b6b2da990b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 06:54:56 -0700 Subject: [PATCH 1/5] Add plan for tabu search, min-conflicts and adaptive restarts Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0175jMLxrP7wZ3vfj6UfeDZx --- docs/new-strategies-plan.md | 191 ++++++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/new-strategies-plan.md diff --git a/docs/new-strategies-plan.md b/docs/new-strategies-plan.md new file mode 100644 index 0000000..b449fbb --- /dev/null +++ b/docs/new-strategies-plan.md @@ -0,0 +1,191 @@ +# Plan: strategies for constraint problems (tabu search, min-conflicts, adaptive restarts) + +Status: proposal, nothing built yet. + +Decisions already made: new APIs are additive in 2.1-SNAPSHOT, and defaults keep today's behavior. A new strategy is +judged on the benchmark, with solve rate first and time second (see "Does it earn its place?"). + +## Why + +Puzzles such as the Red puzzle, Tantrix and sliding puzzles have one right answer: fitness 0, or nothing. For those the +question is not how good a result we reach in a fixed time but how often we reach 0 at all. The measurements from the +earlier phases show where we stand. On the 9 piece Red puzzle, genetic search found the solution in 5 of 12 runs and +annealing in 0 of 12 before guided moves were added (PR #66 in bb4-puzzles), and Tantrix genetic search solved 4 of 6 +runs of 9 tiles in about 8 seconds. Genetic search keeps a population and recombines it, which pays off when good +partial solutions can be combined. Constraint problems mostly do not work that way: a solution is a single assignment +in which a few constraints are violated, and the useful step is to repair one violation without breaking others. +Local search methods built for that are tabu search, min-conflicts and restarts. + +What exists today: + +- Hill climbing, global hill climbing, simulated annealing, iterated local search (ILS) and variable neighborhood search + (VNS). ILS and VNS already restart from a random sample after `restartAfterStagnantIterations`. Global hill climbing + and global annealing start a fixed number of searches from the best of a set of samples. None of them adapts when to + give up on a search. +- Moves (`Move`, `Swap`, `SwapSequence`), `Optimizee.evaluateMove` for delta evaluation, `proposeGuidedMove`, `repair`, + `BudgetedOptimizee` (which stops a search after a number of evaluations by throwing + `EvaluationBudgetExceededException`). These are the building blocks for everything below. +- Moves exist only for permutations (`PermutedParameterArray`). Arrays of integers or booleans, which is what coloring, + scheduling and SAT style problems use, have `getRandomNeighbor` but no moves. +- Nothing in the benchmark is a constraint satisfaction problem with a known solution. `tsp-*` and `tour-*` are + optimization problems, `subset-sum` and `dominating-set` are small. + +## The three techniques, and what each needs + +**Adaptive restarts.** Run a search for a budget of evaluations, and if it has not reached the goal, start a new one +from a fresh random sample (or from a perturbed best), with budgets that grow, for example the Luby sequence 1, 1, 2, +1, 1, 2, 4, ... times a base. Heavy tailed run times are the reason: a search that is going to succeed usually does so +early, and a long run is more often stuck than close. The "adaptive" part is that the base budget and the choice +between a random and a perturbed start follow what was seen (how soon earlier runs reached their best, whether the best +keeps improving between runs). It is a wrapper around any existing strategy, so it is cheap to build and helps all of +them. It needs no new hooks, because `BudgetedOptimizee` already ends a run. + +**Tabu search.** From the current solution look at a set of moves (all of them for a small neighborhood, a random +sample for a large one), take the best move that is not tabu even if it is worse than where we are, and forbid undoing it +for a few iterations. An aspiration rule allows a tabu move that beats the best solution so far. This walks out of +local optima without randomness, and works well on permutation problems. It needs: + +- `Move.attributes`, a small description of what a move changes that the tabu list stores (for `Swap(i, j)` the two + positions with the values they held). The default is the move itself. +- A way to get a set of candidate moves. `proposeMove` gives random ones. For permutations a full enumeration of the + swaps is also worth having when n is small, so a `ParameterArray.enumerateMoves(limit)` with a default of "not + supported" (the strategy then samples). +- Delta evaluation (`evaluateMove`) so that looking at many moves is cheap. Without it tabu search is slow, since it + evaluates a whole neighborhood every step. + +**Min-conflicts.** Pick a variable that violates a constraint, and change it to the value that violates the fewest, with +a small chance of a random move so it does not cycle. It is the standard method for N-queens and coloring and works well +whenever the problem can say which variables are in conflict. It needs: + +- `Optimizee.conflictedPositions(current): Seq[Int]`, default empty, meaning "unknown". Strategies fall back to a + random position. This is the same kind of hook as `proposeGuidedMove` and `GuidedSwap` in the Red puzzle, which is + min-conflicts by another name, and the Red result (9 of 20 solved to 15 of 20) is the best evidence that it pays off. +- A move that sets one position to one value, for arrays that are not permutations: `SetValue(index, value)`. For + permutations the equivalent is a swap of the conflicted position with the best partner, found by scoring the + swaps with `evaluateMove`. +- Which values to try at a position. For integer and boolean parameters that is the parameter's range; for + permutations it is the other positions. + +## Does it earn its place? (decision rule) + +Build the benchmark first, then add a strategy to `OptimizationStrategyType` only if, on the constraint problems, it +solves more runs than the best existing strategy on at least one of them, or solves as many in clearly less time, and it +does not get worse on the others by more than the noise. A strategy that fails the rule stays as a class, off the enum +and off the default benchmark list, and the plan says so. This is the same discipline as the guided moves for +Tantrix, which did not pass and were left off. + +## Phases + +### Phase A: a way to measure (first, because everything else is judged by it) + +- Constraint problems in `BenchmarkProblems`, each with a known solution and `getOptimalFitness` of 0: + - N-queens as a permutation of 100 and 400 queens (conflicts counted on the diagonals; with an allocation free swap + delta). The classic case for min-conflicts and tabu. + - Graph coloring: a random graph that is built around a hidden coloring, so a solution exists, as integer parameters. + Uses the `SetValue` move from phase B, so a version without delta evaluation comes first. + - A hard permutation problem such as all-interval series, to keep the benchmark from being tuned to N-queens. + - The Red puzzle lives in bb4-puzzles, so it is measured there (as in the Phase 4 work), not here. +- The benchmark reports, per problem and strategy, the number of runs that reached fitness 0, and the median time + and evaluations of those that did. The current "fitness reached in a fixed time" column stays for optimization + problems. Runs are repeated over more seeds (20), since solve rates need more samples than medians. +- Baseline of today's strategies on these problems, committed in `docs/benchmark-results.md` like the others. + +### Phase B: shared pieces (additive, no behavior change) + +- `Move.attributes` and a `SetValue(index, value)` move. +- Moves for `NumericParameterArray` when every parameter is an integer (and for boolean parameters), with + `proposeMove`, `applyMove` and `supportsMoves`, consistent with `getRandomNeighbor`: applying the proposed move + gives the same array as `getRandomNeighbor` for the same random numbers (the rule used for permutations). +- `Optimizee.conflictedPositions` with a default of empty, forwarded by the wrappers (`BudgetedOptimizee`, + `DeltaCheckingOptimizee`, `LimitedOptimizee`) as `repair` and `proposeGuidedMove` are. +- `ParameterArray.enumerateMoves(limit)` for permutations, with a default of "unsupported". +- `DeltaCheckingOptimizee` already checks any move against a full evaluation, so the new move types are covered by it. + +### Phase C: adaptive restarts + +- `RestartingStrategy(innerType, config)`: runs the inner strategy through `BudgetedOptimizee` for a budget taken from + a Luby sequence times `baseBudget`, keeps the best result, and stops at fitness 0, at the overall evaluation budget, or + when the iteration limit is reached. A restart starts from `getRandomSample`, or from a perturbed copy of the best (a + `getRandomNeighbor` with a large radius) with a probability that rises when fresh starts have not beaten the best. +- The base budget is adapted from the first runs (a fraction of the evaluations that the first run took to reach its + best), unless it is given. +- Enum entries for the useful wrapped combinations are decided by the rule above, probably restarting hill climbing + and restarting annealing, not every pair. A `RestartingStrategy` can be built for any type by code. +- Tests: the Luby sequence, restarts stop at the goal, the budget is shared and never exceeded, the best result of + all runs is returned, and a seeded run repeats. + +### Phase D: tabu search + +- `TabuSearchStrategy` with `TabuSearchConfig`: `maxIterations`, `candidateMoves` (how many moves to look at per + step; all of them when the neighborhood is smaller), `tenureMin`/`tenureMax` (random per move, which avoids cycles + of a fixed length), `aspiration` on or off, `restartAfterStagnantIterations`. +- Uses moves and `evaluateMove`; for an array without moves it builds neighbors and scores them, like the other + strategies, but is documented as slow there. +- Tests: a move that is tabu is not taken, the aspiration rule lets a tabu move through when it beats the best, + the tenure expires, a seeded run repeats, budget exhaustion returns the best so far, and a solved problem ends early. + +### Phase E: min-conflicts + +- `MinConflictsStrategy` with `MinConflictsConfig`: `maxSteps`, `noiseProbability` (a random move instead of the best + one), `tieBreaking` (random among equally good values), and `restartAfterStagnantSteps`. +- Each step: take `optimizee.conflictedPositions(current)`, pick one at random, and look at the values or swap partners + for that position through `evaluateMove`, taking the best, or a random one with the noise probability. If the + optimizee does not report conflicts the step picks a random position, so the strategy still runs but is only a + randomized greedy search. That case is documented, not hidden. +- Tests: with a problem that reports conflicts it only touches conflicted positions, the noise probability of 0 is + deterministic given the seed, it solves small N-queens and a small coloring, and the fallback runs. + +### Phase F: register, document, adopt + +- `OptimizationStrategyType` entries, descriptions and `appliesTo` (min-conflicts and tabu only for arrays with moves; + min-conflicts is useful only when the optimizee reports conflicts, so `appliesTo` stays permissive and the + description says so). Update the benchmark default list, the README strategy table and the results file. +- In bb4-puzzles: give the Red puzzle `conflictedPositions` (its loose edges, which `GuidedSwap` already finds) and try + tabu and min-conflicts there, as in Phase 4. Tantrix and the sliding puzzle only if the benchmark and Red results + say they are worth it. The puzzle results go in that repo's `docs/benchmarks.md`. + +## Pull requests + +1. Phase A: constraint problems, solve rate reporting, baseline results. (Docs and test code only.) +2. Phase B: moves for integer arrays, `attributes`, `conflictedPositions`, `enumerateMoves`. +3. Phase C: adaptive restarts. +4. Phase D: tabu search. +5. Phase E: min-conflicts. +6. Phase F: registration and docs, then the bb4-puzzles PRs. + +Phases C, D and E depend on A and B but not on each other, so they can be done in any order. I would do C first: it is +the smallest, it can use the existing strategies, and it tells us early whether solve rate responds to restarts on +these problems. + +## Risks + +- **Tuning.** Tabu tenure and the noise probability matter a lot. Defaults come from the benchmark problems and could be + wrong for a puzzle. Mitigation: the configs are public, and the results in the doc are from named problems and seeds. +- **Min-conflicts without conflict information is just greedy search.** It only earns its place when a problem reports + conflicts, which is extra work for the problem author. The fallback keeps it correct but not useful. +- **Solve rates need many runs.** Twenty seeds per cell makes the full benchmark longer. A `--quick` option or a + smaller list for the solve rate section may be needed. +- **Evaluation counting.** `BudgetedOptimizee` counts `evaluateMove` as an evaluation, so a tabu step that looks at 50 moves + uses 50 of the budget. That is the honest cost and the restart budgets are in the same units, but strategies must not + be compared with a count of steps. +- **Overlap with ILS/VNS.** Restarting ILS may add little, since ILS already restarts. The decision rule applies there too. +- **Determinism.** Restarts and tabu tenure use the strategy's random generator, so a seeded run repeats. Any threaded + variant (parallel restarts) is out of scope for now, since it would not. + +## Not in this plan + +- Constraint propagation or backtracking (the exact solvers in bb4-puzzles keep that role). +- Parallel restarts, or a portfolio that runs several strategies at once. +- New parameter array types. Boolean and integer arrays get moves, nothing else changes. +- Changing the defaults of existing strategies. + +## Open questions + +These have defaults, so the plan can proceed without answers: + +1. Should tabu search and min-conflicts be separate enum entries, as proposed, or options of an existing strategy? Default: separate + entries, which matches how ILS and VNS were added. +2. Is N-queens plus coloring plus all-interval series the right benchmark set? Default: yes, and add a problem from + bb4-puzzles later if one is needed. +3. Is a 20 seed solve rate section acceptable for the length of a full benchmark run (about 10 minutes now)? Default: + yes, with the section as its own `--solve-rate` run if it makes the full run too long. From 84ca2f82bdb55e7fdfe4a14db20d5e4a385190a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 07:10:47 -0700 Subject: [PATCH 2/5] Benchmark: constraint problems (queens, coloring, all-interval) and solve rate reporting Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0175jMLxrP7wZ3vfj6UfeDZx --- .../benchmark/AllIntervalProblem.scala | 50 +++++++++++ .../benchmark/AllIntervalProblemSuite.scala | 36 ++++++++ .../benchmark/BenchmarkConfig.scala | 32 +++++-- .../benchmark/BenchmarkConfigSuite.scala | 17 ++++ .../benchmark/BenchmarkProblems.scala | 21 ++++- .../benchmark/BenchmarkReport.scala | 46 +++++++++- .../benchmark/BenchmarkReportSuite.scala | 30 +++++++ .../optimization/benchmark/BenchmarkRun.scala | 18 ++-- .../benchmark/GraphColoringProblem.scala | 72 ++++++++++++++++ .../benchmark/GraphColoringProblemSuite.scala | 43 ++++++++++ .../benchmark/LimitedOptimizee.scala | 20 ++++- .../benchmark/LimitedOptimizeeSuite.scala | 29 +++++++ .../benchmark/NQueensProblem.scala | 85 +++++++++++++++++++ .../benchmark/NQueensProblemSuite.scala | 71 ++++++++++++++++ .../benchmark/OptimizationBenchmark.scala | 6 +- .../OptimizationBenchmarkSuite.scala | 25 ++++++ 16 files changed, 578 insertions(+), 23 deletions(-) create mode 100644 scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblem.scala create mode 100644 scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblemSuite.scala create mode 100644 scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblem.scala create mode 100644 scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblemSuite.scala create mode 100644 scala-test/com/barrybecker4/optimization/benchmark/NQueensProblem.scala create mode 100644 scala-test/com/barrybecker4/optimization/benchmark/NQueensProblemSuite.scala diff --git a/scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblem.scala b/scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblem.scala new file mode 100644 index 0000000..be3385e --- /dev/null +++ b/scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblem.scala @@ -0,0 +1,50 @@ +// Copyright by Barry G. Becker, 2026. Licensed under MIT License: http://www.opensource.org/licenses/MIT +package com.barrybecker4.optimization.benchmark + +import com.barrybecker4.optimization.optimizee.optimizees.OptimizeeProblem +import com.barrybecker4.optimization.parameter.types.IntegerParameter +import com.barrybecker4.optimization.parameter.{ParameterArray, ParameterArrayWithFitness, PermutedParameterArray} + +import scala.util.Random + +/** + * The all-interval series: arrange the numbers 0 to n - 1 so that the differences between neighbors are all + * different, which makes them exactly 1 to n - 1. It comes from music (a twelve tone row that uses every interval). + * The fitness is the number of those differences that no pair of neighbors has, and 0 is a solution. + * + * It is a good counterweight to N-queens for a benchmark: there are many plateaus, because changing one number moves + * two differences at once and rarely helps, and there is no conflict to repair directly. + * @param size how many numbers; at least 3 + * @param seed decides the order in the initial guess + */ +class AllIntervalProblem(size: Int, seed: Long) extends OptimizeeProblem { + + require(size >= 3, s"The series needs at least 3 numbers, but got $size") + + override def getName: String = s"All-interval series of $size" + + override def evaluateByComparison: Boolean = false + + /** The number of differences from 1 to n - 1 that are missing. */ + override def evaluateFitness(series: ParameterArray): Double = { + val present = new Array[Boolean](size) + (1 until size).foreach(i => present(Math.abs(numberAt(series, i) - numberAt(series, i - 1))) = true) + (1 until size).count(!present(_)).toDouble + } + + private def numberAt(series: ParameterArray, position: Int): Int = series.get(position).getValue.toInt + + override def getInitialGuess: ParameterArray = seriesOf(new Random(seed + 1).shuffle((0 until size).toList), seed + 2) + + /** Alternating the smallest and the largest number that is left gives the differences n - 1, n - 2, ..., 1. */ + override def getExactSolution: ParameterArrayWithFitness = { + val series = seriesOf((0 until size).map(i => if (i % 2 == 0) i / 2 else size - 1 - i / 2), seed) + ParameterArrayWithFitness(series, evaluateFitness(series)) + } + + private def seriesOf(numbers: Seq[Int], randomSeed: Long): ParameterArray = + PermutedParameterArray(numbers.indices.map(i => new IntegerParameter(numbers(i), 0, size - 1, "n" + i)), new Random(randomSeed)) + + /** A random series misses about a third of the differences. */ + override def getFitnessRange: Double = size.toDouble +} diff --git a/scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblemSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblemSuite.scala new file mode 100644 index 0000000..fde7c0f --- /dev/null +++ b/scala-test/com/barrybecker4/optimization/benchmark/AllIntervalProblemSuite.scala @@ -0,0 +1,36 @@ +// Copyright by Barry G. Becker, 2026. Licensed under MIT License: http://www.opensource.org/licenses/MIT +package com.barrybecker4.optimization.benchmark + +import org.scalatest.funsuite.AnyFunSuite + +class AllIntervalProblemSuite extends AnyFunSuite { + + private def arrayOf(values: Seq[Int]) = com.barrybecker4.optimization.parameter.PermutedParameterArray( + values.indices.map(i => new com.barrybecker4.optimization.parameter.types.IntegerParameter(values(i), 0, values.size - 1, "p" + i)), + new scala.util.Random(1)) + + test("the known series has every difference once, for any size") { + (3 to 40).foreach(size => assertResult(0.0, s"size $size")(new AllIntervalProblem(size, 1).getExactSolution.fitness)) + } + + test("numbers in order have just one difference, so all the others are missing") { + val problem = new AllIntervalProblem(10, 1) + assertResult(8.0)(problem.evaluateFitness(arrayOf(0 until 10))) + } + + test("the initial guess has each number once, and is not already a solution") { + val problem = new AllIntervalProblem(14, 2) + val guess = problem.getInitialGuess + assertResult((0 until 14).toList)((0 until 14).map(guess.get(_).getValue.toInt).sorted.toList) + assert(problem.evaluateFitness(guess) > 0) + } + + test("the same seed gives the same initial guess") { + def values(seed: Long) = { val g = new AllIntervalProblem(14, seed).getInitialGuess; (0 until 14).map(g.get(_).getValue) } + assertResult(values(3))(values(3)) + } + + test("a series needs at least 3 numbers") { + intercept[IllegalArgumentException](new AllIntervalProblem(2, 1)) + } +} diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfig.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfig.scala index b31813d..93624e2 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfig.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfig.scala @@ -13,6 +13,11 @@ object BenchmarkConfig { val DEFAULT_REPORT_FILE = "docs/benchmark-results.md" private val NO_REPORT = "none" + /** The solve rate run is separate, with more seeds and a shorter limit, since a failed run takes the whole limit. */ + val DEFAULT_SOLVE_RATE_REPORT_FILE = "docs/solve-rate-results.md" + val DEFAULT_SOLVE_RATE_SEEDS = 20 + val DEFAULT_SOLVE_RATE_TIME_LIMIT_MILLIS = 2000L + /** The strategies that make sense on permutations and other discrete problems, and finish on their own. */ val DEFAULT_STRATEGIES: Seq[OptimizationStrategyType] = Seq( SIMULATED_ANNEALING, GLOBAL_SIMULATED_ANNEALING, HILL_CLIMBING, GLOBAL_HILL_CLIMBING, @@ -23,29 +28,41 @@ object BenchmarkConfig { * (the last two are comma separated lists), report (the Markdown summary file, or `none`) and out (a file to * write the tab separated results of every run to). * + * The option `solve-rate=true` runs the constraint problems instead (see [[BenchmarkProblems.CONSTRAINT]]), with + * 20 seeds and 2 seconds as the defaults, and reports how many runs solved each, and how fast. Its full run writes + * [[DEFAULT_SOLVE_RATE_REPORT_FILE]]. + * * A full run (all problems and the default strategies) writes the summary to [[DEFAULT_REPORT_FILE]]. A run * restricted to some problems or strategies writes no summary unless `report` is given, so it cannot replace * the full one with a partial one. */ def parse(args: Seq[String]): BenchmarkConfig = { val options = args.filter(_.startsWith("--")).map(_.drop(2).split("=", 2)).collect { case Array(k, v) => k -> v }.toMap + val solveRate = options.get("solve-rate").contains("true") val problemNames = options.get("problems").map(_.split(",").toSeq).getOrElse(Seq.empty) val strategies = options.get("strategies").map(_.split(",").toSeq.map(OptimizationStrategyType.valueOf)) .getOrElse(DEFAULT_STRATEGIES) + val isFullRun = problemNames.isEmpty && strategies == DEFAULT_STRATEGIES BenchmarkConfig( - timeLimitMillis = options.get("time-ms").map(_.toLong).getOrElse(DEFAULT_TIME_LIMIT_MILLIS), + timeLimitMillis = options.get("time-ms").map(_.toLong).getOrElse(defaultTimeLimit(solveRate)), maxEvaluations = options.get("max-evals").map(_.toLong).getOrElse(DEFAULT_MAX_EVALUATIONS), - numSeeds = options.get("seeds").map(_.toInt).getOrElse(DEFAULT_NUM_SEEDS), - problemNames = problemNames, + numSeeds = options.get("seeds").map(_.toInt).getOrElse(defaultSeeds(solveRate)), + problemNames = if (solveRate && problemNames.isEmpty) BenchmarkProblems.CONSTRAINT.map(_.name) else problemNames, strategies = strategies, outputFile = options.get("out"), - reportFile = reportFile(options.get("report"), isFullRun = problemNames.isEmpty && strategies == DEFAULT_STRATEGIES)) + reportFile = reportFile(options.get("report"), isFullRun, solveRate), + solveRate = solveRate) } - private def reportFile(requested: Option[String], isFullRun: Boolean): Option[String] = requested match { + private def defaultTimeLimit(solveRate: Boolean): Long = + if (solveRate) DEFAULT_SOLVE_RATE_TIME_LIMIT_MILLIS else DEFAULT_TIME_LIMIT_MILLIS + + private def defaultSeeds(solveRate: Boolean): Int = if (solveRate) DEFAULT_SOLVE_RATE_SEEDS else DEFAULT_NUM_SEEDS + + private def reportFile(requested: Option[String], isFullRun: Boolean, solveRate: Boolean): Option[String] = requested match { case Some(NO_REPORT) => None case Some(file) => Some(file) - case None => Option.when(isFullRun)(DEFAULT_REPORT_FILE) + case None => Option.when(isFullRun)(if (solveRate) DEFAULT_SOLVE_RATE_REPORT_FILE else DEFAULT_REPORT_FILE) } } @@ -57,7 +74,8 @@ object BenchmarkConfig { * @param strategies the strategies to run * @param outputFile where to write the results of every run as tab separated values, if anywhere * @param reportFile where to write the summary as a Markdown document, if anywhere + * @param solveRate true to report how many runs reached the optimal fitness, for the constraint problems */ case class BenchmarkConfig(timeLimitMillis: Long, maxEvaluations: Long, numSeeds: Int, problemNames: Seq[String], strategies: Seq[OptimizationStrategyType], - outputFile: Option[String], reportFile: Option[String]) + outputFile: Option[String], reportFile: Option[String], solveRate: Boolean = false) diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfigSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfigSuite.scala index cd549fc..1c0283d 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfigSuite.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkConfigSuite.scala @@ -53,4 +53,21 @@ class BenchmarkConfigSuite extends AnyFunSuite { val error = intercept[IllegalArgumentException](BenchmarkProblems.named(Seq("nope"))) assert(error.getMessage.contains("tour-50")) } + + test("a solve rate run is of the constraint problems, with more seeds and a shorter limit") { + val config = BenchmarkConfig.parse(Seq("--solve-rate=true")) + assert(config.solveRate) + assertResult(BenchmarkProblems.CONSTRAINT.map(_.name))(config.problemNames) + assertResult((20, 2000L))((config.numSeeds, config.timeLimitMillis)) + assertResult(Some("docs/solve-rate-results.md"))(config.reportFile) + } + + test("the solve rate defaults can be overridden, and some problems can be chosen") { + val config = BenchmarkConfig.parse(Seq("--solve-rate=true", "--seeds=3", "--time-ms=100", "--problems=queens-100")) + assertResult((3, 100L, Seq("queens-100"), None))((config.numSeeds, config.timeLimitMillis, config.problemNames, config.reportFile)) + } + + test("an ordinary run is not a solve rate run") { + assert(!BenchmarkConfig.parse(Seq.empty).solveRate) + } } diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkProblems.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkProblems.scala index 0f399f9..75f3946 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkProblems.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkProblems.scala @@ -6,8 +6,10 @@ import com.barrybecker4.optimization.optimizee.optimizees.problems.* /** * A problem to benchmark. `create` makes a fresh problem for the seed, so that repeated runs start from the same place. + * @param isConstraint true when the problem has a solution that is known to exist and the goal is to reach it. Such a + * problem is judged by how many runs solve it and how quickly, and a run stops when it does. */ -case class BenchmarkProblem(name: String, create: Long => OptimizeeProblem) +case class BenchmarkProblem(name: String, create: Long => OptimizeeProblem, isConstraint: Boolean = false) /** The problems that are benchmarked. The tours of increasing length show how a strategy scales with permutation size. */ object BenchmarkProblems { @@ -21,9 +23,22 @@ object BenchmarkProblems { BenchmarkProblem("dominating-set", seed => new DominatingSetProblem(TYPICAL_DS)) ) + /** + * Problems where the goal is to find a solution that is known to exist. They are run on their own (see the + * `--solve-rate` option), with more seeds, since a solve rate needs more samples than a median does. + * The sizes are ones where the existing strategies solve some runs and not others in a couple of seconds. + */ + val CONSTRAINT: Seq[BenchmarkProblem] = Seq( + BenchmarkProblem("queens-100", seed => new NQueensProblem(100, seed), isConstraint = true), + BenchmarkProblem("queens-400", seed => new NQueensProblem(400, seed), isConstraint = true), + BenchmarkProblem("coloring-200", seed => new GraphColoringProblem(200, 4, 7.0, seed), isConstraint = true), + BenchmarkProblem("all-interval-14", seed => new AllIntervalProblem(14, seed), isConstraint = true) + ) + + /** @return the problems with these names, or all of the ordinary ones if there are no names */ def named(names: Seq[String]): Seq[BenchmarkProblem] = - if (names.isEmpty) ALL else names.map(n => ALL.find(_.name == n).getOrElse(throw new IllegalArgumentException( - s"Unknown problem '$n'. The problems are ${ALL.map(_.name).mkString(", ")}"))) + if (names.isEmpty) ALL else names.map(n => (ALL ++ CONSTRAINT).find(_.name == n).getOrElse(throw new IllegalArgumentException( + s"Unknown problem '$n'. The problems are ${(ALL ++ CONSTRAINT).map(_.name).mkString(", ")}"))) /** The TSP variations pick their initial guess from one shared random generator, so make it start the same. */ private def resetSharedRandom(seed: Long): Unit = TravelingSalesmanVariation.RANDOM.setSeed(seed) diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReport.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReport.scala index cf7f724..85abbaa 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReport.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReport.scala @@ -6,6 +6,16 @@ case class Summary(problem: String, strategy: String, runs: Int, evaluationsPerS bytesPerEvaluation: Option[Double], medianFitness: Double, bestFitness: Double, medianEvaluations: Double, stoppedBy: String) +/** + * How often one strategy solved one constraint problem. + * @param solved the number of runs that reached the optimal fitness + * @param medianMillis the median time of the runs that solved it, or NaN if none did + * @param medianEvaluations the median number of evaluations of the runs that solved it, or NaN if none did + * @param medianUnsolvedFitness the median best fitness of the runs that did not, or NaN if all did + */ +case class SolveSummary(problem: String, strategy: String, runs: Int, solved: Int, medianMillis: Double, + medianEvaluations: Double, medianUnsolvedFitness: Double) + /** Turns the results of runs into tables. */ object BenchmarkReport { @@ -32,6 +42,16 @@ object BenchmarkReport { } } + /** One summary for each problem and strategy, in the order they first appear. */ + def summarizeSolves(results: Seq[RunResult]): Seq[SolveSummary] = { + val groups = results.groupBy(r => (r.problem, r.strategy.toString)) + results.map(r => (r.problem, r.strategy.toString)).distinct.map { key => + val (solved, unsolved) = groups(key).partition(_.solved) + SolveSummary(key._1, key._2, solved.size + unsolved.size, solved.size, median(solved.map(_.millis)), + median(solved.map(_.evaluations.toDouble)), median(unsolved.map(_.fitness))) + } + } + private def bytesPerEvaluation(runs: Seq[RunResult]): Option[Double] = { val values = runs.flatMap(_.bytesPerEvaluation) Option.when(values.nonEmpty)(median(values)) @@ -60,14 +80,36 @@ object BenchmarkReport { def document(settings: String, summaries: Seq[Summary]): String = Seq(DOCUMENT_TITLE, DOCUMENT_INTRO, settings, markdown(summaries)).mkString("\n\n") + "\n" + private val SOLVE_TITLE = "# Solve rate results" + private val SOLVE_INTRO = + "Written by `./gradlew benchmark -Pargs=\"--solve-rate=true\"`; see [new-strategies-plan.md](new-strategies-plan.md).\n" + + "Each run stops when it solves the problem, so the time is the time to a solution. The unsolved fitness is the\n" + + "median number of constraints still broken in the runs that did not. Only compare runs made on the same machine." + + /** The solve rates as a document to keep in git, with no date, like [[document]]. */ + def solveDocument(settings: String, summaries: Seq[SolveSummary]): String = + Seq(SOLVE_TITLE, SOLVE_INTRO, settings, solveMarkdown(summaries)).mkString("\n\n") + "\n" + + def solveMarkdown(summaries: Seq[SolveSummary]): String = { + val header = "| Problem | Strategy | Solved | Median ms (solved) | Median evals (solved) | Median broken (unsolved) |\n" + + "| --- | --- | ---: | ---: | ---: | ---: |" + val rows = summaries.map { s => + s"| ${s.problem} | ${s.strategy} | ${s.solved}/${s.runs} | ${number(s.medianMillis, "%,.0f")} | " + + s"${number(s.medianEvaluations, "%,.0f")} | ${number(s.medianUnsolvedFitness, "%.1f")} |" + } + (header +: rows).mkString("\n") + } + + private def number(value: Double, format: String): String = if (value.isNaN) "-" else format.format(value) + private def bytes(value: Option[Double]): String = value.map(v => f"$v%,.0f").getOrElse("n/a") /** One line for every run, so results can be compared with a spreadsheet or a script. */ def tabSeparated(results: Seq[RunResult]): String = { - val header = Seq("problem", "strategy", "seed", "evaluations", "millis", "fitness", "allocatedBytes", "stoppedBy") + val header = Seq("problem", "strategy", "seed", "evaluations", "millis", "fitness", "allocatedBytes", "stoppedBy", "solved") val rows = results.map { r => Seq(r.problem, r.strategy, r.seed, r.evaluations, f"${r.millis}%.1f", r.fitness, - r.allocatedBytes.getOrElse(""), r.stoppedBy).mkString("\t") + r.allocatedBytes.getOrElse(""), r.stoppedBy, r.solved).mkString("\t") } (header.mkString("\t") +: rows).mkString("\n") } diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReportSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReportSuite.scala index 9eca2d7..060d073 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReportSuite.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkReportSuite.scala @@ -78,4 +78,34 @@ class BenchmarkReportSuite extends AnyFunSuite { assert(document.contains(BenchmarkReport.markdown(summaries))) assertResult(document)(BenchmarkReport.document("Java 21, 2 processors", summaries)) } + + private def solveResult(solved: Boolean, millis: Double, evaluations: Long, fitness: Double) = + RunResult("queens", SIMULATED_ANNEALING, 1, evaluations, millis, fitness, None, + if (solved) LimitedOptimizee.SOLVED else LimitedOptimizee.TIME_LIMIT, solved) + + test("a solve summary counts the runs that solved, and takes the medians of those, and of the fitness of the others") { + val summary = BenchmarkReport.summarizeSolves(Seq( + solveResult(solved = true, 10.0, 100, 0.0), solveResult(solved = true, 30.0, 300, 0.0), + solveResult(solved = false, 2000.0, 9000, 4.0), solveResult(solved = false, 2000.0, 9000, 2.0))).head + assertResult((4, 2))((summary.runs, summary.solved)) + assertResult((20.0, 200.0, 3.0))((summary.medianMillis, summary.medianEvaluations, summary.medianUnsolvedFitness)) + } + + test("a strategy that never solved has no time, and one that always did has no unsolved fitness") { + val never = BenchmarkReport.summarizeSolves(Seq(solveResult(solved = false, 2000.0, 10, 3.0))).head + assert(never.medianMillis.isNaN && never.solved == 0) + val always = BenchmarkReport.summarizeSolves(Seq(solveResult(solved = true, 5.0, 10, 0.0))).head + assert(always.medianUnsolvedFitness.isNaN) + } + + test("the solve table shows solved of runs, and a dash where there is nothing to show") { + val table = BenchmarkReport.solveMarkdown(BenchmarkReport.summarizeSolves(Seq( + solveResult(solved = true, 12.0, 1500, 0.0), solveResult(solved = false, 2000.0, 9000, 4.0)))) + assert(table.contains("| queens | SIMULATED_ANNEALING | 1/2 | 12 | 1,500 | 4.0 |"), table) + } + + test("the solve document has a title, the settings and the table, and no date") { + val document = BenchmarkReport.solveDocument("settings here", Seq.empty) + assert(document.startsWith("# Solve rate results") && document.contains("settings here") && document.endsWith("\n")) + } } diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala index da0be8d..641a76c 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala @@ -11,10 +11,12 @@ import scala.util.Random * What one run of one strategy on one problem did. * @param fitness the best fitness found (lower is better) * @param allocatedBytes bytes allocated by the thread that ran the search, or none if not measured - * @param stoppedBy why it ended: finished, time or evals + * @param stoppedBy why it ended: finished, time, evals or solved + * @param solved true if the optimal fitness was reached (only looked for on constraint problems) */ case class RunResult(problem: String, strategy: OptimizationStrategyType, seed: Long, evaluations: Long, - millis: Double, fitness: Double, allocatedBytes: Option[Long], stoppedBy: String) { + millis: Double, fitness: Double, allocatedBytes: Option[Long], stoppedBy: String, + solved: Boolean = false) { def evaluationsPerSecond: Double = if (millis > 0) evaluations * 1000.0 / millis else 0.0 def bytesPerEvaluation: Option[Double] = allocatedBytes.filter(_ => evaluations > 0).map(_.toDouble / evaluations) } @@ -32,21 +34,23 @@ object BenchmarkRun { val optimizee = problem.create(seed) val initialGuess = optimizee.getInitialGuess if (!strategy.appliesTo(optimizee, initialGuess)) None - else Some(search(problem.name, optimizee, initialGuess, strategy, seed, config)) + else Some(search(problem, optimizee, initialGuess, strategy, seed, config)) } - private def search(name: String, problem: com.barrybecker4.optimization.optimizee.optimizees.OptimizeeProblem, + private def search(benchmarkProblem: BenchmarkProblem, problem: com.barrybecker4.optimization.optimizee.optimizees.OptimizeeProblem, initialGuess: com.barrybecker4.optimization.parameter.ParameterArray, strategyType: OptimizationStrategyType, seed: Long, config: BenchmarkConfig): RunResult = { val start = System.nanoTime() - val limited = new LimitedOptimizee(problem, config.maxEvaluations, start + config.timeLimitMillis * NANOS_PER_MILLI) + val limited = new LimitedOptimizee(problem, config.maxEvaluations, start + config.timeLimitMillis * NANOS_PER_MILLI, + stopWhenSolved = benchmarkProblem.isConstraint) val strategy = strategyType.getStrategy(limited, problem.getFitnessRange, new Random(seed)) val allocatedBefore = AllocationMeter.bytesAllocatedSoFar val fitness = bestFitness(strategy, initialGuess, problem.getFitnessRange) val millis = (System.nanoTime() - start) / NANOS_PER_MILLI.toDouble val allocated = for (before <- allocatedBefore; after <- AllocationMeter.bytesAllocatedSoFar) yield after - before - RunResult(name, strategyType, seed, limited.evaluations, millis, fitness, - allocated.filter(_ => !MULTI_THREADED.contains(strategyType)), limited.stoppedBy) + RunResult(benchmarkProblem.name, strategyType, seed, limited.evaluations, millis, fitness, + allocated.filter(_ => !MULTI_THREADED.contains(strategyType)), limited.stoppedBy, + solved = benchmarkProblem.isConstraint && (limited.reachedOptimal || fitness <= problem.getOptimalFitness)) } /** The best fitness found, even when the search was stopped before it finished. */ diff --git a/scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblem.scala b/scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblem.scala new file mode 100644 index 0000000..24938ac --- /dev/null +++ b/scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblem.scala @@ -0,0 +1,72 @@ +// Copyright by Barry G. Becker, 2026. Licensed under MIT License: http://www.opensource.org/licenses/MIT +package com.barrybecker4.optimization.benchmark + +import com.barrybecker4.optimization.optimizee.optimizees.OptimizeeProblem +import com.barrybecker4.optimization.parameter.types.{IntegerParameter, Parameter} +import com.barrybecker4.optimization.parameter.{NumericParameterArray, ParameterArray, ParameterArrayWithFitness} + +import scala.util.Random + +/** + * Color the nodes of a graph with a few colors so that no edge joins two nodes of the same color. The graph is made + * around a hidden coloring: the colors are chosen first and only edges between nodes of different colors are added, so + * a solution is known to exist. The fitness is the number of edges whose ends have the same color, and 0 is a solution. + * + * It is the example of a problem with integer variables rather than a permutation, and the case min-conflicts was + * made for: a node with a clashing edge can simply be given a color that clashes less. + * @param numNodes the number of nodes + * @param numColors the number of colors available + * @param averageDegree how many edges each node has on average; more makes the problem harder + * @param seed decides the graph, the hidden coloring and the initial guess + */ +class GraphColoringProblem(numNodes: Int, numColors: Int, averageDegree: Double, seed: Long) extends OptimizeeProblem { + + require(numNodes >= 2 && numColors >= 2, s"Need 2 or more nodes and colors, but got $numNodes and $numColors") + require(averageDegree > 0 && averageDegree < numNodes * (numColors - 1.0) / numColors, + s"The average degree must be positive and leave room for the edges, but got $averageDegree") + + private val random = new Random(seed) + private val hiddenColoring: IndexedSeq[Int] = IndexedSeq.fill(numNodes)(random.nextInt(numColors)) + private val edges: IndexedSeq[(Int, Int)] = createEdges() + + override def getName: String = s"Coloring $numNodes nodes with $numColors colors" + + override def evaluateByComparison: Boolean = false + + /** The number of edges that join two nodes of the same color. */ + override def evaluateFitness(coloring: ParameterArray): Double = + edges.count { case (a, b) => colorOf(coloring, a) == colorOf(coloring, b) }.toDouble + + private def colorOf(coloring: ParameterArray, node: Int): Int = coloring.get(node).getValue.toInt + + override def getInitialGuess: ParameterArray = { + val initialRandom = new Random(seed + 1) + coloringOf(IndexedSeq.fill(numNodes)(initialRandom.nextInt(numColors)), seed + 2) + } + + override def getExactSolution: ParameterArrayWithFitness = { + val coloring = coloringOf(hiddenColoring, seed) + ParameterArrayWithFitness(coloring, evaluateFitness(coloring)) + } + + /** A random coloring clashes on about one edge in `numColors`. */ + override def getFitnessRange: Double = edges.size.toDouble / numColors + + def numEdges: Int = edges.size + + private def coloringOf(colors: IndexedSeq[Int], randomSeed: Long): ParameterArray = { + val params: IndexedSeq[Parameter] = colors.indices.map(node => IntegerParameter(colors(node), 0, numColors - 1, "n" + node)) + NumericParameterArray(params, rnd = new Random(randomSeed)) + } + + /** Random pairs of nodes with different hidden colors, until there are enough edges; the same edge is added once. */ + private def createEdges(): IndexedSeq[(Int, Int)] = { + val wanted = (numNodes * averageDegree / 2).toInt + val chosen = scala.collection.mutable.LinkedHashSet.empty[(Int, Int)] + while (chosen.size < wanted) { + val (a, b) = (random.nextInt(numNodes), random.nextInt(numNodes)) + if (hiddenColoring(a) != hiddenColoring(b)) chosen += ((Math.min(a, b), Math.max(a, b))) + } + chosen.toIndexedSeq + } +} diff --git a/scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblemSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblemSuite.scala new file mode 100644 index 0000000..527dfb5 --- /dev/null +++ b/scala-test/com/barrybecker4/optimization/benchmark/GraphColoringProblemSuite.scala @@ -0,0 +1,43 @@ +// Copyright by Barry G. Becker, 2026. Licensed under MIT License: http://www.opensource.org/licenses/MIT +package com.barrybecker4.optimization.benchmark + +import org.scalatest.funsuite.AnyFunSuite + +class GraphColoringProblemSuite extends AnyFunSuite { + + test("the hidden coloring has no edge with the same color at both ends") { + (1 to 10).foreach(seed => assertResult(0.0)(new GraphColoringProblem(60, 4, 6.0, seed).getExactSolution.fitness)) + } + + test("there are as many edges as the average degree asks for") { + assertResult(300)(new GraphColoringProblem(100, 4, 6.0, 1).numEdges) + } + + test("the initial guess uses several colors and clashes about as often as chance says") { + val problem = new GraphColoringProblem(200, 4, 7.0, 2) + val guess = problem.getInitialGuess + assert((0 until 200).map(guess.get(_).getValue.toInt).toSet.size == 4) + val expected = problem.numEdges / 4.0 + assert(Math.abs(problem.evaluateFitness(guess) - expected) < expected * 0.4) + } + + test("a coloring with every node the same color clashes on every edge") { + val problem = new GraphColoringProblem(50, 3, 4.0, 3) + val guess = problem.getInitialGuess + val oneColor = (0 until 50).foldLeft(guess)((g, i) => g.asInstanceOf[com.barrybecker4.optimization.parameter.NumericParameterArray] + .copy(params = g.asInstanceOf[com.barrybecker4.optimization.parameter.NumericParameterArray].params.updated(i, + g.get(i).setValue(0.0)))) + assertResult(problem.numEdges.toDouble)(problem.evaluateFitness(oneColor)) + } + + test("the same seed gives the same graph and the same initial guess") { + def clashes(seed: Long) = { val p = new GraphColoringProblem(80, 4, 6.0, seed); (p.numEdges, p.evaluateFitness(p.getInitialGuess)) } + assertResult(clashes(5))(clashes(5)) + assert(clashes(5) != clashes(6)) + } + + test("an impossible degree or too few colors are refused") { + intercept[IllegalArgumentException](new GraphColoringProblem(10, 4, 50.0, 1)) + intercept[IllegalArgumentException](new GraphColoringProblem(10, 1, 2.0, 1)) + } +} diff --git a/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizee.scala b/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizee.scala index 66b9e90..86c5e4a 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizee.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizee.scala @@ -11,6 +11,7 @@ object LimitedOptimizee { val FINISHED = "finished" val TIME_LIMIT = "time" val EVALUATION_LIMIT = "evals" + val SOLVED = "solved" } /** @@ -20,15 +21,22 @@ object LimitedOptimizee { * @param delegate the problem being optimized * @param maxEvaluations stop after this many evaluations * @param deadlineNanos stop at this time (as in `System.nanoTime`) + * @param stopWhenSolved stop at the next evaluation after one reached the optimal fitness, for a problem where + * that is the goal and the strategy might otherwise carry on */ -class LimitedOptimizee(delegate: Optimizee, maxEvaluations: Long, deadlineNanos: Long) extends Optimizee { +class LimitedOptimizee(delegate: Optimizee, maxEvaluations: Long, deadlineNanos: Long, stopWhenSolved: Boolean = false) + extends Optimizee { import LimitedOptimizee.* private val count = new AtomicLong(0) @volatile private var reason = FINISHED + @volatile private var solved = false def evaluations: Long = count.get + /** True once an evaluation returned the optimal fitness of the problem. */ + def reachedOptimal: Boolean = solved + /** Why the search ended: FINISHED if the strategy decided to stop by itself. */ def stoppedBy: String = reason @@ -41,7 +49,7 @@ class LimitedOptimizee(delegate: Optimizee, maxEvaluations: Long, deadlineNanos: override def evaluateFitness(params: ParameterArray): Double = { checkLimits() - delegate.evaluateFitness(params) + noteIfSolved(delegate.evaluateFitness(params)) } override def compareFitness(params1: ParameterArray, params2: ParameterArray): Double = { @@ -52,12 +60,18 @@ class LimitedOptimizee(delegate: Optimizee, maxEvaluations: Long, deadlineNanos: /** A move evaluation counts as one evaluation, the same as a full one. */ override def evaluateMove(current: ParameterArray, currentFitness: Double, move: Move): Double = { checkLimits() - delegate.evaluateMove(current, currentFitness, move) + noteIfSolved(delegate.evaluateMove(current, currentFitness, move)) + } + + private def noteIfSolved(fitness: Double): Double = { + if (fitness <= delegate.getOptimalFitness) solved = true + fitness } private def checkLimits(): Unit = { if (count.incrementAndGet() > maxEvaluations) stop(EVALUATION_LIMIT) if (System.nanoTime() > deadlineNanos) stop(TIME_LIMIT) + if (stopWhenSolved && solved) stop(SOLVED) } private def stop(why: String): Nothing = { diff --git a/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizeeSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizeeSuite.scala index b97e9b9..d26d5e9 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizeeSuite.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/LimitedOptimizeeSuite.scala @@ -36,4 +36,33 @@ class LimitedOptimizeeSuite extends AnyFunSuite { assertResult((problem.evaluateByComparison, problem.getOptimalFitness, problem.getName))( (limited.evaluateByComparison, limited.getOptimalFitness, limited.getName)) } + + private val queens = new NQueensProblem(10, 1) + + test("it carries on after the problem is solved unless it is asked to stop") { + val limited = new LimitedOptimizee(queens, 100, farFuture) + limited.evaluateFitness(queens.getExactSolution.pa) + limited.evaluateFitness(queens.getInitialGuess) + assert(limited.reachedOptimal) + assertResult(LimitedOptimizee.FINISHED)(limited.stoppedBy) + } + + test("when asked to, it stops at the evaluation after the one that solved the problem") { + val limited = new LimitedOptimizee(queens, 100, farFuture, stopWhenSolved = true) + limited.evaluateFitness(queens.getInitialGuess) + assert(!limited.reachedOptimal) + limited.evaluateFitness(queens.getExactSolution.pa) + assert(limited.reachedOptimal) + intercept[EvaluationBudgetExceededException](limited.evaluateFitness(queens.getInitialGuess)) + assertResult(LimitedOptimizee.SOLVED)(limited.stoppedBy) + } + + test("a move that reaches the optimal fitness counts as solving it") { + val limited = new LimitedOptimizee(queens, 100, farFuture) + val almost = queens.getExactSolution.pa.applyMove(com.barrybecker4.optimization.parameter.Swap(0, 1)) + val broken = queens.evaluateFitness(almost) + assert(broken > 0) + limited.evaluateMove(almost, broken, com.barrybecker4.optimization.parameter.Swap(0, 1)) + assert(limited.reachedOptimal) + } } diff --git a/scala-test/com/barrybecker4/optimization/benchmark/NQueensProblem.scala b/scala-test/com/barrybecker4/optimization/benchmark/NQueensProblem.scala new file mode 100644 index 0000000..49c5b99 --- /dev/null +++ b/scala-test/com/barrybecker4/optimization/benchmark/NQueensProblem.scala @@ -0,0 +1,85 @@ +// Copyright by Barry G. Becker, 2026. Licensed under MIT License: http://www.opensource.org/licenses/MIT +package com.barrybecker4.optimization.benchmark + +import com.barrybecker4.optimization.optimizee.optimizees.OptimizeeProblem +import com.barrybecker4.optimization.parameter.types.IntegerParameter +import com.barrybecker4.optimization.parameter.{Move, ParameterArray, ParameterArrayWithFitness, PermutedParameterArray, Swap} + +import scala.util.Random + +/** + * Place n queens on an n by n board so that none attacks another. The tour is a permutation: the queen in row i + * stands in column `p(i)`, so no two queens share a row or a column, and only the diagonals can conflict. The fitness + * is the number of pairs of queens on the same diagonal, and 0 is a solution. + * + * This is the classic problem for min-conflicts and for tabu search with swaps. A swap changes the diagonals of just + * two queens, so `evaluateMove` finds the new count by looking at those two queens alone. + * @param size the number of queens; the known solution that is used as the exact one needs a size that is not 2 or 3 + * more than a multiple of 6 + * @param seed decides the order in the initial guess + */ +class NQueensProblem(size: Int, seed: Long) extends OptimizeeProblem { + + require(size >= 4 && size % 6 != 2 && size % 6 != 3, + s"The known solution is for sizes that are not 2 or 3 mod 6 (and at least 4), but got $size") + + override def getName: String = s"$size queens" + + override def evaluateByComparison: Boolean = false + + /** Count the queens on each diagonal; a diagonal with c queens has c * (c - 1) / 2 pairs. */ + override def evaluateFitness(board: ParameterArray): Double = { + val rising = new Array[Int](2 * size) + val falling = new Array[Int](2 * size) + (0 until size).foreach { row => + val column = columnOf(board, row) + rising(row + column) += 1 + falling(row - column + size) += 1 + } + (pairs(rising) + pairs(falling)).toDouble + } + + private def pairs(queensOnDiagonal: Array[Int]): Int = queensOnDiagonal.map(c => c * (c - 1) / 2).sum + + private def columnOf(board: ParameterArray, row: Int): Int = board.get(row).getValue.toInt + + /** Only the two swapped queens change which diagonals they are on, so only their conflicts are worked out again. */ + override def evaluateMove(board: ParameterArray, conflicts: Double, move: Move): Double = move match { + case Swap(first, second) if first != second => conflicts + conflictChangeOfSwap(board, first, second) + case Swap(_, _) => conflicts + case _ => super.evaluateMove(board, conflicts, move) + } + + private def conflictChangeOfSwap(board: ParameterArray, first: Int, second: Int): Int = { + val (columnA, columnB) = (columnOf(board, first), columnOf(board, second)) + val before = attacks(board, first, columnA, first, second) + attacks(board, second, columnB, first, second) + + attackIf(first, columnA, second, columnB) + val after = attacks(board, first, columnB, first, second) + attacks(board, second, columnA, first, second) + + attackIf(first, columnB, second, columnA) + after - before + } + + /** The number of queens, other than the two swapped ones, that a queen at (row, column) attacks. */ + private def attacks(board: ParameterArray, row: Int, column: Int, skipFirst: Int, skipSecond: Int): Int = + (0 until size).count(other => other != skipFirst && other != skipSecond && onSameDiagonal(row, column, other, columnOf(board, other))) + + private def attackIf(rowA: Int, columnA: Int, rowB: Int, columnB: Int): Int = + if (onSameDiagonal(rowA, columnA, rowB, columnB)) 1 else 0 + + private def onSameDiagonal(rowA: Int, columnA: Int, rowB: Int, columnB: Int): Boolean = + Math.abs(rowA - rowB) == Math.abs(columnA - columnB) + + override def getInitialGuess: ParameterArray = boardOf(new Random(seed + 1).shuffle((0 until size).toList), seed + 2) + + /** For sizes that are not 2 or 3 mod 6 the even columns followed by the odd ones is a solution. */ + override def getExactSolution: ParameterArrayWithFitness = { + val board = boardOf((1 until size by 2).toList ++ (0 until size by 2).toList, seed) + ParameterArrayWithFitness(board, evaluateFitness(board)) + } + + private def boardOf(columns: Seq[Int], randomSeed: Long): ParameterArray = + PermutedParameterArray(columns.indices.map(row => new IntegerParameter(columns(row), 0, size - 1, "q" + row)), new Random(randomSeed)) + + /** A random board has about as many attacking pairs as there are queens. */ + override def getFitnessRange: Double = size.toDouble +} diff --git a/scala-test/com/barrybecker4/optimization/benchmark/NQueensProblemSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/NQueensProblemSuite.scala new file mode 100644 index 0000000..9790d95 --- /dev/null +++ b/scala-test/com/barrybecker4/optimization/benchmark/NQueensProblemSuite.scala @@ -0,0 +1,71 @@ +// Copyright by Barry G. Becker, 2026. Licensed under MIT License: http://www.opensource.org/licenses/MIT +package com.barrybecker4.optimization.benchmark + +import com.barrybecker4.optimization.parameter.{ParameterArray, Swap} +import org.scalatest.funsuite.AnyFunSuite + +class NQueensProblemSuite extends AnyFunSuite { + + private def arrayOf(values: Seq[Int]) = com.barrybecker4.optimization.parameter.PermutedParameterArray( + values.indices.map(i => new com.barrybecker4.optimization.parameter.types.IntegerParameter(values(i), 0, values.size - 1, "p" + i)), + new scala.util.Random(1)) + + private val validSizes = Seq(4, 5, 6, 7, 10, 11, 12, 13, 100, 400) + + test("the known solution has no queens attacking each other, for every size that has one") { + validSizes.foreach { size => + assertResult(0.0, s"$size queens")(new NQueensProblem(size, 1).getExactSolution.fitness) + } + } + + test("the sizes without that solution are refused") { + Seq(2, 3, 8, 9 + 5, 3 * 6 + 2, 3 * 6 + 3).filter(n => n % 6 == 2 || n % 6 == 3 || n < 4).foreach { size => + intercept[IllegalArgumentException](new NQueensProblem(size, 1)) + } + } + + test("the initial guess puts one queen in each column, and a random board has queens attacking") { + val problem = new NQueensProblem(30, 2) + val guess = problem.getInitialGuess + assertResult((0 until 30).toList)((0 until 30).map(guess.get(_).getValue.toInt).sorted.toList) + assert(problem.evaluateFitness(guess) > 0) + } + + test("the same seed gives the same initial guess, and another seed another one") { + def values(seed: Long) = { val g = new NQueensProblem(30, seed).getInitialGuess; (0 until 30).map(g.get(_).getValue) } + assertResult(values(4))(values(4)) + assert(values(4) != values(5)) + } + + test("queens on the same diagonal are a conflict for each pair") { + val problem = new NQueensProblem(4, 1) + val onePair = arrayOf(List(0, 1, 3, 2)) + assertResult(2.0)(problem.evaluateFitness(onePair)) // 0,1 and 3,2 are both on diagonals + val allOnOneDiagonal = arrayOf(List(0, 1, 2, 3)) + assertResult(6.0)(problem.evaluateFitness(allOnOneDiagonal)) + } + + private def afterSwap(problem: NQueensProblem, board: ParameterArray, i: Int, j: Int): Double = + problem.evaluateFitness(board.applyMove(Swap(i, j))) + + test("the conflicts after a swap from evaluateMove are those of the swapped board, for every pair") { + val problem = new NQueensProblem(12, 3) + val board = problem.getInitialGuess + val start = problem.evaluateFitness(board) + for (i <- 0 until 12; j <- 0 until 12) + assertResult(afterSwap(problem, board, i, j), s"swap $i $j")(problem.evaluateMove(board, start, Swap(i, j))) + } + + test("evaluateMove stays right over a long walk of swaps") { + val problem = new NQueensProblem(22, 4) + var board = problem.getInitialGuess + var conflicts = problem.evaluateFitness(board) + val random = new scala.util.Random(1) + (1 to 300).foreach { _ => + val move = Swap(random.nextInt(22), random.nextInt(22)) + conflicts = problem.evaluateMove(board, conflicts, move) + board = board.applyMove(move) + assertResult(problem.evaluateFitness(board))(conflicts) + } + } +} diff --git a/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmark.scala b/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmark.scala index d0428f4..585ddb6 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmark.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmark.scala @@ -25,12 +25,16 @@ object OptimizationBenchmark { def main(args: Array[String]): Unit = { val config = BenchmarkConfig.parse(args.toSeq) val results = runAll(config) - val report = BenchmarkReport.document(machineDescription(config), BenchmarkReport.summarize(results)) + val report = document(config, results) println(report) config.reportFile.foreach(file => write(file, report)) config.outputFile.foreach(file => write(file, BenchmarkReport.tabSeparated(results))) } + private def document(config: BenchmarkConfig, results: Seq[RunResult]): String = + if (config.solveRate) BenchmarkReport.solveDocument(machineDescription(config), BenchmarkReport.summarizeSolves(results)) + else BenchmarkReport.document(machineDescription(config), BenchmarkReport.summarize(results)) + private def write(file: String, text: String): Unit = { val path = Paths.get(file) Option(path.getParent).foreach(Files.createDirectories(_)) diff --git a/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala index c765f42..92ff9c7 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala @@ -43,4 +43,29 @@ class OptimizationBenchmarkSuite extends AnyFunSuite { val b = BenchmarkProblems.named(Seq("tour-50")).head.create(3) assertResult(a.evaluateFitness(a.getInitialGuess))(b.evaluateFitness(b.getInitialGuess)) } + + test("the constraint problems have their own names, are marked, and are not in the ordinary list") { + assertResult(BenchmarkProblems.CONSTRAINT.size)(BenchmarkProblems.CONSTRAINT.map(_.name).distinct.size) + assert(BenchmarkProblems.CONSTRAINT.forall(_.isConstraint) && BenchmarkProblems.ALL.forall(!_.isConstraint)) + assert(BenchmarkProblems.CONSTRAINT.forall(p => !BenchmarkProblems.ALL.exists(_.name == p.name))) + } + + test("every constraint problem has a known solution that has the optimal fitness") { + BenchmarkProblems.CONSTRAINT.foreach { p => + val problem = p.create(1) + assertResult(problem.getOptimalFitness, p.name)(problem.getExactSolution.fitness) + } + } + + test("a run on a constraint problem that finds a solution is solved and stops there") { + val config = quick.copy(problemNames = Seq("queens-100"), strategies = Seq(SIMULATED_ANNEALING), timeLimitMillis = 3000, + numSeeds = 3, solveRate = true) + val results = OptimizationBenchmark.runAll(config) + assertResult(3)(results.size) + results.filter(_.solved).foreach(r => assert(r.stoppedBy == LimitedOptimizee.SOLVED || r.fitness == 0.0, r.toString)) + } + + test("an ordinary problem is never marked solved") { + assert(OptimizationBenchmark.runAll(quick).forall(!_.solved)) + } } From c216f4a90190c1a5b729b359cef51db346eb038e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 07:39:41 -0700 Subject: [PATCH 3/5] Benchmark: skip a strategy that rejects the problem when it starts Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0175jMLxrP7wZ3vfj6UfeDZx --- .../optimization/benchmark/BenchmarkRun.scala | 13 +++++++++++-- .../benchmark/OptimizationBenchmarkSuite.scala | 5 +++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala index 641a76c..c152952 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/BenchmarkRun.scala @@ -28,13 +28,22 @@ object BenchmarkRun { /** Strategies that use several threads, where the allocations of the calling thread are not the whole story. */ private val MULTI_THREADED = Set(OptimizationStrategyType.CONCURRENT_GENETIC_SEARCH) - /** @return the result, or none if the strategy cannot be used on the problem. */ + /** + * @return the result, or none if the strategy cannot be used on the problem, either because it says so up front + * or because it rejects the problem when it starts (global sampling cannot sample 200 integer parameters) + */ def run(problem: BenchmarkProblem, strategy: OptimizationStrategyType, seed: Long, config: BenchmarkConfig): Option[RunResult] = { MathUtil.RANDOM.setSeed(seed) val optimizee = problem.create(seed) val initialGuess = optimizee.getInitialGuess if (!strategy.appliesTo(optimizee, initialGuess)) None - else Some(search(problem, optimizee, initialGuess, strategy, seed, config)) + else + try Some(search(problem, optimizee, initialGuess, strategy, seed, config)) + catch { + case e: IllegalArgumentException => + Console.err.println(s"Skipping $strategy on ${problem.name}: ${e.getMessage}") + None + } } private def search(benchmarkProblem: BenchmarkProblem, problem: com.barrybecker4.optimization.optimizee.optimizees.OptimizeeProblem, diff --git a/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala b/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala index 92ff9c7..0c1c595 100644 --- a/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala +++ b/scala-test/com/barrybecker4/optimization/benchmark/OptimizationBenchmarkSuite.scala @@ -68,4 +68,9 @@ class OptimizationBenchmarkSuite extends AnyFunSuite { test("an ordinary problem is never marked solved") { assert(OptimizationBenchmark.runAll(quick).forall(!_.solved)) } + + test("a strategy that rejects the problem when it starts is left out, not a failure of the whole benchmark") { + val config = quick.copy(problemNames = Seq("coloring-200"), strategies = Seq(GLOBAL_SIMULATED_ANNEALING), solveRate = true) + assert(OptimizationBenchmark.runAll(config).isEmpty) + } } From 887c8ac9f3fa1d70731d9dac5386b5919a451dac Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 07:50:26 -0700 Subject: [PATCH 4/5] Baseline solve rate results; plan updated with what phase A found Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0175jMLxrP7wZ3vfj6UfeDZx --- docs/solve-rate-results.md | 40 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 docs/solve-rate-results.md diff --git a/docs/solve-rate-results.md b/docs/solve-rate-results.md new file mode 100644 index 0000000..70eb3e1 --- /dev/null +++ b/docs/solve-rate-results.md @@ -0,0 +1,40 @@ +# Solve rate results + +Written by `./gradlew benchmark -Pargs="--solve-rate=true"`; see [new-strategies-plan.md](new-strategies-plan.md). +Each run stops when it solves the problem, so the time is the time to a solution. The unsolved fitness is the +median number of constraints still broken in the runs that did not. Only compare runs made on the same machine. + +Java 21.0.12.1, 2 processors, 20 seeds, 2000 ms or 5000000 evaluations per run + +| Problem | Strategy | Solved | Median ms (solved) | Median evals (solved) | Median broken (unsolved) | +| --- | --- | ---: | ---: | ---: | ---: | +| queens-100 | SIMULATED_ANNEALING | 20/20 | 60 | 17,517 | - | +| queens-100 | GLOBAL_SIMULATED_ANNEALING | 20/20 | 62 | 18,800 | - | +| queens-100 | HILL_CLIMBING | 0/20 | - | - | 39.0 | +| queens-100 | GLOBAL_HILL_CLIMBING | 0/20 | - | - | 36.0 | +| queens-100 | GENETIC_SEARCH | 0/20 | - | - | 2.0 | +| queens-100 | CONCURRENT_GENETIC_SEARCH | 0/20 | - | - | 2.0 | +| queens-100 | ITERATED_LOCAL_SEARCH | 0/20 | - | - | 20.0 | +| queens-100 | VARIABLE_NEIGHBORHOOD_SEARCH | 0/20 | - | - | 20.0 | +| queens-400 | SIMULATED_ANNEALING | 20/20 | 818 | 97,017 | - | +| queens-400 | GLOBAL_SIMULATED_ANNEALING | 20/20 | 993 | 94,068 | - | +| queens-400 | HILL_CLIMBING | 0/20 | - | - | 206.0 | +| queens-400 | GLOBAL_HILL_CLIMBING | 0/20 | - | - | 199.0 | +| queens-400 | GENETIC_SEARCH | 0/20 | - | - | 155.5 | +| queens-400 | CONCURRENT_GENETIC_SEARCH | 0/20 | - | - | 155.0 | +| queens-400 | ITERATED_LOCAL_SEARCH | 0/20 | - | - | 155.0 | +| queens-400 | VARIABLE_NEIGHBORHOOD_SEARCH | 0/20 | - | - | 153.0 | +| coloring-200 | SIMULATED_ANNEALING | 0/20 | - | - | 141.0 | +| coloring-200 | HILL_CLIMBING | 0/20 | - | - | 107.5 | +| coloring-200 | GENETIC_SEARCH | 0/20 | - | - | 116.0 | +| coloring-200 | CONCURRENT_GENETIC_SEARCH | 0/20 | - | - | 115.0 | +| coloring-200 | ITERATED_LOCAL_SEARCH | 0/20 | - | - | 131.0 | +| coloring-200 | VARIABLE_NEIGHBORHOOD_SEARCH | 0/20 | - | - | 132.0 | +| all-interval-14 | SIMULATED_ANNEALING | 1/20 | 2 | 1,771 | 1.0 | +| all-interval-14 | GLOBAL_SIMULATED_ANNEALING | 1/20 | 9 | 9,764 | 1.0 | +| all-interval-14 | HILL_CLIMBING | 0/20 | - | - | 2.0 | +| all-interval-14 | GLOBAL_HILL_CLIMBING | 0/20 | - | - | 1.0 | +| all-interval-14 | GENETIC_SEARCH | 10/20 | 205 | 58,450 | 1.0 | +| all-interval-14 | CONCURRENT_GENETIC_SEARCH | 10/20 | 219 | 58,451 | 1.0 | +| all-interval-14 | ITERATED_LOCAL_SEARCH | 5/20 | 3 | 2,436 | 1.0 | +| all-interval-14 | VARIABLE_NEIGHBORHOOD_SEARCH | 4/20 | 17 | 9,609 | 1.0 | From 3a321d1fb8261611c53fd1768273951132fdf016 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 24 Oct 2026 07:50:54 -0700 Subject: [PATCH 5/5] Plan: record what phase A found and other problems to add Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0175jMLxrP7wZ3vfj6UfeDZx --- docs/new-strategies-plan.md | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/docs/new-strategies-plan.md b/docs/new-strategies-plan.md index b449fbb..5c7c319 100644 --- a/docs/new-strategies-plan.md +++ b/docs/new-strategies-plan.md @@ -1,8 +1,11 @@ # Plan: strategies for constraint problems (tabu search, min-conflicts, adaptive restarts) -Status: proposal, nothing built yet. +Status: phase A (the constraint benchmark) is done; see [solve-rate-results.md](solve-rate-results.md). Phases B to F +are not started. -Decisions already made: new APIs are additive in 2.1-SNAPSHOT, and defaults keep today's behavior. A new strategy is +Decisions already made: new APIs are additive in 2.1-SNAPSHOT, and defaults keep today's behavior. Tabu search and +min-conflicts are separate enum entries. A 20 seed solve rate is acceptable. N-queens, graph coloring and all-interval +series are the starting set of problems. A new strategy is judged on the benchmark, with solve rate first and time second (see "Does it earn its place?"). ## Why @@ -74,9 +77,33 @@ does not get worse on the others by more than the noise. A strategy that fails t and off the default benchmark list, and the plan says so. This is the same discipline as the guided moves for Tantrix, which did not pass and were left off. +## What phase A found, and other problems worth adding + +The baseline (20 seeds, 2 s each, two processors) says that annealing is already good at permutation problems with a +swap delta: it solves both N-queens sizes every time, in 60 ms and 0.8 s. Genetic search, hill climbing and iterated +local search solve none of them. Nothing solves the 200 node coloring, where the best result still has about 108 of +roughly 700 edges clashing, because a neighbor of an integer array changes every variable at once. That is the gap that +single variable moves and min-conflicts should close. The all-interval series of 14 is the one problem that genetic +search wins (10 of 20, against 1 for annealing), so it is the check that a new strategy is not tuned to queens. +Two strategies cannot run on integer arrays at all: global annealing and global hill climbing refuse 200 parameters, +because the sampling grid would be too large, so the benchmark skips them there. + +Problems I would add once the boolean and integer moves of phase B exist, because they cover cases the three do not: + +- **Random 3-SAT with a planted solution** (boolean variables, a few hundred of them, clauses at a ratio near 4.2). The + problem WalkSAT and min-conflicts were made for, and the one that tests boolean arrays. +- **Costas array or Langford pairing** (permutations with a very different constraint structure from queens, no easy + local repair). A second all-interval-like check against tuning. +- **A Latin square or Sudoku completion** (several permutations at once, with some cells fixed). It needs a way to + say that cells are fixed, so it waits until a use for that comes up; the Red puzzle in bb4-puzzles is the practical + stand-in for it. + +I did not add magic squares or the Golomb ruler: the first is well solved by the same methods as queens and the second is +an optimization problem, so neither adds information. + ## Phases -### Phase A: a way to measure (first, because everything else is judged by it) +### Phase A: a way to measure (done) - Constraint problems in `BenchmarkProblems`, each with a known solution and `getOptimalFitness` of 0: - N-queens as a permutation of 100 and 400 queens (conflicts counted on the diagonals; with an allocation free swap