Calibrate benchmarks against Library Checker problems

[shakayami]

Summary

Completely redesigned the benchmark suite to predict real-world performance by using actual Library Checker problems as workloads, rather than synthetic query mixes. This addresses systematic disagreements between benchmark results and judge performance.

Key Changes

• Workload redesign: Replaced synthetic benchmarks with transcriptions of actual Library Checker problems:

  • point_add_range_sum (fenwicktree) - Point Add Range Sum problem
  • staticrmq (segtree) - Static RMQ problem
  • point_set_range_composite (segtree) - Point Set Range Composite problem
  • range_affine_range_sum (lazysegtree) - Range Affine Range Sum problem
  • unionfind (dsu) - Unionfind problem
  • scc (scc) - Strongly Connected Components problem
  • two_sat (two_sat) - 2 SAT problem
  • bipartitematching (maxflow) - Matching on Bipartite Graph problem
  • assignment (mincostflow) - Assignment problem
  • Plus polynomial, string, and number theory workloads from Library Checker

• Scaling system: Introduced ACL_BENCH_SCALE environment variable and per-workload COST_FACTOR to normalize runtime across different problem complexities while staying within CI budgets. Each workload targets 1-4 seconds at default 0.2 scale.

• Improved comparison logic (compare.py):

  • Support for multiple time measurement files (interleaved passes for better noise estimation)
  • Uses minimum time across passes (not mean) to filter out CI runner interference
  • Noise-aware delta reporting: only flags changes larger than observed run-to-run variation
  • Minimum significance threshold (3%) to avoid reporting noise as improvements
  • Links benchmark names to their Library Checker problems

• Comprehensive test coverage (test_benchmark_workloads.py, test_benchmark_compare.py):

  • Correctness tests for all workload implementations against naive references
  • Tests for comparison report thresholds and noise handling
  • Validates that benchmarks actually solve the problems they claim to

• Documentation: Added extensive docstrings explaining why each design choice matters for predictive accuracy, plus benchmarks/README.md with problem mappings and historical context.

• CI workflow updates (benchmark.yml):

  • Interleaved base/PR measurement passes for better noise characterization
  • Environment variables for scale and round configuration
  • Increased timeout to 90 minutes for full-scale runs

Notable Implementation Details

• _uniform_pair(): Implements Library Checker's Random::uniform_pair distribution for query ranges, which produces mean width of ~n/3 (more realistic than uniform random).

• Workload class: Unified interface with metadata (problem URL, full_size, effective_scale) while maintaining backward compatibility with existing (name, build, run) tuple consumers.

• Operator realism: Benchmarks use actual Python operators (e.g., affine map composition in Point Set Range Composite) rather than C builtins, so overhead measurements match real submissions.

• Answer accumulation: All run_* functions accumulate results rather than discarding them, matching how real submissions work and preventing dead-code elimination.

• Noise estimation: spread() function measures relative variation between passes to set per-benchmark significance thresholds, preventing false positives from CI runner variance.

https://claude.ai/code/session_0178GKMBABNdFEX6E1PcXR4n

[github-actions]

📊 Performance Benchmark Results

Workloads ran at ACL_BENCH_SCALE=0.2.

⏱️ Library Checker workloads (best of N, lower is better)

Benchmark	Base	PR	Delta	Noise
assignment	n/a	3.062s	n/a	n/a
bipartitematching	n/a	3.720s	n/a	n/a
convolution_mod	n/a	2.199s	n/a	n/a
exp_of_formal_power_series	n/a	2.896s	n/a	n/a
factorize	n/a	122.97ms	n/a	n/a
inv_of_formal_power_series	n/a	3.435s	n/a	n/a
log_of_formal_power_series	n/a	2.853s	n/a	n/a
number_of_substrings	n/a	1.698s	n/a	n/a
point_add_range_sum	n/a	1.088s	n/a	n/a
point_set_range_composite	n/a	2.013s	n/a	n/a
primality_test	n/a	361.33ms	n/a	n/a
range_affine_range_sum	n/a	2.353s	n/a	n/a
scc	399.03ms	1.462s	+266.5% ⚠️	±13.4%
staticrmq	n/a	1.586s	n/a	n/a
suffixarray	n/a	1.364s	n/a	n/a
sum_of_floor_of_linear	n/a	827.46ms	n/a	n/a
two_sat	1.543s	1.515s	-1.8% ≈	±10.5%
unionfind	n/a	111.26ms	n/a	n/a
zalgorithm	n/a	170.23ms	n/a	n/a

🧪 API coverage workloads (not judge-calibrated)

Benchmark	Base	PR	Delta	Noise
extra_acl_math_crt	n/a	856.90ms	n/a	n/a
extra_dsu_groups	n/a	332.42ms	n/a	n/a
extra_maxflow_edges	n/a	186.48ms	n/a	n/a
extra_mincostflow_slope	n/a	1.721s	n/a	n/a
extra_prime_fact_divisors	n/a	1.521s	n/a	n/a
extra_segtree_binary_search	n/a	178.88ms	n/a	n/a

📏 Workload sizes

Benchmark	Library Checker max	Ran at	Note
assignment	N = 500	50%	O(N^3)-ish in CPython; ACL_BENCH_SCALE=1 is far past the judge's TL
bipartitematching	L = R = 100,000, M = 200,000	3.0%
convolution_mod	N = M = 524,288	20%
exp_of_formal_power_series	N = 500,000	10%
factorize	Q = 100, A <= 10^18	100%
inv_of_formal_power_series	N = 500,000	20%
log_of_formal_power_series	N = 500,000	10%
number_of_substrings	N = 500,000	100%
point_add_range_sum	N = Q = 500,000	60%
point_set_range_composite	N = Q = 500,000	40%	segtree with a Python-level op; measures op-dominated cost
primality_test	Q = 100,000, N <= 10^18	100%
range_affine_range_sum	N = Q = 500,000	10%
scc	N = M = 500,000	100%
staticrmq	N = Q = 500,000	60%	segtree with a C-level op (min); measures tree overhead
suffixarray	N = 500,000	100%
sum_of_floor_of_linear	T = 100,000	100%
two_sat	N = M = 500,000	60%
unionfind	N = Q = 200,000	100%
zalgorithm	N = 500,000	100%
extra_acl_math_crt	Q = 100,000	100%	inv_mod / inv_gcd / crt
extra_dsu_groups	N = 200,000	100%	size / leader / groups
extra_maxflow_edges	N = 20,000, M = 60,000	100%	get_edge / change_edge / edges / min_cut
extra_mincostflow_slope	N = 2,000, M = 8,000	100%	slope / get_edge / edges
extra_prime_fact_divisors	Q = 10,000	20%	divisors / totient / lcm
extra_segtree_binary_search	N = Q = 200,000	20%	max_right / min_left

💾 Peak memory (lower is better)

Benchmark	Base	PR	Delta
assignment	n/a	32.8MiB	n/a
bipartitematching	n/a	4.3MiB	n/a
convolution_mod	n/a	30.2MiB	n/a
exp_of_formal_power_series	n/a	21.7MiB	n/a
extra_acl_math_crt	n/a	484.0B	n/a
extra_dsu_groups	n/a	30.5MiB	n/a
extra_maxflow_edges	n/a	29.4MiB	n/a
extra_mincostflow_slope	n/a	5.5MiB	n/a
extra_prime_fact_divisors	n/a	8.6KiB	n/a
extra_segtree_binary_search	n/a	2.3MiB	n/a
factorize	n/a	836.0B	n/a
inv_of_formal_power_series	n/a	21.4MiB	n/a
log_of_formal_power_series	n/a	24.6MiB	n/a
number_of_substrings	n/a	82.5MiB	n/a
point_add_range_sum	n/a	12.1MiB	n/a
point_set_range_composite	n/a	34.9MiB	n/a
primality_test	n/a	804.0B	n/a
range_affine_range_sum	n/a	14.0MiB	n/a
scc	25.1MiB	139.5MiB	+456.4% ⚠️
staticrmq	n/a	8.1MiB	n/a
suffixarray	n/a	82.5MiB	n/a
sum_of_floor_of_linear	n/a	516.0B	n/a
two_sat	174.0MiB	169.8MiB	-2.4% ✅
unionfind	n/a	1.7MiB	n/a
zalgorithm	n/a	3.8MiB	n/a

_{Only the workloads whose module the change touches are measured; a scheduled sweep covers the rest. Each workload mirrors the linked Library Checker problem: same query mix, same operators, same value ranges as that problem's max_random generator. Times are the best observation across interleaved passes; Noise is the spread between passes. ✅ improvement / ⚠️ regression are only shown when the delta exceeds both 3% and the measured noise; ≈ means the difference is not distinguishable from noise. API-coverage workloads exist to catch regressions in methods no judge problem exercises — do not quote their timings as speedups. "n/a" rows mean one side has no such benchmark.}