Repository navigation
Repeated @btime with different setup behaves bad #403
Description
function sumN(N)
s = zero(N)
for i in 1:N
s += i
end
s
end
julia> @btime sumN(N) setup=(N=1_000_000)
2.424 ns (0 allocations: 0 bytes)
500000500000
julia> @btime sumN(N) setup=(N=1_000_000e0)
2.424 ns (0 allocations: 0 bytes)
500000500000
julia> @btime sumN(N) setup=(N=1_000_001e0)
1.330 ms (0 allocations: 0 bytes)
5.00001500001e11
julia> @btime sumN(N) setup=(N=1_000_001)
1.330 ms (0 allocations: 0 bytes)
5.00001500001e11
julia> @btime sumN(M) setup=(M=1_000_001)
1.993 ns (0 allocations: 0 bytes)
500001500001The Float64 trial should run slowly, and return a Float64, whereas the Int trial should run in a couple of ns and return an Int. Is there some caching of the setup going on, which uses == instead of ===?
Activity
vchuravy commentedon Oct 9, 2025
There is a cache here
BenchmarkTools.jl/src/execution.jl
Lines 557 to 558 in 8b64635
sgaure commentedon Oct 9, 2025
Ah, that explains it, I think.
samplefunc_keyis aVectorcontainingExprs,samplefunc_cacheis aDict, andExpr(:tuple, 1) == Expr(:tuple, 1e0). Making it anIdDictwould defeat its purpose sinceExpr(:tuple, 1) !== Expr(:tuple, 1). Changing this behaviour requires some more coding which probably isn't worth it.vchuravy commentedon Oct 9, 2025
I wonder if we should make the key a tuple instead of a vector.
vtjnash commentedon Oct 9, 2025
This is probably fixed in v1.12
ederag commentedon Mar 16, 2026
Thanks @sgaure for filing this issue with a reproducer ! I spent some time chasing a similar case.
Similar output with
Julia Version 1.12.5.