perf: add SIMD integer formatting paths

[geeknoid]

Use SWAR conversion for four decimal digits and let LLVM combine repeated conversions into packed SIMD operations. Add specialized u16, u64, and u128 chunk encoders while preserving the portable and no-panic fallbacks.

Add Gungraun coverage, exhaustive parity tests, SIMD CI, and benchmark documentation.

Times are Criterion medians. Variable-workload instruction counts are Gungraun totals normalized per integer over 1,024 inputs.

Portable x86-64

Name	Instructions before	Instructions after	Time before	Time after
u64(0)	30	21	2.949 ns	2.285 ns
u64(u32::MAX)	79	79	5.766 ns	5.214 ns
i16(0)	38	30	3.038 ns	2.668 ns
i16::MIN	57	57	4.701 ns	4.609 ns
u128(0)	40	40	4.683 ns	4.569 ns
u128::MAX	275	275	24.289 ns	23.878 ns

SSE4.1 + LZCNT

Name	Instructions before	Instructions after	Time before	Time after
u64(0)	30	21	2.958 ns	2.565 ns
u64(u32::MAX)	79	64	5.272 ns	5.330 ns
i16(0)	38	20	3.441 ns	1.375 ns
i16::MIN	57	45	4.682 ns	4.653 ns
u128(0)	40	40	4.523 ns	4.602 ns
u128::MAX	274	215	24.847 ns	19.325 ns

AVX2 + BMI2

Name	Instructions before	Instructions after	Time before	Time after
u64(0)	29	20	3.109 ns	2.797 ns
u64(u32::MAX)	76	66	5.966 ns	5.247 ns
i16(0)	36	19	3.898 ns	1.332 ns
i16::MIN	56	43	4.817 ns	4.284 ns
u128(0)	39	39	4.629 ns	4.605 ns
u128::MAX	249	196	26.900 ns	19.935 ns

[dtolnay]

Thanks!

[dtolnay]

I tried running this on my machine (Linux, x86_64, AMD Ryzen Threadripper 9975WX) and it crashes like this on the second test. Any suggestions how to debug this?

$ valgrind --version
valgrind-3.25.1

$ gungraun-runner --version
gungraun-runner 0.18.2

$ RUSTFLAGS="-C target-cpu=native" cargo +1.97.1 bench --bench gungraun
...
     Running benches/gungraun.rs (target/release/deps/gungraun-02e34c357121e992)
gungraun::benches::itoa_u64 zero:(0)
  Instructions:                          20|N/A                  (*********)
  L1 Hits:                               26|N/A                  (*********)
  LL Hits:                                1|N/A                  (*********)
  RAM Hits:                               3|N/A                  (*********)
  Total read+write:                      30|N/A                  (*********)
  Estimated Cycles:                     136|N/A                  (*********)
gungraun::benches::itoa_u64 half:(u64 :: from(u32 :: MAX))
gungraun_runner: Error: Error in task: Error running 'callgrind': Terminated by a signal '4'
error: bench failed, to rerun pass `--bench gungraun`

Caused by:
  process didn't exit successfully: `./target/release/deps/gungraun-02e34c357121e992 --bench` (exit status: 1)

[geeknoid]

I've never used the lower level tooling, all I do is:

cargo install --version 0.18.2 gungraun-runner
cargo bench --bench gungraun

I also run this on Linux with some AMD processor.

[dtolnay]

I am able to run both of those, just not this command in the readme.

I tried a different machine running a different Linux distribution and a different valgrind version (3.22.0) and a different processor (AMD Turin / EPYC 9005 series 158-core) and that also reproduces the same crash.

[geeknoid]

Thanks for the detailed repro — I can reproduce it now and tracked down the root cause.

It's not a miscompilation: with  -C target-cpu=native  on an AVX-512-capable CPU (Zen 4/5), LLVM autovectorizes the repeated four-digit conversions into EVEX-encoded AVX-512 instructions (the faulting one is a  vpmulld inside  ::fmt ). The code runs fine on real hardware, but Valgrind can't decode AVX-512, so Gungraun aborts with  SIGILL . That's why the two commands you ran work (Gungraun without  native  never enables the SIMD path) but the README command doesn't.

Since the optimized path is gated only on  sse4.1  +  lzcnt , I've updated the README to recommend  RUSTFLAGS="-C target-feature=+sse4.1,+lzcnt"  instead of  target-cpu=native . That exercises exactly the same optimized code, emits no AVX-512, and runs cleanly under Valgrind.

I confirmed a  +sse4.1,+lzcnt  build contains zero AVX-512 instructions and completes under  valgrind .

[geeknoid]

@dtolnay Thanks for your feedback. The commit has been updated and should be ready for another look.

[dtolnay]

If this test reveals panics in some #[no_panic] function, please fix the panics instead of excluding this test from panic checking.

Same for the other tests also.

[geeknoid]

I've removed the  #[cfg(not(feature = "no-panic"))]  exclusions from all of these tests, and they did reveal real panic paths, so I fixed those rather than hiding them.

The paths turned out to be pre-existing (only ever hidden because the old tests used compile-time constants, which let LLVM const-fold the panic paths away; the generic  check  helper passes runtime values and keeps them). I made the affected functions provably panic-free:

• signed  write : replaced  (&mut buf[offset..]).try_into().unwrap()  with the same pointer-cast pattern used in  Buffer::format , and the sign-byte write with  get_unchecked_mut

•  impl_Unsigned!  /  u128::fmt  /  enc_16lsd  /  enc_7msd : switched the provably in-bounds  buf[offset + N] writes to  get_unchecked_mut , each with a SAFETY comment; the u8  .expect()  conversions became non-panicking  .unwrap_or(...)
  Verified:  --features no-panic  now builds with the tests enabled (both portable and  +sse4.1,+lzcnt ), the full suite passes on debug/release/ --no-default-features /SIMD, Miri passes with  -Zmiri-strict-provenance , and clippy pedantic is clean.  powers_of_ten_boundaries  runs under Miri and exercises the new signed pointer-cast; the two heavy exhaustive tests keep  #[cfg_attr(miri, ignore)] .

[dtolnay]

Could you confirm whether to_bcd4 and the logic in this function is newly developed for this PR or is copied from somewhere / based on an existing algorithm in a different library?

[geeknoid]

Good question — it's not newly developed. It's adapted from existing code:

The immediate source is your own itoa-benchmark  repo. The  to_bcd4  function here is copied byte-for-byte from  src/bcd4.rs , and the  u16 / u64  chunk encoders are a near-verbatim adaptation of its  u64toa_bcd4  (same  bcd_top / bcd_hi_hi / bcd_hi_lo / bcd_lo_hi / bcd_lo_lo  split, same  | 0x30303030 , same  leading_zeros() / 8 ). The only change is writing into the  MaybeUninit  buffer via unaligned stores instead of returning a  &str :
https://github.com/dtolnay/itoa-benchmark/blob/master/src/bcd4.rs#L1-L6

The underlying algorithm appears to be xjb714's "xjb" dec-to-ASCII / BCD routine (Apache-2.0), where the magic constants come from —  0x147b  is the divide-by-100 multiplier ( #define DIV_100 0x147b ) and  0x67  the divide-by-10:
https://github.com/xjb714/xjb/blob/main/bench/xjb/other/dec_to_bcd.cpp#L118,
https://github.com/xjb714/xjb/blob/main/bench/xjb/other/dec_to_bcd.cpp#L188

Licensing: the whole lineage is Apache-2.0 ( xjb714/xjb  and  itoa-benchmark  are both Apache-2.0). Since  itoa is  MIT OR Apache-2.0 , this fits the Apache-2.0 arm.

[geeknoid]

@dtolnay Sorry for the delay, I lost track of this PR :-) I hope I've addressed your feedback correctly, please let me know if you'd like more changes.

Thanks.