Re-run benchmarks with recent Go version

[pascaldekloe]

Go version 1.11 includes cl47152 and cl98440. The changes when compared to Go version 1.10 are significant.

benchmark                                  old ns/op     new ns/op     delta
BenchmarkDecodeStdStructMedium-12          32986         26947         -18.31%
BenchmarkEncodeStdStructMedium-12          1569          1192          -24.03%
BenchmarkEncodingJsonStructSmall-12        5451          3699          -32.14%

benchmark                                  old allocs     new allocs     delta
BenchmarkDecodeStdStructMedium-12          99             8              -91.92%
BenchmarkEncodeStdStructMedium-12          4              2              -50.00%
BenchmarkEncodingJsonStructSmall-12        31             6              -80.65%

[codesoap]

I have checked a recent version of go 1.14 (commit ebe49b2c2999a7d4128c44aed9602a69fdc53d16, 2020-02-22) and used benchstat to compare against the current version of jsoniter. I have modified the benchmarks a bit, because the benchmarks in this repo don't fully parse the JSON (incomplete structs). You can find the benchmark I used here.

"old" is encoding/json, "new" is github.com/json-iterator/go (using ConfigCompatibleWithStandardLibrary):

name                      old time/op    new time/op    delta
UnmarshalSmall-2            24.9µs ± 1%     4.9µs ± 2%    -80.23%  (p=0.000 n=10+9)
UnmarshalSmallOnlyId-2      15.4µs ± 1%     4.3µs ± 4%    -71.81%  (p=0.000 n=10+10)
UnmarshalMedium-2            217µs ± 1%      52µs ± 1%    -76.06%  (p=0.000 n=10+10)
UnmarshalMediumOnlyIds-2     135µs ± 1%      33µs ± 1%    -75.86%  (p=0.000 n=8+9)
UnmarshalLarge-2            2.87ms ± 1%    0.80ms ± 2%    -72.04%  (p=0.000 n=10+10)
UnmarshalLargeOnlyIds-2     1.64ms ± 0%    0.50ms ± 1%    -69.27%  (p=0.000 n=9+9)

name                      old alloc/op   new alloc/op   delta
UnmarshalSmall-2              288B ± 0%       80B ± 0%    -72.22%  (p=0.000 n=10+10)
UnmarshalSmallOnlyId-2        248B ± 0%       48B ± 0%    -80.65%  (p=0.000 n=10+10)
UnmarshalMedium-2           1.46kB ± 0%    1.25kB ± 0%    -14.29%  (p=0.000 n=10+10)
UnmarshalMediumOnlyIds-2      392B ± 0%      400B ± 0%     +2.04%  (p=0.000 n=10+10)
UnmarshalLarge-2            13.4kB ± 0%    22.5kB ± 0%    +66.99%  (p=0.000 n=10+8)
UnmarshalLargeOnlyIds-2       402B ± 0%    11937B ± 0%  +2869.40%  (p=0.000 n=10+10)

name                      old allocs/op  new allocs/op  delta
UnmarshalSmall-2              8.00 ± 0%      3.00 ± 0%    -62.50%  (p=0.000 n=10+10)
UnmarshalSmallOnlyId-2        4.00 ± 0%      1.00 ± 0%    -75.00%  (p=0.000 n=10+10)
UnmarshalMedium-2             49.0 ± 0%      64.0 ± 0%    +30.61%  (p=0.000 n=10+10)
UnmarshalMediumOnlyIds-2      8.00 ± 0%     51.00 ± 0%   +537.50%  (p=0.000 n=10+10)
UnmarshalLarge-2               536 ± 0%      1247 ± 0%   +132.65%  (p=0.000 n=10+10)
UnmarshalLargeOnlyIds-2       35.0 ± 0%    1035.0 ± 0%  +2857.14%  (p=0.000 n=10+10)

[pascaldekloe]

Data initialization (var) in the loop would get us more realistic numbers I think.

[pascaldekloe]

Can you even reuse a structure like that? Does Unmarshal clear/zero omitted fields for example?

[codesoap]

I'm not sure about the zeroing of omitted fields in consecutive calls of Unmarshal with the same variable. I think, however, that reusing a variable is a realistic use case, since the test-suit of encoding/json includes tests for both scenarios (BenchmarkCodeUnmarshal and BenchmarkCodeUnmarshalReuse).

Here is the same comparison, when re-initializing the target every iteration (you can find the code here):

name                      old time/op    new time/op    delta
UnmarshalSmall-2            25.6µs ± 1%     5.9µs ± 1%    -77.10%  (p=0.000 n=9+9)
UnmarshalSmallOnlyId-2      15.7µs ± 1%     4.5µs ± 2%    -71.07%  (p=0.000 n=10+10)
UnmarshalMedium-2            245µs ± 1%      64µs ± 2%    -73.85%  (p=0.000 n=10+10)
UnmarshalMediumOnlyIds-2     135µs ± 3%      33µs ± 1%    -75.47%  (p=0.000 n=9+10)
UnmarshalLarge-2            3.27ms ± 1%    0.96ms ± 1%    -70.57%  (p=0.000 n=10+10)
UnmarshalLargeOnlyIds-2     1.71ms ± 1%    0.52ms ± 1%    -69.24%  (p=0.000 n=8+10)

name                      old alloc/op   new alloc/op   delta
UnmarshalSmall-2              400B ± 0%      192B ± 0%    -52.00%  (p=0.000 n=10+10)
UnmarshalSmallOnlyId-2        264B ± 0%       64B ± 0%    -75.76%  (p=0.000 n=10+10)
UnmarshalMedium-2           2.50kB ± 0%    2.23kB ± 0%    -10.58%  (p=0.000 n=10+10)
UnmarshalMediumOnlyIds-2      416B ± 0%      424B ± 0%     +1.92%  (p=0.000 n=10+10)
UnmarshalLarge-2            30.1kB ± 0%    36.5kB ± 0%    +20.97%  (p=0.000 n=10+10)
UnmarshalLargeOnlyIds-2     2.09kB ± 0%   12.73kB ± 0%   +509.58%  (p=0.000 n=10+9)

name                      old allocs/op  new allocs/op  delta
UnmarshalSmall-2              9.00 ± 0%      4.00 ± 0%    -55.56%  (p=0.000 n=10+10)
UnmarshalSmallOnlyId-2        5.00 ± 0%      2.00 ± 0%    -60.00%  (p=0.000 n=10+10)
UnmarshalMedium-2             67.0 ± 0%      81.0 ± 0%    +20.90%  (p=0.000 n=10+10)
UnmarshalMediumOnlyIds-2      10.0 ± 0%      53.0 ± 0%   +430.00%  (p=0.000 n=10+10)
UnmarshalLarge-2               751 ± 0%      1439 ± 0%    +91.61%  (p=0.000 n=10+10)
UnmarshalLargeOnlyIds-2       82.0 ± 0%    1074.0 ± 0%  +1209.76%  (p=0.000 n=10+10)

[win5do]

go1.19.13, Considering GC factors, jsoniter is no longer as good as encoding/json.

bench code: https://github.com/win5do/play-go/blob/0c79bdc76dd37be3db9fc69df20c2a922a11ae49/bench/json/bench_test.go

goos: darwin
goarch: arm64
pkg: play-go/bench/json
          │     std     │                 jtr                 │
          │   sec/op    │   sec/op     vs base                │
Unmarshal   1.954µ ± 4%   1.416µ ± 1%  -27.54% (p=0.000 n=10)

          │     std      │                 jtr                 │
          │     B/op     │     B/op      vs base               │
Unmarshal   1.376Ki ± 0%   1.360Ki ± 0%  -1.14% (p=0.000 n=10)

          │    std     │                jtr                 │
          │ allocs/op  │ allocs/op   vs base                │
Unmarshal   29.00 ± 0%   34.00 ± 0%  +17.24% (p=0.000 n=10)
---
goos: darwin
goarch: arm64
pkg: play-go/bench/json
          │     std     │                 jtr                 │
          │   sec/op    │   sec/op     vs base                │
Unmarshal   17.01µ ± 1%   11.22µ ± 2%  -34.03% (p=0.000 n=10)

          │     std      │                 jtr                 │
          │     B/op     │     B/op      vs base               │
Unmarshal   10.10Ki ± 0%   10.99Ki ± 0%  +8.86% (p=0.000 n=10)

          │    std     │                jtr                 │
          │ allocs/op  │ allocs/op   vs base                │
Unmarshal   205.0 ± 0%   268.0 ± 0%  +30.73% (p=0.000 n=10)
---
goos: darwin
goarch: arm64
pkg: play-go/bench/json
          │     std     │                 jtr                 │
          │   sec/op    │   sec/op     vs base                │
Unmarshal   241.3µ ± 1%   192.6µ ± 2%  -20.19% (p=0.000 n=10)

          │     std      │                 jtr                 │
          │     B/op     │     B/op      vs base               │
Unmarshal   204.5Ki ± 0%   223.4Ki ± 0%  +9.28% (p=0.000 n=10)

          │     std     │                 jtr                 │
          │  allocs/op  │  allocs/op   vs base                │
Unmarshal   2.783k ± 0%   3.901k ± 0%  +40.17% (p=0.000 n=10)