TOML: lookup table for ASCII string categorization; cheaper writeRaw() variants

[pjfanning]

Two small hot-path improvements in the TOML generator.

StringOutputUtil.categorize() — the per-character category (unquoted-key / literal / basic / basic-no-escape / ASCII-only) was computed via a chain of range checks for every character of every String value and key. ASCII categories are now pre-computed into a 128-entry int[] at class init, from the same logic (which stays in place, as _categorize(), for non-ASCII, surrogates and BOM). The String and char[] scanning loops take the ASCII branch before the surrogate check, so the common case is a load-and-AND per char.

TomlGenerator.writeRaw() — writeRaw(String, int, int) allocated via substring(); it now does getChars() straight into the output buffer, only falling back to the String path when the range is longer than the buffer. writeRaw(SerializableString) went through toString(); now getValue(). (3.x's SerializableString has no unquoted char[] accessor — asQuotedChars() is JSON-escaped — so the String overload is the right one.)

Tests: StringOutputUtilTest gains a consistency check that int, String and char[] categorization agree for every BMP char plus surrogate-pair / lone-surrogate cases, and a per-String AND-of-chars check; the existing exhaustive all-code-points write/read test now runs through the table for ASCII. TomlGeneratorTest gains writeRaw overload coverage including a >2000-char range. Full TOML suite passes.

While here I noticed a pre-existing bug in _writeStringImpl(int, char[], int, int) (text[offset + len] instead of text[offset + i] in the escaped-basic-string branch). Not touched in this PR — separate fix incoming against 2.18.

🤖 Generated with Claude Code

[github-actions]

🧪 Code Coverage Report

Metric	Coverage	Change
Instructions	79.18%	📈 +0.380%
Branches	73.11%	📈 +0.410%

Coverage data generated from JaCoCo test results

[yawkat]

Is this actually faster in practice?

[pjfanning]

Before/after numbers (JMH 1.37, JDK 17.0.19, 3 forks × 5 warmup + 5 measurement iterations). "before" = 3.x at b5b62c9, "after" = this PR.

1. StringOutputUtil.categorize() in isolation (the actual change; benchmark lives in the tools.jackson.dataformat.toml package to reach the package-private method)

Input	chars	before (ns/op)	after (ns/op)	Δ
asciiKey — property_name_17	16	28.4 ± 1.2	22.7 ± 3.4	−20%
asciiValue — plain sentence	39	78.4 ± 9.3	38.8 ± 2.5	−50%
asciiEscapes — quotes + tab	30	70.3 ± 6.8	34.9 ± 4.1	−50%
asciiValue via char[] overload	39	119.1 ± 6.1	32.1 ± 3.6	−73%
nonAscii — café naïve 日本語 value 17	24	70.7 ± 4.4	48.0 ± 2.5	−32%

Roughly 2 ns/char → 1 ns/char for ASCII. Non-ASCII input also improves (the ASCII chars in it take the table path; the others pay one extra compare before the unchanged full logic). The old char[] overload was oddly slower than the String one; now they're on par.

2. End-to-end generator (TomlMapper.writeValueAsString of a 20-table × 25-key document, ~40-char string values; -prof gc)

Benchmark		Time (µs/op)	Alloc (B/op)
writeAscii	before	137.5 ± 27.5 (rerun: 134.6 ± 10.7)	31,489
	after	122.4 ± 23.6 (rerun: 121.1 ± 24.2)	31,489
	Δ	~−10% (consistent across two runs, CIs overlap)	0
writeMixed (25% non-ASCII, 25% needing escapes)	before	131.4 ± 10.3 (rerun: 154.7 ± 13.0)	91,513
	after	142.1 ± 9.7 (rerun: 158.5 ± 12.1)	91,513
	Δ	within noise	0
writeRawRanges — 500 × writeRaw(String, off, len)	before	20.5 ± 1.8	40,544
	after	15.2 ± 1.9	12,512
	Δ	−26%	−69% (the substring() per call: 500 × 56 B = 28 KB)

End-to-end, categorization is a minority of generator time (map iteration, key-path StringBuilder, escaping and buffer writes dominate), so the ~10% on ASCII documents is about what the micro numbers predict; writeMixed is dominated by the escape path (getBasicStringEscape per char, unchanged here) and the difference between runs is machine noise — the categorize micro shows the non-ASCII case is faster, not slower. Allocation is unchanged for the generator paths, as expected: this change is CPU-only apart from the writeRaw range fix.

Benchmark source

// package tools.jackson.dataformat.toml
@BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark) @Fork(3) @Warmup(iterations = 5, time = 1) @Measurement(iterations = 5, time = 1)
public class CategorizeBench {
    String asciiKey = "property_name_17";
    String asciiValue = "plain ascii value number 17 for table 3";
    String asciiEscapes = "value with \"quotes\" and a\ttab";
    String nonAscii = "café naïve 日本語 value 17";
    char[] asciiValueChars = asciiValue.toCharArray();

    @Benchmark public int asciiKey() { return StringOutputUtil.categorize(asciiKey); }
    @Benchmark public int asciiValue() { return StringOutputUtil.categorize(asciiValue); }
    @Benchmark public int asciiEscapes() { return StringOutputUtil.categorize(asciiEscapes); }
    @Benchmark public int nonAscii() { return StringOutputUtil.categorize(nonAscii); }
    @Benchmark public int asciiValueCharArray() { return StringOutputUtil.categorize(asciiValueChars, 0, asciiValueChars.length); }
}

@BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark) @Fork(3) @Warmup(iterations = 5, time = 1) @Measurement(iterations = 5, time = 1)
public class TomlWriteBench {
    static final int TABLES = 20, KEYS_PER_TABLE = 25;
    TomlMapper mapper; Map<String, Object> asciiDoc, mixedDoc;

    @Setup public void setup() { mapper = TomlMapper.builder().build(); asciiDoc = doc(false); mixedDoc = doc(true); }

    static Map<String, Object> doc(boolean mixed) {
        Map<String, Object> root = new LinkedHashMap<>();
        for (int t = 0; t < TABLES; t++) {
            Map<String, Object> table = new LinkedHashMap<>();
            for (int k = 0; k < KEYS_PER_TABLE; k++) {
                String value;
                switch ((t * KEYS_PER_TABLE + k) % 4) {
                case 0: value = "plain ascii value number " + k + " for table " + t; break;
                case 1: value = "value with 'apostrophes' inside, item " + k; break;
                case 2: value = mixed ? "café naïve 日本語 value " + k : "value with \"double quotes\" inside " + k; break;
                default: value = mixed ? "tab\there and newline\nthere " + k : "just-a-plain-slug-" + t + "-" + k; break;
                }
                table.put("property_" + t + "_" + k, value);
            }
            root.put("table_" + t, table);
        }
        return root;
    }

    @Benchmark public int writeAscii() throws Exception { return mapper.writeValueAsString(asciiDoc).length(); }
    @Benchmark public int writeMixed() throws Exception { return mapper.writeValueAsString(mixedDoc).length(); }
    @Benchmark public int writeRawRanges() throws Exception {
        StringWriter w = new StringWriter(); String line = "xxkey = 'value'yy";
        try (JsonGenerator g = mapper.createGenerator(w)) {
            for (int i = 0; i < 500; i++) { g.writeRaw(line, 2, line.length() - 4); g.writeRaw('\n'); }
        }
        return w.getBuffer().length();
    }
}