Skip to content

Speed up UUID inserts in the Python codec - #1152

Merged
joe-clickhouse merged 7 commits into
ClickHouse:mainfrom
sebastianbreguel:speed-up-uuid-inserts
Nov 17, 2026
Merged

joe-clickhouse merged 7 commits into
ClickHouse:mainfrom
sebastianbreguel:speed-up-uuid-inserts

Conversation

@sebastianbreguel

@sebastianbreguel sebastianbreguel commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Python codec writes a UUID insert column one value at a time. For every value it does two to_bytes(8, "little") calls, one concatenation and one bytearray append:

x = v.int
dest += (x >> 64).to_bytes(8, "little") + (x & 0xFFFFFFFFFFFFFFFF).to_bytes(8, "little")

This change builds the big-endian 16 byte value for each row in a list comprehension (v.bytes, x.to_bytes(16, "big"), or int(v.replace("-", ""), 16).to_bytes(16, "big") for strings), joins them once, loads the result into array.array("Q") and calls byteswap(). Reversing each 8 byte half of the big-endian value gives exactly the wire layout, two little-endian UInt64 with the high half first. byteswap() reverses the raw bytes of each element, so the result is the same on little- and big-endian hosts.

  • uuid.UUID, int and str inputs use the new path. Falsy values (None, 0, "") still write 16 zero bytes.
  • Error behavior is unchanged: negative ints and values of 2**128 or more raise OverflowError, non-hex strings raise ValueError, short hex strings such as "1234" are still zero-padded. The int branch calls int.to_bytes(x, ...) unbound so a column like [13, "user_1"] still raises TypeError, as x >> 64 did.
  • The bytes/bytearray/memoryview branch is unchanged on purpose. A short bytes value there currently writes fewer than 16 bytes, and joining would turn that into a ValueError or a shifted column, so changing it is a behavior change that does not belong in a perf PR.
  • Only the Python codec changes. native_codec="rust" inserts reach this method only when they fall back to the Python encoder.

Performance

Serialization only, get_from_name("UUID").write_column(values, bytearray(), ctx) on 500k random UUIDs, median of 9 runs, main -> this PR:

Input main this PR
uuid.UUID 50.2 ms 28.8 ms
int 48.2 ms 23.9 ms
str 102.4 ms 76.3 ms

End to end, client.insert of 500k rows into (id UInt64, u UUID, nu Nullable(UUID)) ENGINE MergeTree, a quarter of nu NULL, median of 7 runs:

Input main this PR
uuid.UUID 165 ms 128 ms
int 161 ms 130 ms
str 265 ms 233 ms

After each run SELECT count(), sum(cityHash64(u)), sum(cityHash64(nu)) returned the same values on both branches.

Setup: ClickHouse 26.3 in Docker from the repo's docker-compose.yml, Python 3.12.13 on Apple Silicon, C extensions built, native codec python.

Memory: the column is now held as a list of 16 byte objects before the join, so the transient peak is higher. For the 500k row insert above, the tracemalloc peak during client.insert went from 12.2 MB to 14.8 MB, since inserts are split into blocks of about 2 MB. Calling write_column directly on one 500k value column peaks much higher, about 77 MB.

Tests

The first commit adds tests in test_native_write.py that pass on main and on this branch. They pin the wire bytes against struct.pack("<QQ", high, low) and round trip through the reader for uuid.UUID, str (lower, upper, no dashes) and int inputs, Nullable(UUID) and LowCardinality(Nullable(UUID)) with leading and all None, Array(UUID), Tuple(UUID, UUID) and Array(Tuple(UUID, String)), the lenient inputs above and the error cases.

  • pytest tests/unit_tests: 5525 passed, 957 skipped
  • pytest tests/integration_tests -k uuid: 8 passed, 3 skipped (local server)
  • ruff check, ruff format --check, mypy: clean

Risk

If the byte layout were wrong, UUID inserts with the Python codec would store different UUIDs without an error. The wire-byte tests and the server-side hash check above guard against that. Reverting the second commit restores the old loop, and the tests from the first commit still pass on it.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

Note

Low Risk
Wire-format change is localized to Python UUID inserts; extensive round-trip and wire-byte tests reduce risk of silent data corruption.

Overview
The Python codec speeds up UUID column inserts by replacing per-row little-endian to_bytes loops with a batched path: build 16-byte big-endian values for uuid.UUID, int, and str inputs, join them once, then use array.array("Q").byteswap() to emit ClickHouse’s two-little-endian-UInt64 wire layout.

Falsy values still write 16 zero bytes; the raw bytes/bytearray/memoryview branch is unchanged. Unit tests in test_native_write.py lock wire bytes to struct.pack("<QQ", ...), cover nullable/low-cardinality, arrays/tuples, lenient inputs, and invalid-value errors.

Reviewed by Cursor Bugbot for commit 73b797d. Bugbot is set up for automated code reviews on this repo. Configure here.

@joe-clickhouse

Copy link
Copy Markdown
Contributor

Hey @sebastianbreguel could you run the same benchmark with native_codec="rust_strict"? Our Rust codec already handles UUID inserts, and it beat both main and this PR in my local tests. Strict mode rules out silent fallback to Python.

In my benchmark I did did find some bottlenecks around nullable date/datetime DataFrame that you improved on the python side. I address similarly on the rust side in #1153. But if you can, I'd like to see how using the Rust codec compares across your workloads.

@sebastianbreguel

sebastianbreguel commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

@joe-clickhouse confirmed, rust_strict beats this PR on every input. Same insert benchmark: 500k rows into (id UInt64, u UUID, nu Nullable(UUID)), median of 3 interleaved rounds, same server-side hashes on all three.

Input main (python) this PR (python) rust_strict
uuid.UUID 169.7 ms 123.9 ms 101.5 ms
int 163.2 ms 117.7 ms 101.5 ms
str 273.3 ms 213.9 ms 173.0 ms

I used clickhouse-connect-core 0.2.3 from PyPI, so #1153 is not in these numbers. Reads point the same way: on a 300k-row, 14-column query, rust_strict took query_df from 0.485 s to 0.296 s, result_rows from 0.415 s to 0.359 s and query_np from 0.759 s to 0.723 s.

This PR only speeds up the python codec, which is still the default. If python stays the default for a while, I'll keep it open. If you'd rather put the effort into Rust, I'll close it.

@joe-clickhouse

Copy link
Copy Markdown
Contributor

@sebastianbreguel awesome! Happy to hear that. I'm still good taking the python PR for now so I'll give it a review. The reason I'va asked you the give the rust codec a try is because eventually, I'll be deprecating the python/cython codec altogether so the more real-world use the better. If you hit any problems on the rust codec (performance or otherwise) please feel free to open an issue or PR.

@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 17, 2026 00:41

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sebastianbreguel, looks good. I ran it locally and the numbers hold up. I merged main, moved the changelog entry under UNRELEASED since 1.13.0 just went out, and dropped the percentage from it since it moves around between Python versions.

@github-actions

Copy link
Copy Markdown

AI review gate — Human approval in place of AI review

Approved on the head commit by joe-clickhouse (write access, not the author), standing in for the AI review:

  • Cursor Bugbot is still reviewing this commit.

What to do:

  • Nothing to do yet if the review is running. The gate turns green when it finishes clean.
  • No review showing up? Post bugbot run as a new top-level PR comment. If that does nothing, Bugbot is probably not enabled for this repository: ask a repository administrator.

Evaluated at 73b797d0. This required workflow does not re-run on its own once the AI review later concludes — check the Cursor Bugbot check on this commit directly for its current state.

@joe-clickhouse
joe-clickhouse merged commit 5cea0cc into ClickHouse:main Nov 17, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants