Repository navigation
Speed up UUID inserts in the Python codec - #1152
Conversation
|
Hey @sebastianbreguel could you run the same benchmark with In my benchmark I did did find some bottlenecks around nullable date/datetime DataFrame that you improved on the python side. I address similarly on the rust side in #1153. But if you can, I'd like to see how using the Rust codec compares across your workloads. |
|
@joe-clickhouse confirmed, rust_strict beats this PR on every input. Same insert benchmark: 500k rows into
I used clickhouse-connect-core 0.2.3 from PyPI, so #1153 is not in these numbers. Reads point the same way: on a 300k-row, 14-column query, rust_strict took query_df from 0.485 s to 0.296 s, result_rows from 0.415 s to 0.359 s and query_np from 0.759 s to 0.723 s. This PR only speeds up the python codec, which is still the default. If python stays the default for a while, I'll keep it open. If you'd rather put the effort into Rust, I'll close it. |
|
@sebastianbreguel awesome! Happy to hear that. I'm still good taking the python PR for now so I'll give it a review. The reason I'va asked you the give the rust codec a try is because eventually, I'll be deprecating the python/cython codec altogether so the more real-world use the better. If you hit any problems on the rust codec (performance or otherwise) please feel free to open an issue or PR. |
# Conflicts: # tests/unit_tests/test_driver/test_native_write.py
joe-clickhouse
left a comment
There was a problem hiding this comment.
Thanks @sebastianbreguel, looks good. I ran it locally and the numbers hold up. I merged main, moved the changelog entry under UNRELEASED since 1.13.0 just went out, and dropped the percentage from it since it moves around between Python versions.
AI review gate — Human approval in place of AI reviewApproved on the head commit by joe-clickhouse (write access, not the author), standing in for the AI review:
What to do:
Evaluated at |
Summary
The Python codec writes a
UUIDinsert column one value at a time. For every value it does twoto_bytes(8, "little")calls, one concatenation and onebytearrayappend:This change builds the big-endian 16 byte value for each row in a list comprehension (
v.bytes,x.to_bytes(16, "big"), orint(v.replace("-", ""), 16).to_bytes(16, "big")for strings), joins them once, loads the result intoarray.array("Q")and callsbyteswap(). Reversing each 8 byte half of the big-endian value gives exactly the wire layout, two little-endian UInt64 with the high half first.byteswap()reverses the raw bytes of each element, so the result is the same on little- and big-endian hosts.uuid.UUID,intandstrinputs use the new path. Falsy values (None,0,"") still write 16 zero bytes.OverflowError, non-hex strings raiseValueError, short hex strings such as"1234"are still zero-padded. The int branch callsint.to_bytes(x, ...)unbound so a column like[13, "user_1"]still raisesTypeError, asx >> 64did.bytes/bytearray/memoryviewbranch is unchanged on purpose. A short bytes value there currently writes fewer than 16 bytes, and joining would turn that into aValueErroror a shifted column, so changing it is a behavior change that does not belong in a perf PR.native_codec="rust"inserts reach this method only when they fall back to the Python encoder.Performance
Serialization only,
get_from_name("UUID").write_column(values, bytearray(), ctx)on 500k random UUIDs, median of 9 runs, main -> this PR:uuid.UUIDintstrEnd to end,
client.insertof 500k rows into(id UInt64, u UUID, nu Nullable(UUID)) ENGINE MergeTree, a quarter ofnuNULL, median of 7 runs:uuid.UUIDintstrAfter each run
SELECT count(), sum(cityHash64(u)), sum(cityHash64(nu))returned the same values on both branches.Setup: ClickHouse 26.3 in Docker from the repo's
docker-compose.yml, Python 3.12.13 on Apple Silicon, C extensions built, native codecpython.Memory: the column is now held as a list of 16 byte objects before the join, so the transient peak is higher. For the 500k row insert above, the
tracemallocpeak duringclient.insertwent from 12.2 MB to 14.8 MB, since inserts are split into blocks of about 2 MB. Callingwrite_columndirectly on one 500k value column peaks much higher, about 77 MB.Tests
The first commit adds tests in
test_native_write.pythat pass onmainand on this branch. They pin the wire bytes againststruct.pack("<QQ", high, low)and round trip through the reader foruuid.UUID,str(lower, upper, no dashes) andintinputs,Nullable(UUID)andLowCardinality(Nullable(UUID))with leading and allNone,Array(UUID),Tuple(UUID, UUID)andArray(Tuple(UUID, String)), the lenient inputs above and the error cases.pytest tests/unit_tests: 5525 passed, 957 skippedpytest tests/integration_tests -k uuid: 8 passed, 3 skipped (local server)ruff check,ruff format --check,mypy: cleanRisk
If the byte layout were wrong, UUID inserts with the Python codec would store different UUIDs without an error. The wire-byte tests and the server-side hash check above guard against that. Reverting the second commit restores the old loop, and the tests from the first commit still pass on it.
Checklist
Note
Low Risk
Wire-format change is localized to Python UUID inserts; extensive round-trip and wire-byte tests reduce risk of silent data corruption.
Overview
The Python codec speeds up
UUIDcolumn inserts by replacing per-row little-endianto_bytesloops with a batched path: build 16-byte big-endian values foruuid.UUID,int, andstrinputs, join them once, then usearray.array("Q").byteswap()to emit ClickHouse’s two-little-endian-UInt64wire layout.Falsy values still write 16 zero bytes; the raw
bytes/bytearray/memoryviewbranch is unchanged. Unit tests intest_native_write.pylock wire bytes tostruct.pack("<QQ", ...), cover nullable/low-cardinality, arrays/tuples, lenient inputs, and invalid-value errors.Reviewed by Cursor Bugbot for commit 73b797d. Bugbot is set up for automated code reviews on this repo. Configure here.