Repository navigation
Speed up mixed-type query_np by allocating blocks with np.zeros - #1193
Open
sebastianbreguel wants to merge 3 commits into
Open
sebastianbreguel wants to merge 3 commits into
sebastianbreguel wants to merge 3 commits into
Conversation
np.empty on a structured dtype with object fields sets every object field of every row to None. np.zeros fills them several times faster, and every field is assigned right after, so results are unchanged. A 1M-row, 10-column query_np takes about 18% less time on both the Python and Rust codecs.
joe-clickhouse
approved these changes
Nov 17, 2026
joe-clickhouse
left a comment
Contributor
There was a problem hiding this comment.
Thanks, this checks out. I got about 19% faster on a 1M-row mixed query on both codecs, with identical results. I merged main and moved the changelog entry to UNRELEASED.
AI review gate — Human approval in place of AI reviewApproved on the head commit by joe-clickhouse (write access, not the author), standing in for the AI review:
What to do:
Evaluated at |
joe-clickhouse
approved these changes
Nov 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NumpyResultbuilds a structured array per block when aquery_npresult has mixed column types. Each block is now allocated withnp.zerosinstead ofnp.empty. Every field is still assigned from the block's column data right after, so the returned arrays are the same.Why
For a structured dtype with object fields,
np.emptysets each object field of each row toNone, element by element.np.zerosfills them much faster. The column values then replace those placeholders in both cases.One 8192-row block, 10 fields with 5 of them
object:np.emptynp.zerosThat allocation was about 150 ms of a 1M-row
query_np. Because this code runs after decoding, the gain is the same on the Python and Rust codecs.query_npon 1M rows, local ClickHouse 25.8, numpy 2.3.4, pandas 3.0.6,clickhouse-connect-core0.2.3, Python 3.12, macOS arm64. Each run is the p50 of 5 queries. Base and branch are interleaved, 3 pairs:scout_wide:UInt64, Int32, Float64, String, LowCardinality(String), DateTime, Date, Nullable(String), Nullable(Int64), UUIDThat is about 18% less time on the Python codec and 19% on the Rust codec.
scout_arr:UInt64, Array(UInt64), Array(String), Map(String, UInt64)That is about 4% less time on the Python codec and 6% on the Rust codec.
The digest covers the dtype and every column of the full result. It was identical between base and branch for each table and codec.
How to test
Run it on
mainand on this branch, and again withscout_arr.Tests:
tests/integration_tests/test_numpy.py,test_map_pairs.pyandtest_dynamic.pypass with the Python codec underCLICKHOUSE_CONNECT_USE_C=1and=0(656 passed each). WithCLICKHOUSE_CONNECT_NATIVE_CODEC=rust, the same files plustest_rust_codec.pyshow the same 22 failures onmainand on this branch. I ran those with core 0.2.3 and server 25.8 and did not look into the cause. The unit tests intests/unit_tests/test_driverpass (5629).ruffandmypyare clean.Risk
Low. The change touches one allocation in the structured
query_nppath. Every field is assigned before a block is yielded, so callers never see the initial values. If an assignment raises, the block is never yielded, as before.Structured results without object columns pay for a memset: about 2-3 µs more per 8192-row block in a microbenchmark, or about 0.3 ms per 1M rows.
To revert, change
np.zerosback tonp.empty.Checklist
Note
Low Risk
Single allocation change on the structured query_np path; all fields are overwritten before blocks are returned, so callers should see identical data with minor perf tradeoff on non-object structured dtypes.
Overview
query_npmixed-type results (structured NumPy dtypes with at least one object column) now allocate each result block withnp.zerosinstead ofnp.emptyinNumpyResult._np_stream. Column data is still assigned field-by-field before each block is yielded, so output is unchanged.np.emptywas spending significant time setting every object field toNonerow by row;np.zerosfills those slots much faster. Benchmarks in the PR show roughly ~18% less time on a 1M-row wide mixed query (Python codec) and similar gains on the Rust codec path, since this step runs after decode.Mixed-type results without object columns pay a small zero-init cost (~2–3 µs per 8192-row block). The UNRELEASED changelog documents the improvement.
Reviewed by Cursor Bugbot for commit 01e2eb9. Bugbot is set up for automated code reviews on this repo. Configure here.