Skip to content

Speed up mixed-type query_np by allocating blocks with np.zeros - #1193

Open
sebastianbreguel wants to merge 3 commits into
ClickHouse:mainfrom
sebastianbreguel:speed-up-structured-query-np
Open

sebastianbreguel wants to merge 3 commits into
ClickHouse:mainfrom
sebastianbreguel:speed-up-structured-query-np

Conversation

@sebastianbreguel

@sebastianbreguel sebastianbreguel commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

NumpyResult builds a structured array per block when a query_np result has mixed column types. Each block is now allocated with np.zeros instead of np.empty. Every field is still assigned from the block's column data right after, so the returned arrays are the same.

Why

For a structured dtype with object fields, np.empty sets each object field of each row to None, element by element. np.zeros fills them much faster. The column values then replace those placeholders in both cases.

One 8192-row block, 10 fields with 5 of them object:

numpy np.empty np.zeros
1.26.4 1.23 ms 0.15 ms
2.0.2 1.24 ms 0.15 ms
2.2.6 1.11 ms 0.17 ms
2.3.4 1.19 ms 0.16 ms

That allocation was about 150 ms of a 1M-row query_np. Because this code runs after decoding, the gain is the same on the Python and Rust codecs.

query_np on 1M rows, local ClickHouse 25.8, numpy 2.3.4, pandas 3.0.6, clickhouse-connect-core 0.2.3, Python 3.12, macOS arm64. Each run is the p50 of 5 queries. Base and branch are interleaved, 3 pairs:

scout_wide: UInt64, Int32, Float64, String, LowCardinality(String), DateTime, Date, Nullable(String), Nullable(Int64), UUID

codec pair base branch
python 1 711 ms 581 ms
python 2 704 ms 571 ms
python 3 706 ms 577 ms
rust_strict 1 642 ms 523 ms
rust_strict 2 644 ms 521 ms
rust_strict 3 644 ms 518 ms

That is about 18% less time on the Python codec and 19% on the Rust codec.

scout_arr: UInt64, Array(UInt64), Array(String), Map(String, UInt64)

codec pair base branch
python 1 1005 ms 968 ms
python 2 1010 ms 959 ms
python 3 997 ms 973 ms
rust_strict 1 770 ms 718 ms
rust_strict 2 755 ms 722 ms
rust_strict 3 755 ms 709 ms

That is about 4% less time on the Python codec and 6% on the Rust codec.

The digest covers the dtype and every column of the full result. It was identical between base and branch for each table and codec.

How to test

CREATE TABLE scout_wide (id UInt64, i32 Int32, f64 Float64, s String, lc LowCardinality(String),
  dt DateTime, d Date, ns Nullable(String), ni Nullable(Int64), u UUID) ENGINE MergeTree ORDER BY id;
INSERT INTO scout_wide SELECT number, toInt32(number % 100000) - 50000, number / 7,
  concat('name_', toString(number % 50000)), ['red','green','blue','cyan'][number % 4 + 1],
  toDateTime('2024-01-01 00:00:00') + number, toDate('2020-01-01') + number % 3000,
  if(number % 5 = 0, NULL, concat('n', toString(number))), if(number % 3 = 0, NULL, toInt64(number)),
  generateUUIDv4(number) FROM numbers(1000000);

CREATE TABLE scout_arr (id UInt64, a Array(UInt64), s Array(String), m Map(String, UInt64)) ENGINE MergeTree ORDER BY id;
INSERT INTO scout_arr SELECT number, range(number % 8), arrayMap(x -> concat('t', toString(x)), range(number % 5)),
  map('a', number, 'b', number % 7) FROM numbers(1000000);
import time, clickhouse_connect
for codec in ("python", "rust_strict"):
    client = clickhouse_connect.get_client(native_codec=codec, compress=False)
    client.query_np("SELECT * FROM scout_wide LIMIT 10")
    times = []
    for _ in range(5):
        start = time.perf_counter()
        client.query_np("SELECT * FROM scout_wide ORDER BY id")
        times.append(time.perf_counter() - start)
    print(codec, sorted(times)[2])

Run it on main and on this branch, and again with scout_arr.

Tests: tests/integration_tests/test_numpy.py, test_map_pairs.py and test_dynamic.py pass with the Python codec under CLICKHOUSE_CONNECT_USE_C=1 and =0 (656 passed each). With CLICKHOUSE_CONNECT_NATIVE_CODEC=rust, the same files plus test_rust_codec.py show the same 22 failures on main and on this branch. I ran those with core 0.2.3 and server 25.8 and did not look into the cause. The unit tests in tests/unit_tests/test_driver pass (5629). ruff and mypy are clean.

Risk

Low. The change touches one allocation in the structured query_np path. Every field is assigned before a block is yielded, so callers never see the initial values. If an assignment raises, the block is never yielded, as before.

Structured results without object columns pay for a memset: about 2-3 µs more per 8192-row block in a microbenchmark, or about 0.3 ms per 1M rows.

To revert, change np.zeros back to np.empty.

Checklist

  • A human-readable description of the changes was provided to include in CHANGELOG

Note

Low Risk
Single allocation change on the structured query_np path; all fields are overwritten before blocks are returned, so callers should see identical data with minor perf tradeoff on non-object structured dtypes.

Overview
query_np mixed-type results (structured NumPy dtypes with at least one object column) now allocate each result block with np.zeros instead of np.empty in NumpyResult._np_stream. Column data is still assigned field-by-field before each block is yielded, so output is unchanged.

np.empty was spending significant time setting every object field to None row by row; np.zeros fills those slots much faster. Benchmarks in the PR show roughly ~18% less time on a 1M-row wide mixed query (Python codec) and similar gains on the Rust codec path, since this step runs after decode.

Mixed-type results without object columns pay a small zero-init cost (~2–3 µs per 8192-row block). The UNRELEASED changelog documents the improvement.

Reviewed by Cursor Bugbot for commit 01e2eb9. Bugbot is set up for automated code reviews on this repo. Configure here.

np.empty on a structured dtype with object fields sets every object
field of every row to None. np.zeros fills them several times faster,
and every field is assigned right after, so results are unchanged.
A 1M-row, 10-column query_np takes about 18% less time on both the
Python and Rust codecs.
@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 17, 2026 06:29

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this checks out. I got about 19% faster on a 1M-row mixed query on both codecs, with identical results. I merged main and moved the changelog entry to UNRELEASED.

@github-actions

Copy link
Copy Markdown

AI review gate — Human approval in place of AI review

Approved on the head commit by joe-clickhouse (write access, not the author), standing in for the AI review:

  • Cursor Bugbot is still reviewing this commit.

What to do:

  • Nothing to do yet if the review is running. The gate turns green when it finishes clean.
  • No review showing up? Post bugbot run as a new top-level PR comment. If that does nothing, Bugbot is probably not enabled for this repository: ask a repository administrator.

Evaluated at 01e2eb9c. This required workflow does not re-run on its own once the AI review later concludes — check the Cursor Bugbot check on this commit directly for its current state.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants