Skip to content

Keep Nullable Int128 and Int256 values exact in query_df - #1182

Merged
joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/query-df-nullable-bigint
Nov 12, 2026
Merged

joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/query-df-nullable-bigint

Conversation

@mansur-akhmadi

Copy link
Copy Markdown
Contributor

Summary

Fixes #1181

BigInt has no _finalize_column, so a nullable column reaches NumpyResult.close_df as a Python list of ints and None. pd.Series makes that float64 when the ints fit in 64 bits, and pd.concat of an int64 block and a uint64 block also gives float64. Values above 2**53 are rounded in both cases.

df = client.query_df("SELECT CAST(arrayJoin([toInt128(9007199254740993), NULL]) AS Nullable(Int128)) AS v")
# before: float64 [9007199254740992.0, nan]

This change adds BigInt._finalize_column. For pandas results it returns a nullable BigInt column as an object array, so the values stay Python int and NULL stays None. That is the same null object as Nullable(UUID) and Nullable(Decimal) columns in query_df. It follows the Nullable(UInt64) fix in #1127, but pandas has no extension dtype for 128 and 256 bit integers, so object is the only exact choice.

Behavior change: a Nullable big integer block without NULL values used to be int64 or uint64, and is now object. I kept that so the dtype of the column does not depend on the data in each block. Non-nullable Int128/Int256 columns are not changed. They can still hit the int64 plus uint64 concat case across blocks. Fixing that means changing their dtype from int64 to object for every user, so I left it for a separate decision.

Tests: tests/integration_tests/test_pandas.py covers the four Nullable types and LowCardinality(Nullable(Int128)), negative values, NULL in different blocks, an all-NULL block, the no-NULL int64/uint64 case, both use_extended_dtypes settings and query_df_stream, through sync and async clients. The tests fail on main and pass with this change, with CLICKHOUSE_CONNECT_USE_C=0 and 1.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

@joe-clickhouse

Copy link
Copy Markdown
Contributor

Hi @mansur-akhmadi some if your issues might be alleviated by using the rust codec native_codec="rust_strict"? Before raising more issues and PRs can you give your workloads a go with that and see if any problems are resolved? If you still have problems feel free to raise. Out of curiosity are you fuzz testing? Or are you hitting all these problems with actual workloads?

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good. This was present in the Rust codec as well, so nice work. I made a small update to the changelog.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Changing NULL-free nullable big-integer results from int64/uint64 to object requires an explicit maintainer compatibility decision under the 1.x policy, and no approval is documented.

No open issues 🎉

What changed in this PR

Fixes #1181 by preserving exact nullable 128-bit and 256-bit integers in the driver's pandas results. Non-nullable big integer behavior remains unchanged.

Changes:

  • Uses object arrays to preserve Python integers and None, including across result blocks.
  • Adds sync and async regression coverage for precision, NULL placement, dtype settings, and streaming.
  • Documents the fix and the change to object dtype.

No review findings. Tests, linting, and type checks were not run here.

File Description
tests/​integration_tests/​test_pandas.py Adds nullable big integer precision and streaming regression tests.
clickhouse_connect/​datatypes/​numeric.py Finalizes nullable big integer pandas columns as object arrays.
CHANGELOG.md Records the precision fix and observable dtype change.

Review effort: Balanced

@joe-clickhouse

Copy link
Copy Markdown
Contributor

Note: had to close and then re-open to PR to re-trigger the half-dead CI workflow runs that were stuck somewhere in actions purgatory.

@joe-clickhouse
joe-clickhouse merged commit 7d60f68 into ClickHouse:main Nov 12, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query_df rounds Nullable(Int128/UInt128/Int256/UInt256) values above 2**53

3 participants