Repository navigation
Keep Nullable Int128 and Int256 values exact in query_df - #1182
Conversation
|
Hi @mansur-akhmadi some if your issues might be alleviated by using the rust codec native_codec="rust_strict"? Before raising more issues and PRs can you give your workloads a go with that and see if any problems are resolved? If you still have problems feel free to raise. Out of curiosity are you fuzz testing? Or are you hitting all these problems with actual workloads? |
joe-clickhouse
left a comment
There was a problem hiding this comment.
Thanks, looks good. This was present in the Rust codec as well, so nice work. I made a small update to the changelog.
There was a problem hiding this comment.
🔵 Needs a closer look
Changing NULL-free nullable big-integer results from int64/uint64 to object requires an explicit maintainer compatibility decision under the 1.x policy, and no approval is documented.
No open issues 🎉
What changed in this PR
Fixes #1181 by preserving exact nullable 128-bit and 256-bit integers in the driver's pandas results. Non-nullable big integer behavior remains unchanged.
Changes:
- Uses object arrays to preserve Python integers and
None, including across result blocks. - Adds sync and async regression coverage for precision, NULL placement, dtype settings, and streaming.
- Documents the fix and the change to object dtype.
No review findings. Tests, linting, and type checks were not run here.
| File | Description |
|---|---|
tests/integration_tests/test_pandas.py |
Adds nullable big integer precision and streaming regression tests. |
clickhouse_connect/datatypes/numeric.py |
Finalizes nullable big integer pandas columns as object arrays. |
CHANGELOG.md |
Records the precision fix and observable dtype change. |
Review effort: Balanced
|
Note: had to close and then re-open to PR to re-trigger the half-dead CI workflow runs that were stuck somewhere in actions purgatory. |
Summary
Fixes #1181
BigInthas no_finalize_column, so a nullable column reachesNumpyResult.close_dfas a Python list of ints andNone.pd.Seriesmakes thatfloat64when the ints fit in 64 bits, andpd.concatof anint64block and auint64block also givesfloat64. Values above2**53are rounded in both cases.This change adds
BigInt._finalize_column. For pandas results it returns a nullableBigIntcolumn as an object array, so the values stay Pythonintand NULL staysNone. That is the same null object asNullable(UUID)andNullable(Decimal)columns inquery_df. It follows theNullable(UInt64)fix in #1127, but pandas has no extension dtype for 128 and 256 bit integers, soobjectis the only exact choice.Behavior change: a
Nullablebig integer block without NULL values used to beint64oruint64, and is nowobject. I kept that so the dtype of the column does not depend on the data in each block. Non-nullableInt128/Int256columns are not changed. They can still hit theint64plusuint64concat case across blocks. Fixing that means changing their dtype fromint64toobjectfor every user, so I left it for a separate decision.Tests:
tests/integration_tests/test_pandas.pycovers the fourNullabletypes andLowCardinality(Nullable(Int128)), negative values, NULL in different blocks, an all-NULL block, the no-NULLint64/uint64case, bothuse_extended_dtypessettings andquery_df_stream, through sync and async clients. The tests fail on main and pass with this change, withCLICKHOUSE_CONNECT_USE_C=0and1.Checklist