Skip to content

Reject NaN and infinity in direct integer inserts - #1173

Merged
joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/int-nan-insert
Nov 10, 2026
Merged

joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/int-nan-insert

Conversation

@mansur-akhmadi

@mansur-akhmadi mansur-akhmadi commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1172

With the Python codec, client.insert wrote float NaN and infinity into integer columns as 0, including Nullable columns. There was no error or warning.

Direct inserts into 8 to 64 bit integer columns now raise DataError for float NaN and infinity. This covers nullable columns and values inside arrays, tuples, maps and LowCardinality columns. A None value in a non-nullable integer column also raises DataError instead of being written as 0.

insert_df keeps its current behavior. Missing values are written as NULL in a nullable integer column and as 0 in a non-nullable one. The Rust codec now does the same for insert_df. Before, it raised DataError for missing values in non-nullable integer columns.

Why NaN raises instead of becoming NULL

NaN is a float value, not a missing value, and ClickHouse keeps that distinction. A Float64 column stores NaN as a value. The server rejects NaN and infinity for every integer type with code 70, nullable or not. This holds for CAST, VALUES, INSERT SELECT, and Parquet or Arrow files sent with clickhouse-client. A real NULL into a non-nullable column gets the column default. The Rust codec also rejects NaN. Raising keeps the Python codec consistent with both, and it can be relaxed later without breaking anyone.

pandas is the exception. It uses NaN to mark missing values, and DataFrame.to_parquet writes NaN as a Parquet null. So insert_df treats NaN as missing, which gives the same result as inserting df.to_parquet() output through clickhouse-client.

Tests

Unit tests in test_native_write.py and test_insert.py cover lists, float16, float32 and float64 NumPy arrays, nested types and the DataFrame paths. Integration tests cover sync and async clients with CLICKHOUSE_CONNECT_USE_C=0 and 1.

@joe-clickhouse joe-clickhouse changed the title Insert NaN as NULL into Nullable integer columns and reject infinity Reject NaN and infinity in direct integer inserts Nov 10, 2026
@joe-clickhouse

Copy link
Copy Markdown
Contributor

Thanks for the fix and the detailed notes. I pushed changes on top that take the stricter option you mentioned. Direct inserts now raise DataError for NaN in a Nullable integer column instead of storing NULL. NaN and None in non-nullable integer columns raise as well.

The reasoning is that NaN is a float value and not a missing marker. ClickHouse rejects NaN for integer columns even when they are nullable, in CAST, VALUES, INSERT SELECT and Parquet or Arrow input, and the Rust codec does the same. Storing NULL would mean the client guessing at intent. Raising matches both, and it can be relaxed later if needed. insert_df keeps NaN as missing because that is the pandas convention, and pandas also writes NaN as null in Parquet.

I also added unit tests and updated the title and description to match.

@joe-clickhouse
joe-clickhouse merged commit 0eefca8 into ClickHouse:main Nov 10, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Float NaN and inf are inserted as 0 into integer columns, also into Nullable ones

2 participants