Repository navigation
Reject NaN and infinity in direct integer inserts - #1173
Conversation
33d8a1e to
ffed029
Compare
|
Thanks for the fix and the detailed notes. I pushed changes on top that take the stricter option you mentioned. Direct inserts now raise DataError for NaN in a Nullable integer column instead of storing NULL. NaN and None in non-nullable integer columns raise as well. The reasoning is that NaN is a float value and not a missing marker. ClickHouse rejects NaN for integer columns even when they are nullable, in CAST, VALUES, INSERT SELECT and Parquet or Arrow input, and the Rust codec does the same. Storing NULL would mean the client guessing at intent. Raising matches both, and it can be relaxed later if needed. insert_df keeps NaN as missing because that is the pandas convention, and pandas also writes NaN as null in Parquet. I also added unit tests and updated the title and description to match. |
Summary
Fixes #1172
With the Python codec,
client.insertwrote float NaN and infinity into integer columns as 0, includingNullablecolumns. There was no error or warning.Direct inserts into 8 to 64 bit integer columns now raise
DataErrorfor float NaN and infinity. This covers nullable columns and values inside arrays, tuples, maps and LowCardinality columns. ANonevalue in a non-nullable integer column also raisesDataErrorinstead of being written as 0.insert_dfkeeps its current behavior. Missing values are written as NULL in a nullable integer column and as 0 in a non-nullable one. The Rust codec now does the same forinsert_df. Before, it raisedDataErrorfor missing values in non-nullable integer columns.Why NaN raises instead of becoming NULL
NaN is a float value, not a missing value, and ClickHouse keeps that distinction. A
Float64column stores NaN as a value. The server rejects NaN and infinity for every integer type with code 70, nullable or not. This holds for CAST, VALUES, INSERT SELECT, and Parquet or Arrow files sent with clickhouse-client. A real NULL into a non-nullable column gets the column default. The Rust codec also rejects NaN. Raising keeps the Python codec consistent with both, and it can be relaxed later without breaking anyone.pandas is the exception. It uses NaN to mark missing values, and
DataFrame.to_parquetwrites NaN as a Parquet null. Soinsert_dftreats NaN as missing, which gives the same result as insertingdf.to_parquet()output through clickhouse-client.Tests
Unit tests in
test_native_write.pyandtest_insert.pycover lists, float16, float32 and float64 NumPy arrays, nested types and the DataFrame paths. Integration tests cover sync and async clients withCLICKHOUSE_CONNECT_USE_C=0and1.