Skip to content

Convert NumPy datetime64 inserts to the column unit - #1175

Merged
joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/numpy-datetime64-unit
Nov 11, 2026
Merged

joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/numpy-datetime64-unit

Conversation

@mansur-akhmadi

Copy link
Copy Markdown
Contributor

Summary

Fixes #1174

InsertContext._convert_numpy turned datetime64 columns into integers with astype(int) and marked them with the "int" format. Those integers are ticks in the array unit, but the writer expects ticks in the column unit: seconds for DateTime, days for Date and Date32, and 10**scale ticks for DateTime64. Only an array with the same unit as the column worked.

data = np.array([[np.datetime64("2024-05-06T07:08:09.123456789")]], dtype="datetime64[ns]")
client.insert("t", data, column_names=["v"])  # v DateTime64(3)
# before: 9999-12-31 23:59:59.789

This change casts each date or time column to col_type.np_type before it takes the ticks, the same as the pandas path in _convert_pandas. The 2-D branch now converts each column on its own, so a 2-D array can feed Date and DateTime64(6) columns together, and it keeps the row or column orientation the caller asked for. Values before the epoch round down, which matches toDateTime64 on the server.

Not changed: NaT in a NumPy array is still not written as NULL, a cast to a finer unit outside the NumPy range still wraps, DateTime64 scales other than 0, 3, 6 and 9 still raise the existing ProgrammingError, and a numpy.datetime64 scalar inside row data still fails as before. I can send those separately if you want them.

Tests: tests/integration_tests/test_numpy.py covers Date, Date32, DateTime, DateTime64(3/6/9) and Nullable(DateTime64(3)) with units from D to ns, a pre-epoch value, and structured, row and column-oriented 2-D arrays, through sync and async clients. A unit test in tests/unit_tests/test_driver/test_insert.py checks the ticks and the orientation of a mixed 2-D array. The new tests fail on main and pass with this change, with CLICKHOUSE_CONNECT_USE_C=0 and 1.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

@mansur-akhmadi
mansur-akhmadi force-pushed the fix/numpy-datetime64-unit branch from 5b05926 to e31ad9b Compare November 10, 2026 18:18

@joe-clickhouse joe-clickhouse left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Looks good.

I pushed two commits on top:

  • Renamed _datetime64_ticks to _np_datetime_ticks. The old name matched an unrelated method on DateTime64.
  • Added type hints to the helper and removed its comment.
  • Merged main.

NaT as NULL and the other gaps you listed can go in a separate PR.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Using np_array.T reverses nested cell dimensions, causing previously accepted row-oriented (2, 1, 3) datetime arrays for an Array(DateTime) column to raise ProgrammingError.

1 open issue
What changed in this PR

This PR addresses #1174 by converting NumPy datetime64 values to the target column's time unit before insertion, rather than writing raw array ticks.

Changes:

  • Adds per-column conversion for structured and two-dimensional NumPy arrays.
  • Adds unit and sync/async integration tests for unit conversion, mixed column types, and orientation.
  • Documents the user-visible fix in the changelog.
File Description
tests/​unit_tests/​test_driver/​test_insert.py Checks converted ticks, orientation, and column-count validation.
tests/​integration_tests/​test_numpy.py Covers date/time units, array layouts, and mixed columns.
clickhouse_connect/​driver/​insert.py Converts NumPy date/time columns to target units before extracting ticks.
CHANGELOG.md Describes the insertion fix and previous failure modes.

Review effort: Balanced

Comment thread clickhouse_connect/driver/insert.py Resolved
@joe-clickhouse
joe-clickhouse merged commit 6296050 into ClickHouse:main Nov 11, 2026
91 of 92 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NumPy datetime64 inserts write wrong dates when the array unit differs from the column

3 participants