Skip to content

Truncate Decimal inserts consistently in the Python codec - #1165

Merged
joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
patrick-immometrica:fix-decimal-insert-rounding
Nov 10, 2026
Merged

joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
patrick-immometrica:fix-decimal-insert-rounding

Conversation

@patrick-immometrica

Copy link
Copy Markdown
Contributor

Summary

The Python codec's Decimal write runs int(Decimal(str(x)) * 10**scale) inside a context with prec = self.prec. When the scaled value has more digits than the column precision, the multiply rounds (half-even) before int() gets to truncate. So the result depends on the magnitude:

Decimal(9, 2):  1.999         -> 1.99
                1234567.899   -> 1234567.90
                -1234567.899  -> -1234567.90

Same for Decimal128/256 via BigDecimal. The Rust codec and the server (toDecimal*, text input) truncate in every case.

The multiply by 10**scale is exact once precision isn't capped, so this sets prec to decimal.MAX_PREC for the write and lets int() truncate. Overflow behaviour is unchanged since the int still has to fit the column width. The read path is untouched.

Checklist

  • Unit tests for Decimal(9, 2), nullable and Decimal(38, 2) (fail on main)
  • CHANGELOG entry

@patrick-immometrica
patrick-immometrica marked this pull request as ready for review November 9, 2026 12:06
# Conflicts:
#	CHANGELOG.md
#	tests/unit_tests/test_driver/test_native_write.py
joe-clickhouse
joe-clickhouse previously approved these changes Nov 9, 2026

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I added wider regression tests, clarified the changelog, and removed the comment. These cover the carry case and both clients. I also merged current main. The fix itself is unchanged.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Both decimal writers inherit clamp=1, which can cause MemoryError on previously valid inserts after precision is raised to MAX_PREC; they need clamp disabled and regression coverage.

1 open issue
What changed in this PR

This PR updates Python codec Decimal inserts to truncate excess fractional digits consistently, without rounding first at larger magnitudes. The read path is unchanged.

Changes:

  • Use decimal.MAX_PREC during scaling in both Decimal writers.
  • Add unit and integration coverage for decimal widths, input types, nullable values, and nested types.
  • Document the behavior correction in the changelog.

Validation was limited to a focused standard-library check. Project tests, linting, and type checks were not run.

File Description
tests/​unit_tests/​test_driver/​test_native_write.py Check encoded integers for Decimal truncation.
tests/​integration_tests/​test_numeric.py Add round-trip tests, including nullable and nested Decimal values.
clickhouse_connect/​datatypes/​numeric.py Increase scaling precision in both Decimal write paths.
CHANGELOG.md Describe the truncation fix and previous rounding behavior.

Review effort: Balanced

Comment thread clickhouse_connect/datatypes/numeric.py Resolved
@joe-clickhouse
joe-clickhouse merged commit 3e77220 into ClickHouse:main Nov 10, 2026
@patrick-immometrica
patrick-immometrica deleted the fix-decimal-insert-rounding branch November 10, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants