Skip to content

Decode JSON shared data values in query_df and query_np - #1177

Merged
joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/query-df-json-shared-data
Nov 11, 2026
Merged

joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
mansur-akhmadi:fix/query-df-json-shared-data

Conversation

@mansur-akhmadi

Copy link
Copy Markdown
Contributor

Summary

Fixes #1176

JSON shared data is read as a Map(String, String) whose value type is SharedDataString. That class reads raw bytes, but it inherits String._finalize_column. With use_extended_dtypes, which query_df uses by default, or with max_str_len, the raw bytes become pandas or NumPy strings. decode_shared_data_value then sees a str and returns it unchanged, so the type byte and payload end up in the result.

query = """SELECT '{"a": 13, "s": "user_1"}'::JSON(max_dynamic_paths=0) AS j"""
client.query_df(query).iloc[0, 0]
# before: {'a': '\n\r\x00\x00\x00\x00\x00\x00\x00', 's': '\x15\x06user_1'}

This change:

  • adds SharedDataString._finalize_column that returns the raw column, the same as SharedVariant already does.
  • changes _decode_variant to check len(result) instead of result. In NumPy mode the single decoded value is a one-element array, and 0 or False made it falsy, so the value became None. This dropped 0 and false from JSON shared data and also turned Dynamic(max_types=0) values 0 and false into None in query_df.

Not changed: with max_str_len, shared data path names and string values are still cut to that length, and decoded numbers are NumPy scalars in NumPy and pandas results. Both come from decoding with the query context and were the same before. I kept this PR small, but I can follow up with a neutral decode context if you want.

Tests: tests/integration_tests/test_dynamic.py covers JSON(max_dynamic_paths=0) as a scalar and inside Array, Map, Tuple and Array(Tuple), with numbers, strings, floats, arrays, nested objects, 0, false and "", for query, query_df and query_np with and without extended dtypes and max_str_len. It also adds 0 and false to the Dynamic shared value DataFrame test. The new cases fail on main and pass with this change, through sync and async clients, with CLICKHOUSE_CONNECT_USE_C=0 and 1.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

@mansur-akhmadi
mansur-akhmadi force-pushed the fix/query-df-json-shared-data branch from c4706a8 to 759e181 Compare November 10, 2026 18:20

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, approving. I merged main and updated docs/rust-codec.mdx, which listed this bug as a Python vs Rust difference.

The Rust codec doesn't have this bug, but Python is the default, so it's worth fixing there.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

JSON string formatting fails on decoded NumPy scalars with the standard-library serializer, requiring recursive conversion to Python values while preserving native object results and regression coverage for query_df and query_np.

1 open issue
What changed in this PR

Updates the Python codec to decode JSON shared data correctly in pandas and NumPy query results, addressing #1176.

Changes:

  • Preserves raw bytes until decoding and retains valid 0 and false values in JSON and Dynamic results.
  • Adds regression tests for scalar and nested JSON across query APIs and dtype options.
  • Updates the changelog and documents remaining Python/Rust codec differences.

Validation was limited to code inspection; tests were not run.

File Description
tests/​integration_tests/​test_dynamic.py Adds shared-data decoding and zero/false regression cases.
docs/​rust-codec.mdx Clarifies scalar-type and string-truncation differences between codecs.
clickhouse_connect/​datatypes/​dynamic.py Preserves binary shared values and checks decoded result length.
CHANGELOG.md Records the JSON and Dynamic decoding fixes.

Review effort: Balanced

Comment thread clickhouse_connect/datatypes/dynamic.py Resolved
@joe-clickhouse
joe-clickhouse merged commit c3c1e8a into ClickHouse:main Nov 11, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query_df returns undecoded binary for JSON values stored in shared data

3 participants