Skip to content

Quote Arrow field names that are not plain identifiers in generated DDL - #1202

Open
Shriraj1901 wants to merge 1 commit into
ClickHouse:mainfrom
Shriraj1901:quote-arrow-field-names-in-ddl
Open

Shriraj1901 wants to merge 1 commit into
ClickHouse:mainfrom
Shriraj1901:quote-arrow-field-names-in-ddl

Conversation

@Shriraj1901

@Shriraj1901 Shriraj1901 commented Nov 17, 2026 •

Copy link
Copy Markdown

arrow_schema_to_column_defs() copied each Arrow field name straight into the column definition, so create_table_from_arrow_schema() produced invalid DDL for names that are common in Parquet and pandas data, such as "unit price", "order-id" or "2nd". A dotted name like "a.b" was read by the server as a Nested column instead of a single column.

Arrow field names are literal, so any name that is not a plain identifier is now back-quoted (and escaped) with the same helper the SQLAlchemy dialect uses. Plain names such as "id" are returned unchanged, so existing DDL output does not change.

Summary

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

Note

Low Risk
Localized DDL helper change with unit tests; plain identifiers are unchanged so existing output is preserved.

Overview
PyArrow → ClickHouse DDL now back-quotes Arrow column names when they are not plain SQL identifiers.

arrow_schema_to_column_defs() and create_table_from_arrow_schema() route each field name through _arrow_field_to_column_name(), which leaves names like id or _ok9 unchanged and wraps others with the existing _format_identifier() helper (spaces, hyphens, dots, leading digits, Unicode, embedded backticks). Generated CREATE TABLE statements therefore match common Parquet/pandas column names instead of producing rejected DDL or misinterpreting a.b as Nested.

Reviewed by Cursor Bugbot for commit 0fbf6ff. Bugbot is set up for automated code reviews on this repo. Configure here.

arrow_schema_to_column_defs() copied each Arrow field name straight into the
column definition, so create_table_from_arrow_schema() produced invalid DDL
for names that are common in Parquet and pandas data, such as "unit price",
"order-id" or "2nd". A dotted name like "a.b" was read by the server as a
Nested column instead of a single column.

Arrow field names are literal, so any name that is not a plain identifier is
now back-quoted (and escaped) with the same helper the SQLAlchemy dialect
uses. Plain names such as "id" are returned unchanged, so existing DDL output
does not change.
@Shriraj1901
Shriraj1901 requested a review from a team as a code owner November 17, 2026 10:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant