Skip to content

Fix query classification for comments and quoted text - #1186

Merged
joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
sankalpsthakur:fix/925-sql-comment-classification
Nov 13, 2026
Merged

joe-clickhouse merged 3 commits into
ClickHouse:mainfrom
sankalpsthakur:fix/925-sql-comment-classification

Conversation

@sankalpsthakur

@sankalpsthakur sankalpsthakur commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

remove_sql_comments used one regex. It knew -- line comments, block comments without nesting, and plain quotes. The server lexer also reads //, # and #! line comments, nested block comments, backtick identifiers, heredocs and backslash escapes. The wrong text reached QueryContext.is_select, has_limit and is_command. The client then skipped query_limit, added a second LIMIT (server error 62), or missed a command that follows a comment.

Closes #925.

This follows the plan in the maintainer comment on #925 from 2026-09-08. adelbordbari, who took the issue first, released it on 2026-11-08.

What changed

  • binding.py: _sql_opaque_regions yields the comments, quoted strings, quoted identifiers and heredocs of a query, with the existing lexer. A query without $ jumps from one possible region start to the next. A query with $ is lexed token by token, because $ can continue a bareword or open a heredoc.
  • query.py: remove_sql_comments removes the comment regions. is_select and has_limit keep select_re and limit_re and ignore a match inside a quoted region. is_command is not changed. It reads the text that remove_sql_comments returns, so it now finds a command that follows a //, # or #! comment. The separator fix from Fix query type detection: a removed comment must still separate tokens #929 is kept.
  • client.py: _prep_query tests query_limit first. With the default of 0 it does not classify the query.
  • The old comment_re is removed. Nothing else used it.

Text outside comments and quotes is classified as before. For example DESCRIBE (SELECT 13) still gets no LIMIT. I compared 1997 SQL strings from tests/ and edge cases against main. 32 differ, and each one contains a comment, quote or heredoc. remove_sql_comments keeps its output and its separators where the old regex read the SQL correctly. Over the 9,927 string constants in tests/, its output changes for 155, and each of them contains a //, # or #! comment, a nested comment, a backtick identifier, a heredoc, a backslash escape or an unterminated quote. Parameter binding, INSERT detection and inline INSERT data are not changed. The LIMIT 0 metadata path from #1038 now ignores a LIMIT 0 in a comment or a quote, and a real LIMIT 0 query still takes it.

Tests

  • New unit tests cover remove_sql_comments, the classification, _prep_query and the lexer helper. 19 + 27 + 9 of them fail on main in the parser, query and chDB tests. All pass here.
  • One helper test compares the jump scan with lexing every token, for every combination of up to three lexer-relevant fragments.
  • chDB test (ClickHouse 26.9.2.1): 11 cases with query_limit=2. On main, 9 fail as in remove_sql_comments misses // and # comments, nested block comments, backtick identifiers and heredocs, so queries are misclassified #925: all rows come back, or error 62.
  • test_query_limit_ignores_limit_in_comments_and_quotes in tests/integration_tests/test_client.py uses param_client for sync and async. I did not run it, because it needs a server. The same cases pass on chDB.
  • Unit suite: 7264 passed against 7151 on main. The same 18 errors in both, from test_httpclient.py::TestQuery, which needs a server.
  • ruff format --check, ruff check and mypy pass.

Performance

Time to build a QueryContext and run Client._prep_query, main -> this branch, Python 3.13.

input 1 KiB 64 KiB 1 MiB
plain SQL 32 -> 10 us 1.84 -> 0.46 ms 27.6 -> 8.8 ms
string literals 43 -> 10 us 2.23 -> 0.42 ms 32.5 -> 6.8 ms
one large quoted value 22 -> 15 us 1.11 -> 0.67 ms 18.4 -> 11.6 ms
leading block comment 35 -> 19 us 1.79 -> 0.43 ms 28.6 -> 7.9 ms
line comments 28 -> 27 us 1.65 -> 1.45 ms 25.8 -> 24.3 ms
nested comments 197 -> 48 us 612 -> 2.7 ms main is quadratic
unmatched heredoc candidates 29 -> 11 us 1.86 -> 0.47 ms 30.8 -> 8.1 ms
unterminated quote 38 -> 13 us 2.24 -> 0.48 ms 35.5 -> 9.8 ms
unterminated comment 37 -> 60 us 2.16 -> 3.45 ms 34.2 -> 56.1 ms

This is the default query_limit=0. With query_limit=2 most rows are within 1.2x of main. Literals with a trailing LIMIT are 1.4x slower, and an unterminated comment is 2x slower. Both grow linearly. An unterminated comment is slower because _skip_block_comment reads it one character at a time. I did not change that helper.

AI tools assisted with this change.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG

Note

Medium Risk
Touches core query prep and limit/command classification paths used on every query; behavior changes for SQL with non-trivial comments or quotes, but coverage is extensive in unit/integration tests.

Overview
Replaces regex-based comment stripping with a ClickHouse-aligned lexer so SELECT, LIMIT, and command detection ignore text inside comments, string literals, backtick identifiers, heredocs, and escapes.

remove_sql_comments now handles //, # , #!, and nested /* */ comments via new _sql_opaque_regions in binding.py. QueryContext.is_select and has_limit only match keywords outside opaque regions, fixing false limits in comments (duplicate LIMIT / error 62), skipped query_limit, and wrong columns-only LIMIT 0 probes. Client._prep_query checks query_limit before touching classification when the default limit is off.

Comment removal scans opaque regions only (faster on typical SQL; some edge cases with late comment markers can be slower). comment_re remains exported for compatibility.

Reviewed by Cursor Bugbot for commit ede2c01. Bugbot is set up for automated code reviews on this repo. Configure here.

remove_sql_comments used one regex that knew only "--" line comments,
block comments without nesting, and plain quotes. The server lexer also
has "//", "# " and "#!" comments, nested block comments, backtick
identifiers, heredocs and backslash escapes. The wrong result fed
QueryContext.is_select, has_limit and is_command, so the client skipped
or doubled query_limit.

Lex only the comments and the quoted regions, with the shared lexer in
binding.py, and match the existing keyword patterns outside them. Text
outside comments and quotes is classified as before. A query without "$"
jumps from one possible region start to the next, so plain text is not
visited.

_prep_query now tests query_limit first. With the default of 0 it does
not classify the query at all.

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Thanks. This matches the approach I asked for on the issue, and it does fix #925. I pushed one commit on top to:

  • Speed up block comment and $ scanning. Both were much slower than main on large queries.
  • Restore comment_re as a compatibility export.
  • Merge the duplicated test cases into one shared fixture.
  • Reword the changelog and add the LIMIT 0 row loss case.

Some very large queries still prepare slower than main. I have a prototype Cython scanner that removes that cost. I will open it myself in a separate PR.

@github-actions

Copy link
Copy Markdown

AI review gate — Human approval in place of AI review

Approved on the head commit by joe-clickhouse (write access, not the author), standing in for the AI review:

  • Cursor Bugbot is still reviewing this commit.

What to do:

  • Nothing to do yet if the review is running. The gate turns green when it finishes clean.
  • No review showing up? Post bugbot run as a new top-level PR comment. If that does nothing, Bugbot is probably not enabled for this repository: ask a repository administrator.

Evaluated at ede2c011. This required workflow does not re-run on its own once the AI review later concludes — check the Cursor Bugbot check on this commit directly for its current state.

@joe-clickhouse
joe-clickhouse merged commit a392bf1 into ClickHouse:main Nov 13, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

remove_sql_comments misses // and # comments, nested block comments, backtick identifiers and heredocs, so queries are misclassified

3 participants