Repository navigation
Fix query classification for comments and quoted text - #1186
Merged
joe-clickhouse merged 3 commits intoNov 13, 2026
Merged
joe-clickhouse merged 3 commits into
joe-clickhouse merged 3 commits into
Conversation
remove_sql_comments used one regex that knew only "--" line comments, block comments without nesting, and plain quotes. The server lexer also has "//", "# " and "#!" comments, nested block comments, backtick identifiers, heredocs and backslash escapes. The wrong result fed QueryContext.is_select, has_limit and is_command, so the client skipped or doubled query_limit. Lex only the comments and the quoted regions, with the shared lexer in binding.py, and match the existing keyword patterns outside them. Text outside comments and quotes is classified as before. A query without "$" jumps from one possible region start to the next, so plain text is not visited. _prep_query now tests query_limit first. With the default of 0 it does not classify the query at all.
joe-clickhouse
approved these changes
Nov 13, 2026
joe-clickhouse
left a comment
Contributor
There was a problem hiding this comment.
Approved. Thanks. This matches the approach I asked for on the issue, and it does fix #925. I pushed one commit on top to:
- Speed up block comment and
$scanning. Both were much slower than main on large queries. - Restore
comment_reas a compatibility export. - Merge the duplicated test cases into one shared fixture.
- Reword the changelog and add the
LIMIT 0row loss case.
Some very large queries still prepare slower than main. I have a prototype Cython scanner that removes that cost. I will open it myself in a separate PR.
AI review gate — Human approval in place of AI reviewApproved on the head commit by joe-clickhouse (write access, not the author), standing in for the AI review:
What to do:
Evaluated at |
chernser
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
remove_sql_commentsused one regex. It knew--line comments, block comments without nesting, and plain quotes. The server lexer also reads//,#and#!line comments, nested block comments, backtick identifiers, heredocs and backslash escapes. The wrong text reachedQueryContext.is_select,has_limitandis_command. The client then skippedquery_limit, added a secondLIMIT(server error 62), or missed a command that follows a comment.Closes #925.
This follows the plan in the maintainer comment on #925 from 2026-09-08. adelbordbari, who took the issue first, released it on 2026-11-08.
What changed
binding.py:_sql_opaque_regionsyields the comments, quoted strings, quoted identifiers and heredocs of a query, with the existing lexer. A query without$jumps from one possible region start to the next. A query with$is lexed token by token, because$can continue a bareword or open a heredoc.query.py:remove_sql_commentsremoves the comment regions.is_selectandhas_limitkeepselect_reandlimit_reand ignore a match inside a quoted region.is_commandis not changed. It reads the text thatremove_sql_commentsreturns, so it now finds a command that follows a//,#or#!comment. The separator fix from Fix query type detection: a removed comment must still separate tokens #929 is kept.client.py:_prep_querytestsquery_limitfirst. With the default of 0 it does not classify the query.comment_reis removed. Nothing else used it.Text outside comments and quotes is classified as before. For example
DESCRIBE (SELECT 13)still gets noLIMIT. I compared 1997 SQL strings fromtests/and edge cases against main. 32 differ, and each one contains a comment, quote or heredoc.remove_sql_commentskeeps its output and its separators where the old regex read the SQL correctly. Over the 9,927 string constants intests/, its output changes for 155, and each of them contains a//,#or#!comment, a nested comment, a backtick identifier, a heredoc, a backslash escape or an unterminated quote. Parameter binding, INSERT detection and inline INSERT data are not changed. TheLIMIT 0metadata path from #1038 now ignores aLIMIT 0in a comment or a quote, and a realLIMIT 0query still takes it.Tests
remove_sql_comments, the classification,_prep_queryand the lexer helper. 19 + 27 + 9 of them fail on main in the parser, query and chDB tests. All pass here.query_limit=2. On main, 9 fail as in remove_sql_comments misses // and # comments, nested block comments, backtick identifiers and heredocs, so queries are misclassified #925: all rows come back, or error 62.test_query_limit_ignores_limit_in_comments_and_quotesintests/integration_tests/test_client.pyusesparam_clientfor sync and async. I did not run it, because it needs a server. The same cases pass on chDB.test_httpclient.py::TestQuery, which needs a server.ruff format --check,ruff checkandmypypass.Performance
Time to build a
QueryContextand runClient._prep_query, main -> this branch, Python 3.13.This is the default
query_limit=0. Withquery_limit=2most rows are within 1.2x of main. Literals with a trailing LIMIT are 1.4x slower, and an unterminated comment is 2x slower. Both grow linearly. An unterminated comment is slower because_skip_block_commentreads it one character at a time. I did not change that helper.AI tools assisted with this change.
Checklist
Note
Medium Risk
Touches core query prep and limit/command classification paths used on every query; behavior changes for SQL with non-trivial comments or quotes, but coverage is extensive in unit/integration tests.
Overview
Replaces regex-based comment stripping with a ClickHouse-aligned lexer so
SELECT,LIMIT, and command detection ignore text inside comments, string literals, backtick identifiers, heredocs, and escapes.remove_sql_commentsnow handles//,#,#!, and nested/* */comments via new_sql_opaque_regionsinbinding.py.QueryContext.is_selectandhas_limitonly match keywords outside opaque regions, fixing false limits in comments (duplicateLIMIT/ error 62), skippedquery_limit, and wrong columns-onlyLIMIT 0probes.Client._prep_querychecksquery_limitbefore touching classification when the default limit is off.Comment removal scans opaque regions only (faster on typical SQL; some edge cases with late comment markers can be slower).
comment_reremains exported for compatibility.Reviewed by Cursor Bugbot for commit ede2c01. Bugbot is set up for automated code reviews on this repo. Configure here.