Skip to content

Add a compiled SQL scanner for query classification - #1192

Merged
joe-clickhouse merged 3 commits into
mainfrom
joe/cython-sql-scanner
Nov 16, 2026
Merged

joe-clickhouse merged 3 commits into
mainfrom
joe/cython-sql-scanner

Conversation

@joe-clickhouse

@joe-clickhouse joe-clickhouse commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #1186 (#925). Moves the SQL lexer that skips comments and
quoted text into a Cython module, driverc/sqlscan.pyx. The pure Python
path stays as the fallback.

  • Wired through ctypes.connect_c_modules like the other C modules, CPython only
  • Only exact str takes the compiled path
  • Heredoc tags use a seeded hash table with a probe budget, falling back to Python if it runs out
  • Differential tests compare the compiled and Python paths on edge cases and seeded fuzz, plus subprocess fallback tests

Query prep is faster than main on every everyday shape we measured.
Results match the Python path on all fuzz cases, on 3.10 to 3.14t.


Note

Low Risk
Performance optimization with strict parity fallback to the existing Python lexer; no API or wire-protocol changes.

Overview
Adds a fourth C extension, clickhouse_connect.driverc.sqlscan, that speeds up comment stripping and regex matching outside quoted/heredoc regions during query classification. query.py and connect_c_modules() route plain CPython str queries through the compiled path when C optimizations are enabled; PyPy, missing extensions, str subclasses, or a heredoc probe budget exhaustion transparently fall back to the existing Python lexer in binding.py (kept in sync via comments).

The C scanner walks Unicode buffers directly, indexes $tag$ heredocs with a bounded hash table, and returns None when probing would be too expensive so behavior stays identical to Python. Docs and changelog note the C extension count; parity/fuzz subprocess tests and a freethreading check cover the new module.

Reviewed by Cursor Bugbot for commit a6fbbf1. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

AI review gate — AI review passed

Clean review from: Cursor Bugbot.

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scope: the compiled scanner commit 500dadd only. The changes that come from #1186 are not reviewed here.

Findings, most important first. Details are in the inline comments.

  1. Test gap: no test fails when sqlscan is not built or not wired (test_sqlscan_parity.py). On this head, with the sqlscan .so removed, the new tests and test_ctypes.py still pass (10 passed, 63 skipped).
  2. Nit: sqlscan.pyx copies _SQL_COMMENT_MARKERS and has no sync pointer for it.
  3. Nit: char is used as a variable name next to C char types.
  4. Question: is the custom heredoc hash table worth its code, compared with a dict?

I found no correctness defect in the scanner.

Performance evidence. The PR body says query prep is faster than main on every everyday shape, but it gives no numbers or environment. .agents/architecture.md asks to record the environment with any claimed result, and it names stresshouse-benchmark-compare.yml for PR comparisons. My measurements of remove_sql_comments support the claim. Times are per call, best of 3. Environment: Python 3.12.16, Cython 3.3.1, gcc 12.2 -O3, Xeon Platinum 8175M, Linux x86_64. main is the regex at 9fa76e5. "Python fallback" is this head with _c_sqlscan = None.

Query shape Length main This PR, compiled This PR, Python fallback
Short SELECT 42 2.6 us 0.5 us 1.7 us
Short SELECT, 2 comments 55 4.8 us 0.9 us 9.3 us
IN list, 20k quoted values 169k 13.7 ms 0.70 ms 0.71 ms
Same, trailing -- 169k 13.5 ms 0.95 ms 20.5 ms
Same, Cyrillic values, trailing -- 169k 16.0 ms 0.82 ms 20.6 ms
20k {p:String}, trailing -- 309k 15.8 ms 12.0 ms 45.8 ms
20k $tN$, trailing -- 169k 8.5 ms 3.8 ms 131 ms
1 MB INSERT ... VALUES, trailing -- 998k 80 ms 6.4 ms 74 ms

Suggest adding numbers like these, or a stresshouse-benchmark-compare run, to the PR body, together with the environment.

Checked, no problem found

  • Parity, compared with _sql_opaque_regions and with the full classification (remove_sql_comments plus _matches_outside_quotes for the three patterns). I ran about 1.9M random queries from a wider fragment set than the one in the test. The set had 1-, 2- and 4-byte str kinds, lone surrogates, curly quotes, non-ASCII placeholder names (Latin letter U+00E9, superscript digit U+00B2, Roman numeral U+216B, a combining mark), {$x:T} and overlapping $a$b$ tags. I also ran about 32k heredoc-heavy queries with up to 400 tags each, to reach the hash-table path, with small probe budgets so that about 30% of them returned None. No differences.
  • C level: bounds in close_tag, find_char and the backslash skip, the output buffer size in remove_sql_comments, the load factor in grow_tags, and that memory is freed on every exit path, including MemoryError.
  • Import wiring: _c_sqlscan is set for every entry point I tried first (clickhouse_connect, driver.binding, driver.query, driverc.sqlscan, dbapi, cc_sqlalchemy, datatypes).
  • Free threading: the module has no shared mutable state except the seed, which is written once at import. So freethreading_compatible = True is correct.
  • 743 targeted unit tests pass (test_sqlscan_parity, test_query, test_binding, test_parser, test_ctypes, test_freethreading). ruff 0.15.8 and mypy report nothing for the changed files. CI is green, except the AI gate, which waits because this is a draft, and the cloud leg, which is still pending.

Not checked: the Windows and macOS compilers, and the PyPy wheel builds. Those wheels are built only in publish.yml, and that job does not require the C extensions.

Review by Polyglot AI of 500dadd.

Comment thread clickhouse_connect/driverc/sqlscan.pyx Resolved
Comment thread clickhouse_connect/driverc/sqlscan.pyx Resolved
Comment thread clickhouse_connect/driverc/sqlscan.pyx Resolved
@joe-clickhouse
joe-clickhouse marked this pull request as ready for review November 13, 2026 23:09

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on ed5bee0. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

Copy link
Copy Markdown

AI review gate — auto-approved

ed5bee030f86 is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • ed5bee030f86 approved — green: Cursor Bugbot

@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 16, 2026 22:24
@joe-clickhouse
joe-clickhouse merged commit 3d50bcf into main Nov 16, 2026
57 checks passed
@joe-clickhouse
joe-clickhouse deleted the joe/cython-sql-scanner branch November 16, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants