Repository navigation
Add a compiled SQL scanner for query classification - #1192
Conversation
AI review gate — AI review passedClean review from: Cursor Bugbot. |
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Scope: the compiled scanner commit 500dadd only. The changes that come from #1186 are not reviewed here.
Findings, most important first. Details are in the inline comments.
- Test gap: no test fails when
sqlscanis not built or not wired (test_sqlscan_parity.py). On this head, with thesqlscan.so removed, the new tests andtest_ctypes.pystill pass (10 passed, 63 skipped). - Nit:
sqlscan.pyxcopies_SQL_COMMENT_MARKERSand has no sync pointer for it. - Nit:
charis used as a variable name next to Cchartypes. - Question: is the custom heredoc hash table worth its code, compared with a dict?
I found no correctness defect in the scanner.
Performance evidence. The PR body says query prep is faster than main on every everyday shape, but it gives no numbers or environment. .agents/architecture.md asks to record the environment with any claimed result, and it names stresshouse-benchmark-compare.yml for PR comparisons. My measurements of remove_sql_comments support the claim. Times are per call, best of 3. Environment: Python 3.12.16, Cython 3.3.1, gcc 12.2 -O3, Xeon Platinum 8175M, Linux x86_64. main is the regex at 9fa76e5. "Python fallback" is this head with _c_sqlscan = None.
| Query shape | Length | main |
This PR, compiled | This PR, Python fallback |
|---|---|---|---|---|
| Short SELECT | 42 | 2.6 us | 0.5 us | 1.7 us |
| Short SELECT, 2 comments | 55 | 4.8 us | 0.9 us | 9.3 us |
| IN list, 20k quoted values | 169k | 13.7 ms | 0.70 ms | 0.71 ms |
Same, trailing -- |
169k | 13.5 ms | 0.95 ms | 20.5 ms |
Same, Cyrillic values, trailing -- |
169k | 16.0 ms | 0.82 ms | 20.6 ms |
20k {p:String}, trailing -- |
309k | 15.8 ms | 12.0 ms | 45.8 ms |
20k $tN$, trailing -- |
169k | 8.5 ms | 3.8 ms | 131 ms |
1 MB INSERT ... VALUES, trailing -- |
998k | 80 ms | 6.4 ms | 74 ms |
Suggest adding numbers like these, or a stresshouse-benchmark-compare run, to the PR body, together with the environment.
Checked, no problem found
- Parity, compared with
_sql_opaque_regionsand with the full classification (remove_sql_commentsplus_matches_outside_quotesfor the three patterns). I ran about 1.9M random queries from a wider fragment set than the one in the test. The set had 1-, 2- and 4-bytestrkinds, lone surrogates, curly quotes, non-ASCII placeholder names (Latin letter U+00E9, superscript digit U+00B2, Roman numeral U+216B, a combining mark),{$x:T}and overlapping$a$b$tags. I also ran about 32k heredoc-heavy queries with up to 400 tags each, to reach the hash-table path, with small probe budgets so that about 30% of them returnedNone. No differences. - C level: bounds in
close_tag,find_charand the backslash skip, the output buffer size inremove_sql_comments, the load factor ingrow_tags, and that memory is freed on every exit path, includingMemoryError. - Import wiring:
_c_sqlscanis set for every entry point I tried first (clickhouse_connect,driver.binding,driver.query,driverc.sqlscan,dbapi,cc_sqlalchemy,datatypes). - Free threading: the module has no shared mutable state except the seed, which is written once at import. So
freethreading_compatible = Trueis correct. - 743 targeted unit tests pass (
test_sqlscan_parity,test_query,test_binding,test_parser,test_ctypes,test_freethreading).ruff0.15.8 andmypyreport nothing for the changed files. CI is green, except the AI gate, which waits because this is a draft, and the cloud leg, which is still pending.
Not checked: the Windows and macOS compilers, and the PyPy wheel builds. Those wheels are built only in publish.yml, and that job does not require the C extensions.
Review by Polyglot AI of 500dadd.
500dadd to
6c24028
Compare
There was a problem hiding this comment.
Auto-approved by the AI review gate on ed5bee0. Details in the approver's comment on this pull request.
AI review gate — auto-approved
History (latest 5 heads, newest first)
|
Follow-up to #1186 (#925). Moves the SQL lexer that skips comments and
quoted text into a Cython module, driverc/sqlscan.pyx. The pure Python
path stays as the fallback.
Query prep is faster than main on every everyday shape we measured.
Results match the Python path on all fuzz cases, on 3.10 to 3.14t.
Note
Low Risk
Performance optimization with strict parity fallback to the existing Python lexer; no API or wire-protocol changes.
Overview
Adds a fourth C extension,
clickhouse_connect.driverc.sqlscan, that speeds up comment stripping and regex matching outside quoted/heredoc regions during query classification.query.pyandconnect_c_modules()route plain CPythonstrqueries through the compiled path when C optimizations are enabled; PyPy, missing extensions,strsubclasses, or a heredoc probe budget exhaustion transparently fall back to the existing Python lexer inbinding.py(kept in sync via comments).The C scanner walks Unicode buffers directly, indexes
$tag$heredocs with a bounded hash table, and returnsNonewhen probing would be too expensive so behavior stays identical to Python. Docs and changelog note the C extension count; parity/fuzz subprocess tests and a freethreading check cover the new module.Reviewed by Cursor Bugbot for commit a6fbbf1. Bugbot is set up for automated code reviews on this repo. Configure here.