Skip to content

Commit c3e1714

Browse files
committed
feat: extend PostgreSQL expressions, query composition and object DDL
Expand ParserSQL's own PostgreSQL grammar with structured productions for VARIADIC arguments, array slices, ILIKE/SIMILAR TO and ESCAPE, POSITION, OVERLAY, IS JSON and ORDER BY USING in queries, aggregates and windows. Preserve bit/hex literal prefixes, field/subscript indirection and pattern precedence through SQL reconstruction. Reject incomplete binary operands, invalid fields and malformed nonassociative predicate chains. Represent nested joins, USING aliases, TABLESAMPLE/REPEATABLE, ROWS FROM, row-lock strengths/targets/wait policies and SELECT INTO explicitly. Retain the established AST representation for compatible simple joins and preserve comma-join scope for richer trees. Distinguish PostgreSQL EXCEPT query operands from ParserSQL's existing star-column exclusion extension. EXPLAIN now retains grouped options and forwards inner completion/error state instead of reporting an incomplete or failed child as successful. Add CREATE/ALTER DOMAIN, shell/composite/enum/range/base TYPE definitions, CREATE/ALTER SEQUENCE, routine properties and SET/RESET clauses, common object identity changes and additional DROP families. Validate object names, option boundaries, defaults, constraints and quoted prefixes while retaining traversable names, types, expressions and clause operands. Map PostgreSQL DefineStmt to CREATE in the development compatibility tool. Append AST node kinds without changing the 48-byte layout. Guard local execution of newly represented semantics, including PostgreSQL bit/hex literals, and keep parameterization conservative. Production parsing has no PostgreSQL runtime dependency or fallback. Benchmarks and benchmark-only documentation/configuration edits are intentionally outside this commit. Validation: - Forced make all/build-corpus-test: 1,455 passed; 37 external-backend skips. - ASan/UBSan: all 27 new tests passed, including bounded-arena sweeps. - 87 Python PostgreSQL compatibility harness tests passed. - PostgreSQL 18.4 raw AST comparisons: all 1,838 relevant object-DDL corpus statements and 405 newly covered query/DML/EXPLAIN cases matched after removing source positions; positive/negative and mutation controls passed. - Independent review confirmed pattern precedence, EXPLAIN, allocation handling, AST emission, execution guards and MySQL compatibility. The separately committed correctness snapshot records the complete corpus replay and remaining unsupported grammar; this does not claim full parity.
1 parent 91880dc commit c3e1714

18 files changed

Lines changed: 1910 additions & 46 deletions

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
5555
$(TEST_DIR)/test_pg_expressions.cpp \
5656
$(TEST_DIR)/test_pg_special_expressions.cpp \
5757
$(TEST_DIR)/test_pg_query_gaps.cpp \
58+
$(TEST_DIR)/test_pg_query_continuation.cpp \
59+
$(TEST_DIR)/test_pg_expression_continuation.cpp \
60+
$(TEST_DIR)/test_pg_object_ddl.cpp \
5861
$(TEST_DIR)/test_pg_dml_extensions.cpp \
5962
$(TEST_DIR)/test_pg_ddl.cpp \
6063
$(TEST_DIR)/test_pg_json_xml.cpp \

‎docs/postgresql-analysis-features.md‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -139,11 +139,25 @@ RETURNING and behavior options, table columns, serialization and default
139139
expressions. Their new node kinds support traversal and SQL reconstruction;
140140
parameterization remains conservative about unsupported contexts.
141141

142-
Coverage is still a subset of PostgreSQL. Remaining examples include CREATE
143-
TYPE/DOMAIN/SEQUENCE, embedded CREATE SCHEMA elements, CREATE DATABASE options,
144-
most non-table ALTER commands, unquoted SQL function bodies, exclusion
145-
constraints, less common DDL options, assignment slices,
146-
full ON CONFLICT index-inference options, and `IS JSON` predicates. PostgreSQL remains responsible
142+
Additional expression forms include `VARIADIC` arguments, array slices,
143+
`ILIKE`/`SIMILAR TO` and `ESCAPE`, SQL `POSITION`/`OVERLAY`, `IS JSON` predicates,
144+
and `ORDER BY ... USING` in queries, aggregates and windows. PostgreSQL bit and
145+
hexadecimal literals preserve their prefixes. Nested joins, join USING aliases,
146+
`TABLESAMPLE ... REPEATABLE`, `ROWS FROM`, row-lock strengths/targets/wait policies,
147+
and `SELECT INTO` have explicit AST nodes. EXPLAIN retains grouped options and
148+
propagates its inner statement's completion and error status. Unsupported local
149+
execution is guarded; parameterization conservatively rejects the new contexts.
150+
151+
Object DDL includes CREATE/ALTER DOMAIN, shell/composite/enum/range/base TYPE
152+
forms, CREATE/ALTER SEQUENCE, routine properties and SET/RESET clauses,
153+
object renaming/ownership/schema changes, and additional DROP families.
154+
Defaults, constraints, names and options remain traversable AST components.
155+
156+
Coverage is still a subset of PostgreSQL. Remaining examples include embedded
157+
CREATE SCHEMA elements, CREATE DATABASE options, other object-specific ALTER
158+
commands, aggregate/operator definitions, roles, comments, cursors, unquoted SQL
159+
function bodies, exclusion constraints, less common DDL options, assignment
160+
slices and full ON CONFLICT index-inference options. PostgreSQL remains responsible
147161
for name resolution, types, privileges and semantic checks. Corpus acceptance
148162
alone does not establish grammar parity or round-trip equivalence for every
149163
accepted statement.
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# PostgreSQL grammar continuation implementation plan
2+
3+
> **For agentic workers:** Use superpowers:subagent-driven-development for independent expression and DDL work. Parent owns query integration and final verification.
4+
5+
**Goal:** Extend the native parser against the remaining PostgreSQL corpus, prioritizing SELECT composition and common object DDL.
6+
**Architecture:** Preserve the current tokenizer/arena/AST architecture; add validated productions and explicit nodes where existing AST shapes cannot preserve semantics. Unsupported local engine behavior must be rejected before execution. Use the pinned PostgreSQL parser only as a development oracle.
7+
**Tech stack:** C++17, GoogleTest, Python compatibility harness, PostgreSQL 18.4/libpg_query pinned source.
8+
**Spec:** Continues the user-approved native grammar work documented in 2026-09-26-pg-grammar-gaps.md and the user's request to continue filling remaining gaps.
9+
10+
## Global constraints
11+
12+
- Continue branch feat/postgresql-coverage-and-ast-apis and update PR #67 with detailed commits.
13+
- Do not commit benchmarks, benchmark README/.gitignore edits, or generated binaries.
14+
- Preserve MySQL behavior, appended enum values, and the 48-byte AST layout.
15+
- No runtime PostgreSQL dependency or unvalidated opaque-tail acceptance.
16+
- Base 91880dc: 43,373 deep parses of 51,415 pinned corpus statements; preserve prior accepted coverage.
17+
18+
## Review focus
19+
20+
- Nested joins retain association and ON/USING ownership through emission.
21+
- Keywords used as names and quoted names remain distinguishable from grammar.
22+
- New syntax operands stay traversable; syntax constants do not become bind values.
23+
- Malformed separators/clauses and exhausted arenas cannot produce a truncated successful AST.
24+
- Planner guards reject new semantics rather than executing a simplified statement.
25+
26+
## Tasks
27+
28+
- [x] Expressions: tests/test_pg_expression_continuation.cpp; expression_parser.h and owned shared markers. Start with failing positive/negative fixtures for VARIADIC calls, array slices, ILIKE/SIMILAR TO/ESCAPE, POSITION/OVERLAY and IS JSON. Examples: SELECT a[:3], b[1:]; SELECT POSITION('a' IN s); SELECT OVERLAY(s PLACING 'x' FROM 2 FOR 1); SELECT f(VARIADIC ARRAY[1,2]); SELECT doc IS JSON OBJECT WITH UNIQUE KEYS. Compare emitted PostgreSQL AST, reject malformed bounds/operands and guard local execution.
29+
- [x] DDL: tests/test_pg_object_ddl.cpp; pg_ddl_parser.h. Start with ALTER FUNCTION f(int) PARALLEL SAFE, CREATE DOMAIN d AS int CHECK(VALUE>0), CREATE TYPE t AS ENUM('a','b'), CREATE TYPE t AS(a int,b text), CREATE SEQUENCE s START WITH 3, ALTER SEQUENCE s RESTART WITH 4. Add routine options/ownership/schema/renaming, common ALTER domain/type/sequence and validated remaining DROP families. Keep names/types/expressions structural. Verify positive/negative/oracle cases.
30+
- [x] Query composition: tests/test_pg_query_continuation.cpp; table_ref_parser.h, select_parser.h, compound_query_parser.h and pg_query_clauses.h. Tests first for parenthesized/nested joins, TABLESAMPLE/REPEATABLE, ROWS FROM, ORDER BY USING operators, row-lock strength/OF/wait policies and SELECT INTO. Preserve EXPLAIN options and inner completion while validating new query forms inside EXPLAIN. Preserve existing simple-join execution representation where possible; use appended guarded nodes for richer forms. Validate PostgreSQL AST equivalence and malformed syntax.
31+
- [ ] Integration: register tests, run independent cross-reviews, force full build and corpus runner, focused ASan/UBSan, oracle comparisons and Python harness. Replay all 51,415 statements and inspect losses/status transitions. Commit implementation first, refresh correctness snapshot against that commit, verify full compatibility workflow, commit snapshot and push/update PR #67. Report concrete remaining grammar variants without claiming complete PostgreSQL parity.
32+
33+
Implementation workers must not commit, run full make or change shared files outside their owned markers. Parent consolidates commits and verifies the final tree.

‎include/sql_engine/plan_builder.h‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,29 @@ class PlanBuilder {
6969
bool dml_values = false) {
7070
using sql_parser::NodeType;
7171
switch (node->type) {
72+
// PG_CONT_EXPRESSION_GUARD
73+
case NodeType::NODE_LITERAL_BIT:
74+
case NodeType::NODE_LITERAL_HEX:
75+
if constexpr (D == sql_parser::Dialect::PostgreSQL) return true;
76+
break;
77+
case NodeType::NODE_PG_VARIADIC_ARGUMENT:
78+
case NodeType::NODE_PG_ARRAY_SLICE:
79+
case NodeType::NODE_PG_PATTERN_PREDICATE:
80+
case NodeType::NODE_PG_POSITION:
81+
case NodeType::NODE_PG_OVERLAY:
82+
case NodeType::NODE_PG_JSON_PREDICATE:
83+
return true;
84+
// PG_CONT_QUERY_GUARD
85+
case NodeType::NODE_PG_EXPLAIN_OPTION:
86+
case NodeType::NODE_PG_JOIN_TREE:
87+
case NodeType::NODE_PG_TABLE_GROUP:
88+
case NodeType::NODE_PG_JOIN_USING:
89+
case NodeType::NODE_PG_TABLESAMPLE:
90+
case NodeType::NODE_PG_ROWS_FROM:
91+
case NodeType::NODE_PG_SORT_USING:
92+
case NodeType::NODE_PG_SELECT_INTO:
93+
case NodeType::NODE_PG_ROW_LOCK:
94+
return true;
7295
// PG_GAPS_EXPRESSION_GUARD
7396
case NodeType::NODE_PG_EXTRACT:
7497
case NodeType::NODE_PG_SUBSTRING:

‎include/sql_parser/common.h‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,24 @@ enum class NodeType : uint16_t {
310310
// PG_GAPS_JSON_XML_NODES
311311
NODE_PG_JSON_XML,
312312
NODE_PG_JSON_XML_SYNTAX,
313+
// PG_CONT_EXPRESSION_NODES
314+
NODE_PG_VARIADIC_ARGUMENT, // one value or named-argument child; final call argument
315+
NODE_PG_ARRAY_SLICE, // base, optional lower/upper; flags 1/2 mark bounds
316+
NODE_PG_PATTERN_PREDICATE, // value=operator; subject, pattern, optional escape
317+
NODE_PG_POSITION, // needle, haystack
318+
NODE_PG_OVERLAY, // source, replacement, start, optional length; flag 1=plain call
319+
NODE_PG_JSON_PREDICATE, // validated IS [NOT] JSON suffix; one expression child
320+
// PG_CONT_QUERY_NODES
321+
NODE_PG_EXPLAIN_OPTION,
322+
NODE_PG_JOIN_TREE,
323+
NODE_PG_TABLE_GROUP,
324+
NODE_PG_JOIN_USING,
325+
NODE_PG_TABLESAMPLE,
326+
NODE_PG_ROWS_FROM,
327+
NODE_PG_SORT_USING,
328+
NODE_PG_SELECT_INTO,
329+
NODE_PG_ROW_LOCK,
330+
313331
};
314332

315333
} // namespace sql_parser

‎include/sql_parser/compound_query_parser.h‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,14 +44,14 @@ class CompoundQueryParser {
4444
result->add_child(order);
4545
}
4646
if constexpr (D == Dialect::PostgreSQL) {
47-
if (!PgQueryClauses<D>(tok_, arena_, expr_parser_).pagination(result)) return nullptr;
47+
if (!PgQueryClauses<D>(tok_, arena_, expr_parser_).tail(result)) return nullptr;
4848
} else if (tok_.peek().type == TokenType::TK_LIMIT) {
4949
tok_.skip();
5050
AstNode* limit = parse_limit(require_operands);
5151
if (!limit || !limit->first_child) return nullptr;
5252
result->add_child(limit);
5353
}
54-
if (result->type == NodeType::NODE_SELECT_STMT &&
54+
if (D == Dialect::MySQL && result->type == NodeType::NODE_SELECT_STMT &&
5555
tok_.peek().type == TokenType::TK_FOR) {
5656
tok_.skip();
5757
AstNode* lock = make_node(arena_, NodeType::NODE_LOCKING_CLAUSE);
@@ -222,15 +222,22 @@ class CompoundQueryParser {
222222
}
223223

224224
if constexpr (D == Dialect::PostgreSQL) {
225+
if (dir.type == TokenType::TK_USING) {
226+
auto* op = expressions.parse_sort_operator();
227+
if (!op) return nullptr;
228+
item->add_child(op);
229+
}
225230
if (ExpressionParser<D>::keyword(tok_.peek(), "NULLS")) {
226231
tok_.skip();
227232
Token placement = tok_.peek();
228233
bool first = ExpressionParser<D>::keyword(placement, "FIRST");
229234
if (!first && !ExpressionParser<D>::keyword(placement, "LAST")) return nullptr;
230235
tok_.skip();
231236
item->flags |= FLAG_ORDER_NULLS;
232-
item->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER,
233-
first ? StringRef{"NULLS FIRST", 11} : StringRef{"NULLS LAST", 10}));
237+
auto* nulls = make_node(arena_, NodeType::NODE_IDENTIFIER,
238+
first ? StringRef{"NULLS FIRST", 11} : StringRef{"NULLS LAST", 10});
239+
if (!nulls) return nullptr;
240+
item->add_child(nulls);
234241
}
235242
}
236243
order_by->add_child(item);

‎include/sql_parser/emitter.h‎

Lines changed: 117 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,56 @@ class Emitter {
197197
if (mode_ == EmitMode::DIGEST) { sb_.append_char('?'); break; }
198198
emit_string_literal(node); break;
199199

200+
// PG_CONT_EXPRESSION_DISPATCH
201+
case NodeType::NODE_PG_VARIADIC_ARGUMENT:
202+
sb_.append("VARIADIC "); emit_node(node->first_child); break;
203+
case NodeType::NODE_PG_ARRAY_SLICE:
204+
case NodeType::NODE_PG_PATTERN_PREDICATE:
205+
case NodeType::NODE_PG_POSITION:
206+
case NodeType::NODE_PG_OVERLAY:
207+
case NodeType::NODE_PG_JSON_PREDICATE:
208+
emit_pg_continuation_expression(node); break;
209+
// PG_CONT_QUERY_DISPATCH
210+
case NodeType::NODE_PG_EXPLAIN_OPTION:
211+
emit_value(node);
212+
if (node->first_child) { sb_.append_char(' '); emit_node(node->first_child); }
213+
break;
214+
case NodeType::NODE_PG_JOIN_TREE: emit_pg_join_tree(node); break;
215+
case NodeType::NODE_PG_TABLE_GROUP:
216+
sb_.append_char('('); emit_node(node->first_child); sb_.append_char(')'); break;
217+
case NodeType::NODE_PG_JOIN_USING:
218+
sb_.append("USING (");
219+
for (const auto* col = node->first_child; col; col = col->next_sibling) {
220+
if (col != node->first_child) sb_.append(", ");
221+
emit_node(col);
222+
}
223+
sb_.append_char(')');
224+
if (!node->value().empty()) { sb_.append(" AS "); emit_value(node); }
225+
break;
226+
case NodeType::NODE_PG_TABLESAMPLE: emit_pg_tablesample(node); break;
227+
case NodeType::NODE_PG_ROWS_FROM:
228+
sb_.append("ROWS FROM (");
229+
for (const auto* item = node->first_child; item; item = item->next_sibling) {
230+
if (item != node->first_child) sb_.append(", ");
231+
emit_node(item);
232+
}
233+
sb_.append_char(')'); break;
234+
case NodeType::NODE_PG_SORT_USING:
235+
sb_.append("USING "); emit_value(node); break;
236+
case NodeType::NODE_PG_SELECT_INTO:
237+
sb_.append(" INTO "); emit_value(node); emit_node(node->first_child); break;
238+
case NodeType::NODE_PG_ROW_LOCK:
239+
sb_.append(" FOR "); emit_value(node);
240+
if (node->first_child) {
241+
sb_.append(" OF ");
242+
for (const auto* rel = node->first_child; rel; rel = rel->next_sibling) {
243+
if (rel != node->first_child) sb_.append(", ");
244+
emit_node(rel);
245+
}
246+
}
247+
if (node->flags == 1) sb_.append(" NOWAIT");
248+
if (node->flags == 2) sb_.append(" SKIP LOCKED");
249+
break;
200250
// PG_GAPS_EXPRESSION_DISPATCH
201251
case NodeType::NODE_PG_EXTRACT:
202252
case NodeType::NODE_PG_SUBSTRING:
@@ -248,6 +298,63 @@ class Emitter {
248298
}
249299
}
250300

301+
// PG_CONT_EXPRESSION_EMITTER
302+
void emit_pg_continuation_expression(const AstNode* node) {
303+
const AstNode* first = node->first_child;
304+
if (node->type == NodeType::NODE_PG_ARRAY_SLICE) {
305+
emit_node(first); sb_.append_char('[');
306+
const AstNode* bound = first ? first->next_sibling : nullptr;
307+
if (node->flags & 1) { emit_node(bound); bound = bound ? bound->next_sibling : nullptr; }
308+
sb_.append_char(':');
309+
if (node->flags & 2) emit_node(bound);
310+
sb_.append_char(']');
311+
} else if (node->type == NodeType::NODE_PG_PATTERN_PREDICATE) {
312+
emit_node(first); sb_.append_char(' '); emit_value(node); sb_.append_char(' ');
313+
const AstNode* pattern = first ? first->next_sibling : nullptr;
314+
emit_node(pattern);
315+
if (pattern && pattern->next_sibling) {
316+
sb_.append(" ESCAPE "); emit_node(pattern->next_sibling);
317+
}
318+
} else if (node->type == NodeType::NODE_PG_JSON_PREDICATE) {
319+
emit_node(first); sb_.append_char(' '); emit_value(node);
320+
} else if (node->type == NodeType::NODE_PG_POSITION) {
321+
sb_.append("POSITION("); emit_node(first); sb_.append(" IN ");
322+
emit_node(first ? first->next_sibling : nullptr); sb_.append_char(')');
323+
} else {
324+
sb_.append("OVERLAY(");
325+
if (node->flags & 1) emit_list(node, ", ");
326+
else {
327+
const AstNode* replacement = first ? first->next_sibling : nullptr;
328+
const AstNode* start = replacement ? replacement->next_sibling : nullptr;
329+
emit_node(first); sb_.append(" PLACING "); emit_node(replacement);
330+
sb_.append(" FROM "); emit_node(start);
331+
if (start && start->next_sibling) { sb_.append(" FOR "); emit_node(start->next_sibling); }
332+
}
333+
sb_.append_char(')');
334+
}
335+
}
336+
// PG_CONT_QUERY_EMITTER
337+
void emit_pg_join_tree(const AstNode* node) {
338+
const auto* left = node->first_child;
339+
const auto* right = left->next_sibling;
340+
const auto* qual = right->next_sibling;
341+
sb_.append_char('('); emit_node(left); sb_.append_char(' ');
342+
emit_value(node); sb_.append_char(' '); emit_node(right);
343+
if (qual) {
344+
sb_.append(qual->type == NodeType::NODE_PG_JOIN_USING ? " " : " ON ");
345+
emit_node(qual);
346+
}
347+
sb_.append_char(')');
348+
}
349+
void emit_pg_tablesample(const AstNode* node) {
350+
const auto* name = node->first_child;
351+
const auto* args = name->next_sibling;
352+
sb_.append(" TABLESAMPLE "); emit_node(name); emit_node(args);
353+
if (args->next_sibling) {
354+
sb_.append(" REPEATABLE ("); emit_node(args->next_sibling); sb_.append_char(')');
355+
}
356+
}
357+
251358
// PG_GAPS_EXPRESSION_EMITTER
252359
void emit_pg_special_expression(const AstNode* node) {
253360
const AstNode* first = node->first_child;
@@ -1220,7 +1327,7 @@ class Emitter {
12201327
}
12211328
}
12221329

1223-
if (!has_inner_stmt && !has_options) {
1330+
if (D == Dialect::MySQL && !has_inner_stmt && !has_options) {
12241331
sb_.append("DESCRIBE");
12251332
} else {
12261333
sb_.append("EXPLAIN");
@@ -1233,6 +1340,11 @@ class Emitter {
12331340
}
12341341

12351342
void emit_explain_options(const AstNode* node) {
1343+
if constexpr (D == Dialect::PostgreSQL) {
1344+
if (node->flags & 1) {
1345+
sb_.append_char('('); emit_list(node, ", "); sb_.append_char(')'); return;
1346+
}
1347+
}
12361348
bool first = true;
12371349
for (const AstNode* child = node->first_child; child; child = child->next_sibling) {
12381350
if (!first) sb_.append_char(' ');
@@ -1687,7 +1799,10 @@ class Emitter {
16871799
void emit_field_access(const AstNode* node) {
16881800
const AstNode* expr = node->first_child;
16891801
const AstNode* field = expr ? expr->next_sibling : nullptr;
1690-
if (expr && expr->type == NodeType::NODE_EXPRESSION) emit_node(expr);
1802+
if (expr && (expr->type == NodeType::NODE_EXPRESSION ||
1803+
(D == Dialect::PostgreSQL && (expr->type == NodeType::NODE_COLUMN_REF ||
1804+
expr->type == NodeType::NODE_QUALIFIED_NAME || expr->type == NodeType::NODE_ARRAY_SUBSCRIPT ||
1805+
expr->type == NodeType::NODE_PG_ARRAY_SLICE || expr->type == NodeType::NODE_FIELD_ACCESS)))) emit_node(expr);
16911806
else {
16921807
sb_.append_char('(');
16931808
if (expr) emit_node(expr);

0 commit comments

Comments
 (0)