feat: expand native PostgreSQL expressions, queries, DML and DDL

Replace shallow handling of common PostgreSQL statements with validated,
traversable AST productions. Keep all production parsing inside ParserSQL;
libpg_query remains a test oracle, with no runtime dependency or fallback.

Add EXTRACT, SQL SUBSTRING/TRIM, time-zone expressions, interval qualifiers,
qualified operators, expression-valued type modifiers, ARRAY subqueries,
IS DISTINCT FROM, quantified comparison operands and NORMALIZE. Preserve
special datetime precision and MERGE_ACTION grammar while rejecting reserved
function names and malformed delimiters or wildcard operands.

Add grouping sets, ROLLUP/CUBE, grouping quantifiers, OFFSET/LIMIT ordering,
FETCH, table-function ordinality and typed output columns. Preserve ONLY and
explicit inheritance sources, nested WITH operands, quoted names and CASE
branches in reconstructed SQL. Enforce restricted FETCH/OFFSET ROWS operands.

Parse MERGE actions and RETURNING, modifying CTE bodies and main statements,
SEARCH/CYCLE, INSERT OVERRIDING and compound sources, assignment indirection,
and RETURNING options. Guard unsupported local DML/query semantics before
execution and propagate arena exhaustion instead of dropping syntax.

Add structured common table/schema/database creation, ALTER TABLE, DROP,
index/view/function/procedure/trigger definitions, privileges, TRUNCATE and
maintenance statements. Add explicit SQL/JSON and XML constructors, query
functions, aggregates and table columns, including behavior/default clauses.
Unsupported variants remain errors or incomplete input, not opaque AST tails.

Append statement/node kinds without changing the existing AST layout. Update
compatibility type mappings, focused positive/negative/AST tests, allocation
regressions and coverage documentation. Leave benchmark code/results and
benchmark-only README/.gitignore edits outside this commit.

Validation: force-rebuilt full GoogleTest suite and corpus runner, focused
ASan/UBSan grammar tests, PostgreSQL 18.4 parse/emit AST comparisons, malformed
input mutation checks and a replay of the pinned 51,415-statement corpus.

Author: renecannao

Makefile

@@ -53,6 +53,11 @@ TEST_SRCS = $(TEST_DIR)/test_main.cpp \
             $(TEST_DIR)/test_compound.cpp \
             $(TEST_DIR)/test_pg_query_features.cpp \
             $(TEST_DIR)/test_pg_expressions.cpp \
+            $(TEST_DIR)/test_pg_special_expressions.cpp \
+            $(TEST_DIR)/test_pg_query_gaps.cpp \
+            $(TEST_DIR)/test_pg_dml_extensions.cpp \
+            $(TEST_DIR)/test_pg_ddl.cpp \
+            $(TEST_DIR)/test_pg_json_xml.cpp \
             $(TEST_DIR)/test_pg_utilities.cpp \
             $(TEST_DIR)/test_parse_all.cpp \
             $(TEST_DIR)/test_ast_utilities.cpp \

docs/postgresql-analysis-features.md

@@ -23,8 +23,7 @@ validation and other semantic checks.
 
 The new clauses remain bounded by the existing expression grammar. PostgreSQL
 subqueries now share compound-query parsing, including SELECT, TABLE, VALUES and
-WITH operands in derived tables and LATERAL. Interval-valued frame offsets remain
-outside the supported expression subset.
+WITH operands in derived tables and LATERAL. Interval-valued frame offsets are supported through structured interval literals.
 
 Window expressions previously disappeared from emitted SQL and digests because
 their AST nodes had no emitter handlers. They now appear correctly. Digest
@@ -66,11 +65,10 @@ names, operators or function overloads, and does not validate catalog semantics.
 Consumers must also handle the appended node/token kinds. `?` is a PostgreSQL
 operator; MySQL continues to tokenize it as an anonymous bind marker.
 
-This phase does not cover qualified `OPERATOR(schema.op)` syntax, arbitrary
-expression-valued type modifiers, every keyword in an unquoted type-name
-position, interval prefix-literal qualifiers, or the remaining full expression
-and DDL grammar. These additions improve the native ParserSQL implementation;
-there is no PostgreSQL runtime fallback or production library dependency.
+Qualified prefix and binary `OPERATOR(schema.op)` syntax, expression-valued
+type modifiers and interval prefix-literal qualifiers are also supported.
+These additions improve the native ParserSQL implementation; there is no
+PostgreSQL runtime fallback or production library dependency.
 
 ## Qualified calls, aggregates and CTEs
 
@@ -99,12 +97,57 @@ recursive/nested CTEs, CTE output-column lists, materialization hints and
 VALUES/TABLE query bodies before CTE materialization. These additions do not extend CTE parameterization, which
 continues to return an unsupported-context error.
 
-Remaining gaps include data-modifying CTEs and main statements, SEARCH/CYCLE,
-special EXTRACT/SUBSTRING forms, AT TIME ZONE, advanced grouping, and broader
-SQL/JSON, XML and DDL grammar. See PostgreSQL's
+See PostgreSQL's
 [aggregate syntax](https://www.postgresql.org/docs/18/sql-expressions.html#SYNTAX-AGGREGATES)
 and [WITH queries](https://www.postgresql.org/docs/18/queries-with.html).
 
+## Further native grammar coverage
+
+Common PostgreSQL expressions now include `EXTRACT`, SQL `SUBSTRING`, `TRIM`,
+`AT TIME ZONE`, `AT LOCAL`, interval literal precision/field qualifiers,
+`IS [NOT] DISTINCT FROM`, `ARRAY(query)`, quantified comparison operands
+(`ANY`/`ALL`/`SOME`) and Unicode `NORMALIZE`. Operands remain AST children.
+Qualified operators and syntax modifiers retain their spelling during emission.
+
+Queries support `GROUPING SETS`, `ROLLUP`, `CUBE`, grouping `ALL`/`DISTINCT`,
+`LIMIT ALL`, either LIMIT/OFFSET order, offset-only pagination and
+`FETCH FIRST/NEXT ... ROWS ONLY/WITH TIES`. Table functions support
+`WITH ORDINALITY` and typed output-column definitions. `ONLY` relation sources
+and parenthesized WITH operands are retained. New constructs that the local
+engine cannot execute are rejected by its planner.
+
+`MERGE` has structured target/source, join condition, match branches, optional
+conditions, UPDATE/DELETE/INSERT/DO NOTHING actions and RETURNING. CTE bodies
+and main statements may use supported INSERT/UPDATE/DELETE/MERGE forms.
+Recursive `SEARCH` and `CYCLE` clauses, INSERT `OVERRIDING`, compound query
+sources, assignment indirection and RETURNING options retain their structure.
+The local DML planner rejects these new semantics before execution.
+
+Common DDL now has validated productions for CREATE TABLE (including CTAS and
+partition bounds), basic CREATE SCHEMA/DATABASE, ALTER TABLE actions, DROP
+families, indexes,
+views/materialized views, functions/procedures, triggers, privileges, role
+membership, TRUNCATE, VACUUM and ANALYZE. Names, expressions, lists and clauses
+are traversable; type names and fixed syntax are validated leaves. Quoted
+function bodies remain string literals. This is syntax parsing, not catalog
+validation or execution, and unsupported options must leave an error or
+incomplete input rather than being accepted as an arbitrary tail.
+
+SQL/JSON and XML productions retain constructors, JSON_ARRAYAGG/JSON_OBJECTAGG,
+query/value/exists calls,
+RETURNING and behavior options, table columns, serialization and default
+expressions. Their new node kinds support traversal and SQL reconstruction;
+parameterization remains conservative about unsupported contexts.
+
+Coverage is still a subset of PostgreSQL. Remaining examples include CREATE
+TYPE/DOMAIN/SEQUENCE, embedded CREATE SCHEMA elements, CREATE DATABASE options,
+most non-table ALTER commands, unquoted SQL function bodies, exclusion
+constraints, less common DDL options, assignment slices,
+full ON CONFLICT index-inference options, and `IS JSON` predicates. PostgreSQL remains responsible
+for name resolution, types, privileges and semantic checks. Corpus acceptance
+alone does not establish grammar parity or round-trip equivalence for every
+accepted statement.
+
 ## Multiple statements
 
 ```cpp

docs/superpowers/plans/2026-09-26-pg-grammar-gaps.md

@@ -0,0 +1,62 @@
+# Remaining PostgreSQL grammar gaps
+
+The user authorized filling the remaining gaps in ParserSQL's own grammar.
+Continue on feat/postgresql-coverage-and-ast-apis and update PR #67. Keep detailed
+commits and leave all benchmark work uncommitted. Baseline: de06d72, 27,014 of
+51,415 pinned PostgreSQL 18.4 statements completely parsed.
+
+## Implementation groups
+
+1. Expressions/types: special EXTRACT and SUBSTRING syntax, AT TIME ZONE,
+   interval literals and qualifiers, qualified OPERATOR and type modifiers.
+   Preserve structural operands and source spelling; verify precedence,
+   malformed input and equivalent PostgreSQL ASTs after SQL emission.
+2. Query clauses: GROUPING SETS/ROLLUP/CUBE, both LIMIT/OFFSET orders and FETCH,
+   WITH ORDINALITY and typed table-function output columns. Expressions remain
+   AST children; syntax modifiers must not become bind parameters.
+3. DML/CTEs: MERGE branches/actions/RETURNING, WITH preceding DML and modifying
+   CTE bodies, SEARCH/CYCLE, and remaining practical INSERT variants. Distinguish
+   query subexpressions from DML bodies; reject unsupported local execution.
+4. DDL/utility: structure common CREATE/ALTER/DROP tables, views, functions,
+   indexes and triggers, privilege and maintenance forms. Do not claim complete
+   parsing by storing an unvalidated tail or skipping arbitrary tokens.
+5. SQL/JSON and XML: add explicit productions for structured constructors,
+   query/value functions, JSON_TABLE/XMLTABLE and serialization options. Keep
+   default expressions traversable and preserve server-defined syntax.
+6. Integration: test first in each group, obtain independent review, force the
+   full C++ rebuild, run sanitizers and pinned oracle AST comparisons, inspect
+   every compatibility transition, refresh correctness fixtures, then commit
+   and publish the PR update. Report remaining variants honestly.
+
+## Coordination and review
+
+Use the subagent-development skill for independent grammar implementations.
+Each group owns its parser headers and tests. Shared AST/emitter/planner edits
+must be confined to distinct marked sections; no global rewrites. The parent
+owns integration, query clauses and final corpus validation. Existing MySQL
+behavior and the 48-byte AST layout remain unchanged. Append node/statement
+kinds. No production PostgreSQL dependency or parser fallback.
+
+Review focus: syntax rejection versus semantic validation; SQL emission
+preserving names, precedence and options; valid nesting; parameterization
+contexts; local execution refusing unsupported semantics before side effects.
+Full PostgreSQL grammar parity must not be claimed from corpus counts alone.
+
+## Completed verification
+
+Implemented all five grammar groups with focused malformed-input and AST tests.
+Independent reviews covered DML/DDL and expressions/query clauses; confirmed
+findings were fixed and rechecked. ASan/UBSan passes all 47 new grammar tests,
+including arena-exhaustion regressions. The forced full rebuild passes 1,428
+tests with 37 backend-dependent tests skipped. The compatibility harness passes
+87 tests, and 215 representative PostgreSQL 18.4 AST round trips match after
+removing source-position fields.
+
+Pinned corpus replay: 43,373/51,415 deeply parsed, versus 27,014 before this
+phase; zero baseline regressions. All 1,274 former classification-only cases
+now have complete ASTs. Reviewed 5,919 changes between unsupported categories:
+5,165 trailing-input to error, 723 type-mismatch to error, and 31 trailing-input
+to partial. These reflect stricter productions rather than lost accepted SQL;
+unsupported CREATE/ALTER and expression variants remain documented limitations.
+The generated correctness snapshot is refreshed separately after the code commit
+so its parser revision identifies the implementation being measured.

include/sql_engine/dml_plan_builder.h

@@ -8,6 +8,7 @@
 
 #include "sql_engine/plan_node.h"
 #include "sql_engine/catalog.h"
+#include "sql_engine/plan_builder.h"
 #include "sql_parser/ast.h"
 #include "sql_parser/common.h"
 #include "sql_parser/arena.h"
@@ -23,7 +24,7 @@ class DmlPlanBuilder {
         : catalog_(catalog), arena_(arena) {}
 
     PlanNode* build(const sql_parser::AstNode* stmt_ast) {
-        if (!stmt_ast) return nullptr;
+        if (!supported(stmt_ast)) return nullptr;
         switch (stmt_ast->type) {
             case sql_parser::NodeType::NODE_INSERT_STMT:
                 return build_insert(stmt_ast);
@@ -37,7 +38,7 @@ class DmlPlanBuilder {
     }
 
     PlanNode* build_insert(const sql_parser::AstNode* insert_ast) {
-        if (!insert_ast) return nullptr;
+        if (!supported(insert_ast)) return nullptr;
 
         PlanNode* node = make_plan_node(arena_, PlanNodeType::INSERT_PLAN);
         if (!node) return nullptr;
@@ -104,7 +105,7 @@ class DmlPlanBuilder {
     }
 
     PlanNode* build_update(const sql_parser::AstNode* update_ast) {
-        if (!update_ast) return nullptr;
+        if (!supported(update_ast)) return nullptr;
 
         PlanNode* node = make_plan_node(arena_, PlanNodeType::UPDATE_PLAN);
         if (!node) return nullptr;
@@ -161,7 +162,7 @@ class DmlPlanBuilder {
     }
 
     PlanNode* build_delete(const sql_parser::AstNode* delete_ast) {
-        if (!delete_ast) return nullptr;
+        if (!supported(delete_ast)) return nullptr;
 
         PlanNode* node = make_plan_node(arena_, PlanNodeType::DELETE_PLAN);
         if (!node) return nullptr;
@@ -201,6 +202,32 @@ class DmlPlanBuilder {
     }
 
 private:
+    static bool supported(const sql_parser::AstNode* ast) {
+        if (!ast) return false;
+        if constexpr (D == sql_parser::Dialect::PostgreSQL) {
+            if (!PlanBuilder<D>::supports_dml_features(ast) || unsupported_dml(ast)) return false;
+        }
+        return true;
+    }
+
+    static bool unsupported_dml(const sql_parser::AstNode* ast) {
+        using sql_parser::NodeType;
+        if (ast->type == NodeType::NODE_CTE || ast->type == NodeType::NODE_COMPOUND_QUERY ||
+            ast->type == NodeType::NODE_ON_CONFLICT ||
+            (ast->type == NodeType::NODE_IDENTIFIER && !(ast->flags & sql_parser::FLAG_IDENT_DELIMITED) &&
+             ast->value().equals_ci("DEFAULT", 7))) return true;
+        if (ast->type == NodeType::NODE_UPDATE_SET_ITEM && ast->first_child &&
+            ast->first_child->type != NodeType::NODE_IDENTIFIER &&
+            ast->first_child->type != NodeType::NODE_COLUMN_REF) return true;
+        if (ast->type == NodeType::NODE_INSERT_COLUMNS) {
+            for (const auto* col = ast->first_child; col; col = col->next_sibling)
+                if (col->type != NodeType::NODE_IDENTIFIER) return true;
+        }
+        for (const auto* child = ast->first_child; child; child = child->next_sibling)
+            if (unsupported_dml(child)) return true;
+        return false;
+    }
+
     const Catalog& catalog_;
     sql_parser::Arena& arena_;
 

include/sql_engine/plan_builder.h

@@ -56,13 +56,64 @@ class PlanBuilder {
         return ast && !has_unsupported_query_feature(ast);
     }
 
+    // DML owns its VALUES rows; all expression/query restrictions still apply.
+    static bool supports_dml_features(const sql_parser::AstNode* ast) {
+        return ast && !has_unsupported_query_feature(ast, true, true);
+    }
+
 private:
     const Catalog& catalog_;
     sql_parser::Arena& arena_;
 
-    static bool has_unsupported_query_feature(const sql_parser::AstNode* node, bool root = true) {
+    static bool has_unsupported_query_feature(const sql_parser::AstNode* node, bool root = true,
+                                               bool dml_values = false) {
         using sql_parser::NodeType;
         switch (node->type) {
+            // PG_GAPS_EXPRESSION_GUARD
+            case NodeType::NODE_PG_EXTRACT:
+            case NodeType::NODE_PG_SUBSTRING:
+            case NodeType::NODE_PG_TIME_ZONE:
+            case NodeType::NODE_PG_INTERVAL:
+            case NodeType::NODE_PG_TRIM:
+            case NodeType::NODE_PG_ARRAY_QUERY:
+            case NodeType::NODE_PG_QUANTIFIED_OPERAND:
+            case NodeType::NODE_PG_NORMALIZE:
+                return true;
+            // PG_GAPS_DML_GUARD
+            case NodeType::NODE_MERGE_STMT:
+            case NodeType::NODE_CTE_SEARCH:
+            case NodeType::NODE_CTE_CYCLE:
+            case NodeType::NODE_PG_DML_CLAUSE:
+            case NodeType::NODE_PG_RETURNING_OPTIONS:
+                return true;
+            case NodeType::NODE_INSERT_STMT:
+            case NodeType::NODE_UPDATE_STMT:
+            case NodeType::NODE_DELETE_STMT:
+                if (!root) return true; // A modifying CTE must never be materialized as a query.
+                break;
+            // PG_GAPS_DDL_GUARD
+            case NodeType::NODE_PG_DDL_STMT:
+            case NodeType::NODE_PG_DDL_CLAUSE:
+            case NodeType::NODE_PG_DDL_LIST:
+            case NodeType::NODE_PG_DDL_SYNTAX:
+                return true;
+            // PG_GAPS_QUERY_GUARD
+            case NodeType::NODE_TABLE_REF:
+                if (node->flags & (sql_parser::FLAG_TABLE_ONLY | sql_parser::FLAG_TABLE_INHERIT)) return true;
+                break;
+            case NodeType::NODE_GROUPING_SET:
+            case NodeType::NODE_OFFSET_CLAUSE:
+            case NodeType::NODE_FETCH_CLAUSE:
+            case NodeType::NODE_ORDINALITY:
+            case NodeType::NODE_FUNCTION_COLUMN:
+                return true;
+            case NodeType::NODE_GROUP_BY_CLAUSE:
+                if (!node->value().empty()) return true;
+                break;
+            // PG_GAPS_JSON_XML_GUARD
+            case NodeType::NODE_PG_JSON_XML:
+            case NodeType::NODE_PG_JSON_XML_SYNTAX:
+                return true;
             case NodeType::NODE_CTE:
                 // Nested WITH requires its own materialization scope, which
                 // neither build_cte nor the subquery executor implements.
@@ -74,8 +125,10 @@ class PlanBuilder {
                 break;
             case NodeType::NODE_CTE_COLUMNS:
             case NodeType::NODE_TABLE_QUERY:
-            case NodeType::NODE_VALUES_CLAUSE:
                 return true;
+            case NodeType::NODE_VALUES_CLAUSE:
+                if (!dml_values) return true;
+                break;
             case NodeType::NODE_BINARY_OP:
             case NodeType::NODE_UNARY_OP:
                 if (node->flags & sql_parser::FLAG_PG_OPERATOR) return true;
@@ -101,7 +154,7 @@ class PlanBuilder {
             default: break;
         }
         for (const auto* child = node->first_child; child; child = child->next_sibling)
-            if (has_unsupported_query_feature(child, false)) return true;
+            if (has_unsupported_query_feature(child, false, dml_values)) return true;
         return false;
     }
 

include/sql_parser/common.h

@@ -119,6 +119,9 @@ enum class StmtType : uint8_t {
     DO_STMT,
     COPY,
     RELEASE_SAVEPOINT,
+    VACUUM,
+    ANALYZE,
+    MERGE,
 };
 
 // -- AST node types --
@@ -276,6 +279,37 @@ enum class NodeType : uint16_t {
     NODE_NAMED_ARGUMENT,       // value=argument name; one expression child
     NODE_AGGREGATE_ORDER_BY,   // ORDER_BY_ITEM children; constants are not ordinals
     NODE_CTE_COLUMNS,          // identifier children; follows body in CTE_DEFINITION
+    // PG_GAPS_EXPRESSION_NODES
+    NODE_PG_EXTRACT,
+    NODE_PG_SUBSTRING,
+    NODE_PG_TIME_ZONE,
+    NODE_PG_INTERVAL,
+    NODE_PG_TRIM,
+    NODE_PG_ARRAY_QUERY,
+    NODE_PG_QUANTIFIED_OPERAND,
+    NODE_PG_NORMALIZE,
+    // PG_GAPS_DML_NODES
+    NODE_MERGE_STMT,            // target, USING source, ON expression, WHEN actions, RETURNING
+    NODE_MERGE_WHEN,            // match spelling; optional AND expression, THEN action
+    NODE_PG_DML_CLAUSE,         // validated fixed syntax; structured children separated by spaces
+    NODE_CTE_SEARCH,            // value=DEPTH/BREADTH; columns then sequence column
+    NODE_CTE_CYCLE,             // columns, mark column, optional values, path column
+    NODE_PG_RETURNING_OPTIONS,  // OLD/NEW AS alias options
+    NODE_PG_ASSIGNMENT_FIELD,   // target.field, without expression parentheses
+    // PG_GAPS_DDL_NODES
+    NODE_PG_DDL_STMT,           // command value; structured clause children
+    NODE_PG_DDL_CLAUSE,         // grammar production, optional keyword prefix
+    NODE_PG_DDL_LIST,           // comma-separated children; flag 1 = no parentheses
+    NODE_PG_DDL_SYNTAX,         // validated syntax keyword or option, never an expression
+    // PG_GAPS_QUERY_NODES
+    NODE_GROUPING_SET,
+    NODE_OFFSET_CLAUSE,
+    NODE_FETCH_CLAUSE,
+    NODE_ORDINALITY,
+    NODE_FUNCTION_COLUMN,
+    // PG_GAPS_JSON_XML_NODES
+    NODE_PG_JSON_XML,
+    NODE_PG_JSON_XML_SYNTAX,
 };
 
 } // namespace sql_parser