feat: expand PostgreSQL aggregate calls and CTE query composition

Parse schema-qualified and quoted expression function names, DISTINCT/ALL
aggregate arguments, in-call ORDER BY with direction and null placement, and
WITHIN GROUP. Preserve these forms through FILTER/OVER and SQL emission.
Represent aggregate ordering with its own AST node so literal sort expressions
are parameterized as values instead of mistaken for SELECT output ordinals.
Reject incomplete calls, malformed ordering, incompatible aggregate modifiers,
and bare-star placements that PostgreSQL's grammar does not permit.

Extend PostgreSQL CTEs with output-column lists, materialization hints,
SELECT/TABLE/VALUES/compound bodies, and nested WITH queries. Share this query
parser with scalar and derived subqueries, including LATERAL, and preserve CTE
identifier source spelling and the existing body-first child layout. Emit full
WITH clauses, preserving RECURSIVE and quoted names. Keep MySQL query parsing
behavior and the 48-byte AST layout; append the new node kinds.

Propagate subquery errors without dereferencing null ASTs. Reject unsupported
aggregate and CTE semantics in the local engine before materialization, rather
than skipping an unbuildable CTE and accidentally reading a same-named physical
table. Nested/recursive CTE execution, column remapping, VALUES/TABLE execution,
and materialization hints remain unsupported. DML CTEs, SEARCH/CYCLE and full
PostgreSQL grammar parity remain outside this increment.

Add parse/emission, malformed-input, parameterization, AST-shape and real
executor shadowing regressions. Document supported forms and remaining gaps.
Validation: forced make all/build-corpus-test passed 1,381 tests (37 external
backend skips); 52 focused expression/AST sanitizer tests passed; 24 new
original/emitted SQL pairs produced equivalent PostgreSQL ASTs after removing
source positions. Independent review rechecked 20 grammar cases against the
pinned oracle and all three physical-table shadowing reproductions.

Benchmark harnesses, reports, samples and their README/ignore changes are
intentionally excluded from this commit.

Author: renecannao

docs/postgresql-analysis-features.md

@@ -21,10 +21,10 @@ transformation. They support normal SQL emission and digest generation.
 PostgreSQL still performs name resolution, type checking, window inheritance
 validation and other semantic checks.
 
-The new clauses remain bounded by the existing expression/subquery grammar.
-In particular, LATERAL currently uses the simple SELECT subquery callback;
-WITH/VALUES/compound derived-query operands and interval-valued frame offsets
-are not covered by this change.
+The new clauses remain bounded by the existing expression grammar. PostgreSQL
+subqueries now share compound-query parsing, including SELECT, TABLE, VALUES and
+WITH operands in derived tables and LATERAL. Interval-valued frame offsets remain
+outside the supported expression subset.
 
 Window expressions previously disappeared from emitted SQL and digests because
 their AST nodes had no emitter handlers. They now appear correctly. Digest
@@ -72,6 +72,39 @@ position, interval prefix-literal qualifiers, or the remaining full expression
 and DDL grammar. These additions improve the native ParserSQL implementation;
 there is no PostgreSQL runtime fallback or production library dependency.
 
+## Qualified calls, aggregates and CTEs
+
+Expression calls accept schema-qualified names, including quoted components.
+Aggregate calls preserve `DISTINCT`/`ALL`, in-call `ORDER BY` (including direction
+and NULLS placement), and `WITHIN GROUP (ORDER BY ...)`. FILTER and OVER can
+follow the supported aggregate forms. This is syntactic parsing; PostgreSQL
+still checks aggregate signatures and semantic restrictions.
+
+`NODE_FUNCTION_CALL` retains ordinary argument children and node-specific flags
+for qualification, DISTINCT, ALL and WITHIN GROUP. An appended
+`NODE_AGGREGATE_ORDER_BY` child contains `NODE_ORDER_BY_ITEM` children. Its sort
+expressions are values, not SELECT output ordinals: parameterization binds a
+literal `1` inside `array_agg(x ORDER BY 1)` but preserves the query-level
+`ORDER BY 1`. Malformed calls and missing closing delimiters report errors.
+
+PostgreSQL CTE definitions accept output-column lists, `[NOT] MATERIALIZED`,
+SELECT/TABLE/VALUES/compound bodies and nested WITH queries. RECURSIVE is retained
+in the AST and emitted SQL; it does not enable recursive local execution.
+The definition body remains its first child, followed by `NODE_CTE_COLUMNS`
+when present. CTE names and column names preserve quoting. The CTE emitter
+reconstructs the complete WITH clause and main query.
+
+The local engine rejects qualified calls, aggregate modifiers/ordering,
+recursive/nested CTEs, CTE output-column lists, materialization hints and
+VALUES/TABLE query bodies before CTE materialization. These additions do not extend CTE parameterization, which
+continues to return an unsupported-context error.
+
+Remaining gaps include data-modifying CTEs and main statements, SEARCH/CYCLE,
+special EXTRACT/SUBSTRING forms, AT TIME ZONE, advanced grouping, and broader
+SQL/JSON, XML and DDL grammar. See PostgreSQL's
+[aggregate syntax](https://www.postgresql.org/docs/18/sql-expressions.html#SYNTAX-AGGREGATES)
+and [WITH queries](https://www.postgresql.org/docs/18/queries-with.html).
+
 ## Multiple statements
 
 ```cpp

docs/superpowers/plans/2026-09-22-pg-select-cte-coverage.md

@@ -0,0 +1,54 @@
+# PostgreSQL SELECT and CTE coverage increment
+
+Continue the user-approved grammar expansion on PR #67. Execute inline, retain
+MySQL behavior and the 48-byte AST layout, and exclude all benchmark files from
+commits. Preserve syntax in ASTs and emitted SQL; reject newly parsed features
+in the local engine unless its existing implementation supports them.
+
+1. Add failing PostgreSQL expression tests for qualified/quoted function names,
+   DISTINCT/ALL aggregates, aggregate ORDER BY with direction/null placement,
+   WITHIN GROUP, composition with FILTER/OVER, malformed argument boundaries,
+   and parameterization of aggregate sort constants (which are not ordinals).
+2. Implement expression parsing and emission with node-specific flags and an
+   appended aggregate order node. Retain ordinary function argument children.
+3. Add failing CTE tests for output columns, VALUES/TABLE/compound bodies,
+   nested WITH, materialization hints, and strict delimiters. Preserve the body
+   as the first definition child and explicit columns as a separate child.
+   Add CTE emission and guard unsupported execution. DML CTEs remain deferred
+   if their integration requires changes beyond this query-focused increment.
+4. Force rebuild and run full C++ tests, corpus harness, and PostgreSQL oracle
+   comparisons. Inspect transitions before refreshing correctness snapshots.
+5. Obtain an independent code review, document behavior and remaining gaps,
+   make detailed commits, push the branch, and update PR #67.
+
+Review focus: quoted names, call versus column ambiguity, missing delimiters,
+aggregate order constants versus SELECT ordinals, CTE child layout and engine
+materialization behavior. Do not equate complete parsing with AST equivalence.
+
+## Progress and review
+
+- Qualified calls, aggregate modifiers/order and CTE composition implemented.
+  Focused tests first demonstrated the missing syntax, then passed.
+- Ruling: extend the shared PostgreSQL subquery callback and its entry checks
+  to support VALUES/TABLE/WITH in derived tables and scalar subqueries. This
+  avoids different grammars for the same query in CTE and derived contexts.
+- Review exposed malformed EXISTS null dereference, quoted CTE name emission,
+  incomplete operand acceptance in nested queries, duplicate bare WITH, ALL
+  WITHIN GROUP acceptance, and aggregate star restrictions. Added regressions
+  and corrected each against pinned PostgreSQL grammar.
+- Ruling: add NULLS FIRST/LAST in compound-query ordering because shared
+  subquery parsing must preserve ordering clauses. Unsupported local execution
+  remains explicitly rejected.
+- Review demonstrated physical-table shadowing when unsupported CTE bodies were
+  skipped during execution. Reject VALUES/TABLE query bodies and nested CTE
+  materialization before execution, with an executor regression test.
+- Final corpus replay: 27,014 complete parses (+771), no previously complete
+  case lost. Gains: 638 SELECT, 123 EXPLAIN, nine UPDATE and one INSERT. Another
+  79 trailing and 42 type-mismatch results now report explicit errors; these
+  remain unsupported queries, not lost complete parses.
+- Forced full rebuild: 1,381 passing C++ tests, 37 backend skips; corpus harness
+  built. Focused sanitizer run: 52 passing tests. All 24 new original/emitted
+  PostgreSQL AST pairs match after source locations are removed.
+- Independent final review: no outstanding findings after 20 targeted grammar
+  checks and three CTE physical-table shadowing reproductions.
+- Correctness snapshot refresh and PR publication follow the grammar commit.

include/sql_engine/plan_builder.h

@@ -52,36 +52,56 @@ class PlanBuilder {
         return nullptr;
     }
 
+    static bool supports_query_features(const sql_parser::AstNode* ast) {
+        return ast && !has_unsupported_query_feature(ast);
+    }
+
 private:
     const Catalog& catalog_;
     sql_parser::Arena& arena_;
 
-    static bool has_unsupported_query_feature(const sql_parser::AstNode* node) {
+    static bool has_unsupported_query_feature(const sql_parser::AstNode* node, bool root = true) {
         using sql_parser::NodeType;
         switch (node->type) {
+            case NodeType::NODE_CTE:
+                // Nested WITH requires its own materialization scope, which
+                // neither build_cte nor the subquery executor implements.
+                if (!root || (node->flags & sql_parser::FLAG_CTE_RECURSIVE)) return true;
+                break;
+            case NodeType::NODE_CTE_DEFINITION:
+                if (node->flags & (sql_parser::FLAG_CTE_MATERIALIZED |
+                    sql_parser::FLAG_CTE_NOT_MATERIALIZED)) return true;
+                break;
+            case NodeType::NODE_CTE_COLUMNS:
+            case NodeType::NODE_TABLE_QUERY:
+            case NodeType::NODE_VALUES_CLAUSE:
+                return true;
             case NodeType::NODE_BINARY_OP:
             case NodeType::NODE_UNARY_OP:
                 if (node->flags & sql_parser::FLAG_PG_OPERATOR) return true;
                 break;
             case NodeType::NODE_NAMED_ARGUMENT:
             case NodeType::NODE_TYPE_CAST:
             case NodeType::NODE_DISTINCT_ON:
+            case NodeType::NODE_AGGREGATE_ORDER_BY:
             case NodeType::NODE_AGGREGATE_FILTER:
             case NodeType::NODE_LATERAL:
             case NodeType::NODE_WINDOW_CLAUSE:
             case NodeType::NODE_WINDOW_REFERENCE:
             case NodeType::NODE_WINDOW_FRAME:
                 return true;
             case NodeType::NODE_FUNCTION_CALL:
-                if (node->flags & sql_parser::FLAG_FUNCTION_TABLE) return true;
+                if (node->flags & (sql_parser::FLAG_FUNCTION_TABLE | sql_parser::FLAG_FUNCTION_DISTINCT |
+                    sql_parser::FLAG_FUNCTION_ALL | sql_parser::FLAG_FUNCTION_QUALIFIED |
+                    sql_parser::FLAG_FUNCTION_WITHIN_GROUP)) return true;
                 break;
             case NodeType::NODE_ORDER_BY_ITEM:
                 if (node->flags & sql_parser::FLAG_ORDER_NULLS) return true;
                 break;
             default: break;
         }
         for (const auto* child = node->first_child; child; child = child->next_sibling)
-            if (has_unsupported_query_feature(child)) return true;
+            if (has_unsupported_query_feature(child, false)) return true;
         return false;
     }
 

include/sql_engine/plan_executor.h

@@ -115,6 +115,9 @@ class PlanExecutor {
             return execute(plan);
         }
 
+        // Reject unsupported CTE semantics before any materialization occurs.
+        if (!PlanBuilder<D>::supports_query_features(ast)) return {};
+
         // Materialize each CTE definition
         std::vector<std::unique_ptr<InMemoryDataSource>> cte_sources;
         for (const sql_parser::AstNode* child = ast->first_child; child; child = child->next_sibling) {

include/sql_parser/common.h

@@ -63,11 +63,18 @@ static constexpr uint16_t FLAG_QUERY_PARENTHESIZED = 0x01;
 static constexpr uint16_t FLAG_TABLE_ONLY = 0x01;
 static constexpr uint16_t FLAG_TABLE_INHERIT = 0x02;
 static constexpr uint16_t FLAG_FUNCTION_TABLE = 0x01;
+static constexpr uint16_t FLAG_FUNCTION_DISTINCT = 0x02;
+static constexpr uint16_t FLAG_FUNCTION_ALL = 0x04;
+static constexpr uint16_t FLAG_FUNCTION_QUALIFIED = 0x08;
+static constexpr uint16_t FLAG_FUNCTION_WITHIN_GROUP = 0x10;
 // On binary/unary expressions: PostgreSQL operation without local engine support.
 static constexpr uint16_t FLAG_PG_OPERATOR = 0x01;
 static constexpr uint16_t FLAG_WINDOW_BETWEEN = 0x01;
 static constexpr uint16_t FLAG_ORDER_NULLS = 0x01;
 static constexpr uint16_t FLAG_LIMIT_COMMA = 0x01;
+static constexpr uint16_t FLAG_CTE_RECURSIVE = 0x01;
+static constexpr uint16_t FLAG_CTE_MATERIALIZED = 0x02;
+static constexpr uint16_t FLAG_CTE_NOT_MATERIALIZED = 0x04;
 
 // -- Flags for NODE_IDENTIFIER / NODE_COLUMN_REF --
 // Set when the identifier was source-delimited (backtick `name` for MySQL,
@@ -267,6 +274,8 @@ enum class NodeType : uint16_t {
     NODE_TYPE_CAST,            // expression, NODE_TYPE_NAME; canonical CAST emission
     NODE_TYPE_NAME,            // validated type syntax, including modifiers/bounds
     NODE_NAMED_ARGUMENT,       // value=argument name; one expression child
+    NODE_AGGREGATE_ORDER_BY,   // ORDER_BY_ITEM children; constants are not ordinals
+    NODE_CTE_COLUMNS,          // identifier children; follows body in CTE_DEFINITION
 };
 
 } // namespace sql_parser

include/sql_parser/compound_query_parser.h

@@ -210,6 +210,18 @@ class CompoundQueryParser {
                 item->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER, dir.text));
             }
 
+            if constexpr (D == Dialect::PostgreSQL) {
+                if (ExpressionParser<D>::keyword(tok_.peek(), "NULLS")) {
+                    tok_.skip();
+                    Token placement = tok_.peek();
+                    bool first = ExpressionParser<D>::keyword(placement, "FIRST");
+                    if (!first && !ExpressionParser<D>::keyword(placement, "LAST")) return nullptr;
+                    tok_.skip();
+                    item->flags |= FLAG_ORDER_NULLS;
+                    item->add_child(make_node(arena_, NodeType::NODE_IDENTIFIER,
+                        first ? StringRef{"NULLS FIRST", 11} : StringRef{"NULLS LAST", 10}));
+                }
+            }
             order_by->add_child(item);
 
             if (tok_.peek().type == TokenType::TK_COMMA) {

include/sql_parser/emitter.h

@@ -54,7 +54,8 @@ class Emitter {
             case NodeType::NODE_WHERE_CLAUSE:    emit_where_clause(node); break;
             case NodeType::NODE_GROUP_BY_CLAUSE: emit_group_by(node); break;
             case NodeType::NODE_HAVING_CLAUSE:   emit_having(node); break;
-            case NodeType::NODE_ORDER_BY_CLAUSE: emit_order_by(node); break;
+            case NodeType::NODE_ORDER_BY_CLAUSE:
+            case NodeType::NODE_AGGREGATE_ORDER_BY: emit_order_by(node); break;
             case NodeType::NODE_ORDER_BY_ITEM:   emit_order_by_item(node); break;
             case NodeType::NODE_LIMIT_CLAUSE:    emit_limit(node); break;
             case NodeType::NODE_LOCKING_CLAUSE:  emit_locking(node); break;
@@ -108,6 +109,9 @@ class Emitter {
             case NodeType::NODE_STAR_REPLACE: emit_star_replace(node); break;
             case NodeType::NODE_REPLACE_ITEM: emit_replace_item(node); break;
 
+            case NodeType::NODE_CTE: emit_cte(node); break;
+            case NodeType::NODE_CTE_COLUMNS: emit_list(node, ", "); break;
+
             // ---- Compound query ----
             case NodeType::NODE_COMPOUND_QUERY:  emit_compound_query(node); break;
             case NodeType::NODE_SET_OPERATION:   emit_set_operation(node); break;
@@ -1277,20 +1281,60 @@ class Emitter {
         if (child) emit_node(child);
     }
 
+    void emit_cte(const AstNode* node) {
+        sb_.append("WITH ");
+        if (node->flags & FLAG_CTE_RECURSIVE) sb_.append("RECURSIVE ");
+        const AstNode* child = node->first_child;
+        bool first = true;
+        for (; child && child->type == NodeType::NODE_CTE_DEFINITION; child = child->next_sibling) {
+            if (!first) sb_.append(", ");
+            first = false;
+            emit_identifier(child);
+            const AstNode* body = child->first_child;
+            const AstNode* columns = body ? body->next_sibling : nullptr;
+            if (columns && columns->type == NodeType::NODE_CTE_COLUMNS) {
+                sb_.append_char('(');
+                emit_node(columns);
+                sb_.append_char(')');
+            }
+            sb_.append(" AS ");
+            if (child->flags & FLAG_CTE_MATERIALIZED) sb_.append("MATERIALIZED ");
+            else if (child->flags & FLAG_CTE_NOT_MATERIALIZED) sb_.append("NOT MATERIALIZED ");
+            sb_.append_char('(');
+            emit_node(body);
+            sb_.append_char(')');
+        }
+        sb_.append_char(' ');
+        emit_node(child);
+    }
+
     void emit_function_call(const AstNode* node) {
         const AstNode* arg = node->first_child;
         if (node->flags & FLAG_FUNCTION_TABLE) {
             emit_node(arg);
             arg = arg ? arg->next_sibling : nullptr;
         } else emit_value(node);
         sb_.append_char('(');
+        if (node->flags & FLAG_FUNCTION_DISTINCT) sb_.append("DISTINCT ");
+        else if (node->flags & FLAG_FUNCTION_ALL) sb_.append("ALL ");
         bool first = true;
+        const AstNode* order = nullptr;
         for (; arg; arg = arg->next_sibling) {
+            if (arg->type == NodeType::NODE_AGGREGATE_ORDER_BY) {
+                order = arg;
+                continue;
+            }
             if (!first) sb_.append(", ");
             first = false;
             emit_node(arg);
         }
+        if (order && !(node->flags & FLAG_FUNCTION_WITHIN_GROUP)) emit_node(order);
         sb_.append_char(')');
+        if (order && (node->flags & FLAG_FUNCTION_WITHIN_GROUP)) {
+            sb_.append(" WITHIN GROUP (ORDER BY ");
+            emit_list(order, ", ");
+            sb_.append_char(')');
+        }
     }
 
     void emit_is_null(const AstNode* node) {