docs: describe PostgreSQL analysis syntax and AST ownership contracts

Document the new query grammar, supported transaction and COPY forms, and
multi-statement parsing API with source offsets and per-statement results.
State the lifetime requirements for input text and parser-arena ASTs, the
limits of batch recovery, and the distinction between parsing a construct
and executing it in the local query engine. Explain that window expressions
now appear in emitted SQL and therefore change affected digest values.

Add examples and contracts for bounded traversal, owning deep copies,
validated subtree replacement and parameterization of copied ASTs. Cover
PostgreSQL numbered binds, MySQL emitted parameter positions, literal/source
mappings, preserved syntax constants, allocation behavior and contexts that
must fail explicitly instead of yielding misleading rewritten SQL.

Link both API guides from the README, correct its stale AstNode size claim
to 48 bytes, and distinguish historical acceptance figures from complete-input
AST coverage. Retain the implementation design and completion evidence with
the user's current dedicated-branch instructions and benchmark exclusion.

Documentation reflects the final forced build: 1,354 active C++ tests pass,
37 backend-dependent tests are skipped, 87 compatibility harness tests pass,
and 33,534 refreshed CI fixtures replay successfully. Benchmark reports,
harness code, build-ignore changes and new benchmark links remain uncommitted.

Author: renecannao

README.md

@@ -52,6 +52,10 @@ Dialect selection is compile-time (`Parser<Dialect::MySQL>` / `Parser<Dialect::P
 
 See [`docs/benchmarks/latest.md`](docs/benchmarks/latest.md) for the full report, and [`REPRODUCING.md`](docs/benchmarks/REPRODUCING.md) for reproduction instructions.
 
+The acceptance figures above are historical and include classification-only
+results; use the PostgreSQL compatibility report below for complete-input AST
+coverage and remaining syntax gaps.
+
 ## Quick Start
 
 ### Build
@@ -322,14 +326,19 @@ auto report = recovery.recover();
 
 - **Arena allocator** — 64 KB bump allocator per parser, O(1) reset. All AST nodes and plan nodes live in the arena. No per-node new/delete.
 - **Zero-copy `StringRef`** — tokens point into the original input buffer.
-- **32-byte `AstNode`** — half a cache line; intrusive linked list (first_child + next_sibling).
+- **48-byte `AstNode`** — intrusive linked list (first_child + next_sibling), with value and source spans.
 - **Compile-time dialect dispatch** — `if constexpr` for MySQL vs PostgreSQL differences. Zero runtime overhead.
 - **Header-only parsers & operators** — maximum inlining. Only `arena.cpp`, `parser.cpp`, a few engine `.cpp` files compile separately.
 
 ## Features
 
 ### Parser
 
+See [PostgreSQL analysis features](docs/postgresql-analysis-features.md) for
+DISTINCT ON, FILTER, LATERAL, named windows/frames, transaction/COPY ASTs and
+`parse_all()`. [AST utility APIs](docs/ast-utilities.md) document traversal,
+owning copies, subtree replacement and context-aware parameterization.
+
 - **Tier 1 deep parse:** SELECT, INSERT, UPDATE, DELETE, SET, REPLACE, EXPLAIN, CALL, DO, LOAD DATA
 - **Compound queries:** UNION / INTERSECT / EXCEPT with SQL-standard precedence and parenthesized nesting
 - **CTEs:** `WITH ... [RECURSIVE] AS (...)` — non-recursive materialized, recursive planned

docs/ast-utilities.md

@@ -0,0 +1,125 @@
+# AST traversal, transformation, and parameterization
+
+These header-only C++17 APIs use the existing 48-byte `AstNode` and `Arena`.
+Include `sql_parser/ast_walk.h`, `sql_parser/ast_transform.h`, or
+`sql_parser/parameterize.h` as needed.
+
+## Traverse one subtree
+
+```cpp
+auto result = sql_parser::walk_ast(root,
+    [](const sql_parser::AstNode& node,
+       const sql_parser::AstVisitContext& context) {
+        // context.parent, context.depth, context.child_index
+        return sql_parser::AstVisitAction::Continue;
+    });
+```
+
+Traversal is preorder, children in sibling order. `SkipChildren` skips only the
+current node's descendants; `Stop` terminates the whole walk. The root's own
+`next_sibling` is outside the traversal. A null root is a completed empty walk.
+Do not change links during a walk.
+
+The iterative implementation uses O(depth) auxiliary storage. `AstWalkLimits`
+defaults to 1,000,000 visited nodes and a maximum depth of 4,096 (root depth zero).
+`AstWalkResult` contains the visit count and `Completed`, `Stopped`, or
+`LimitExceeded`. Bounds also prevent unbounded traversal of cyclic input.
+
+## Copy and replace
+
+```cpp
+sql_parser::Arena destination;
+auto copied = sql_parser::clone_ast(root, destination);
+if (!copied.ok()) { /* inspect copied.error */ }
+```
+
+`clone_ast` copies one subtree and every value/source string into `destination`.
+The input SQL and original arena may then be destroyed. The clone's root has no
+sibling; internal child and sibling order and flags are retained. Cycles and
+shared nodes are rejected with `InvalidTree`; exhaustion and traversal bounds
+return explicit errors. Cloning null succeeds with a null result.
+
+`make_owned_node(arena, type, value, flags, source)` is the owning counterpart of
+`make_node`: it copies text and returns null on arena allocation failure.
+
+`replace_ast_subtree(root, target, replacement)` changes the caller's root pointer
+when necessary, preserving the target's sibling position. Pass null replacement
+to remove the target. On success the old target is detached from its siblings.
+The input root must be a standalone tree, with no root sibling. Replacement must
+be a detached subtree with no root sibling and share no nodes with the input
+tree. Validation happens before mutation; errors leave input links unchanged.
+Nodes are never freed individually, and callers retain responsibility for the
+lifetimes of every participating arena.
+
+Failed operations can consume arena space; they do not reset or roll back an
+arena that might contain other live nodes. Temporary STL allocation failures use
+the usual C++ exception behavior.
+
+## Parameterize complete statements
+
+```cpp
+sql_parser::Parser<sql_parser::Dialect::PostgreSQL> parser;
+auto parsed = parser.parse(sql.data(), sql.size());
+sql_parser::Arena output;
+auto rewritten = sql_parser::parameterize_ast<sql_parser::Dialect::PostgreSQL>(
+    parsed, output);
+if (rewritten.ok()) {
+    sql_parser::Emitter<sql_parser::Dialect::PostgreSQL> emitter(output);
+    emitter.emit(rewritten.ast);
+    auto sql_with_binds = emitter.result();
+}
+```
+
+Prefer the `ParseResult` overload: it rejects non-OK results, unconsumed input,
+and results without `full_input`. The raw `AstNode*` overload requires a complete,
+valid AST in the specified dialect. Neither overload performs catalog lookup,
+type inference, or a second SQL grammar validation.
+
+The original tree is unchanged. The result contains a copied AST and an ordered
+`parameters` vector of `ExtractedParameter { index, literal_type, value, source }`.
+Indices are one-based. Values retain their lexical spelling, including any
+escape sequences; they are not decoded strings or converted numeric values.
+Source retains the original literal spelling when recorded by the parser, and
+may be empty for manually built nodes or keywords without source metadata.
+Mapping text and AST nodes live in the output arena; the result owns its vectors.
+Repeated literals produce separate parameters.
+
+PostgreSQL preserves existing `$n` spellings and indices and generates new
+indices after the largest existing index. For `SELECT $3, 7, $1, 'a'`, the result
+is `SELECT $3, $4, $1, $5`, with new mapping indices 4 and 5 and
+`existing_parameters` equal to `[3, 1]`. Invalid indices and signed 32-bit index
+overflow fail explicitly.
+
+MySQL emits `?` and counts positions in the resulting SQL. For
+`SELECT ?, 7, ?, 'a'`, new mapping indices are 2 and 4 and
+`existing_parameters` is `[1, 3]`. The latter identifies the positions where
+callers should interleave their original bindings. MySQL comma LIMIT syntax is
+normalized from `LIMIT offset, count` to `LIMIT count OFFSET offset`, so extracted
+literals follow the normalized emission order. If that comma LIMIT contains an
+existing `?`, parameterization returns `UnsupportedContext` rather than silently
+reordering the caller's original anonymous bindings. Existing binds elsewhere in
+the statement and `LIMIT count OFFSET offset` remain supported.
+
+Supported roots are SELECT, INSERT, UPDATE, DELETE, compound queries, and TABLE
+queries. Bindable literals include integers, floats, strings, hex/bit literals,
+and standalone TRUE/FALSE. NULL remains literal. Direct integer ordinals in
+query GROUP BY, ORDER BY, and DISTINCT ON remain unchanged, including
+parenthesized integers. Integers inside larger expressions and window ORDER BY
+are values. `IS [NOT] TRUE/FALSE` keywords and PostgreSQL binary `::` type
+subtrees remain unchanged. The precision argument of CURRENT_TIMESTAMP,
+CURRENT_TIME, LOCALTIME, and LOCALTIMESTAMP remains a syntax constant. MySQL also
+preserves precision arguments of NOW, CURTIME, SYSDATE, UTC_TIME, and
+UTC_TIMESTAMP; these ordinary function names retain bindable arguments in
+PostgreSQL. Quoted function names retain bindable arguments in both dialects.
+Aggregate FILTER predicates and window frame offsets are bindable. A manually constructed binary `::` AST is supported; this does not
+extend the parser's existing cast grammar.
+
+Unsupported roots and contexts return `UnsupportedRoot` or `UnsupportedContext`
+with no AST or partial mapping. These include SET/transaction/COPY/DDL/other
+utility statements, SELECT INTO, CTEs, generic function-shaped CAST nodes,
+opaque subquery/interval fragments, and unknown future node types. PostgreSQL
+string-shaped aliases are rejected, including typed string literals currently
+misparsed as aliases (for example, `SELECT TIMESTAMP '2020-01-01'`). These
+restrictions reflect contexts where replacing literals or the current emitter
+would not produce reliable executable SQL. Parameterization does not change the
+existing digest API.

docs/postgresql-analysis-features.md

@@ -0,0 +1,103 @@
+# PostgreSQL query analysis additions
+
+These additions extend parsing and SQL reconstruction. They do not add local
+execution support for the new PostgreSQL query constructs. The plan builder
+rejects unsupported constructs instead of silently executing a simpler query.
+
+## Query syntax
+
+- `SELECT DISTINCT ON (expressions)`.
+- Aggregate `FILTER (WHERE expression)`, also followed by `OVER`.
+- `LATERAL` subqueries and table functions, including qualified names and alias
+  column lists.
+- Named window declarations (`WINDOW w AS (...)`), references (`OVER w`), and
+  inherited specifications (`OVER (w ...)`).
+- `ROWS`, `RANGE`, and `GROUPS` frames; single bounds or `BETWEEN ... AND ...`;
+  `UNBOUNDED`, offset `PRECEDING`/`FOLLOWING`, `CURRENT ROW`; and `EXCLUDE CURRENT
+  ROW`, `GROUP`, `TIES`, or `NO OTHERS`.
+
+These forms retain expressions and options as AST children for traversal and
+transformation. They support normal SQL emission and digest generation.
+PostgreSQL still performs name resolution, type checking, window inheritance
+validation and other semantic checks.
+
+The new clauses remain bounded by the existing expression/subquery grammar.
+In particular, LATERAL currently uses the simple SELECT subquery callback;
+WITH/VALUES/compound derived-query operands and interval-valued frame offsets
+are not covered by this change.
+
+Window expressions previously disappeared from emitted SQL and digests because
+their AST nodes had no emitter handlers. They now appear correctly. Digest
+strings/hashes for affected queries change as a consequence. Quoted identifiers
+and PostgreSQL string-source boundaries are also preserved more accurately.
+
+## Multiple statements
+
+```cpp
+sql_parser::Parser<sql_parser::Dialect::PostgreSQL> parser;
+std::string sql = "BEGIN READ ONLY; SELECT ';'; COMMIT";
+auto batch = parser.parse_all(sql.data(), sql.size());
+for (const auto& statement : batch.statements) {
+    // statement.offset: byte offset in the original input
+    // statement.source: SQL span, including its terminating semicolon if any
+    // statement.result: classification, AST, status and full_input
+}
+```
+
+All returned ASTs remain alive until the next `parse`, `parse_all`, or `reset`
+on that parser. Input SQL must remain alive too. Use `clone_ast` to retain a tree
+independently. The result vector owns records, not the ASTs or source bytes.
+
+The tokenizer identifies boundaries, so semicolons in strings, quoted
+identifiers and comments do not split statements. Empty statements/comments are
+skipped. Malformed statements remain in the result; scanning continues at the
+next lexical statement boundary when available. Error offsets are relative to
+the original batch input.
+
+`batch.ok()` requires every nonempty statement to have `OK`, complete input and
+a known statement type. It does not require every statement to have a deep AST
+and is not a substitute for PostgreSQL syntax/semantic validation. Empty batches
+succeed. The API handles SQL text, not inline COPY data, psql commands, or
+unquoted procedural bodies with internal statement terminators.
+
+## Transaction and COPY ASTs
+
+PostgreSQL transaction commands now produce `NODE_TRANSACTION_STMT`, whose value
+is the canonical command and whose children contain options, savepoint names or
+prepared-transaction identifiers:
+
+- `BEGIN` / `START TRANSACTION`, isolation levels, read-only/read-write mode,
+  and `[NOT] DEFERRABLE`.
+- `COMMIT` / `ROLLBACK`, `[AND [NO] CHAIN]`, and `END` / `ABORT` aliases.
+- `SAVEPOINT`, `ROLLBACK TO [SAVEPOINT]`, `RELEASE [SAVEPOINT]`.
+- `PREPARE TRANSACTION`, `COMMIT PREPARED`, `ROLLBACK PREPARED`.
+
+`COPY` produces `NODE_COPY_STMT` with direction in its value (`FROM` or `TO`).
+Children are the table/query, optional column list, `NODE_COPY_ENDPOINT`, zero
+or more `NODE_COPY_OPTION` nodes, and an optional WHERE clause. Endpoints
+distinguish `STDIN`, `STDOUT`, `FILE`, and `PROGRAM`; filenames/commands are
+string children. Parsing does not execute any command or open any file.
+
+Supported COPY forms include table/column lists, query `TO`, `FROM ... WHERE`,
+and modern parenthesized options: FORMAT, FREEZE, DELIMITER, NULL, DEFAULT,
+HEADER, QUOTE, ESCAPE, FORCE_QUOTE, FORCE_NOT_NULL, FORCE_NULL, ENCODING,
+ON_ERROR, REJECT_LIMIT and LOG_VERBOSITY. Query operands use the supported
+SELECT/TABLE/VALUES grammar. Legacy COPY option syntax and DML query operands
+are not implemented. Unsupported tails must not be treated as complete input.
+Option compatibility, privileges and data conversion remain server concerns.
+
+The new `StmtType::COPY` and `StmtType::RELEASE_SAVEPOINT` values are appended;
+existing enum values and the 48-byte AST layout are unchanged. Consumers should
+handle the new statement types explicitly.
+
+See [PostgreSQL COPY syntax](https://www.postgresql.org/docs/18/sql-copy.html)
+and [BEGIN syntax](https://www.postgresql.org/docs/18/sql-begin.html) for the server
+grammar and semantics.
+
+## AST APIs and parameterization
+
+See [AST utility APIs](ast-utilities.md) for traversal with skip/stop controls,
+owned cloning, validated subtree replacement, and parameterization with literal
+and existing-bind mappings. Parameterization operates on a copied AST, preserves
+query ordinals, and rejects unsupported contexts explicitly. It does not decode
+literal values or infer PostgreSQL types.

docs/superpowers/plans/2026-09-22-parser-features.md

@@ -0,0 +1,49 @@
+# ParserSQL analysis features implementation plan
+
+> Execute the authorized plan using test-driven development and independent
+> parallel tasks; integrate and review before reporting completion.
+
+**Goal:** Improve PostgreSQL coverage and expose useful query-analysis APIs.
+**Architecture:** Extend the existing AST without changing its layout. Add
+header-only utilities and preserve the reusable parser's fast single-query path.
+**Tech stack:** C++17, arenas, GoogleTest, pinned PostgreSQL compatibility harness.
+**Spec:** ../specs/2026-09-22-parser-features-design.md
+
+## Constraints and review focus
+
+- Preserve existing uncommitted changes; append enum values; no new dependencies.
+- Commit implementation, tests and documentation on a dedicated branch with
+  detailed messages; leave benchmark changes uncommitted. Raw build and timing
+  artifacts stay in /tmp.
+- Check malformed new syntax, quoted identifiers, nested queries, parser reuse,
+  allocation failure, and unsupported-engine behavior.
+- Parameter indices must remain correct when existing binds and literals mix.
+- Multi-statement lexical boundaries must ignore quoted/comment semicolons.
+
+## Tasks
+
+- [x] Query grammar: expression/select/table-reference parsers, common tags,
+  emitter, planner guards, and focused grammar tests. Establish failing tests
+  for DISTINCT ON, FILTER, LATERAL, named windows and frame boundaries; implement
+  structured nodes; verify emission and invalid-input rejection.
+- [x] AST utilities: new ast_walk.h / ast_transform.h / parameterize.h and
+  focused tests. Test traversal order, skip/stop, copied text lifetime, subtree
+  replacement, ordinals, and bind mapping before implementation.
+- [x] Batch and utility parsing: parser.h / parser.cpp, utility parser header,
+  tests. Test quoted semicolons, multiple preserved ASTs, malformed middle
+  statements, transaction options and COPY direction/options; implement
+  parse_all and utility dispatch, coordinate tags/emitter with grammar task.
+- [x] Integrate: register tests in Makefile, run forced full tests and corpus
+  build, differential oracle validation and review; correct identified issues.
+- [x] Document public APIs, scope and limitations; rerun direct benchmark
+  against the same pinned pg_raw_parse API and report actual coverage/timings.
+
+## Completion evidence
+
+1,354 active C++ tests passed (37 backend-dependent skips); corpus harness built;
+87 compatibility-harness tests and 33,534 regenerated CI cases passed; 26 AST
+utility tests passed ASan/UBSan. Independent review findings were fixed and
+reviewer signed off. Public APIs and limitations are documented in
+[the feature guide](../../postgresql-analysis-features.md) and
+[the AST utility guide](../../ast-utilities.md). Benchmark reports and harness
+changes are retained locally and excluded from the implementation commits.

docs/superpowers/specs/2026-09-22-parser-features-design.md

@@ -0,0 +1,36 @@
+# ParserSQL PostgreSQL analysis features
+
+The user authorized the recommended implementation sequence: common PostgreSQL
+query grammar, AST traversal/transformation, multi-statement and utility parsing,
+and context-aware parameterization. Existing uncommitted coverage and IN-list
+performance improvements are the starting point and must be preserved.
+
+Use the existing C++17 parser and arenas. Keep AstNode at 48 bytes and append
+node tags without renumbering old ones. Existing parse(), digest and cache APIs
+keep their behavior. No dependency on PostgreSQL in the production library.
+
+Grammar additions: DISTINCT ON, aggregate FILTER, LATERAL references, named
+windows and ROWS/RANGE/GROUPS frames with boundaries/exclusions. Each must have
+structured AST representation, emission, malformed-input checks and tests. The
+execution planner rejects newly parsed constructs it cannot execute.
+
+AST utilities: preorder traversal with stop/skip, explicit arena-backed cloning
+that copies text, and controlled subtree replacement. Parameterization returns
+a copied AST plus an ordered parameter mapping; preserve original bind indices
+and SQL syntax constants/ordinals. Fail explicitly for unsupported roots or
+contexts instead of returning misleading executable SQL. Existing digest is
+unchanged.
+
+Multi-statement API: return ordered per-statement parse results and source
+boundaries, retain all ASTs until parser reset, respect quoted strings/comments,
+report errors without accepting an unparsed tail. Utility parsing: structured
+PostgreSQL transaction options/savepoints and COPY relation/query, direction,
+endpoint and common options. COPY data payloads and execution are outside scope.
+
+Validation: meaningful failing tests before implementation; focused tests per
+task; full forced C++ rebuild; corpus harness; PostgreSQL differential checks;
+direct pg_raw_parse benchmark for existing and new supported AST cases.
+Retain unsupported syntax explicitly. No claim of full PostgreSQL grammar or
+semantic validation. Commit relevant implementation, tests and documentation
+on a dedicated branch with detailed messages; keep benchmark changes uncommitted
+as requested by the user.