Repository navigation
feat(introspection)!: the dialect owns the SQL, and capability predicates answer before adaptation - #127
Open
vistart wants to merge 60 commits into
Open
feat(introspection)!: the dialect owns the SQL, and capability predicates answer before adaptation#127vistart wants to merge 60 commits into
vistart wants to merge 60 commits into
Conversation
… declared An expression collects its parameters and stops; the formatter that knows the backend decides what it can honour. The `unsigned` shape check therefore moves out of the integer and numeric constructors into each dialect's formatter, and the attribute is read directly rather than probed for with a default -- whether a field exists is settled by the class declaration, so a missing one means the wrong class was passed, which deserves an error rather than a silently signed column. The mutually-exclusive clause options collapse into one helper, so that message has a single definition instead of thirty-one. The expression-dialect skill is rebuilt around this: the protocols a derived class obeys, the four shapes a DataType takes, and the one-to-one pairing between an expression and its formatter. CI installs the testsuite from release/v1.0.0.dev17, which is where typed-column-contracts has landed.
Twenty-one getattr probes turned into direct attribute reads, each one where the parameter's annotation names a class whose constructor signature declares the field -- so the field's existence is settled by the declaration and not by what the caller happened to pass. Two forms had been hiding in plain sight. Several lines already touched the same attribute directly one line after probing it for it, which is the shape of a probe whose answer was never in doubt. And _format_constraint_enforcement carried `constraint: Any`, so the probe there could not be checked against anything; its only caller passes a ColumnConstraint, whose signature declares enforced, not_enforced and constraint_type, so the annotation now says that. What is deliberately left alone: getattrs that ask whether a capability exists, such as `getattr(self, "supports_constraint_enforced", None)`. That one answers "did this backend implement the method at all", and reading it directly would raise on every backend that did not. The rule about probing is about identity fields on expressions, and a capability probe is not one.
…n real probes Column constraints were rendered by fixed-slot logic that ignored what each dialect actually declares. This makes the expression layer carry the missing shapes, and routes rendering through the capability probes backends already implement. Expression layer: - New ColumnClause base. Collate and CharacterSet become real nodes instead of attributes smuggled through the AR layer; format_column_attribute now dispatches on format_method. - ForeignKeyMatchType enum plus normalize_foreign_key_match_type, so MATCH FULL / PARTIAL / SIMPLE is expressible and renderable. - on_delete / on_update / match_type move up to TableConstraint; ForeignKeyConstraint is now only sugar over it. - ColumnDefinition.attributes is properly typed. Rendering: - Column constraints are gated on the probes each dialect already declares: PRIMARY KEY, NOT NULL, NULL, UNIQUE, CHECK, REFERENCES, COLLATE, CHARACTER SET, DEFERRABLE. - CHECK predicates are inlined, and a predicate-less CHECK now raises instead of rendering an empty CHECK (). - format_primary_key_constraint added, so SQLite can hook AUTOINCREMENT placement. SQLite: - Mixin collapsed to one dispatch table; five duplicated handlers and three redundant probes dropped in favour of the shared renderers. - supports_deferrable_constraint is True, with live-SQLite proof that DEFERRABLE INITIALLY DEFERRED really defers the check to COMMIT. - AUTOINCREMENT has two valid forms (column level, and table-level PRIMARY KEY (id AUTOINCREMENT)); placement is checked, not assumed. Docs: state what parallel test execution requires instead of forbidding it.
…ming
The SQL:1999/2016 modifiers (DEFERRABLE, ENFORCED) defaulted to True, so
dialects without them silently rendered clauses their servers reject.
Constructors keep their True default; modifiers now default False and the
dialects that measured the feature opt in (SQLite keeps its measured
FK-only deferral; dummy declares both to keep the render paths exercised).
supports_constraint_naming() joins the protocol with the True default the
SQL-92 grammar gives it (SQLite measured accepting it on 3.50.4, at both
the column and the table level), and both render paths consult it: the
column path used to drop 'name' entirely, the table path rendered it
ungated.
ENFORCED's kind check moves from a hardcoded {CHECK, FOREIGN KEY} to the
standard-derived whitelist {PRIMARY KEY, UNIQUE, FOREIGN KEY, CHECK} as a
static export; the trait now renders on UNIQUE and PRIMARY KEY instead
of being validated and then discarded, and the formatter re-checks the
exclusive pair as the last line of defence.
_call_constraint_capability and its inspect.signature sniffing are gone
(every probe is (self) with no exceptions), and so is the constraint-kind
runtime normalisation: the renderers read the declared enum field and
fail explicitly on anything else.
Eight of the mixin's methods were copies of behaviour the core already renders: the three IF [NOT] EXISTS probes default to the False answers SQLite gives, the ADD/DROP INDEX actions are refused by the core's supports_alter_table_index_actions() default, and the ADD/DROP COLUMN and DROP CONSTRAINT formatters re-raised what the core refuses through the same probes. What remains is what SQLite genuinely spells differently: the AUTOINCREMENT tail, booleans as 1/0, and the placement rules only the column definition can check. The shared DROP CONSTRAINT action gains the if_exists probe gate its ADD/DROP COLUMN siblings already had, so the last per-dialect copy could go too.
…a parameter The rules file answers the node-or-parameter question mechanically: one expression one formatter (A), node-or-parameter from the grammar production (B), count paths not keywords (C), the four parameter shapes (D), and the prefixing the owning parent node's level dictates (E). The catalogue section records the anti-patterns each rule replaced, with the measured evidence, so the next one is recognised rather than reinvented. Indexed from rules/README.md alongside the existing rules.
…nders _format_with_clause took already-rendered CTE strings plus a bool, so no expression corresponded to it: no backend could override the clause without re-implementing the whole query composition, and the supports_basic_cte / supports_recursive_cte probes were read nowhere -- a dialect that declared no CTE support rendered WITH anyway, and a RECURSIVE request on a dialect without recursive CTEs was silently spelled. WithClause now carries the ctes and the RECURSIVE modifier, and format_with_clause is its public formatter: the two probes are consulted there and refuse by name (an empty clause renders nothing). format_with_query_expression composes the node and the rendering is byte-identical: sqlite and dummy declare both probes True, so their output does not move; the only behavior change is the refusal on dialects that declared the probes False. The contract tests render the clause directly (including the two refusals via probed dialects) in addition to the query composition.
…predicate _format_in_value_list took the expression, two already-rendered pieces and the Literal -- three parameters that were not the expression, which is what disqualified it from being a formatter: no protocol could pair with it and no backend could override the value-list spelling. The body moves into format_in_predicate's isinstance branch, the whole Literal is passed in rather than just its value so the inline_literals flag the DDL preparation pass sets is still honoured, and the helper is deleted. Rendering is unchanged move-for-move; the IN-inside-CHECK cases (the six regression tests in test_constraint_protocol.py) are the evidence.
…bility The WITH construct had three classes where two suffice, and the split it took was not the one the grammar draws. - ``CTEExpression`` -> ``CTEClause``: a single ``cte-table-name ... AS (...)`` binds a query-local name and cannot execute alone, so it is a clause of the statement that follows it. Its formatter becomes ``format_cte_clause``. - ``WithQueryExpression`` -> ``WithStatement``: what ``to_sql()`` yields can be sent to the database as is, so it is a statement, not an expression. Its formatter becomes ``format_with_statement``, which gates on ``supports_basic_cte`` / ``supports_recursive_cte`` where the construct is spelled and composes each ``format_cte_clause`` with the trailing query's own rendering. - The ``WITH`` side is held by the statement rather than by a separate clause node. The ``WITH`` keyword is grammar structure the formatter emits, not a node, and apart from ``ctes`` + ``recursive`` the clause has no state, lifecycle, or probe family of its own -- so a clause node over it would only add an indirection over the same data. With no CTE definitions the statement renders the bare query, so a plain query needs no wrapper at all. Call sites follow the new names: ``WithStatement(dialect, query=q, ctes=[...], recursive=r)`` replaces ``WithQueryExpression(dialect, ctes=[...], main_query=q, recursive=r)`` -- ``main_query`` is now ``query``, matching the attribute it names. Breaking: the expression class names and the two formatter names change; no compatibility layer is kept.
The three querying XML expressions (XMLQUERY, XMLEXISTS, XMLTABLE) each carried the ``PASSING`` clause as two unpacked fields and handed them to a private dialect helper. That gave the shared state no home and backends nothing to override. - ``PassingClause`` is the clause node, with the dialect's public ``format_passing_clause`` as its formatter; the three expressions now hold a clause reference instead of ``passing`` + ``passing_mechanism``, so the data lives in one place (规范 4.9.2: a clause with independent data, independent spelling, and a real override point is a node). - The private ``_format_xml_passing_clause`` helper is gone. Every protocol member is public, so the shared spelling is a declared member of ``SQLXMLQueryingSupport`` rather than an internal step of three formatters. - ``passing_clause(dialect, passing, mechanism)`` is a public factory alongside the three XML factories, for callers that build the expressions directly; the factories themselves keep their signatures. Also publicises ``translate_ar_attribute`` (was ``_translate_ar_attribute``, added earlier on this branch): backend ``format_column_definition`` overrides are expected to call it, so it belongs to ``ColumnAttributeSupport``. No protocol member is private, and this branch adds none. Breaking: ``XMLQueryExpression`` / ``XMLExistsExpression`` / ``XMLTableExpression`` take ``passing_clause=`` instead of ``passing=`` + ``passing_mechanism=``; the attributes are gone with it.
…butes The dialect layer asked objects whether they carried an attribute in a dozen places, so a missing field silently became a default and the feature it belonged to was unreachable. Each fix declares the field (or names the type the probe was really asking about) and reads it directly; behaviour that silently defaulted now either works or fails by name. Fields declared where the probes were: - ``AliasableMixin.alias`` -- ``as_()`` attached the attribute to a copy, so only aliased expressions had it. Declared on the mixin, so every aliased expression has the field (and a catalogue entry such as ``Table``, which is not aliasable, still has none). - ``InsertExpression.or_replace`` / ``.or_ignore`` -- SQLite's ``INSERT OR REPLACE`` / ``INSERT OR IGNORE`` were rendered from attributes nothing ever set, so the feature was dead. Now constructor arguments, with the mutual-exclusion rules enforced where the node is built instead of on every render. Dialects that spell neither ignore both fields. Probes replaced by type questions: - ``FunctionCall.niladic`` / ``.filter_predicate`` / ``.window_spec`` are declared on the node; the window guard now checks ``WindowFunctionSupport``, as the FILTER guard already checked ``FilterClauseSupport``, instead of probing for the method. - ``AlterColumn.operation`` / ``new_value`` / ``cascade`` are dataclass fields; ``ExplainOptions.type`` is a declared default; ``DomainAlter- Action.action_kind`` and ``TypeAlterAction.action_kind`` / ``TypeDefinition.definition_kind`` are abstract properties, so the class-name fallback was dead. - ``JSONTableColumn`` / ``XMLTableColumn`` entries are dataclasses, so the projection formatters take the declared type and read ``name``, ``data_type``, ``path`` and ``default`` off it. The ``for_ordinality`` branch goes with them: no node declares that field, so nothing could ever reach it. - Enum-or-spelling parameters (``COMMENT ON`` object type, ``ALTER COLUMN`` operation, constraint types, constraint validation) are discriminated with ``isinstance(..., Enum)`` instead of probed for ``value`` / ``name``. - ``Literal`` is the node carrying a Python value, so the quantified ``ANY`` / ``ALL`` array path tests that rather than asking for ``value``; and the ``RETURNING`` guard unwraps ``WrappedCallMixin.call`` explicitly before reading ``is_aggregate``. What remains in the dialect layer is deliberate: formatter dispatch by ``format_method``, the per-type ``format_data_type_*`` / ``supports_data- type_*`` families, named capability checks, and a few optional class attributes. Breaking: ``INSERT OR REPLACE`` / ``OR IGNORE`` now reject an invalid combination when the statement is built rather than when it is rendered, and passing such a node to a formatter without construction no longer works -- which is the point.
规范 4.9 puts statements in their own naming family: a ``Statement`` is a complete executable unit, and the table has no generic ``*Expression`` category to fall back on. Fifty-seven classes rendered executable SQL while carrying the suffix of values and fragments. Their ``format_method`` names were already right -- ``format_query_statement``, ``format_insert_statement``, ``format_create_table_statement`` and so on -- so the formatters led and the class names followed a release later. Renamed by family: - DQL: ``QueryStatement``, ``SetOperationStatement`` - DML: ``InsertStatement``, ``UpdateStatement``, ``DeleteStatement``, ``MergeStatement`` - transactions: ``BeginTransactionStatement``, ``CommitStatement``, ``RollbackStatement``, ``SavepointStatement``, ``ReleaseSavepointStatement`` - DDL: every ``Create*`` / ``Alter*`` / ``Drop*`` / ``RefreshMaterializedView`` / ``Truncate`` / ``CommentOn`` statement - property graphs: ``Create`` / ``Alter`` / ``DropPropertyGraphStatement`` - SQLite: ``SQLiteInsertStatement``, ``SQLiteReindexStatement``, ``SQLiteVacuumStatement``, ``SQLiteAnalyzeStatement``, ``CreateVirtualTableStatement``, ``DropVirtualTableStatement`` Nothing about the rendered SQL, the formatter names, or the probe contracts changes: only the class names, and every reference to them -- expressions, annotations, exports, docs, and the testsuite's constructor registry included. Formatter names follow in a separate commit so the two halves of the adaptation can be reviewed apart.
The class renames reached the README, the expression-lineage skill and the two experiment scripts, which document and drive the library as it is now. ``CHANGELOG.md`` is deliberately untouched: its entries record what shipped under the names it shipped with, and rewriting them would falsify the history of those releases. The unreleased ``changelog.d/`` fragments do follow, since they describe this cycle.
Four statements still rendered through a formatter named after something other than a statement, and the node/formatter correspondence (规范 4.2, extended by 4.9.1: a renamed class takes its formatter name with it) was therefore broken in both directions. - ``format_set_operation_expression`` -> ``format_set_operation_statement`` - ``format_begin_transaction`` -> ``format_begin_transaction_statement``, and the same for commit / rollback / savepoint / release savepoint - ``format_create_table_using_template`` -> ``format_create_table_from_template_statement``, so the formatter names the node instead of restating the construct in prose - ``format_create_virtual_table`` / ``format_drop_virtual_table`` -> ``..._virtual_table_statement`` Only names move: every formatter still takes the node it took before and renders the same SQL. The protocol declarations, the core mixins, the SQLite and dummy dialects, and the tests follow.
…their category 规范 4.9 gives predicates, clauses and full-text matches their own naming families. Eight nodes sat outside them while being exactly one of the three: a predicate that ended in ``Expression`` can be told apart from a value expression by nothing but the fact that it inherits ``SQLPredicate``. - ``AnyExpression`` / ``AllExpression`` / ``ExistsExpression`` / ``ILIKEExpression`` -> ``AnyPredicate`` / ``AllPredicate`` / ``ExistsPredicate`` / ``ILikePredicate``, and ``XMLExistsExpression`` / ``SQLiteFTS5MatchExpression`` likewise - ``FilterClauseExpression`` -> ``FilterClause`` (its formatter was already ``format_filter_clause``) - ``FulltextMatchExpression`` -> ``FulltextMatch`` The formatter names follow their nodes, as 4.9.1 requires: ``format_any_predicate``, ``format_all_predicate``, ``format_exists_predicate``, ``format_ilike_predicate``, ``format_xmlexists_predicate``. ``LogicalPredicate`` already holds that name, so ``BooleanLogicExpression`` keeps its own: it is the boolean **value** half of the pair, deliberately distinct from the predicate half (it may be aliased into a SELECT list, compared, and cast), and the two share one formatter by design rather than by accident.
…efactor The two-level constraint nodes (S3) will move which node carries which field and where the constraint-level prefix and the trailing traits are rendered. What must not move is a single character of the output, and there was no single place recording it -- the constraint tests each asserted a handful of forms, so a diff after the refactor would have said "something changed" without saying what. Every form is now rendered once, with the dialect capability each form needs supplied explicitly, so a post-refactor diff names the form that moved. Two entries measure current behaviour that S3 is meant to change, and say so where they sit: a column-level ``PRIMARY KEY`` accepts the deferral traits and then drops them -- silently, even on a dialect that declares ``supports_deferrable_constraint()`` -- because only the foreign-key formatter reads them; and a column-level ``FOREIGN KEY`` renders the ``REFERENCES`` half without its ``FOREIGN KEY (col)`` prefix.
…evels BREAKING CHANGE: the fat tag union is gone. ColumnConstraint and TableConstraint are now production-level clauses that name a branch; the branch says which kind the constraint is and carries that kind's own fields. - Delete ColumnConstraintType, TableConstraintType and GeneratedColumnExpression. The kind is a class now, so it cannot be spelled as a string, and a request for a trait a branch cannot carry is refused where the node is built instead of being accepted and dropped. - Add 14 branch nodes: ColumnPrimaryKeyConstraint, ColumnUniqueConstraint, ColumnNotNullConstraint, ColumnNullableConstraint, ColumnCheckConstraint, ColumnDefaultConstraint, ColumnForeignKeyConstraint, ColumnGeneratedConstraint, TablePrimaryKeyConstraint, TableUniqueConstraint, TableCheckConstraint, TableForeignKeyConstraint, TableExcludeConstraint. - ColumnConstraint(dialect, branch=..., name=..., deferrable=..., initially_deferred=..., enforced=..., not_enforced=...) and the table counterpart carry the production-level traits; DEFERRABLE/INITIALLY ride PRIMARY KEY, UNIQUE and FOREIGN KEY, ENFORCED/NOT ENFORCED ride those plus CHECK. - A payload the branch requires is refused at construction: DEFAULT without a value, CHECK without a condition, FOREIGN KEY without a reference. - AlterConstraint.constraint_type= becomes constraint_kind= (a branch class); UseConstraint takes the branch class and keeps its declared fields. - The constraint formatters are renamed for what they render: format_column_primary_key_constraint, format_column_default_constraint, format_table_primary_key_constraint, format_table_unique_constraint, format_table_foreign_key_constraint, and the shared format_constraint_branch / format_constraint_traits. - The dummy dialect's column-definition formatter delegates to the shared constraint formatter rather than keeping a second implementation of the same grammar. - Column-level PRIMARY KEY now renders the deferral traits it was silently dropping; that is the one behaviour change, and the byte baseline records it.
The tests speak the tag language the nodes used to speak; the branch nodes replace it, so the suite is rewritten against them. - Constraint construction goes through branch=; a branch class is required, and a bare tag is refused. - "Which kind is this constraint" is asked of the branch, not of a field. - A missing payload is asserted where the branch is built, not where the clause is rendered. - The byte baseline is rewritten against the branch nodes and records the one behaviour change: a column-level PRIMARY KEY now renders the deferral traits it used to drop. - The roundtrip contract classifies the two branch bases as category names that declare no formatter, and the testsuite gains special constructors for the branches whose payload a heuristic cannot supply.
The S3 branch nodes need the special constructors the testsuite branch registers; installing testsuite from release would leave the roundtrip contract unable to build them.
…ming spec BREAKING CHANGE: the *Expression suffix no longer denotes values or function calls; kind is now carried by the suffix itself (4.9 naming table). - Value family takes the Value suffix: SQLValueExpression -> SQLValue, TemporalValueExpression -> TemporalValue (with its Date/Time/Timestamp/ Interval derivatives), JSONValueExpression -> JSONValue, ArrayValueExpression -> ArrayValue, BinaryValueExpression -> BinaryValue, XMLValueExpression -> XMLValue, UUIDValueExpression -> UUIDValue, NumericValueExpression -> NumericValue, IntegerValueExpression -> IntegerValue, StringValueExpression -> StringValue, BooleanValueExpression -> BooleanValue, WildcardExpression -> Wildcard, QualifiedIdentifierExpression -> QualifiedIdentifier. - Boolean logic joins the value family as LogicalValue, not Predicate: it renders AND/OR/NOT as a value usable in a SELECT list, where LogicalPredicate is the WHERE-clause form of the same operators. Both docstrings state the distinction so the two are never conflated. - Function family takes the Function suffix: CastExpression -> CastFunction, CaseExpression -> CaseFunction, Trim/Repeat/Lpad/Rpad/Collate *Expression -> *Function. - JSON access keeps value semantics: JSONDocumentExpression -> JSONDocumentValue, JSONTextExpression -> JSONTextValue. - ArrayExpression becomes ArrayOperation, not ArrayValue: its subject is the ANY/ALL/constructor/access operation, which would have collided with the array value type that ArrayValueExpression already claims in core. The two docstrings now point at each other. - Introspection containers gain the Introspection prefix: DatabaseInfo, TableInfo, ColumnInfo, ViewInfo, TriggerInfo and ForeignKeyInfo become IntrospectionDatabaseInfo and friends, matching the rule that introspection is named per-capability rather than as a generic container. Known follow-up: the roundtrip contract still mis-builds LikePredicate (a raw string reaches an expression slot); to be fixed in the next commit.
…c 4.10 BREAKING CHANGE: Wildcard is gone. The asterisk is one lexical token in two contracts, and each now has its own class, formatter entry and parameter contract. - AllColumnsWildcard(dialect, relation=None) is the select-list and RETURNING form: a bare * when no relation is set (the surrounding clause's formatter injects what the expansion spans), rel.* when one is. The qualified form's quoting rides the relation object; the node carries no flags of its own. A dialect without schema qualification now refuses a schema-qualified wildcard explicitly instead of emitting a name the engine cannot resolve -- the old formatter bypassed that gate. - RowCountWildcard(dialect) is COUNT(*)'s parameter: bare * only, no relation slot. COUNT is the only aggregate whose contract admits it; sum_/avg_/min_/max_ refuse it (and a bare "*" string) at construction, and count() refuses the all-columns wildcard the same way -- COUNT(table.*) is backend territory (a non-NULL row count), defined by supporting backends as their own derived node. - format_wildcard splits into format_all_columns_wildcard and format_row_count_wildcard, keeping the class/kind/formatter three-place homology; the WildcardSupport protocol declares both entries. - The roundtrip contract registers special constructors for the wildcard pair and the LIKE/ILIKE predicates, whose operands are value nodes the introspective guess was supplying as strings.
The typed-column rebuild (0245b29) put the issubclass check in UseColumnType.__init__ but left ColumnBase imported only under TYPE_CHECKING. The annotation is a string and needed nothing at runtime; the check did, so every marker declaration raised NameError -- which is what turned the whole CI matrix red at e5e0689. Resolved at the point of use, the way this module keeps backend imports lazy and the same way ddl_table.ColumnConstraint.__init__ resolves its branch classes.
The guard scans the expression core for the typed-value family by suffix, and it still scanned for ValueExpression. 5dc64fd renamed that family to the Value suffix (规范 4.9), so the scan came back empty and the only assertion left to fail was 'the typed expressions moved' -- the single red test in the matrix at ba05ec7. Scans Value now, and the test name and docstring follow the same vocabulary: a value declares its family by being a class, not by an Expression suffix.
Thirty commits collapsed into one. They were one change arriving in pieces: the
typed value wrapper, which held a FunctionCall so that the result would say what
it was rather than what produced it, was the last thing standing between a
grammar and the nodes that state it. This removes it and everything built to
prop it.
What the wrapper did: NumericValue, StringValue, TemporalValue and ten siblings
each did two jobs. They said what a value *was*, and they carried every operation
that kind of value supports. A node that *was* a number and a node that *yielded*
one therefore had to be the same class -- so a function result could not be its
own node unless it was a wrapper around a generic call, and the artifact read
{"kind": "numeric_value", "params": {"call": {...}}}.
What replaced it is two names for two claims. expression/surfaces.py holds the
eight surfaces: what can be done *to* a value, carrying no __init__ and no
to_sql, because a surface is not a node and cannot render. Seventy-two node
classes name the surface they yield; ten typed value literals carry theirs; a
column maps to the annotation typing an operation over it.
The surfaces declare an abstract format_method, which does two jobs.
auto_register_builtins skips a class with unimplemented abstract methods, so a
surface never enters the kind whitelist as a name nothing can produce -- and a
node that forgets its own format_method now fails when its class is built rather
than when it is rendered, which is the difference between a mistake at import and
a mistake at query time.
Two places where order decides whether a thing works, both learned the hard way:
a node's concrete formatter must precede the surface in the base list, or the
surface's abstract one wins and the class cannot be built; and a dialect that
overrides a mixin must be listed before the mixin, or the override is dead code.
The second is now a written clause rather than folklore.
Three gates were dead and now run. _is_interval_value asked about a class nothing
in the repository ever constructed, so it was always False and every temporal
refusal below it was unreachable; an interval is a node now, so the node says it
is a span, and mixins.IntervalMixin is that declaration. The operation-agreed
result types -- GREATEST, LEAST, NULLIF, CASE, CAST, the window value functions
-- reached a wrapper through a table walk and now answer with an annotation or a
surface.
The registry had the same defect one level up: it registered every
BaseExpression subclass it walked past, which is why a surface would have entered
the kind whitelist had it not been abstract first.
The namespace is the one the naming spec asked for, arrived at over thirty
commits: Pivot/Unpivot lose the Expression suffix, RawSQLExpression becomes
RawSQL, the ORDER BY entry becomes OrderByItem, the standard scalar productions
go back to *Expression, FunctionCall is reserved for user-defined functions, and
the arithmetic and concatenation operations are named as the core names them.
Two clauses were written down because the code kept arguing them: override order,
the reflection ban and enum-first now live in the organisation's
expression-dialect.md §11, language-neutral, with the per-language spelling on
each side. Nine closed value domains that were private tuples, frozensets and
string-keyed dicts became five enums -- including a sort-direction set that had
been declared in the expression layer and again in the dialect layer with nothing
keeping them in step. Four probes that asked whether an object had a field became
isinstance assertions.
Not included, and deliberately: uuid._require still looks its probe up by name.
Handing it the bound method breaks two round-trip cases for a reason not yet
established, and that is not a change to make without knowing why. DDL literal
and inline_params were expected to be dead and are not -- inline_params is read at
dialect/mixins/expression.py:127 and DdlLiteral is constructed by two test
modules.
SQL is unchanged throughout. Rendering probes against the pre-squash tree cover
the function families, the literal encodings, and every path that used to go
through a wrapper -- set operations, sort directions, arithmetic, the datetime
aliases, the UUID constants, GREATEST / LEAST / NULLIF / LAG / LEAD /
FIRST_VALUE / CAST / CASE / COALESCE -- and are byte-identical, which is the
point: only the knowledge of what a node yields moved.
Per-commit messages are kept at .claude/plan/2026-10-17/squashed-commits.md.
get_params() walked the constructor signature and probed for `_{name}` and then
`name`, choosing between them by testing whether the public one was callable.
There were only ever two reasons for that, and neither survives.
The first was eighteen fluent setters named exactly like the constructor
parameter they set -- schema, table_name, view_name, trigger_name, table_type,
include_columns, include_indexes, include_foreign_keys, include_views,
include_system, include_hidden, begin_type, isolation_level, deferrable,
not_deferrable, wait, no_wait, session. `self.schema = x` in __init__ would have
shadowed the method, so the value went to `self._schema` and the probe read it
back. The setters are `with_*` now: the name states that they mutate and return
self, which is what they do, and it frees the parameter to own its own name.
Eleven call sites followed, in introspection/base.py, the two sqlite transaction
implementations, the sqlite examples, and the tests.
The second was a handful of values stored privately with no method to account
for it: `_mode` on the transaction statements, `_savepoint` on RollbackStatement,
`_values` in one test fixture. Those are just stored under the wrong name.
Removing the probe turned three silent omissions into loud ones, which is the
point. BeginTransactionStatement never stored `mode` under any reachable name;
RollbackStatement never stored `savepoint`; a test fixture never stored its
`*values`. The old branch warned and skipped, which put the omission into the
artifact as an absent parameter -- a serialized node that reconstructs into a
different tree, with nothing to show for it. Reading the name has no fallback to
reach for, so the AttributeError names the class and the parameter.
Two tests pinned the warn-and-skip behaviour and now pin the contract instead:
test_missing_attribute_warns became test_missing_attribute_raises in
test_expression_serialization.py and in test_expression_edge_branches.py.
Two judgement calls worth recording. `_precision` on the five niladic datetime
nodes is a private *helper method*, not a competing store -- `self.precision`
was already correct, and the probe was taking the public branch, so nothing there
changed. And backend/transaction.py has an `isolation_level` property on
TransactionManagerBase which is not an expression parameter at all; an early pass
renamed it and broke the property pair, and it was reverted.
Verified: dummy2 (including every round-trip test), sqlite4 except one
pre-existing Windows temp-file failure that fails identically on HEAD, and twelve
query and backend feature files pass. Sixteen introspection and transaction
probes -- rendering and get_params -- are byte-identical against HEAD.
`string_concatenation_function()` returned `self.STRING_CONCATENATION` verbatim, so a dialect declaring anything other than a string answered with that value -- an int reaching SQL text where a function name belongs. The return type says `Optional[str]` and the test already said a non-string declaration is ignored; the implementation simply never checked. Read as "no override", which is the answer a dialect that never declared one gets, so a wrong type degrades to the standard's `||` instead of interpolating the author's slip into the statement.
Seven commits collapsed into one. They are one change: an artifact stopped
naming a Python class and started naming a kind, and everything that followed
from that -- the whitelist, what a value looks like on the wire, and who
converts it.
An expression artifact used to carry `{type, module, params}` and be read back by
importing whatever module the string named. That is the wrong shape for a value
object on an untrusted channel: a dotted path forces a reader to import an
attacker-chosen module, and resolution by import means the answer to "what class
is this?" lives in a table nobody maintains. A node is now a bare
`{kind, params}`, where the kind is the class name's snake_case form, computed by
`BaseExpression.kind()` and never stored, so the two cannot drift. Resolution is
by whitelist: `ExpressionRegistry` starts empty, imports no expression package,
and therefore cannot discover anything. Whatever a package did not register
answers `kind_not_registered`, which is what makes selective assembly real -- a
deployment that wants no DDL simply does not register it.
The registry owns one mapping and nothing else. It does not own naming, does not
construct, and -- this took a correction -- does not dispatch rendering. It had
carried a `(dialect, kind) -> render` table with no callers, left over from a
design in which formatting went through the registry; an expression declares its
own `format_method` and the dialect supplies it. Three primitives were missing
and are now present: `unregister_node`, and `disable`/`enable` with `is_enabled`,
kept distinct because "absent from the build" and "present and declined" are
different answers and folding them together loses the distinction.
Values carry themselves. Wire §5 reserves exactly two keys, `__b64__` and
`__dur__`, and everything else is a bare scalar: a date is "2026-10-17", a
Decimal is "1.50", a UUID its canonical string. Twelve kinds of value probe
round-trip byte-for-byte through both the dict and the JSON path. The conversion
that makes that readable lives in the constructor, in one method --
`TypedValue.normalise` -- because the expression declares what it accepts and is
the one place that says so. Three mechanisms that had accumulated around this
were removed rather than reconciled: a `codec.py` whose scalar table restated
what seven `normalise` overrides say (deleted), a `to_wire` that only delegated
to it and whose one caller then re-tested for `None`, and a `from_wire` that
duplicated what the constructor already did.
A nested node is its own artifact. `{"__expr__": {"kind": ...}}` is gone: a node
is `{kind, params}` at every depth, so the reader identifies one by shape and
asks nothing else. Whether a position *wants* a node is answered by the
constructor it is handed to. The first attempt at this asked the registry whether
the kind was known, which failed the shape test for an unregistered kind and let
it through as a plain mapping -- bypassing the whitelist by consulting it too
early. The shape test is shape only; an unregistered kind reaches the decoder and
is refused by name.
Strictness found real defects rather than cosmetic ones. `get_params()` no longer
probes `_{name}` then `name` and skips what it cannot find; it reads the
parameter's own name, so a parameter the class never stored raises an
`AttributeError` naming both. Eighteen fluent setters were named exactly like
their constructor parameters -- `schema`, `table_name`, `include_columns`,
`begin_type` -- so the value could not be written as `self.schema = ...` without
shadowing the method; they are `with_*` now and the parameter owns its name. An
annotation produced by `as_text()` could not be serialized at all, because
`_Annotation` called `BaseExpression.__init__` without inheriting from it and so
had no `kind()`; it inherits now.
Verified: expression/ and dummy2 pass, as do the value-lattice, temporal,
document, operation-boundary, symmetry and numeric feature files. Per-commit
messages are kept at .claude/plan/2026-10-18/squashed-commits.md.
Five SQLite nodes overrode ``to_sql`` to call ``dialect.format_*`` themselves. That is the ``format_method`` mechanism with the declaration left out: the node looked up its own renderer instead of naming one, so the node knew where its formatting lived. An expression declares the name; the dialect supplies the method; nothing in the expression calls the dialect. Changed: SQLiteVacuumStatement and SQLiteAnalyzeStatement, SQLiteAttachExpression and SQLiteDetachExpression, SQLiteReindexStatement. Each now returns ``format_vacuum_statement`` / ``format_analyze_statement`` / ``format_attach_expression`` / ``format_detach_expression`` / ``format_reindex_statement`` from ``format_method``, and the ``to_sql`` override is gone. Rendering is unchanged -- VACUUM, VACUUM INTO 'b.db', ANALYZE and the rest still come out byte-identical -- which is the point: the two spellings were always doing the same thing, and only one of them lets the registry see the node. That last part is why this was worth doing. These five were among thirty kinds that the registry accepted but which could not render: their ``format_method`` resolved to BaseExpression's sentinel, so ``to_sql()`` raised NotImplementedError. Nothing noticed, because nothing enumerates "can every registered kind render?" -- the artifact path was the only consumer, and these nodes are not read back from artifacts in practice. 25 kinds still cannot render. Eleven are SQLite nodes that build SQL by hand in ``to_sql`` (FTS5, R*Tree, Geopoly, MATCH) and need a real split; fourteen are internal and contract bases -- SQLValue, SQLPredicate, SchemaObject, RelationObject and friends -- which should never have been registered at all. Verified: expression/, dummy2 and sqlite4 pass (one Windows temp-file failure deselected, which fails identically on HEAD).
Sixteen commits, one change: an expression artifact is plain JSON, and every
spelling it needs belongs to the object that owns it.
**The artifact carries no SQL and no markers.**
A node's kind is its identity -- derived from the class name, computed and never
stored, so the two cannot drift. A literal keeps kinds of its own
(`integer_value`, `date_value`, ...) because a date's spelling and a string's
are the same bytes and only the kind tells them apart.
The marker vocabulary is gone: `__expr__`, `__tuple__`, `__enum__`,
`__dataclass__`, `__b64__`, `__dur__`. A marker is a dispatch -- the reader
must recognise the key and know what it means -- so every language reimplements
it and they drift. The artifact has to be byte-identical across four
implementations, so anything a reader dispatches on has to be agreed by
construction. A JSON scalar needs no agreement.
Operators name the operation; the formatter writes the symbol. The comparison
family carried `=` / `>=` / `<>` in a payload meant to be language-neutral, so
its artifacts read as SQL and no other port could reproduce them without
hardcoding the same characters. `ComparisonOperator` now mirrors
`ArithmeticOperator` exactly, and the constructor normalises, so all 63 call
sites keep passing ">=" unchanged and get the same node. `!=` and `<>` are one
operation, so both normalise to `NE` and the formatter writes the standard
`<>`.
The enums follow the same split: `SelectModifier`, `JoinType`, `LockStrength`,
`SortDirection` and a dozen more write words, and the formatter supplies the
SQL. `SelectModifier` was not writable at all -- a plain `Enum` is not a JSON
scalar, so `SELECT DISTINCT` could not be captured. `LockStrength`'s values
were the clauses themselves ("FOR UPDATE"), so a strength was a spelling; a
dialect that cannot support a strength has to be able to say so by name.
NaN and the infinities are ordinary members of a float, and JSON's number
grammar (RFC 8259) has no literal for them. `json.dumps` writes a bare `NaN`
anyway -- its own documentation calls that non-compliant -- and every other
port refused the bytes. They now travel as text under kinds that name how to
read them back, the way `binary_value` and `interval_value` already did.
Infinity carries its sign in the text rather than being two kinds: the sign is
the content of the value, not a different concept.
**The vocabulary belongs to the object that owns it.**
Nothing lives at module level. A spelling table belongs to the object that
owns the vocabulary, and a coercion belongs to the class whose input invariant
it guards.
* `ComparisonOperator.symbols()` / `.characters()` and
`LikeOperator.keywords()` replace the module-level tables. An Enum turns
any non-descriptor in its class body into a member, so a mapping cannot be
a class attribute there without adding a spurious kind to the artifact; a
classmethod is a descriptor, so the table lives on the enum as a method.
* `ComparisonPredicate.coerce_operator()` and
`LikePredicate.coerce_operator()` replace the module-level normalisers.
The coercion guards the predicate's own input invariant, so it belongs
with the attribute it guards.
* `quote_flag` and `require_exclusive_options` had sixteen callers across
the tree and all of them inherit `BaseExpression`, so they became
classmethods there. `quote_flag` used to take the owner's name as an
argument, precisely because it could not be a method.
* An annotation's value type is read off its bases
(`_Annotation.value_type_of()`) instead of being declared beside them,
which took the eleven hand-copied mixin lists and the ten redundant
fields with it.
**The registry registers only what can render.**
A class entered the kind whitelist without ever being able to say how it
renders: thirty of the registered kinds resolved their `format_method` to the
sentinel on `BaseExpression`, so they could be constructed, resolved from an
artifact, and then raised `NotImplementedError` on render. The test is now the
*resolved* `format_method`, asked by definition site -- a base supplying one
for its subclasses is a real declaration, and only `BaseExpression`'s own
sentinel means "none".
The remaining eleven could render but reached the dialect themselves: their
`to_sql` called `dialect.format_*`, which is the `format_method` mechanism
with the declaration omitted. Eleven SQLite nodes -- FTS5, R*Tree, Geopoly,
MATCH -- now return that name from `format_method` instead, and the `to_sql`
override is gone. The node no longer knows where its formatting lives.
**What the strictness found.**
Imposing it turned up real defects rather than cosmetic ones:
* `_restrict` silently dropped all fourteen value kinds, because it rebuilt
its view node by node and `register_node` keys on the *class* name, so
every value kind filed under `literal`. The table looked populated and
refused every artifact carrying a value.
* Ten enums were plain `Enum`, a violation the old `__enum__` wrapper had
been hiding -- with a tag, any `Enum` was writable.
* `RawSQL`'s bind values did not survive a round-trip at all: `b"\x00"` and
a string are the same three bytes with no kind to tell them apart.
* `nowait` was rendered unconditionally while its sibling `skip_locked` was
gated, so SQLite received `FOR UPDATE NOWAIT` -- rejected at execution
time, with the transaction already open.
* `get_params()` probed `_{name}` then `name` and warned past what it could
not find.
* `LateralExpression` carries its own `join_type` and had been missed by the
enum conversion entirely.
**SQLite gets its schema back.**
The introspection SELECT-class formatters are now expression trees rather than
string concatenation, which exposed a correctness defect: dropping the `main.`
qualifier made `list_views(schema='other')` silently return the primary
database's rows. Naming an attached database is not the SQL-standard
construct, so it is answered by a SQLite-specific override keyed on node
identity, and the DDL probes stay False. A core table carrying a schema is
still refused.
Also in this commit: `.worktree/` and the branch toolchain directories are
ignored. Feature and fix branches live in `.worktree/<name>/` per the
organisation-wide worktree rule; both worktrees share one `.git` directory but
each has its own checked-out `.gitignore`, so the rule has to be recorded on
the branch being developed as well.
BREAKING CHANGE: the artifact is a bare `{kind, params}` node -- no envelope,
no markers -- and artifacts written before this commit cannot be read by a
reader after it. `comparison_predicate.op` writes the operation word (`gte`)
rather than the SQL character (`>=`), and accepts either at the constructor;
`join_clause.join_type` writes `left` rather than the SQL phrase;
`for_update_clause.strength` writes `update` rather than `FOR UPDATE`;
`query_statement.select_modifier` writes `distinct` rather than `DISTINCT`;
an ORDER BY direction writes `desc` rather than `DESC`. `interval_value` and
`binary_value` write the bare text their kind names rather than a wrapper
object. `RawSQL` and `RawSQLPredicate` are no longer registered by default.
The module docstring was the last place still describing the format the codec
had already stopped speaking. It documented two things that no longer exist:
* A **class index** -- a deserialization support table keyed by fully
qualified class name, with a note that "the kind whitelist arrives with
the bare-node rewrite". That rewrite is this branch. The reader resolves
a kind through the registry now, and there is no class index to consult.
* A **reserved-key vocabulary** -- `__expr__`, `__tuple__`, `__vdc__`,
`__b64__`, `__dur__` -- together with a warning that user-defined
parameters must not use them. The markers are gone: a nested node is a
nested `{kind, params}` recognised by its shape, and a value JSON cannot
spell travels as the text its kind names. Nothing is reserved any more,
so there is nothing to warn about.
The docstring was also quietly wrong about the registry: it said the registry
does registration, checking and dispatch of *rendering*, "not of classes",
which is true of what the registry does and false of what it is for -- it is
the table that turns a kind back into a class.
Only prose changes here. The codec was already right, and so were the two
mentions of the old markers that remain in the tree: the ones in
`docs/*/backend/expression/serialization.md` and the one in `values.py` say
why the wrapper was dropped, which is the reason to keep them.
Verified: ruff reports nothing; black's complaints on this file are the same
before and after (they are all below line 250 and none of them is in the
docstring); the expression artifact contract tests pass.
Geopoly is an optional module, so a SQLite build may or may not carry it. The cases skip where it is absent -- but whether it is there is asked of the SQLite itself rather than of the dialect, because the dialect's probe falls back to a version check when it has no compile options to read, and a build can carry a version number without the module. The skip is not the interesting part; the cases are. Geopoly's statement was the one the framework used to render with an option Geopoly does not have, and nothing checked it against a server that had the module. Now it does.
The dialect's surface is two kinds of member: capability switches (``supports_*()``) and formatters (``format_*(self, expr)``). This change makes the code match that, and settles what a type position holds while it is there. **A formatter owns its production.** Six private renderers in core's dialect mixins -- ``_format_function``, ``_format_scalar_function``, ``_format_aggregate_function``, ``_format_niladic_function``, ``_format_optional_length_function``, ``_format_window_rank`` -- served families of thin ``format_*(self, expr)`` wrappers. The wrappers were formatters; the bodies were not, and being private they could not be overridden, which is what ``architecture.md`` §8 and rule A5 both say is wrong. The bodies moved into the formatters that used them, and the six renderers are gone. The duplication is the price: a formatter a backend can override is worth more than a helper it cannot. **A type-name position holds a type.** The positions that carried a raw string -- XMLSERIALIZE's target type, an XMLTABLE column's type, ALTER COLUMN ... SET DATA TYPE's type -- guarded the string with a character class in the base class. That class neither matched what the engines accept nor kept the value inside its syntactic position: ``TEXT AS (SELECT 1`` passed it. A position now holds a type node; a name the framework does not model becomes the framework's name node, whose name the grammar validates. Whether the type exists is the server's answer to give. **Binding is the caller's call.** A DDL position that cannot bind says so through its dialect's own capability switch, and a value the caller chose to inline is inlined. The framework does not refuse on the caller's behalf: a literal that was not marked inline stays a bind value, and the engine that has no room for one is the one that refuses. **The backend's own answers are the effective ones.** A backend's mixin now precedes the core mixin it overrides, so the override is not dead code, and a test holds that in place. **The rules say what the code does.** ``architecture.md`` and the two skills described a ``requires_inline_literals()`` switch that exists nowhere in the code -- it was abandoned, and the same files called it a current contract. They now describe the switch that is there: the caller marks the literal, and the engine refuses what it cannot take. A6 says where shared rendering logic lives, and the function examples reach their constructors through ``functions as F``, so a reader sees at a glance which names the framework provides.
The expansion that folded core's private renderers into the formatters that used them replaced the call in place. In these two the call was one branch of an ``if``, and the branch after it -- the arithmetic SQLite needs for a precision -- became dead code, so a two-argument TRUNC or LOG rendered as a plain call instead of the scaled-and-truncated form. Each branch now holds its own production, and the branch structure is the one the code had.
…tement The convention is already in place: a node that renders a whole statement is a ``*Statement`` (51 in core), a fragment is an ``*Expression``. Four statement nodes still carried the old name: TransactionExpression -> TransactionStatement (the base of the six) SetTransactionExpression -> SetTransactionStatement SQLiteAttachExpression -> SQLiteAttachStatement SQLiteDetachExpression -> SQLiteDetachStatement The formatters already said ``format_attach_statement`` and ``format_detach_statement``; the nodes were the only part still calling themselves expressions. A transaction statement is a statement, and the class that the six of them share is named for what they are. Nothing changes but the names: no behaviour, no rendering, no signature.
The SQL Server and Firebird variants derive from ``CreateTableStatement`` and render a whole statement; the docs still named them ``*Expression``. The names now match what the classes are.
A node now carries one name in three places: the class name, the kind derived from it, and the dialect formatter it names. - Datetime* replaces DateTime*: datetime is one word, so its kind is datetime_add_expression, not date_time_add_expression. - The introspection nodes render a SELECT, so they are *QueryStatement (DatabaseInfoQueryStatement and the rest) and their formatters carry the same stem. - Formatters that added, dropped or respelled a word now match the kind: storage/temporal options, uuid, XML, geopoly, the binary and unary operators, COMMENT ON, COMMIT, ROLLBACK and SET TRANSACTION. - kind_name is gone. It was a stored second source of truth nothing read, and two entries had already drifted from the derived kind. - A guard test derives each *Statement/*Expression kind and requires its formatter to be format_<kind>, with the backend prefix optional.
Ten names carried a leading underscore, which says "internal to this module". Nine of the ten are called from outside it -- from another core module, or from a backend, or both -- and the tenth, ``check_spelling``, is called from all nine backends and from no core code at all. # ``check_spelling``: 130 backend calls, 0 core calls. # ``set_cached`` / ``get_cached`` / ``make_cache_key``: 24 core calls each # plus 17-20 backend calls. # ``register_type``: 19 core, 32 backend. ``is_sql_expression``: 6 core, # 13 backend. ``remap_row_columns`` 2/13, ``adapt_row_types`` 2/17, # ``refuse_ddl_bind_parameters`` 3/8, ``type_class_for`` 1/2. The underscore bought nothing and cost the thing that matters. It marks a name as private to its own tree, so nothing outside could tell that these were contract points -- and when the testsuite's adapters called ``self._register_type`` from another package, there was no tool, no type checker and no reader that could flag the coupling. ``is_sql_expression`` was the plainest case: Oracle imported it as ``from rhosocial.activerecord.backend.base.operations import _is_sql_expression``. A private name does not belong in another package's import list. Each new name was checked against every public definition and call site in all ten repositories before it was used, so no public name is shadowed by moving these out from behind the underscore.
…n layer stops holding state Introspection had one class doing four jobs. `IntrospectorMixin` held the TTL cache, the SQL builders and the parsing contracts at once, so the layer whose only business is turning a question into a statement was carrying mutable state and an abstract I/O contract for results it never sees. The split follows the responsibilities instead: * `IntrospectionCache` is an interface. `InMemoryIntrospectionCache` is the local implementation that ships here -- in-memory, TTL on read. What a cache does about expiry or sharing is the implementation's business, and a backend can put its own behind the manager. * `IntrospectorMixin` keeps only what builds SQL: the dialect shortcut, the schema helper and the `_build_*` methods. No cache, no lock, no state, no parsing contract. * `_IntrospectionManagerBase` holds `backend`, `executor` and the cache, declares the `_parse_*` contracts and answers for them, because it is the part with a database behind it. Sync and async stay separate and equal, as before; only how they run the SQL differs. The cache is reachable as `introspector.cache`, and injectable, so a backend that would rather have always-fresh metadata than a fast answer can supply a cache with no TTL or one of its own. Key policy is on the interface on purpose. It is the correctness contract: two callers asking about materialized views and ordinary views must agree on a key despite sharing `IntrospectionScope.VIEW`, and the `extra` parameter is how they differ. One home for that rule, not a convention repeated at every call site.
The CLI reaches the introspection manager already -- `backend.introspector` is the manager -- so there was nothing to wire. What was unverified is the property that lets the CLI stop thinking about the cache: a command is one process in normal use, but a script can drive `main()` twice, and if two runs reached the same metadata a table created between them would be invisible. Nothing in the CLI invalidates anything, so the guarantee has to come from each run building its own manager. Now it is tested: a table created by a separate connection between two runs shows up in the second. The assertion reads JSON rather than the default table, because the rendered table wraps every cell to five characters and `added_later` never appears as one contiguous string -- asserting on that text would test the renderer's wrapping instead of the answer. That wrapping is its own defect and is recorded in the plan, not fixed here.
`AsyncAbstractIntrospector` subclassed `IntrospectorMixin` rather than the manager base, so it had no cache, no `invalidate_cache`, no `clear_cache`, and its `__init__` fell through to `object.__init__` -- every async introspector raised `TypeError` on construction. Nothing said so, because the cache tests only ever built the sync manager: async introspection had been broken since the cache became a component, while the sync suite stayed green. It now shares `_IntrospectionManagerBase`, which is where the cache, the parsing contract and the two invalidation methods live. This also completes the injection seam the base already documented: * `cache` gains a setter. The docstring promised a caller could replace what the manager caches with, but only reading it worked. * both abstract introspectors -- and sqlite's pair -- accept `cache=` and forward it, so a backend chooses its cache at construction instead of assigning to a private field afterwards. Tests: a new async suite pins what the sync suite pins -- the second walk asks the database nothing, a ttl=0 cache asks every time, invalidation makes a new table visible, and the cache is replaceable through the property and at construction. The sync tests now use those public seams instead of assigning `_cache` and `_executor`.
A field-by-field review against the other CLIs found values the library computed and then dropped. Four of them were the library asking the engine a question and ignoring the answer, and one was not asking at all: * `list_views` projected `name` but not `sql`, so every view reported `definition=None` even though `_parse_views` reads it. A caller had no way to tell "this view has no definition" from "we never asked". * `is_unique` was left False on every column, because SQLite records uniqueness on the index and the parser only read the column row. That is a wrong answer rather than a missing one: a caller building a model from it drops a constraint that exists. * a trigger's `timing` and `events` were left empty although both are in the DDL, which the introspector already fetches. * the database's `encoding`, `size_bytes` and `table_count` were left None although each is one PRAGMA or one count away. The column list now also reads the index pragmas and passes the unique column set to the parser, as a set rather than by parsing the DDL text: a partial unique index is excluded, because it constrains only the rows that pass its WHERE and is not the promise `is_unique` makes. The two directions share one reduction so they cannot disagree. `get_database_info` is overridden on both managers to gather the pragmas and hand the same dict to the same parser -- `_parse_database_info` stays pure, so the sync and async paths answer identically. `pragma.get` returns a whole row, so the scalar is picked out of it rather than reported as a dict. `introspect table -o json` also emitted three concatenated JSON arrays, which is not a document: a parser gets `Extra data` and nothing. It now emits one object keyed by section, while the rendered table and csv/tsv keep their own sections -- csv has no single-document shape for three record families. Eight tests pin each answer, including that a partial unique index does not make a column unique and that the two new cache entries stay distinct.
A backend's CLI listed its types by hand, and a hand-written list drifts from what the backend can actually answer. ClickHouse shipped `foreign-keys` and `triggers` for an engine that declares it has neither, and `introspect triggers` failed with a TypeError rather than the capability refusal it was written to produce -- the user found that out from an error, not from --help. `available_introspect_types(dialect, declared)` filters a declared list against the dialect, one capability per family. Types with no switch are offered as before; types whose switch method does not exist on the dialect are dropped, because an absent switch is not evidence the family works. `TYPE_CAPABILITIES` is the single place the mapping lives, and it names the same switch for the same family in every backend -- `supports_sequence_introspection` answers for sequences in Postgres, Oracle, MariaDB and SQL Server alike, so a caller learns whether sequences can be asked about without trying. The core module is new and shared; no backend changes behaviour through it yet, because a derived list has to be proved on one backend before it is spread to nine. ClickHouse is that one.
`backend/cli/introspect_capabilities.py` mapped each introspection family to a `supports_*` switch and filtered a backend's declared list against it. That made core the arbiter of a vocabulary it does not own: a backend adding a family had to come back and edit core's dict, and the dict assumed the nine engines share a naming scheme when their family names are merely similar. A single `str -> str` mapping also could not express the differences that actually decide the answer. ClickHouse's `triggers` are structurally absent, MariaDB's `sequences` are gated on 10.3+, and `schemas` means different things in SQL Server and Snowflake; all three collapse into one lookup. And the filter ran at parser construction, before connecting, so it could not see the server version it was trying to reason about. Which families a CLI offers is the backend's own hand-written `INTROSPECT_TYPES`; version gating belongs after `introspect_and_adapt()`, in the one backend that needs it. The module is deleted, and with it the `backend/cli/` package, which existed only to hold this file. Nothing in core imported it, so no call sites change. The `supports_*` switches stay on the dialects: they serve library callers, not the CLI list.
…s its underscores Introspection used to spell its own SQL. PRAGMA was built by hand in PragmaInfo.get_sql, in the pragma introspector's _pragma_sql and again in the backend's set path; the SQLite status introspector read sqlite_master through a literal; and DDL lookups glued a name into a statement with an f-string. None of that is dialect-neutral, all of it sat in modules that only know SQLite, and none of it could be checked without a database. Every question now names a statement instead. A statement holds its parameters and renders nothing; the dialect's format_*_query_statement decides the SQL; the backend only chooses which statement to ask for. SQLite gains statements of its own for PRAGMA reads and writes, for the catalogue counts and for table and index DDL, so "PRAGMA" and "sqlite_master" now appear in formatters and nowhere else. Statements collect every parameter in __init__ -- the chained builder that added one parameter per call is gone. Three names were private that were never optional: default_schema, the parse_* row parsers and the manager base class. Every backend implements the parsers and the base calls them, so the underscores described a visibility the code never had. default_schema becomes a property, beside dialect and cache: it states a fact about the database rather than performing an action. Two discarded "schema or default_schema" expressions in get_table_info go with it, sync and async -- they never assigned anything, so nothing changed behaviourally. The four principles behind this are recorded in .claude/rules/introspection.md: the dialect owns the SQL, a backend owns its own vocabulary, a statement carries its parameters from construction, and the public surface states facts as properties.
format_logical_predicate joined its operands with the bare connective, so a tree and the SQL it produced disagreed whenever the nesting did not follow SQL's own precedence. "a AND (b OR c)" rendered as "a AND b OR c", which SQL reads as "(a AND b) OR c": a different predicate, usually a wider one, with nothing to show for the difference -- the statement parses, the server answers, and the rows belong to another question. The realistic shape is a caller's filter AND-ed with a default exclusion written as an OR: "object = ? AND (relation IS NULL OR relation NOT LIKE 'RDB$%')". Flattened, the OR swallows the filter, so the answer stops depending on the object the caller named. That is how it surfaced: a Firebird introspection read passed its object parameter and returned every non-system grant anyway. An operand is now bracketed when its connective binds looser than its parent's. Equal or tighter stays flat, because "a AND b OR c" already means "(a AND b) OR c" and brackets there would state a rule this layer does not have. NOT keeps the brackets it always wrote, and a predicate that is not a connective still binds tightest of all. The rule is asserted where the SQL is written: nine cases in test_logical_predicate_precedence, five of which failed before this change. They pin both halves -- the brackets that must appear and the ones that must not -- because "bracket every nesting" would pass the five and quietly churn the SQL of every other shape.
"table" was the only introspect type that answered several catalogue questions at once: the relation's row, its columns, its indexes and its foreign keys. Each of those is an item the introspector reports on its own, and each already had a type, so the aggregate was a second way to ask for the same rows -- and the only way to get an unparseable answer: three arrays printed one after another is not a JSON document. display_table_detail existed to patch that one case into a single JSON object. Removing the aggregate removes the reason for it, so the helper goes with it. get_table_info stays: assembling the three parts in code is a reasonable convenience for a caller, and it is not a display concern.
TableType could only say BASE_TABLE or VIEW, so PostgreSQL and BigQuery had nowhere to put a materialized view and mapped it onto BASE_TABLE -- the one distinction the catalogue made, erased at the only layer that saw it. PARTITIONED TABLE met the same fate, and FOREIGN TABLE had to be spelled EXTERNAL beside genuinely external tables. Name the three kinds. Spellings that belong to one engine -- BigQuery's CLONE and SNAPSHOT, ClickHouse's engines, Snowflake's MANAGED and EVENT tables -- stay in `extra` beside the member they map to instead of widening the enum, and the introspection rules now say which of the two a new kind is.
A supports_* method decides whether SQL may use a feature, so it cannot raise when the version is unknown: the caller is asking precisely because it cannot assume the answer is yes, and DialectNotAdaptedException tells it nothing it can act on. Two of core's own predicates raised anyway -- SQLite's foreign-key check and the DQL FETCH ... WITH TIES default -- and the backends copied the pattern, so an introspection CLI that skipped introspect_and_adapt() died on a capability probe instead of reporting "not supported". Add version_or_none and version_at_least to SQLDialectBase. The strict version property still raises, because a renderer gating SQL on a version must know which version it is gating against; the predicates answer "not confirmed" until something resolves one. _declared_version() lets a dialect name the version it knows without a handshake -- SQLite's engine is its driver's library -- and SQLite's own predicates now read it. For the other direction, reporting a version that is already known: StorageBackendBase gains get_cached_server_version(), the I/O-free counterpart of get_server_version(), and introspectors expose server_version, which reads the dialect, then that cache, and never raises. A backend can fill IntrospectionDatabaseInfo.version from it without crashing when a CLI never adapted.
…tion A transaction belongs to the caller that opened it. When a second coroutine began one on the same connection, both drove the level counter: two "Beginning transaction (level 0)" in a row, then a commit that found level 0 and raised "No active transaction to commit" -- a message that names neither the cause nor the caller. Only transaction() warned about a shared backend, and pool.transaction() bypasses it by calling begin_transaction() directly. begin() itself had no check at all. It does now, for both the async manager (task identity) and the sync one (thread identity), and ownership is released when the outermost transaction ends. The tests drive the managers directly: a second task is refused while the owner may still nest, the connection is usable by another task after the commit, and the same for a second thread.
…e answers first Two leftovers from the version/kind work. SQLite's own predicates in virtual_table.py still read the strict version property, so supports_virtual_table() and the extension probes raised DialectNotAdaptedException on a dialect that had not been adapted -- the very defect the rest of the module just lost. They answer through version_at_least now, which is what _declared_version exists for. parse_tables folded every kind other than table/view into EXTERNAL. PRAGMA table_list reports four: shadow tables are the storage a virtual-table module maintains (an FTS5 table's fts_data/fts_idx), and `virtual` is the module's own table. Shadow tables are system-owned, so they are SYSTEM_TABLE; the virtual table the user created stays EXTERNAL.
9.2 removed the private `node._nodeid` alias that pytest-benchmark 5.3.0 still reads, so all 20 benchmark tests in tests/benchmark/relation/ errored at setup with `'Function' object has no attribute '_nodeid'` and the FastAPI step then failed for a missing output file. The same release also changed how the suite behaves under `-n auto --dist=loadgroup`, which turned 14 shared-state tests (worker hooks, the deep named-series CLI) red -- on a day when not one line of the library had changed, and on release/v1.0.0.dev30 exactly the same way. Nothing here pins a dependency to make CI pass: the toolchain was unpinned, so every run silently took whatever upstream published that morning, and the last green run (2026-10-27) had pytest 9.1.1 while the first red one (2026-10-28) had 9.2.0. The pin holds it where CI was last green; a bump is a deliberate act with a run to prove it.
The first pin said `pytest~=9.1.1` for every interpreter, which is wrong: 9.x requires Python 3.10, so the 3.8 and 3.9 jobs failed at install time with "Could not find a version that satisfies the requirement" in 36 seconds. Those jobs were never running 9.x anyway -- on the last green run 3.9 resolved pytest 8.4.2 and 3.10+ resolved 9.1.1. The pin now says exactly that, split by environment marker, so each interpreter keeps the version it was green on.
`pool.transaction()` rolled back in its `except` and let whatever the rollback raised become the caller's exception. A rollback can fail for reasons that say nothing about the original failure -- a statement error aborts the transaction, so a second end finds nothing to end; the connection may be gone; another coroutine may have finished it -- and when that happened the body's exception was lost. Observed in a downstream service's webhook ledger: a duplicate-key `IntegrityError` on a concurrent delivery was reported as `TransactionError: No active transaction to rollback`, and the operator was pointed at the transaction rather than at the duplicate. The caller lost the only fact it could act on. Both pools now log the rollback failure and re-raise the original exception. Tests cover path 2 (connection-only) and path 3 (acquire) in the sync and async twins, and assert the context tokens are still cleared afterwards.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Lands the
fix/column-constraint-gatescycle onrelease/v1.0.0.dev30.Scope up front: this branch carries the whole cycle started on 2026-10-11 (the S3 expression/introspection refactor), not only the last two days of fixes. 56 commits, 17 of them marked breaking. The recent fixes are the last six of them.
Headline changes:
format_*_query_statementplus a public manager pair; the expression layer no longer holds state, and the result cache is an injectable component (235b034b,e9c48948,6dd304da,505303d8,6e34c75e).version_or_none/version_at_least/_declared_versionon the dialect,get_cached_server_version()on the backend,server_versionon an introspector. Askingsupports_*()no longer raisesDialectNotAdaptedExceptionwhen the version is not known; the strictdialect.versionstill raises, because a renderer gating SQL must know which version it gates against (c59fc9e5).TableTypegainedMATERIALIZED_VIEW,FOREIGN_TABLE,PARTITIONED_TABLE; SQLite's FTS shadow tables areSYSTEM_TABLErather thanEXTERNAL(c7d3a182,b38bc759).introspect tableis gone; the type list is derived from capability, so a listed type is always answerable (4f3162c5,6dd304da,4d2e0766).pool.transaction()no longer bypasses the check (5f000666).format_logical_predicaterenders nested groups (52105a13).Type of change
feat: A new featurerefactor: Internal restructuring with no intended behaviour change (added: not in the template list, required by Conventional Commits for the S3 cycle)test: Adding missing tests or correcting existing testsdocs/perf/chore/ci/build/revert(not relevant)Breaking Change
Impact on End-Users:
introspect tableno longer exists in any backend's CLI; usetables,views,columns,indexes,foreign-keys,triggers,databaseinstead.IntrospectionTableInfo.table_typemay now beMATERIALIZED_VIEW/PARTITIONED_TABLE/FOREIGN_TABLE, and on SQLite the storage tables behind an FTS or R-Tree virtual table areSYSTEM_TABLErather thanEXTERNAL.Falseinstead of raisingDialectNotAdaptedExceptionbefore adaptation. Code that caught that exception to detect an unadapted dialect must readdialect.version_or_noneinstead.Impact on Backend Developers (Custom Implementations):
*Statement,parse_*lost their leading underscores, and helpers called from another module became public (e9c48948,b7016bab,34e1fb45,890aaaa3,93ad24b9,101f71a5,fc4f2469).IntrospectionCacheis a component the manager owns and a caller may replace, rather than a hidden attribute (235b034b).8448dfb5); value and function families were renamed per the naming spec (5dc64fd5,556568ef); the wildcard node was split by grammar position (61a1a458).a2e8481f) — a dialect surface another implementation may have mirrored.Related Repositories/Packages
The matching branches of the other repositories are green at the commit recorded in each:
python-activerecord-testsuite32891035f000666python-activerecord-mysql65db7e2c_versioncrash, CLI adaptation, predicate vocabularypython-activerecord-mariadb05e944aapython-activerecord-postgres6eb017fetable_countpython-activerecord-oraclec478ce46python-activerecord-clickhouseccce0d14python-activerecord-snowflake8c49645dinclude_viewsno longer hides managed tablespython-activerecord-sqlserver7daabc30--include-systemworkspython-activerecord-bigquery579e18dfpython-activerecord-firebird61ba4b8eRDB$RELATION_TYPEprojectionAll ten still install core and testsuite from
@fix/column-constraint-gatesin CI; those references must be repointed atrelease/v1.0.0.dev30as each one merges, or the release branch keeps testing the fix branch.python-github-ci-trackerchecks out core and the postgres backend at this branch explicitly; it needs the same repointing once both have merged.Testing
Test Plan: CI runs the full matrix on this exact commit: 12 jobs (11 Python versions/builds plus benchmarks), all green — run
40477536642. The backend matrices exercise the same core against MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, ClickHouse and Firebird service containers. A local full run was skipped deliberately: the organisation caps a single task at 180 seconds, and the CI result on this SHA is the stronger evidence.Checklist
README.md,.claude/rules/introspection.md).format_*).Additional Notes
release/v1.0.0.dev30has 3 toolchain/documentation commits this branch does not have (04e1178d,fb4aab98,86ddfbe3). A dry-run merge auto-merged the three overlapping files (.gitignore,CLAUDE.md,.claude/testing.md) with zero conflicts, so GitHub generates a merge commit rather than rewriting history.python-schedule-manager,python-contact-manager,python-mcpserverandpython-userall declarerhosocial-activerecord>=1.0.0.dev30,<2.0.0, so a publish from this branch upgrades them without a version decision on their side. The fragment and the release notes need to name the breaking changes they inherit.