Repository navigation
feat(expression): rebuild the column layer - typed columns, node formatters, and the bare-Column reference - #126
Merged
Conversation
…lution Rebuild the column-type side around the agreed protocol split: DataTypeSupport reports data types, ColumnTypeSupport reports column classes. - ColumnTypeSupport carries only the two tables: suggested_column_types() (the common Python types, each answered with a column class, or None as a last resort) and suggested_extra_column_types() (the backend's own Python types). No format_column, no resolution method on the dialect. - Resolution is the model layer's (base/fields.resolve_column_class): explicit UseColumnType, then the common-type table, then the extras table, then ColumnTypeResolutionError. Normalisation walks the answering table's own key order; there is no shared core key list. - Deleted: both column_suggestion modules, the expression-layer column_suggestions module, the UNSUPPORTED sentinel, supports_column_operation, value_type, WholeNumberResultMixin, the numeric sub-families (Float/Decimal/Real/Double columns), and the inherited neutral table. - SQLite and dummy state their own tables; the dummy keeps the portable baseline explicitly. - The tests for the new shape are still being rewritten (the three former test modules do not import yet).
…Accessor - DDLSourceMixin presents the three dialect-free field facts: field_python_type(); column_type(), the column-type declaration as written (UseColumnType | class | sequence | None); and column_data_type(), renamed from the old column_type (UseSqlType | DataType | sequence | None). - Column-class selection is the field accessor's own business: FieldAccessor is a public top-level class and resolves the declaration -- or, absent one, the annotation against the backend's tables -- into a single class. - base/fields.py is annotation-only again; ColumnTypeResolutionError moved to base/field_proxy.py; columns_type() -> columns_data_type(). - Tests rewritten for the new API (test_column_type_support.py, test_sqlite_column_type.py, column_helpers.py, test_typed_columns.py); the two outdated *_suggestion* modules are deleted. 225 targeted tests pass. - docs: column_type -> column_data_type renames across en/zh.
…rfaces - DummyDialect and SQLiteDialect name the ColumnTypeSupport protocol in their bases, alongside DataTypeSupport, so the MRO-based conformance tests classify it as implemented; - SQLITE_PROTOCOLS lists it, so the positive/negative partition test sees it as classified; - the sqlite2 column-type test imports the shared query helpers through their package path (rhosocial.activerecord_test...) instead of a bare sibling import whose resolution depended on test collection order; the eight same-directory importers use the same package path so the helper has a single import name.
…. 3.13t) pydantic-core stopped publishing cp313t wheels after 2.41.4, and its newer sdists cannot build for free-threaded 3.13: PyO3 0.29 refuses the free-threaded build of CPython below 3.14. The python_version marker distinguishes release lines, not GIL configuration, so the pin covers both 3.13 builds -- the same shape as the existing 3.8 pins. 2.12.2 is the last pydantic whose core (2.41.4) has a cp313t wheel; verified by resolving against a real 3.13t interpreter (pydantic 2.12.2 / pydantic-core 2.41.4).
…lue surface length/ascii/strpos/position/octet_length/bit_length were implemented twice: once on StringValueMixin (string receiver) and once on IntegerValueMixin, which was attached to IntegerValueExpression as well. The integer-side copy is the bug: LENGTH(name).length() constructed, and only the database rejected it; the string copy had position()'s arguments swapped relative to the factory signature. The operations now live once, in StringToIntegerMixin, attached to the string classes only (StringColumn, StringValueExpression). Integer results keep the numeric surface: IntegerValueExpression swaps the string operations for NumericValueMixin + TranscendentalMixin (the same groups IntegerColumn carries), and NumericValueExpression gains TranscendentalMixin, so the two sides of every family match.
…ithmeticMixin + - * on a temporal value were either unconditionally delegated to BinaryArithmeticExpression (the column side, so created + created rendered "created" + "created") or refused outright (the value side, so interval + interval was an AttributeError). Neither matched temporal arithmetic: - a temporal value +/- an interval is that kind of temporal value; - a temporal value - a temporal value is an interval; - an interval +/- an interval is an interval; - an interval x a number is an interval. TemporalArithmeticMixin performs exactly those and refuses the rest at construction in the package's refusal vocabulary (it inherits NotANumberMixin for the reflected operators and / %), and replaces ArithmeticMixin on DateTimeColumn and NotANumberMixin on TemporalValueExpression. The operand check recognises both temporal columns and temporal values (DateTimeMixin), so the column and the value follow one gate.
The metadata-based accessors answered None/[] for a field the model does not have, while the model_fields-based ones (field_python_type, column_type) raised KeyError. A misspelled field name could therefore be ignored in silence on the accessors that answer None -- the failure the explicit answers exist to prevent. Every accessor now reads its metadata through one guarded lookup (`_field_metadata`): a name the model lacks raises KeyError, and None (or an empty collection) always means "the field exists and declares nothing". The batch accessors inherit the rule per named field. column_name() stays a name mapping and is unchanged. Docs (en/zh) record the rule.
& | ~ on a truth value returned LogicalPredicate, exactly like combining two predicates does, so (flag & other) could not be aliased into a SELECT list or compared as a value -- the plan's P1 item 3. The two receivers now share one implementation (the private _LogicalOperators) and differ only in the result class: - predicate receivers keep LogicalPredicate (WHERE-building stays as it was); - value receivers build BooleanLogicExpression, which renders the same SQL through the same dialect entry point (format_logical_predicate) but is a boolean value: aliasing, comparison, casting and further connectives all stay available. BooleanValueExpression switches from LogicalMixin to BooleanLogicMixin for the same reason. New batteries pin the two result classes, negation, chaining, aliasing and comparison (test_boolean_connectives.py), and the family symmetry contract -- same-family column and value classes offer the same operation set -- passes for all ten families (test_operation_symmetry.py).
…rameter name The dummy2 expression contract resolves every __init__ parameter by its own name -- VAR_POSITIONAL included -- and the dict round-trip rebuilds the node from those attributes. BooleanLogicExpression stored its operands only as `predicates` while the parameter was `operands`, so get_params() could not resolve the attribute and the round-trip changed the SQL. The parameter is now `*predicates`, exactly the shape LogicalPredicate uses: the renderer's attribute name and the contract's parameter name are the same thing.
…rvey measured - refuse at construction the string edges the ten backends answer four ways: left/right negative n, lpad/rpad negative length and empty pad, substr start<1 and negative length, and multi-character trim sets - repeat(-n) is the empty string, the answer PostgreSQL, MySQL, MariaDB and ClickHouse give natively; lpad/rpad always spell out the omitted pad - add DistinctFromPredicate (IS [NOT] DISTINCT FROM) on ComparisonMixin with the standard rendering as the default and a SQLite 3.39.0 gate that refuses rather than substitutes - sign returns a numeric value rather than an integer one - log/truncate/position/strpos/temporal contracts carried in docstrings, the baseline in .claude/rules/operation-contracts.md
test_add_check_constraint_enforces_non_null referenced a bare 'dialect' that is not defined anywhere in the module. The test is gated on supports_add_constraint(), which only returns True on SQLite >= 3.53, so the broken path never ran before: on SQLite 3.50 the test skipped, and the NameError went unnoticed. Use backend_with_users.dialect, matching every other call site in the file. Also rename test_sqlite_353_integration.py to test_sqlite_3_53_integration.py so the 3.53 naming is consistent with test_sqlite_3_53_features.py.
A factory that needs a dialect's help builds a node, never a SQL string --
raw SQL is text by the time any hook could reach it, so the standard trim
form being a syntax error on SQLite was unfixable while the factory
assembled the string itself. TrimExpression + TrimMixin make it a formatter's
decision: the default renders the standard form unchanged, and SQLite
overrides format_trim_expression with its function forms (TRIM/LTRIM/RTRIM).
Alongside:
- count('*') goes through WildcardExpression rather than a raw '*' string
- dummy refuses unsigned REAL -- the invariant caught the silent drop once
dummy was enumerated; SQLite's formatter already did
The paradigm-invariants test listed the nine sibling backends and imported whichever were installed. Every project installs every sibling locally, so on a developer machine that pulled nine foreign distributions into this project's test process -- filling the process-wide expression registry with classes this suite has no allowlist for, which is how test_all_builtin_expressions failed here and passed in CI with none of them installed. A project tests its own code: dummy and sqlite, both in this repository. The CITEXT self-render probe moves to the postgres repository (recorded in the plan, not ported -- its branch does not yet import against this one).
The survey's SQLite dispositions become real renderings, each a node with one formatter and no raw SQL anywhere: - renames: ASCII->UNICODE, STRPOS->INSTR, and POSITION reordered to INSTR's haystack-first order - LEFT/RIGHT become SUBSTR with computed bounds; RIGHT clamps its start with MAX(..., 1) because a negative start reads from the end and would hand back a tail instead of the whole string when n exceeds the length - repeat, lpad and rpad are nodes now (RepeatExpression/LpadExpression/ RpadExpression), emulated out of ordinary nodes: REPLACE(HEX(ZEROBLOB(n)), '00', x), a two-branch CASE, and a single SUBSTR respectively - truncate: SQLite's TRUNC takes one argument, so a precision is carried by scaling, truncating toward zero and scaling back -- exact for negatives, which rounding would not be - log swaps its two arguments, and log() no longer renders LOG(x, NULL) Every emulation is verified twice: the SQL is asserted, and then executed against a real SQLite with the answer PostgreSQL, MySQL and ClickHouse give for the same call. Two survey rows were wrong and are corrected -- the RIGHT form and SQLite's TRUNC arity.
…n rename
The bare `Column` becomes what it should have been: a reference that renders
an identifier and keeps only the shared surface (aliasing, null tests,
casting). It carries no capability mixins, so `Column(...) == x`, `.like()`,
`.in_()`, arithmetic no longer exist on it. That was previously a silent
trap - the fallback `object.__eq__` returned Python `False` rather than
raising, so `find_one`'s scalar-PK path, the optimistic-lock check and
`restore()` all built a broken predicate without complaining. Those three
now build an explicit `ComparisonPredicate`.
`ColumnBase` is an ABC with abstract `format_method`; each concrete column
declares its own one-line `format_method`. `DateTimeColumn` is renamed
`TimestampColumn`: the column layer models values, and a timestamp is a
value. `DateTimeMixin` (both the expression-layer and the per-dialect one)
keeps its name - `TimestampMixin` is already taken by the unrelated
field-maintenance class in `field/timestamp.py`.
The query layer builds its predicates explicitly (`_in_predicate` in
`query/range.py` mirrors `ComparisonMixin.in_` branch for branch), so a bare
column reference needs no mixin at all.
Tests: exercising a mixin capability now constructs a typed column;
exercising statement machinery builds explicit nodes. The roundtrip test's
params-dict comparison moves to `to_sql()` equality because a bare column has
no `__eq__` any more.
Verified: dummy2 (2728 passed), sqlite2 + operation boundaries, query
top-level + range_queries, basic/{fields,ddl,crud}, mixins,
relation/derived. Ruff: zero new findings against baseline, ten
pre-existing E501s resolved.
`_create_query_method` built the foreign-key condition as `query.where(Column(...) == value)`. With the bare `Column` no longer carrying `ComparisonMixin`, that falls through to `object.__eq__` and hands `where()` a Python `False` - which it rejects with `TypeError: Condition must be str or SQLPredicate, got <class 'bool'>`. Ten tests in `feature/relation/test_descriptor_db_coverage.py` caught it, in both the sync and async descriptors. Both now build a `ComparisonPredicate`, matching what the same file already does for its composite-key paths. This is the follow-on of the P2 trap: the failure is loud here only because `where()` type-checks. The same shape in `find_one`, the optimistic lock and `restore()` failed silently instead. Verified: `test_descriptor_db_coverage.py` 40 passed (was 10 failed). CI 38042800801 ran the full suite green apart from these 10 - 11041 passed per job, no collection errors - so this is the whole of it.
… time `supports_functions()` has always been able to *answer* whether a name is available on this server version -- SQLite records `json_array_insert` and `jsonb_array_insert` as 3.53.0+, and the check was implemented in `_is_sqlite_function_supported` -- but nothing consulted it while rendering. Measured before this change: on a 3.32 dialect, `supports_functions()["json_array_insert"]` reports `False` **and** `FunctionCall(..., "json_array_insert", ...).to_sql()` emits `JSON_ARRAY_INSERT(...)`, which the server rejects. The gate was advisory. `FunctionCallMixin.format_function_call` now calls the new `SQLDialectBase.check_function_version` first, so a backend that records a floor gets a refusal instead of unreachable SQL. The message names the function, the floor and the actual version, so the caller can act on it. The hook is deliberately a *data* table, not a method override: the base precedes every mixin in a dialect's MRO, so a mixin method overriding it here would be shadowed by the base and silently dead. SQLite derives its `function_version_floors` from the min column of the existing `_SQLITE_FUNCTION_VERSIONS`, so the query surface and the gate can no longer disagree. Ceilings stay with the query surface only -- a ceiling says the spelling does not exist on this backend at all, which is a different question from being too old. `supports_functions()` is unchanged in shape and values. Twelve tests were pinning the old behaviour: they asserted that `pow`/`power`/`sqrt`/`mod`/`trunc` render on a `sqlite_dialect_3_8_0` fixture when those functions arrived in 3.35.0, so each one was asserting SQL a 3.8 server rejects. They now use the version the functions actually arrived at, and a refusal test pins the floor so the fixture cannot drift back. Verified: feature/backend/sqlite2 978 passed, feature/backend/sqlite 736 passed unchanged, feature/backend/dummy and feature/backend/dialect identical, and the SQLite LPAD/RPAD/REPEAT emulations render byte-identically.
The 57 probe definitions (dummy 29 + sqlite 28) were unconditional `return True` in exact 1:1 correspondence with their formatters, and their only consumer was `DataTypeMixin.supports_data_types()`. The derivation now treats the formatter itself as the support declaration, and a probe — where a dialect still owns one — as the sole opt-out, so backends that gate a feature by version (firebird FB3, mariadb, mysql, clickhouse) keep their behaviour byte-for-byte while the core dialects carry no second, easily-stale answer to the same question. Kept deliberately (all live): supports_data_types() and its lookup chain (_type_class_for / _annotated_data_type / _load_backend_type_module / _BACKEND_TYPE_MODULE), the format_data_type dispatcher, substitute_advice / _suggested_type_advice, and DateTimeSupport / supports_microsecond_timestamp (called by the testsuite field tests and a clickhouse test, overridden by firebird, enumerated by seven conformance harnesses). Tests: the 1:1 correspondence harness becomes the family-retirement guard (formatter is the declaration, no core probe may remain); direct probe callers now assert through their formatters.
The recorded floor for `json_extract` / `json_extract_text` said 3.38.0, and the render-time gate added in the previous commit started enforcing it. CI then failed `test_json_extract_execution` on a **3.35.0 server that executes the function fine** -- which is the disproof: the JSON1 functions have existed since SQLite 3.9.0 behind a compile-time option, and 3.38.0 is only when they were promoted into core. Availability on an old build depends on how SQLite was compiled, not on the version number, so a hard floor refuses SQL the server accepts. The rest of the JSON1 family in the same table (`json_type`, `json_valid`, `json_quote`, `json_remove`, `json_set`, ...) already carried no floor, which is also why the table looked inconsistent -- these two entries were the mistake. They now carry none either, with a comment recording the evidence. The gate is unchanged: a correctly recorded floor still refuses at render time. `json_array_insert` / `jsonb_array_insert` (3.53.0), `unhex` (3.45.0), `pow` (3.35.0) and `iif` (3.32.0) all still do, and the boundary test's parameter list keeps pinning them -- with a note explaining why `json_extract` must stay off it.
The protocol declares one switch, `supports_microsecond_timestamp`, and its own docstring already says the switches are "about precision rather than about whether date and time functions exist at all". So the name claimed a capability it never declared: date/time *operations* live elsewhere, in `DateTimeMixin` and `TemporalArithmeticMixin` on the column side and in each dialect's formatters. The name was also doing double duty. Oracle's P7-generated `OracleDateTimeSupport` is a protocol of the same shape, but *it* really does declare date-time operations -- `format_date_trunc_expression`, `format_interval_expression`, `format_datetime_add_expression` and the rest. Two different meanings under one name; renaming this one to `TimestampSupport` names the precision question it asks and leaves the operations family its own. The module moves with the class, since it held nothing else. Nothing about the method changes -- `supports_microsecond_timestamp` keeps its name, so the testsuite contract that calls it and every dialect that overrides it are unaffected. Only the protocol-class name and the conformance harness lists that enumerate it move: 16 sites across core and nine backends. `OracleDateTimeSupport` is deliberately left alone: it means the other thing.
setup-python now ships Python 3.15.0, so the release-candidate pins are replaced with plain 3.15 (both the GIL and the free-threaded legs), and the exemptions that made the leg optional are removed: - the "(experimental)" tag in the job names, and the experimental job ids; - `continue-on-error: true`, which let the leg report failure without failing the run; - postgres' disable gate (`if:` set to false) and its comment. The remaining matrix entries are unchanged: every backend keeps a GIL and a free-threaded leg on 3.15, now on equal footing with the other interpreters.
… Python The benchmark job was pinned to 3.14 while the test matrix had already moved to 3.15, so the newest interpreter this package supports was never exercised under load. It now runs on 3.15 (core, postgres and mysql) -- job id, job name and the summary text included -- while the per-version test legs stay on their own interpreters. The README carried the Pydantic requirement as one coarse line. It now carries the actual pairing: 3.8 pins an exact pair (the last installable there), 3.13 and 3.13t pin the pair whose pydantic-core still has a wheel, and the rest stay open. The reason column records why the 3.13 row cannot simply track the newest release, and the free-threaded paragraphs now state per backend which legs exist and why.
The test-python-315 matrix renders its free-threaded leg as "Run tests with Python 3.15 free-threaded". Prepend the `t` so the check reads "Run tests with Python 3.15t free-threaded", matching the 3.13t / 3.14t naming used by test-free-threaded.
Python 3.15.0 was released (2026-10-09), so it leaves the "in CI/classifiers" caveat and joins the Supported list (plus 3.15t among the free-threaded builds). Python 3.10 reached end-of-life on 2026-10-01, so the matrix records it as past-EOL, and the 1.1.x floor plan moves from ">=3.10" to 3.11+ since an EOL version cannot stay the floor.
|
18 of 22 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Rebuilds the column layer so that which operations a column offers is decided by its
class rather than by a universal base, and so that every operation is rendered through
exactly one hook instead of being assembled from a function name.
Three things change together and cannot be separated:
ColumnBaseis an ABC with an abstractformat_method; each concrete column class composes the capability mixins its valuefamily genuinely has (
StringColumncarries the string surface,NumericColumnthearithmetic one,
TimestampColumnthe temporal one, and so on). The old design letevery operation be called on every column, which was not a feature but a silent
trap:
Column(...) == valuefell through toobject.__eq__and returned a Pythonbool, sofind_one, the optimistic-lock check andrestore()all built a brokenpredicate without raising anything.
Columnis a reference, not an operator surface. It renders an identifierand keeps the shared surface (aliasing, null tests, casting). Operations are reached
through a typed column, or built explicitly as nodes (
ComparisonPredicate,InPredicate,BinaryArithmeticExpression, …), which is what the query layer now does.ColumnTypeSupportexposes
suggested_column_types()/suggested_extra_column_types(); the model layernormalises a field's annotation against those tables and fails, naming the field, when
neither answers.
DateTimeColumnis renamedTimestampColumn: the column layer modelsvalues, and
datetimeis a vendor spelling, not a different value.The temporal family is one class, by decision (方案甲): date/time and timestamp share the
same operations, and the three storage semantics stay in the DataType layer
(
DateTimeType/TimestampType/TimestampTzType), selected throughUseSqlType.Also in this branch, as follow-on work that the rebuild made necessary:
trim/lpad/rpad/repeatare nodes (TrimExpression,LpadExpression,RpadExpression,RepeatExpression), each with oneformat_*hook, rather than functionnames routed through
format_function_call. SQLite spells the four it lacks by emulation.(
SQLDialectBase.check_function_version), so a floor below the connected server's versionproduces a refusal naming the function, the floor and the version instead of SQL the
server rejects.
supports_data_type_*probe family is retired;supports_data_types()nowderives from formatter presence, with a probe as the only opt-out.
format_*'sexprparameters are annotated with their concrete expression types.DateTimeSupportis renamedTimestampSupport(it declares timestamp precision, notdate/time operations — a separate auto-generated protocol,
OracleDateTimeSupport, meansthe latter and keeps that name).
Type of change
feat: A new featuredocsperftest: Adding missing tests and correcting existing tests for the new featurechoreci: Changes to CI configuration (Python 3.15 legs, benchmark version)build: Pydantic pinning for Python 3.13 / 3.13trevertBreaking Change
Impact on End-Users — yes. Six changes are visible to end-user code:
DateTimeColumnrenamedTimestampColumnColumnlost its operatorsStringColumn,NumericColumn, …) or build the node explicitly (ComparisonPredicate,InPredicate,BinaryArithmeticExpression)DDLSource.column_type()is now a declaration accessorcolumn_data_type()/columns_data_type();columns_typeis gonetrim/lpad/rpad/repeatreturn nodes, not function callsUnsupportedFeatureError, or target a server that has the functionsupports_data_type_*probes are gonesupports_data_types(), which now answers from formatter presenceThe silent
boolfrom a bare-column comparison is the reason for the second row: it isbetter to fail loudly than to emit a query that cannot work.
Impact on Backend Developers (Custom Implementations) — yes.
(
TrimMixin/LpadMixin/RepeatMixin/RpadMixinfor the new nodes) or write aformat_*override where its spelling differs.ColumnTypeSupportis a new protocol a backend must implement.format_*signature changes:expris now typed rather thanAnyor absent.ColumnTypeSupportand, after the rename,TimestampSupport.Related Repositories/Packages
This change cannot be used alone; the coordinated branch in each repo is
feature/typed-columns(testsuite:test/typed-column-contracts).python-activerecord-testsuitetest/typed-column-contractsrelease/v1.0.0.dev17python-activerecord-{postgres,mysql,mariadb,oracle,sqlserver,firebird,clickhouse,snowflake,bigquery}feature/typed-columnsNote on the install references: every backend's release branch currently installs the core
from
@release/v1.0.0.dev30and the testsuite from@feature/expression-schema-object-refactor.Merging the backends therefore has to happen after core and the testsuite are on their
release branches, and their release-branch install lines must be re-pointed — otherwise the
release branch keeps depending on a feature branch that will be deleted.
Testing
Test Plan:
Verified locally (targeted runs, within the project's 180-second-per-run rule):
dummy22728 passed;sqlite2+ boundaries green;querytop-level +range_queries,basic/{fields,ddl,crud},mixins,relation/derived,dummy,expression,dialectgreen. Full-suite verification is left to CI — the projectforbids local full-suite runs.
Checklist
README.md).is created after the PR exists, named with the real PR number (Steps 14–15).
(all rendering stays parameterised; the change removes a path that silently produced
a bool rather than a parameterised predicate).
dict lookup on the render path).
Additional Notes
Deliberately out of scope for this branch (recorded so reviewers do not expect it):
backends that still carry their own
supports_data_type_*probes (~305 definitions:mysql ~90, mariadb ~115, firebird ~35, clickhouse ~65) are part of it.
format_*annotations org-wide. 44 are done here (core); the remaining 647 inthe backends are follow-up work.
dict/list/set/frozenset/tuplemap to one columnclass per backend (JSON or Array), so the JSON-vs-Array capability split is per-backend
rather than per-type. This is a design trade-off, not an oversight; converging it needs a
decision on whether a column class is chosen by Python type or by backend capability.
Also note: the per-backend branches are each ahead of their own latest release branch
(verified: the release tip is an ancestor, 0 behind), so no rebase is needed on merge.