Repository navigation
feat(mariadb): adapt to derived DDL and rework version boundaries - #5
Merged
Merged
Conversation
…y formatting - Drop the private supports_inline_index/format_inline_index declarations from the MariaDB table protocol; the capability now comes from the core TableSupport protocol (default False, MariaDB overrides True) - Converge inline index rendering onto the core format_index_definition override (returns (sql, params)); remove the parallel method - Point CI at the core feature branch that carries the protocol
Add MariaDB-owned partition DDL expressions (strategies, boundary values, definitions derived from the generic base declarations) and replace the partition formatting stubs in MariaDBPartitionMixin with real implementations for PARTITION BY strategies, inline definitions, definition options, subpartitions, and boundary values.
Introduce the MariaDB-specific column definition (and options) derived from the generic ColumnDefinition / ColumnOptions, carrying typed CHARACTER SET / COLUMN_FORMAT / STORAGE / INVISIBLE attributes with no dialect_options bag. The MariaDB format_column_definition (mixin and dialect) accepts both the generic and MariaDB column definitions and renders the MariaDB-only attributes. Implemented independently of MySQL; export the new classes and cover them with tests.
…ions Introduce MariaDBCreateTableOptions (deriving the generic CreateTableOptions) carrying the MariaDB-only ENGINE / CHARSET / table COLLATE options with no dialect_options bag. The MariaDB CREATE TABLE formatter reads these typed fields and the generic comment field, dropping the dialect_options comment read. Implemented independently of MySQL.
The MariaDB formatter now reads typed table_options.comment instead of the dialect_options bag.
…forced Move ALTER TABLE IF EXISTS/NOWAIT/WAIT onto MariaDBAlterTableExpression and read typed constraint 'enforced' instead of the dialect_options bag.
…ptions Add MariaDBInsertExpression (ignore/replace) and MariaDBTruncateExpression; read typed MariaDBCreateTableOptions only in the CREATE TABLE formatter.
…ect_options) MariaDB was the only backend whose DDL formatters read expr.dialect_options. Promote those keys to typed fields on MariaDBRenameTableExpression (if_exists/wait/nowait) and MariaDBTableMaintenanceExpression (no_write_to_binlog/local/persistent/check_mode/checksum_mode/repair_mode), and update the formatters and tests.
…ename-index expressions
Core DataType no longer carries a dialect_options bag; backend type subclasses stop forwarding it. Update the type-protocol tests.
…l + validation Add MariaDBCharsetCollationSupport protocol and MariaDBCharsetCollationMixin implementing a version-filtered whitelist (MariaDBCharset, MariaDBStorageEngine, MariaDBCollation) plus validate_* helpers. MariaDBCreateTableOptions validates engine/charset/collate at construction. Fold expression-level COLLATE into the mixin; remove the standalone collation.py / mixins/collation.py modules.
…options Add the three typed table options to MariaDBCreateTableOptions with validation and capability gating for system versioning; remove the duplicate format_create_table_statement in the mixin (the dialect's own method is authoritative) and the dead dict-based format_system_versioning_clause.
The protocol-conformance test requires MariaDBTableMixin to implement MariaDBTableSupport.format_create_table_statement. Restore the mixin implementation (with the new typed options) and drop the duplicate on the dialect class.
Remove identity/identity_start/identity_increment/identity_clause from XxxColumnDefinition and XxxColumnOptions to match core's column-attribute migration; XxxColumnDefinition now forwards attributes.
Render the dialect-free column attributes channel in each backend's format_column_definition (it was silently dropped), and give backends whose identity syntax differs from the SQL-standard form a format_identity_clause override (MySQL/MariaDB AUTO_INCREMENT, SQL Server IDENTITY(seed, inc)). Oracle gates column COLLATE to 12.2+; MariaDB drops the duplicate dialect formatter so the attribute-aware mixin is authoritative.
Core added the ColumnAttributeSupport protocol and renamed the deriver plan entry to creation_plan(); update protocol conformance and derived-DDL tests.
MariaDB renders the table comment as an inline table option and the column comment inside the column definition; declare supports_table_comment() and supports_column_comment() as True so the capability bits match the actual rendering behavior.
Inline table/column comments are now TableCommentClause / ColumnCommentClause nodes; render them through format_table_comment_clause / format_column_comment_clause. Tests updated to build the clause nodes.
Verify MariaDB DDLSource declarations, backend types, column options, and dialect rendering.
dialect.py rebound MARIADB_VERSION_BOUNDARIES to a local 19-key copy that omitted CHECK_CONSTRAINT, shadowing the 20-key dict imported from .mixins. Every other consumer imports the mixins copy, so the two had silently diverged. The visible symptom was supports_check_constraint() raising KeyError: 'CHECK_CONSTRAINT' on every call. It is reached from the live DDL path in mixins/ddl_table.py when formatting a table CHECK constraint, and the same incomplete dict was what __init__ re-exported as the public MARIADB_VERSION_BOUNDARIES. Drop the duplicate so dialect.py, the public export and every mixin resolve to the single dict in mixins/backend.py.
12.3 (LTS) and 13.0 are both GA, so they move into the gating matrix alongside the existing 12.x rows. 13.1 is still a release candidate, so it goes to the non-gating experimental job, matching how 13.0-rc was handled before it reached GA. Two defects made the previous 12.3 / 13.0-rc rows worthless: - The experimental job exported PYTHONPATH=src, missing tests, so the shared testsuite could not import providers.registry and collection failed outright. continue-on-error swallowed it, so the job reported green while testing nothing. Use src:tests, matching the main job. - Both rows were pinned to Python 3.15.0-rc.2, so MariaDB version coverage was blocked on an unrelated interpreter release. The new 13.1-rc row runs on released 3.14 instead, and the stale 13.0-rc image tag is replaced with 13.0 now that it is GA.
MDEV-30041 stopped old_mode from setting UTF8_IS_UTF8MB3 by default in
MariaDB 13.1, so "utf8" resolves to utf8mb3 on every release up to 13.0
and to utf8mb4 from 13.1. The same DDL therefore produced a different
column charset depending on the server:
CREATE TABLE t (s VARCHAR(50)) CHARACTER SET utf8
<= 13.0 -> utf8mb3_uca1400_ai_ci (4-byte chars rejected, 1366)
>= 13.1 -> utf8mb4_uca1400_ai_ci (4-byte chars accepted)
utf8 was absent from _CHARSET_MIN_VERSIONS, so the version guard was
skipped and validate_charset_name accepted it on every version while
supported_charsets advertised it. Resolve it instead, to utf8mb3 -- the
meaning it always had before 13.1. Resolving forward to utf8mb4 would
silently widen the column on every pre-13.1 server, trading one drift
for another.
Below MariaDB 10.6 the explicit utf8mb3 name is not documented, so the
alias passes through unchanged there; it is unambiguous at that point.
Verified across all 15 CI versions: the emitted character set is now
utf8mb3 (or the equivalent utf8 spelling) everywhere, and 4-byte
characters are rejected consistently rather than only on <= 13.0.
Also fill _COLLATION_MIN_VERSIONS, which was empty and made the version
branch in supported_collations()/validate_collation_by_name()
unreachable, and correct the utf8mb4 threshold note: 5.5.3 is a real
MariaDB release, so that gate is meaningful for a 10.2+ backend.
The prior design note in .claude/plan/2026-09-13 asserted "MariaDB uses
utf8 (not utf8mb3)"; that assumption is exactly what 13.1 invalidates,
so it is corrected there too.
…source Add the boundaries the 12.3 / 13.0 / 13.1 work needs, grouped by release, and record the invariant that broke in cb4d1e5: a module that rebinds this name shadows the imported table and diverges silently, turning a missing key into a KeyError at call time rather than at import time.
The list was a static frozenset taken from the MariaDB 11.0 docs, with no version coupling. MariaDB adds reserved words in otherwise-minor releases, so it went stale: SELECT to_date FROM ... -- parses on 12.2, errno 1064 on 12.3+ SELECT conversion FROM ... -- parses on 12.2, errno 1064 on 12.3+ `to_date` in particular is a very common column name. Both are now added at 12.3, and `deny` at 13.1 for the DENY privilege statement. MariaDBDialect also overrides the version setter so the word set follows a late assignment: backend.introspect_and_adapt() re-assigns version after construction, which would otherwise leave quoting decisions made for the old version. This narrows the blast radius rather than being the root fix -- it only affects the unquoted path, and every identifier field defaults to need_quote=True, so default DDL was never exposed.
CREATE TABLE t (y YEAR(2)) succeeds on 12.2 and 12.3 and fails with errno
1064 on 13.0 and 13.1. YEAR(2) was deprecated in 2012; the only way back
is old_mode=2_DIGIT_YEAR, which is itself deprecated.
MariaDBYearType accepted any integer for display_width despite its
docstring saying YEAR[(4)], and parse_type() round-tripped a
server-reported YEAR(2) straight back into an emitted YEAR(2) -- so a
schema read from a pre-13.0 server produced DDL that 13.0 rejects.
Constrain the width to {4, None} at construction, and have parse_type()
degrade a reported YEAR(2) to bare YEAR with a DeprecationWarning rather
than failing introspection or silently rewriting the schema.
MDEV-5092 added the clause in 13.0. Verified against live servers: it is a syntax error on 12.2 and 12.3, and works on 13.0 and 13.1, where it also accepts expressions and OLD_VALUE() so pre- and post-update values come back in one round trip. Only single-table UPDATE is accepted. Both supports_returning_for_update implementations returned False unconditionally and never consulted the version, so the capability was affirmatively denied even where the server supports it. Gate it on a RETURNING_UPDATE boundary, and add the un-suffixed supports_returning_update alias -- the core format_update_statement consults that name, so without it the clause is rejected regardless of what the _for_ variant reports. SQLAlchemy, Prisma and Drizzle all document INSERT/DELETE RETURNING for MariaDB but not UPDATE.
… 10.6 supports_json_table() returned False unconditionally and format_json_table_expression() always raised, on the stated grounds that MariaDB lacks JSON_TABLE. That is wrong: verified working on 12.2, 12.3, 13.0 and 13.1. Meanwhile the backend's own MariaDBJSONTableExpression docstring said "Supported in MariaDB 10.6+", examples/query shipped a JSON_TABLE walkthrough, and a test asserted the feature was absent -- so the contradiction was locked in from three directions at once. Gate on a 10.6 boundary and render the statement. The MariaDB node carries FOR ORDINALITY, EXISTS, ON EMPTY / ON ERROR and NESTED PATH, none of which the core formatter handles, so it is rendered here rather than delegated. Both node shapes are accepted: the MariaDB node's `json_doc` is a JSON document literal while the core node's `json_column` names a column. Column types are allow-listed rather than escaped, since a type is not data. Everything else is identifier-quoted or quote-escaped. A live server caught that MariaDB's grammar has no alias on NESTED PATH, so one is dropped. The security tests in test_mariadb_dialect_security.py asserted UnsupportedFeatureError, which meant the escaping was never actually executed. They now assert the escaping.
…tion _MARIADB_FUNCTION_VERSIONS had an upper-bound branch, but all 32 entries carried max_version=None, so nothing could ever be marked as removed in a later release. Give DES_ENCRYPT / DES_DECRYPT the 13.0 upper bound they were removed in, and add supports_function_name() so the table is reachable for an arbitrary name rather than only for the functions happening to be constructible here. The backend does not emit DES_* today, so this changes no current behaviour; it exists so a future caller cannot generate SQL that 13.0+ rejects without noticing.
…a list MariaDBDatetimeAdapter stored a server version and never read it; ShowFunctionality computed an invisible-column flag and never read it either (the SHOW CREATE TABLE parser matches INVISIBLE unconditionally, which is correct because a server too old to emit the keyword never produces it). Collapse an if/else whose two arms were identical. The system-schema exclusion list was an inline 4-tuple repeated at four call sites, so a new MariaDB release adding one would need four edits and a missed site would silently leak the schema into user-facing listings. Hoist it to SYSTEM_SCHEMAS / SYSTEM_SCHEMAS_SQL_PREDICATE. It stays a fixed list rather than a discovery query: information_schema .SCHEMATA carries no flag separating a system schema from a user schema, so the only reliable discriminator is a round trip against the server's bootstrap state.
…licy The table topped out at 10.6 and recommended 10.6+, while CI has covered 11.x and 12.x for some time. It also contradicted the code in six rows: SKIP LOCKED was documented at 10.6 but gated at 10.3, JSON at 10.2 but gated at 10.2.3, and GROUPING SETS, INET6 and DEFAULT expressions had no implementation at all. Correct the thresholds, drop the unimplemented rows, and list what is deliberately not supported so callers are not left guessing. Add a support policy (12.3 LTS recommended, 13.0 opt-in, 13.1 not for production) and describe what happens on an unrecognised newer version, which was previously undefined. Feature gates resolve to "enabled" when the server is newer than an introduction boundary, so new syntax is assumed available; removals are declared per feature; and behaviour changes that are not new syntax are handled by resolving the ambiguity rather than branching on version, as the utf8 alias now does. Document the utf8 alias change, the JSON arrow situation, and the version-aware reserved words, since all three are user-visible.
The suite ran 3506 passed / 93 skipped identically on 10.2 through 13.1, which means no test changes behaviour with the server version -- so a green run was not evidence that those versions were safe, it was evidence that nothing about them was exercised. supports_returning_for_update returning False on 13.0 and 13.1 is exactly the shape of bug that hides behind an identical test count. Add boundary tests on both sides of each new gate, including one that asserts the boundary table is not shadowed by a second copy (the cb4d1e5 regression). The integration tests compare each gate against what a live server actually accepts, so a threshold that drifts from reality fails here rather than in production, and the charset tests assert that the resolved character set denotes utf8mb3 on every version.
CI caught this: gating utf8mb4_bin at 10.6 broke test_collate_executes_case_sensitive_match on 10.2 / 10.4 / 10.5. Both directions were wrong. Checked against live servers: 10.2, 10.5 -> utf8_general_ci, utf8_unicode_ci, utf8mb4_* 10.6, 12.3 -> utf8mb3_general_ci, utf8mb3_unicode_ci, utf8mb4_* MariaDB 10.6 renamed the 3-byte family from utf8_* to utf8mb3_* and dropped the old spellings, so those names are a removal at 10.6 rather than an addition, and the utf8mb4_* collations have existed since 5.5.3 rather than 10.6. Model both bounds per collation, add the utf8mb3_* enum members, and point the removed-name error at the replacement spelling.
Expression classes moved under backend.expression/ and several were renamed during the derived-DDL refactor, leaving relative-import depths and forward references in TYPE_CHECKING blocks pointing at modules and classes that no longer exist. Type checkers and any runtime evaluation of those blocks fail to resolve them. Verified statically: every TYPE_CHECKING import in the touched files now resolves to an existing module and an existing top-level symbol, and no string annotation references a name absent from the workspace.
The suite applies markers that were never declared in [tool.pytest.ini_options] markers. pytest only warns for an unknown marker, so the warning scrolls past unnoticed and a dropped or misspelt registration is silent. Registers the ones in use, with descriptions that say what they do: collection grouping, not ordering. `feature` labels feature-area tests and `benchmark_relation` marks relation benchmarks; neither affects scheduling, so neither substitutes for the xdist_group marker that actually serialises a suite. usefixtures needs no entry and is not added: pytest provides it, so it was a false positive in the isolation audit rather than a missing registration.
… revisions The installed hook exec'd scripts/preflight.sh unconditionally. That hook is a local, untracked artifact, so it survives a checkout of any commit that predates the script, and on such a commit exec failed with a bare "No such file or directory" — blocking the commit with nothing to act on. In this repository scripts/preflight.sh first appears at 5f542e5, so every earlier commit is affected. The failure is also silent in a dangerous way: it swallowed a `git commit --amend`, so a commit-message rewrite reported as applied when it had not been, and had to be redone. The hook now checks the script is executable and exits 0 otherwise, so a revision without the pre-flight simply skips it. Normal operation is unchanged. Also honours core.hooksPath. git ignores .git/hooks entirely once that is set, so the hook was being written to a location that would never run while this script still reported success. No repository sets it today, so that part is latent.
Both dependencies were pinned to feature/activerecord-derived-ddl, a branch that exists only to carry the derived-DDL work. Now that the core change has landed in release/v1.0.0.dev30 and the matching testsuite change in release/v1.0.0.dev17, pinning to the feature branch means CI keeps resolving a branch that will be deleted once the backend PRs merge. This is a prerequisite for merging: leaving the old ref in place means the first run after that branch disappears fails to resolve, not the code.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/v1.0.0.dev3 #5 +/- ##
=======================================================
+ Coverage 63.48% 66.50% +3.02%
=======================================================
Files 123 132 +9
Lines 7446 8160 +714
Branches 1104 1225 +121
=======================================================
+ Hits 4727 5427 +700
+ Misses 2412 2356 -56
- Partials 307 377 +70
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two things land together here: the migration of the MariaDB backend onto the derived-DDL
model from rhosocial/python-activerecord#121, and a rework of how the backend reasons about
MariaDB version differences.
The version work is the distinctive part. Version knowledge was previously scattered and
partly wrong, so behaviour depended on where a check happened to live.
MARIADB_VERSION_BOUNDARIESis now the single source, and a function can be gated by a removal rather than only by an
addition. That corrects several things that were wrong or missing:
JSON_TABLEis implemented; MariaDB has supported it since 10.6.UPDATE ... RETURNINGis supported on MariaDB 13.0+.YEAR(2)is rejected, since MariaDB 13.0 removed it.utf8charset alias resolves to an explicit spelling.The compatibility table was rewritten and now states an explicit support policy. CI covers
MariaDB 12.3, 13.0 and 13.1-rc in addition to the existing versions.
On the DDL side the backend gains typed options and capability reporting:
MariaDBColumnDefinitionwith typed column options and backend identity syntax.MariaDBCreateTableOptionswith typedAUTO_INCREMENT,ROW_FORMATandSYSTEM VERSIONING, read from the generic declaration path.MariaDBAlterTableExpressionwith a typedcheckenforcement option, typedRENAME TABLEand maintenance options, typed insert/truncate variants.table and column comment capabilities.
DDLSourcecollection, tableinheritance, tablespace and inline-index rendering.
dialect_optionsis removed from data-type value objects and from theadmin/routine/rename-index expressions, along with the generated DDL references. The
create-table formatter is kept on
MariaDBTableMixin,MariaDBDatabaseMixinis wired intothe dialect, and
format_index_definitionis declared in the table protocol whileinline-index capability is inherited from core.
Type of change
feat: A new featuredocs: Documentation only changes (if related to the new feature)test: Adding missing tests or correcting existing tests (for the new feature)chore: Changes to the build process or auxiliary tools and libraries (if related to the new feature)Breaking Change
No commit carries a
!marker. The public model and query APIs are unchanged.Impact on End-Users:
No backward-incompatible changes are expected for typical end-user applications. Two
behaviours do change, both toward rejecting input that MariaDB would have rejected anyway:
YEAR(2)is now refused rather than sent to the server, and a feature that a givenMariaDB version has removed is now gated by that removal rather than by an addition
threshold.
Impact on Backend Developers (Custom Implementations):
dialect_optionsbag is removed from data-type value objects and from theadmin, routine and rename-index expressions, and the generated DDL references are gone.
Anything reading or writing it must move to a declared option or capability bit.
nodes from core.
MARIADB_VERSION_BOUNDARIES. A check writtenad-hoc against a version number will drift from the corrected windows, which is the
specific class of defect this rework exists to remove.
supports_column_commentis declared, and inline table andcolumn comment capabilities are advertised. A conformance suite asserting the old values
needs updating.
were repaired so type checkers resolve them.
Related Repositories/Packages
The core half is rhosocial/python-activerecord#121 (merged into
release/v1.0.0.dev30) andthe shared contract half is rhosocial/python-activerecord-testsuite#21 (merged into
release/v1.0.0.dev17).This PR also repoints this repository's workflow at those release branches. Both dependencies
were pinned to
feature/activerecord-derived-ddl, a branch that only exists to carry thederived-DDL work; leaving the old ref in place would have made the first run after that
branch is deleted fail to resolve.
Merged or in flight, and still to merge in the same sequence:
91b89e0f7dbc39a5f9a9fff98cf00870faf03c6d8c0c573289e7e2824d2e9f0Testing
Local full-suite execution was skipped at the user's instruction. Verification relied on
remote CI, which ran on this exact tip
a5f9a9fand covers MariaDB 10.2 through 13.1-rc onPython 3.9 through 3.15 including free-threaded builds. All sixteen jobs pass.
That run also covers the workflow change in this PR: it is the first execution against
release/v1.0.0.dev30andrelease/v1.0.0.dev17rather than the feature branches, so thenew dependency refs are verified to resolve.
Checklist
changelog.d/(to be added after this PR isopened, named by the actual PR number)
Additional Notes
Four commits are not part of the DDL or version work and are called out so a reviewer is not
surprised:
ci: install core and testsuite from release branches— the workflow repoint describedabove, a prerequisite for merging.
ci(mariadb): cover MariaDB 12.3, 13.0 and 13.1-rc— the matrix expansion that theversion-boundary work needs in order to be verified.
fix(scripts): stop the pre-commit hook from blocking commits on older revisions— arepository-hygiene fix, kept identical across all ten repositories so they do not drift.
fix(backend): repair unresolvable TYPE_CHECKING importsandchore(test): register the pytest markers the suite already uses— mechanical repairs required for type checkingand lint to run at all after the namespace split.