Repository navigation
feat(postgres): complete DDL, partition lifecycle and typed options for derived DDL - #58
Merged
Merged
Conversation
Add a PostgreSQL-owned partition clause type derived from the generic PartitionClause for capability gating and future PG-only partition parameters; concrete partitions remain created via the separate CREATE TABLE ... PARTITION OF statement. Export it and cover it with tests.
Introduce the PostgreSQL-specific column definition (and options) derived from the generic ColumnDefinition / ColumnOptions, carrying typed COMPRESSION / STORAGE / STATISTICS attributes (no dialect_options bag). The PostgreSQL format_column_definition accepts both the generic and PostgreSQL column definitions and renders the PostgreSQL-only attributes; it no longer reads dialect_options["identity"] (the typed identity field is authoritative). Export the new classes and cover them with tests.
Introduce PostgresCreateTableOptions (deriving the generic CreateTableOptions) carrying the PostgreSQL-only UNLOGGED header modifier. The generic format_create_table_options no longer renders UNLOGGED; PostgreSQL overrides it to add UNLOGGED and accepts both the generic and PostgreSQL options. UNLOGGED/TEMPORARY mutual-exclusion is checked via isinstance.
UNLOGGED now lives on the typed PostgresCreateTableOptions; update the CreateTableOptions(unlogged=...) tests and the capability docstring.
…t_options Add PostgresIndexDefinition (opclasses/nulls_not_distinct/with_options) and PostgresCreateIndexExpression; PostgresDropIndexExpression uses the generic typed concurrent flag. The index mixin reads typed fields instead of the dialect_options bag.
Add PostgresAlterColumn (typed using) and PostgresExcludeConstraint (typed elements/using/where); read typed constraint validation; build vector indexes with PostgresCreateIndexExpression.
The materialized-view pass-through and the partition test provider no longer use the dialect_options bag.
…sions Remove the unused/reserved dialect_options parameter and attribute from PostgreSQL DDL expression classes (ALTER INDEX, REINDEX, VACUUM, partition, type/domain/collation/publication/routine/statistics/extension/foreign table/comment). No formatter reads them; typed fields carry real options.
Core DataType no longer carries a dialect_options bag; backend type subclasses stop forwarding it. Update the type-protocol tests.
Remove identity/identity_start/identity_increment/identity_clause from XxxColumnDefinition and XxxColumnOptions to match core's column-attribute migration; XxxColumnDefinition now forwards attributes.
Render the dialect-free column attributes channel in each backend's format_column_definition (it was silently dropped), and give backends whose identity syntax differs from the SQL-standard form a format_identity_clause override (MySQL/MariaDB AUTO_INCREMENT, SQL Server IDENTITY(seed, inc)). Oracle gates column COLLATE to 12.2+; MariaDB drops the duplicate dialect formatter so the attribute-aware mixin is authoritative.
Core added the ColumnAttributeSupport protocol and renamed the deriver plan entry to creation_plan(); update protocol conformance and derived-DDL tests.
PostgreSQL supports table inheritance via INHERITS (...); enable the core capability gate.
The capability is already declared as supports_table_inheritance on PostgresTableMixin/PostgresTableSupport.
…_comment_on format_column_definition no longer inlines COMMENT 'x' (syntax PostgreSQL itself rejects); a declared column comment raises UnsupportedFeatureError instead of producing broken DDL. PostgresCommentMixin declares supports_comment_on() and is composed before TableMixin so format_comment_statement is not shadowed by the generic one; it accepts any comment expression and quotes dotted targets segment-by-segment. The comment protocol declares supports_comment_on as well.
Adapt the inline column-comment guard to the ColumnCommentClause node and update the corresponding test.
Model base already defaults to "id"; fixtures should exercise the unspecified-PK default path.
Verify PostgreSQL DDLSource declarations, backend types, column options, and dialect rendering.
PostgresMaterializedViewMixin was previously ordered after the core ViewMixin, so its format_create_materialized_view_statement never took effect and storage_options was silently dropped. Landed the failing tests first and watched them fail (T-01/T-02/T-05, 7 red), then reordered the bases in dialect.py to turn them green. Fixed: - MRO order: PostgresMaterializedViewMixin moved ahead of ViewMixin, with PostgresMaterializedViewMixin(ViewMixin) declaring the dependency explicitly - PG syntax order: WITH (storage_parameter) must precede TABLESPACE - The CONCURRENTLY version gate moved down into format_refresh_materialized_view_statement, which also gates the core RefreshMaterializedViewExpression; the raised exception is now uniformly UnsupportedFeatureError instead of ValueError - Schema qualification now goes through format_qualified_identifier, fixing mixed-case/reserved identifiers broken by bare concatenation - Fixed the unescaped percent sign in LIKE 'pg_%' in the new introspection query - MV capability probes moved off PostgresViewMixin/PostgresViewSupport into a materialized-view-specific mixin/protocol pair, now returning by version (PG 9.3/9.4 thresholds) Added: - PostgresStorageParameter enum (the server's RELOPT_KIND_HEAP standard relation options, per the fillRelOptions mapping for RELKIND_MATVIEW) with value_type, min_version and enum_values; storage_options keys are validated strictly against the enum rather than by regex, and table access method / toast.* namespaced parameters require an explicit allow_unlisted_storage_parameters=True - PostgresCreate/Drop/AlterMaterializedViewExpression plus 5 ALTER actions (RENAME TO / SET SCHEMA / SET / RESET / OWNER TO); PG does not support ALTER MATERIALIZED VIEW ... SET TABLESPACE - list_materialized_views() / get_materialized_view_info() / materialized_view_exists(), with symmetric sync/async implementations, reporting is_populated and has_unique_index Tests: L1 SQL-generation unit tests, L2 doc/example consistency, L3 live-database integration (sync + async), plus a real run of the extended examples/ddl/materialized_view.py against PG 19.
CI (run 36170586225) exposed 4 cases that had encoded the old, incorrect behaviour: - REFRESH MATERIALIZED VIEW identifiers are now quoted per repository convention (the old PG formatter used bare f-string concatenation, the only case that did not quote) - Schema qualification is now quoted per segment: "analytics"."monthly_sales_summary" - The CONCURRENTLY 9.4 version gate now raises UnsupportedFeatureError instead of ValueError - supports_materialized_view() is now version-aware (PG 9.3+); an unadapted dialect cannot answer version questions. The contract matches the existing supports_table_partitioning(), so the cases now pass version explicitly Assertions for the 9.2/9.3/15 version points were added at the same time.
PostgreSQL 19 Beta 4 withdrew SQL/PGQ before the 19.0 release, and no earlier release shipped it either. Every graph grammar production was pulled from the parser, so CREATE PROPERTY GRAPH, CREATE VERTEX TABLE, CREATE EDGE TABLE, MATCH, GRAPH_TABLE and PROPERTIES are now syntax errors. Verified against the servers in tests/config/postgres_scenarios.yaml. All of the above are rejected with SQLSTATE 42601 on 19beta4, and the supporting catalog objects are gone as well (pg_proc graph/vertex/edge functions 1 -> 0, pg_type graph/agtype types 14 -> 0, pg_class property-graph relations 2 -> 0, and the GRAPH_TABLE/VERTEX/EDGE/PROPERTY parser keywords 4 -> 0 when compared against 19beta3). The same holds for 9.6 through 18, which never had the feature. The generated statements were also executed against a 19beta3 container, which did have it: 8 of 9 were still rejected, so the formatters were never validated against a real server. The capability is therefore resolved from explicit opt-ins rather than from self.version, which is how PostgresPropertyGraphQueryMixin already behaved; this change only documents the reasoning. No expression, formatter or capability behaviour is altered. - Expand the mixin module/class/method docstrings with the catalog evidence and per-probe rationale - Document graph_feature_overrides in PostgresDialect.__init__, and why the mixin must precede GraphMixin/GraphTableMixin in the MRO - Fix the stale path in the formatter test module header, and record in both test modules why the formatter tests drive the dialect through the explicit opt-in instead of relying on the default - Add the beta3/beta4 catalog comparison to the availability doc (en_US, zh_CN)
Expression classes moved under backend.expression/ and several were renamed during the derived-DDL refactor, leaving relative-import depths and forward references in TYPE_CHECKING blocks pointing at modules and classes that no longer exist. Type checkers and any runtime evaluation of those blocks fail to resolve them. Verified statically: every TYPE_CHECKING import in the touched files now resolves to an existing module and an existing top-level symbol, and no string annotation references a name absent from the workspace.
The suite applies markers that were never declared in [tool.pytest.ini_options] markers. pytest only warns for an unknown marker, so the warning scrolls past unnoticed and a dropped or misspelt registration is silent. Registers the ones in use, with descriptions that say what they do: collection grouping, not ordering. `feature` labels feature-area tests and `benchmark_relation` marks relation benchmarks; neither affects scheduling, so neither substitutes for the xdist_group marker that actually serialises a suite. usefixtures needs no entry and is not added: pytest provides it, so it was a false positive in the isolation audit rather than a missing registration.
… revisions The installed hook exec'd scripts/preflight.sh unconditionally. That hook is a local, untracked artifact, so it survives a checkout of any commit that predates the script, and on such a commit exec failed with a bare "No such file or directory" — blocking the commit with nothing to act on. In this repository scripts/preflight.sh first appears at ef1a19d, so every earlier commit is affected. The failure is also silent in a dangerous way: it swallowed a `git commit --amend`, so a commit-message rewrite reported as applied when it had not been, and had to be redone. The hook now checks the script is executable and exits 0 otherwise, so a revision without the pre-flight simply skips it. Normal operation is unchanged. Also honours core.hooksPath. git ignores .git/hooks entirely once that is set, so the hook was being written to a location that would never run while this script still reported success. No repository sets it today, so that part is latent.
Both dependencies were pinned to feature/activerecord-derived-ddl, a branch that exists only to carry the derived-DDL work. Now that the core change has landed in release/v1.0.0.dev30 and the matching testsuite change in release/v1.0.0.dev17, pinning to the feature branch means CI keeps resolving a branch that will be deleted once the backend PRs merge. This is a prerequisite for merging: leaving the old ref in place means the first run after that branch disappears fails to resolve, not the code.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/v1.0.0.dev17 #58 +/- ##
========================================================
- Coverage 77.32% 76.92% -0.40%
========================================================
Files 314 330 +16
Lines 12884 15360 +2476
Branches 2042 2606 +564
========================================================
+ Hits 9962 11816 +1854
- Misses 2211 2580 +369
- Partials 711 964 +253
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Brings the PostgreSQL backend up to the derived-DDL model in
rhosocial/python-activerecord#121, including the PostgreSQL 19 support that came with it.
The largest part of this is completing DDL expression coverage, which is why the diff is
substantial across
expression/ddl. On top of the typed-option migration:PostgresColumnDefinitionwith typed column options and backend identity syntax.PostgresCreateTableOptions, includingUNLOGGEDrendering.PostgresPartitionClause, derived from the generic clause, with a partition lifecycleprovider and declared partition support capabilities.
ALTER ... USING,EXCLUDEconstraintsand vector indexes.
dialect_optionsbag.Capability reporting is corrected as part of this:
supports_table_inheritsis declared(after removing a duplicate declaration),
supports_comment_onis declared and the invalidinline column
COMMENTremoved,CommentSupportis classified for PostgreSQL conformance,and generated-column capability bits are pinned by tests.
PostgresDatabaseMixinis wiredinto the dialect, dialect base initialisation is restored, and unrelated
COPY/REPACKimports are removed.
Materialized-view support keeps its
storage_optionsand gains complete DDL coverage, andthe MV tests are realigned with the corrected contracts. A live
Model.truncate()test(sync and async) is added, along with coverage for
DDLSourcecollection, tableinheritance and tablespace.
dialect_optionsis removed from data-type value objects, theILIKEexpression, DDLexpressions and index expressions, including a dead pass-through. The generated DDL
lifecycle is removed.
Type of change
feat: A new featuredocs: Documentation only changes (if related to the new feature)test: Adding missing tests or correcting existing tests (for the new feature)chore: Changes to the build process or auxiliary tools and libraries (if related to the new feature)Breaking Change
No commit carries a
!marker. The public model and query APIs are unchanged.Impact on End-Users:
No backward-incompatible changes are expected for typical end-user applications. DDL is now
generated from model declarations rather than a free-form option bag, so schema output comes
from the same declarations the ORM already exposes.
Impact on Backend Developers (Custom Implementations):
Relevant to anyone extending or forking this backend:
dialect_optionsbag is removed from data-type value objects, theILIKEexpression, DDL expressions and index expressions, along with a dead pass-through.
Anything reading or writing it must move to a declared option or capability bit.
nodes from core.
supports_comment_onis now declared, the invalid inline column
COMMENTis gone, and a duplicatesupports_table_inheritsdeclaration was removed. A conformance suite that asserted theold values will need updating.
fix(postgres): drop index by name without ON <table>corrects index-drop rendering, soSQL emitted for a bare index name changes.
were repaired so type checkers resolve them.
Related Repositories/Packages
The core half is rhosocial/python-activerecord#121 (merged into
release/v1.0.0.dev30) andthe shared contract half is rhosocial/python-activerecord-testsuite#21 (merged into
release/v1.0.0.dev17).This PR also repoints this repository's workflow at those release branches. Both dependencies
were pinned to
feature/activerecord-derived-ddl, a branch that only exists to carry thederived-DDL work; leaving the old ref in place would have made the first run after that
branch is deleted fail to resolve.
Merged so far, and still to merge in the same sequence:
91b89e04e9d1f4a5f9a9fff98cf00870faf03c6d8c0c573289e7e2824d2e9f0Testing
Local full-suite execution was skipped at the user's instruction. Verification relied on
remote CI, which ran on this exact tip
4e9d1f4and covers PostgreSQL 9 through 19 Beta 4on Python 3.8 through 3.14, plus a benchmarks job. All twelve executed jobs pass; the
experimental matrix entry is skipped by the workflow.
That run also covers the workflow change in this PR: it is the first execution against
release/v1.0.0.dev30andrelease/v1.0.0.dev17rather than the feature branches, so thenew dependency refs are verified to resolve.
Checklist
changelog.d/(to be added after this PR isopened, named by the actual PR number)
Additional Notes
Four commits are not part of the DDL adaptation and are called out so a reviewer is not
surprised:
ci: install core and testsuite from release branches— the workflow repoint describedabove, a prerequisite for merging.
fix(scripts): stop the pre-commit hook from blocking commits on older revisions— arepository-hygiene fix, kept identical across all ten repositories so they do not drift.
fix(backend): repair unresolvable TYPE_CHECKING importsandchore(test): register the pytest markers the suite already uses— mechanical repairs required for type checkingand lint to run at all after the namespace split.
docs(postgres): record why SQL/PGQ stays disabled after 19 Beta 4documents a deliberatenon-change: SQL/PGQ usage documentation was dropped, and this records that the decision
still holds on PostgreSQL 19 Beta 4 rather than being an oversight.