diff --git a/README.md b/README.md index 4a5c6bb7..2acaffa5 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,8 @@ refusal — never a silently wrong or incomplete result: through to a blocking rewrite, and the route's preflight refuses table shapes and environments it cannot run against. The shadow build also refuses a statement the copy cannot carry: a `USING` expression it cannot reproduce, - or a dropped-and-re-added column it cannot fill. + a dropped-and-re-added column it cannot fill, or a new identity or `serial` + column whose sequence the swap cannot carry. - **Row security** is included in exports when present, with reviewable before/after differences. The [atomic executor](docs/atomic-row-security.md) applies RLS-only changes to existing tables through `migrate --desired` or the Go API. Only the Go API supports binding execution to diff --git a/docs/capabilities.md b/docs/capabilities.md index 68328175..b4d2bf8b 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -207,7 +207,7 @@ or the only online form is a direct statement the operator runs). | Operation | Status | Engine path | Online-safety problem? | Behavior and why | | --- | --- | --- | --- | --- | | `ADD COLUMN` (no default, a constant default, or a non-volatile default such as `now()`) | ✅ | native, as-is | Yes | Metadata-only / fast default (PG 11+), proven by adding the column to an empty temporary table first; executes instantly under bounded locks | -| `ADD COLUMN` with volatile default (`gen_random_uuid()`, `clock_timestamp()`, identity, `serial`, …) | 🟡 | copy-and-swap | Yes | Table rewrite; routes to copy-and-swap, which does not carry a volatile default yet (the shadow fill would have to evaluate the default per row and the stream would have to agree) | +| `ADD COLUMN` with volatile default (`gen_random_uuid()`, `clock_timestamp()`, identity, `serial`, …) | 🟡 | copy-and-swap | Yes | Table rewrite; routes to copy-and-swap, which does not carry a volatile default yet (the shadow fill would have to evaluate the default per row and the stream would have to agree). An identity or `serial` column is further refused at the shadow build itself with `shadow-statement-new-sequence`, as a sequence the swap cannot carry to the live table — the engine's guard for library callers that reach the build without this route check — see [limitations.md](limitations.md) | | `ADD COLUMN` of a domain type with a `CHECK` or `NOT NULL`, with or without a default | 🟡 | copy-and-swap | Yes | Table rewrite to check every row against the domain; routes to copy-and-swap, which does not carry an added column yet (the shadow fill and the stream would each fill it from the default) | | `ADD COLUMN ... GENERATED ... STORED` | 🟡 | copy-and-swap | Yes | Table rewrite; routes to copy-and-swap, which does not carry stored generated columns yet — the copy must **recompute, never copy,** them on the shadow table | | `ALTER COLUMN ... SET EXPRESSION` (changed stored generated expression) | 🟡 | copy-and-swap | Yes | Planned but not executed. `diff` shows a changed stored generated expression as one `SET EXPRESSION` statement of kind `set-generated-expression`, refused with `unsupported-statement`, rather than a drop and re-add that would also drop the column's indexes. It recomputes every row under `ACCESS EXCLUSIVE`, and the statement needs PostgreSQL 17 or later | diff --git a/docs/invariants.md b/docs/invariants.md index 762d3ed7..6efcc8e8 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -1173,9 +1173,19 @@ IDENTITY`, and the dependents RF-2 names), and `CheckCopySwapEnvironment` (`wal_level`, the derived slot name's availability, slot and WAL-sender headroom, caller-measured disk headroom against the shadow copy's requirement), chained by type — each check accepts only the previous one's proof and the `CopySwapTarget` the writing steps -require is minted by the last; for the native route's sequences, `pkg/executor` -`admitSequence`, which `RunSequence` calls to admit every step's shape and target before the -first step runs. `migrate --dry-run` reports the same shape refusals through +require is minted by the last; inside the shadow build's transaction, `pkg/schemachange` +`readSequenceOwnedColumns` and `refuseNewSequenceColumns` refuse a gated statement that +gives the shadow an identity or sequence-owned column the source does not have +(`shadow-statement-new-sequence`), since the swap carries only the source's sequences and +would otherwise stop on the new one after the whole copy — a prerequisite knowable before +the first row moves, caught before the build commits (tests +`TestBuildShadowRefusesANewIdentityColumnBesideARetype`, +`TestBuildShadowRefusesANewSerialColumn`, +`TestBuildShadowRefusesMakingAKeptColumnAnIdentity`, +`TestBuildShadowAdmitsARetypeBesideTheSourcesSerialKey`, +`TestRunnerRefusesANewIdentityColumnBeforeAnyRowOrFence`); for the native route's sequences, +`pkg/executor` `admitSequence`, which `RunSequence` calls to admit every step's shape and +target before the first step runs. `migrate --dry-run` reports the same shape refusals through `migrate.AdmitRoutedSteps`, a read-only surface of that admission rather than a second enforcement; the size guard and privilege preflight need live facts and stay the apply's. *Planned enforcement:* wiring those checks into the route, and @@ -1269,7 +1279,9 @@ field for field against the checkpointed proof. A re-added column the copy canno its default — the primary key, the replacement of a source identity column, or a column that refuses NULL, by its own `NOT NULL` or a domain's, with no default of its own or on its type — is refused inside the build's transaction with `shadow-statement-readd`, so no shadow is -committed for a statement whose copy could not run. +committed for a statement whose copy could not run. A column re-added as an identity or +`serial` is fillable by the server, so it passes this check and is refused by ST-6's +`shadow-statement-new-sequence` instead. *Enforced:* `pkg/schemachange` (`readColumnPositions` reads both relations' live attribute numbers inside the build's and the inspection's transaction; `sameColumns` keeps a column only when its live position equals its shadow attribute number; `copyColumns` takes only those; @@ -1288,7 +1300,7 @@ refuses the three re-adds before the build commits). Tests: `TestBuildShadowRefusesAReaddedNotNullColumnWithNoDefault`, `TestBuildShadowRefusesAReaddedPrimaryKey`, `TestBuildShadowRefusesAStatementThatReplacesAnIdentityColumn`, -`TestBuildShadowDoesNotRefuseAPlainColumnReaddedAsAnIdentity`, +`TestBuildShadowRefusesAPlainColumnReaddedAsAnIdentity`, `TestBuildShadowRefusesAReaddedColumnWhoseDomainRefusesNull`, `TestBuildShadowRefusesAReaddedColumnWhoseBaseDomainRefusesNull`, `TestBuildShadowRefusesAReaddedColumnWhoseDefaultIsOnlyOnTheBaseDomain`, diff --git a/docs/limitations.md b/docs/limitations.md index 47818db8..897c6f73 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -28,7 +28,8 @@ not escape hatches: | `ALTER COLUMN TYPE ... USING ` the copy cannot reproduce | The copy-and-swap route (above) runs the retype, but the shadow builder admits a `USING` expression only when it can evaluate it again, identically, in the chunk copy, the verifier's recopy, and every change the catch-up flushes. Three rules, each refused as `shadow-statement-conversion` before any shadow is committed: the expression must be **immutable** by the server's own classification (no `now()`, `random()`, `current_setting()`, session-dependent casts such as `text` to `timestamptz`, or functions declared `STABLE` or `VOLATILE`; checked by trying it as a stored generated column's expression on the empty shadow inside the build's transaction, then rolling that attempt back), it must **name only the column it retypes**, bare and unqualified (no neighbouring column, no `t.col`, no whole-row `t.*`), and it must not retype the **primary key**, whose values every chunk range, watermark, and resume clear are expressed in. The expression is also refused when the deparser cannot prove it rendered the text faithfully. Rewrite the expression in terms of the retyped column alone and of `IMMUTABLE` functions, or retype without `USING` so the server's assignment cast converts the value; the refusal names the column, the expression, and that rewrite. The spellings most often refused, and what to write instead, are in [`USING` expressions the copy reproduces](#using-expressions-the-copy-reproduces) below. | | A schema change to the source table while a copy-and-swap runs | Unsupported: logical decoding does not stream DDL, and the copy does not lock it out — a chunk holds `ACCESS SHARE` on the source only for its own transaction, so `ALTER TABLE` on the source completes between chunks. The copy selects the source by column name, so what happens depends on whether every copied name still resolves. A copied column renamed or dropped stops the copy at its next chunk with the server's `undefined_column` error (SQLSTATE `42703`), which is not a typed refusal; that chunk stays in flight and nothing from it reaches the shadow. A change that leaves every copied name resolving to a column of a compatible type — two columns swapping names, a column renamed and re-added under the old name, a widened type — is not detected by the copy, which completes, and after a swap or a re-add the rows copied since hold another column's values or the new column's default. Once the replication stream has decoded a change to the table, a later relation description that differs stops it with `ErrSourceShapeChanged`. None of these reaches the swap: the verifier compares source and shadow rows by column name, and the cutover gate re-fingerprints the source and refuses any changed column as `cutover-schema-drift` ([refusal-classes.md](refusal-classes.md#shadow-operation-refusals-keyed-on-refusalcause)), after which the shadow is dropped and the change re-planned. | | A concurrent index build whose name is occupied by an invalid index, or whose table carries quarantined debris | The build refuses before running and never drops the occupant: an invalid entry is either abandoned debris or someone's build still in progress, and a drop by name cannot tell them apart. The refusal is typed by proof (`invalid-index-abandoned`, `invalid-index-build-in-flight`, `invalid-index-other-table`, `invalid-index-not-droppable`, `invalid-index-builder-unobservable`), and a table carrying an entry a previous recovery quarantined (`pgsprite_abandoned_`) but did not drop is refused the same way, so debris never goes quiet. The proven removal — `executor.RebuildAbandonedIndex`, or `executor.DropAbandonedIndex` to remove the leftover without running the build; both library-only today, no CLI verb — takes a bounded `SHARE UPDATE EXCLUSIVE` lock on the table (which any running concurrent index command holds), re-verifies the entry by OID, renames it to an identity-derived name and drops that, all within one shared budget; it refuses a visible in-flight build, another table's entry, or an entry the server will not drop concurrently (a partitioned table's index, an index partition, a constraint's index), and reports a lock budget rather than wait behind a build it cannot see. From the CLI, the [runbook](invalid-index-recovery.md) applies. | -| A copy-and-swap statement that drops a column and adds one of the same name | Like any drop, it runs only under `--allow-destructive`; without it the statement is refused with `destructive-change` and the `alter-column-type-in-place` guidance before the shadow build starts. Under the acknowledgement, the re-added column is a new column: every row holds its default (or NULL), as under a direct `ALTER TABLE`, and the dropped column's values are not copied into it ([copy-and-swap D1](copy-and-swap-design.md#d1--no-durable-scratch-database)). Three such re-adds have no default to take or are the key the copy is driven by, and the shadow build refuses them with `shadow-statement-readd` inside its own transaction, before any shadow, fence, or checkpoint row exists: re-adding the primary key column, re-adding a source identity column, and re-adding a column that refuses NULL (by its own `NOT NULL` or its domain's) with no default of its own or on its type. The remedy depends on which: a primary key is refused whatever the new column declares, so a default does not help — change the key in place with `ALTER COLUMN TYPE`; a source identity column is refused whatever the new column declares too — drop it in one schema change and add the new column in the next; any other column runs once the re-added column has a default (its own, or its domain's). A nullable, defaulted, or generated re-add builds and runs. A re-add declared as an identity or `serial` builds, copies, and then stops at the swap with `cutover-swap-mismatch`, leaving the shadow and the hold on the source for `--restart`, as any new identity or sequence-owned column added through copy-and-swap does. The refusal covers only these named re-add shapes: a new column the server cannot fill in other ways — a `CHECK` that rejects the column's default or NULL, a `NOT NULL` column added without a drop beside a retype, or the key replaced under a new name — still builds and stops at the first chunk or the copier's shape check, failing closed; a first-chunk stop is the server's class `23` error, a data fault, so the run abandons itself and the verdict is `failed` with `code: data-fault`. A run checkpointed by an earlier build for a drop-and-re-add statement no longer resumes (`resume-row-mismatch`, because the copy list it recorded names the re-added column); abandon it and run again rather than retrying the resume. | +| A copy-and-swap statement that drops a column and adds one of the same name | Like any drop, it runs only under `--allow-destructive`; without it the statement is refused with `destructive-change` and the `alter-column-type-in-place` guidance before the shadow build starts. Under the acknowledgement, the re-added column is a new column: every row holds its default (or NULL), as under a direct `ALTER TABLE`, and the dropped column's values are not copied into it ([copy-and-swap D1](copy-and-swap-design.md#d1--no-durable-scratch-database)). Three such re-adds have no default to take or are the key the copy is driven by, and the shadow build refuses them with `shadow-statement-readd` inside its own transaction, before any shadow, fence, or checkpoint row exists: re-adding the primary key column, re-adding a source identity column, and re-adding a column that refuses NULL (by its own `NOT NULL` or its domain's) with no default of its own or on its type. The remedy depends on which: a primary key is refused whatever the new column declares, so a default does not help — change the key in place with `ALTER COLUMN TYPE`; a source identity column is refused whatever the new column declares too — drop it in one schema change and add the new column in the next; any other column runs once the re-added column has a default (its own, or its domain's). A nullable, defaulted, or generated re-add builds and runs. A re-add declared as an identity or `serial` is refused in the same transaction with `shadow-statement-new-sequence` (next row). The refusal covers only these named re-add shapes: a new column the server cannot fill in other ways — a `CHECK` that rejects the column's default or NULL, a `NOT NULL` column added without a drop beside a retype, or the key replaced under a new name — still builds and stops at the first chunk or the copier's shape check, failing closed; a first-chunk stop is the server's class `23` error, a data fault, so the run abandons itself and the verdict is `failed` with `code: data-fault`. A run checkpointed by an earlier build for a drop-and-re-add statement no longer resumes (`resume-row-mismatch`, because the copy list it recorded names the re-added column); abandon it and run again rather than retrying the resume. | +| A copy-and-swap statement that gives the table a new identity or `serial` column | The swap carries only the source's sequences to the live table — each re-owned by the shadow column that kept its source column's name — so a sequence the gated statement mints on the shadow (`ADD COLUMN … GENERATED … AS IDENTITY`, `ADD COLUMN … serial`, `ALTER COLUMN … ADD GENERATED … AS IDENTITY` on a kept column, or a dropped column re-added as one) has no source counterpart to carry. The shadow build refuses such a statement with `shadow-statement-new-sequence` inside its own transaction, after the statement has run on the empty shadow and the shadow's columns and sequence-ownership edges have been read, so no shadow, fence, or checkpoint row exists; the source's own serial and identity columns are not new sequences and pass. From the CLI these statements are routed before the build reaches them — a new identity or `serial` column is a volatile-default `ADD COLUMN`, which the copy-and-swap backend does not carry yet (`backend-unavailable`), and `ADD GENERATED` on an existing column is `unsupported-operation` — so the build-time refusal is the library path's guard (`schemachange.Runner`, `BuildShadow`) and the engine's own defence should a route later admit them. Declare the column plain in this schema change and add the identity, or attach the sequence, in a later one that the server applies in place. | ## `USING` expressions the copy reproduces diff --git a/docs/refusal-classes.md b/docs/refusal-classes.md index cd833aa1..e67b8349 100644 --- a/docs/refusal-classes.md +++ b/docs/refusal-classes.md @@ -165,10 +165,11 @@ reads it through wrapping. Every one of these refusals is fail-closed under `ErrInvariantViolation` ([SAFETY.md](../SAFETY.md#rules-inside-the-core)): the sentinel is the mechanism that stops the operation, and the cause is what tells the importer which way to react. Only the causes that mean the caller handed the engine an incoherent input, or that the -engine's own write did not take, are `invariant-violation`; two, `shadow-statement-conversion` -and `shadow-statement-readd`, are `capability-boundary`, since each describes a statement the -engine declines to carry rather than a fault; the rest describe a lock, a proof, or a relation -whose state moved under the operation, and the importer retries, re-plans, or cleans up. +engine's own write did not take, are `invariant-violation`; three, `shadow-statement-conversion`, +`shadow-statement-readd`, and `shadow-statement-new-sequence`, are `capability-boundary`, since +each describes a statement the engine declines to carry rather than a fault; the rest describe a +lock, a proof, or a relation whose state moved under the operation, and the importer retries, +re-plans, or cleans up. `RefusalCause.Class()` answers the `class` column below as a `refusal.Class` (`pkg/refusal`, the core's declaration of the classes the cause tables in this document use), so an importer triages on the class without reading the cause's text, and a periphery adapter converts it to @@ -194,7 +195,8 @@ is `RefusalCause.Class()` converted by token (`pkg/migrate/copy_swap_refusal.go` | `shadow-foreign-relation` | The relation wearing the shadow's name is not a plain table owned by the source's owner ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | The engine did not build it and will not touch it; an operator removes or renames the relation. | | `shadow-grants-differ` | The shadow's grants still differ from the source's after the build synchronised them ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `invariant-violation` | The build wrote the grants under the table lock and read back something else; that is the engine's write, not the environment. | | `shadow-identity-handoff` | A source identity column the change kept on the shadow does not carry `DEFAULT nextval()` there, or the shadow declares an identity of its own ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | The change itself, or a later edit to the shadow, replaced the handoff; change the statement or drop the shadow and rebuild. | -| `shadow-statement-readd` | The gated statement drops a column and adds one under the same name that the copy cannot fill: the primary key, whose values the copy's chunk ranges and watermark are expressed in; a source identity column, whose sequence is handed only to the same column kept on the shadow; or a column that refuses NULL — by its own `NOT NULL` or a domain's, including a domain its type is built on — with no default of its own, no default on its type, no generation expression, and no identity, so the first chunk would have no value to give it ([ST-11](invariants.md#st-11--the-shadow-takes-a-columns-values-only-from-the-same-source-column)) | `capability-boundary` | A re-added column is a new column that takes its default, never the dropped column's values, and these three have no default to take or are the key the copy is driven by. The build refuses inside its own transaction, after the statement has run on the empty shadow and its column positions have been read, so nothing is committed and no checkpoint row is written. Re-plan the statement rather than retry it; the remedy depends on the column. A primary key is refused whatever the new column declares — a default does not lift it — so change the key in place with `ALTER COLUMN TYPE`. A source identity column is refused whatever the new column declares too: drop it in one schema change and add the new column in the next. For any other column, give the re-added column a default (its own, or its domain's). A nullable, defaulted, or generated re-add builds and runs; a re-add declared as an identity or `serial` builds but stops at the swap (`cutover-swap-mismatch`), as any new identity or sequence-owned column added through copy-and-swap does. The message names the column and the remedy; the cause carries no column. | +| `shadow-statement-readd` | The gated statement drops a column and adds one under the same name that the copy cannot fill: the primary key, whose values the copy's chunk ranges and watermark are expressed in; a source identity column, whose sequence is handed only to the same column kept on the shadow; or a column that refuses NULL — by its own `NOT NULL` or a domain's, including a domain its type is built on — with no default of its own, no default on its type, no generation expression, and no identity, so the first chunk would have no value to give it ([ST-11](invariants.md#st-11--the-shadow-takes-a-columns-values-only-from-the-same-source-column)) | `capability-boundary` | A re-added column is a new column that takes its default, never the dropped column's values, and these three have no default to take or are the key the copy is driven by. The build refuses inside its own transaction, after the statement has run on the empty shadow and its column positions have been read, so nothing is committed and no checkpoint row is written. Re-plan the statement rather than retry it; the remedy depends on the column. A primary key is refused whatever the new column declares — a default does not lift it — so change the key in place with `ALTER COLUMN TYPE`. A source identity column is refused whatever the new column declares too: drop it in one schema change and add the new column in the next. For any other column, give the re-added column a default (its own, or its domain's). A nullable, defaulted, or generated re-add builds and runs; a re-add declared as an identity or `serial` is refused by the next check, `shadow-statement-new-sequence`, as any new identity or sequence-owned column through copy-and-swap is. The message names the column and the remedy; the cause carries no column. | +| `shadow-statement-new-sequence` | The gated statement gives the shadow an identity column, or a column owning a sequence (`serial`), that the source does not have: a new column declared so, a kept column given `ADD GENERATED … AS IDENTITY`, or a dropped column re-added as one ([ST-6](invariants.md#st-6--preflight-before-the-first-write)) | `capability-boundary` | The swap carries only the source's sequences to the live table, re-owning each by the shadow column that kept its source column's name; a sequence the statement minted on the shadow has no source counterpart to carry, so the swap's catalog re-read would stop on it after the whole copy. The build refuses inside its own transaction instead, after the statement has run on the empty shadow and the shadow's columns and `pg_depend` ownership edges have been read, so nothing is committed and no checkpoint row is written. The source's own serial and identity columns do not trip the check: the shadow draws them from the source's sequences without owning them until the swap. Declare the column plain in this schema change and add the identity, or attach the sequence, in a later one that the server applies in place. The message names the column and the remedy; the cause carries no column. | | `cutover-unverified` | The cutover gate was handed an empty built-shadow or verified-shadow proof, two proofs naming different relations or different relation OIDs (a shadow rebuilt under the same name after the checksum ran), or a verified watermark that does not cover the whole key space ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Only the copier and verifier mint those proofs for the same table; a caller that reaches the gate without them skipped the checksum, which no retry corrects. | | `cutover-undrained` | The cutover was handed a verified-shadow proof a live pass minted — the shadow equal to the source as of a position the stream had reached, with every change since still in the stream — and no drain to apply those changes under the swap lock ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Pass a `DrainFunc` that applies the stream through the final position under the swap lock, or stop the stream and re-verify with a quiesced `Check`; the proof is sound, the hand-off to the swap is not. | | `cutover-drain-deferred` | The shadow refused a batch the swap's drain had to apply under the swap lock: a unique-key move among the changes committed since the live verification pass that cannot be ordered into the shadow, with no copy left to change the outcome ([CO-6](invariants.md#co-6--unique-secondary-key-moves-must-converge-postgresql-specific-gap)) | `environmental` | The swap transaction is rolled back and the source stays live; the row is left in `cutover` with the position the drain reopened from, so a resume drains the same changes again once the rows the move touched have settled. A move the shadow cannot ever order is re-planned. | diff --git a/pkg/capabilities/capabilities.yaml b/pkg/capabilities/capabilities.yaml index 068ffda1..c069bdd6 100644 --- a/pkg/capabilities/capabilities.yaml +++ b/pkg/capabilities/capabilities.yaml @@ -21,7 +21,7 @@ rows: migrate: refused diff: refused refusal_reason: backend-unavailable - reason_notes: "Table rewrite; routes to copy-and-swap, which does not carry a volatile default yet (the shadow fill would have to evaluate the default per row and the stream would have to agree)" + reason_notes: "Table rewrite; routes to copy-and-swap, which does not carry a volatile default yet (the shadow fill would have to evaluate the default per row and the stream would have to agree). An identity or `serial` column is further refused at the shadow build itself with `shadow-statement-new-sequence`, as a sequence the swap cannot carry to the live table — the engine's guard for library callers that reach the build without this route check — see [limitations.md](limitations.md)" - id: add-column-of-a-domain-with-constraints area: "column_changes" operation: "`ADD COLUMN` of a domain type with a `CHECK` or `NOT NULL`, with or without a default" diff --git a/pkg/schemachange/new_sequence_column.go b/pkg/schemachange/new_sequence_column.go new file mode 100644 index 00000000..c5ba3a88 --- /dev/null +++ b/pkg/schemachange/new_sequence_column.go @@ -0,0 +1,84 @@ +package schemachange + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5" +) + +// sequenceOwnedColumn is one shadow column that owns a sequence after the +// gated statement ran on the shadow: an identity column, whose sequence +// depends on it internally, or a serial-style column whose sequence is +// OWNED BY it. +type sequenceOwnedColumn struct { + column string + identity bool +} + +// readSequenceOwnedColumns lists the shadow's columns that declare an +// identity or own a sequence, in attribute order. The shadow is cloned +// LIKE … EXCLUDING IDENTITY, a source identity column is handed off as a +// plain column with DEFAULT nextval(), and a source +// serial column's default is copied without the OWNED BY edge, so before +// the gated statement no shadow column declares an identity or owns a +// sequence; every one this read returns was declared by the statement. +func readSequenceOwnedColumns(ctx context.Context, tx pgx.Tx, shadowOID uint32) ([]sequenceOwnedColumn, error) { + // INV: CO-9 — every relation, operator, and type is pg_catalog-qualified, + // so a planted oid = regclass operator cannot hide a column's sequence + // edge from the check. + rows, err := tx.Query(ctx, ` + SELECT a.attname, a.attidentity OPERATOR(pg_catalog.<>) '' + FROM pg_catalog.pg_attribute a + WHERE a.attrelid OPERATOR(pg_catalog.=) $1 + AND a.attnum OPERATOR(pg_catalog.>) 0 + AND NOT a.attisdropped + AND (a.attidentity OPERATOR(pg_catalog.<>) '' + OR EXISTS ( + SELECT 1 + FROM pg_catalog.pg_depend d + JOIN pg_catalog.pg_class s + ON s.oid OPERATOR(pg_catalog.=) d.objid + AND s.relkind OPERATOR(pg_catalog.=) 'S' + WHERE d.classid OPERATOR(pg_catalog.=) 'pg_catalog.pg_class'::pg_catalog.regclass + AND d.refclassid OPERATOR(pg_catalog.=) 'pg_catalog.pg_class'::pg_catalog.regclass + AND d.refobjid OPERATOR(pg_catalog.=) a.attrelid + AND d.refobjsubid OPERATOR(pg_catalog.=) a.attnum + AND (d.deptype OPERATOR(pg_catalog.=) 'a' OR d.deptype OPERATOR(pg_catalog.=) 'i'))) + ORDER BY a.attnum`, shadowOID) + if err != nil { + return nil, fmt.Errorf("read sequence-owning columns: %w", err) + } + owned, err := pgx.CollectRows(rows, func(row pgx.CollectableRow) (sequenceOwnedColumn, error) { + var c sequenceOwnedColumn + err := row.Scan(&c.column, &c.identity) + return c, err + }) + if err != nil { + return nil, fmt.Errorf("read sequence-owning columns: %w", err) + } + return owned, nil +} + +// refuseNewSequenceColumns refuses a gated statement that gives the shadow +// an identity or sequence-owned column the source does not have: an added +// identity column, an added serial column, or a kept column the statement +// made an identity. The swap carries the source's sequences and identity +// columns to the live table and proves the live table has exactly those; +// a sequence the statement minted on the shadow has no place in that +// handoff, so a run carrying one would copy every row and then stop at +// the swap. The build refuses it instead, inside its own transaction, +// before any shadow, fence, or checkpoint row exists. owned is the +// shadow's readSequenceOwnedColumns after the statement ran; the message +// names the first such column in attribute order. +func refuseNewSequenceColumns(schema, table string, owned []sequenceOwnedColumn) error { + if len(owned) == 0 { + return nil + } + // INV: ST-6 + c := owned[0] + if c.identity { + return refuse(CauseStatementNewSequence, nil, "the statement for %s.%s declares %s as an identity column, which the swap cannot carry to the live table; declare it a plain column here and add the identity in a later schema change", schema, table, c.column) + } + return refuse(CauseStatementNewSequence, nil, "the statement for %s.%s gives %s a sequence of its own, which the swap cannot carry to the live table; declare it a plain column here and attach the sequence in a later schema change", schema, table, c.column) +} diff --git a/pkg/schemachange/new_sequence_column_integration_test.go b/pkg/schemachange/new_sequence_column_integration_test.go new file mode 100644 index 00000000..5f07f1d2 --- /dev/null +++ b/pkg/schemachange/new_sequence_column_integration_test.go @@ -0,0 +1,99 @@ +package schemachange_test + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/block/pg-sprite/pkg/checkpoint" + "github.com/block/pg-sprite/pkg/schemachange" +) + +// Adding an identity column beside a retype gives the shadow a sequence the +// source does not have. The swap carries only the source's sequences to the +// live table, so the build refuses the statement inside its own transaction +// and commits nothing. The source's own serial key (id) is not a new +// sequence and does not trip the check. +func TestBuildShadowRefusesANewIdentityColumnBesideARetype(t *testing.T) { + f := newShadowFixture(t) + f.createOrders(t) + + _, err := f.build(t, "orders", ` + ALTER TABLE %s.orders + ADD COLUMN seqno bigint GENERATED BY DEFAULT AS IDENTITY, + ALTER COLUMN qty TYPE bigint`) + + assert.ErrorIs(t, err, schemachange.ErrInvariantViolation) + assert.Equal(t, schemachange.CauseStatementNewSequence, schemachange.RefusalCauseOf(err)) + assert.False(t, f.relationExists(t, schemachange.ShadowName(f.schema, "orders")), "a refused build creates nothing") +} + +// A serial column is a plain column that owns a sequence created for it, so +// it is refused for the same reason as an identity column: the swap has no +// source sequence to carry for it. +func TestBuildShadowRefusesANewSerialColumn(t *testing.T) { + f := newShadowFixture(t) + f.createOrders(t) + + _, err := f.build(t, "orders", ` + ALTER TABLE %s.orders + ADD COLUMN seqno serial`) + + assert.ErrorIs(t, err, schemachange.ErrInvariantViolation) + assert.Equal(t, schemachange.CauseStatementNewSequence, schemachange.RefusalCauseOf(err)) + assert.False(t, f.relationExists(t, schemachange.ShadowName(f.schema, "orders")), "a refused build creates nothing") +} + +// Making a kept column an identity column mints a sequence for it on the +// shadow without adding a column, so the check is on the sequence, not on +// the column list. The server accepts the statement on the empty shadow +// because qty is already NOT NULL; the build still refuses it. +func TestBuildShadowRefusesMakingAKeptColumnAnIdentity(t *testing.T) { + f := newShadowFixture(t) + f.createOrders(t) + + _, err := f.build(t, "orders", ` + ALTER TABLE %s.orders + ALTER COLUMN qty ADD GENERATED ALWAYS AS IDENTITY`) + + assert.ErrorIs(t, err, schemachange.ErrInvariantViolation) + assert.Equal(t, schemachange.CauseStatementNewSequence, schemachange.RefusalCauseOf(err)) + assert.False(t, f.relationExists(t, schemachange.ShadowName(f.schema, "orders")), "a refused build creates nothing") +} + +// A statement that only retypes beside the source's own serial key builds: +// the shadow's id still draws from the source's sequence rather than owning +// one, so the check sees no new sequence. +func TestBuildShadowAdmitsARetypeBesideTheSourcesSerialKey(t *testing.T) { + f := newShadowFixture(t) + f.createOrders(t) + + built, err := f.build(t, "orders", ` + ALTER TABLE %s.orders + ALTER COLUMN qty TYPE bigint`) + require.NoError(t, err) + + assert.Equal(t, []string{"id", "qty", "note", "sku"}, built.CopyColumns()) +} + +// Through the Runner the refusal lands before the run has touched anything: +// the shadow build is the first write, and it rolled back, so there is no +// shadow, no write fence on the source, and no checkpoint row to abandon. +func TestRunnerRefusesANewIdentityColumnBeforeAnyRowOrFence(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + + _, err := f.run(t, "orders", ` + ALTER TABLE %s.orders + ADD COLUMN seqno bigint GENERATED BY DEFAULT AS IDENTITY, + ALTER COLUMN qty TYPE bigint`, schemachange.RunOptions{}) + + assert.ErrorIs(t, err, schemachange.ErrInvariantViolation) + assert.Equal(t, schemachange.CauseStatementNewSequence, schemachange.RefusalCauseOf(err)) + assert.False(t, f.relationExists(t, schemachange.ShadowName(f.schema, "orders")), "no shadow was committed") + assert.False(t, f.hasFence(t, "orders"), "the source was never fenced") + _, err = f.store.Inspect(t.Context(), f.schema, "orders") + assert.ErrorIs(t, err, checkpoint.ErrNotFound, "the refused run writes no row") + f.assertWritable(t, "orders") +} diff --git a/pkg/schemachange/new_sequence_column_test.go b/pkg/schemachange/new_sequence_column_test.go new file mode 100644 index 00000000..6815a491 --- /dev/null +++ b/pkg/schemachange/new_sequence_column_test.go @@ -0,0 +1,35 @@ +package schemachange + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// A shadow with no identity or sequence-owning column after the statement +// is admitted: the source's handed-off identity columns and copied serial +// defaults never appear in the list, so an empty list is the usual case. +func TestRefuseNewSequenceColumnsAdmitsAShadowWithoutOne(t *testing.T) { + assert.NoError(t, refuseNewSequenceColumns("public", "orders", nil)) +} + +// A column the statement declared an identity is refused as a new sequence +// the swap cannot carry, and the message names the column. +func TestRefuseNewSequenceColumnsRefusesANewIdentityColumn(t *testing.T) { + err := refuseNewSequenceColumns("public", "orders", []sequenceOwnedColumn{{column: "sku", identity: true}}) + + require.ErrorIs(t, err, ErrInvariantViolation) + assert.Equal(t, CauseStatementNewSequence, RefusalCauseOf(err)) + assert.ErrorContains(t, err, "sku", "the message names the column") +} + +// A column the statement gave a sequence of its own — serial, or an +// explicit OWNED BY — is refused the same way. +func TestRefuseNewSequenceColumnsRefusesANewSerialColumn(t *testing.T) { + err := refuseNewSequenceColumns("public", "orders", []sequenceOwnedColumn{{column: "seqno"}}) + + require.ErrorIs(t, err, ErrInvariantViolation) + assert.Equal(t, CauseStatementNewSequence, RefusalCauseOf(err)) + assert.ErrorContains(t, err, "seqno", "the message names the column") +} diff --git a/pkg/schemachange/readded_column_integration_test.go b/pkg/schemachange/readded_column_integration_test.go index a196a84e..7f5d67b8 100644 --- a/pkg/schemachange/readded_column_integration_test.go +++ b/pkg/schemachange/readded_column_integration_test.go @@ -192,23 +192,23 @@ func TestBuildShadowRefusesAReaddedPrimaryKey(t *testing.T) { } // A plain column re-added as an identity column is a new column the server -// numbers itself, so the server fills it and the re-add refusal does not -// fire: the copy leaves it out and the build commits. This pins the build -// only. A run carrying it still stops at the swap, which refuses any new -// identity or sequence-owned column the source did not have -// (cutover-swap-mismatch), as it does for such a column added without a -// drop. -func TestBuildShadowDoesNotRefuseAPlainColumnReaddedAsAnIdentity(t *testing.T) { +// numbers itself, so the re-add refusal does not fire — the copy could leave +// it out. The build refuses it all the same, as a new sequence the swap +// cannot carry to the live table (shadow-statement-new-sequence, not +// shadow-statement-readd), inside its own transaction, so no shadow is left +// behind. +func TestBuildShadowRefusesAPlainColumnReaddedAsAnIdentity(t *testing.T) { f := newShadowFixture(t) f.createOrders(t) - built, err := f.build(t, "orders", ` + _, err := f.build(t, "orders", ` ALTER TABLE %s.orders DROP COLUMN sku, ADD COLUMN sku bigint GENERATED BY DEFAULT AS IDENTITY`) - require.NoError(t, err) - assert.Equal(t, []string{"id", "qty", "note"}, built.CopyColumns(), "the re-added identity column is not a copy column") + assert.ErrorIs(t, err, schemachange.ErrInvariantViolation) + assert.Equal(t, schemachange.CauseStatementNewSequence, schemachange.RefusalCauseOf(err)) + assert.False(t, f.relationExists(t, schemachange.ShadowName(f.schema, "orders")), "a refused build creates nothing") } // Dropping qty and adding it back NOT NULL as a domain with a default gives diff --git a/pkg/schemachange/readded_column_test.go b/pkg/schemachange/readded_column_test.go index 089df190..9909f5ce 100644 --- a/pkg/schemachange/readded_column_test.go +++ b/pkg/schemachange/readded_column_test.go @@ -27,9 +27,11 @@ func newReaddFixture(readded string) readdFixture { return readdFixture{source: source, shadow: append(shadow, columnPosition{name: readded, attnum: 4})} } -// A re-added column the copy can leave to its default is not refused: +// A re-added column the copy can leave to its default is not refused here: // nullable, with a default, generated, or an identity of its own replacing a // plain column. Each is a new column the copy omits and the server fills. +// The identity case is refused by the build's new-sequence check instead, +// which runs after this one. func TestRefuseUnfillableReaddsAdmitsAColumnTheServerFills(t *testing.T) { for name, column := range map[string]schemadiff.Column{ "nullable": {Name: "note", Type: "text"}, diff --git a/pkg/schemachange/refusal.go b/pkg/schemachange/refusal.go index 6712097f..c5d4c82b 100644 --- a/pkg/schemachange/refusal.go +++ b/pkg/schemachange/refusal.go @@ -82,6 +82,14 @@ const ( // the replacement of a source identity column whose sequence the // handoff cannot carry to it. CauseStatementReadd RefusalCause = "shadow-statement-readd" + // CauseStatementNewSequence means the gated statement gives the shadow + // an identity or sequence-owned column the source does not have — an + // added identity or serial column, or a kept column made an identity + // (ST-6): the swap hands the live table exactly the source's sequences + // and identity columns, so a sequence the statement minted would copy + // every row and then stop at the swap; the build refuses it before any + // shadow exists. + CauseStatementNewSequence RefusalCause = "shadow-statement-new-sequence" // CauseCutoverUnverified means the cutover gate was handed no usable // data-equality proof (CO-1): the built-shadow proof or the verified // shadow proof is empty, the two name different relations, or the @@ -249,6 +257,7 @@ func RefusalCauses() []RefusalCause { CauseGrantsDiffer, CauseIdentityHandoff, CauseStatementReadd, + CauseStatementNewSequence, CauseCutoverUnverified, CauseCutoverUndrained, CauseCutoverUnfenced, @@ -283,8 +292,8 @@ func (c RefusalCause) Invariant() string { switch c { case CauseLockUnproven, CauseLockLost, CauseLockHeldElsewhere, CauseLockUnconfirmed: return "LK-1" - case CauseProofEmpty, CauseSourceShape, CauseRelationReplaced, CauseSwapMismatch, - CauseCopyShadowMismatch, CauseCopyChunkUnknown: + case CauseProofEmpty, CauseSourceShape, CauseStatementNewSequence, CauseRelationReplaced, + CauseSwapMismatch, CauseCopyShadowMismatch, CauseCopyChunkUnknown: return "ST-6" case CauseCopyFrontierBroken: return "CO-4" @@ -322,7 +331,8 @@ func (c RefusalCause) Invariant() string { // operation is environmental — state outside the engine stopped a // supportable change. A statement the engine declines to carry as written // — a conversion the copy cannot reproduce, a re-added column it cannot -// fill — is a capability boundary. Every other cause is an invariant violation: an +// fill, a new identity or sequence the swap cannot carry — is a capability +// boundary. Every other cause is an invariant violation: an // input the caller handed in that cannot have come from this engine, or a // write of the engine's own that the catalog does not show; nothing on the // server changes that, so the importer reports it. The class is a triage @@ -341,7 +351,7 @@ func (c RefusalCause) Class() refusal.Class { CauseResumeShadowGone, CauseResumeSlotForeign, CauseResumeSlotHeld, CauseRestartSwapCommitted, CauseCopyConversionDrift: return refusal.ClassEnvironmental - case CauseStatementConversion, CauseStatementReadd: + case CauseStatementConversion, CauseStatementReadd, CauseStatementNewSequence: return refusal.ClassCapabilityBoundary case CauseLockUnproven, CauseProofEmpty, CauseStatementTarget, CauseGrantsDiffer, CauseCutoverUnverified, CauseCutoverUndrained, CauseSwapMismatch, CauseResumeTerminal, diff --git a/pkg/schemachange/refusal_test.go b/pkg/schemachange/refusal_test.go index 0e2fd583..b4646dd4 100644 --- a/pkg/schemachange/refusal_test.go +++ b/pkg/schemachange/refusal_test.go @@ -34,6 +34,7 @@ func TestEveryRefusalCauseNamesItsInvariant(t *testing.T) { CauseGrantsDiffer: "ST-5", CauseIdentityHandoff: "ST-5", CauseStatementReadd: "ST-11", + CauseStatementNewSequence: "ST-6", CauseSchemaDrift: "ST-5", CauseFidelityDrift: "ST-5", CauseIndexInvalid: "ST-5", @@ -90,6 +91,7 @@ func TestEveryRefusalCauseNamesItsClass(t *testing.T) { CauseGrantsDiffer: refusal.ClassInvariantViolation, CauseIdentityHandoff: refusal.ClassEnvironmental, CauseStatementReadd: refusal.ClassCapabilityBoundary, + CauseStatementNewSequence: refusal.ClassCapabilityBoundary, CauseCutoverUnverified: refusal.ClassInvariantViolation, CauseCutoverUndrained: refusal.ClassInvariantViolation, CauseCutoverUnfenced: refusal.ClassEnvironmental, diff --git a/pkg/schemachange/shadow.go b/pkg/schemachange/shadow.go index 1189864c..923f885f 100644 --- a/pkg/schemachange/shadow.go +++ b/pkg/schemachange/shadow.go @@ -307,6 +307,17 @@ func buildShadow(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLock if err := refuseUnfillableReadds(target.Schema(), target.Table(), target.PKColumn(), sourcePositions, shadowPositions, targetModel, shadowDomains, identities); err != nil { return BuiltShadow{}, err } + // A sequence the statement minted on the shadow has no place in the + // swap's handoff, which carries the source's sequences and identity + // columns alone; it is refused here, before a copy would run to the + // swap only to stop there. + owned, err := readSequenceOwnedColumns(ctx, tx, shadowOID) + if err != nil { + return BuiltShadow{}, err + } + if err := refuseNewSequenceColumns(target.Schema(), target.Table(), owned); err != nil { + return BuiltShadow{}, err + } // The handoff is proven on the shadow the statement left, not the one // the defaults were applied to: the proof records exactly what an // inspection of this shadow would accept.