schemachange: refuse a new identity or serial column at the shadow build

README.md

@@ -70,7 +70,8 @@ refusal — never a silently wrong or incomplete result:
   through to a blocking rewrite, and the route's preflight refuses table shapes
   and environments it cannot run against. The shadow build also refuses a
   statement the copy cannot carry: a `USING` expression it cannot reproduce,
-  or a dropped-and-re-added column it cannot fill.
+  a dropped-and-re-added column it cannot fill, or a new identity or `serial`
+  column whose sequence the swap cannot carry.
 - **Row security** is included in exports when present, with reviewable before/after differences. The
   [atomic executor](docs/atomic-row-security.md) applies RLS-only changes to existing tables
   through `migrate --desired` or the Go API. Only the Go API supports binding execution to

docs/capabilities.md

@@ -207,7 +207,7 @@ or the only online form is a direct statement the operator runs).
 | Operation | Status | Engine path | Online-safety problem? | Behavior and why |
 | --- | --- | --- | --- | --- |
 | `ADD COLUMN` (no default, a constant default, or a non-volatile default such as `now()`) | ✅ | native, as-is | Yes | Metadata-only / fast default (PG 11+), proven by adding the column to an empty temporary table first; executes instantly under bounded locks |
-| `ADD COLUMN` with volatile default (`gen_random_uuid()`, `clock_timestamp()`, identity, `serial`, …) | 🟡 | copy-and-swap | Yes | Table rewrite; routes to copy-and-swap, which does not carry a volatile default yet (the shadow fill would have to evaluate the default per row and the stream would have to agree) |
+| `ADD COLUMN` with volatile default (`gen_random_uuid()`, `clock_timestamp()`, identity, `serial`, …) | 🟡 | copy-and-swap | Yes | Table rewrite; routes to copy-and-swap, which does not carry a volatile default yet (the shadow fill would have to evaluate the default per row and the stream would have to agree). An identity or `serial` column is further refused at the shadow build itself with `shadow-statement-new-sequence`, as a sequence the swap cannot carry to the live table — the engine's guard for library callers that reach the build without this route check — see [limitations.md](limitations.md) |
 | `ADD COLUMN` of a domain type with a `CHECK` or `NOT NULL`, with or without a default | 🟡 | copy-and-swap | Yes | Table rewrite to check every row against the domain; routes to copy-and-swap, which does not carry an added column yet (the shadow fill and the stream would each fill it from the default) |
 | `ADD COLUMN ... GENERATED ... STORED` | 🟡 | copy-and-swap | Yes | Table rewrite; routes to copy-and-swap, which does not carry stored generated columns yet — the copy must **recompute, never copy,** them on the shadow table |
 | `ALTER COLUMN ... SET EXPRESSION` (changed stored generated expression) | 🟡 | copy-and-swap | Yes | Planned but not executed. `diff` shows a changed stored generated expression as one `SET EXPRESSION` statement of kind `set-generated-expression`, refused with `unsupported-statement`, rather than a drop and re-add that would also drop the column's indexes. It recomputes every row under `ACCESS EXCLUSIVE`, and the statement needs PostgreSQL 17 or later |

docs/invariants.md

@@ -1173,9 +1173,19 @@ IDENTITY`, and the dependents RF-2 names), and `CheckCopySwapEnvironment`
 (`wal_level`, the derived slot name's availability, slot and WAL-sender headroom,
 caller-measured disk headroom against the shadow copy's requirement), chained by type —
 each check accepts only the previous one's proof and the `CopySwapTarget` the writing steps
-require is minted by the last; for the native route's sequences, `pkg/executor`
-`admitSequence`, which `RunSequence` calls to admit every step's shape and target before the
-first step runs. `migrate --dry-run` reports the same shape refusals through
+require is minted by the last; inside the shadow build's transaction, `pkg/schemachange`
+`readSequenceOwnedColumns` and `refuseNewSequenceColumns` refuse a gated statement that
+gives the shadow an identity or sequence-owned column the source does not have
+(`shadow-statement-new-sequence`), since the swap carries only the source's sequences and
+would otherwise stop on the new one after the whole copy — a prerequisite knowable before
+the first row moves, caught before the build commits (tests
+`TestBuildShadowRefusesANewIdentityColumnBesideARetype`,
+`TestBuildShadowRefusesANewSerialColumn`,
+`TestBuildShadowRefusesMakingAKeptColumnAnIdentity`,
+`TestBuildShadowAdmitsARetypeBesideTheSourcesSerialKey`,
+`TestRunnerRefusesANewIdentityColumnBeforeAnyRowOrFence`); for the native route's sequences,
+`pkg/executor` `admitSequence`, which `RunSequence` calls to admit every step's shape and
+target before the first step runs. `migrate --dry-run` reports the same shape refusals through
 `migrate.AdmitRoutedSteps`, a read-only surface of that admission rather than a second
 enforcement; the size guard and privilege preflight need live facts and stay the apply's.
 *Planned enforcement:* wiring those checks into the route, and
@@ -1269,7 +1279,9 @@ field for field against the checkpointed proof. A re-added column the copy canno
 its default — the primary key, the replacement of a source identity column, or a column that
 refuses NULL, by its own `NOT NULL` or a domain's, with no default of its own or on its type —
 is refused inside the build's transaction with `shadow-statement-readd`, so no shadow is
-committed for a statement whose copy could not run.
+committed for a statement whose copy could not run. A column re-added as an identity or
+`serial` is fillable by the server, so it passes this check and is refused by ST-6's
+`shadow-statement-new-sequence` instead.
 *Enforced:* `pkg/schemachange` (`readColumnPositions` reads both relations' live attribute
 numbers inside the build's and the inspection's transaction; `sameColumns` keeps a column only
 when its live position equals its shadow attribute number; `copyColumns` takes only those;
@@ -1288,7 +1300,7 @@ refuses the three re-adds before the build commits). Tests:
 `TestBuildShadowRefusesAReaddedNotNullColumnWithNoDefault`,
 `TestBuildShadowRefusesAReaddedPrimaryKey`,
 `TestBuildShadowRefusesAStatementThatReplacesAnIdentityColumn`,
-`TestBuildShadowDoesNotRefuseAPlainColumnReaddedAsAnIdentity`,
+`TestBuildShadowRefusesAPlainColumnReaddedAsAnIdentity`,
 `TestBuildShadowRefusesAReaddedColumnWhoseDomainRefusesNull`,
 `TestBuildShadowRefusesAReaddedColumnWhoseBaseDomainRefusesNull`,
 `TestBuildShadowRefusesAReaddedColumnWhoseDefaultIsOnlyOnTheBaseDomain`,

docs/limitations.md

@@ -28,7 +28,8 @@ not escape hatches:
 | `ALTER COLUMN TYPE ... USING <expr>` the copy cannot reproduce | The copy-and-swap route (above) runs the retype, but the shadow builder admits a `USING` expression only when it can evaluate it again, identically, in the chunk copy, the verifier's recopy, and every change the catch-up flushes. Three rules, each refused as `shadow-statement-conversion` before any shadow is committed: the expression must be **immutable** by the server's own classification (no `now()`, `random()`, `current_setting()`, session-dependent casts such as `text` to `timestamptz`, or functions declared `STABLE` or `VOLATILE`; checked by trying it as a stored generated column's expression on the empty shadow inside the build's transaction, then rolling that attempt back), it must **name only the column it retypes**, bare and unqualified (no neighbouring column, no `t.col`, no whole-row `t.*`), and it must not retype the **primary key**, whose values every chunk range, watermark, and resume clear are expressed in. The expression is also refused when the deparser cannot prove it rendered the text faithfully. Rewrite the expression in terms of the retyped column alone and of `IMMUTABLE` functions, or retype without `USING` so the server's assignment cast converts the value; the refusal names the column, the expression, and that rewrite. The spellings most often refused, and what to write instead, are in [`USING` expressions the copy reproduces](#using-expressions-the-copy-reproduces) below. |
 | A schema change to the source table while a copy-and-swap runs | Unsupported: logical decoding does not stream DDL, and the copy does not lock it out — a chunk holds `ACCESS SHARE` on the source only for its own transaction, so `ALTER TABLE` on the source completes between chunks. The copy selects the source by column name, so what happens depends on whether every copied name still resolves. A copied column renamed or dropped stops the copy at its next chunk with the server's `undefined_column` error (SQLSTATE `42703`), which is not a typed refusal; that chunk stays in flight and nothing from it reaches the shadow. A change that leaves every copied name resolving to a column of a compatible type — two columns swapping names, a column renamed and re-added under the old name, a widened type — is not detected by the copy, which completes, and after a swap or a re-add the rows copied since hold another column's values or the new column's default. Once the replication stream has decoded a change to the table, a later relation description that differs stops it with `ErrSourceShapeChanged`. None of these reaches the swap: the verifier compares source and shadow rows by column name, and the cutover gate re-fingerprints the source and refuses any changed column as `cutover-schema-drift` ([refusal-classes.md](refusal-classes.md#shadow-operation-refusals-keyed-on-refusalcause)), after which the shadow is dropped and the change re-planned. |
 | A concurrent index build whose name is occupied by an invalid index, or whose table carries quarantined debris | The build refuses before running and never drops the occupant: an invalid entry is either abandoned debris or someone's build still in progress, and a drop by name cannot tell them apart. The refusal is typed by proof (`invalid-index-abandoned`, `invalid-index-build-in-flight`, `invalid-index-other-table`, `invalid-index-not-droppable`, `invalid-index-builder-unobservable`), and a table carrying an entry a previous recovery quarantined (`pgsprite_abandoned_<oid>`) but did not drop is refused the same way, so debris never goes quiet. The proven removal — `executor.RebuildAbandonedIndex`, or `executor.DropAbandonedIndex` to remove the leftover without running the build; both library-only today, no CLI verb — takes a bounded `SHARE UPDATE EXCLUSIVE` lock on the table (which any running concurrent index command holds), re-verifies the entry by OID, renames it to an identity-derived name and drops that, all within one shared budget; it refuses a visible in-flight build, another table's entry, or an entry the server will not drop concurrently (a partitioned table's index, an index partition, a constraint's index), and reports a lock budget rather than wait behind a build it cannot see. From the CLI, the [runbook](invalid-index-recovery.md) applies. |
-| A copy-and-swap statement that drops a column and adds one of the same name | Like any drop, it runs only under `--allow-destructive`; without it the statement is refused with `destructive-change` and the `alter-column-type-in-place` guidance before the shadow build starts. Under the acknowledgement, the re-added column is a new column: every row holds its default (or NULL), as under a direct `ALTER TABLE`, and the dropped column's values are not copied into it ([copy-and-swap D1](copy-and-swap-design.md#d1--no-durable-scratch-database)). Three such re-adds have no default to take or are the key the copy is driven by, and the shadow build refuses them with `shadow-statement-readd` inside its own transaction, before any shadow, fence, or checkpoint row exists: re-adding the primary key column, re-adding a source identity column, and re-adding a column that refuses NULL (by its own `NOT NULL` or its domain's) with no default of its own or on its type. The remedy depends on which: a primary key is refused whatever the new column declares, so a default does not help — change the key in place with `ALTER COLUMN TYPE`; a source identity column is refused whatever the new column declares too — drop it in one schema change and add the new column in the next; any other column runs once the re-added column has a default (its own, or its domain's). A nullable, defaulted, or generated re-add builds and runs. A re-add declared as an identity or `serial` builds, copies, and then stops at the swap with `cutover-swap-mismatch`, leaving the shadow and the hold on the source for `--restart`, as any new identity or sequence-owned column added through copy-and-swap does. The refusal covers only these named re-add shapes: a new column the server cannot fill in other ways — a `CHECK` that rejects the column's default or NULL, a `NOT NULL` column added without a drop beside a retype, or the key replaced under a new name — still builds and stops at the first chunk or the copier's shape check, failing closed; a first-chunk stop is the server's class `23` error, a data fault, so the run abandons itself and the verdict is `failed` with `code: data-fault`. A run checkpointed by an earlier build for a drop-and-re-add statement no longer resumes (`resume-row-mismatch`, because the copy list it recorded names the re-added column); abandon it and run again rather than retrying the resume. |
+| A copy-and-swap statement that drops a column and adds one of the same name | Like any drop, it runs only under `--allow-destructive`; without it the statement is refused with `destructive-change` and the `alter-column-type-in-place` guidance before the shadow build starts. Under the acknowledgement, the re-added column is a new column: every row holds its default (or NULL), as under a direct `ALTER TABLE`, and the dropped column's values are not copied into it ([copy-and-swap D1](copy-and-swap-design.md#d1--no-durable-scratch-database)). Three such re-adds have no default to take or are the key the copy is driven by, and the shadow build refuses them with `shadow-statement-readd` inside its own transaction, before any shadow, fence, or checkpoint row exists: re-adding the primary key column, re-adding a source identity column, and re-adding a column that refuses NULL (by its own `NOT NULL` or its domain's) with no default of its own or on its type. The remedy depends on which: a primary key is refused whatever the new column declares, so a default does not help — change the key in place with `ALTER COLUMN TYPE`; a source identity column is refused whatever the new column declares too — drop it in one schema change and add the new column in the next; any other column runs once the re-added column has a default (its own, or its domain's). A nullable, defaulted, or generated re-add builds and runs. A re-add declared as an identity or `serial` is refused in the same transaction with `shadow-statement-new-sequence` (next row). The refusal covers only these named re-add shapes: a new column the server cannot fill in other ways — a `CHECK` that rejects the column's default or NULL, a `NOT NULL` column added without a drop beside a retype, or the key replaced under a new name — still builds and stops at the first chunk or the copier's shape check, failing closed; a first-chunk stop is the server's class `23` error, a data fault, so the run abandons itself and the verdict is `failed` with `code: data-fault`. A run checkpointed by an earlier build for a drop-and-re-add statement no longer resumes (`resume-row-mismatch`, because the copy list it recorded names the re-added column); abandon it and run again rather than retrying the resume. |
+| A copy-and-swap statement that gives the table a new identity or `serial` column | The swap carries only the source's sequences to the live table — each re-owned by the shadow column that kept its source column's name — so a sequence the gated statement mints on the shadow (`ADD COLUMN … GENERATED … AS IDENTITY`, `ADD COLUMN … serial`, `ALTER COLUMN … ADD GENERATED … AS IDENTITY` on a kept column, or a dropped column re-added as one) has no source counterpart to carry. The shadow build refuses such a statement with `shadow-statement-new-sequence` inside its own transaction, after the statement has run on the empty shadow and the shadow's columns and sequence-ownership edges have been read, so no shadow, fence, or checkpoint row exists; the source's own serial and identity columns are not new sequences and pass. From the CLI these statements are routed before the build reaches them — a new identity or `serial` column is a volatile-default `ADD COLUMN`, which the copy-and-swap backend does not carry yet (`backend-unavailable`), and `ADD GENERATED` on an existing column is `unsupported-operation` — so the build-time refusal is the library path's guard (`schemachange.Runner`, `BuildShadow`) and the engine's own defence should a route later admit them. Declare the column plain in this schema change and add the identity, or attach the sequence, in a later one that the server applies in place. |
 
 ## `USING` expressions the copy reproduces