fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores by robinnsc · Pull Request #384 · ExtendDB/extenddb

robinnsc · 2026-10-05T02:45:34Z

What

PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. Three commits:

1. fix(postgres): back up and restore tables with a sort key (unchanged from the first version of this PR)

create_backup and restore_table_from_backup checked for a column named sk, which data tables do not have (they use sk_s, sk_n, sk_b). Backups of every table with a sort key lost their sort keys, and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item and writes it with a plain INSERT. Backups written by earlier binaries carry the whole item in item_data and restore correctly.

2. fix(backup): restore indexes and throughput, bound memory, recover abandoned restores

What a restore now carries across, from a new per-backup table definition:

global and local secondary indexes, with key schemas, projections, and GSI throughput, filled during the copy;
billing mode and provisioned throughput (it was always 5/5);
table class, SSE specification, and on-demand throughput.

Streams, TTL, tags, and deletion protection are not restored, as on the service.

Catalog 0.0.4:

crates/storage-postgres/migrations/003_backup_definitions.sql and the SQLite schema add backup_definitions, one row per backup. It holds the definition as JSON in the wire's shape behind a version marker; see crates/storage/src/backup_definition.rs.
A backup taken before the upgrade has no row and restores as before: keys and items, no secondary indexes, and 5/5 throughput for a provisioned table.
The migration runner writes the compiled catalog version after every walk (SET_CATALOG_VERSION_SQL), the same change fix(backup): report point-in-time recovery as unsupported #335 makes. Without it, a replayed earlier migration can leave the stored version behind the schema.

Consistency:

PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken. It locks the source data table ACCESS SHARE before releasing the row.
- UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read.
- A GSI whose catalog row exists but whose data table UpdateTable has not built yet is left out, along with any attribute definition only it used.
SQLite CreateBackup reads the table row, its definition, and its items in one write transaction.
DeleteBackup removes a backup's rows and marks it DELETED in one transaction. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException.
DeleteTable on a table that is being restored returns ResourceInUseException, as the service does for a table it is still creating.

Memory and concurrency:

Backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON, whichever comes first.
SQLite restores commit per batch, so other writers wait for a batch, not the whole restore. SQLite backups still hold the write lock for their duration, which is what makes them consistent.
PostgreSQL runs up to four restores at once per process. A fifth waits up to 30 s and is then refused with LimitExceededException.

Failure and crash:

A failed copy removes its target by table id, synchronously:
- The target is claimed CREATING → DELETING in one statement, so removal and the flip to ACTIVE cannot both happen.
- Its data and index tables are dropped, then the catalog row is deleted.
- If a drop fails, the row stays DELETING and the control plane finishes the job.
PostgreSQL: a restore holds a session advisory lock keyed by account and target name, from before the target is created until the copy ends. The lock lives on a dedicated connection with idle_session_timeout disabled. The control-plane pass removes targets that are CREATING with no scheduled transition, are older than 60 s, and whose lock is free.
SQLite: the same targets are removed at startup, before the server takes requests. This assumes one server process per database file, which the backend's in-process write lock already requires.

3. fix(backup): report RestoreSummary from DescribeTable, refuse DeleteBackup during a restore

The same 0.0.4 migration also adds table_restores: one row per restored table, with the source backup ARN and the restore time. It is not a foreign key to backups, so the summary keeps naming a backup that is deleted later, as on the service.
DescribeTable reports RestoreSummary for a restored table. RestoreInProgress is true while the table is CREATING and false once it is ACTIVE. The restore response and later DescribeTable calls carry the same RestoreDateTime.
DeleteBackup returns BackupInUseException (HTTP 400) while a restore from that backup is still running, and succeeds once the restore is done.
- On PostgreSQL, DeleteBackup locks the backup row FOR UPDATE and the restore holds it FOR SHARE while it records itself.
- On SQLite, both run under the write lock.
New error types: DynamoDbError::BackupInUseException and StorageError::BackupInUse.

Refused rather than restored wrongly:

backups of tables with vector indexes, as before;
backups of multi-part-key tables: a preview behind enable_multipart_keys that the item read and write paths address only by their first HASH and RANGE attribute;
table definitions with a version this build does not know.

RestoreTableFromBackup now validates TargetTableName.

Docs: the upgrade manual has a 0.0.4 section, and the version literals in the install and quickstart guides are updated.

Why

No issue filed. This is the 1.0 readiness review's P0-2, which found:

PostgreSQL backups of composite-key tables could not be restored.
PostgreSQL and SQLite restores dropped GSIs, LSIs, throughput, table class, and encryption settings.
Restores held every item in memory.
A crash mid-restore left the target CREATING with its name taken.

The backup-store design in #348 replaces this path. Until it lands, this is the path users have.

Testing done

tests/test_backup_restore_fidelity.py (7 tests, any backend):

Hash-only tables with S, N, and B keys, and composite tables with S, N, and B sort keys. Key schema, attribute definitions, and every item are compared attribute for attribute, by Scan and by GetItem.
A provisioned table with four GSIs and an LSI:
- projections ALL, INCLUDE, and KEYS_ONLY; S and N hash keys; N and B range keys; sparse items;
- checks table and GSI throughput, IndexStatus, and every index's contents;
- checks projected attribute sets against the projection definitions, not just against the source table;
- checks ordered, ranged index reads in both directions.
RestoreSummary is checked on the restore response and on the later DescribeTable.
The tests wait for backups and restores the way the service needs, so they are valid against DynamoDB. I have not run them against the service.
7/7 on PostgreSQL and on SQLite. The index test fails against commit 1 alone.

crates/storage-postgres/tests/backup_restore.rs (15 tests, added to the PostgreSQL storage-level CI step):

Old backup formats: the pre-fix row shape, and a pre-0.0.4 backup after migration 003 is replayed twice.
Failure handling: failed-copy cleanup under a non-zero control-plane delay; duplicate rows; the multi-part refusal.
Restored definition: indexes and throughput; table class, SSE, and on-demand throughput; more than 1,000 rows including 300 KB items.
Abandoned-restore sweep: inside the grace period, with the lock held, and unowned; a restore into a name whose lock is held.
Concurrency: the backup barrier against an uncommitted UpdateTable (fails on the previous approach); a half-built GSI; DeleteTable during a restore.
RestoreSummary and DeleteBackup: no summary on an ordinary table; in progress while CREATING; done when ACTIVE; the delete refused, then allowed; the summary kept after the backup is deleted.
The tests run one at a time, and their scratch databases are dropped even when a test panics.

crates/storage-sqlite/src/backup.rs restore_tests (10):

Indexes and throughput; the startup sweep; failed-copy cleanup; the multi-part refusal; table class, SSE, and on-demand throughput.
730 rows across batch boundaries. Physical rows and index rows must equal what the write path produced, including the orderable N encoding.
A database with the exact 0.0.3 schema (testdata/schema_0_0_3.sql) holding a backup in the 0.0.3 row format.
Byte-bounded batch cuts, DeleteTable during a restore, and the RestoreSummary and DeleteBackup cases above.

By hand, against servers built from this branch:

kill -9 during a 100,000-item restore:
- PostgreSQL: the target was removed about 60 s later.
- SQLite: the target was removed at the next startup.
- On both, the name could be restored into again.
Memory: 100,000 items of 4 KB (about 400 MB of data) peaked at 87 MB server RSS during the backup and 109 MB during the restore. 150 items of about 2.9 MB of JSON each (435 MB) peaked at 147 MB and 168 MB with MALLOC_ARENA_MAX=1; glibc's per-thread arenas otherwise retain freed memory and RSS reads higher.
40 restores with a DeleteTable fired at each: every restore completed and no target was left behind, on both backends.
PutItem against another table on SQLite during a 60,000-item restore: p50 137 ms, max 276 ms.
DeleteBackup during a 60,000-item restore (SQLite) and a 100,000-item restore (PostgreSQL) returned BackupInUseException 400, and the restore completed.
A c178814 SQLite database with backups was upgraded by extenddb migrate (0.0.3 → 0.0.4) and its backups restored.

Full pytest (tests/, import/export excluded as in CI) and the comprehensive suite (tests/python), on PostgreSQL and SQLite, against this branch and main:

No test fails on this branch that passes on main.
The comprehensive suite passes 331/331 on both backends.
TestGSIOnHashOnlyBaseTable pagination tests fail intermittently under parallel load on both builds and pass alone.
I ran pytest directly, not through devtools/run-tests, so the CLI lifecycle and GSI-queue suites errored the same way on both builds. The CLI migration suites pass when run with a PostgreSQL connection string (7/7).

cargo fmt --all -- --check
cargo +1.97.0 clippy --all-targets -- -D warnings   # the CI toolchain
cargo test --workspace                              # 1,253 passed
cargo +1.88.0 check --workspace --locked
EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test --release -p extenddb-storage-postgres \
  --test vector_control_plane --test backup_restore  # 42 + 15 passed
tests/rust: backup 18 passed, restore 14 passed (PostgreSQL)

Review: five rounds of independent adversarial review, two reviewers each (one per backend). Every round's findings were fixed or are listed below.

Not changed here:

Vector index restore is still refused.
If only a restore's lock connection dies while its copy continues (for example, an operator terminates that backend), the PostgreSQL sweep can claim the target after 60 s. The restore then fails with ResourceNotFoundException; the claim and the activation cannot both succeed, so no wrong table results.
Two SQLite server processes on one database file: the second's startup sweep could remove a restore target the first is still copying. fix(sqlite): allow one extenddb serve per database file #391 makes extenddb serve take an exclusive lock on the file so a second server cannot start.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic)
I have added or updated tests for new functionality
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk
format, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #348 (backup and restore design) describes the target this moves toward. The on-disk change is one new catalog table; there are no wire, trait, auth, or CLI changes.

Breaking changes

Catalog 0.0.4. Every PostgreSQL and SQLite deployment must run extenddb migrate; the server refuses to start against 0.0.3. fix(backup): report point-in-time recovery as unsupported #335 also claims 0.0.4 and migration 003, so whichever of the two merges second must renumber to 0.0.5 and 004.
RestoreTableFromBackup refuses backups of multi-part-key tables, which previously restored with the wrong layout.
DeleteTable on a table being restored returns ResourceInUseException.
DeleteBackup on a backup being restored from returns BackupInUseException.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

CreateBackup decided whether a table had a sort key by looking for a column named `sk`. Data tables name their sort key columns by type (`sk_s`, `sk_n`, `sk_b`), so the check was false for every table with a sort key: the backup recorded no sort key, and RestoreTableFromBackup then failed its first insert on the NOT NULL sort key column. The failure came after the target had been created CREATING with no scheduled transition, so the target stayed CREATING forever and its name could not be reused. `item_data` holds the whole item, key attributes included, so neither side needs the physical key columns: - CreateBackup records `pk` and `item_data` only. - RestoreTableFromBackup derives every key column from the item (all HASH parts into `pk`, each RANGE part into its typed column) and writes it with a plain INSERT inside one data transaction. Two backup rows for one key fail the restore instead of collapsing into one item. - The restored table reports its item count and size when it turns ACTIVE, instead of 0 bytes until the size worker runs. - The flip to ACTIVE happens only from CREATING, so a DeleteTable issued while the copy runs is not undone. - If the copy fails, the target is removed by table id, synchronously, whatever the control-plane delay, and the error is returned. A failed restore leaves no CREATING or DELETING table and frees the name. Backups written by earlier binaries already carry the full item in `item_data`, so they restore correctly with this change; no migration is needed. Tests: - tests/test_backup_restore_fidelity.py round-trips a hash-only table and S, N, and B sort-key tables and compares key schema, attribute definitions, and every item attribute for attribute, by Scan and by point read. It waits for the backup to be AVAILABLE so it also holds against the service. On main the three composite-key cases fail on PostgreSQL; all four pass on PostgreSQL and SQLite with this change. - crates/storage-postgres/tests/backup_restore.rs: a backup in the pre-fix row shape restores with its sort keys and size; a restore that fails mid-copy removes its target under a non-zero control-plane delay; a backup with two rows for one key fails; a two-HASH, two-RANGE key backup restores every key part. Added to the PostgreSQL storage-level CI step. Readiness assessment P0-2 (sort-key defect). GSI and LSI restore follow separately. Signed-off-by: Scott Robinson <robinnsc@amazon.com>

…andoned restores Follows the sort-key fix. Restores on PostgreSQL and SQLite now reproduce what the service reproduces, and the copy is bounded and crash-safe. Restored, from a new per-backup table definition: - global and local secondary indexes, with their key schemas, projections, and GSI throughput, filled during the copy; - billing mode and provisioned throughput (was always 5/5); - table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions`, one row per backup holding that definition as JSON in the wire's shape behind a version marker (`extenddb_storage::backup_definition`). Backups taken before the upgrade have no row and restore as before: keys and items, no secondary indexes, 5/5 for a provisioned table. The migration runner now writes the compiled catalog version after every walk, as #335 also does, so a replayed earlier migration cannot leave the version behind the schema. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and locks the source data table before releasing it. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose catalog row exists but whose data table is not yet built is left out, with any attribute definition only it used. - SQLite CreateBackup reads the table, its definition, and its items in one write transaction. - DeleteBackup removes a backup's rows and marks it DELETED in one transaction; a restore reads from a snapshot (PostgreSQL) or re-checks under the write lock per batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a restore target returns ResourceInUseException, as the service does for a table it is still creating, instead of deleting it out from under the copy. Memory: items are streamed and buffered up to 500 items or 4 MiB of stored JSON, whichever comes first, on both sides of both backends. Measured on PostgreSQL: 100,000 items of 4 KB (about 400 MB), peak server RSS 87 MB during the backup and 109 MB during the restore; 150 items of about 2.9 MB JSON each (435 MB), 147 MB and 168 MB with MALLOC_ARENA_MAX=1 (glibc's per-thread arenas otherwise retain freed memory and RSS reads higher). Concurrency: SQLite restores commit per batch, so other writers wait for a batch rather than the whole restore (PutItem p50 137 ms, max 276 ms during a 60,000-item restore). SQLite backups still hold the write lock for their duration, as they must to be consistent. PostgreSQL allows four concurrent restores per process; a fifth waits up to 30 s and is then refused with LimitExceededException. Failure and crash: - A failed copy removes the target by table id, synchronously: it is claimed CREATING -> DELETING in one statement (so removal and the flip to ACTIVE are mutually exclusive), its data and index tables are dropped, and its row deleted. If a drop fails the row stays DELETING and the control plane finishes the job. - PostgreSQL: a restore holds a session advisory lock keyed by account and target name from before the target is created until the copy ends, on a dedicated connection with idle_session_timeout disabled. The control-plane pass removes targets that are CREATING with no scheduled transition, older than 60 s, and whose lock is free. Verified with kill -9 mid-restore: the target is removed about 60 s later and the name can be restored into again. - SQLite: the same targets are removed at startup, before serving; this assumes one server process per database file, which the backend's in-process write lock already requires. Verified with kill -9. Refused rather than restored wrongly: backups of tables with vector indexes (as before), backups of multi-part-key tables (a preview the item paths address only by their first parts), and definitions with an unknown version. RestoreTableFromBackup now validates TargetTableName. Tests: - tests/test_backup_restore_fidelity.py: hash-only tables with S, N, and B keys; composite keys with S, N, and B sort keys; a provisioned table with four GSIs (ALL, INCLUDE, KEYS_ONLY; S and N hash keys, N and B range keys; sparse) and an LSI, comparing definitions, throughput, IndexStatus, every index's contents, projected attribute sets against the projection definitions, and ordered ranged index reads in both directions. Waits for backups and restores as the service needs. 7/7 on PostgreSQL and SQLite; the index test fails on the previous commit. - crates/storage-postgres/tests/backup_restore.rs (14): legacy row shape, pre-0.0.4 backup after replaying migration 003, failed-copy cleanup under a non-zero control-plane delay, duplicate rows, multi-part refusal, indexes and throughput, table class/SSE/on-demand, >1,000 rows with oversized items, the abandoned-restore sweep (grace, held lock, unowned), a name whose lock is held, the backup barrier against an uncommitted UpdateTable (fails on the previous approach), a half-built GSI, and DeleteTable during a restore. Run one at a time, and scratch databases are dropped even when a test panics. - crates/storage-sqlite/src/backup.rs restore_tests (9): indexes and throughput, the startup sweep, failed-copy cleanup, multi-part refusal, table class/SSE/on-demand, 730 rows across batches with physical row and index-row equality against the write path, a database with the exact 0.0.3 schema (testdata/schema_0_0_3.sql) holding a 0.0.3-format backup, byte-bounded batch cuts, and DeleteTable during a restore. - tests/test_cli_vector_catalog_migration.py covers 0.0.2 -> 0.0.4 and the replay case. Not changed here: RestoreSummary appears only in the RestoreTableFromBackup response and not in later DescribeTable output (pre-existing; needs a persisted column). DeleteBackup during a restore fails the restore with BackupNotFoundException rather than refusing with BackupInUseException. Vector index restore is still refused. Readiness assessment P0-2. Signed-off-by: Scott Robinson <robinnsc@amazon.com>

+    }
+
+    async fn count(engine: &SqliteEngine, table: &str) -> i64 {
+        sqlx::query_scalar(&format!("SELECT COUNT(*) FROM {table}"))


…ackup during a restore RestoreSummary was only fabricated on the RestoreTableFromBackup response; DescribeTable never reported it. And DeleteBackup during a restore deleted the backup and failed the restore, where the service refuses with BackupInUseException. Catalog 0.0.4 (same migration as the previous commit; nothing extra to migrate) adds `table_restores`: one row per restored table with the source backup ARN and the restore time. Not a foreign key to `backups`, because the summary keeps naming a backup that is deleted later, as on the service. - PostgreSQL and SQLite DescribeTable report RestoreSummary for a restored table, RestoreInProgress true while it is CREATING and false once ACTIVE. The restore response and later DescribeTable calls carry the same RestoreDateTime. Tables restored before the upgrade have no row and no summary. - DeleteBackup returns BackupInUseException (HTTP 400) while a table still CREATING names the backup. On PostgreSQL the backup row is locked FOR UPDATE by DeleteBackup and FOR SHARE by the restore while it records itself; on SQLite both run under the write lock. Once the restore is done the backup can be deleted. - New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Tests: restore_summary_and_backup_in_use on both backends (summary absent on an ordinary table, in progress while CREATING, done when ACTIVE, delete refused then allowed, summary kept after the backup is deleted), and the wire fidelity test checks RestoreSummary on the response and on DescribeTable. By hand, DeleteBackup during a 60,000-item (SQLite) and 100,000-item (PostgreSQL) restore returned BackupInUseException 400 and the restore completed. Signed-off-by: Scott Robinson <robinnsc@amazon.com>

robinnsc requested review from LeeroyHannigan, amrith, c33howard, jcshepherd, pdf-amzn and yesyayen as code owners October 5, 2026 02:45

github-advanced-security Bot found potential problems Oct 5, 2026

View reviewed changes

Comment thread crates/storage-postgres/src/backup_engine.rs Fixed

Comment thread crates/storage-postgres/src/backup_engine.rs Fixed

Comment thread crates/storage-postgres/src/backup_engine.rs Fixed

robinnsc changed the title ~~fix(postgres): back up and restore tables with a sort key~~ fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores Oct 5, 2026

github-advanced-security Bot found potential problems Oct 5, 2026

View reviewed changes

robinnsc mentioned this pull request Oct 6, 2026

fix(sqlite): allow one extenddb serve per database file #391

Open

8 tasks done

Navigation Menu

fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores - #384

fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores#384
robinnsc wants to merge 3 commits into
mainfrom
fix/backup-restore-fidelity

robinnsc commented Oct 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

robinnsc commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing done

Checklist

Breaking changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robinnsc commented Oct 5, 2026 •

edited

Loading