Skip to content

fix(streams): derive shard ids from the table id and label streams to the millisecond - #383

Open
robinnsc wants to merge 1 commit into
mainfrom
fix/pg-stream-shard-ids
Open

robinnsc wants to merge 1 commit into
mainfrom
fix/pg-stream-shard-ids

Conversation

@robinnsc

@robinnsc robinnsc commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

What

Stream shard ids on PostgreSQL and SQLite are built from the table id instead of the table name, and stream labels on all three backends carry milliseconds.

  • crates/storage-postgres/src/stream_engine.rs, crates/storage-sqlite/src/stream.rs: shardId-{table_id}-{i:016} in place of shardId-{table_name}-{i:016}. The table id is a UUID, so new shard ids are 61 characters. MongoDB already builds its shard ids this way.
  • stream_label is now YYYY-MM-DDThh:mm:ss.sss, the shape the service uses: to_char(clock_timestamp(), 'YYYY-MM-DD"T"HH24:MI:SS.MS') on PostgreSQL (stream_engine.rs, update_table.rs), strftime('%Y-%m-%dT%H:%M:%f','now') on SQLite (stream.rs, update_table.rs), and format_stream_label on MongoDB (table_engine.rs).
  • Existing shard rows and labels are not rewritten. They keep their ids and ARNs and keep working, because shard ids and labels are looked up as opaque values. Shards created from now on get the new form, including on a pre-existing table that enables a stream for the first time.
  • Docs: docs/manuals/02-design-guide.md (shard id format), docs/design/13-storage-mongodb.md (label format).

Why

No issue filed. Found in the 1.0 readiness review (P0-5).

stream_shards is keyed by shard id alone, and the id contained only the table name, so any second stream-enabled table under one name collided:

  • PostgreSQL keeps a deleted table's shard rows. DeleteTable followed by CreateTable with the same name and a stream failed with InternalServerError every time (duplicate key value violates unique constraint "stream_shards_pkey"). Delete-and-recreate under one name is routine in test harnesses and works in DynamoDB.
  • On PostgreSQL and SQLite, a second account creating a stream-enabled table with a name another account already uses failed the same way.
  • A table name longer than about 40 characters gave a ShardId over the 65 characters the AWS SDKs accept, so the SDK rejected DescribeStream and GetShardIterator calls carrying it client-side. [Bug] GetShardIterator/GetRecords rejected client-side by AWS SDKs for tables named 3–6 characters long #247 fixed the lower bound for short names; this fixes the upper one.

With recreate working, a second defect became reachable. Labels had one-second resolution, so a table deleted and recreated within the same second (control_plane_delay_seconds of 0 makes this easy) got the same stream ARN as the old table, and DescribeStream on the old ARN resolved to the new table's stream. On MongoDB, where recreate already worked, the new test hit this in 1 of 3 runs on main.

Testing done

New tests/test_stream_table_reuse.py:

  • recreate under the same name: the new stream ARN differs, the new stream carries only the new table's records, and the old ARN returns ResourceNotFoundException;
  • two accounts, one table name: each stream carries only its own account's records;
  • a 221-character table name: every ShardId is within 28..65 and GetShardIterator and GetRecords work through boto3;
  • disable then re-enable a stream on one table: records written after re-enabling are delivered.

On main, the first three fail on PostgreSQL and the second and third fail on SQLite. On this branch all four pass on PostgreSQL (control_plane_delay_seconds 0, 0.05, and 0.25), SQLite, and MongoDB, together with test_streams.py (20 passed, 1 xfailed on each).

The file is ExtendDB-only (module skip when EXTENDDB_TEST_ENDPOINT is unset), like test_streams.py: it signs Streams requests against the ExtendDB endpoint, and the two-account case needs the management API.

Full PostgreSQL pytest run (tests/, import/export excluded as in CI) and the comprehensive suite (tests/python), against servers built from this branch and from main: no test fails on the branch that passes on main, and the comprehensive suite passes 331 of 331 on both. I ran pytest directly rather than through devtools/run-tests, so the CLI lifecycle and GSI queue suites, which need the runner's PostgreSQL connection string and server restarts, errored the same way on both builds.

cargo fmt --all -- --check
cargo +1.97.0 clippy --all-targets -- -D warnings   # the CI toolchain
cargo test --workspace                       # 1,222 passed
cargo +1.88.0 check --workspace --locked

Not changed here: PostgreSQL never removes a deleted table's shard rows (four per stream). They no longer block anything. Removing them safely needs a delete marker in the data database, because the catalog and the data database are separate and a sweep that infers "deleted" from a missing catalog row races CreateTable.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. Shard ids and stream labels are opaque server-issued values; existing rows are left as they are, and no trait, schema, or CLI change.

Breaking changes

New streams get ShardIds and StreamLabels in a different shape. Both are opaque in the DynamoDB API, and the new label shape is the service's own. A client that parsed the table name out of a ShardId, or expected a label with no fractional seconds, would see the difference. Existing streams are unchanged.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

… the millisecond

Shard ids were `shardId-<table name>-<n>`, while `stream_shards` is keyed by
shard id alone. Any second stream-enabled table under one name collided:

- PostgreSQL keeps a deleted table's shards, so DeleteTable followed by
  CreateTable with the same name and a stream failed with
  InternalServerError every time.
- On PostgreSQL and SQLite, a second account creating a stream-enabled
  table with a name another account already uses failed the same way.
- A table name over about 40 characters produced a ShardId longer than the
  65 characters the AWS SDKs accept, so DescribeStream and GetShardIterator
  calls carrying it were rejected client-side.

Use the table id (a fresh UUID per table) in place of the name on
PostgreSQL and SQLite, as the MongoDB backend already does. New shard ids
are 61 characters. Shard rows that already exist keep their ids and keep
working; shards created from now on, including on a pre-existing table
that enables a stream for the first time, get the new form.

With recreate working, a second defect becomes reachable: stream labels had
one-second resolution on all three backends, so a table deleted and
recreated within the same second got the same stream ARN, and the old ARN
resolved to the new table's stream. Labels now carry milliseconds
(`2026-10-05T01:54:50.312`), the shape the service uses. Existing labels
are unchanged. On MongoDB, where recreate already worked, this reproduced
in one of three runs of the new test.

PostgreSQL still never removes a deleted table's shard rows (four per
stream). That is unchanged by this commit.

tests/test_stream_table_reuse.py covers recreate under the same name
(including that the old stream ARN no longer resolves), two accounts with
one name, a long name, and disabling then re-enabling a stream. On main the
first three fail on PostgreSQL and the second and third fail on SQLite; all
pass on PostgreSQL, SQLite, and MongoDB with this change.

Readiness assessment P0-5.

Signed-off-by: Scott Robinson <robinnsc@amazon.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant