Skip to content

A join on a declared reference joins on hidden integer keys, not the UUID or text it names - #1159

Merged
tagspace merged 4 commits into
mainfrom
issue-1144-reference-keys
Nov 18, 2026
Merged

tagspace merged 4 commits into
mainfrom
issue-1144-reference-keys

Conversation

@tagspace

@tagspace tagspace commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

From Claude (Opus 5.5)

Closes #1144

The decision

  • Each references: entry whose referenced key is a UUID, a text, several columns, or a sparse integer (range at least 64 times its rows) gets numbers. Mason keeps a dictionary of the key's values (_mason.ref_keys_…), each value numbered once, and a hidden BIGINT on both tables. A dense integer key (a serial) gets none. Whether references: is written is unchanged: mason add still writes every declared foreign key.
  • The rewrite (referenceJoins) writes an equality of a reference's key columns as the equality of the two numbers. It does so only for a top-level conjunct of a JOIN's ON or of a WHERE. Every column of the key must be compared, each a column of a table of the same query, by Postgres's own = for uuid, text/varchar or the integers. A NULL key is numbered -1 on the referenced side and 0 on the other, so it never matches. That gives FALSE where = gives NULL, which only those two places treat alike. A * over a synced table, USING, NATURAL, an outer query's column, = as a value, NOT and OR all still read the keys.
  • Why a dictionary, not the referenced row's number (a change from the issue's plan). The issue proposed numbering the referenced rows, with a uniqueness check on the referenced key and a fallback to the plain join for orphans. With a dictionary, a number belongs to a value: two rows' numbers are equal exactly when their keys are. So a key held twice, an orphan's key, and a row deleted and inserted again all answer as = does, with no check needed. The cost: the dictionary keeps a deleted key's number for ever.
  • Exactness while rows are still unnumbered. A statement that joins on numbers runs in a transaction. It first checks that no row of those tables is unnumbered, then runs in the same snapshot (SessionRunner.numbersHold). Otherwise it runs the plan without numbers, which is planned beside it (PlannedSql.numbers.plain). If the check cannot tell, it fails closed the same way. The check could not go inside the statement: DuckDB evaluates a WHERE only on rows that reach it, and an inner join whose only pair was an unnumbered row answered nothing and raised nothing. Every way a statement ends rolls the transaction back.
  • Numbers are given on the writer alone, so no number is given twice. A stream batch and a writer's poll number their rows in their own transaction. An update in place that sets a key column unnumbers that row in the same statement. A first sync, a catch-up and a compaction are numbered on the writer's turn after, 2 M rows a turn. A side table is numbered from the dictionary, read only, before its swap. A key that is new since then is left for the writer.
  • Whether a number is still right is read from the store, never from what the numbering found before. Each write reads the hidden columns its table holds, in its own transaction. A dictionary belongs to its key's table, columns and types, and a hidden column's name begins with its dictionary's. A dictionary is dropped only once no column holds its numbers. A row is numbered only where its key's columns and the dictionary are of the same family's types, so no cast can fail or make two values equal. So a number cannot outlive its key, whatever a swap, a compaction, a timer, a restart or a type change does.

How it was tested

  • tests/duck-reference-keys.test.ts (unit, PGlite as the truth). It covers inner, LEFT, RIGHT, FULL and comma-WHERE joins, a key of two columns, and a table that references itself. The data has orphans, NULL keys, a referenced key held twice, and a key of two columns with a NULL in it. It checks that each of these reads the numbers. It checks that the shapes that must read the keys do not. It checks that an unnumbered row runs the statement on the keys. And it checks that a statement that fails or is cancelled leaves the session usable. scripts/fails-without.ts 234cd9b3 tests/duck-reference-keys.test.ts reports PROVED (2 tests).
  • tests/integration/duck-reference-keys.itest.ts covers uuid, text-from-varchar, two-column, sparse bigint, dense serial and self-reference keys. It runs through the first sync, the poll (inserts, a key updated on either side, set to NULL, a parent deleted and inserted again), the stream, a re-sync after a column change, and a store synced before this change (numbered in place at the next start, no table synced again). All 4 pass against Postgres 18.
  • The oracle has 14 new statements, "type matrix: reference: …", over the new tables rparent and rchild. Their keys repeat and include NULLs, a case pair, a trailing blank, '', and a composed é beside a decomposed é. Both mutation rounds change keys. IT_ORACLE_ONLY="reference: " asked 208 times with the shapes and found 0 differences. 152 of the 204 asks over rchild ran on numbers. The rest are the four shapes that must read the keys.
  • npm run typecheck is clean. npm run check locally: the only failures are macOS-only (SIGSTOP/ps, xdg-open, the socket path length, case-insensitive file names). duck-spill-ceiling fails the same way at main on this machine.

Measured

Synthetic data: 100 M orders, and 5 M customers for each key kind, synced from a scratch Postgres 18 on the same laptop (10 cores, 32 GiB RAM, NVMe disk). The query is revenue by order month and signup year over the join (72 rows). "Before" is the same build with no references:, which is what main does.

key orders' key column the join reads warm, min of 5 cold, 3 runs
UUID 1.60 GiB → 0.31 GiB 0.83 s → 0.83 s 1.07–1.20 s → 0.83–0.88 s
text (cus_ + 16 hex) 1.20 GiB → 0.31 GiB 1.42 s → 0.83 s 1.63–1.71 s → 0.87–0.93 s
sparse 60-bit integer 0.79 GiB → 0.31 GiB 0.82 s → 0.85 s 0.89–0.92 s → 0.84–0.88 s
dense serial (none) 0.40 GiB, the same 0.82 s → 0.85 s 0.86–0.90 s → 0.85–0.89 s
  • First sync of the five tables: 40.6 s → 53.5 s. Numbering finished 0.2 s after it. Synced data: 4.04 GiB → 5.19 GiB, about 312 MiB per reference on the orders.
  • For "cold", 42 GB was read before each run to push the tables out of the page cache. Then the SQL Mason sent ran in a new engine instance. On this NVMe disk a cold read is cheap, so the cold gain is small here. A disk that reads 125 MB/s would spend 4 to 10 s less per query at this size. That was not measured.
  • Warm, a UUID or sparse-integer join runs about as fast on its numbers as on its key. A text key gains (1.42 s → 0.83 s).

After the first review (b3344d3)

The review found three wrong answers or stuck states at b3344d3. Each is fixed by the rule above, and each has a test that fails at b3344d3 (tests/integration/duck-reference-keys-stale.itest.ts; scripts/fails-without.ts b3344d3a reports PROVED, 6 tests):

  • A stream update of a key (a large value left unchanged) right after a compaction or a swap, before the numbering's next turn, kept the old key's number. At b3344d3 a FULL JOIN answers 20001 20000 19013 where Postgres answers 20001 20000 18923.
  • A key's type changed on both sides (uuid to text, bigint to text), under the poll and the stream. At b3344d3, '{<uuid>}', '01000003', ' 2000006' and '+3000009' join keys they are not equal to (62 rows where Postgres has 60).
  • The same change with a value the old type cannot hold ('not a uuid'). At b3344d3 the tables never sync again (Conversion Error … to UUID). Now they sync, and each join answers as Postgres does.

Two more found on the way:

  • A crash. Numbering a batch by UPDATE … FROM (…) s WHERE t.rowid = s.r aborted the process in about 1 run in 4 of the compaction test, at b3344d3 too. Guard Malloc showed DuckDB 2.0.0's join filter on the row id reading past the scan's vector (BloomFilterExecutor::FilterSelection). The update now joins on the key's own columns. 0 aborts in 3 runs under Guard Malloc and 10 under load, where the join on the row id aborted in 2 runs of 2 under Guard Malloc.
  • The schema could miss a reference. The numbering rebuilt the schema only when its own turn changed the references. A swap's own rebuild between two turns could leave it stale, so joins kept reading the keys (seen in 2 runs of about 10). It is now rebuilt whenever the references differ from those it was last built with.

Merged with main's 15-minute buckets (#1146): a table with a reference and a timestamptz holds both hidden columns, and hiddenColumns tells them apart. The reference-key tests give pings a timestamptz and check both after the first sync, each poll and each stream write.

The sweep and the probe, at 70fef9e:

  • The oracle's type matrix: reference: statements: 208 asked, 0 differ, 152 of 204 ran on numbers. With the bucket statements too: 1,172 asked, 0 differ.
  • The timing probe: 2 M orders and 200 k customers (uuid keys), 60 s of stream writes (pgbench, 20 transactions a second, 345 k key updates in all: rows moved in place with a large value unchanged, whole rows, the referenced side's key changed). In each snapshot, the check ran, then both the join on numbers and the join on keys. 1,044 snapshots, all numbered, 0 answers differing. Median times: the check 0.9 ms; an inner join grouped by region 18.5 ms on numbers against 19.1 ms on keys; a FULL JOIN count 6.9 ms against 10.5 ms.

Left out

  • A join where one side is an outer query's column (an EXISTS (… WHERE o.customer_id = c.id) semi-join) still reads the keys. A name could be shadowed in the subquery.
  • The dictionary is never pruned.
  • A store made by an earlier commit of this branch (before 70fef9e) holds hidden columns of the old name (__mason_ref_ and 12 hex digits). This Mason does not know them: they are never read or dropped. No release made them.
  • Nothing tunes SPARSE_RANGE (64). It is a choice, not a measured edge.

🤖 Generated with Claude Code

tagspace and others added 2 commits November 18, 2026 18:26
…UUID or text it names (#1144)

Each reference the sources file names, whose referenced key is a UUID, a text, several columns or a sparse
integer, gets a dictionary of its key's values (a number given once) and a hidden BIGINT on both tables.
A join on the reference's columns, as a conjunct of an ON or a WHERE, joins on the two numbers. The
statement first checks, in its own transaction, that every row it reads numbers of is numbered, and
runs as written otherwise. Numbers are given on the writer alone: in a stream batch's and a writer's
poll's transaction, on the writer's turn for the rest, and from the dictionary before a side table's swap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…keys

# Conflicts:
#	.claude/rules/duck-rewrite.md
#	src/duck/plan.ts
#	tests/integration/duck-oracle.itest.ts
@tagspace
tagspace marked this pull request as ready for review November 18, 2026 17:52
…erence's hidden key and a timestamptz's bucket side by side

hiddenColumns tells the two apart ("number", "bucket"). The stream's updateCarried sets both, the planner hands the
rewrite both lookups, and the oracle syncs both its reference and its bucket tables. The reference-key integration
test gives pings a timestamptz, and checks both hidden columns after the first sync, each poll and each stream write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tagspace
tagspace marked this pull request as draft November 18, 2026 18:23
…m the store, a dictionary belongs to its key's types

Whether a row's number is still right was decided from what the numbering's last turn had found (`present`, `made`).
A swap or a compaction cleared that until the next turn, and a stream update of a key in that window kept its old
number: a FULL JOIN answered wrong, and stayed wrong after catch-up. And a dictionary was named for its key's table
and columns alone: after a type change on both sides (uuid or bigint to text), new values were cast into the old
dictionary and joined keys they are not equal to, and a value the old type could not hold stopped the table's sync.

- An update in place reads the table's hidden columns in its own transaction and unnumbers each whose key columns it
  sets (one it cannot place, always). A write's numbering reads the store too (`writable`).
- A dictionary is named for its key's types, and a hidden column's name carries its dictionary's. A dictionary is
  dropped only once no column holds its numbers. A row is numbered only where its key's columns and the dictionary are
  of the families' types, so no cast can fail. A side table's numbering never fails the sync.
- Numbering joins on the key's own columns, never on rowid: DuckDB 2.0.0's join filter on a row id corrupted the heap
  (an abort in about 1 run in 4 of a stream batch; Guard Malloc showed the read).
- The schema is rebuilt whenever the numbering's references differ from those it was last built with.
- The tests read the store on a connection of their own, never the sync's writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tagspace
tagspace marked this pull request as ready for review November 18, 2026 20:48
@tagspace
tagspace merged commit 9828b97 into main Nov 18, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A join on a declared reference joins on hidden integer keys, not the UUID it names

1 participant