Repository navigation
A join on a declared reference joins on hidden integer keys, not the UUID or text it names - #1159
Merged
Merged
Conversation
…UUID or text it names (#1144) Each reference the sources file names, whose referenced key is a UUID, a text, several columns or a sparse integer, gets a dictionary of its key's values (a number given once) and a hidden BIGINT on both tables. A join on the reference's columns, as a conjunct of an ON or a WHERE, joins on the two numbers. The statement first checks, in its own transaction, that every row it reads numbers of is numbered, and runs as written otherwise. Numbers are given on the writer alone: in a stream batch's and a writer's poll's transaction, on the writer's turn for the rest, and from the dictionary before a side table's swap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…keys # Conflicts: # .claude/rules/duck-rewrite.md # src/duck/plan.ts # tests/integration/duck-oracle.itest.ts
tagspace
marked this pull request as ready for review
November 18, 2026 17:52
…erence's hidden key and a timestamptz's bucket side by side
hiddenColumns tells the two apart ("number", "bucket"). The stream's updateCarried sets both, the planner hands the
rewrite both lookups, and the oracle syncs both its reference and its bucket tables. The reference-key integration
test gives pings a timestamptz, and checks both hidden columns after the first sync, each poll and each stream write.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tagspace
marked this pull request as draft
November 18, 2026 18:23
…m the store, a dictionary belongs to its key's types Whether a row's number is still right was decided from what the numbering's last turn had found (`present`, `made`). A swap or a compaction cleared that until the next turn, and a stream update of a key in that window kept its old number: a FULL JOIN answered wrong, and stayed wrong after catch-up. And a dictionary was named for its key's table and columns alone: after a type change on both sides (uuid or bigint to text), new values were cast into the old dictionary and joined keys they are not equal to, and a value the old type could not hold stopped the table's sync. - An update in place reads the table's hidden columns in its own transaction and unnumbers each whose key columns it sets (one it cannot place, always). A write's numbering reads the store too (`writable`). - A dictionary is named for its key's types, and a hidden column's name carries its dictionary's. A dictionary is dropped only once no column holds its numbers. A row is numbered only where its key's columns and the dictionary are of the families' types, so no cast can fail. A side table's numbering never fails the sync. - Numbering joins on the key's own columns, never on rowid: DuckDB 2.0.0's join filter on a row id corrupted the heap (an abort in about 1 run in 4 of a stream batch; Guard Malloc showed the read). - The schema is rebuilt whenever the numbering's references differ from those it was last built with. - The tests read the store on a connection of their own, never the sync's writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tagspace
marked this pull request as ready for review
November 18, 2026 20:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From Claude (Opus 5.5)
Closes #1144
The decision
references:entry whose referenced key is a UUID, a text, several columns, or a sparse integer (range at least 64 times its rows) gets numbers. Mason keeps a dictionary of the key's values (_mason.ref_keys_…), each value numbered once, and a hidden BIGINT on both tables. A dense integer key (a serial) gets none. Whetherreferences:is written is unchanged:mason addstill writes every declared foreign key.referenceJoins) writes an equality of a reference's key columns as the equality of the two numbers. It does so only for a top-level conjunct of a JOIN's ON or of a WHERE. Every column of the key must be compared, each a column of a table of the same query, by Postgres's own=for uuid, text/varchar or the integers. A NULL key is numbered -1 on the referenced side and 0 on the other, so it never matches. That gives FALSE where=gives NULL, which only those two places treat alike. A*over a synced table, USING, NATURAL, an outer query's column,=as a value, NOT and OR all still read the keys.=does, with no check needed. The cost: the dictionary keeps a deleted key's number for ever.SessionRunner.numbersHold). Otherwise it runs the plan without numbers, which is planned beside it (PlannedSql.numbers.plain). If the check cannot tell, it fails closed the same way. The check could not go inside the statement: DuckDB evaluates a WHERE only on rows that reach it, and an inner join whose only pair was an unnumbered row answered nothing and raised nothing. Every way a statement ends rolls the transaction back.How it was tested
tests/duck-reference-keys.test.ts(unit, PGlite as the truth). It covers inner, LEFT, RIGHT, FULL and comma-WHERE joins, a key of two columns, and a table that references itself. The data has orphans, NULL keys, a referenced key held twice, and a key of two columns with a NULL in it. It checks that each of these reads the numbers. It checks that the shapes that must read the keys do not. It checks that an unnumbered row runs the statement on the keys. And it checks that a statement that fails or is cancelled leaves the session usable.scripts/fails-without.ts 234cd9b3 tests/duck-reference-keys.test.tsreports PROVED (2 tests).tests/integration/duck-reference-keys.itest.tscovers uuid, text-from-varchar, two-column, sparse bigint, dense serial and self-reference keys. It runs through the first sync, the poll (inserts, a key updated on either side, set to NULL, a parent deleted and inserted again), the stream, a re-sync after a column change, and a store synced before this change (numbered in place at the next start, no table synced again). All 4 pass against Postgres 18.rparentandrchild. Their keys repeat and include NULLs, a case pair, a trailing blank,'', and a composed é beside a decomposed é. Both mutation rounds change keys.IT_ORACLE_ONLY="reference: "asked 208 times with the shapes and found 0 differences. 152 of the 204 asks overrchildran on numbers. The rest are the four shapes that must read the keys.npm run typecheckis clean.npm run checklocally: the only failures are macOS-only (SIGSTOP/ps,xdg-open, the socket path length, case-insensitive file names).duck-spill-ceilingfails the same way at main on this machine.Measured
Synthetic data: 100 M orders, and 5 M customers for each key kind, synced from a scratch Postgres 18 on the same laptop (10 cores, 32 GiB RAM, NVMe disk). The query is revenue by order month and signup year over the join (72 rows). "Before" is the same build with no
references:, which is what main does.cus_+ 16 hex)After the first review (b3344d3)
The review found three wrong answers or stuck states at b3344d3. Each is fixed by the rule above, and each has a test that fails at b3344d3 (
tests/integration/duck-reference-keys-stale.itest.ts;scripts/fails-without.ts b3344d3areports PROVED, 6 tests):20001 20000 19013where Postgres answers20001 20000 18923.'{<uuid>}','01000003',' 2000006'and'+3000009'join keys they are not equal to (62 rows where Postgres has 60).'not a uuid'). At b3344d3 the tables never sync again (Conversion Error … to UUID). Now they sync, and each join answers as Postgres does.Two more found on the way:
UPDATE … FROM (…) s WHERE t.rowid = s.raborted the process in about 1 run in 4 of the compaction test, at b3344d3 too. Guard Malloc showed DuckDB 2.0.0's join filter on the row id reading past the scan's vector (BloomFilterExecutor::FilterSelection). The update now joins on the key's own columns. 0 aborts in 3 runs under Guard Malloc and 10 under load, where the join on the row id aborted in 2 runs of 2 under Guard Malloc.Merged with main's 15-minute buckets (#1146): a table with a reference and a timestamptz holds both hidden columns, and
hiddenColumnstells them apart. The reference-key tests givepingsa timestamptz and check both after the first sync, each poll and each stream write.The sweep and the probe, at 70fef9e:
type matrix: reference:statements: 208 asked, 0 differ, 152 of 204 ran on numbers. With the bucket statements too: 1,172 asked, 0 differ.Left out
EXISTS (… WHERE o.customer_id = c.id)semi-join) still reads the keys. A name could be shadowed in the subquery.__mason_ref_and 12 hex digits). This Mason does not know them: they are never read or dropped. No release made them.SPARSE_RANGE(64). It is a choice, not a measured edge.🤖 Generated with Claude Code