improvement(tables): bump rows version once at commit and serialize same-value unique writes by waleedlatif1 · Pull Request #8335 · simstudioai/sim

waleedlatif1 · 2026-09-26T18:02:33Z

Summary

bump_user_table_rows_version ran as a statement-level trigger inside every row-update statement, so each writer locked its table's user_table_definitions row across the rest of its transaction (executions patch, provenance write and capture, synchronous commit), and concurrent writers to one table queued behind each other. Provenance-only updates bumped it too, even though they don't change the table's content
The UPDATE bump is now a row-level DEFERRABLE INITIALLY DEFERRED constraint trigger AFTER UPDATE OF data, order_key, with a WHEN on real changes. It bumps once per table per transaction at commit, deduped through one transaction-local setting. INSERT/DELETE statement bumps are unchanged. The only readers (the CSV snapshot cache and mount safety) read outside write transactions, so a commit-time bump keeps their read, scan, re-read checks intact
user_table_rows.table_id → user_table_definitions FK becomes DEFERRABLE INITIALLY DEFERRED (catalog-only). The write path updates a row twice in one transaction (data, then the provenance marker, which the rolling-deploy demote trigger requires to be separate), and the second update re-checked the FK, taking a key-share lock on the definition row that the commit-time bump then waited behind. ON DELETE CASCADE stays immediate, and nothing catches this FK's error inline
The trigger swap runs inside an explicit BEGIN, so it stays atomic when an earlier migration in the same deploy used a COMMIT breakpoint
CREATE STATISTICS (dependencies) ON workspace_id, table_id plus ANALYZE, so the planner stops underestimating per-table row counts and serves data @> filters from the tenant GIN
Unique columns are an application-level check, so two concurrent writes of the same value could both pass it and both commit. Every write path now takes transaction-scoped advisory locks (tag user_table_unique_value) before its check: the table's unique lock shared, then an exclusive lock per value, keyed on the same normalization the check uses (so '8' and 8 share a key). Writers of different values, on any column, never wait on each other
- Each transaction takes all its locks at once, in one sorted order. The global order is schema lock, then unique locks, then row-order lock, then definition row
- A json object or array value, or a transaction that would take more than 64 value locks, takes the table's unique lock exclusively instead. Whole-table replace and every import batch always do. A transaction holds at most 1 + 64 unique locks, however many unique columns the table has. 64 is the default per-connection budget of Postgres's shared lock table, so wide schemas and large batches can't exhaust it and fail unrelated queries
- The unique check for updateRow, batchUpdateRows, and single-row filter updates now runs inside the write transaction, under the locks. batchUpdateRows also rejects two updates in one batch that set the same value, and it locks and checks only the unique columns each update changes
- Tables without unique columns take no extra round trips

Type of Change

Improvement

Testing

A/B against staging on identical fresh DBs, real service functions, runs alternated:
- 20 concurrent updateRow writers on one table: 943–1071 → 1585–1606 ops/s, p99 77–107 → ~18 ms, lock wait 521–525 → 4–8 backend-s
- 50 writers: 886–937 → 1398–1450 ops/s, p99 274–292 → 124–159 ms
- Provenance-only writes: 5185–5482 → 7979–8713 ops/s, p99 ~15 → ~4 ms
- Inserts with a unique column and executions-only writes: within noise
- Large single-transaction rewrites are ~20% faster, with ~0.6s of commit-time trigger work at 1M rows
Filter queries on 4,000 tables with skewed sizes: large-table data @> filters 1.7–5.7 → 0.5 ms
Migration-during-writes: 0 missed bumps across 3 runs (without the BEGIN, 40/40 missed)
row-writes.integration.ts: data and reorder edits bump once per transaction, including two tables in one transaction; provenance-only, timestamp-only, no-op, and executions-only writes don't bump; a second writer commits while the first is uncommitted; the snapshot re-keys when a writer commits mid-materialization. These fail on the old trigger
Fresh DB migrate plus replay, secret-provenance and table integration suites, table unit suites, type-check, lint, check:audits, and check:migrations origin/staging pass
Unique races (deterministic, driven by pg_locks wait states), all ending with one row:
- same-value single inserts
- '8' vs 8
- batch vs single insert
- updateRow vs insert
- import batch vs insert
- a batch over the value-lock cap vs a per-value writer
- a replace adding the first unique column vs a writer on the older schema (no deadlock)
- a 100-unique-column table holding a single lock
- two updates in one batch setting the same value
Different values don't wait on each other. Each race test fails on the pre-fix code
Unique-column throughput vs staging is within noise: distinct-value inserts, batch inserts, and batch upserts at 10 and 50 writers. updateRow keeps the gains above
Migration is 0390 on top of staging

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-09-26T18:02:36Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 28, 2026 4:04am UTC

cubic-dev-ai

All reported issues were addressed across 9 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-09-26T18:06:57Z

Confidence Score: 5/5

[Critical risk] Restructures row-write locking and adds database migration.

The PR appears safe to merge; no new actionable issue or outstanding previous finding remains.

Summary

The PR defers row-update version bumps until commit and adds transaction-scoped locks around application-level unique checks.

UPDATEs that change row data or order now bump each table’s version once per transaction; provenance-only writes do not.
Unique-value writes serialize before checking for conflicts, with a table-wide lock for large batches and imports.
The latest changes canonicalize JSON object keys for batch duplicate detection and release a test writer when its lock-wait assertion fails.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Begin row-write transaction] --> B[Acquire unique-value or table-wide lock]
  B --> C[Check uniqueness]
  C --> D[Acquire row-order lock if needed]
  D --> E[Write rows]
  E --> F[Commit]
  F --> G[Deferred UPDATE trigger bumps rows_version once per table]

_{Reviews (10) · Last reviewed commit: "fix(tables): key unique JSON values by c..."}

waleedlatif1 · 2026-09-26T18:28:01Z

@greptile

waleedlatif1 · 2026-09-26T18:28:02Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-26T18:28:09Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 9 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-26T19:04:51Z

@greptile

waleedlatif1 · 2026-09-26T19:04:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-26T19:04:58Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 9 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-26T19:08:57Z

@greptile

waleedlatif1 · 2026-09-26T20:16:53Z

@greptile

waleedlatif1 · 2026-09-26T20:16:54Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-26T20:17:00Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 8 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-26T23:29:49Z

@greptile

waleedlatif1 · 2026-09-26T23:29:50Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-26T23:29:54Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

waleedlatif1 · 2026-09-27T01:50:21Z

@greptile re the outside-diff note on service.ts:2325 (sparse filters may time out): this PR doesn't change that query. The capped update/delete limit branches are byte-for-byte the staging queries; the earlier fenced-CTE version was removed before this head, because it regressed dense filters. So there's no 'previous GIN-first query' to lose relative to staging. The only thing this PR changes for these plans is the dependency statistic, which corrects the per-table row estimate the planner uses to choose between the ordered walk and the tenant GIN. In the A/B (4,000 tables, skewed sizes, 2.1M rows), rare and late-matching filters had no plan trap after the statistic, and large-table data @> filters went from 1.7–5.7 ms to 0.5 ms.

greptile-apps · 2026-09-27T01:50:47Z

You're right. apps/sim/lib/table/rows/service.ts is unchanged in this PR (the head and staging revisions are byte-for-byte identical), so there is no PR-introduced query regression or “previous GIN-first query” to flag. The dependency statistic can change the planner's estimates and plan choice, but that is the intended improvement here, and your skewed-table A/B results directly exercise the sparse/late-match cases.

I'm withdrawing the outside-diff note; no action is needed for service.ts.

waleedlatif1 · 2026-09-27T02:01:29Z

@greptile re the concurrent-unique-insert risk in the summary: this PR doesn't change unique-constraint behavior. insertRow, batch insert, and upsert's insert branch are byte-identical to staging. An earlier revision moved the unique check under the per-table append lock, which closed the race but serialized every insert into a table with a unique column (2.5–10× slower under concurrency). That revision was reverted, so this PR doesn't touch the race either way. The race exists on staging today; fixing it properly needs per-value locks (table, column, normalized value), and that goes in a separate PR with its own benchmark. This PR only changes the rows_version trigger, the row FK's deferrability, and the planner statistic.

greptile-apps · 2026-09-27T02:01:56Z

You’re right: the concurrent unique-insert race is not introduced or changed by this PR. The changed application code contains no insert, batch-insert, or upsert-branch edits, and the migration only changes the rows-version triggers, FK deferrability, and planner statistics.

I’m withdrawing that critical-risk finding from this review; it should be addressed separately with the per-value locking design and its own concurrency benchmarks.

…h per-value advisory locks

…ck unique columns on every import batch Lower the per-transaction value-lock cap to 64 so concurrent writers cannot exhaust the shared lock table; larger writes keep the exclusive column-lock fallback. Import batches now take the unique-column locks and run their unique check inside their transaction, batchUpdateRows rejects two updates writing the same unique value and locks and checks only the unique columns each update changes, and the 0390 comment describes the mechanism only.

…ocks

waleedlatif1 · 2026-09-28T03:34:17Z

@greptile

waleedlatif1 · 2026-09-28T03:34:19Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-28T03:34:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 13 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

… schemas stay within the lock budget

waleedlatif1 · 2026-09-28T03:52:51Z

@greptile

waleedlatif1 · 2026-09-28T03:52:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-28T03:52:58Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 13 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…ver hides a batch duplicate

waleedlatif1 · 2026-09-28T04:04:22Z

@greptile

waleedlatif1 · 2026-09-28T04:04:23Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-28T04:04:29Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 13 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

vercel Bot deployed to Preview September 26, 2026 18:04 View deployment

cubic-dev-ai Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/service.ts Outdated

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts Outdated

Comment thread apps/sim/lib/table/rows/service.ts Outdated

vercel Bot temporarily deployed to Preview September 26, 2026 18:27 Inactive

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts Outdated

cubic-dev-ai Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts Outdated

vercel Bot temporarily deployed to Preview September 26, 2026 19:04 Inactive

cubic-dev-ai Bot reviewed Sep 26, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts Outdated

vercel Bot deployed to Preview September 26, 2026 20:18 View deployment

cubic-dev-ai Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts Outdated

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts

This comment has been minimized.

Sign in to view

waleedlatif1 force-pushed the improvement/table-rows-version-at-commit branch from dba8085 to ecb119e Compare September 26, 2026 23:29

vercel Bot deployed to Preview September 26, 2026 23:31 View deployment

waleedlatif1 mentioned this pull request Sep 28, 2026

fix(knowledge): resume the member resurrection walk from a persisted cursor #8365

Merged

6 tasks

waleedlatif1 added 5 commits September 27, 2026 19:29

improvement(tables): bump a table's rows version once at commit

65895bf

fix(tables): serialize concurrent writes of the same unique value wit…

49d4904

…h per-value advisory locks

fix(tables): take the batched advisory locks on a transaction

16ee1d8

chore(tables): share one unique-value key between the check and the l…

97cb317

…ocks

waleedlatif1 force-pushed the improvement/table-rows-version-at-commit branch from 1d520f6 to 97cb317 Compare September 28, 2026 03:34

waleedlatif1 changed the title ~~improvement(tables): bump a table's rows version once at commit~~ improvement(tables): bump rows version once at commit and serialize same-value unique writes Sep 28, 2026

vercel Bot deployed to Preview September 28, 2026 03:36 View deployment

cubic-dev-ai Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts

Comment thread apps/sim/lib/table/rows/unique-locks.ts Outdated

Comment thread packages/db/migrations/0390_table_rows_version_at_commit.sql

fix(tables): lock a table's unique values under one table key so wide…

fbc3db1

… schemas stay within the lock budget

vercel Bot temporarily deployed to Preview September 28, 2026 03:52 Inactive

cubic-dev-ai Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/service.ts

greptile-apps Bot reviewed Sep 28, 2026

View reviewed changes

Comment thread apps/sim/lib/table/rows/row-writes.integration.ts Outdated

fix(tables): key unique JSON values by canonical form so key order ne…

1c00cf5

…ver hides a batch duplicate

vercel Bot temporarily deployed to Preview September 28, 2026 04:04 Inactive

cubic-dev-ai Bot reviewed Sep 28, 2026

View reviewed changes

Conversation

waleedlatif1 commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 26, 2026

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 26, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 26, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

waleedlatif1 commented Sep 26, 2026

Uh oh!

waleedlatif1 commented Sep 26, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 26, 2026

Uh oh!

waleedlatif1 commented Sep 27, 2026

Uh oh!

greptile-apps Bot commented Sep 27, 2026

Uh oh!

waleedlatif1 commented Sep 27, 2026

Uh oh!

greptile-apps Bot commented Sep 27, 2026

Uh oh!

waleedlatif1 commented Sep 28, 2026

Uh oh!

waleedlatif1 commented Sep 28, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 28, 2026

waleedlatif1 commented Sep 26, 2026 •

edited

Loading

vercel Bot commented Sep 26, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading