Skip to content

perf: read the catalog by join rather than by a literal id list - #27

Merged
JeePeeTee merged 1 commit into
mainfrom
perf/schema-join
Sep 2, 2026
Merged

JeePeeTee merged 1 commit into
mainfrom
perf/schema-join

Conversation

@JeePeeTee

Copy link
Copy Markdown
Owner

Closes #22. Benchmarked first — the numbers and the fixture are in
the issue.

The problem

A schema-only export of a 12,900-object database:

cold (plan cache cleared)   6.27 s   6.32 s   6.20 s
warm                        0.94 s   0.96 s

Six catalog queries each spliced every object id into a literal IN list — 31 KB of SQL text
for 3,000 tables. A list that size costs ~900 ms to compile; the same query then runs in 32 ms.

That is why it only bites the case it was reported from: a nightly job over seventy databases.
Object ids differ per database, so the SQL text differs, so every database compiles its own plans
and nothing is reused. Roughly seven minutes a night, all of it compilation.

The fix, and the one that does not work

The queries now join sys.objects instead of naming ids, so the text is constant.

cold   6.32 s  →  1.55 s     ~4x
warm   0.97 s  →  0.92 s     unchanged, as expected

Batching the id list — which the issue suggested, and so did I — was measured and does not
help:
917 ms against 931 ms. Each batch is a different literal list, so a different query text,
so another compilation. It trades one large compile for several smaller ones.

approach cold warm
literal IN list 931 ms 32 ms
batched 1,000 at a time 917 ms 47 ms
STRING_SPLIT of one parameter 109 ms 67 ms
join to sys.objects 70 ms 38 ms

I chose the join over STRING_SPLIT because the latter needs database compatibility level 130,
and this repo documents no minimum version and gates nothing on one. That floor should be a
deliberate decision, not a side effect of a performance fix.

Correctness

The queries now return rows for tables the filter excluded, so every lookup by object id checks
the map
instead of trusting its zero value. Without that, a row for an excluded table would
attach its columns, indexes or constraints to tables[0].

Verified against the previous build by exporting the same database with both and comparing bytes:

(no filter)                                          identical
--include dbo.Crm*                                   identical
--include sales.* --exclude-data CrmProjectActivity  identical

A trap the test suite caught

My first version filtered the columns query on o.is_ms_shipped = 0. SQL Server reports a
user-created table type with is_ms_shipped = 1
— verified on a live instance — so every table
type lost its columns and CREATE TYPE ... AS TABLE () came out empty. Four integration tests
failed with Incorrect syntax near ')'. Table types are now matched on type alone, with a comment
recording why, since the flag looks like the obvious thing to filter on.

Checks

  • full suite under -race against a live SQL Server: 8/8 packages
  • round trip on a 403-table database: export 11.9 s, import 21.7 s, verify: OK
  • gofmt, go vet, go build clean

A schema-only export of a 12,900-object database took 6.3s cold and 0.9s warm.
The gap was query plan compilation: six catalog queries each spliced every
object id into a literal IN list - 31 KB of SQL text for 3,000 tables - and a
list that size costs roughly 900ms to compile. Once cached the same query runs
in 32ms, which is why the cost only shows on a first run.

That makes it a problem for the case it was reported from: a nightly job over
seventy databases. Object ids differ per database, so the SQL text differs, so
every database compiles its own plans and none of the work is reused. Seven
minutes a night, all of it compilation.

The queries now join sys.objects instead of naming ids, so the text is
constant and one plan serves every database and every --include filter. Cold
drops from 6.3s to 1.55s, about 4x. Warm is unchanged at ~0.9s, as expected -
there was never anything wrong with the warm path.

Batching the id list, which the issue and my own first comment on it suggested,
was measured and does not work: 917ms against 931ms. Each batch is a different
literal list, so it is a different query text, so it is another compilation.
It trades one large compile for several smaller ones.

The queries now return rows for tables the filter excluded, so every lookup by
object id checks the map rather than trusting its zero value. Without that, a
row for an excluded table would attach its columns, indexes or constraints to
tables[0]. Verified by exporting with and without --include and comparing the
archives byte for byte against the previous build.

One trap found by the test suite: SQL Server reports a user-created table type
with is_ms_shipped = 1, so filtering the columns query on that flag dropped
every table type and their CREATE TYPE came out with no columns at all. Table
types are matched on type alone.
@JeePeeTee
JeePeeTee merged commit 65961da into main Sep 2, 2026
2 checks passed
@JeePeeTee
JeePeeTee deleted the perf/schema-join branch September 2, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schema extraction performance on very large schemas (8000+ objects)

1 participant