perf: read the catalog by join rather than by a literal id list - #27
Merged
Merged
Conversation
A schema-only export of a 12,900-object database took 6.3s cold and 0.9s warm. The gap was query plan compilation: six catalog queries each spliced every object id into a literal IN list - 31 KB of SQL text for 3,000 tables - and a list that size costs roughly 900ms to compile. Once cached the same query runs in 32ms, which is why the cost only shows on a first run. That makes it a problem for the case it was reported from: a nightly job over seventy databases. Object ids differ per database, so the SQL text differs, so every database compiles its own plans and none of the work is reused. Seven minutes a night, all of it compilation. The queries now join sys.objects instead of naming ids, so the text is constant and one plan serves every database and every --include filter. Cold drops from 6.3s to 1.55s, about 4x. Warm is unchanged at ~0.9s, as expected - there was never anything wrong with the warm path. Batching the id list, which the issue and my own first comment on it suggested, was measured and does not work: 917ms against 931ms. Each batch is a different literal list, so it is a different query text, so it is another compilation. It trades one large compile for several smaller ones. The queries now return rows for tables the filter excluded, so every lookup by object id checks the map rather than trusting its zero value. Without that, a row for an excluded table would attach its columns, indexes or constraints to tables[0]. Verified by exporting with and without --include and comparing the archives byte for byte against the previous build. One trap found by the test suite: SQL Server reports a user-created table type with is_ms_shipped = 1, so filtering the columns query on that flag dropped every table type and their CREATE TYPE came out with no columns at all. Table types are matched on type alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22. Benchmarked first — the numbers and the fixture are in
the issue.
The problem
A schema-only export of a 12,900-object database:
Six catalog queries each spliced every object id into a literal
INlist — 31 KB of SQL textfor 3,000 tables. A list that size costs ~900 ms to compile; the same query then runs in 32 ms.
That is why it only bites the case it was reported from: a nightly job over seventy databases.
Object ids differ per database, so the SQL text differs, so every database compiles its own plans
and nothing is reused. Roughly seven minutes a night, all of it compilation.
The fix, and the one that does not work
The queries now join
sys.objectsinstead of naming ids, so the text is constant.Batching the id list — which the issue suggested, and so did I — was measured and does not
help: 917 ms against 931 ms. Each batch is a different literal list, so a different query text,
so another compilation. It trades one large compile for several smaller ones.
INlistSTRING_SPLITof one parametersys.objectsI chose the join over
STRING_SPLITbecause the latter needs database compatibility level 130,and this repo documents no minimum version and gates nothing on one. That floor should be a
deliberate decision, not a side effect of a performance fix.
Correctness
The queries now return rows for tables the filter excluded, so every lookup by object id checks
the map instead of trusting its zero value. Without that, a row for an excluded table would
attach its columns, indexes or constraints to
tables[0].Verified against the previous build by exporting the same database with both and comparing bytes:
A trap the test suite caught
My first version filtered the columns query on
o.is_ms_shipped = 0. SQL Server reports auser-created table type with
is_ms_shipped = 1— verified on a live instance — so every tabletype lost its columns and
CREATE TYPE ... AS TABLE ()came out empty. Four integration testsfailed with
Incorrect syntax near ')'. Table types are now matched on type alone, with a commentrecording why, since the flag looks like the obvious thing to filter on.
Checks
-raceagainst a live SQL Server: 8/8 packagesverify: OKgofmt,go vet,go buildclean