Repository navigation
fix(snapshot): drop a leftover __snapshot_upsert table before recreating it - #806
Open
gedaliakliger wants to merge 2 commits into
Open
gedaliakliger wants to merge 2 commits into
gedaliakliger wants to merge 2 commits into
Conversation
…ing it A snapshot run that died after filling <snapshot>__snapshot_upsert left the table behind, and the next run reused it through create table if not exists, so its rows were inserted twice. Fixes ClickHouse#600
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
clickhouse__snapshot_merge_sqlcreates<snapshot>__snapshot_upsertwithcreate table if not existsand drops it only at the end of a successful run. A run that died after filling it left the table behind, and the next run inserted into it again, so every snapshot row was duplicated.The upsert table is now dropped before it is created, the same way
__dbt_tmpis. The new test leaves a filled upsert table behind and checks that the next snapshot run keeps 10 rows; without the fix it has 20.Ran here on ClickHouse 26.3, single node, over HTTP and native: the snapshot tests in
tests/integration/adapter/basic, plus ruff and mypy on the changed test file. The cluster and Cloud paths, newer ClickHouse versions and the fullmake lintare left to CI.Fixes #600
Checklist
Note
Low Risk
Small, targeted snapshot merge change aligned with existing tmp-table cleanup, with a regression test; no auth or broad behavioral shifts.
Overview
Fixes duplicate snapshot rows when a prior run failed after populating
<snapshot>__snapshot_upsert.clickhouse__snapshot_merge_sqlnow callsdrop_relation_if_existson that upsert relation beforeCREATE TABLE IF NOT EXISTS, matching staging cleanup for__dbt_tmp.Adds a regression integration test that simulates a leftover filled upsert table and asserts row and
dbt_scd_idcounts stay correct, plus a changelog note under Bugs.Reviewed by Cursor Bugbot for commit 25edcea. Bugbot is set up for automated code reviews on this repo. Configure here.