[bot] Merge 26.9 to develop - #997
Merged
Merged
Conversation
…985) ## Rationale Make it possible to tell from the ETL job log which event data rows ended up holding no attribute values. The _SND Event Data step clears attribute values as a side effect of its merge and only the _SND Attribute Data step restores them; because the two steps decide independently which rows to process, a mismatch leaves event data silently stripped. Newly inserted rows carry the same exposure, arriving with no attribute values and depending on that same second step to supply them. Until now nothing in the log identified the rows either step handled, so the problem could only be found by querying the database after the fact. ## Changes - Both ETL steps log the EventDataIds they handled, so the two logs can be diffed against each other. - The event data step logs on every path that touches exp.Object: merge, insert, and delete. - The merge also logs which rows are about to have their attribute values cleared. - The attribute data step logs the rows it received, the rows it wrote, and the difference between them. - The attribute data step reports when its source rows arrive ungrouped, which is one way the values fail to be restored. - Failures that abort the attribute data step now name the row that failed and note that the batch was left with its attribute values cleared. - Both steps also log the source rowversion span of each batch, so the rows a step actually handled can be placed against the incremental window the ETL logs for the run. - The attribute data step pairs each EventDataId it left unwritten with its rowversion; one falling inside the window points at the source view dropping the row rather than at a window mismatch. - Id lists are sorted and chunked for diffing, and omitted above 2000 ids so the full batches of an initial load don't flood the log. - Everything logs at debug, which the ETL job logger runs at by default; the diagnostics that cost queries or memory are skipped when debug is off. - No change to what the merge writes.
cnathe
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated automatically.
Merging changes from: 6234410
Approve all matching PRs simultaneously.
Approval will trigger automatic merge.
Verify all PRs before approving: https://internal.labkey.com/Scrumtime/Backlog/harvest-gitOpenPullRequests.view?branch=fb_bot_merge_26.9