Fix SND attribute values lost or left stale across ETL runs - #998
Open
labkey-martyp wants to merge 2 commits into
Open
Fix SND attribute values lost or left stale across ETL runs#998labkey-martyp wants to merge 2 commits into
labkey-martyp wants to merge 2 commits into
Conversation
The merge deleted and recreated each row's exp.Object, cascading its attribute values away for only the _SND Attribute Data step to re-insert; because that step computes its incremental window independently, it can skip rows this merge just cleared and the values are then lost permanently. Also drops the diagnostic that reported which rows were about to be cleared, which no longer describes anything.
v_snd_attributeData filtered blank values out, so a cleared attribute never reached the ETL and its old value survived; the view now passes those rows through with NULL values and the attribute step deletes the property they replace. Lookup misses still keep the old value.
labkey-martyp
requested review from
RameshRapa,
ankurjuneja,
jallentxbiomed and
labkey-bpatel
September 9, 2026 16:10
labkey-bpatel
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
Stop the SND ETL losing attribute values when the event data and attribute data steps' incremental windows drift apart, and make a blanked source attribute clear its stored value instead of leaving it stale. The event data step deleted and recreated each merged row's exp object, which discarded its attribute values, and only the attribute data step restores them; because that step computes its own incremental window, a row whose change landed in the seconds between the two steps computing their windows was cleared by the first and never seen by the second, and its values were gone permanently. The delete dates from the days before the attribute data step existed and has been redundant since that step began replacing values property by property.
The attribute data source view also filtered blank values out, so a cleared attribute never reached the ETL and its old value survived. That was masked by the same delete, which is why it surfaces alongside this fix.
The view change is a source-database ALTER VIEW and deploys separately from the module. The module change is safe to deploy first, since no null-valued rows arrive until the view changes.
Related Pull Requests
Changes