Skip to content

Fix SND attribute values lost or left stale across ETL runs - #998

Open
labkey-martyp wants to merge 2 commits into
release26.3-SNAPSHOTfrom
26.3_fb_snd_attribute_data_loss
Open

Fix SND attribute values lost or left stale across ETL runs#998
labkey-martyp wants to merge 2 commits into
release26.3-SNAPSHOTfrom
26.3_fb_snd_attribute_data_loss

Conversation

@labkey-martyp

Copy link
Copy Markdown
Contributor

Rationale

Stop the SND ETL losing attribute values when the event data and attribute data steps' incremental windows drift apart, and make a blanked source attribute clear its stored value instead of leaving it stale. The event data step deleted and recreated each merged row's exp object, which discarded its attribute values, and only the attribute data step restores them; because that step computes its own incremental window, a row whose change landed in the seconds between the two steps computing their windows was cleared by the first and never seen by the second, and its values were gone permanently. The delete dates from the days before the attribute data step existed and has been redundant since that step began replacing values property by property.

The attribute data source view also filtered blank values out, so a cleared attribute never reached the ETL and its old value survived. That was masked by the same delete, which is why it surfaces alongside this fix.

The view change is a source-database ALTER VIEW and deploys separately from the module. The module change is safe to deploy first, since no null-valued rows arrive until the view changes.

Related Pull Requests

Changes

  • The event data merge no longer deletes and recreates each row's exp object, so attribute values survive a merge that the attribute data step's window does not also cover.
  • The attribute data source view passes blank values through as null-valued rows instead of filtering them out, so a cleared attribute reaches the ETL.
  • The attribute data step removes the stored value for a null-valued source row; a lookup value that fails to resolve still leaves the old value in place.
  • The diagnostic that reported which values a merge was about to clear is removed, since nothing is cleared now.

The merge deleted and recreated each row's exp.Object, cascading its attribute values away for only the _SND Attribute Data step to re-insert; because that step computes its incremental window independently, it can skip rows this merge just cleared and the values are then lost permanently.

Also drops the diagnostic that reported which rows were about to be cleared, which no longer describes anything.
v_snd_attributeData filtered blank values out, so a cleared attribute never reached the ETL and its old value survived; the view now passes those rows through with NULL values and the attribute step deletes the property they replace. Lookup misses still keep the old value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants