feat: ✨ stage basic raw data#108
Conversation
…easibility-data into feat/data-raw-to-staging
There was a problem hiding this comment.
This mapping tells us what forms/resources are filled in at which events. We can use it to select relevant rows for each resource when staging data.
There was a problem hiding this comment.
Similar addition as in #106. It might make sense to pull out common logic once both are merged.
…feasibility-data into feat/raw-to-staging-basic
lwjohnst86
left a comment
lwjohnst86
left a comment
There was a problem hiding this comment.
As I've written in other PRs, there's not enough clarity about what data gets converted into what resource. I think we need to take a step back and write some docs to make that link clearer. Plus, as @joelostblom suggested, wrangle data first without using/depending on the data dictionary, and only after we've created them into staging/ will we extract the properties from them and link them to the data dictionary. I'm going to convert this to draft for now.
2 participants
Description
This PR stages raw data that does not require special transformations (like, e.g., VAS data).
I split this into a separate PR because we are already handling repeating and non-repeating resources, so that's enough complexity.
This works only with #106.
It ran fine on the newest version of the real data and metadata on GenomeDK.
Closes #109
This PR needs an in-depth review.
Checklist
just run-all