chore: ♻️ add download metadata to pytask pipeline#153
Conversation
There was a problem hiding this comment.
If we don't want to keep passing the paths around that declare task dependencies, we can move them into a data catalog
There was a problem hiding this comment.
Would you like me to do that now?
There was a problem hiding this comment.
Yea, let's see how it looks
There was a problem hiding this comment.
I had a deeper look and now I think we don't want to use this actually 😅 .
It looks like the main use case for a data catalog is to decouple input/output objects from their storage location, but, for us, the entire flow is structured around files being at particular locations. Plus, the default "node" they use to represent arbitrary objects stores them in binary, so that's not great for inspecting or version controlling.
It is possible to use "path nodes" in a data catalog, but that is just a slightly more verbose way of defining a constant for each reused path.
Now, the one thing we could potentially do is define custom Pytask "nodes" for the file types we work with (CSV, JSON, Parquet). These nodes could encapsulate read/write operations, which is the most cumbersome thing about passing around paths. This would obviously add complexity, so let me know if you think it's worth it.
| def task_download_field_metadata( | ||
| field_metadata_path: Annotated[Path, Product] = BLD_REDCAP / "field_metadata.json", | ||
| ) -> None: | ||
| """Download field metadata to `BLD_REDCAP`.""" | ||
| download_redcap_metadata(field_metadata_path, "metadata") |
There was a problem hiding this comment.
As we will be downloading other metadata as well, I'm calling this field metadata because it contains metadata about all fields in REDCap.
|
|
||
|
|
||
| def task_download_field_metadata( | ||
| field_metadata_path: Annotated[Path, Product] = BLD_REDCAP / "field_metadata.json", |
There was a problem hiding this comment.
I'm a bit confused. What is metadata in this case? The dictionary?
There was a problem hiding this comment.
Yes: we will be downloading multiple dictionaries from REDCap that all contain metadata (about events, repeating instruments, fields). So I'm calling what REDCap calls "data dictionary" "field metadata" here because it contains metadata about the form fields.
There was a problem hiding this comment.
What is the reason to switch to using "field metadata" when we were using "data dictionary" before?
There was a problem hiding this comment.
Do you mean more than the above?
That is: initially we thought we would only need to make one metadata request to REDCap. This was for the "data dictionary" that contains metadata about the form fields. "data dictionary" is (one of) REDCap's term for this metadata, so we used it as well. Later (while looking into primary keys), we found out that this is not enough and that, in fact, we need 3 pieces of metadata: about fields, events, and repeating instruments. So now that we are building things up from square one, I thought it would be better to name these 3 kinds of metadata consistently and descriptively.
(I also think that calling the metadata "dictionary" is quite vague because in Python many things are dictionaries. Plus, the data dict is actually a list of dictionaries.)
github-project-automation
Bot
moved this from In review
to In progress
in Data development
lwjohnst86
changed the title
|
|
||
| BLD_REDCAP = BLD / "redcap" | ||
|
|
||
| FIELD_METADATA_PATH = BLD_REDCAP / "field_metadata.json" |
There was a problem hiding this comment.
If we don't want the constants floating around, we could pack them in a class.
2 participants
Description
This PR sets up pytask and downloads the data dict from REDCap. As we will be downloading other metadata as well, I'm calling it field metadata because it contains metadata about all fields in REDCap.
Related to #97
Needs a thorough review.
Checklist
just run-all