Summary: Define the shared, source-neutral discovery AST and lineage contract; the implementation is delivered by open PR #84 and must merge before source mappings re-land.
Depends on: None.
Blocks: #62, #63, #64, #79, #80, #81, #82, #85, #86, #87.
Current status
This is an open issue whose implementation is delivered by open PR #84. #84 targets main, has passed cross-review, and has green checks; it has not merged yet.
On #84's branch:
- src/flowx/models/discovery.py defines SourceGraph, SourceNode, ContainerNode, GapNode, SourceDependency, ScheduleSpec, ParameterSpec, and PolicySpec, including raw/properties/extensions escape seams.
- src/flowx/models/ir.py defines the reusable DataAsset, ControlEdge, DataEdge, MotifAnnotation, and Lineage types.
- src/flowx/discovery_serde.py:source_graph_to_dict/source_graph_from_dict owns discovery serialization.
- src/flowx/discovery_lineage.py:walk_nodes and src/flowx/lineage.py:control_edges_from_calls/data_edges_from_endpoints provide the source-neutral lineage traversal/derivation seams.
main still lacks models/discovery.py and the shared lineage substrate. ADF mapping is staged separately in #79-#81; Airflow adoption is Pete's #63.
What needs to change
Merge #84 to main, then close this issue as delivered. No new source wiring or conversion work is required in #61 itself.
The accepted standard is a source-faithful discovery AST, not a Databricks target model: source-specific detail must survive in raw/properties/extensions; data identities are best-effort and never guessed; control lineage is cross-workflow while v1 data lineage is per workflow.
How to approach
Review and merge #84 as the single source of truth. Rebase #79-#82 onto it rather than reconciling parallel type definitions later.
Acceptance/verification:
This change is additive and source-neutral.
Summary: Define the shared, source-neutral discovery AST and lineage contract; the implementation is delivered by open PR #84 and must merge before source mappings re-land.
Depends on: None.
Blocks: #62, #63, #64, #79, #80, #81, #82, #85, #86, #87.
Current status
This is an open issue whose implementation is delivered by open PR #84. #84 targets main, has passed cross-review, and has green checks; it has not merged yet.
On #84's branch:
main still lacks models/discovery.py and the shared lineage substrate. ADF mapping is staged separately in #79-#81; Airflow adoption is Pete's #63.
What needs to change
Merge #84 to main, then close this issue as delivered. No new source wiring or conversion work is required in #61 itself.
The accepted standard is a source-faithful discovery AST, not a Databricks target model: source-specific detail must survive in raw/properties/extensions; data identities are best-effort and never guessed; control lineage is cross-workflow while v1 data lineage is per workflow.
How to approach
Review and merge #84 as the single source of truth. Rebase #79-#82 onto it rather than reconciling parallel type definitions later.
Acceptance/verification:
This change is additive and source-neutral.