Summary: Add uncollapsed motif annotations to discovery inventory as the deterministic-capability signal used by routing; this is Phase 1 and is not built.
Depends on: #84 and the main re-land of #79-#81.
Blocks: #77.
Current status
This is an open issue with no implementation yet.
Today:
- src/flowx/motifs/detector.py:detect_motifs detects ADF motifs from AdfPipeline/AdfDefinitions and returns DetectedMotif values.
- src/flowx/sources/adf/loader.py:build_profile_rows reduces detection to the numeric collapsible_patterns profile column; inventory.json does not receive the matched members or replacement.
- src/flowx/sources/adf/translate.py:translate_pipeline/_filter_motifs_for_collapse detects again during conversion and may pass matches to src/flowx/motifs/collapser.py:collapse_motifs, which replaces multiple IR tasks with MotifActivity nodes.
- src/flowx/models/ir.py:MotifAnnotation/Lineage provides a source-neutral place for motif metadata, but src/flowx/discovery_lineage.py:build_graph_lineage currently derives control/data edges only; the inventory's lineage.motifs list is not populated.
- Lakeflow Connect is currently reachable only after conversion through CopyActivity flags in preparer/activity_preparers/copy.py:prepare/_prepare_lakeflow_connect_copy, or through a consolidated metadata-driven MotifActivity in preparer/activity_preparers/motif.py:_prepare_consolidated_metadata_driven.
What needs to change
During ADF discovery, add one additive, uncollapsed annotation per detected motif containing:
- stable motif id/name;
- matched/member source activity keys;
- databricks_replacement;
- confidence tier and explanatory notes.
These annotations are a capability signal: they tell #77 where the deterministic engine already has a recognized route. They must not collapse SourceNodes or change conversion in Phase 1. Existing inventory keys and coverage output must remain compatible.
The shape stays source-neutral so Pete can later populate it from Airflow, but this issue must not change Airflow's current path.
How to approach
Run motif detection alongside ADF SourceGraph construction in sources/adf/loader.py:main or sources/adf/discovery_mapping.py:adf_definitions_to_source_graphs, map DetectedMotif into shared MotifAnnotation values, and attach them to each graph's Lineage before discovery_inventory.py:build_source_inventory calls ir_serde.py:lineage_to_dict.
Extend models/ir.py:MotifAnnotation and its serde additively for confidence if the field is not already present. Keep motifs/detector.py authoritative for detection and motifs/collapser.py authoritative for optional convert-time collapse; do not duplicate heuristics in the inventory emitter.
Acceptance/verification:
- Golden tests prove all prior inventory/coverage keys are unchanged.
- Fixture tests compare inventory annotations with detect_motifs results: same ids, members, replacements, confidence, and notes.
- Structural tests prove SourceGraph task count/order/dependencies are unchanged (no discover-time collapse).
- Existing convert motif-collapse tests remain green.
- Empty/no-match pipelines emit an empty or omitted additive motif block consistently with the shared lineage contract.
- Airflow discover/convert/package tests stay green without source-path changes.
This is additive and non-breaking.
Summary: Add uncollapsed motif annotations to discovery inventory as the deterministic-capability signal used by routing; this is Phase 1 and is not built.
Depends on: #84 and the main re-land of #79-#81.
Blocks: #77.
Current status
This is an open issue with no implementation yet.
Today:
What needs to change
During ADF discovery, add one additive, uncollapsed annotation per detected motif containing:
These annotations are a capability signal: they tell #77 where the deterministic engine already has a recognized route. They must not collapse SourceNodes or change conversion in Phase 1. Existing inventory keys and coverage output must remain compatible.
The shape stays source-neutral so Pete can later populate it from Airflow, but this issue must not change Airflow's current path.
How to approach
Run motif detection alongside ADF SourceGraph construction in sources/adf/loader.py:main or sources/adf/discovery_mapping.py:adf_definitions_to_source_graphs, map DetectedMotif into shared MotifAnnotation values, and attach them to each graph's Lineage before discovery_inventory.py:build_source_inventory calls ir_serde.py:lineage_to_dict.
Extend models/ir.py:MotifAnnotation and its serde additively for confidence if the field is not already present. Keep motifs/detector.py authoritative for detection and motifs/collapser.py authoritative for optional convert-time collapse; do not duplicate heuristics in the inventory emitter.
Acceptance/verification:
This is additive and non-breaking.