Summary: Pete-owned Phase 2 work to make Airflow conversion consume the shared discovery AST and decision plan, mapping directly to the common IR without changing the ADF implementation.
Depends on: #84, Pete's Airflow discovery alignment #63, routing #77, and #85's shared minimal IR/validation surface.
Blocks: Airflow adoption of the decision-fed shared-AST conversion architecture.
Current status
This is an open issue and is not built by the flowx ADF workstream; owner is Pete.
Today Airflow bypasses SourceGraph:
- src/flowx/sources/airflow/loader/api.py:load_airflow_dags/load_pipelines parses DAG files and returns models.ir.Pipeline objects directly.
- src/flowx/sources/airflow/loader/lowering.py:_load_airflow_module lowers captured operators, TaskFlow, groups/mapping, dependencies, schedules, and policies straight into target IR.
- src/flowx/sources/airflow/discover.py:build_inventory_dict builds discovery inventory from that already-lowered Pipeline IR.
- src/flowx/sources/airflow/convert.py:main calls the same load_pipelines path and serializes Pipeline with ir_serde.pipeline_to_dict.
There is no Airflow SourceGraph mapper, persisted source AST, discovery lineage population, or decision-fed AST-to-IR conversion. The existing Airflow discover/convert/package and fingerprint-bound leaf resolver in agentic.py must not be changed by the ADF workstream.
What needs to change
After #63 maps Airflow's pre-lowering captures onto the shared AST, Pete should make Airflow convert consume the persisted SourceGraph plus the user-approved connected-component decision and map it to Pipeline/Activity IR.
Reuse #85's shared IR fields, serde/rehydration, typed ingestion/resource abstractions, and #87 structural validation. Do not add a parallel Airflow-only lineage or routing model.
Carry Airflow-specific semantics faithfully: operator/native type, TaskFlow and task groups, dynamic mapping, dependency trigger rules/run_condition, policies/default_args, schedule, raw source/captures, data assets, and graph lineage. Reconcile source task keys to normalized IR keys and preserve lineage across any motif/group transformation.
How to approach
Pete should build the mapper beside sources/airflow/loader from the SourceGraph artifact produced by #63, using current captures.py:TaskCapture/EdgeCapture, visitor.py:_DagVisitor, and lowering helpers only behind explicit source-to-target seams.
Change sources/airflow/convert.py:main to load the persisted SourceGraph and decision rather than call load_pipelines in the enabled Phase 2 path. Apply decisions per connected component at conversion time; keep the current direct-lowering path available until migration is proven and do not couple ADF modules into Airflow.
Acceptance/verification:
- Airflow fixture golden tests cover AST -> IR -> serde -> rehydrate -> package for classic operators, TaskFlow, TaskGroups, dynamic mapping, trigger rules, schedules, and policies.
- DataAsset/Lineage endpoints survive task-key normalization and transformations.
- Decision/component validation matches ADF behavior and rejects split cross-DAG references.
- Existing Airflow audit/reconciliation and agentic leaf-resolver evidence remains valid or has an explicit migration.
- ADF discovery/convert/package golden tests remain unchanged.
- The change is gated/additive until Pete opts the Airflow path in.
This issue is Pete-owned; the ADF workstream supplies the source-neutral standard and shared seams only.
Summary: Pete-owned Phase 2 work to make Airflow conversion consume the shared discovery AST and decision plan, mapping directly to the common IR without changing the ADF implementation.
Depends on: #84, Pete's Airflow discovery alignment #63, routing #77, and #85's shared minimal IR/validation surface.
Blocks: Airflow adoption of the decision-fed shared-AST conversion architecture.
Current status
This is an open issue and is not built by the flowx ADF workstream; owner is Pete.
Today Airflow bypasses SourceGraph:
There is no Airflow SourceGraph mapper, persisted source AST, discovery lineage population, or decision-fed AST-to-IR conversion. The existing Airflow discover/convert/package and fingerprint-bound leaf resolver in agentic.py must not be changed by the ADF workstream.
What needs to change
After #63 maps Airflow's pre-lowering captures onto the shared AST, Pete should make Airflow convert consume the persisted SourceGraph plus the user-approved connected-component decision and map it to Pipeline/Activity IR.
Reuse #85's shared IR fields, serde/rehydration, typed ingestion/resource abstractions, and #87 structural validation. Do not add a parallel Airflow-only lineage or routing model.
Carry Airflow-specific semantics faithfully: operator/native type, TaskFlow and task groups, dynamic mapping, dependency trigger rules/run_condition, policies/default_args, schedule, raw source/captures, data assets, and graph lineage. Reconcile source task keys to normalized IR keys and preserve lineage across any motif/group transformation.
How to approach
Pete should build the mapper beside sources/airflow/loader from the SourceGraph artifact produced by #63, using current captures.py:TaskCapture/EdgeCapture, visitor.py:_DagVisitor, and lowering helpers only behind explicit source-to-target seams.
Change sources/airflow/convert.py:main to load the persisted SourceGraph and decision rather than call load_pipelines in the enabled Phase 2 path. Apply decisions per connected component at conversion time; keep the current direct-lowering path available until migration is proven and do not couple ADF modules into Airflow.
Acceptance/verification:
This issue is Pete-owned; the ADF workstream supplies the source-neutral standard and shared seams only.