Populate ADF data lineage + Switch-aware walking on the shared source AST (#62b)

[matthewmoorcroft]

Summary: Populate ADF source-graph data/control lineage and make discovery traversal cover every Switch and other control-flow branch.

Depends on: #84 and the main re-land of #79.

Blocks: #62, #81, #77, #85.

Current status

This PR is merged into integration/discovery, not main (merge commit 3064f3d). Its checks passed when merged. The implementation exists only on staging and therefore is not available from main.

On staging:

• src/flowx/sources/adf/dataset_lineage.py — activity_data_assets and resolve_dataset_identity collect all activity input/output references and emit shared DataAsset values using resolved identity first and a structural signature only when supportable; identities are never guessed.
• src/flowx/sources/adf/discovery_mapping.py — _activity_to_node attaches data_reads/data_writes; _record_invocation records ExecutePipeline targets and waitOnCompletion under neutral invocation properties; adf_definitions_to_source_graphs derives lineage after mapping.
• src/flowx/discovery_lineage.py — walk_nodes recursively traverses every ContainerNode branch, while build_graph_control_edges, build_graph_data_edges, and build_graph_lineage derive lineage from the shared AST.
• src/flowx/lineage.py — control_edges_from_calls and data_edges_from_endpoints hold the common dedupe, tier matching, and self-edge rules.
• src/flowx/discovery_serde.py and src/flowx/ir_serde.py:lineage_to_dict round-trip the per-graph lineage block.

The scope is per-workflow data edges because DataEdge has task endpoints only; control edges may cross workflows.

What needs to change

Re-land this work on main after #84 and #79. Rebase onto the final shared types rather than redefining lineage. Preserve the corrected signature rule: unresolved opaque references without a path anchor must not join merely because an ADF dataset name matches.

Conversion and packaging remain unchanged in this slice.

How to approach

Layer dataset_lineage.py and the ADF population hooks onto the re-landed #79 mapper. Keep discovery traversal source-neutral: ADF-specific extraction belongs in sources/adf, while edge construction stays in discovery_lineage.py/lineage.py.

Acceptance/verification:

• make test and make fmt pass.
• Tests cover all Copy/Lookup/activity-level inputs and outputs, distinct parameter bindings, strong identity vs weak signature matching, and no false fallback between distinct resolved identities.
• ExecutePipeline creates a ControlEdge with source/target/via task/wait/resolved fields.
• walk_nodes covers If true/false, ForEach/Until body, every Switch case, and Switch default, including empty branches.
• Discovery serde round-trips populated and empty lineage lists.
• Existing coverage golden remains unchanged.
• ADF convert/package and Airflow paths remain untouched and green.

This remains additive discover-only behavior.

[matthewmoorcroft]

Addressed both blocking findings in 1e539ba (lineage.py refactor + Switch recursion left as-is, per review).

BLOCKING 1 — name-only fallback → false edges: signatures are now path-derived only (resolved identity, else structural path signature). With neither available the signature is left empty; an empty signature is falsy so _match_assets cannot join on it — the bare dataset name is never a join key again (#36's rule). New test: an opaque writer+reader produce no data edge and carry empty signatures.

BLOCKING 2 — distinct parameterized refs collapsed: de-dup now keys on (reference_name, parameter binding) instead of name alone, so ds(tbl=orders) and ds(tbl=customers) are both retained while an identical binding in both a slot and typeProperties still collapses to one. New tests cover both.

Gates: 1283 unit tests pass (+3), ruff format/check + mypy clean. Recomputed fixture lineage: 11 control edges, 0 data edges, zero false edges — the 47 previously name-signature'd assets now carry non-joining empty signatures.

This pull request and its description were written by Isaac.