Phase 2: Cross-pipeline / multi-pipeline motif detection (deterministic expansion)

[matthewmoorcroft]

Summary

Extend the deterministic motif system to detect and collapse cross-pipeline / multi-pipeline patterns (Phase 2). Today motifs are strictly intra-pipeline; this issue teaches the engine to recognize a pattern that spans multiple source pipelines (e.g. a metadata-driven multi-pipeline ingestion framework) so the routing agent can recommend a deterministic conversion where today the only option is the agentic fallback.

Phase: 2 (deterministic expansion). Depends on: #64 (additive motifs as the routing capability signal), #77 (per-connected-component routing), and the Phase 1 discovery + lineage stack (#62 / #79–#81). Relates to: #87 (the in-engine agentic escape hatch this incrementally reclaims cases from) and the typed Lakeflow Connect IR issue (the collapse target).

Current status

• Motif detection lives in motifs/detector.py and operates within a single pipeline — it structurally matches patterns over one pipeline's activity tree.
• Collapse happens at convert (motifs/collapser.py), replacing the active task list with a MotifActivity (models/ir.py); original_activities is preserved on the node but dropped on serialize.
• The shared control lineage (ControlEdge, derived by discovery_lineage.py) already captures cross-pipeline invocations, but no motif consumes it — there is no cross-pipeline aggregate and no (pipeline, task_key)-qualified membership.
• Net: a multi-pipeline framework has no motif today and correctly surfaces as a gap → routed agentic.

What needs to change

• A cross-pipeline detector pass that reasons over a connected component (using ControlEdge / lineage) rather than a single pipeline.
• Qualified membership — motif members addressed by (pipeline, task_key), not task_key within one pipeline.
• A cross-pipeline collapse/compose that emits one target construct (e.g. a single Lakeflow Connect job) spanning multiple source pipelines, with consumed-source accounting / provenance.
• Surface these additively in the inventory (per Phase 1: Additive motifs in inventory as the routing capability signal #64) so the router recommends deterministic.

How to approach

• Build on the connected-component routing unit from Phase 1: Routing agent — per-connected-component deterministic/agentic decision #77 (control lineage defines the boundary).
• Start with the highest-value concrete pattern (metadata-driven multi-pipeline → single LFC) and generalize.
• Pairs with the typed LFC IR issue (the collapse target) — sequence that first or together.
• Verify: golden tests that the new motif is detected on a representative multi-pipeline fixture, members are correctly qualified, and collapse produces the expected single target with full source accounting; existing intra-pipeline motifs unchanged.