Summary
Add a first-class typed Lakeflow Connect (LFC) ingestion IR type + preparer (Phase 2), upgrading LFC from an indirectly-reachable / raw-dict construct into a typed, auditable, testable IR activity. This moves agent-authored or motif-collapsed LFC targets from "opaque passthrough" (#87) into first-class deterministic, validated output.
Phase: 2 (typed IR expansion). Relates to: #87 (the passthrough escape hatch this upgrades), the cross-pipeline motif issue (its collapse target), and #85 (AST → IR).
Current status
- There is no first-class LFC ingestion IR type. LFC is only reachable indirectly:
- a
CopyActivity flagged for the Lakeflow connector (preparer/activity_preparers/copy.py), or
- a consolidated metadata-driven
MotifActivity (preparer/activity_preparers/motif.py), which emits one LFC pipeline resource with one object per lookup row.
- The LFC resource is emitted as a fairly raw
pipeline_resources dict (bundler/dab_writer.py), and the motif config models a limited field set — richer connector settings / per-object transforms aren't typed.
- So an agent (or a motif) can produce an LFC pipeline, but not as a typed, schema-validated construct.
What needs to change
- A typed
LakeflowConnectActivity (or equivalent) IR dataclass in models/ir.py carrying the connector fields that matter (source system, objects, destination, connection, schedule/settings).
- A preparer lowering it to the LFC
pipeline_task + resource (generalizing the existing Copy/Motif LFC emission).
- Serde + rehydration round-trip; registry / dispatch wiring.
- Optional: richer connector config beyond the current consolidation shape.
How to approach
- Follow the "Adding a New Deterministic Translator" pattern (IR dataclass → preparer → code_generator if needed → registry), scoped to the preparer/packaging side (this is a target type; discovery does not need it).
- Reuse the existing LFC resource-emission logic from the Copy/Motif preparers as the lowering target.
- Verify: a typed LFC activity round-trips through serde and packages to a valid LFC pipeline resource that passes
bundle_invariants / databricks bundle validate; existing Copy/Motif LFC paths unchanged.
Summary
Add a first-class typed Lakeflow Connect (LFC) ingestion IR type + preparer (Phase 2), upgrading LFC from an indirectly-reachable / raw-dict construct into a typed, auditable, testable IR activity. This moves agent-authored or motif-collapsed LFC targets from "opaque passthrough" (#87) into first-class deterministic, validated output.
Phase: 2 (typed IR expansion). Relates to: #87 (the passthrough escape hatch this upgrades), the cross-pipeline motif issue (its collapse target), and #85 (AST → IR).
Current status
CopyActivityflagged for the Lakeflow connector (preparer/activity_preparers/copy.py), orMotifActivity(preparer/activity_preparers/motif.py), which emits one LFC pipeline resource with one object per lookup row.pipeline_resourcesdict (bundler/dab_writer.py), and the motif config models a limited field set — richer connector settings / per-object transforms aren't typed.What needs to change
LakeflowConnectActivity(or equivalent) IR dataclass inmodels/ir.pycarrying the connector fields that matter (source system, objects, destination, connection, schedule/settings).pipeline_task+ resource (generalizing the existing Copy/Motif LFC emission).How to approach
bundle_invariants/databricks bundle validate; existing Copy/Motif LFC paths unchanged.