Skip to content

Phase 2: First-class Lakeflow Connect ingestion IR type + preparer #89

Description

@matthewmoorcroft

Summary

Add a first-class typed Lakeflow Connect (LFC) ingestion IR type + preparer (Phase 2), upgrading LFC from an indirectly-reachable / raw-dict construct into a typed, auditable, testable IR activity. This moves agent-authored or motif-collapsed LFC targets from "opaque passthrough" (#87) into first-class deterministic, validated output.

Phase: 2 (typed IR expansion). Relates to: #87 (the passthrough escape hatch this upgrades), the cross-pipeline motif issue (its collapse target), and #85 (AST → IR).

Current status

  • There is no first-class LFC ingestion IR type. LFC is only reachable indirectly:
    • a CopyActivity flagged for the Lakeflow connector (preparer/activity_preparers/copy.py), or
    • a consolidated metadata-driven MotifActivity (preparer/activity_preparers/motif.py), which emits one LFC pipeline resource with one object per lookup row.
  • The LFC resource is emitted as a fairly raw pipeline_resources dict (bundler/dab_writer.py), and the motif config models a limited field set — richer connector settings / per-object transforms aren't typed.
  • So an agent (or a motif) can produce an LFC pipeline, but not as a typed, schema-validated construct.

What needs to change

  • A typed LakeflowConnectActivity (or equivalent) IR dataclass in models/ir.py carrying the connector fields that matter (source system, objects, destination, connection, schedule/settings).
  • A preparer lowering it to the LFC pipeline_task + resource (generalizing the existing Copy/Motif LFC emission).
  • Serde + rehydration round-trip; registry / dispatch wiring.
  • Optional: richer connector config beyond the current consolidation shape.

How to approach

  • Follow the "Adding a New Deterministic Translator" pattern (IR dataclass → preparer → code_generator if needed → registry), scoped to the preparer/packaging side (this is a target type; discovery does not need it).
  • Reuse the existing LFC resource-emission logic from the Copy/Motif preparers as the lowering target.
  • Verify: a typed LFC activity round-trips through serde and packages to a valid LFC pipeline resource that passes bundle_invariants / databricks bundle validate; existing Copy/Motif LFC paths unchanged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions