Emit discovery-AST lineage into inventory.json (#62) - #81
Merged
Merged
Conversation
The source-agnostic inventory emitter serialized only
{source, source_dir, pipelines, summary} and dropped each graph's
SourceGraph.lineage, so the control/data edges #62b derived never reached
inventory.json.
Add an additive per-pipeline `lineage` block, placed beside each pipeline's
`activities` to mirror the shared discovery serde (where lineage is a per-graph
field). It is produced by the one shared serialiser
(flowx.ir_serde.lineage_to_dict, the same one discovery_serde consumes), so an
emitted block is byte-identical to the serde's and round-trips through
source_graph_from_dict. A graph with no derived lineage omits the key entirely,
leaving the historical consumer keys (coverage/results/dashboard/mcp read only
source/pipelines/activities/summary) untouched.
On the ADF fixtures this surfaces the 11 control edges / 0 data edges #62b
computed. Tests: a round-trip assertion via the shared serde, a
backward-compat assertion that a lineage-less graph omits the key, and an
aggregate count over the fixtures; the existing coverage golden still passes.
Co-authored-by: Isaac <no-reply@databricks.com>
This was referenced Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary: Surface each SourceGraph lineage block in the unified inventory without changing any pre-existing inventory keys.
Depends on: #84 and the main re-lands of #79 and #80.
Blocks: #62, #82, #77, #85.
Current status
This PR is merged into integration/discovery, not main (merge commit c42e47b). Its checks passed when merged. The staging code exposes lineage, but main does not.
The concrete staging change is in src/flowx/discovery_inventory.py:build_source_inventory: when graph.lineage is present, the per-pipeline entry gains a lineage block serialized by src/flowx/ir_serde.py:lineage_to_dict. A graph with lineage=None omits the key. src/flowx/discovery_serde.py:source_graph_from_dict provides the round-trip seam.
This preserves all existing pipeline/activity/summary fields. The ADF fixture inventory on staging carries the control edges produced by #80; reporting/coverage.py continues to read only its established keys.
What needs to change
Re-land this small emitter change on main after #79/#80 are re-landed on #84. Keep it additive and reuse the single shared lineage serializer; do not introduce a second inventory-specific encoding.
How to approach
Apply the conditional lineage projection in discovery_inventory.build_source_inventory and retain the existing omission behavior for lineage=None.
Acceptance/verification:
This is an additive inventory field only.