Phase 1 stack #5: routing + in-engine agentic conversion - #97
Open
matthewmoorcroft wants to merge 15 commits into
Open
matthewmoorcroft wants to merge 15 commits into
matthewmoorcroft wants to merge 15 commits into
Conversation
Group pipelines into connected components over control lineage and recommend a deterministic/agentic strategy per component (routing.py), recording the user's decision as an additive conversion plan (models/conversion_plan.py). Drive the in-engine agentic conversion flow (route_agentic.py) that routes, alters and fills the remaining gaps, with plan-bound gap combine/dedupe scoped to the routed pipeline's own tasks and mandatory bundle validation. Expose route and fill through the adapter CLI and MCP, and update AGENTS.md. Consolidates PRs #77 (per-connected-component routing agent), #92 and #93 (in-engine agentic conversion flow + CLI/MCP parity + cross-review fixes). Co-authored-by: Isaac <no-reply@databricks.com>
Propagate the flowx-enrich skill (from #95) up the stack ahead of adding the routing skill changes on this branch. Co-authored-by: Isaac <no-reply@databricks.com>
…ills Adds the first-class flowx-route skill (interactive route → decide → record plan → fill routed-agentic groups) and updates flowx-migrate to the discover → enrich → route → convert/fill → package flow. Registers flowx-route and the already-present flowx-resolve-airflow-gaps in the manifest, so it now lists both flowx-enrich (from the agentic-insights PR) and flowx-route. Convert runs as the deterministic baseline before routing; no plain convert re-run after routing (only the additive merge/combine fills), so routed-agentic placeholders survive to package. Customer decides the route per component; the agent presents options and authors the approved fill. Skill/manifest docs only; no src/ changes. Co-authored-by: Isaac <no-reply@databricks.com>
…rder 1. Airflow per-gap fill: convert --merge-agentic is ADF-only. Airflow's --merge-agentic is disabled and exits 2, so direct Airflow gaps to the flowx-resolve-airflow-gaps skill; removed the false --source airflow suggestion (route + migrate). 2. Merge behavior: describe it accurately — merge_agentic_results replaces the FIRST task whose name matches activity_name (matched by name, not verified to be a PlaceholderActivity). Removed the unfounded "replaces only placeholder tasks" guarantee (route + migrate). 3. Migrate sequencing: make the whole skill consistently linear — discover -> enrich -> convert (deterministic baseline) -> route (decide + edit) -> fill agentic gaps -> package. Fixed frontmatter, intro, and phase list; reordered and renumbered the workflow steps (convert now Step 5 before route Step 7, fill Step 8) so it reads top-to-bottom in execution order with no forward/back jumps; reordered the MCP command sequence to put convert before route. Grounded against adapter/__main__.py, sources/airflow/convert.py, and ir_serde.py. Skill-doc edits only; no src/ changes. make fmt + make test green. Co-authored-by: Isaac <no-reply@databricks.com>
merge_agentic_results / _find_and_replace_task (ir_serde.py) replaces the matched task with the authored task definition verbatim and does not set or mutate any `status` field. Reword the flowx-route merge note to describe the in-place replacement accurately and drop the false "status becomes translated" claim, consistent with the name-match description above it. Skill-doc edit only; no src/ changes. make fmt + make test green. Co-authored-by: Isaac <no-reply@databricks.com>
…e claim Airflow discovery does not yet emit control lineage or motifs (sources/airflow/discover.py), so it cannot form routing components, and convert --merge-agentic --source airflow is disabled (sources/airflow/convert.py). The route -> edit -> fill agentic-conversion flow is therefore ADF-only today; Airflow aligns via #63. - flowx-route: removed the false "Airflow (RunJob) emit control edges, source-neutral routing" claim; added an "ADF only (current scope)" note near the top directing Airflow agentic gaps to flowx-resolve-airflow-gaps. - flowx-migrate: added the ADF-only scope caveat to the Context intro and Step 7 (route); deterministic discover/convert/package still described as working for both sources. Skill-doc edits only; no src/ changes. make fmt + make test green. Co-authored-by: Isaac <no-reply@databricks.com>
…A-grounded channel FIX 2: apply_combine_fill now asserts every agent-authored pipeline carries tags.source == 'adf' (routing/agentic is ADF-only). A missing/wrong tag fails the combine closed (nothing written) with a clear message, instead of being accepted at combine and only rejected later at package preflight. The required tag is now documented in the flowx-route AgenticComponentActivity / combine authoring example. FIX 3: Make combine idempotent. The recorded plan still lists a component's members after a prior combine collapsed them out of the report, so the plan/membership check passes on a re-run; a naive second run then re-appends the authored pipeline (duplicate). Detect the already-combined state (members gone from the report, authored pipeline(s) already present) and no-op (already_combined=true). Regression test: running combine twice with the same members + authored pipelines yields a single authored pipeline. FIX 5: Persist a compact metadata/route_audit.json (routed components + decisions + gaps introduced) before package prunes .work/, so the "what did routing change?" trail survives the default prune. Additive and non-breaking: it writes nothing when no conversion_plan.json was recorded, so the no-route path stays byte-identical. .work/ pruning is unchanged. FIX 8b: In flowx-route's fill authoring guidance, default the Lakeflow pipeline/Connect resource channel to 'current' (stable/GA); only use 'preview' with a cited GA-status justification + confirmed workspace availability. Require the same GA-vs-Preview verification/citation as flowx-enrich (FIX 8a) before recommending any connector / Lakeflow Connect pattern. Co-authored-by: Isaac <no-reply@databricks.com>
…entic Bring the discover warning/units, enrich edge_type guidance, and GA-grounding into the tip so it carries all Phase-1 fixes (1, 7, 4, 8a, 2, 3, 5, 8b). Co-authored-by: Isaac <no-reply@databricks.com>
…name sets
Cross-review BLOCKING 1: the prior FIX 3 detection inferred "already combined"
from pipeline-name sets, which broke two ways:
(a) re-running combine with the SAME --members but a differently-named authored
replacement fell through to combine_group_fill (members already gone, so
nothing dropped) and appended a second authored pipeline -> duplicate;
(b) an authored pipeline whose name collided with a former member defeated the
detection entirely.
Fix: each successful combine now stamps an additive `_combine_provenance` marker
onto the report recording (component_id, inventory_sha256). Idempotency keys off
that recorded state -- never off comparing authored names to member names -- so a
re-run is detected regardless of how the authored replacement is named or whether
a name collides. The marker lives in the report, so a fresh convert (which
rewrites the report) clears it and combine re-applies after a genuine re-convert.
The marker is an extra top-level report key: ignored by the package phase (not a
recognized report shape key) and by ir_serde (which reads per-pipeline IR, not the
report wrapper), so the convert->package contract shape is unchanged.
Regression tests cover both failure modes: (a) a differently-named authored re-run
does not duplicate, and (b) an authored name colliding with a former member is
still detected as already-combined.
Co-authored-by: Isaac <no-reply@databricks.com>
…g-agentic Bring the --adf-source-path warning correction and the enrich GA-grounding rework into the tip, so it carries all cross-review re-fixes (FIX 1, 8a) alongside its own FIX 3 idempotency rework. Co-authored-by: Isaac <no-reply@databricks.com>
…nto stack/routing-agentic Co-authored-by: Isaac <no-reply@databricks.com>
…osure
Mirror the per-pattern release_state / release_state_source onto AgenticPattern
(they already ride into the agentic option via the {**pattern} spread; this makes
the parity explicit and typed). Add a neutral, per-pattern release_disclosures list
to AgenticOption: one entry per recommended pattern whose state is disclosed
(public_preview / private_preview / beta), carrying the pipeline, pattern, state, a
factual label, and a human-readable message. It is pure factual labelling, not a
warning:
- ga / unknown -> silent (no entry; unknown is treated exactly like ga);
- public_preview -> label "Public Preview (production-ready)";
- private_preview -> label "Private Preview";
- beta -> label "Beta".
The route skill points its channel guidance at the structured release_state and
reflects the neutral labels. The separate recommendation-eligibility rule (treat
Private Preview as doNotSuggest without confirmed enrollment) and the
verify-before-recommend grounding rule are unchanged.
Co-authored-by: Isaac <no-reply@databricks.com>
… stack/routing-agentic Co-authored-by: Isaac <no-reply@databricks.com>
…ack/routing-agentic Co-authored-by: Isaac <no-reply@databricks.com>
…lease_state examples) into stack/routing-agentic Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the Phase 1 stack (top). Base:
stack/agentic-component. (Content of the former #92 + #93.)What this adds — the in-engine agentic conversion flow
route(one command): groups pipelines by connected component over control lineage; presentsoptions.deterministic+options.agentic(incl. simplification) per group; takes the decision interactively (TTY) /--plan-path/ stdin / MCP; records the fingerprint-boundmetadata/conversion_plan.json(ownership split: agent authors onlydecision; library recomputes members/recommended/options); and edits the report so routed-agentic groups become standardizedPlaceholderActivity+AgenticGap.merge_agentic_results; cross-pipeline COMBINE (fill-agentic combine) swaps a routed group's N pipelines for the agent-authored M (e.g. multi-pipeline → one LFC), plan-membership-bound and structurally validated (check_bundle_dir) before writing.convertis untouched; no decision / all-deterministic = today's behavior exactly.Release-state surfacing in the agentic option (neutral disclosure)
Builds on the structured
release_statefield added in #95 and surfaces it per agentic conversion option.What.
AgenticPatternmirrorsrelease_state+release_state_sourcefor parity (they already ride into the option via the{**pattern}spread; this makes the parity explicit and typed).AgenticOptiongains a neutralrelease_disclosureslist.How it works.
release_disclosurescarries one entry per recommended pattern whose state is disclosed —{pipeline, pattern, release_state, label, message}:gaandunknown→ silent (no entry;unknownis treated exactly likega, since the two are indistinguishable in the surfaced output);public_preview→ label "Public Preview (production-ready)";private_preview→ label "Private Preview";beta→ label "Beta".It is pure factual labelling — no alarm flag, severity, or warning framing. The
flowx-routeskill's channel guidance now points at the structuredrelease_stateand reflects these neutral labels; the separatedoNotSuggest-Private-Preview recommendation-eligibility rule and the verify-before-recommend grounding rule are unchanged. The no-route inventory/report path stays byte-identical.This pull request and its description were written by Isaac.