Make discover → enrich → route a first-class phase1 skill flow - #99
Closed
matthewmoorcroft wants to merge 2 commits into
Closed
matthewmoorcroft wants to merge 2 commits into
matthewmoorcroft wants to merge 2 commits into
Conversation
Wire the phase1 skill experience so enrichment runs by default and routing is invocable as a skill, without changing any CLI command semantics (the "always enrich" guarantee lives at the SKILL layer only; `discover`-the-command stays deterministic, headless, always-valid). - Add a first-class `flowx-enrich` skill wrapping the agent-authors-insights → `enrich` validate/merge loop. Lifts the insights authoring guide out of flowx-discover (moved `insights.md`) and reframes it: no LLM in the library, default-on but skippable for a deterministic-only pass. - Reframe `flowx-discover` Step 3 to chain into enrich by default (points at flowx-enrich), with an explicit deterministic-only skip path; drop the "optional / changes nothing" skip-by-default framing. - Add a first-class `flowx-route` skill covering interactive `route` (recommend + per-connected-component deterministic/agentic decision, plan shape + validation rules), and both fills: per-pipeline via `convert --merge-agentic` and cross-pipeline via `fill-agentic combine`, including how to author `AgenticComponentActivity` nodes. Flags verified against the adapter CLI and MCP server. - Orchestrate discover → enrich (default) → route → convert/fill → package in `flowx-migrate` (steps 3.5, 3.6, 5.05 + MCP command sequence + workflow ref), non-breaking. - Register flowx-enrich and flowx-route in the plugin manifest (and add the already-present flowx-resolve-airflow-gaps so the list matches the skills/ directory). make test: 1410 passed, 5 skipped. make fmt: clean (ruff + mypy). Co-authored-by: Isaac <no-reply@databricks.com>
Blocking fixes from the codex cross-review: 1. Add mandatory --source adf to the per-pipeline convert --merge-agentic command (route + migrate); the convert phase exits 2 without it. 2. Correct the route MCP source-path param: use adf_source_path / airflow_source_path (generic source_path is ignored by the server). 3. Fix the destructive migrate sequence: convert now runs as the deterministic baseline BEFORE routing (or is triggered by route in-process); no second plain convert after routing (it would overwrite .work/translation_report.json and erase routed-agentic placeholders). Only the additive convert --merge-agentic / fill-agentic combine runs after routing. Updated the route flow diagram to match. 4. Mark plan component_id as required (validator requires a non-empty string that matches the computed component). 5. Fix decision ownership: the CUSTOMER decides deterministic-vs-agentic per component; the agent presents options and serializes only the approved decision, then authors the agentic fill. Non-blocking: 6. insights.md: recommended_patterns ordering is a convention, not validator-enforced (the validator checks count + fields only). 7. flowx-migrate frontmatter/intro/phase list now reflect the five-step discover -> enrich -> route -> convert/fill -> package flow. Skill/manifest docs only; no src/ changes. make fmt + make test green. Co-authored-by: Isaac <no-reply@databricks.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Makes the phase1 flowx discover → enrich → route experience first-class, with enrich running by default at the skill layer. No CLI command semantics change — the "always enrich" guarantee lives only in the skills;
discover-the-command stays deterministic, headless, and always-valid.New skills
flowx-enrich(skills/flowx-enrich/SKILL.md+insights.md) — a first-class skill wrapping the agent-authors-insights →enrichvalidate/merge loop. The insights authoring guide was lifted out of flowx-discover (movedinsights.mdhere) and reframed: no LLM in the library (author → validate → merge, sha-bound), default-on but skippable for a deterministic-only pass.flowx-route(skills/flowx-route/SKILL.md) — covers interactiveroute(recommend + per-connected-component deterministic/agentic decision, plan shape + validation rules) and both fills: per-pipeline viaconvert --merge-agenticand cross-pipeline viafill-agentic combine, including how to authorAgenticComponentActivitynodes.Reframed / updated
flowx-discover— after the deterministic inventory, Step 3 now chains into enrich by default (points atflowx-enrich) with an explicit deterministic-only skip path. The old "optional / changes nothing" skip-by-default framing is removed.flowx-migrate— end-to-end orchestration now runs discover → enrich (default) → route → convert/fill → package (Steps 3.5, 3.6, 5.05 + the MCP command sequence + a workflow reference note), non-breaking..claude-plugin/plugin.json— registersflowx-enrichandflowx-route(and the already-presentflowx-resolve-airflow-gaps), so the manifest matches theskills/directory and all skills load.Design constraints honored
enrichCLI command is not merged intodiscover;discover-the-command stays deterministic/headless/always-valid.enrichis agentic only in that the agent authors the insights; the library has no LLM — it validates + merges (sha-bound). The skills state this accurately.src/flowx/adapter/__main__.py,src/flowx/sources/adf/translate.py, andsrc/flowx/mcp/server.py(enrich--output-dir/--insights-path/--out; route--output-dir/--plan-path/--source/--source-path/--out;fill-agentic combine --output-dir/--members/--pipelines-path/--out;convert --merge-agentic --report/--agentic-results/--output; MCP commandsenrich/route/fill_agentic/merge_agenticand their param names).Verification
make fmt— clean (ruff format, ruff check, mypy: no issues in 128 files).make test— 1410 passed, 5 skipped.namematches directory and is listed inplugin.json.flowx-discoverno longer frames enrich as an optional skip-by-default footnote;insights.mdnow lives underflowx-enrich, notflowx-discover.This pull request and its description were written by Isaac.