Agentic insights on the unified inventory (discover phase) (#25)

[matthewmoorcroft]

What this adds

This PR adds an optional, source-neutral, agent-authored insights layer to discovery. The deterministic inventory still records what the source workflows are; an external agent reads that inventory and authors judgment about intent, target patterns, and cross-pipeline relationships. flowx contains no LLM for this step: the library validates the authored object and atomically merges it under one additive top-level insights key.

The PR is merged into integration/discovery, not main; this description documents the merged contract.

Merged inventory.json.insights schema

The complete merged block is:

{
  "schema_version": "1",
  "inventory_sha256": "<sha256 of the deterministic inventory base>",
  "overview": "Short factory-wide narrative",
  "system_recommendation": {
    "headline": "The architectural decision to make first",
    "recommended_patterns": [
      {
        "pattern": "Lakeflow Connect SQL Server connector",
        "fit": "Replaces the child extractor family",
        "simplification_pattern": true
      }
    ],
    "cascade": ["5 child extractors -> managed connector pipelines"],
    "decision_driver": "Is this connector approved for the source?"
  },
  "pipeline_insights": [
    {
      "pipeline": "IngestCustomers",
      "pattern_name": "Metadata-driven ingestion",
      "intent": "Land customer source tables",
      "databricks_pattern": "Managed ingestion",
      "recommended_patterns": [
        {
          "pattern": "Lakeflow Connect",
          "fit": "Removes the bespoke copy loop",
          "simplification_pattern": true
        }
      ],
      "conversion_notes": ["Preserve source object selection"],
      "risk_if_ignored": "The custom watermark/control-table tier carries forward"
    }
  ],
  "pipeline_relationships": [
    {
      "from_pipeline": "Orchestrator",
      "to_pipeline": "IngestCustomers",
      "lineage_edge": {
        "edge_type": "control",
        "edge_identity": "<real via_task_key>"
      },
      "relationship_summary": "Orchestrator invokes ingestion",
      "databricks_pattern": "Run job dependency",
      "risk_if_ignored": "The invocation is lost"
    }
  ]
}

Top-level fields

• schema_version — library-owned schema version.
• inventory_sha256 — library-owned fingerprint binding the insights to the deterministic inventory they describe.
• overview — optional factory-wide narrative.
• system_recommendation — optional whole-factory architectural decision.
• pipeline_insights[] — optional per-pipeline judgments.
• pipeline_relationships[] — optional cross-pipeline judgments.

Recommended patterns and simplification

system_recommendation.recommended_patterns[] and each pipeline_insights[].recommended_patterns[] are ranked, best-first lists of 1–4 objects:

{"pattern": "<real Databricks capability>", "fit": "<why it fits>", "simplification_pattern": true}

simplification_pattern: true is reserved for a distinctive capability that removes/collapses a legacy pattern—such as Lakeflow Connect, Auto Loader, declarative AUTO CDC, or system tables replacing custom logging—not a like-for-like port. The whole-system object also carries headline, optional cascade[], and optional decision_driver.

Pipeline relationships

Every relationship names real inventory pipelines and contains lineage_edge:

• edge_type: "control" annotates a deterministic control edge already present in inventory. edge_identity must exactly match that edge's via_task_key for the same (from_pipeline, to_pipeline) pair; evidence and confidence must be omitted.
• edge_type: "inferred" records a coupling deterministic lineage could not prove. edge_identity is an agent-authored descriptor, and non-empty evidence plus confidence: high|medium|low are required.

The v1 deterministic data edges are intra-pipeline/task-scoped, so an agent-observed cross-pipeline data relationship uses the inferred tier.

Author -> validate -> merge contract

1. Author: the agent reads metadata/inventory.json, its lineage, and the source artifacts. It authors only overview, system_recommendation, pipeline_insights, and pipeline_relationships. It does not supply schema_version or inventory_sha256.
2. Validate: validate_insights checks the object shape, pipeline foreign keys, ranked-pattern limits/content, exact control-edge triples, and the evidence/confidence requirements for inferred edges. Violations are aggregated so the author can fix them together.
3. Merge: only a valid object is merged. The library removes any previous insights block when computing the deterministic base fingerprint, injects schema_version and inventory_sha256, and replaces the whole block atomically. Repeating the same enrichment is byte-idempotent; a failure leaves inventory.json untouched.

Insights are descriptive and additive. This PR does not choose a deterministic/agentic route, mutate IR, or alter conversion.

Agent-facing skill structure and decision ownership

This mirrors the “Agent-facing skill structure and decision ownership” section of the local docs/agentic-conversion-design.md.

The flowx-enrich workflow is documented in skills/flowx-discover/insights.md:

1. The agent reads the deterministic inventory and enough source material to understand intent.
2. The agent decides which recommendations and ranked target patterns to propose, which patterns are genuine simplifications, and what evidence supports inferred relationships.
3. The agent calls enrich through CLI or MCP.
4. The library does not invent recommendations. It validates, fingerprints, and merges the agent-authored judgment.

This is intentionally distinct from routing: enrich owns agent-authored recommendations; the customer later owns the per-connected-component deterministic/agentic choice.

Surfaces

• CLI: python -m flowx.adapter enrich --output-dir <dir> --insights-path insights.json
• MCP: flowx("enrich", {"output_dir": "<dir>", "insights": {...}}) or insights_path
• Authoring guide: skills/flowx-discover/insights.md

Non-breaking behavior

• One additive top-level key; deterministic inventory content is preserved.
• Atomic replacement, fingerprint binding, and idempotent output.
• No embedded LLM and no conversion/routing decision.
• Validation failure writes nothing.

The branch's unit coverage exercises malformed schemas, foreign keys, control-edge resolution, inferred evidence/confidence, ranked-pattern constraints, aggregate violations, fingerprinting, atomic/idempotent merge, CLI, and MCP behavior. The acceptance commands were make test and make fmt.

[matthewmoorcroft]

Addressed both blocking review findings in d6e4afa (validation strictness only; no changes to the merge/atomicity/fingerprint/fences that reviewed clean):

• BLOCKING 1 — forbid inferred-only keys by presence. A deterministic control edge now rejects evidence/confidence when the key is present at all (was .get(...) is not None), so an explicit evidence: null / confidence: null is a violation instead of slipping through. New test: test_control_edge_rejects_present_but_null_evidence_or_confidence.
• BLOCKING 2 — aggregate per-edge errors. _validate_edge no longer early-returns on a bad edge_type/edge_identity; tier-specific checks run independently, so an inferred edge that is both missing a valid identity AND missing evidence/confidence surfaces all three errors in one pass. New test: test_inferred_edge_aggregates_identity_and_evidence_errors.

Gates: make fmt clean (ruff + mypy, 124 files); full unit suite with the mcp extra installed 1335 passed, 0 skipped (MCP wiring tests run and pass). Not merged.