Surface detected ADF motifs additively in discovery inventory (#64)

[matthewmoorcroft]

What this adds

Discovery already ran the ADF motif detector for the profile report, but only retained a count. This PR adds the full detections to metadata/inventory.json as an additive, per-pipeline motifs[] field so routing can see multi-activity deterministic capabilities.

The field is deliberately separate from lineage and discovery does not collapse any activities.

pipelines[].motifs[] schema

The key is emitted only when a pipeline has at least one detected motif. Each entry is:

{
  "motif_id": "metadata_driven_bulk_copy",
  "display_name": "Metadata-Driven Bulk Copy",
  "member_task_keys": ["Lookup ETL Config", "ForEach Table In Config"],
  "databricks_replacement": "for_each_ingestion",
  "source_type_hint": "database",
  "confidence_notes": [
    "Lookup 'Lookup ETL Config' appears to read a control/metadata table"
  ]
}

• motif_id — stable detector definition id.
• display_name — human-readable motif name.
• member_task_keys — the exact source activities participating in this match; these remain individually present in activities[].
• databricks_replacement — the Databricks-native replacement associated with the definition.
• source_type_hint — optional detector hint about the source involved.
• confidence_notes — human-readable detection rationale; the detector does not invent a numeric confidence score.

The first four names align with the shared MotifAnnotation vocabulary while retaining detector-specific source_type_hint and confidence_notes.

How it works

1. The ADF discover path calls detect_motifs_by_pipeline once for every pipeline. Detection is best-effort: a detector exception is logged and treated as no motifs rather than failing discovery.
2. The same detection map is passed to both the source-neutral inventory projector and build_profile_rows, keeping the full inventory entries and the profile report's collapsible_patterns count consistent from one pass.
3. build_source_inventory projects matches beside each pipeline's activities and lineage keys. Pipelines without a match omit motifs entirely, preserving the prior shape.
4. Exact duplicates—same motif_id and the same member set—deduplicate to the first occurrence. Overlapping matches with different member sets remain separate detections.

Decoupled and non-collapsing

pipelines[].motifs is its own discovery concept. It is not nested under pipelines[].lineage, and lineage.motifs remains untouched/empty because that slot belongs to convert-time IR lineage. The discovery emitter only surfaces the match: it does not call motifs/collapser.py, does not replace member nodes, and does not remove them from activities[].

For example, a Lookup + ForEach bulk-copy motif produces one motifs[] entry while both Lookup ETL Config and ForEach Table In Config remain separate activity entries.

Why routing consumes it

Routing assesses the deterministic option for a connected component from two signals:

• activities whose strategy is already deterministic; and
• non-deterministic activities claimed by a detected motif's member_task_keys.

The component is deterministic-capable only when every activity is covered by one of those signals. Routing also surfaces the de-duplicated motif ids in options.deterministic.motifs. The inventory field is therefore a capability signal, not a request to collapse or convert anything during discovery.

Non-breaking behavior

• Existing inventory keys and values remain unchanged.
• motifs is additive and omitted when empty.
• The lineage block remains independent.
• Coverage consumers continue to read the historical activity shape.

Verification

The branch adds source-neutral emitter tests for additive/no-collapse behavior, omission when empty, decoupling from lineage, and duplicate/overlap handling, plus real ADF discover-path coverage proving motif members remain intact. The PR was verified with make test and make fmt.

This pull request and its description were written by Isaac.