Preserve conversion documents and expand validated graph formats

.changeset/conversion-fidelity.md

@@ -0,0 +1,23 @@
+---
+"@statelyai/graph": minor
+---
+
+Expose separate Graph, native semantic and source-document preservation assessments in format support entries. Correct unrestricted round-trip claims and document known conversion limits. Existing feature keys remain available; the additive preservation field is optional for third-party entries.
+
+Repair existing converter semantics: scoped XML attributes/defaults and edge modes, native GEXF weight/mutual edges, DOT declaration/cluster/port behavior, GML standard identity options, structured metadata/edit precedence, collision-free ELK endpoints and explicit hyperedge rejection, JGF version support, and adjacency/TGF boundaries.
+
+Retain supported D2 key/reference/source abstractions and diagram annotations. Fix Mermaid styles/properties/edge identity, ordered sequence notes/lifecycle with stable note references, official participant configuration, ER entities/constraints, block shapes/columns and class/tree metadata. New metadata fields remain JSON-serializable; legacy input remains readable. Source annotations may be reformatted, and unsupported event references/topology fail explicitly.
+
+
+Add `@statelyai/graph/document` and explicit native `*DocumentConverter` registrations. Plain JSON-serializable envelopes retain exact original text/native JSON values after persistence, independently of partial or unavailable semantic projections. Supported edits patch source with reparse proof; unsupported/conflicting edits fail explicitly. Custom codecs, including VisualGraph codecs, can register without eager optional peer imports. Source validation rejects non-JSON values rather than silently dropping them. Explicit lossy regeneration reparses its output and reports unrepresentable Graph field paths or an unavailable projection, alongside possible source loss. Expand the optional ELK peer range through 0.12 after runtime/type package verification.
+
+
+Add a separate Node 24 native-validation suite with exact dev-only parser/compiler/schema dependencies and converter coverage gates. Independently validate external and emitted profiles, with native database/compiler semantic assertions. Correct native Mermaid class multiplicity placement, nested state transition scope, and literal backslash encoding/import counts discovered by those oracles. Core runtime support remains Node 20+; validators are absent from published runtime imports.
+
+Add dependency-free `csv`, `graphology`, `graphlib` and `networkx` subpaths with explicit document registrations. CSV/TSV use separate node/edge tables, configurable typed columns and default JSON cells to distinguish missing/null/empty values. Bounded document cell edits preserve unrelated source columns and quoting. Native JSON adapters preserve selected attribute metadata and reject unsupported uniform-direction mappings. Validate emitted CSV cells with pinned csv-parse and NetworkX profiles with isolated Python oracles; these checks establish scoped interoperability, not unrestricted semantic losslessness.
+
+Add scoped Mermaid Architecture, Requirement, Sankey, GitGraph, C4 and UseCase converters with serializable document registrations and native Mermaid12.1 database proof. Architecture sides map ports; GitGraph preserves ordered DAG actions and canonical main-branch configuration. Bounded label patches preserve comments; unsupported syntax/edits retain source or reject explicitly. Scope Sankey to the native parser's printable ASCII/trimmed-name profile; omit Requirement empty text tokens that native syntax rejects. Repair injective empty Mermaid IDs and empty-parent hierarchy references.
+
+Add adjacency matrix, Matrix Market, Graphviz JSON, draw.io, Pajek and scoped PlantUML subpaths, explicit mappings/aggregation/direction policies, source document registrations and independent native oracles. draw.io compression uses optional fflate; native validation tools remain development-only, including checksum-pinned PlantUML/JRE and isolated Python dependencies.
+
+Reject native display/wire strings that independent oracles replace: Mermaid NUL/lone-surrogate labels and D2 lone-surrogate labels/IDs. Preserve supported astral strings, D2 compiler-level NUL labels, safe encoded Mermaid IDs, JSON-escaped opaque data and exact unprojectable source strings.

.gitattributes

@@ -0,0 +1,11 @@
+# Preserve official vendored schema bytes; their upstream whitespace is hash checked.
+tests/native/fixtures/graphml/1.0/*.xsd -text -whitespace
+tests/native/fixtures/gexf/1.2draft/*.xsd -text
+tests/native/fixtures/gexf/1.3/*.xsd -text
+tests/native/fixtures/drawio/mxfile.xsd -text
+tests/native/fixtures/graphviz/schema/graphviz_json_schema.json -text -whitespace
+tests/native/fixtures/graphviz/schema/LICENSE.txt -text -whitespace
+
+# Decoded draw.io payload snapshots intentionally retain source whitespace and EOF.
+tests/native/fixtures/drawio/page-first.mxgraph.xml -text -whitespace
+tests/native/fixtures/drawio/page-second.mxgraph.xml -text -whitespace

.github/workflows/ci.yml

@@ -37,6 +37,12 @@ jobs:
       - name: Typecheck
         run: pnpm typecheck
 
+      - name: Typecheck Repository
+        run: pnpm typecheck:repo
+
+      - name: Check API Conventions
+        run: pnpm check:conventions
+
       - name: Check Generated Artifacts
         run: pnpm check:generated
 
@@ -71,3 +77,41 @@ jobs:
 
       - name: Validate Published Package Surface
         run: pnpm validate:package
+
+
+  native-validation:
+    name: Native Format Validation
+    runs-on: ubuntu-latest
+    env:
+      GRAPH_NATIVE_PYTHON: ${{ github.workspace }}/.venv-native/bin/python
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v4
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: 24
+          cache: pnpm
+      - name: Install Dependencies
+        run: pnpm install --frozen-lockfile
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+          cache-dependency-path: requirements/native.txt
+      - name: Install Native Python Oracles
+        run: |
+          python -m venv .venv-native
+          .venv-native/bin/python -m pip install -r requirements/native.txt
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '21'
+      - name: Install Pinned PlantUML Oracle
+        run: pnpm setup:native:plantuml
+      - name: Validate Native Formats and Converter Coverage
+        run: pnpm test:coverage

.gitignore

@@ -6,3 +6,10 @@ ASSESSMENT.md
 .thoughts
 
 docs/superpowers
+
+coverage/
+
+.venv-native/
+__pycache__/
+
+.tools-native/

README.md

@@ -54,6 +54,97 @@ if (path) {
 }
 ```
 
+## Format preservation
+
+<!-- format preservation capabilities and exports derived from src/formats/support.ts and src/format-support.ts -->
+
+The table describes bare Graph converters. Format mappings, Graph round-tripping,
+native semantics and source preservation are separate contracts. No converter currently guarantees 100% losslessness for
+arbitrary Graphs or complete native documents. Selected-field tests characterize
+supported mappings; they do not establish conformance to an entire format.
+
+| Formats | Graph → format → Graph | Native → Graph → native | Source details |
+| --- | --- | --- | --- |
+| Adjacency list, Cytoscape, D3, DOT, ELK, GEXF, GML, GraphML, JGF, TGF, xyflow | Partial | Partial | None |
+| CSV/TSV, Graphology export JSON, Graphlib JSON, NetworkX node-link JSON, adjacency matrix, Matrix Market, Graphviz JSON, draw.io, Pajek, scoped PlantUML | Partial | Partial | None |
+| Edge list | Partial | Full for endpoint tuples | None |
+| D2 | Partial | Partial | Partial comments |
+| Mermaid architecture, block, C4, class, ER, flowchart, GitGraph, ishikawa, mindmap, requirement, Sankey, sequence, state, UseCase | Partial | Partial | Partial annotations |
+
+Graph preservation includes JSON-serializable fields, optional-field presence and
+entity order. Native preservation includes format-specific topology, attributes
+and semantics. Source preservation additionally covers comments, declaration
+order, spelling and whitespace. Adapter metadata can carry selected Graph fields
+without making them native features. Use a GraphDocument when those details matter.
+
+Additional adapters: [adjacency matrix](src/formats/adjacency-matrix/README.md),
+[Matrix Market](src/formats/matrix-market/README.md),
+[Graphviz JSON](src/formats/graphviz-json/README.md), [draw.io](src/formats/drawio/README.md),
+[Pajek](src/formats/pajek/README.md), [PlantUML](src/formats/plantuml/README.md).
+
+```ts
+import { getFormatSupportEntry } from '@statelyai/graph/format-support';
+
+const support = getFormatSupportEntry('graphml');
+console.log(support?.preservation); // { graph: 'partial', native: 'partial', document: 'none' }
+```
+
+The existing `features` API remains available; `features.roundTrip` describes the
+legacy Graph round-trip assessment. `preservation` is optional for third-party
+entries and present on every built-in entry.
+
+<!-- repaired native semantics and remaining limits from src/formats/support.ts -->
+
+Native imports now retain supported key/default/namespace semantics, edge modes,
+weights, hierarchy, ports and structured metadata. DOT declarations and clusters,
+ELK endpoint identity, JGF versions, D2 source abstractions and Mermaid ordered sequence events,
+properties and entity declarations have dedicated regression coverage.
+Canonical and visible native edits take precedence over unchanged source hints
+where the adapter can resolve the mapping.
+
+Source-only annotations remain a separate limitation: diagram comments,
+frontmatter and directives may be reformatted; XML/DOT comments and complete
+declaration order are not guaranteed. Unsupported Graph topology such as
+hyperedges is rejected instead of silently truncated. Independent native validators cover selected emitted profiles. Broader structural source editing remains bounded by each document adapter.
+
+## Native documents
+
+<!-- document contract and exports from src/formats/document/index.ts and native converter modules -->
+
+```ts
+import { graphMLDocumentConverter } from '@statelyai/graph/graphml';
+
+const document = graphMLDocumentConverter.from(xml);
+const persisted = JSON.parse(JSON.stringify(document));
+const exactSource = graphMLDocumentConverter.to(persisted);
+```
+
+CSV/TSV use explicit node and edge tables with a serializable schema; default optional cells encode JSON, including visible labels. Graphology, Graphlib and NetworkX subpaths convert their native JSON profiles without runtime library dependencies. Mapping limits: [CSV/TSV](src/formats/csv/README.md), [Graphology](src/formats/graphology/README.md), [Graphlib](src/formats/graphlib/README.md), [NetworkX](src/formats/networkx/README.md).
+
+Every native converter subpath offers a `*DocumentConverter` registration. The
+`@statelyai/graph/document` factory also accepts explicit custom codecs, including
+VisualGraph codecs. Documents retain exact original text/native JSON values on
+no-op export, including comments, extensions and unsupported constructs, after
+JSON persistence. Projection can be partial or unavailable and carries explicit
+diagnostics. This guarantees source retention, not full semantic interpretation.
+
+Supported Graph edits patch native source and must pass reparse proof. Unsupported
+or conflicting edits throw instead of replaying stale source. Native source edits
+remain authoritative when Graph is unchanged. Regeneration is an explicit,
+diagnostic-bearing operation that reports differing Graph fields after reparse,
+as well as possible source loss. See [document API](src/formats/document/README.md)
+for the validated JSON domain and editing boundary.
+
+## Native validation
+
+<!-- native-validation scripts and CI from package.json and .github/workflows/ci.yml -->
+
+`pnpm test:native` runs pinned native parsers, compilers, schemas and runtime
+oracles on Node 24. `pnpm test:coverage` enforces measured converter coverage
+floors. These tools are dev-only; ordinary core tests still cover Node 20/22/24.
+Selected native assertions supplement conversion regressions without promising
+complete language coverage. See [native validation](docs/native-validation.md).
+
 ## Optional peer dependencies
 
 The core (`@statelyai/graph`) and the pure-JSON formats have no runtime
@@ -65,6 +156,7 @@ for the peer — install it and the import works.
 | Subpath import | Peer dependency to install |
 | --- | --- |
 | `@statelyai/graph/dot` | `dotparser` |
+| `@statelyai/graph/drawio` | `fflate`, `fast-xml-parser` |
 | `@statelyai/graph/graphml`, `@statelyai/graph/gexf` | `fast-xml-parser` |
 | `@statelyai/graph/layout/cytoscape` | `cytoscape` |
 | `@statelyai/graph/xstate` | `xstate` |

docs/adr/003-serializable-native-documents.md

@@ -0,0 +1,17 @@
+# Serializable native documents
+
+Status: accepted, additive API.
+
+Graph models editable graph semantics. It cannot represent every comment, XML extension, unsupported hyperedge, multiple native graph, or syntax spelling. Existing format converters continue returning Graph and keep their existing defaults.
+
+`@statelyai/graph/document` adds a plain `GraphDocument<TSource, TGraph>` envelope. It stores native source, a possibly partial Graph projection, independent native/Graph baselines, diagnostic records and optional serializable adapter state. Explicit factory registrations avoid importing optional peers through the core module. VisualGraph codecs retain their stronger projected type.
+
+Source belongs to the JSON value/string domain. Validation rejects nonfinite numbers, undefined source values, functions, symbols, BigInt, class instances, accessors, sparse arrays and cycles before cloning. Object alias identity and prototypes are outside this contract. Optional undefined Graph object fields compare using their JSON view. No-op output exactly preserves source strings or native JSON values, including unknown fields, key order and value presence, after JSON persistence. This guarantees source retention, not complete native semantic interpretation.
+
+Native source edits are authoritative when Graph remains unchanged. Graph edits compare against the baseline projection, then merge with a fresh native projection by stable entity ID. Different mapped fields merge; conflicting changes to the same field return diagnostics. Object-valued Graph fields currently conflict conservatively as whole fields. Entity order changes count as structural changes. Unsupported edits never replay stale source.
+
+`.to(document)` remains strict. A verified format-specific patch must reproduce the merged requested projection when reparsed. Without a patch, structural support, a unique lexical span, or a successful proof, export throws an ordinary Error carrying diagnostics. Explicit `getGraphDocumentUpdate(document, converter, { mode: 'regenerate' })` may regenerate source and always returns a loss warning covering source and Graph details. Regenerated source is reparsed; differing Graph field paths or an unavailable projection receive additional warning diagnostics. Projection failures retain source with `graph: null`; strict import throws.
+
+Comparison hooks may normalize only verified adapter provenance. Editable user data and foreign reserved-key payloads remain observable. A comparator that removes private state needs a separate `getComparisonState` extractor to permit edits in normalization-altered fields; otherwise those field edits are conservatively rejected. Stored adapter state and baseline projection provenance must match the baseline native source. Import hooks may expose one editable label where a native format duplicates it in configuration, while retaining the complete native configuration in source.
+
+Initial text editing supports bounded, verified label spans, preserving unedited source text. JSON adapters update supported stable entity fields while retaining native extensions. This is an extensible editing boundary, not a universal syntax tree or native validation API. Native parsers and independent semantic oracles remain separate work.