feat: decoupled parser adapter architecture for continuous Dart SDK version support #96

Activity

1. caverav commented on Aug 27, 2026

   

   caverav

   on Aug 27, 2026

   OwnerAuthor

   More actions

   cc @kalixtez
2. caverav commented on Aug 28, 2026

   

   caverav

   on Aug 28, 2026

   Last edited by caverav

   OwnerAuthor

   More actions

   Recomendation

   This issue identifies the correct architectural boundary: keep one version-independent flutterdec core and isolate Dart snapshot parsing behind an adapter boundary.

   One important refinement: the exact snapshot hash must remain the compatibility key, but it should not automatically define a separate parser implementation or executable. Parser code should be versioned by snapshot grammar family. Layout constants and other data should be versioned by profile. Exact hash records should provide the compatibility evidence connecting them.

   The target architecture should be:

   Target binary
       |
       v
   Loader extracts SnapshotIdentity
       |
       v
   Exact compatibility registry lookup
       |
       +-- parser family
       +-- data profile
       +-- verified adapter artifact
       |
       v
   Adapter protocol
       |
       v
   ProgramModel
       |
       v
   Strict semantic validator
       |
       v
   Version-independent core
   

   Unknown identities must use an explicit machine-code-only fallback. They must never select a parser by proximity, Dart semantic version, filename, or heuristic hash matching.

   1. Compatibility identity

   The registry should match an exact identity containing:

   • the 32-character lowercase hexadecimal snapshot hash;
   • snapshot kind, such as FullAOT;
   • target architecture;
   • normalized layout-affecting feature tuple, including compressed pointers;
   • optionally, a digest of the normalized feature set.

   The Dart snapshot hash is an MD5 digest represented as 32 hexadecimal characters. It is distinct from the snapshot magic number and from the Dart semantic version.

   dart_version and flutter_version should remain informational metadata. They must not determine parser compatibility.

   Dart generates this hash from VM snapshot serializer sources, and the VM performs exact snapshot-version equality checks when loading a snapshot:

   • https://github.com/dart-lang/sdk/blob/3.11.1/tools/make_version.py
   • https://github.com/dart-lang/sdk/blob/3.11.1/runtime/vm/app_snapshot.cc#L9337-L9366

   A changed hash means that compatibility cannot be assumed. It does not prove that the serialization grammar changed. Conversely, an unchanged or reused profile is not enough to authorize parsing without an exact registry record.

   2. Separate parser families, profiles, and compatibility records

   These concepts should remain independent.

   Parser family

   A parser family owns behavior that depends on serialized snapshot grammar:

   • clustered serialization;
   • reference encoding;
   • cluster ordering;
   • ReadAlloc and ReadFill;
   • object roots;
   • dispatch tables;
   • post-load behavior;
   • version-specific parser logic.

   One parser family may support multiple exact snapshot identities, but only after real fixtures and invariant checks prove that the implementation remains valid.

   Data profile

   A profile contains data-only layout information:

   • CIDs;
   • object-header rules;
   • field offsets;
   • alignment;
   • pointer width;
   • pool geometry;
   • instruction-table layout;
   • feature-dependent constants.

   Profiles should be independently updateable and content-addressed. They should not be compiled into the core with include_str!; adding or correcting a profile should not require rebuilding the main decompiler.

   Compatibility record

   Every supported hash should have an exact registry record containing:

   snapshot_hash
   snapshot_kind
   target_arch
   features_digest
   dart_version                 # informational
   flutter_version              # informational
   parser_family_id
   profile_id
   profile_sha256
   protocol_major
   model_major
   compatibility_evidence
   trust_tier
   host artifact variants:
     host_os
     host_arch
     artifact_sha256
     artifact_size
     provenance
   

   Multiple exact records may point to the same parser artifact and profile. A new hash therefore does not automatically require a new executable.

   The registry is the authority. Adapter output must not be allowed to declare itself authoritative merely by returning a matching hash or an adapter_kind string.

   3. ProgramModel contract

   The current Schema v3 contract is too weak for this boundary. Provenance, capabilities, unavailable values, and semantic validation change the meaning of the model, not just its JSON shape.

   Because this project is still pre-1.0, a clean ProgramModel v4 cutover is preferable to permanently accepting v2, v3, and v4.

   ProgramModel v4 should include:

   • explicit schema name and major/minor version;
   • producer ID and producer version;
   • adapter artifact digest and trust tier;
   • observed snapshot identity;
   • snapshot kind and normalized features;
   • input region names, sizes, virtual addresses, and SHA-256 digests;
   • selected compatibility-record ID and digest;
   • parser-family ID and digest;
   • profile ID and digest;
   • per-domain capability state:
     • complete;
     • partial;
     • unavailable;
   • provenance for facts that are exact, derived, or heuristic;
   • stable IDs and referential integrity;
   • code ranges separate from semantic function records;
   • diagnostics and parser limitations;
   • one controlled extensions object.

   Placeholder strings must not masquerade as semantic values. For example, an unresolved class, library, owner, or name should be represented as unavailable with a diagnostic, not as an apparent exact name such as <unresolved ...>.

   Likewise, heuristic pool information must not be assigned confidence: 1.0. Use typed provenance such as exact, derived, and heuristic. Numeric confidence should only be added if it has a defined and calibrated meaning.

   JSON Schema should be generated from canonical Rust DTOs. The schema validates shape; Rust validation must enforce semantic invariants.

   Required invariants include:

   • input identity equals adapter identity;
   • adapter identity equals the selected registry record;
   • parser family and profile match the selected record;
   • all IDs are unique;
   • all references are valid;
   • virtual-address arithmetic is checked;
   • code ranges stay within executable regions;
   • function ranges are valid and non-overlapping where required;
   • pool indices and pool geometry are bounded;
   • capability states agree with populated fields;
   • authoritative names cannot originate from heuristic sources;
   • ordering is deterministic;
   • unavailable data is not represented as fabricated semantic metadata.

   4. Adapter protocol

   A full JSON-RPC protocol is unnecessary for the initial one-process-per-analysis design.

   Start with protocol v1:

   • one JSON request on stdin;
   • request includes protocol/model versions;
   • request includes the selected compatibility-record digest;
   • snapshot data is passed through immutable files, not base64-encoded JSON;
   • request includes input paths, sizes, SHA-256 digests, and virtual addresses;
   • adapter writes its model to an assigned output path;
   • adapter returns one structured status result on stdout;
   • stderr is diagnostic-only and bounded;
   • the host validates output size, digest, schema, semantics, and identity before consuming it.

   The host should create private input/output directories for every invocation.

   The protocol should not allow an adapter to choose its own compatibility record. The host resolves the exact registry entry first and passes that decision to the adapter.

   Add JSON-RPC later only if long-lived adapters, streaming, remote workers, or concurrent calls become real requirements.

   Native subprocesses should be the first execution format. In-process shared libraries reduce startup overhead but allow parser bugs to crash or compromise the main process. WASM may be useful later for distributing third-party adapters, but it does not solve grammar classification or compatibility admission and should not block the native design.

   5. Adapter containment and artifact verification

   Before executing an adapter, the host should verify:

   • registry record exists for the exact identity;
   • artifact SHA-256 matches;
   • host OS and architecture match;
   • target architecture matches;
   • parser family and profile digests match;
   • artifact path cannot escape the installation directory;
   • artifact is installed atomically;
   • protocol and model versions are supported.

   The execution boundary should provide:

   • wall-clock timeout;
   • bounded stdout and stderr;
   • bounded output file size;
   • memory and process limits where available;
   • private working directory;
   • cleared environment except for an allow-list;
   • no inherited unrelated file descriptors;
   • no network access where sandbox support exists;
   • process-tree termination on timeout;
   • cleanup on success and failure.

   Checksums provide integrity, not publisher authenticity. Automatic remote installation should wait until the registry and artifacts support signing, rollback protection, and freeze protection through TUF, Sigstore, or an equivalent mechanism.

   Initial distribution should be local/bundled and install-only. Do not add automatic network installation before the trust model is complete.

   6. Honest fallback behavior

   Unknown or unsupported snapshot identities should not require a fake hash-named adapter.

   The fallback may provide:

   • executable regions;
   • architecture-derived facts;
   • recovered function candidates;
   • direct branch and call destinations;
   • dispatch-table call detection;
   • explicit heuristic function IDs;
   • clear capability limitations.

   The fallback must mark these unavailable unless independently recovered:

   • Dart libraries;
   • class hierarchy;
   • original semantic names;
   • authoritative ObjectPool meaning;
   • arbitrary object identities;
   • semantic entry functions.

   It must not synthesize values such as:

   package:app/main.dart
   Global
   main
   

   merely to satisfy a non-empty model check.

   This fallback is a useful product behavior, not an adapter failure. It lets unknown versions remain analyzable without emitting plausible but untrusted metadata.

   7. Serwalker integration

   The current Serwalker work is a good first parser candidate. It already demonstrates several useful properties:

   • exact Dart 3.11.1 snapshot-hash checking;
   • rejection of unsupported hashes;
   • FullAOT-only validation;
   • compressed-pointer validation;
   • separation of snapshot, cluster, root, instruction-table, raw-object, and model-producer code.

   However, it should not be merged with its current core integration shape.

   Required changes:

   1. Package Serwalker as an adapter executable implementing protocol v1.
   2. Remove the direct flutterdec-core dependency on the Serwalker crate.
   3. Do not route Serwalker through AdapterBackend::Internal.
   4. Preserve Internal for machine-code-only fallback.
   5. Select Serwalker through an exact registry record.
   6. Emit ProgramModel v4 rather than hardcoded Schema v3.
   7. Move data-only layout constants into a referenced profile where possible.
   8. Preserve exact hash and feature checks inside the parser as defense in depth.
   9. Replace unresolved placeholder strings with typed unavailable values and diagnostics.
   10. Do not mark unresolved names as exact.
   11. Do not assign full confidence to unresolved or heuristic pool entries.
   12. Add a real Dart 3.11.1 snapshot fixture and an end-to-end adapter test.
   13. Compare libraries, classes, functions, code ranges, and pool facts against an SDK-matched reference.
   14. Prove deterministic output for the same input and adapter artifact.
   15. Fix all workspace checks before merge.

   The parser must also preserve stream alignment across every snapshot section. In particular, root parsing must consume the initial-field-table and shared-initial-field-table counts and references before continuing to the dispatch table, even if those tables are discarded by the current model.

   The object-store layer should remain format-oriented and return references or domain-neutral values. Conversion into LibraryInfo, ClassInfo, and other ProgramModel types belongs in the model producer layer. A resolver that returns Result<()> while claiming to resolve object-store state should be replaced with a real state-producing API.

   Serwalker should become the first external adapter artifact, not define the generic host contract.

   8. Other parser backends

   r2flutter can be useful as:

   • an independent parser;
   • a reference oracle;
   • a source of exact object-pool information;
   • a guarded adapter for identities it can prove exactly.

   Its fingerprint mode must never provide authoritative metadata for an unknown hash.

   Blutter can be useful as:

   • an SDK-matched reference parser;
   • an expensive CI oracle;
   • an explicit fallback where installation and build costs are acceptable.

   Neither backend should be selected from a substring in adapter_kind. Backend authority must come from the verified registry record and provenance fields.

   9. Landing order

   The work should land as stacked changes.

   PR 1: Semantic contract

   Start from main:

   • add typed SnapshotIdentity;
   • define protocol v1 request/result types;
   • introduce ProgramModel v4;
   • add capability and provenance states;
   • add strict relational validation;
   • migrate the current core consumer;
   • migrate the checked-in reference adapter;
   • remove synthetic semantic fallback records;
   • stop inferring authority from adapter_kind.

   PR 2: Adapter host and registry

   Add:

   • exact identity-to-family/profile/artifact resolution;
   • runtime-loaded profiles;
   • subprocess execution;
   • SHA-256 verification;
   • host/target validation;
   • schema and semantic validation;
   • resource containment;
   • explicit machine-code fallback;
   • deterministic error reporting.

   Do not add automatic network installation yet.

   PR 3: Serwalker adapter

   After the first two contracts stabilize:

   • rebase or transplant PR Flutterdec dart serwalker #74;
   • package Serwalker as protocol v1;
   • emit ProgramModel v4;
   • remove direct core linkage;
   • stop overloading Internal;
   • add real fixtures and reference comparisons;
   • preserve exact identity checks;
   • fix parser alignment and model-provenance issues;
   • complete all CI checks.

   PR #74 can temporarily target the foundation branch as a stacked PR, but the merge order should remain contract, host/registry, then Serwalker. Building the foundation directly on the current Serwalker branch would mix generic architecture with parser-specific implementation and make the core contract depend on one parser.

   PR 4: Grammar-family proof

   Add a second exact snapshot identity.

   For that identity:

   • run the existing parser family unchanged;
   • compare output against an SDK-matched reference;
   • run semantic invariants;
   • add a profile-only change if only layout data differs;
   • split or update the parser family if serialization behavior differs.

   Never infer family compatibility from Dart version proximity.

   PR 5: CI and distribution

   For every Dart stable or beta candidate:

   1. acquire the exact SDK/source;
   2. generate or obtain real snapshot fixtures;
   3. extract the exact identity;
   4. generate a candidate profile;
   5. test existing parser families;
   6. run semantic invariants;
   7. compare overlapping facts with an SDK-matched reference;
   8. run wrong-hash and wrong-feature tests;
   9. run corruption, truncation, timeout, and oversized-output tests;
   10. verify artifact digests and host variants;
   11. publish a compatibility record only after all checks pass;
   12. otherwise publish a compatibility-break artifact containing the SDK diff, failed fixtures, and failed invariants.

   CI should automate SDK acquisition, identity extraction, profile generation, comparison, and publication. It should not attempt general-purpose C++-to-Rust parser translation; Dart serialization logic is procedural and conditional, not a stable declarative schema.

   10. Acceptance gates

   Issue #96 should not be considered complete until all of these behaviors are covered:

   1. A known exact identity selects the documented parser family, profile, and artifact.
   2. A wrong hash fails before adapter output reaches the core.
   3. A wrong snapshot kind, architecture, or feature tuple fails.
   4. A heuristically discovered hash cannot select an authoritative parser.
   5. An unknown identity uses machine-code fallback without fabricated semantic metadata.
   6. Corrupt or truncated adapter output fails relational validation.
   7. Adapter crashes, timeouts, malformed JSON, and oversized output remain contained.
   8. Artifact digest, host variant, target architecture, and registry mismatches fail before execution.
   9. The same input and adapter produce deterministic canonical output.
   10. Existing supported inputs preserve valid decompilation goldens.
   11. A newly admitted hash/profile can be added without rebuilding the core binary.
   12. Multiple exact registry records can reference one parser artifact.
   13. At least two materially different snapshot families have real fixtures.
   14. Failed compatibility candidates never enter the production registry.
   15. Parser limitations and unavailable fields are visible to callers.
   16. Adapter provenance is preserved through the complete pipeline.
   17. Machine-code fallback remains separate from snapshot deserialization.
   18. Security limits apply to every adapter invocation.

   Final decision

   The recommended architecture is:

   Exact snapshot identity
       + verified compatibility registry
       + grammar-family parser
       + independently versioned data profile
       + external adapter protocol
       + ProgramModel v4
       + strict semantic validation
       + honest machine-code fallback
   

   Reject:

   • one complete flutterdec binary per Dart or Flutter version;
   • one copied parser executable per snapshot hash;
   • one parser implementation per Dart major/minor version;
   • a monolithic in-core deserializer as the primary architecture;
   • heuristic hash matching as authoritative compatibility;
   • synthetic metadata for unknown snapshots;
   • automatic remote installation before artifact trust is implemented;
   • automatic general-purpose C++-to-Rust parser translation.

   Recommended first implementation priority:

   1. semantic contract and ProgramModel v4;
   2. adapter host and exact registry;
   3. Serwalker as the first external adapter;
   4. second-family compatibility proof;
   5. CI-driven profile and registry updates;
   6. signed remote distribution only after the local trust model is complete.
3. caverav

   self-assigned this

   on Aug 28, 2026
4. kalixtez commented on Aug 28, 2026

   

   kalixtez

   on Aug 28, 2026

   Collaborator

   More actions

   Small things to consider:

   • There might never be a chance for us to implement a JSON-RPC IPC model for adapters, given the task is not complex enough. Subprocess spawning works and I think will work fine for as long as the project lives.

   • For now, we don't really need to use the snapshot kind to key a future profile rwgistry. We assume from the beginning that we are gonna be receiving a FullAOT binary. The other proposed components of the composite key, snapshot and target arch, are very valid and indeed should be implemented in my opinion.

   • It is possible don't need to take into consideration alignment details when it comes to specifying a data profile. Dart doesn't insert padding or forces alignment constraints during serialization with the exception of the boundaries between snapshot section such as the clustered stream and the ROData image, where the output stream is aligned to image header-sized values.