Skip to content

perf(serialization): reuse explicitly declared neutral capture keys #53

Closed

Description

@FLwolfy

Verified on branch / awaiting merge

Production implementation: dd5979c613b0fe8ebd9f7026f25001ceee9f8cee. Delivered head: dbae93e8f211d45a69cdffdb6afa3a33a0af0fa5; the subsequent diff contains documentation/acceptance evidence only. Exact-head CI: Engine CI and NuGet dependency graph passed; all six checks on this exact head are successful. All applicable acceptance conditions for this issue are verified; unavailable physical/manual device checks are explicitly scoped, not reported passed.

Owner: Core immutable snapshot/Writer proof, Scene complete neutral row inputs. Implementation commits: be44532f, 2434cd96, 1917b253.

Evidence: complete report, exact-head delivery attachment, portable test-case/raw-measurement/requirement indexes linked by the report. Specific proof: Core97: callback retirement/thread/metadata/identity/converter/effect/mutation/GC guards; Scene67: complete writes/restore/references/Missing/Prefab and removed-System GC; Ten complete-inspection processes, architecture/Wiki/products/CI.

Commands: official tests/Invoke-EditorUiAcceptance.ps1 Contracts/Workflows/Visual/Lifecycle/Products with SDK-selected independent outputs; complete source/entry preparation; formal InnoEngine.Build scripts --ide for the three companions; tests/Invoke-WikiChecks.ps1; controlled public performance fixtures where applicable. The report records original command owners, TRX/binlogs, negative tests, failures, skips, input hashes and device limits.

Profiles overlap; do not sum their pass counts into a distinct-test claim. Windows-only sharing fixtures and Unix-only filesystem fixtures are scoped to the executing host. Real native contexts/simulated display metrics and hidden product feature journeys are not a claim of physical display, OS IME or manual Mac testing. No BGCS/vendor/package/generated-binding modification, parallel Core system, test hook, IVT or private reflection was used.

Keep this issue Open. PR #26 includes an individual Fixes #53 and will close it only when the user merges to master. No automatic merge or premature master-resolution claim.

Original defect, ownership and accepted work

The following baseline observations describe the issue as opened. The verification above supersedes intermediate pending checkpoints; original failed/negative logs remain preserved. Acceptance checkboxes are checked only after the exact delivery-head verification completes.

Approved-plan mapping

Parent #27; required by #46, E07 #34 and E11 #38. This issue owns the minimal Core serialization boundary needed to eliminate redundant complete large-scene materialization without weakening authoritative state detection.

Baseline confirmed facts and risks

Complete due inspection still takes roughly 35–43 ms before the independent experiment. A historical whole-root prototype demonstrated lower idle cost but still incurred roughly 70 ms and 16.8 MB after an edit. The current branch instead certifies complete array-element inputs through the existing Writer and memoizes only bounded immutable canonical blocks. Preliminary ordinary-producer edit medians are 12.8–14.7 ms and about 2.51 MB; these do not replace final quiet process pairs. Earlier interfaces, primitive slots, context projections, property-only shortcuts and cached graph-ID maps are not retained.

Owners and dependency direction

  • Core Serialization owns the borrowed builder, exact neutral key comparison, metadata/Converter guards, immutable snapshot hints and publication validation.
  • Core Identity remains the only object authority; keys retain domain-qualified runtime values without resolving or retaining live objects.
  • The metadata provider certifies actual pure primitive fields and explicit complete source revision access.
  • Scene owns its existing topology/hierarchy, GameObject metadata/activation writes and Transform local/world/reset/restore writes.
  • Editor Workspace reads current source catalog status and continues authoritative dirty/saved-baseline handling.

No UI/backend/platform branch, event system, identity registry, serializer, persistence format or global pool is introduced. BGCS/vendor/packages/generated bindings are unchanged.

Implementation plan and files

  1. Add an optional protected Converter capture-key callback, declining by default. Regular Serialize never uses it; arbitrary converters keep normal capture.
  2. Add a synchronous borrowed Core builder for capacity, owner revisions, proven property revisions and exact neutral text/numeric inputs. Failed TryAdd invalidates the entire key. Retire operation/context/metadata references on every outcome and reject later/cross-thread use.
  3. Reuse the current immutable snapshot/node pipeline. Guard the key by current Core runtime identity, immutable metadata and pinned Converter generation. Recheck a changed key after the normal write; changed inputs reject publication. Never mutate a prior node/key.
  4. Add explicit metadata capability methods and a revision-member declaration attribute. Sealed identity sources use private nonserialized nullable counters; exhausted/unavailable counters decline reuse. No public Transform interface/property is added.
  5. Extend the existing WriteObjectArray operation with an optional borrowed capture-key callback. Scene declares complete per-object rows: owner revision, source GUID, included parent source GUID, sibling position and every eligible component revision/source identity. The root, ordering and Systems retain ordinary writing. Missing/Prefab/untracked/contextual rows decline only their own reuse. Exact GUID values are compared without lossy hashes.
  6. Prevent enclosing-key publication whenever nested converters execute, including independently captured property payloads. Use only a transient neutral effect marker; arbitrary converters retain normal evaluation.
  7. Let the same canonical encoder memoize keyed immutable object blocks up to 64 KiB. Larger blocks continue streaming; prior snapshots remain immutable and public output arrays are independent.
  8. Cover every public and internal metadata/activation/local/world/reset/restore mutation. Current catalog queries do not import source data during label/dirty queries; source reconciliation remains the Asset owner.
  9. Update actual consumers, scripting exports, project references, XML, Wiki, architecture checks and the original-plan ledger together.
src/foundation/core/Inno.Core.Serialization/
  Capture/SerializationCaptureKeyBuilder.cs
  Metadata/SerializationRevisionAttribute.cs
  Metadata/SerializationTypeMetadata.cs
  Converters/SerializationConverter.cs
  Internal/{ConverterRegistry,SerializationOperation,SerializationNode,ValuePipeline}.cs
  Properties/ScriptingApi.cs
backends/DotNet/runtime/Inno.Adapter.Serialization.DotNet/
  ReflectionSerializationMetadataSource.cs
  Discovery/ReflectionStructMetadata.cs
src/content/scene/Inno.Scene/
  Core/GameObject.cs
  Concrete/Transform.cs
  Serialization/{Converters/GameSceneConverter,Graph/SceneGraphSerialization}.cs
src/composition/editor/features/Inno.Editor.Scene/Documents/
  EditorSceneWorkspace.SourceChanges.cs
tests/core/Inno.Core.Serialization.Tests/
tests/scene/Inno.Scene.Tests/
tests/editor/Inno.Editor.PlayMode.Tests/
docs/core/Inno.Core.Serialization.md
records/architecture/acceptance/

Acceptance

  • Canonical bytes and dirty/clean results agree for all local/world/reset/restore writes, names/tags/layers/active state, hierarchy/topology, references, Missing, prefab, source reconciliation, History, Play and reload.
  • Untracked fields/properties and stateful/contextual/ambiguous converters keep authoritative capture; opt-in is explicit and complete.
  • Metadata/Converter/identity/domain changes, key decline, partial errors and mutation during capture never reuse or publish a false clean result.
  • Retained snapshots/builders do not retain extension objects, metadata, delegates, Assembly or ALC; strict GC/finalizer/GC monitors remain enforced.
  • Borrowed builder retirement, thread misuse, failures and nullable/exhausted revision semantics pass through public converter/provider boundaries without IVT/private reflection/testing APIs.
  • Five alternating independent process pairs on fixed input pass median <16 ms, P95 <25 ms and <8 MiB per complete due inspection with the unchanged 100 ms policy. Raw data, CPU, GC, allocations and failures are retained.
  • Full relevant Core/Scene/Editor, SDK/script exports, architecture, Wiki and final CI pass on the same candidate.

Failure, lifecycle and closure

Keys are transient neutral optimization facts, never encoded project data or a schema version. Unknown capability declines reuse; exceptions retain the prior snapshot/last-good owner and diagnostics. Source mutation during a declared capture rejects that capture. Default and unsupported cases use the same authoritative writer, not a parallel implementation or legacy path.

Keep Open until full acceptance is verified on branch, then add an individual Fixes reference to PR #26. Close only when that PR merges to master. No automatic merge is authorized.

Activity

  1. FLwolfy commented on Oct 29, 2026

    @FLwolfy
    MemberAuthor

    Production correctness currently passes Core 90 / Scene 56, and the first complete twelve-suite SDK build completed with zero warnings/errors. Quiet due-inspection medians are 6.7–7.1 ms, but the first ten-sample P95 is 39.6 ms; this is not latency acceptance. The measured identity-check scratch allocates about 2.16 MB per inspection. A further Core-owned refinement borrows a cleared BCL buffer only during the synchronous callback, validates all identities before unchanged-node reuse, and copies checks only when an actual write needs post-write validation. No thresholds, timers, samples or ordinary writer semantics are relaxed. The next build was explicitly canceled for this refinement; its log remains canceled, not passed. Full final-source tests, controlled groups and product/CI reruns remain open.

  2. FLwolfy commented on Oct 29, 2026

    @FLwolfy
    MemberAuthor

    The editing-path check prevents closure: changing one Transform per complete 10,000-object inspection still costs about 69–71 ms / 16.8 MB, despite the idle improvement. A property-only proof experiment passes Core contracts but reaches only 57–60 ms and initially violates unchanged-snapshot reuse; it will not be adopted as the solution. The next design applies the same neutral capture proof at existing structured-array element boundaries. Core Writer owns optional row-key capture/verification; Scene contributes complete object/component state, exact source/parent identities and sibling placement. Ordinary serialization and uncertified/Missing/Prefab/reference rows still execute the same writer. Removing the all-or-nothing Scene root hint avoids rescanning its complete key twice on edits; independently unchanged rows can reuse their existing immutable nodes. This requires one optional existing Writer callback parameter and exact Guid key input, public-boundary non-Scene row tests, canonical/retirement/failure tests and fresh idle/edit measurements. No new serializer, identity index, mutation service or platform/backend dependency is introduced. Existing failed experiments remain evidence, not production fallback paths.

  3. FLwolfy commented on Oct 29, 2026

    @FLwolfy
    MemberAuthor

    Commit 2434cd9 replaces the all-or-nothing Scene root hint with optional keyed elements in the existing Writer array protocol, and adds exact Guid key values. Scene owns complete object/component, source/parent ID and sibling inputs. Core rejects enclosing reuse when nested converters execute (including neutral effects propagated from independent property payloads), preserving stateful conversion. The rejected property-only experiment is removed. The same canonical encoder memoizes immutable keyed blocks up to 64 KiB; larger nodes remain streamed. Core 97 passes, including independent non-Scene element reuse, exact IDs, failures, converter effects, concurrency, failed streams and large-node bounded allocation. Scene 56 and ten focused native cases pass at their recorded stages. Preliminary edit medians are 12.8–14.7 ms / ~2.51 MB, idle about 9.5 ms / ~2.36 MB; raw spikes remain retained. Complete frozen SDK graph, quiet five-process idle/edit pairs, profiles/products and exact-head CI are now required before closure. No BGCS, vendor, generated binding, platform policy or parallel foundational service was changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions