Reduce memory pressure during bulk import processing

[stalep]

Problem

During bulk imports (load-legacy-runs), h5m can OOM on large test suites (e.g., 20 tests × 200 runs at 8 GB heap). Allocation profiling with async-profiler and heap dump analysis identified three independent retention/allocation issues in the processing pipeline that compound to exhaust available heap.

Root causes and fixes

1. New GraalJS Engine created per JavaScript evaluation

NodeService.calculateJsValues() and evaluateFingerprintFilter() both create a new Engine AND Context on every invocation:

Context.newBuilder("js")
    .engine(Engine.newBuilder("js").option(...).build())  // NEW Engine every call
    .build()

The Engine is the heavyweight object that allocates Truffle internal data structures (JSFunctionData, ShapeExt, FrameDescriptor, DefaultFrameInstance, etc.). Even though the Context is closed via try-with-resources, the inline Engine relies on GC for cleanup.

Profiling evidence: async-profiler allocation profiling showed:

• Baseline (engine-per-eval): 9,255 MB DefaultFrameInstance + 6,831 MB FrameWithoutBoxing per 15s window
• With shared engine: 1,288 MB + 956 MB — 86% reduction

By snapshots where no JS nodes were active, Truffle allocation dropped to near zero, confirming the shared Engine only allocates for actual evaluations.

Fix: Single static Engine shared across all JS evaluations. Engine is thread-safe and designed for reuse. Context is still created per-evaluation for isolation.

2. Work objects retain heavy JqValue data after processing

When WorkService.execute() completes, cascade Work items are created carrying sourceValues — full ValueEntity instances with their parsed JqValue data field (e.g., 3.2 MB per rhivos run). These sit in the work queue until a thread picks them up. The cascade workers reload via em.find() anyway, so the original data is unnecessary.

After execute() returns, the completed Work object's sourceValues, sourceNodes, and activeNodes fields remain populated until the Work object is garbage collected. With many items in the queue, this retains gigabytes of parsed JSON data.

Heap dump evidence: 79,074 Object[] references to ProxyJqObject instances held via Work objects in the queue.

Fix: Null out sourceValues, sourceNodes, and activeNodes in Work.run() after execute() returns.

3. Hibernate persistence context accumulates entities within WorkService.execute()

Each WorkService.execute() call runs in its own transaction. During execution, it loads source values via em.find(), calculates new values (which may load historical values via getDescendantValueByPath(), findMatchingFingerprint(), etc.), and persists results. All loaded and created entities accumulate in the persistence context for the duration of the transaction.

For change detection nodes (RelativeDifference, StdDevAnomaly, EDivisive), a single execute() call can load hundreds of historical ValueEntity instances. With @Immutable + READ_ONLY caching, these use ImmutableManagedEntityHolder which keeps references alive.

Heap dump evidence: EntityEntryContext (6,167 refs), StatefulPersistenceContext (1,044 refs), PersistEvent (26,556 refs), MergeEvent (7,007 refs) all holding ValueEntity references.

Allocation profiling evidence: SessionImpl.persistOnFlush — 672 MB of cascade persist event allocations per 15s window.

Fix: em.flush() + em.clear() at the end of execute(), after cascade work is created. All new entities have been flushed, cascade Work items carry entity IDs and reload in their own transactions, and change-detected events have already fired.

Testing

All 408 tests pass with all three fixes applied.