Skip to content

perf(export): streaming chunked JSON export without intermediate Map and list allocations #1535

Closed as not planned

Description

@ZhuchkaTriplesix

Problem

In DataExportService.formatJson (lib/shared/services/data_export_service.dart), exporting table data creates heavy memory pressure:

  1. Allocates a new Map<String, Object?> for every row (100,000 maps for 100,000 rows).
  2. Collects all maps into a list of 100,000 items.
  3. Encodes the whole structure via JsonEncoder.convert, creating a single massive string in RAM before writing to disk, leading to potential OOM on large datasets.

Solution

  • Stream JSON chunks directly into StringSink / IOSink using zero-allocation chunked formatting:
    buffer.write('{\n "'), buffer.write(col), buffer.write('": '), _writeJsonValue(buffer, val).
  • Eliminate all intermediate Map objects and the outer list.

Benefits

  • 0 intermediate Map allocations during JSON export.
  • Flat memory footprint regardless of row count, avoiding Out-Of-Memory errors on large exports.

Activity

  1. ZhuchkaTriplesix commented on Oct 12, 2026

    @ZhuchkaTriplesix
    MemberAuthor

    Closing after measuring against the current code.

    • Save-to-file already streams: DataExportService.writeToSink -> _writeJsonObjectArray encodes one row at a time and yields every 500 rows. No full list of maps and no giant string on that path.
    • The clipboard path (formatAsync -> formatJson) needs the full string anyway (OS clipboard API), and it already runs in an isolate.
    • Bench, 50k rows x 20 cols: JSON via all-maps + one string is ~259 ms on one isolate, which is why it is already off the UI thread.

    Nothing user-visible is left to win here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions