Skip to content

fix: MapStreamQuery serializes items with the payload serializer, breaking NDJSON/SSE framing and the JSON contract across formats and TFMs #846

Description

@samtrion

User Story

As a developer exposing stream queries via MapStreamQuery, I want every stream format (NDJSON and SSE) to use the same valid framing and the same JSON contract as the rest of my Minimal API, so that clients can parse streamed items reliably and receive the same property names regardless of format or target framework.


Problem

MapStreamQuery injects IPayloadSerializer (src/NetEvolve.Pulse.AspNetCore/EndpointRouteBuilderExtensions.cs:234) and uses it for NDJSON on all TFMs (:249) and for SSE on net8.0/net9.0 (:258). On net10.0, SSE goes through TypedResults.ServerSentEvents(items) (:255) instead.

1. Framing breaks when the serialized item contains newlines.

  • Pre-net10 SSE (:374-377) writes data: once, then the whole output of payloadSerializer.SerializeToBytes(item), then the suffix. The payload is never split per line.
  • NDJSON (:399-401) writes the serialized bytes followed by \n, with no check that the payload itself contains no newline.

The default configuration is not affected, because the default JsonSerializerOptions has WriteIndented = false. The bug appears as soon as an application sets services.Configure<JsonSerializerOptions>(o => o.WriteIndented = true) (the options SystemTextJsonPayloadSerializer reads, src/NetEvolve.Pulse/Serialization/SystemTextJsonPayloadSerializer.cs:34-37) or registers a custom IPayloadSerializer that emits newlines:

  • NDJSON: each item spans several lines, so no single line is a valid JSON text and NDJSON readers fail.
  • SSE (net8/9): the client takes only { as the event data. The following lines (for example "OrderId": 1,) have no data: prefix and are dropped as unknown fields.

2. Different JSON contracts depending on format and TFM.

SystemTextJsonPayloadSerializer uses plain IOptions<JsonSerializerOptions>, falling back to new JsonSerializerOptions(). That means PascalCase and none of the application's HttpJsonOptions converters or naming policy. MapQuery, MapCommand and net10 SSE (ServerSentEventsResult) serialize with HttpJsonOptions, which use web defaults (camelCase). For the same DTO:

Endpoint / format net8.0 / net9.0 net10.0
MapQuery / MapCommand {"orderId":1} {"orderId":1}
MapStreamQuery SSE {"OrderId":1} {"orderId":1}
MapStreamQuery NDJSON {"OrderId":1} {"OrderId":1}

A client reading SSE therefore sees the casing change when the server moves between net8/9 and net10. There is also a smaller difference: net10 ServerSentEvents writes a string item raw, while the pre-net10 SSE and NDJSON paths write it as a quoted JSON string.

Using IPayloadSerializer here was a deliberate choice in #681 ("so streamed items honor the app's configured Pulse serializer"), but no ADR records it. The existing tests stream only primitives (int in tests/.../EndpointRouteBuilderIntegrationTests.cs via MapStreamQuery<NumbersStreamQuery, int>, and string in the unit and correlation tests). Neither the framing nor the casing issue is covered.

Not covered by #794 / #802: those cover only the inspector endpoints and the Azure Queue Storage envelope.


Specification

  • NDJSON spec §3.1/§3.2: "The JSON texts MUST NOT contain newlines or carriage returns." Each JSON text is terminated by \n.
  • WHATWG HTML §9.2.6 Interpreting an event stream: for a field named data, the value plus a LF is appended to the data buffer. Any other field name "is ignored". A line without a colon uses "the whole line as the field name". Multi-line data therefore needs a data: prefix on every line.
  • ASP.NET Core Minimal APIs: JSON options: "By default, Minimal API apps use Web defaults options during JSON serialization." ServerSentEventsResult (ASP.NET Core 10) resolves IOptions<Microsoft.AspNetCore.Http.Json.JsonOptions> and its SseFormatter prefixes every line with data:.

Requirements

  • Choose one JSON contract for MapStreamQuery that applies to every format and TFM. Recommended: align with MapQuery, MapCommand and net10 SSE by serializing with IOptions<Microsoft.AspNetCore.Http.Json.JsonOptions>.Value.SerializerOptions, copied once with WriteIndented = false. This explicitly reverses the IPayloadSerializer choice from refactor(serialization): use IPayloadSerializer instead of raw System.Text.Json #681.
  • If IPayloadSerializer is kept instead: prefix every output line with data: for pre-net10 SSE, and make sure an NDJSON item never contains \n or \r. Document the resulting casing difference from net10 SSE and MapQuery.
  • NDJSON output must be valid NDJSON regardless of the application's indentation settings.
  • Pre-net10 SSE output must deliver the complete item as event data (single-line JSON, or data: per line like SseFormatter).
  • Keep NativeAOT and trimming compatibility (resolve type info through the options' TypeInfoResolverChain, no new trim warnings).
  • Record the decision in an ADR under decisions/, and add a breaking-change note to the NetEvolve.Pulse.AspNetCore README. Moving NDJSON and pre-net10 SSE to HttpJsonOptions changes the casing existing clients receive (PascalCase to camelCase).

Acceptance Criteria

  • A failing test is added first: a multi-property DTO streamed with WriteIndented = true configured produces invalid NDJSON and a truncated pre-net10 SSE event.
  • NDJSON output with indentation enabled contains exactly one valid JSON text per line.
  • Pre-net10 SSE output with indentation enabled delivers the full item as data to a spec-compliant parser.
  • The same DTO has identical property casing across MapQuery, MapStreamQuery NDJSON and MapStreamQuery SSE on net8.0, net9.0 and net10.0 (tests per TFM).
  • The string item handling difference between net10 SSE and the other paths is covered by a test and either aligned or documented.
  • An ADR and a README breaking-change note document the chosen contract.
  • The NativeAOT publish (aot.yml) passes with -warnaserror.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugIndicates an issue or flaw that needs to be fixed.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions