Table service corrupts multi-byte UTF-8 characters split across request body chunks (`readRequestIntoText` uses `segments.join` instead of `Buffer.concat`)

[max3004]

Which service(blob, file, queue, table) does this issue concern?

Table. (The same generated helper exists verbatim in the blob and queue serializers — see the last section; we verified the corruption for the table service.)

Which version of the Azurite was used?

v3.35.0 (latest release). The affected code is unchanged on current main (c9aa6ff).

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm. (The affected code ships identically in all distributions.)

What's the Node.js version?

First hit on Node 20 (Linux CI); reproduced with Node 25 on macOS using the repro below. Not Node-version-specific.

What problem was encountered?

When the request body of a table operation (e.g. an entity upsert) is delivered to Azurite's HTTP server in more than one data event and a multi-byte UTF-8 character straddles a chunk boundary, that character is decoded into U+FFFD replacement characters. The corrupted value is then persisted — every subsequent read returns it — and the string length changes too (a split 4-byte emoji becomes 4 replacement characters instead of 2 UTF-16 code units). Real Azure Table Storage handles the same request correctly.

Root cause — readRequestIntoText in src/table/generated/utils/serializer.ts:

async function readRequestIntoText(req: IRequest): Promise<string> {
  return new Promise<string>((resolve, reject) => {
    const segments: string[] = [];
    const bodyStream = req.getBodyStream();
    bodyStream.on("data", buffer => {
      segments.push(buffer);
    });
    bodyStream.on("error", reject);
    bodyStream.on("end", () => {
      const joined = segments.join(""); // <-- decodes every chunk in isolation
      resolve(joined);
    });
  });
}

getBodyStream() returns the raw Express request with no encoding set (ExpressRequestAdapter.ts#L25-L27), so the data events emit Buffers (the segments: string[] annotation is misleading). Array.prototype.join then stringifies each buffer individually, and a UTF-8 sequence whose bytes are split across two chunks cannot be decoded by either side alone — both halves become U+FFFD.

Example: 🎉 is F0 9F 8E 89. A chunk boundary after F0 yields ���� (1 + 3 replacement characters) instead of 🎉.

Whether the bug fires depends only on how the OS delivers the body. Small loopback requests usually arrive in a single data event, which hides the bug; large bodies (a ~175 KiB entity-upsert JSON in our case) are reliably split into multiple events. For us the same test failed deterministically on Linux CI and passed on macOS loopback — Azurite silently diverges from the real service only under perfectly normal network segmentation, which made this hard to track down.

An Azurite -d debug log confirms the corruption happens while reading the request: the deserialize(): Raw request body string is ... line already contains the replacement characters.

Steps to reproduce the issue?

Self-contained repro: npm install @azure/data-tables, start azurite --inMemoryPersistence, then node repro.js. A tiny TCP proxy re-segments the byte stream so the body reliably arrives in many chunks — it merely makes deterministic what a real network does with large bodies anyway:

// repro.js — Azurite table service corrupts multi-byte UTF-8 split across body chunks
//
//   npm install @azure/data-tables
//   npx azurite --inMemoryPersistence --silent   (table endpoint on 127.0.0.1:10002)
//   node repro.js
//
// The tiny TCP proxy below re-segments the byte stream into 1031-byte slices with a
// small pause, so Azurite's HTTP server sees the request body as many 'data' events —
// it merely makes deterministic what a real network does with large bodies anyway.

const net = require("net");
const { TableClient, AzureNamedKeyCredential } = require("@azure/data-tables");

const AZURITE_PORT = 10002;
const PROXY_PORT = 10102;
const CHUNK = 1031; // odd on purpose: boundaries drift through every byte offset of the 4-byte sequences

const server = net.createServer((client) => {
  const upstream = net.connect(AZURITE_PORT, "127.0.0.1");
  client.on("data", async (data) => {
    client.pause();
    for (let i = 0; i < data.length; i += CHUNK) {
      upstream.write(data.subarray(i, i + CHUNK));
      await new Promise((r) => setTimeout(r, 2)); // each slice arrives as its own 'data' event
    }
    client.resume();
  });
  upstream.pipe(client);
  client.on("error", () => upstream.destroy());
  upstream.on("error", () => client.destroy());
});

server.listen(PROXY_PORT, async () => {
  const client = new TableClient(
    `http://127.0.0.1:${PROXY_PORT}/devstoreaccount1`,
    "utf8repro",
    new AzureNamedKeyCredential(
      "devstoreaccount1",
      "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
    ),
    { allowInsecureConnection: true }
  );
  await client.createTable();

  // 4 string properties, each below the 32K-character property limit,
  // together ~240 KB of 4-byte UTF-8 on the wire
  const value = "🎉".repeat(15000);
  const props = ["p0", "p1", "p2", "p3"];
  const entity = { partitionKey: "p", rowKey: "r" };
  for (const k of props) entity[k] = value;
  await client.upsertEntity(entity, "Replace");
  const read = await client.getEntity("p", "r");

  const intact = props.every((k) => read[k] === value);
  const fffd = props.reduce((n, k) => n + ((read[k].match(/\uFFFD/g) || []).length), 0);
  console.log("roundtrip intact  :", intact);
  console.log("chars/prop written:", value.length);
  console.log("chars/prop read   :", props.map((k) => read[k].length).join(", "));
  console.log("U+FFFD read back  :", fffd);
  server.close();
  process.exit(intact ? 0 : 1);
});

Output against unpatched Azurite v3.35.0 (reproduced on macOS and Linux; exact counts vary with segmentation):

roundtrip intact  : false
chars/prop written: 30000
chars/prop read   : 30045, 30043, 30044, 30045
U+FFFD read back  : 529

Expected (= behavior of real Azure Table Storage):

roundtrip intact  : true
chars/prop written: 30000
chars/prop read   : 30000, 30000, 30000, 30000
U+FFFD read back  : 0

Have you found a mitigation/solution?

Yes — collect Buffers and decode once over the concatenated bytes instead of per chunk:

const segments: Buffer[] = [];
...
bodyStream.on("end", () => {
  resolve(Buffer.concat(segments).toString("utf8"));
});

(or stream-decode with string_decoder.StringDecoder). We verified that with exactly this one-line change applied to dist/src/table/generated/utils/serializer.js, the repro above round-trips losslessly (second output block). The repo already uses the correct pattern in TableBatchUtils.StreamToString, so the table batch path is not affected.

Note: the identical readRequestIntoText also exists in the generated blob and queue serializers and presumably wants the same fix (a queue message XML body with multi-byte text looks like a candidate for the same corruption) — we only verified the table service.

As a workaround we currently patch the installed serializer.js in our CI test setup. Happy to open a PR with the fix.

[jainakanksha-msft]

@max3004 Azurite welcomes Contribution.
Please go ahead and raise a PR with relevant test cases addition and Changelog update.