Skip to content

Decode server-sent events as UTF-8 with BOM handling - #1308

Open
theamodhshetty wants to merge 1 commit into
pydantic:mainfrom
theamodhshetty:fix/sse-leading-bom
Open

theamodhshetty wants to merge 1 commit into
pydantic:mainfrom
theamodhshetty:fix/sse-leading-bom

Conversation

@theamodhshetty

@theamodhshetty theamodhshetty commented Nov 1, 2026 •

Copy link
Copy Markdown

Summary

A leading UTF-8 BOM currently hides the first data, event, or id field in an SSE response. It can also trigger a false event-size error. A non-UTF-8 response charset or client default encoding corrupts UTF-8 event data.

The SSE parsing rules require UTF-8 decoding and ignoring exactly one leading BOM. Decode response content bytes through the existing incremental TextDecoder before line buffering and size checks. Later U+FEFF characters remain intact, and Response.encoding is unchanged.

Added 42 parameterized sync/async regression cases covering fragmented BOMs and UTF-8 characters, event metadata, exact size limits, later/double BOMs, and charset/default-encoding overrides. Updated the SSE documentation.

Local validation on Python 3.15.0:

  • Against untouched upstream b77e3e8, nine sync regressions fail and five controls pass.
  • All 2,138 tests pass with the fix.
  • scripts/coverage reports 100% coverage; scripts/check passes formatting, typing, lint, lockfile and unasync checks.

Hosted CI and maintainer review remain separate gates.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

Implementation and tests were prepared with Codex assistance.

View guided diff

@codspeed

codspeed Bot commented Nov 1, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 18 untouched benchmarks
⏩ 7 skipped benchmarks1


Comparing theamodhshetty:fix/sse-leading-bom (2e05bcd) with main (b77e3e8)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

View guided diff | Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant