Skip to content

Fix JSON5 number and escape grammar conformance - #153

Open
gaoflow wants to merge 1 commit into
Kijewski:mainfrom
gaoflow:fix-number-escape-conformance
Open

Fix JSON5 number and escape grammar conformance#153
gaoflow wants to merge 1 commit into
Kijewski:mainfrom
gaoflow:fix-number-escape-conformance

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 19, 2026

Copy link
Copy Markdown

This tightens the number-literal and string-escape parsing in src/_decoder.pyx to match the JSON5 grammar. The parser currently both accepts inputs the spec forbids and rejects inputs it requires; the defects all cluster in the number scanner and the escape handler, so this fixes them as one class. The expected results below are from the json5 reference implementation (npm 2.2.3) and agree with stdlib json and dpranke's json5 where those apply.

Accepts inputs that should be rejected

input current reference
1_000, 1__0, 1_, 0x1_0, 1_e3 parsed, _ silently stripped SyntaxError — JSON5 has no numeric separators
.e5, .e+5 0.0 SyntaxError — a leading . needs a fractional digit
5.8., .5. 5.8 / 0.5 SyntaxError — at most one ., never after the exponent
++7, +-.4 7 / -0.4 SyntaxError — a sign is only valid after e/E
0e, 0e+ 0.0 SyntaxError — the exponent needs a digit
"\09", "\07", "\00" "\x009", "\x007", … SyntaxError — \0 before a digit is a legacy octal escape

Rejects inputs that should be accepted

input current reference
1e999, -1e999, 1e400 Json5EOF Infinity, -Infinity (ECMAScript ToNumber)
0.e1 Json5EOF 0.0

"\0", "\0a", 1e-999 (→ 0.0), 1e308, .5, 1.e5, 0xFF, 01/00/007 (→ rejected) and every other currently-correct case are unchanged; the official json5-tests and JSONTestSuite suites stay green.

Fix

_decode_number_any now tracks the grammar explicitly — at most one ., at most one exponent, a sign only right after e/E, and a leading . must be followed by a digit — instead of silently dropping stray _/./sign characters. The leading-zero path (0.xxx / 0exxx) reuses the same scanner. _decode_double keeps fast_double_parser as the fast path and falls back to float() only for a grammatically valid token it rejects, which is exactly overflow (→ ±Infinity) and 0.e1. The \0 escape peeks the next code unit and rejects a following decimal digit.

Tests

scripts/run-conformance-test.py is a value-level table covering the whole class, added to CI and make test.

Build (Cython): git submodule update --init --recursive then pip install -e ..

Tighten the number-literal scanner and string-escape handling in
src/_decoder.pyx to match the JSON5 grammar (verified against the json5
reference implementation):

- reject numeric underscores, a leading '.' without a fractional digit,
  doubled or trailing dots, dots inside the exponent, stray/doubled signs
  and empty exponents
- reject '\0' followed by a decimal digit (a forbidden legacy octal escape)
- accept 0.e1 and parse out-of-range magnitudes to +/-Infinity per
  ECMAScript ToNumber

The float scanner now tracks the grammar explicitly (at most one dot, one
exponent, a sign only after e/E), and the leading-zero path reuses it, so
float() is only reached to rescue tokens fast_double_parser rejects but
ECMAScript accepts (overflow and 0.e1).

Add scripts/run-conformance-test.py, a value-level table for the whole
class, to CI and `make test`.
@gaoflow
gaoflow force-pushed the fix-number-escape-conformance branch from d11d339 to f7050c9 Compare July 19, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant