Fix JSON5 number and escape grammar conformance - #153
Open
gaoflow wants to merge 1 commit into
Open
Conversation
Tighten the number-literal scanner and string-escape handling in src/_decoder.pyx to match the JSON5 grammar (verified against the json5 reference implementation): - reject numeric underscores, a leading '.' without a fractional digit, doubled or trailing dots, dots inside the exponent, stray/doubled signs and empty exponents - reject '\0' followed by a decimal digit (a forbidden legacy octal escape) - accept 0.e1 and parse out-of-range magnitudes to +/-Infinity per ECMAScript ToNumber The float scanner now tracks the grammar explicitly (at most one dot, one exponent, a sign only after e/E), and the leading-zero path reuses it, so float() is only reached to rescue tokens fast_double_parser rejects but ECMAScript accepts (overflow and 0.e1). Add scripts/run-conformance-test.py, a value-level table for the whole class, to CI and `make test`.
gaoflow
force-pushed
the
fix-number-escape-conformance
branch
from
July 19, 2026 22:51
d11d339 to
f7050c9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This tightens the number-literal and string-escape parsing in
src/_decoder.pyxto match the JSON5 grammar. The parser currently both accepts inputs the spec forbids and rejects inputs it requires; the defects all cluster in the number scanner and the escape handler, so this fixes them as one class. The expected results below are from thejson5reference implementation (npm 2.2.3) and agree with stdlibjsonand dpranke'sjson5where those apply.Accepts inputs that should be rejected
1_000,1__0,1_,0x1_0,1_e3_silently stripped.e5,.e+50.0.needs a fractional digit5.8.,.5.5.8/0.5., never after the exponent++7,+-.47/-0.4e/E0e,0e+0.0"\09","\07","\00""\x009","\x007", …\0before a digit is a legacy octal escapeRejects inputs that should be accepted
1e999,-1e999,1e400Json5EOFInfinity,-Infinity(ECMAScriptToNumber)0.e1Json5EOF0.0"\0","\0a",1e-999(→0.0),1e308,.5,1.e5,0xFF,01/00/007(→ rejected) and every other currently-correct case are unchanged; the officialjson5-testsand JSONTestSuite suites stay green.Fix
_decode_number_anynow tracks the grammar explicitly — at most one., at most one exponent, a sign only right aftere/E, and a leading.must be followed by a digit — instead of silently dropping stray_/./sign characters. The leading-zero path (0.xxx/0exxx) reuses the same scanner._decode_doublekeepsfast_double_parseras the fast path and falls back tofloat()only for a grammatically valid token it rejects, which is exactly overflow (→±Infinity) and0.e1. The\0escape peeks the next code unit and rejects a following decimal digit.Tests
scripts/run-conformance-test.pyis a value-level table covering the whole class, added to CI andmake test.Build (Cython):
git submodule update --init --recursivethenpip install -e ..