Skip to content

fix(srcml): patch the two non-termination defects - #109

Merged
EllianCarlos merged 1 commit into
masterfrom
fix/srcml-nontermination
Oct 12, 2026
Merged

EllianCarlos merged 1 commit into
masterfrom
fix/srcml-nontermination

Conversation

@EllianCarlos

Copy link
Copy Markdown
Collaborator

What this changes

The census found two srcML defects that make the parser run without end. Each hung file costs 600 s and is then left out of the dataset. This PR builds srcML (develop d382719) with a patch that fixes both: nix/srcml.nix applies nix/srcml-nontermination.patch.

  1. Adjacent string literals (lexer). char *s = "a""b"; (C, C++, ObjC). The old lexer took the run of quotes as the closing quote, so string and code swapped for the rest of the file. On NSS certdata.c this makes the parser superlinear (100 KB 0.28 s, 200 KB 116.6 s, 400 KB and more: over 900 s). Fix in src/parser/CommentTextLexer.g: leave the quote loop when mode == STRING_END and not noescape, in C, C++ and ObjC only.
  2. Function-name loop (parser). X template <class R> auto a; (C++), and namespace n{struct T&f(const struct S);} parsed as C. function_header[0] can match nothing, and start() calls it again forever. Fix in src/parser/srcMLParser.g: when it consumes nothing twice at the same token, clear MODE_FUNCTION_NAME.

The srcML commits are on https://github.com/EllianCarlos/srcML/tree/fix/cregit-nontermination (543fed6af, 50a63193f); the patch equals git diff d38271916 543fed6af. Patch sha256: ff21c1970823ec99b609273c03e4ae1cde1fcb932f2f24c1981256ce69a758b4.

Evidence

  • All 81 time-outs of the census so far finish, in at most 1.02 s of CPU each (64 versions of NSS certdata.c, 17 boost/hana headers).
  • Regression check on 8,492 files: 8,454 byte-identical token streams, 22 that never finished before, 12 corrections of the defect's own markup.
  • Open caveat: 4 copies of sqlite3.c differ, because a separate, older defect (an #error "...\ string continued over a backslash-newline) breaks both builds from that line on.
  • srcML parser test suite: 53,496 / 53,496 pass. cregit: make test, tests/test_token_worker.sh, prove tests/t (239), tests/tokenize_gate.sh (50/0), tests/retokenize_passthrough.sh (34/0) pass.
  • The build is bit-for-bit reproducible (nix build --rebuild); libsrcml.so.1.1.0 sha256 dea4467bb1b0a435e4536be13367325564b82e14ff69b478ad31bf144a8455f3.

Note

A new srcML changes the tokenizer identity digest for all 13 srcML extensions (Rust is unchanged). Existing outputs keep their old tokens; to pick up the fix, resume at step 2 with --retokenize for the srcML extensions. With this patch, all 12 entries of BlobDenylistEntries.scala parse; dropping them is a separate change.

srcML never finishes on two kinds of file, so the census excludes
them after the 600 s blob timeout: 64 versions of NSS certdata.c
and 17 boost/hana headers in the finished runs.

The patch, against the pinned source d38271916, carries:
- the lexer ends an ordinary C, C++ or Objective-C string at its
  closing quote, so "a""b" is two literals and not one open
  string that flips string and code for the rest of the file;
- a statement leaves function-name mode when the function-name
  rule consumes no token twice at the same token, which is the
  only way that rule can loop.

With it, all 81 excluded blobs finish, in at most 1.1 s of CPU.
So do the 4 C++-in-a-C-header blobs on the blob denylist. The 8
Java blobs there (srcML/srcML#2361) already finish at d38271916.

Measured over 8492 C, C++ and Java blobs from the census
originals, NSS at foxhound HEAD, boost/hana headers from
sonic-pi and files picked for the changed rules: 8454 token
streams are byte-identical, 22 are files that never finished
before, and 16 are files with adjacent string literals. In 12 of
them the lexer fix corrects the markup. The other 4 are copies
of sqlite3.c, where an #error string continued over a
backslash-newline already puts the lexer out of step, in both
builds; the fix only splits that wrong text differently.
srcML's own parser test suite passes 53496/53496 with the patch.
Without it, the suite fails the 66 adjacent-string cases and
does not finish the new macro-before-template case.

The tokenizer identity includes libsrcml, so a work directory
with cached tokens needs `--retokenize` for the srcML extensions.
@EllianCarlos
EllianCarlos merged commit 40f865e into master Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant