Repository navigation
fix(srcml): patch the two non-termination defects - #109
Merged
Merged
Conversation
srcML never finishes on two kinds of file, so the census excludes them after the 600 s blob timeout: 64 versions of NSS certdata.c and 17 boost/hana headers in the finished runs. The patch, against the pinned source d38271916, carries: - the lexer ends an ordinary C, C++ or Objective-C string at its closing quote, so "a""b" is two literals and not one open string that flips string and code for the rest of the file; - a statement leaves function-name mode when the function-name rule consumes no token twice at the same token, which is the only way that rule can loop. With it, all 81 excluded blobs finish, in at most 1.1 s of CPU. So do the 4 C++-in-a-C-header blobs on the blob denylist. The 8 Java blobs there (srcML/srcML#2361) already finish at d38271916. Measured over 8492 C, C++ and Java blobs from the census originals, NSS at foxhound HEAD, boost/hana headers from sonic-pi and files picked for the changed rules: 8454 token streams are byte-identical, 22 are files that never finished before, and 16 are files with adjacent string literals. In 12 of them the lexer fix corrects the markup. The other 4 are copies of sqlite3.c, where an #error string continued over a backslash-newline already puts the lexer out of step, in both builds; the fix only splits that wrong text differently. srcML's own parser test suite passes 53496/53496 with the patch. Without it, the suite fails the 66 adjacent-string cases and does not finish the new macro-before-template case. The tokenizer identity includes libsrcml, so a work directory with cached tokens needs `--retokenize` for the srcML extensions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The census found two srcML defects that make the parser run without end. Each hung file costs 600 s and is then left out of the dataset. This PR builds srcML (develop d382719) with a patch that fixes both:
nix/srcml.nixappliesnix/srcml-nontermination.patch.char *s = "a""b";(C, C++, ObjC). The old lexer took the run of quotes as the closing quote, so string and code swapped for the rest of the file. On NSScertdata.cthis makes the parser superlinear (100 KB 0.28 s, 200 KB 116.6 s, 400 KB and more: over 900 s). Fix insrc/parser/CommentTextLexer.g: leave the quote loop whenmode == STRING_ENDand notnoescape, in C, C++ and ObjC only.X template <class R> auto a;(C++), andnamespace n{struct T&f(const struct S);}parsed as C.function_header[0]can match nothing, andstart()calls it again forever. Fix insrc/parser/srcMLParser.g: when it consumes nothing twice at the same token, clearMODE_FUNCTION_NAME.The srcML commits are on https://github.com/EllianCarlos/srcML/tree/fix/cregit-nontermination (543fed6af, 50a63193f); the patch equals
git diff d38271916 543fed6af. Patch sha256:ff21c1970823ec99b609273c03e4ae1cde1fcb932f2f24c1981256ce69a758b4.Evidence
certdata.c, 17 boost/hana headers).sqlite3.cdiffer, because a separate, older defect (an#error "...\string continued over a backslash-newline) breaks both builds from that line on.make test,tests/test_token_worker.sh,prove tests/t(239),tests/tokenize_gate.sh(50/0),tests/retokenize_passthrough.sh(34/0) pass.nix build --rebuild);libsrcml.so.1.1.0sha256dea4467bb1b0a435e4536be13367325564b82e14ff69b478ad31bf144a8455f3.Note
A new srcML changes the tokenizer identity digest for all 13 srcML extensions (Rust is unchanged). Existing outputs keep their old tokens; to pick up the fix, resume at step 2 with
--retokenizefor the srcML extensions. With this patch, all 12 entries ofBlobDenylistEntries.scalaparse; dropping them is a separate change.