Repository navigation
feat(tokenize): add a tokenizer for sh files - #112
isacaselli wants to merge 3 commits into
Conversation
sh_tokenizer.py tokenizes POSIX sh and bash scripts in the line format of the other tokenizers. It parses with tree-sitter-bash and finds the heredocs by the rules of the shell, because tree-sitter-bash reads many heredoc bodies as code, for instance when && follows the delimiter (cat >expect <<-\EOF &&). A string is one literal token and a heredoc body one heredoc token. Function definitions get begin_function, DECL|function|<name> and end_function markers. A file that fails the format checks, or on which the tokenizer raises, comes out as a plain word split of kind unknown, with a warning. --library-paths prints the compiled tree-sitter runtime and bash grammar, which decide the tokens as much as the script does.
CregitLanguages.pm maps .sh to Sh, runs shTokenizer/sh_tokenizer.py for it and adds Sh to the default mask, so a run without --mask now tokenizes shell scripts. tokenize.pl lists Sh in its usage. tokenizerIdentity.pl digests, for .sh, the tree-sitter runtime and bash grammar that sh_tokenizer.py --library-paths names, as it digests the libsrcml that srcml2token names. It stops when the tokenizer does not name them. tokenizerIdentity.t gives its fixture a stub tokenizer that names two library files, and checks that a change to the tokenizer or to either library moves the .sh identity only. The devenv Python gets the tree-sitter and tree-sitter-bash packages. Both tokenize.pl and CregitLanguages.pm are in the identity of every extension, so existing blob maps need --retokenize or a fresh run. The new packages change the devenv shell environment, which is in the RUNPATH of a freshly built srcml2token.
tokenize/shTokenizer/tests has unit tests of the heredoc scanner, the token kinds, positions, the fallback and the command line, and 12 golden fixtures cut from git/git at 18e66859, each with its expected output with and without --position. `make test` runs them, and `make expected` regenerates the expected outputs. CI runs them in a step of its own. tokenizeSh.t runs a fixture through tokenize.pl, with --language=Sh, by the .sh extension and with --position. tokenBySha.t checks that a .sh blob reaches the tokenizer with --language=Sh. tests/fixtures/ tiny.sh puts a .sh file through the tokenizer worker test. devenv test no longer runs shellcheck on the fixtures of the sh tokenizer. They are tokenizer input: some have no shebang or a comment before it, and one ends in the middle of a && chain.
| @@ -0,0 +1,4 @@ | |||
| #!/bin/sh | |||
There was a problem hiding this comment.
Is there some edge cases we not covering by these fixtures?
I would suggest testing:
cat <<'a\b'
single quotes keep the backslash: the delimiter is a\b
a\b
echo after single
cat <<"E\"F"
in double quotes \" is a quote: the delimiter is E"F
E"F
echo after double
cat <<E\\F
unquoted \\ is one backslash: the delimiter is E\F
E\F
echo after unquoted| word = text[word_start:word_end].replace(b"\\\n", b"") # line continuations | ||
| return Heredoc(io_start=io_start, op_start=i, op_end=op_end, word_start=word_start, | ||
| word_end=word_end, strip_tabs=strip_tabs, | ||
| delimiter=bytes(b for b in word if b not in b"\\'\"")) |
There was a problem hiding this comment.
This strips every backslash and quote from the word, which is not shell quote removal. With cat <<'a\b' bash ends the heredoc at a\b, but the tokenizer looks for ab, so the body swallows the rest of the file (same for <<"E\"F" and <<E\\F, see the fixture suggested on tiny.sh). The format check can't catch it, since a swallowed body still rebuilds the file, so it won't show up as a fallback either.
Suggested fix:
def unquote_word(word: bytes) -> bytes:
out, i, n = bytearray(), 0, len(word)
while i < n:
byte = word[i]
if byte == BACKSLASH:
out += word[i + 1:i + 2]
i += 2
elif byte == SINGLE_QUOTE:
close = word.find(b"'", i + 1)
close = n if close < 0 else close
out += word[i + 1:close]
i = close + 1
elif byte == DOUBLE_QUOTE:
i += 1
while i < n and word[i] != DOUBLE_QUOTE:
if word[i] == BACKSLASH and i + 1 < n and word[i + 1] in b'"\\$`':
i += 1
out.append(word[i])
i += 1
i += 1
else:
out.append(byte)
i += 1
return bytes(out)with delimiter=unquote_word(word). skip_quoted_word also needs to skip \x inside double quotes, otherwise "E\"F" ends at the inner quote.
| heredoc.body_start = heredoc.body_end = heredoc.line_end = len(self.text) | ||
| return self.heredocs + self.pending | ||
|
|
||
| def single_quoted(self, i: int) -> int: |
There was a problem hiding this comment.
Nit: the quote/backslash skipping here and in skip_quoted_word (line 261) is the same logic written twice, and the second copy is the one that misses \" inside double quotes. Could they share one helper?
| run: devenv shell -- bash -c 'cd tokenize/rustTokenizer && make && make test' | ||
|
|
||
| - name: Test shell tokenizer (sh_tokenizer.py) | ||
| run: devenv shell -- bash -c 'cd tokenize/shTokenizer && python3 -m unittest discover -s tests' |
There was a problem hiding this comment.
Could this be make -C tokenize/shTokenizer test, like the Rust and srcML steps? Then the Makefile is the only place that knows how to run these tests.
| | `literal` | a whole string: `'...'`, `"..."` or `$'...'` | | ||
| | `heredoc` | a whole heredoc body | | ||
| | `heredoc_delim` | the delimiter word after `<<` / `<<-` and the terminator line | | ||
| | `comment` | a whole comment, `#` to end of line | |
There was a problem hiding this comment.
The fallback emits unknown (line 53), but it is missing from this table. Could you add a row for it?
| reason = f"internal error: {type(exc).__name__}: {exc}" | ||
| if reason is None: | ||
| return records | ||
| print(f"sh_tokenizer: falling back to word split for [{name}]: {reason}", file=sys.stderr) |
There was a problem hiding this comment.
Not blocking: the fallback exits 0 with only a warning on stderr, so a run can degrade blame data without anyone noticing. Would it be worth counting the fallbacks somewhere the pipeline reports them?
| class HeredocScanner: | ||
| def __init__(self, text: bytes): | ||
| self.text = text | ||
| self.stack = [Context("code")] |
There was a problem hiding this comment.
Nit: the scanner modes are bare strings ("code", "single", "double", "arith", "backquote") compared with context.kind == "...". An Enum would catch typos.
| yield m.start(), m.end() | ||
|
|
||
|
|
||
| def flatten(value: bytes) -> bytes: |
There was a problem hiding this comment.
Nit on names: flatten folds CR/LF into spaces and strips trailing whitespace, so something like one_line would say what it does. Same for check (line 517), which returns an error string (validation_error?), and tie in Output.add (line 53).
|
Also, I ran the ci/cd and it failed, if you can check and fix the errors it would be good as well, thanks! |
|
Two more points while you look at the CI failure:
The inline comments of 18 October still apply; the heredoc delimiter (quote removal) is the main one. |
Closes #111.
Summary
Adds
tokenize/shTokenizer/sh_tokenizer.py, a tokenizer for POSIX sh / bash scripts,and routes
.shto it throughCregitLanguages.pm.It parses with tree-sitter-bash, plus its own heredoc locator: tree-sitter-bash reads
many heredoc bodies as code (for instance when
&&follows the delimiter,cat >expect <<-\EOF &&, the usual style of the git test suite), so the heredocs arefound by the rules of the shell and blanked before parsing.
Output
The line format of the other tokenizers (
kind|value,begin_unit/end_unit, ablank line after every
end_*,--positionasLINE:COL|):literaltoken, as srcML emits a string literal; a heredoc body isone
heredoctoken;begin_function,DECL|function|<name>,end_function;keyword,command,word,variable,number,op,literal,heredoc,heredoc_delim,comment; the README lists what each covers;check, or that raises, falls back to a plain word split of kind
unknownwith awarning, so the pipeline never stops on a file.
Commits
feat(tokenize): add a tokenizer for sh files:sh_tokenizer.pyand its README.feat(tokenize): route .sh files to the sh tokenizer:.shinCregitLanguages.pm(routed and in the default mask),Shin the usage oftokenize.pl, the tree-sitter libraries in the.shidentity oftokenizerIdentity.pl, tree-sitter in the devenv Python, the root README. Thechanges to
tokenizerIdentity.tcome with it: once.shis in the default mask,the fixture of that test needs an sh tokenizer.
test(tokenize): test the sh tokenizer: its unit tests, golden fixtures andMakefile, a CI step for them,tokenizeSh.t, the.shtest oftokenBySha.t,a
.shfixture for the tokenizer worker test, and the shellcheck exclusion of thefixtures in
devenv test.prove,pytest, the pipeline shell tests and the tokenizer worker test pass ateach commit.
Compatibility
tree-sitterandtree-sitter-bashPython packages in thedevenv Python. This changes the devenv shell environment, which is in the RUNPATH of
a freshly built
srcml2token..sh, so default runs tokenize shell scripts.tokenize.plandCregitLanguages.pmchange, so theidentity of every extension moves; existing blob maps need
--retokenizeor a freshrun.
tokenize.pl --language=Sh;sh_tokenizer.pytakes the options thedispatcher passes (
--language,--position,--verbose, an output file) and--library-paths.devenv test: the shellcheck gate skipstokenize/shTokenizer/tests/fixtures/.Those files are tokenizer input cut from git/git: they have no shebang, or a header
comment before it, and one ends in the middle of a
&&chain.Validation
make -C tokenize/shTokenizer test: 36 tests, 12 golden fixtures taken fromgit/git at
18e66859, with and without--position.prove tests/t: newtokenizeSh.t(7 tests), 2 new tests intokenBySha.t, 8 intokenizerIdentity.t.bash tests/test_token_worker.shwith the newtests/fixtures/tiny.sh: same tokensand memo file through the worker as through
tokenBySha.pl.pytest -qanddevenv testpass unchanged..shblob of git/git's history, all refs (31,899 blobs): no fallback; everyoutput passes the format check.
.shfiles (truncations, deletions, insertedquotes, heredoc operators, expansions and arithmetic): no fallback.
This tokenizer is the generic part of the one used to build a token-level authorship
dataset of git/git's shell scripts, which was checked against line blame and by hand.
On the 31,899 blobs its output is identical to that version run without the git test
bodies, apart from the markers of strings and heredoc bodies, dropped here, and 6 blobs
in which a string holds a heredoc, now cut into
literalpieces instead of words.Limitations
In the README: a multi-line string or heredoc body is one line of the tokenized file,
so its blame is coarser than git's line blame, and code passed as a string (such as a
git test body) is a string; plus four tree-sitter-bash and heredoc corner cases.