Skip to content

feat(tokenize): add a tokenizer for sh files - #112

Open
isacaselli wants to merge 3 commits into
ccsl-codev:masterfrom
isacaselli:feat/sh-tokenizer
Open

isacaselli wants to merge 3 commits into
ccsl-codev:masterfrom
isacaselli:feat/sh-tokenizer

Conversation

@isacaselli

Copy link
Copy Markdown

Closes #111.

Summary

Adds tokenize/shTokenizer/sh_tokenizer.py, a tokenizer for POSIX sh / bash scripts,
and routes .sh to it through CregitLanguages.pm.

It parses with tree-sitter-bash, plus its own heredoc locator: tree-sitter-bash reads
many heredoc bodies as code (for instance when && follows the delimiter,
cat >expect <<-\EOF &&, the usual style of the git test suite), so the heredocs are
found by the rules of the shell and blanked before parsing.

Output

The line format of the other tokenizers (kind|value, begin_unit / end_unit, a
blank line after every end_*, --position as LINE:COL|):

  • a string is one literal token, as srcML emits a string literal; a heredoc body is
    one heredoc token;
  • markers only for function definitions: begin_function,
    DECL|function|<name>, end_function;
  • kinds keyword, command, word, variable, number, op, literal,
    heredoc, heredoc_delim, comment; the README lists what each covers;
  • the token values rebuild the file, whitespace aside; a file whose output fails that
    check, or that raises, falls back to a plain word split of kind unknown with a
    warning, so the pipeline never stops on a file.

Commits

  1. feat(tokenize): add a tokenizer for sh files: sh_tokenizer.py and its README.
  2. feat(tokenize): route .sh files to the sh tokenizer: .sh in
    CregitLanguages.pm (routed and in the default mask), Sh in the usage of
    tokenize.pl, the tree-sitter libraries in the .sh identity of
    tokenizerIdentity.pl, tree-sitter in the devenv Python, the root README. The
    changes to tokenizerIdentity.t come with it: once .sh is in the default mask,
    the fixture of that test needs an sh tokenizer.
  3. test(tokenize): test the sh tokenizer: its unit tests, golden fixtures and
    Makefile, a CI step for them, tokenizeSh.t, the .sh test of tokenBySha.t,
    a .sh fixture for the tokenizer worker test, and the shellcheck exclusion of the
    fixtures in devenv test.

prove, pytest, the pipeline shell tests and the tokenizer worker test pass at
each commit.

Compatibility

  • New dependency: the tree-sitter and tree-sitter-bash Python packages in the
    devenv Python. This changes the devenv shell environment, which is in the RUNPATH of
    a freshly built srcml2token.
  • Default mask: it now selects .sh, so default runs tokenize shell scripts.
  • Tokenizer identity: tokenize.pl and CregitLanguages.pm change, so the
    identity of every extension moves; existing blob maps need --retokenize or a fresh
    run.
  • CLI: tokenize.pl --language=Sh; sh_tokenizer.py takes the options the
    dispatcher passes (--language, --position, --verbose, an output file) and
    --library-paths.
  • devenv test: the shellcheck gate skips tokenize/shTokenizer/tests/fixtures/.
    Those files are tokenizer input cut from git/git: they have no shebang, or a header
    comment before it, and one ends in the middle of a && chain.

Validation

  • make -C tokenize/shTokenizer test: 36 tests, 12 golden fixtures taken from
    git/git at 18e66859, with and without --position.
  • prove tests/t: new tokenizeSh.t (7 tests), 2 new tests in tokenBySha.t, 8 in
    tokenizerIdentity.t.
  • bash tests/test_token_worker.sh with the new tests/fixtures/tiny.sh: same tokens
    and memo file through the worker as through tokenBySha.pl.
  • The pipeline shell tests, pytest -q and devenv test pass unchanged.
  • Every .sh blob of git/git's history, all refs (31,899 blobs): no fallback; every
    output passes the format check.
  • Fuzzing: 224,000 mutations of git/git's .sh files (truncations, deletions, inserted
    quotes, heredoc operators, expansions and arithmetic): no fallback.

This tokenizer is the generic part of the one used to build a token-level authorship
dataset of git/git's shell scripts, which was checked against line blame and by hand.
On the 31,899 blobs its output is identical to that version run without the git test
bodies, apart from the markers of strings and heredoc bodies, dropped here, and 6 blobs
in which a string holds a heredoc, now cut into literal pieces instead of words.

Limitations

In the README: a multi-line string or heredoc body is one line of the tokenized file,
so its blame is coarser than git's line blame, and code passed as a string (such as a
git test body) is a string; plus four tree-sitter-bash and heredoc corner cases.

sh_tokenizer.py tokenizes POSIX sh and bash scripts in the line
format of the other tokenizers. It parses with tree-sitter-bash and
finds the heredocs by the rules of the shell, because tree-sitter-bash
reads many heredoc bodies as code, for instance when && follows the
delimiter (cat >expect <<-\EOF &&).

A string is one literal token and a heredoc body one heredoc token.
Function definitions get begin_function, DECL|function|<name> and
end_function markers. A file that fails the format checks, or on
which the tokenizer raises, comes out as a plain word split of kind
unknown, with a warning.

--library-paths prints the compiled tree-sitter runtime and bash
grammar, which decide the tokens as much as the script does.
CregitLanguages.pm maps .sh to Sh, runs shTokenizer/sh_tokenizer.py
for it and adds Sh to the default mask, so a run without --mask now
tokenizes shell scripts. tokenize.pl lists Sh in its usage.

tokenizerIdentity.pl digests, for .sh, the tree-sitter runtime and
bash grammar that sh_tokenizer.py --library-paths names, as it digests
the libsrcml that srcml2token names. It stops when the tokenizer does
not name them. tokenizerIdentity.t gives its fixture a stub tokenizer
that names two library files, and checks that a change to the
tokenizer or to either library moves the .sh identity only.

The devenv Python gets the tree-sitter and tree-sitter-bash packages.

Both tokenize.pl and CregitLanguages.pm are in the identity of every
extension, so existing blob maps need --retokenize or a fresh run.
The new packages change the devenv shell environment, which is in
the RUNPATH of a freshly built srcml2token.
tokenize/shTokenizer/tests has unit tests of the heredoc scanner, the
token kinds, positions, the fallback and the command line, and 12
golden fixtures cut from git/git at 18e66859, each with its expected
output with and without --position. `make test` runs them, and
`make expected` regenerates the expected outputs. CI runs them in a
step of its own.

tokenizeSh.t runs a fixture through tokenize.pl, with --language=Sh,
by the .sh extension and with --position. tokenBySha.t checks that a
.sh blob reaches the tokenizer with --language=Sh. tests/fixtures/
tiny.sh puts a .sh file through the tokenizer worker test.

devenv test no longer runs shellcheck on the fixtures of the sh
tokenizer. They are tokenizer input: some have no shebang or a
comment before it, and one ends in the middle of a && chain.
Comment thread tests/fixtures/tiny.sh
@@ -0,0 +1,4 @@
#!/bin/sh

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some edge cases we not covering by these fixtures?

I would suggest testing:

cat <<'a\b'
single quotes keep the backslash: the delimiter is a\b
a\b
echo after single
cat <<"E\"F"
in double quotes \" is a quote: the delimiter is E"F
E"F
echo after double
cat <<E\\F
unquoted \\ is one backslash: the delimiter is E\F
E\F
echo after unquoted

word = text[word_start:word_end].replace(b"\\\n", b"") # line continuations
return Heredoc(io_start=io_start, op_start=i, op_end=op_end, word_start=word_start,
word_end=word_end, strip_tabs=strip_tabs,
delimiter=bytes(b for b in word if b not in b"\\'\""))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strips every backslash and quote from the word, which is not shell quote removal. With cat <<'a\b' bash ends the heredoc at a\b, but the tokenizer looks for ab, so the body swallows the rest of the file (same for <<"E\"F" and <<E\\F, see the fixture suggested on tiny.sh). The format check can't catch it, since a swallowed body still rebuilds the file, so it won't show up as a fallback either.

Suggested fix:

def unquote_word(word: bytes) -> bytes:
    out, i, n = bytearray(), 0, len(word)
    while i < n:
        byte = word[i]
        if byte == BACKSLASH:
            out += word[i + 1:i + 2]
            i += 2
        elif byte == SINGLE_QUOTE:
            close = word.find(b"'", i + 1)
            close = n if close < 0 else close
            out += word[i + 1:close]
            i = close + 1
        elif byte == DOUBLE_QUOTE:
            i += 1
            while i < n and word[i] != DOUBLE_QUOTE:
                if word[i] == BACKSLASH and i + 1 < n and word[i + 1] in b'"\\$`':
                    i += 1
                out.append(word[i])
                i += 1
            i += 1
        else:
            out.append(byte)
            i += 1
    return bytes(out)

with delimiter=unquote_word(word). skip_quoted_word also needs to skip \x inside double quotes, otherwise "E\"F" ends at the inner quote.

heredoc.body_start = heredoc.body_end = heredoc.line_end = len(self.text)
return self.heredocs + self.pending

def single_quoted(self, i: int) -> int:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the quote/backslash skipping here and in skip_quoted_word (line 261) is the same logic written twice, and the second copy is the one that misses \" inside double quotes. Could they share one helper?

Comment thread .github/workflows/ci.yml
run: devenv shell -- bash -c 'cd tokenize/rustTokenizer && make && make test'

- name: Test shell tokenizer (sh_tokenizer.py)
run: devenv shell -- bash -c 'cd tokenize/shTokenizer && python3 -m unittest discover -s tests'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be make -C tokenize/shTokenizer test, like the Rust and srcML steps? Then the Makefile is the only place that knows how to run these tests.

| `literal` | a whole string: `'...'`, `"..."` or `$'...'` |
| `heredoc` | a whole heredoc body |
| `heredoc_delim` | the delimiter word after `<<` / `<<-` and the terminator line |
| `comment` | a whole comment, `#` to end of line |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallback emits unknown (line 53), but it is missing from this table. Could you add a row for it?

reason = f"internal error: {type(exc).__name__}: {exc}"
if reason is None:
return records
print(f"sh_tokenizer: falling back to word split for [{name}]: {reason}", file=sys.stderr)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not blocking: the fallback exits 0 with only a warning on stderr, so a run can degrade blame data without anyone noticing. Would it be worth counting the fallbacks somewhere the pipeline reports them?

class HeredocScanner:
def __init__(self, text: bytes):
self.text = text
self.stack = [Context("code")]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the scanner modes are bare strings ("code", "single", "double", "arith", "backquote") compared with context.kind == "...". An Enum would catch typos.

yield m.start(), m.end()


def flatten(value: bytes) -> bytes:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit on names: flatten folds CR/LF into spaces and strips trailing whitespace, so something like one_line would say what it does. Same for check (line 517), which returns an error string (validation_error?), and tie in Output.add (line 53).

@EllianCarlos

Copy link
Copy Markdown
Collaborator

Also, I ran the ci/cd and it failed, if you can check and fix the errors it would be good as well, thanks!

@EllianCarlos

Copy link
Copy Markdown
Collaborator

Two more points while you look at the CI failure:

  1. The cause of the CI failure: the test every_masked_language_routes_to_a_tokenizer_this_file_pins in tokenize/rustTokenizer/tests/framing.rs panics. Sh is now in %MASKED_LANGUAGES, but shTokenizer/sh_tokenizer.py has no framing pin in the pinned list. A framing check for the sh tokenizer, added to pinned, makes the test pass. Please put it in the same commit that adds Sh to the mask (or before it), so that each commit passes on its own.

  2. Tokenizer identity: this PR changes tokenize.pl and CregitLanguages.pm, and that changes the tokenizer identity of every language, not only Sh. So every existing blob map (C, C++, Java, Rust) needs a full --retokenize on its next run. Can the identity change be limited to Sh files, so that existing blob maps stay valid? If not, please say it in the PR description, so that users know they must retokenize.

The inline comments of 18 October still apply; the heredoc delimiter (quote removal) is the main one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support sh / bash files in the tokenizer

2 participants