Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ jobs:
- name: Build & test Rust tokenizer (rust_tokenizer)
run: devenv shell -- bash -c 'cd tokenize/rustTokenizer && make && make test'

- name: Test shell tokenizer (sh_tokenizer.py)
run: devenv shell -- bash -c 'cd tokenize/shTokenizer && python3 -m unittest discover -s tests'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be make -C tokenize/shTokenizer test, like the Rust and srcML steps? Then the Makefile is the only place that knows how to run these tests.


- name: "Tokenizer worker: same output as tokenBySha.pl"
run: devenv shell -- bash tests/test_token_worker.sh

Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@ the checkout — the repo ships an [`.envrc`](./.envrc), so the devenv shell the
activates automatically whenever you `cd` in, and typing `devenv shell` is no
longer needed.

By default every file the tokenizer can parse is tokenized — C, C++, Java and
Rust, case-insensitively. The default mask is derived from the tokenizer's own
extension table, so it cannot drift from it; print it with `perl
By default every file the tokenizer can parse is tokenized — C, C++, Java,
Rust and sh, case-insensitively. The default mask is derived from the
tokenizer's own extension table, so it cannot drift from it; print it with `perl
tokenize/fileMask.pl`. Pass `--mask` to narrow it to one language. `.am`/`.ac`
are routed to the m4 parser but are not in the default mask, because that parser
mis-lexes real autotools quoting (see `%MASKED_LANGUAGES` in
Expand Down Expand Up @@ -90,6 +90,7 @@ This builds, in dependency order:
| ------------------------------------------------------------- | ------------------- | ------------------------- |
| `tokenize/srcMLtoken/srcml2token` | srcML tokenizer | gcc + xerces-c + libsrcml |
| `tokenize/rustTokenizer` binary | Rust tokenizer | cargo |
| `tokenize/shTokenizer/sh_tokenizer.py` (no build step) | sh tokenizer | python3 + tree-sitter-bash |
| `blobExec/target/scala-2.13/blobExec-0.1.0-assembly.jar` | tokenization driver | sbt, JDK 21 |
| `{slickGitLog,persons,remapCommits}/target/scala-2.10/*-one-jar.jar` | history / persons / remap tools | sbt 0.13, JDK 8 |

Expand Down Expand Up @@ -161,7 +162,7 @@ Flags (see `./run_pipeline_process.sh --help` for the full list):
| `--repo-url` | git URL or local path of the repository to process | **required** |
| `--repo-name` | short name prefixed to the output files | derived from `--repo-url` |
| `--commit-url` | base URL for commit links in the generated HTML | `<repo-url minus .git>/commit/` |
| `--mask` | regex of files to tokenize (C, C++, Java, Rust); quote it | `perl tokenize/fileMask.pl` |
| `--mask` | regex of files to tokenize (C, C++, Java, Rust, sh); quote it | `perl tokenize/fileMask.pl` |
| `--work` | working/output directory | `../cregit-files` |
| `--memo-dir` | where to memoize tokenized blobs; outside `--work` it survives the full-run wipe | `<work>/memo` |
| `--mode` / `--shards` | tokenizer walk mode / shard count for `sharded` | `pipeline` / `4` |
Expand Down
6 changes: 4 additions & 2 deletions devenv.nix
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@ in
perlEnv

# pytest must come from this python: the system one's duckdb will not load.
(pkgs.python3.withPackages (ps: [ ps.duckdb ps.pytest ]))
# tree-sitter + tree-sitter-bash: the shell tokenizer.
(pkgs.python3.withPackages (ps: [ ps.duckdb ps.pytest ps.tree-sitter ps.tree-sitter-bash ]))
];

# Scala 2.13 + sbt 1.x needs a modern JDK; the legacy sbt 0.13 hack is gone.
Expand All @@ -73,7 +74,8 @@ in
# in `packages`: that would change the shell env in srcml2token's RUNPATH, and
# so the tokenizer identity of every C, C++, Java and Rust blob.
enterTest = ''
git ls-files -z '*.sh' | xargs -0 --no-run-if-empty ${pkgs.shellcheck}/bin/shellcheck -S error
# The sh tokenizer's fixtures are its input, and some are invalid shell on purpose.
git ls-files -z -- '*.sh' ':!:tokenize/shTokenizer/tests/fixtures/' | xargs -0 --no-run-if-empty ${pkgs.shellcheck}/bin/shellcheck -S error
echo "shellcheck: no errors"
pytest -q
prove -r $(git ls-files '*/t/*.t' | xargs -n1 dirname | sort -u)
Expand Down
4 changes: 4 additions & 0 deletions tests/fixtures/tiny.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/bin/sh

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some edge cases we not covering by these fixtures?

I would suggest testing:

cat <<'a\b'
single quotes keep the backslash: the delimiter is a\b
a\b
echo after single
cat <<"E\"F"
in double quotes \" is a quote: the delimiter is E"F
E"F
echo after double
cat <<E\\F
unquoted \\ is one backslash: the delimiter is E\F
E\F
echo after unquoted

add () {
echo "$(( $1 + $2 ))" # "x|y"
}
13 changes: 12 additions & 1 deletion tests/t/tokenBySha.t
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

use strict;
use warnings;
use Test::More tests => 24;
use Test::More tests => 26;
use FindBin;
use File::Temp qw(tempdir);
use Digest::SHA qw(sha1_hex);
Expand Down Expand Up @@ -89,6 +89,17 @@ my $memoFile = "$memoDir/" . substr($sha1, 0, 2) . "/" . substr($sha1, 2, 2) . "
like($out, qr/^STUB-TOKENIZER\n--language=C\+\+\n/, ".cpp maps to --language=C++");
}

{
my ($status, $out, $err) = run_tokenbysha("echo hi\n",
BFG_MEMO_DIR => $memoDir,
BFG_TOKENIZE_CMD => $stub,
BFG_BLOB => "6" x 40,
BFG_FILENAME => "t0000-basic.sh",
);
is($status, 0, ".sh run succeeds");
like($out, qr/^STUB-TOKENIZER\n--language=Sh\n/, ".sh maps to --language=Sh");
}

{
my $failedContent = "int failure;\n";
my $failedSha1 = sha1_hex($failedContent);
Expand Down
56 changes: 56 additions & 0 deletions tests/t/tokenizeSh.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env perl

use strict;
use warnings;
use Test::More;
use FindBin;
use File::Temp qw(tempdir);

my $root = "$FindBin::Bin/../.."; # tests/t -> repo root

my $dispatcher = "$root/tokenize/tokenize.pl";
my $shDir = "$root/tokenize/shTokenizer";
my $fixture = "$shDir/tests/fixtures/01_test_body.sh";
my $expected = "$shDir/tests/expected/01_test_body.sh.nopos.token";

plan skip_all => "python3 with tree-sitter-bash not available (run inside devenv shell)"
unless system("python3 -c 'import tree_sitter, tree_sitter_bash' >/dev/null 2>&1") == 0;

plan tests => 7;

my $workdir = tempdir(CLEANUP => 1);

sub slurp {
my ($file) = @_;
open(my $fh, '<', $file) or die "unable to read [$file]: $!";
local $/;
my $content = <$fh>;
return defined $content ? $content : '';
}

sub run_dispatcher {
my (@args) = @_;
my $out = "$workdir/stdout";
my $err = "$workdir/stderr";
my $status = system("perl '$dispatcher' " . join(' ', @args) . " > '$out' 2> '$err'");
return ($status, slurp($out), slurp($err));
}

{
my ($status, $out) = run_dispatcher("--language=Sh", "'$fixture'");
is($status, 0, "dispatching a .sh file as Sh succeeds");
is($out, slurp($expected), "dispatcher output matches the shTokenizer expected output");
}

{
my ($status, $out) = run_dispatcher("'$fixture'");
is($status, 0, "autodetecting .sh succeeds");
is($out, slurp($expected), "autodetected output matches the explicit --language=Sh run");
}

{
my ($status, $out) = run_dispatcher("--language=Sh", "--position", "'$fixture'");
is($status, 0, "--position is forwarded to the shell tokenizer");
like($out, qr/\A-:-\|begin_unit\|/, "markers are prefixed with -:-|");
like($out, qr/^2:1\|command\|test_expect_success$/m, "tokens are prefixed with LINE:COL|");
}
30 changes: 30 additions & 0 deletions tests/t/tokenizerIdentity.t
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ sub fixture {
write_file("$tok/rustTokenizer/target/release/rust_tokenizer", "RUST TOKENIZER v1\n");
make_path("$tok/m4Tokenizer");
write_file("$tok/m4Tokenizer/m4.py", "M4 v1\n");
make_path("$tok/shTokenizer");
write_sh_tokenizer($dir, "v1");
write_file("$dir/tree-sitter", "TREE-SITTER v1\n");
write_file("$dir/tree-sitter-bash", "TREE-SITTER-BASH v1\n");
write_srcml2token($dir, "v1");
write_file("$dir/libsrcml", "LIBSRCML v1\n");
write_file("$dir/ctags", "CTAGS v1\n");
Expand All @@ -49,6 +53,12 @@ sub write_srcml2token {
write_file("$dir/srcml2token", "#!/bin/sh\n# SRCML2TOKEN $version\necho '$dir/libsrcml'\n");
}

sub write_sh_tokenizer {
my ($dir, $version) = @_;
write_file("$dir/tokenize/shTokenizer/sh_tokenizer.py",
"#!/bin/sh\n# SH TOKENIZER $version\necho '$dir/tree-sitter'\necho '$dir/tree-sitter-bash'\n");
}

sub identity {
my ($dir, @extra) = @_;
my @cmd = ("perl", "$dir/tokenize/tokenizerIdentity.pl",
Expand Down Expand Up @@ -113,6 +123,18 @@ for my $component (qw(srcml2token libsrcml ctags)) {
is($a{rs}, $b{rs}, "changing $component leaves .rs alone");
}

for my $component (qw(sh_tokenizer.py tree-sitter tree-sitter-bash)) {
my $d = fixture();
my ($before) = identity($d);
if ($component eq "sh_tokenizer.py") { write_sh_tokenizer($d, "v2") }
else { write_file("$d/$component", "$component v2\n") }
my ($after) = identity($d);
my %b = as_map($before);
my %a = as_map($after);
isnt($a{sh}, $b{sh}, "changing $component moves the .sh identity");
is($a{c}, $b{c}, "changing $component leaves .c alone");
}

# CregitSrcMl.pm is tokenizeSrcMl.pl's copy in the tokenizer worker.
for my $script (qw(tokenizeSrcMl.pl CregitSrcMl.pm)) {
my $d = fixture();
Expand Down Expand Up @@ -169,6 +191,14 @@ for my $shared (qw(tokenize.pl CregitLanguages.pm)) {
is($out, "", "and prints no identity");
}

{
my $d = fixture();
write_file("$d/tokenize/shTokenizer/sh_tokenizer.py", "#!/bin/sh\nexit 0\n");
my ($out, $status) = identity($d);
isnt($status, 0, "an sh tokenizer that does not name its tree-sitter libraries is fatal");
is($out, "", "and prints no identity");
}

# ---------------------------------------------------------------------------
{
my ($all) = identity($dir, "--all-extensions");
Expand Down
4 changes: 4 additions & 0 deletions tokenize/CregitLanguages.pm
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ our %EXT_LANG = (
'am' => 'M4',
# Rust
'rs' => 'Rust',
# Shell
'sh' => 'Sh',
);

# Languages the pipeline's file mask is allowed to select, i.e. the ones whose
Expand Down Expand Up @@ -102,6 +104,7 @@ our %MASKED_LANGUAGES = (
'C++' => 1,
'Java' => 1,
'Rust' => 1,
'Sh' => 1,
);

# Language -> parser, relative to the tokenize/ directory. Every value must
Expand All @@ -112,6 +115,7 @@ our %LANG_PARSER_REL = (
'Java' => 'tokenizeSrcMl.pl',
'M4' => 'm4Tokenizer/m4.py',
'Rust' => 'rustTokenizer/target/release/rust_tokenizer',
'Sh' => 'shTokenizer/sh_tokenizer.py',
);

# The parser every srcML-routed language shares. Named so callers can ask "is
Expand Down
14 changes: 14 additions & 0 deletions tokenize/shTokenizer/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.PHONY: test expected

PYTHON ?= python3

test:
$(PYTHON) -m unittest discover -s tests

expected:
mkdir -p tests/expected
for fixture in tests/fixtures/*.sh; do \
name=$$(basename "$$fixture"); \
$(PYTHON) sh_tokenizer.py --position "$$fixture" > "tests/expected/$$name.token"; \
$(PYTHON) sh_tokenizer.py "$$fixture" > "tests/expected/$$name.nopos.token"; \
done
147 changes: 147 additions & 0 deletions tokenize/shTokenizer/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# shTokenizer

`sh_tokenizer.py` tokenizes POSIX sh / bash scripts into the line-oriented view used
by cregit, the format consumed by `blameRepo` and `generate_dataset`.

It uses [tree-sitter-bash](https://github.com/tree-sitter/tree-sitter-bash) for the
syntax, plus its own heredoc locator: tree-sitter-bash misreads heredoc bodies in common
forms, such as a `&&` after the delimiter (`cat >expect <<-\EOF &&`), and reads their
text as code.

## Requirements

Python 3 with the `tree-sitter` and `tree-sitter-bash` packages (provided by
`devenv shell`).

## How to use

```sh
sh_tokenizer.py [--language=Sh] [--position] [--verbose] <file> [<output>]
sh_tokenizer.py --library-paths
```

The token stream goes to `<output>`, or to _stdout_ without it. `--language` and
`--verbose` are accepted so the `tokenize.pl` dispatcher can call it like the other
tokenizers. `--position` prefixes every line with `LINE:COL|` (tokens) or `-:-|`
(markers).

`--library-paths` prints the files of the compiled tree-sitter runtime and bash
grammar. They decide the tokens as much as `sh_tokenizer.py` does, so
`tokenize/tokenizerIdentity.pl` digests them in the tokenizer identity of `.sh`.

Exit status: 0 whenever the file can be read, including files with syntax errors;
1 if the file cannot be read; 2 on a usage error.

## Output format

One line per token or marker:

```
begin_unit|revision:0.1.0;language:Sh;cregit-version:0.0.1
kind|value
...
end_unit

```

Every `end_*` marker, `end_unit` included, is followed by an empty line (the line
`-:-|` with `--position`), exactly as `tokenizeSrcMl.pl` does for C, C++ and Java.
`generate_dataset` classifies these lines as structural blanks.

If a file cannot be tokenized (an internal error, or an output that fails the checks of
this format), a warning goes to _stderr_ and the output is the plain split of the file
into words, all of kind `unknown`, so the pipeline never stops on a file.

### Token kinds

| kind | what |
|---|---|
| `keyword` | alphabetic reserved words: `if then else elif fi case esac for select while until do done in function` |
| `command` | the command name of a simple command (`git`, `:`, `.`, `return`, `time`), and the builtins `export`, `local`, `readonly`, `declare`, `typeset`, `unset` and `unsetenv` |
| `word` | any other unquoted word: arguments, options, paths, patterns, test operators and the numbers they compare (`-z`, `-eq 1`), the name in a function definition |
| `variable` | variable names in expansions, assignments and declaration commands, special parameters (`1`, `@`, `#`, `?`) |
| `number` | numbers in arithmetic (`$(( ))`, `$[ ]`, `(( ))`, the `(( ))` of `for`), file descriptors in redirections (`2>`, `6<<`, `>&2`) |
| `op` | operators and punctuation: `&& \|\| \| ; ;; & ( ) { } ! = < > >> <& >& << <<- $ ${ $( ) $(( ))`, backquotes, the brackets `[ ] [[ ]]` of a test command, arithmetic operators, the `\` of a line continuation |
| `literal` | a whole string: `'...'`, `"..."` or `$'...'` |
| `heredoc` | a whole heredoc body |
| `heredoc_delim` | the delimiter word after `<<` / `<<-` and the terminator line |
| `comment` | a whole comment, `#` to end of line |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallback emits unknown (line 53), but it is missing from this table. Could you add a row for it?


### Values

- The value is the exact source text of the token, never the value the shell would
compute: quotes, backslashes and escapes are kept.
- CR and LF inside a value are replaced by a space, so every token fits on one line.
- Values are never empty.
- Values are raw bytes: files that are not valid UTF-8 are copied through unchanged.
Tokens are only split at ASCII characters, never inside a multi-byte sequence.
- Concatenating all values after removing whitespace (space, tab, CR, LF) gives the
source file after removing whitespace. No non-whitespace byte is dropped or added.

### Strings

A string is one `literal` token holding its whole source text, quotes and expansions
included, as the other tokenizers emit a string literal. In a concatenation such as
`'a'"b"c`, each quoted piece is a literal of its own and `c` is a `word`. A heredoc
inside a string (in `"$(cat <<EOF ...)"`) is emitted by the heredoc rules and cuts the
string into several literals.

### Heredocs

The body of a heredoc is one `heredoc` token, expansions included. The operator is an
`op`; the delimiter and the terminator are `heredoc_delim`.

Heredoc extents are decided by the tokenizer, not by tree-sitter, following the shell:
operators `<<` and `<<-` in code, `$( )` and backquotes included, but not inside quotes,
comments or arithmetic (`$(( ))`, `(( ))`); `<<<` is not a heredoc. A body starts after
the end of the line holding its operator, or after the previous body started from that
line; it ends before the first line equal to the unquoted delimiter, leading tabs
removed for `<<-`; a body without a terminator runs to the end of the file.

### Structural markers

Markers carry no source text. `generate_dataset` classifies them as structural
(`is_structural = 1`).

| markers | around |
|---|---|
| `begin_unit|...` / `end_unit` | the whole file |
| `begin_function`, `DECL|function|<name>` / `end_function` | a function definition |

### Positions

With `--position`, `LINE:COL` is the 1-based position of the first byte of the token;
`COL` counts the characters of the line decoded as UTF-8 with invalid bytes replaced, as
`generate_dataset` reads the source: a truncated multi-byte sequence counts as one
character, and so does each other invalid byte.

## Limitations

- A string or heredoc body that spans several lines is one line of the tokenized file,
so its blame is coarser than the line blame of git. Code passed as a string is a
string too: in the git test suite, the whole body of `test_expect_success 'title' '...'`
is one `literal`.
- The heredoc locator reads `((` at the start of a word as arithmetic; bash reads it as
two nested subshells when it is not valid arithmetic (`((cd x) )`).
- Inside backquotes, a backquote within quotes opens a nested command substitution;
bash ends the backquotes there.
- tree-sitter-bash cuts an unquoted word at a form feed or a vertical tab, which the shell
reads as part of the word; that byte is then a `word` token of its own.
- When tree-sitter-bash pairs the quotes of a line wrongly (as in the `case` pattern
`?*' '?*)`) and reaches a quote it cannot close, everything from that quote to the end
of the file is one error token, which comes out as a single `word`.

## How to test

```sh
cd tokenize/shTokenizer
make test
```

`make expected` regenerates the expected outputs in `tests/expected/` from
`tests/fixtures/`: `NAME.sh.token` with `--position` and `NAME.sh.nopos.token` without.
Check the diff of every regenerated file.

## License

GPL-3.0+
Loading
Loading