Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,14 +76,47 @@ jobs:
- name: Build & test Rust tokenizer (rust_tokenizer)
run: devenv shell -- bash -c 'cd tokenize/rustTokenizer && make && make test'

- name: "Tokenizer worker: same output as tokenBySha.pl"
run: devenv shell -- bash tests/test_token_worker.sh

- name: Perl tests (prove)
run: devenv shell -- bash -c 'prove tests/t'

- name: Pipeline tests (shell)
run: devenv shell -- bash -c 'tests/tokenize_gate.sh && tests/pipeline_workdir_guard.sh'
run: |
devenv shell -- bash -c 'set -e
for t in tokenize_gate pipeline_workdir_guard ensure_artifacts retokenize_passthrough reblame_passthrough; do
bash "tests/$t.sh"
done'

- name: Dataset tests (pytest)
run: devenv shell -- bash -c 'pytest -q'
run: devenv shell -- pytest -q --junitxml=junit-generate-dataset.xml

# `devenv shell -- pytest` does not flush its summary line; read the JUnit XML.
- name: Re-check the pytest JUnit counts
if: always()
shell: bash
run: |
f=junit-generate-dataset.xml
[ -f "$f" ] || { echo "no $f: pytest did not get far enough to write one"; exit 1; }
python3 - "$f" <<'PY'
import sys, xml.etree.ElementTree as ET
root = ET.parse(sys.argv[1]).getroot()
s = root.find("testsuite") if root.tag == "testsuites" else root
a = s.attrib
print(f"tests={a['tests']} failures={a['failures']} "
f"errors={a['errors']} skipped={a['skipped']}")
if int(a["failures"]) or int(a["errors"]):
sys.exit("failures or errors in the suite")
PY

- name: Upload the pytest JUnit report
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-generate-dataset
path: junit-generate-dataset.xml
if-no-files-found: warn

- name: Build & test legacy onejar modules (sbt test one-jar, Scala 2.10 / JDK 8)
run: |
Expand Down
40 changes: 29 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ HTML views land in the sibling directory `../cregit-files/html`. See
## Preliminaries

- Code is written in Scala, C++, Rust and Perl.
- Platform: Linux x86_64 or macOS arm64 — `devenv` builds the pinned `srcml` parser from source for those platforms (`nix/srcml.nix`). The 1.1.0 release binary segfaults under `--position`, which the pipeline always passes, and the fix is upstream but unreleased, so we pin a revision rather than a release. `srcml --version` still prints `1.1.0`.
- Platform: Linux x86_64 or macOS arm64 — `devenv` builds the pinned `srcml`
1.1.0 parser from its release tag (`nix/srcml.nix`).

## Prerequisites

Expand Down Expand Up @@ -87,7 +88,7 @@ This builds, in dependency order:

| artifact | module | toolchain |
| ------------------------------------------------------------- | ------------------- | ------------------------- |
| `tokenize/srcMLtoken/srcml2token` | C++ transcoder | gcc + xerces-c |
| `tokenize/srcMLtoken/srcml2token` | srcML tokenizer | gcc + xerces-c + libsrcml |
| `tokenize/rustTokenizer` binary | Rust tokenizer | cargo |
| `blobExec/target/scala-2.13/blobExec-0.1.0-assembly.jar` | tokenization driver | sbt, JDK 21 |
| `{slickGitLog,persons,remapCommits}/target/scala-2.10/*-one-jar.jar` | history / persons / remap tools | sbt 0.13, JDK 8 |
Expand All @@ -111,9 +112,12 @@ cd ../tokenize/srcMLtoken && make && make test
cd ../rustTokenizer && make && make test

cd ../..
bash tests/test_token_worker.sh
prove tests/t
pytest -q
tests/tokenize_gate.sh && tests/pipeline_workdir_guard.sh
for t in tokenize_gate pipeline_workdir_guard ensure_artifacts retokenize_passthrough reblame_passthrough; do
bash "tests/$t.sh"
done

for module in slickGitLog persons remapCommits; do
(cd "$module" && sbt --java-home "$LEGACY_JAVA_HOME" -batch test one-jar)
Expand Down Expand Up @@ -162,6 +166,14 @@ Flags (see `./run_pipeline_process.sh --help` for the full list):
| `--memo-dir` | where to memoize tokenized blobs; outside `--work` it survives the full-run wipe | `<work>/memo` |
| `--mode` / `--shards` | tokenizer walk mode / shard count for `sharded` | `pipeline` / `4` |
| `--jobs` | concurrent blame/HTML processes | `CREGIT_JOBS` or up to `4` CPUs |
| `--reblame` | re-blame every file in step 7, replacing existing `.blame` output | off — a resume skips files already blamed |

**`git blame` runs with `-C100` copy detection** (`blameRepo/formatBlame.pl`), so a
token moved between files keeps its original author. It changes up to 25% of a
project's authors, so blame from before 2026-09-22 is not comparable with later blame.

**Pass `--reblame` when the blame itself changed**: without it step 7 keeps every
existing `.blame` file. Do **not** pass it to resume an interrupted run.

A full run starts by **deleting the work directory** — to keep several target
repositories side by side, give each its own `--work`. To resume a failed run
Expand All @@ -174,19 +186,25 @@ returns without invoking `srcml` at all. Put it out of reach with `--memo-dir`,
names neither repository nor extension). The runner refuses a step-1 wipe that
would delete a memo of 10,000 entries or more; `--force-clean` overrides it.

Four blobs in one corpus project are on a **blob denylist**
(`blobExec/src/main/scala/cregit/blobexec/BlobDenylistEntries.scala`): srcML 1.1.0 does
not terminate on them (upstream srcML/srcML#2361, which the revision we now pin closes,
so the entries should be inert). They are never handed to
the tokenizer, are dropped from the rewritten trees rather than kept as raw
source, are counted as `blobsDenylisted` and named with a reason and a citation —
and, unlike a tokenizer timeout, they do **not** change the exit status. Adding an
entry means editing that file and rebuilding the jar.
12 blobs on which srcML 1.1.0 does not terminate (8 Java, srcML/srcML#2361; 4 C) are
on a **blob denylist** (`BlobDenylistEntries.scala`). They never reach the tokenizer,
are dropped from the rewritten trees, and are counted as `blobsDenylisted`.

A blob whose tokenizer times out or reports a parser crash is excluded the same
way without an entry: it is named on an `EXCLUDED failed blob` line, tried once
per run, and the run exits 0. An entry still saves the `--blob-timeout` (600 s by
default) that a known hang costs on each run.

Example run (cregit run on itself):
![Example cregit run](cregit.gif)
p.s.: long pauses are trimmed.

### When a tokenizer is corrected: `--retokenize`

Each run records a tokenizer identity per extension; a changed one with cached rows refuses
the run (exit 3). After `./run_pipeline_process.sh --ensure-artifacts`, resume at step 2 with
`--retokenize rs` to redo only those cached tokens; one that would change nothing exits 7.

### Outputs

Everything lands in the work directory (default: `../cregit-files`, a sibling
Expand Down
6 changes: 3 additions & 3 deletions blameRepo/blameRepoFiles.pl
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ =head1 SYNOPSIS
blameRepoFiles.pl [options] <repository> <outputDirectory> <fileNameRegexp>

Options:
--override overwrite existing files
--overwrite re-blame files whose .blame output already exists
--help brief help message
--man full documentation
--jobs=N maximum concurrent blame processes (default 1)
Expand All @@ -228,9 +228,9 @@ =head1 OPTIONS

Prints the manual page and exits.

=item B<--override>
=item B<--overwrite>

By default, if an output file exists it is skipped. This changes that behaviour.
Re-blame files whose output already exists (by default they are skipped).

=back

Expand Down
4 changes: 2 additions & 2 deletions blobExec/shard_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,9 @@ if [ -z "$THREADS" ]; then
fi

# Default tokenizer dispatcher (srcML for .c/.h, rustTokenizer for .rs, …); the
# --srcml/--ctags paths are forwarded only to the srcML parser.
# --srcml2token/--ctags paths are forwarded only to the srcML parser.
if [ -z "$TOK_CMD" ]; then
TOK_CMD="$CREGIT/tokenize/tokenize.pl --srcml2token=$CREGIT/tokenize/srcMLtoken/srcml2token --srcml=$(command -v srcml) --ctags=$(command -v ctags)"
TOK_CMD="$CREGIT/tokenize/tokenize.pl --srcml2token=$CREGIT/tokenize/srcMLtoken/srcml2token --ctags=$(command -v ctags)"
fi
export BFG_TOKENIZE_CMD="$TOK_CMD"
JAVA="$(command -v java)"
Expand Down
9 changes: 4 additions & 5 deletions blobExec/src/main/scala/cregit/blobexec/BlobDenylist.scala
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
package cregit.blobexec

/** Blobs never handed to the tokenizer, keyed by git blob id: srcML 1.1.0 does not
* terminate on them. Excluded like an oversized blob, and deliberately not gating
* the exit status. The entries live in [[BlobDenylistEntries]]. */
/** Blobs never handed to the tokenizer, keyed by git blob id. Excluded like an
* oversized blob, and deliberately not gating the exit status. */
final class BlobDenylist private (private val bySha: Map[String, BlobDenylist.Entry]) {

def entryFor(sha: String): Option[BlobDenylist.Entry] = bySha.get(sha.toLowerCase)
Expand All @@ -13,8 +12,8 @@ final class BlobDenylist private (private val bySha: Map[String, BlobDenylist.En

object BlobDenylist {

/** Why one blob is excluded. Both fields are required: an exclusion nobody can
* trace back to a defect is indistinguishable from data loss. */
/** Both fields are required: an exclusion nobody can trace back to a defect is
* indistinguishable from data loss. */
final case class Entry(citation: String, reason: String)

/** Where a reader finds the list, quoted in the excluded-blob report. */
Expand Down
31 changes: 20 additions & 11 deletions blobExec/src/main/scala/cregit/blobexec/BlobDenylistEntries.scala
Original file line number Diff line number Diff line change
@@ -1,27 +1,36 @@
package cregit.blobexec

/** The denylisted blobs themselves, keyed by git blob id. Data only: [[BlobDenylist]]
* holds the lookup, and Walker drops these paths from every rewritten tree.
*
* Keyed on content, not path, so one file's history across a move is several
* entries sharing a reason. Each defect class is named once below, which is why the
* shas are grouped rather than each carrying its own prose. */
/** Blobs on which srcML 1.1.0 does not terminate, keyed by git blob id and grouped
* by defect. A tokenizer crash needs no entry: the walk already excludes that blob. */
private[blobexec] object BlobDenylistEntries {

import BlobDenylist.Entry

/** srcML 1.1.0 loops forever on `String @A [] [] s = new String @A [2] [2];`.
* Open upstream against the latest release. */
private val JavaAnnotatedArrays = Entry(
"srcML/srcML#2361",
"srcML 1.1.0 does not terminate on this file's annotated multi-dimensional " +
"array declarations; OpenJDK langtools regression test @bug 8005681"
"srcML 1.1.0 does not terminate on annotated array dimensions, minimal reproducer " +
"`class C { int @A [] a @A []; }`; OpenJDK langtools regression tests " +
"TestNewCastArray (@bug 8005681) and CheckErrorsForSource7"
)

private val CppInCHeader = Entry(
"srcml-nontermination-analysis.md#class-2-the-c-parser-on-c-inside-a-h",
"srcML 1.1.0's C parser does not terminate on C++ in a .h file (no upstream issue), " +
"minimal reproducer `namespace n{struct T&f(const struct S);}`; eden/fs/utils/StatTimes.h"
)

val NonTerminating: Map[String, Entry] = Map(
"2ee2673ad0a8ff2cef0254e7bfdc488cc1d61a65" -> JavaAnnotatedArrays,
"303c1a109f1fdc4b9d84e15111c56931202844ac" -> JavaAnnotatedArrays,
"504aaf9109fc01d6d62d99ac529ded99a8dfa8ca" -> JavaAnnotatedArrays,
"c91924d1e87cb82d372a73b24579ccc34609efaa" -> JavaAnnotatedArrays
"c91924d1e87cb82d372a73b24579ccc34609efaa" -> JavaAnnotatedArrays,
"56c8adf588e697a3e42b600b2e5ec54710592c61" -> JavaAnnotatedArrays,
"5e9c9138ed9c43c20591d1814cb72ad014d1a9e5" -> JavaAnnotatedArrays,
"6bf666550f6cba72f1602ce1b7a6b0c9025fc8c7" -> JavaAnnotatedArrays,
"afbd81e5789ba060147b0de3be4f01759963299d" -> JavaAnnotatedArrays,
"5460d43f1884e3be230e02215151b9b963844a48" -> CppInCHeader,
"720914b27e170b73b38ef662fb04239273dd0a34" -> CppInCHeader,
"99c191e9a208547f5c9b1224c290f106a640de59" -> CppInCHeader,
"dbb885e0bf046755c4fcfbddfc2454f90c550c62" -> CppInCHeader
)
}
84 changes: 43 additions & 41 deletions blobExec/src/main/scala/cregit/blobexec/BlobExec.scala
Original file line number Diff line number Diff line change
Expand Up @@ -5,37 +5,28 @@ import org.eclipse.jgit.lib.{ObjectId, ObjectInserter}

import java.util.{Arrays => JavaArrays}

/**
* Run the per-blob command and map its result onto a git object.
*
* The command reads the original blob on stdin and writes the replacement on
* stdout. `BFG_FILENAME` is the basename rather than the path because cregit's
* `tokenBySha.pl` keys its on-disk memo by it.
*
* - env `BFG_BLOB` / `BFG_FILENAME` / `BFG_PATH`
* - exit != 0 → `Skip` (or `Abort` if `abortOnError`)
* - killed → `Skip`, never `Abort`, never `Replace`
* - stdout == stdin → `Skip`
* - otherwise → `Replace(newBlob)`
*/
/** Runs a per-blob command: original bytes on stdin, replacement on stdout, env
* `BFG_BLOB`, `BFG_PATH` and `BFG_FILENAME` (the basename, which `tokenBySha.pl`
* keys its memo by). */
object BlobExec {

/** Per-blob wall-clock budget for the external command, in seconds. */
val DefaultTimeoutSeconds: Int = 600

/** Must match `$PARSER_CRASH_EXIT` in `tokenize/tokenizeSrcMl.pl`. */
val ParserCrashExitCode: Int = 33

/** The JVM reports a death by signal N as exit status 128 + N. */
private val SignalDeathStatus: Int = 128

sealed trait Outcome
object Outcome {
case object Skip extends Outcome
final case class Replace(newBlob: ObjectId) extends Outcome
case object Skip extends Outcome
final case class Replace(newBlob: ObjectId) extends Outcome
final case class Abort(stderr: String, exitCode: Int) extends Outcome
}

/**
* Run `command` against `bytes`. Pure aside from the child process and the
* jgit inserter, which must be confined to the calling thread.
*/
/** Thread-safe if `inserter` is confined to the calling thread. */
def run(
bytes: Array[Byte],
origSha: String,
Expand All @@ -46,34 +37,59 @@ object BlobExec {
inserter: ObjectInserter,
timeoutSeconds: Int = DefaultTimeoutSeconds,
onTimeout: () => Unit = () => (),
onParserCrash: () => Unit = () => ()
onParserCrash: () => Unit = () => (),
workerPool: Option[TokenizerWorkerPool] = None
): Outcome = {
val env = Seq("BFG_BLOB" -> origSha, "BFG_FILENAME" -> filename, "BFG_PATH" -> fullPath)
val ran = workerPool match {
case Some(pool) => pool.invoke(bytes, filename, timeoutSeconds)
case None => new ChildRunner(timeoutSeconds).run(command, bytes, env)
}

new ChildRunner(timeoutSeconds).run(command, bytes, env) match {
ran match {
case ChildRunner.Outcome.Killed(why) =>
// Not routed through `abortOnError`: one wedged blob must not abort a long run.
System.err.println(
s"Warning: command [$command] on blob $origSha at path [$fullPath] gave no usable " +
s"result ($why): blob left untokenized"
s"Warning: command [$command] timed out on blob $origSha at path [$fullPath] ($why): " +
"blob excluded"
)
onTimeout()
Outcome.Skip

case ChildRunner.Outcome.Exited(ParserCrashExitCode, _, stderr) =>
reportParserCrash(command, origSha, fullPath, s"reported a parser crash (exit $ParserCrashExitCode)", stderr)
// Skip like a timeout, but counted apart: more time never fixes a crash.
System.err.println(
s"Warning: command [$command] reported a parser crash (exit $ParserCrashExitCode) on blob " +
s"$origSha at path [$fullPath]: srcML died or produced no tokens, so this blob is excluded " +
"rather than written as an empty tokenization"
)
printStderr(command, origSha, fullPath, stderr)
onParserCrash()
Outcome.Skip

case ChildRunner.Outcome.Exited(0, stdout, stderr) if stdout.isEmpty && bytes.nonEmpty =>
reportParserCrash(command, origSha, fullPath,
s"reported a parser crash: exited 0 with no output for a ${bytes.length}-byte blob", stderr)
case ChildRunner.Outcome.Exited(status, _, stderr) if status > SignalDeathStatus =>
// A kill from outside (e.g. the OOM killer) must not keep the source as tokens.
System.err.println(
s"Warning: command [$command] was killed by signal ${status - SignalDeathStatus} on blob " +
s"$origSha at path [$fullPath]: blob excluded"
)
printStderr(command, origSha, fullPath, stderr)
onParserCrash()
Outcome.Skip

case ChildRunner.Outcome.Exited(status, _, stderr) if status != 0 =>
logError(command, origSha, fullPath, status, stderr)
if (abortOnError) Outcome.Abort(stderr, status) else Outcome.Skip

case ChildRunner.Outcome.Exited(_, stdout, _) if stdout.isEmpty && bytes.nonEmpty =>
System.err.println(
s"Warning: command [$command] exited 0 but produced no output for the ${bytes.length}-byte " +
s"blob $origSha at path [$fullPath]. Refusing to write an empty tokenization: counting " +
"this as a parser crash and excluding the blob."
)
onParserCrash()
Outcome.Skip

case ChildRunner.Outcome.Exited(_, stdout, _) if JavaArrays.equals(bytes, stdout) =>
Outcome.Skip

Expand All @@ -82,20 +98,6 @@ object BlobExec {
}
}

private def reportParserCrash(
command: String,
origSha: String,
fullPath: String,
what: String,
stderr: String
): Unit = {
System.err.println(
s"Warning: command [$command] $what on blob $origSha at path [$fullPath]: " +
"blob left untokenized rather than written as an empty tokenization"
)
printStderr(command, origSha, fullPath, stderr)
}

private def logError(
command: String,
origSha: String,
Expand Down
Loading
Loading