LogNexus is the open-source implementation for the accepted ISSTA 2026 paper “LogNexus: Effective Log Compression via Unified Redundancy Encoding.” It targets raw software logs whose repeated structures and variable values are correlated. Instead of parsing logs first and compressing extracted variables later, LogNexus mines recurring structure-variable patterns and encodes them in one compression pipeline before applying backend archive compression.
This repository contains the source code, a reduced public smoke-test dataset, reproduction scripts, configuration documentation, claim-boundary documentation, and reference result snapshots. Its initial public release is based on the exact Zenodo v2 source artifact archived at 10.5281/zenodo.21281836. The GitHub repository may continue to evolve; use the Zenodo v2 DOI when you need the evaluated version.
| Item | Details |
|---|---|
| Paper | LogNexus: Effective Log Compression via Unified Redundancy Encoding, ISSTA 2026 |
| Badges requested | Functional, Reusable, Available |
| Method | Unified Redundancy Encoding over stable tokens and recurring structure-variable patterns |
| Implementation | C++20 compressor and decompressor, with Docker and native build paths |
| Artifact DOI | 10.5281/zenodo.21281836 |
| Platform | Packaged and tested for linux/amd64; no GPU required |
| Included data | A small public LogHub Linux log for offline smoke tests |
| Main restoration claim | Equality of whitespace-separated tokens in source order |
The artifact includes the algorithmic source used for the paper experiments, including token-preserving reconstruction, Stage 1 and Stage 3 ablations, the paper's dataset-specific evaluation thresholds, a fixed default threshold for untuned use, stable archive metadata, and optional Stage-2 case-study counters.
The source package is intended for inspection, rebuilding, and reuse. The image package is intended for the fastest getting-started path: load the image, run the offline smoke test, and then optionally run the reduced evaluation. The full LogHub evaluation scripts are included, but the complete 16-dataset LogHub corpus and third-party baseline artifacts are not bundled because of size and provenance constraints.
LogNexus builds a Unified Redundancy Tree to encode stable tokens and frequent structure-variable patterns together. Stage 1 extracts stable structure, Stage 2 encodes recurring variable subtrees, Stage 3 handles residual data, and the resulting streams are packaged with LZMA-based archive compression. The artifact also includes the paper's Stage 1 plus LZMA and Stage 1 plus Stage 3 plus LZMA ablations.
The accepted paper evaluates LogNexus on the 16-dataset LogHub benchmark. The table below lists the paper-aligned aggregate results that are tracked by this artifact. Exact threshold profiles and reproduction rules are documented in artifact/CONFIGURATION.md; the compact source of these aggregate values is artifact/reference/paper_aggregate_results.csv.
| Paper-aligned experiment | Aggregate result | Artifact support |
|---|---|---|
| RQ1: overall compression | LogNexus achieves the highest compression ratio on 14 of 16 datasets, with an average compression ratio of 88.202. | Full LogNexus runs are reproducible with scripts/run_all_datasets.sh after mounting the complete LogHub data. The reduced Linux run checks the same code path. |
| RQ1: compression speed | LogNexus reaches 26.921 MiB/s on average in the default configuration and 36.237 MiB/s with internal parallelism. | The compressor reports elapsed time, input bytes, archive bytes, throughput, compression ratio, and chunk count for every run. |
| RQ2: input granularity | Single-archive LogNexus reaches an average compression ratio of 339.691 and an average compression speed of 14.578 MiB/s. | scripts/run_all_datasets.sh runs the single-archive configuration; make reduced checks this path on the included Linux log. |
| RQ3: stage ablation | LogNexus improves over the direct LZMA baseline as each stage is added. The aggregate values are shown below. | The artifact includes the Stage 1 + LZMA and Stage 1 + Stage 3 + LZMA ablation modes. The direct LZMA row is documented as a baseline measurement in artifact/CLAIMS.md. |
| Threshold sensitivity | The fixed-threshold study reports an average compression ratio of 87.680 at the recommended untuned default tau = 0.02. |
scripts/run_parameter_sensitivity.sh threshold runs the eight fixed-threshold settings on the full LogHub layout; make reduced checks all eight settings on Linux. |
| Restoration audit | LogNexus decompression restores the token stream on all 16 datasets under the paper's restoration criterion. | Restorable configurations are checked with scripts/verify_token_roundtrip.py; reduced and full launchers record token equality in their CSV outputs. |
| RQ3 configuration | Average CR | Average speed |
|---|---|---|
| LZMA only | 31.779 | 17.045 MiB/s |
| Stage 1 + LZMA | 57.587 | 22.454 MiB/s |
| Stage 1 + Stage 3 + LZMA | 81.319 | 24.170 MiB/s |
| Full LogNexus | 88.202 | 36.237 MiB/s |
The reduced evaluation included in this artifact checks the same LogNexus code paths on the public Linux log. Reproducing the full cross-tool aggregate values requires the complete LogHub data and the LogZip, LogReducer, LogShrink, and Denum baseline artifacts, which are not bundled into the reduced image.
The restoration criterion in the paper is equality of the whitespace-separated token stream in source order. The current archive format may normalize whitespace, including omission of blank or whitespace-only lines. Therefore, this artifact does not claim byte-for-byte or physical-line identity. The verifier reports token counts, token-stream SHA-256 values, source/restored line counts, and byte equality so that this boundary is explicit. For large files whose byte counts already differ, the verifier may skip raw-byte SHA-256 computation and report not_computed; the token-stream SHA-256 check is still computed in streaming mode.
The fastest check is the Docker smoke test. Download the prebuilt linux/amd64 Docker/OCI image from the Zenodo v2 record, place it in the repository root, and run the commands below without network access. Loading the image and running the smoke test should take less than 30 minutes.
docker load -i lognexus-issta26-ae.lognexus-image.tar.gz
docker run --rm --network none \
lognexus-issta26-ae:latest ./scripts/smoke_test.shPodman can be used with the corresponding podman load and podman run commands.
Expected final lines include:
PASS: ... ordered tokens match across 1 restored chunk(s).
Byte comparison: different because the current format normalizes whitespace and may omit blank lines; this is outside the paper's token-level criterion.
SMOKE TEST PASSED
The exact token count and timing are printed by the script. Timing can vary across hosts and should not be compared directly with the paper unless the hardware and configuration are comparable.
Users who need to rebuild the image can run:
docker build --platform linux/amd64 -t lognexus-issta26-ae:latest .
docker run --rm --network none \
lognexus-issta26-ae:latest ./scripts/smoke_test.shThe image build downloads Ubuntu packages. The smoke test and experiment runs do not access the internet.
On Ubuntu 22.04:
sudo apt-get install build-essential libpcre2-dev libarchive-dev python3 time coreutils grep xz-utils
make -j"$(nproc)"
make smokeThe container is the primary evaluation environment. See REQUIREMENTS for the complete environment and hardware requirements.
The reduced evaluation exercises the main paper configurations on the included 2.3 MB Linux log:
./scripts/run_reduced_evaluation.shIt runs these configurations:
| Configuration | Paper connection |
|---|---|
rq1_lognexus_paper_tau |
Default LogNexus with the Linux threshold used for the paper table |
rq1_lognexus_p_paper_tau |
LogNexus-P with the Linux threshold used for the paper table |
rq2_single_archive_paper_tau |
Global single-archive mode with the Linux paper threshold |
rq3_s1_s3_lzma |
Stage 1 + Stage 3 + LZMA |
rq3_s1_lzma |
Stage 1 + LZMA |
tau_0_002, tau_0_005, tau_0_01, tau_0_02_fixed_default, tau_0_03, tau_0_05, tau_0_10, tau_0_15 |
All eight fixed-threshold sensitivity settings from the paper |
The script prints progress for every configuration and writes:
artifact-output/reduced/results.csv
artifact-output/reduced/runs/<configuration>/compress.log
artifact-output/reduced/runs/<configuration>/decompress.log
artifact-output/reduced/runs/<configuration>/verification.json
The result CSV includes compression ratio, compression and decompression throughput, archive and restored sizes, CPU utilization, average CPU cores, peak RSS, and restoration status. The complete configurations and S1+S3+LZMA are decompressed and checked against the paper's token-level restoration criterion. S1+LZMA is a compression-only ablation and does not produce a complete restorable format, so its decompression and verification columns are reported as not_available or not_checked. The reduced results demonstrate that the code paths are exercisable. They are not expected to equal the 16-dataset averages in the paper.
scripts/verify_token_roundtrip.py computes a canonical token stream with standard GNU text utilities and hashes that stream incrementally. Set LOGNEXUS_VERIFY_CACHE_DIR=<dir> to cache original-file token statistics across repeated checks on the same dataset; the full-dataset launcher sets this automatically.
Exact timing is host-dependent. Internally parallel configurations can also show small archive-size differences because concurrent processing can change stream ordering before backend compression. This does not change the restored token sequence. artifact/reference/reduced_linux_variability_20260611.csv records the observed range from seven native validation runs; this range is evidence of variability, not an acceptance bound.
If a virtualized host adjusts its clock during a run and GNU time reports a negative duration, the script records not_available for wall_seconds instead of retaining an invalid measurement. The compressor's own monotonic-clock throughput remains available, but timing comparisons with the paper should use comparable native hardware.
Build the binaries and compress a log with 100K-line chunks:
make -j"$(nproc)"
LOG_FILE=/data/Logs/HDFS/HDFS.log
./build/LogNexus_compress \
"$LOG_FILE" HDFS 100000 4 1 1 0.02The output directory is derived from the input filename:
output_HDFS_lognexus/
compressed_0.tar.xz
compressed_1.tar.xz
...
Decompress and verify:
./build/LogNexus_decompress \
output_HDFS_lognexus restored_HDFS HDFS
python3 scripts/verify_token_roundtrip.py \
"$LOG_FILE" restored_HDFSThe paper uses 100K-line chunks and four chunk workers. Default LogNexus keeps each chunk internally single-threaded; LogNexus-P uses four internal workers in Stage 1 and Stage 2. The final paper tables use dataset-specific thresholds selected during the sensitivity study. These values are recorded in configs/paper_thresholds.csv. The source-code default remains 0.02 for untuned use.
DATASET=HDFS
LOG_FILE=/data/Logs/HDFS/HDFS.log
TAU=$(awk -F, -v dataset="$DATASET" \
'NR > 1 && $1 == dataset { print $2 }' configs/paper_thresholds.csv)
# LogNexus
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 1 1 "$TAU"
# LogNexus-P
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 4 4 "$TAU"The compressor prints end-to-end elapsed time, original bytes, archive bytes, compression speed, compression ratio, and chunk count. RQ1 baseline comparisons additionally require the official LogZip, LogReducer, LogShrink, and Denum artifacts and their paper configurations.
DATASET=HDFS
LOG_FILE=/data/Logs/HDFS/HDFS.log
TAU=$(awk -F, -v dataset="$DATASET" \
'NR > 1 && $1 == dataset { print $2 }' configs/paper_thresholds.csv)
# 100K-line chunked mode
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 1 1 "$TAU"
# Single global archive
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 4 4 "$TAU" --single-archiveSingle-archive mode can use substantially more memory on very large datasets. The paper reports both performance and peak memory for this tradeoff.
DATASET=HDFS
LOG_FILE=/data/Logs/HDFS/HDFS.log
TAU=$(awk -F, -v dataset="$DATASET" \
'NR > 1 && $1 == dataset { print $2 }' configs/paper_thresholds.csv)
# Full: Stage 1 + Stage 2 + Stage 3 + LZMA
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 4 4 "$TAU"
# Stage 1 + Stage 3 + LZMA
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 4 4 0.02 --skip-stage2
# Stage 1 + LZMA, compression-only ablation
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 4 4 0.02 --s1-lzmaThe LZMA-only values in the paper are obtained by applying the system LZMA/XZ compressor directly to each input dataset under the same measurement environment.
Pass the threshold ratio as the final numeric argument in chunked mode:
LOG_FILE=/data/Logs/HDFS/HDFS.log
DATASET=HDFS
for tau in 0.002 0.005 0.01 0.02 0.03 0.05 0.10 0.15; do
./build/LogNexus_compress "$LOG_FILE" "$DATASET" 100000 4 1 1 "$tau"
doneRun each threshold in a separate working directory because output directory names do not include the threshold. The accepted paper identifies 0.02 as the recommended fixed threshold because it provides the highest average compression ratio among the tested fixed thresholds while retaining similar throughput to nearby settings. The main paper tables use the dataset-specific values in configs/paper_thresholds.csv; the fixed-threshold sensitivity results are a separate evaluation profile.
The release build can emit the per-chunk Stage-1, Stage-2, and Stage-3 variable counters used by the OpenSSH and HDFS case study. The counters are disabled in ordinary compression runs and do not enter the compressed archives. Run the following with the complete LogHub layout:
./scripts/run_stage2_case_study.sh /data/Logs artifact-output/stage2-case-studyThe launcher sets LOGNEXUS_CASE_STUDY_COUNTERS=1, runs S1+S3+LZMA and Full with the dataset-specific paper threshold, checks the accounting invariant that Stage 2 plus Stage 3 equals Stage 1, and writes artifact-output/stage2-case-study/counter_summary.csv. For a quick check of the instrumented code path using the bundled log, run:
DATASETS=Linux ./scripts/run_stage2_case_study.sh Logs artifact-output/stage2-case-study-linuxThe OpenSSH and HDFS paper values are retained in artifact/reference/stage2_case_study_20260518.csv with their original setup. These are reference observations; a rerun on the current code should be checked by the counter accounting and reported percentages, rather than by timing or archive bytes.
The full evaluation uses the 16 public LogHub datasets. The datasets can be obtained from the public LogHub repository at https://github.com/logpai/loghub, whose README provides download links for each raw log in the Download column. The artifact does not download these datasets automatically because the full set is large and the original dataset terms should remain visible to users.
Download the raw logs, decompress them if needed, and arrange them with the following paths before running the full scripts:
Logs/
Android/Android.log
Apache/Apache.log
BGL/BGL.log
Hadoop/Hadoop.log
HDFS/HDFS.log
HealthApp/HealthApp.log
HPC/HPC.log
Linux/Linux.log
Mac/Mac.log
OpenSSH/OpenSSH.log
OpenStack/OpenStack.log
Proxifier/Proxifier.log
Spark/Spark.log
Thunderbird/Thunderbird.log
Windows/Windows.log
Zookeeper/Zookeeper.log
Use the original LogHub dataset variants listed in the paper setup. In particular, use HDFS_v1 for HDFS, Android_v1 for Android, and the raw logs named by the LogHub table for the other datasets. After placement, the top-level path such as /data/Logs should contain one subdirectory per dataset, and each subdirectory should contain the corresponding <Dataset>.log file shown above.
Only Logs/Linux/Linux.log is included to keep the image small and permit an offline smoke test. Obtain the remaining data from the public LogHub distribution and retain its original terms and provenance. No experiment script downloads data during execution.
The bundled file's source, checksum, use restriction, and handling note are recorded in artifact/DATASET_PROVENANCE.md. The LogHub license notice required for redistributed dataset copies is included in Logs/LICENSE.loghub.
After arranging all datasets, run the complete artifact matrix with progress output:
./scripts/run_all_datasets.sh /data/Logs artifact-output/fullThis applies the dataset-specific paper threshold to RQ1 and RQ2, runs the two compression ablations, checks three representative fixed thresholds, verifies every restorable output, and writes artifact-output/full/combined_results.csv. It records token restoration, decompression throughput, CPU utilization, average CPU cores, and peak RSS for each restorable LogNexus configuration. The eight-value threshold sweep in Section 6 can be used when reproducing the full sensitivity figure.
If the full matrix is interrupted, rerun it with RESUME=1 and the same output directory:
RESUME=1 ./scripts/run_all_datasets.sh /data/Logs artifact-output/fullAfter mounting the full LogHub data, reproduce the eight fixed-threshold settings from the paper with:
./scripts/run_parameter_sensitivity.sh \
threshold /data/Logs artifact-output/threshold-sensitivityThis runs all 16 datasets at 0.002, 0.005, 0.01, 0.02, 0.03, 0.05, 0.10, and 0.15. Every run uses one shared threshold within that run.
Reproduce the chunk-size table on BGL, HDFS, and Spark with:
./scripts/run_parameter_sensitivity.sh \
chunk /data/Logs artifact-output/chunk-sensitivityThis uses 100K, 300K, 1M, and 3M lines per chunk, four chunk workers, one internal Stage 1 worker, one internal Stage 2 worker, and the dataset-specific thresholds in configs/paper_thresholds.csv. Both studies print per-run progress and write a CSV plus individual logs. For a quick script check with the bundled dataset, prefix either command with DATASETS=Linux and use Logs as the data root; that quick check is not part of the paper table.
artifact/CLAIMS.md maps paper claims to reproduction paths and identifies claims that are not covered by the reduced image. In particular, the reduced image does not rerun the four baselines or represent a substitute for the full 77 GB evaluation. It does exercise LogNexus's main modes and verification logic without network access.
artifact/CONFIGURATION.md explains the dataset-specific paper-table profile and the fixed 0.02 recommendation. Both follow the methodology: one shared tau is used by the filtering operations within a run, while its numerical value may differ across dataset runs. Reviewers should use the dataset-specific profile for the main RQ tables and the fixed profile for the threshold-sensitivity result.
Chunked mode:
LogNexus_compress <log_file> <dataset> <block_size> <chunk_threads>
[stage1_threads] [stage2_threads] [threshold] [flags]
Single-archive mode:
LogNexus_compress <log_file> <dataset> <stage1_threads>
[stage2_threads] [threshold] --single-archive
Flags:
| Flag | Meaning |
|---|---|
--single-archive |
Build one global archive instead of independent chunks |
--skip-stage2 |
Run the S1+S3+LZMA ablation |
--s1-lzma |
Run the S1+LZMA ablation |
Decompression:
LogNexus_decompress <archive_directory> <restored_directory> <dataset>
Supported dataset names are the 16 names shown in the data layout. Dataset-specific names select the regular expressions used to preserve known log field formats.
.
|-- .dockerignore
|-- .gitignore
|-- Dockerfile
|-- Makefile
|-- LogNexus_compress.cpp
|-- LogNexus_decompress.cpp
|-- README.md
|-- REQUIREMENTS
|-- STATUS
|-- LICENSE
|-- NOTICE
|-- CITATION.cff
|-- CONTRIBUTING.md
|-- SECURITY.md
|-- Logs/
| |-- LICENSE.loghub
| `-- Linux/
| `-- Linux.log
|-- artifact/
| |-- CLAIMS.md
| |-- CONFIGURATION.md
| |-- DATASET_PROVENANCE.md
| |-- MANIFEST.sha256
| `-- reference/
| |-- README.md
| |-- paper_aggregate_results.csv
| |-- reduced_linux_20260611.csv
| |-- reduced_linux_20260624.csv
| |-- reduced_linux_variability_20260611.csv
| `-- stage2_case_study_20260518.csv
|-- configs/
| `-- paper_thresholds.csv
|-- figures/
| `-- overview_LogNexus.png
`-- scripts/
|-- check_reference_results.py
|-- generate_manifest.sh
|-- package_container.sh
|-- package_source.sh
|-- run_all_datasets.sh
|-- run_parameter_sensitivity.sh
|-- run_reduced_evaluation.sh
|-- run_stage2_case_study.sh
|-- smoke_test.sh
|-- summarize_stage2_counters.py
`-- verify_token_roundtrip.py
Use CITATION.cff for citation metadata. When referring to the evaluated implementation, cite the accepted ISSTA 2026 paper together with the Zenodo v2 artifact DOI, 10.5281/zenodo.21281836. The author-created LogNexus source code and documentation are released under the Apache License 2.0 in LICENSE; NOTICE and Logs/LICENSE.loghub record the separate terms for bundled public benchmark data and third-party materials.
On a machine with Docker:
./scripts/package_container.shThis builds the linux/amd64 image, runs the smoke test, saves a compressed image archive, and writes its SHA-256 checksum. The evaluated artifact is archived on Zenodo at 10.5281/zenodo.21281836 (https://doi.org/10.5281/zenodo.21281836). If this artifact is updated through Zenodo versioning, use the latest version linked from the Zenodo concept DOI.
The source-only submission archive can be generated independently with:
make source-packageThis command validates the paper aggregate snapshot and threshold profiles, regenerates the file manifest, verifies the staged files, and creates a deterministic archive under dist/.