From c66592d94b1bcc77c82f4d2cbced577b7c47916a Mon Sep 17 00:00:00 2001 From: Jad Yehya Date: Tue, 26 May 2026 11:12:06 +0200 Subject: [PATCH] DOC update README to enhance clarity and structure, adding installation instructions, benchmark interface details, and model descriptions --- README.md | 279 ++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 229 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index 2a88df8..af6d479 100644 --- a/README.md +++ b/README.md @@ -1,85 +1,264 @@ # Benchmark repository for Time Series Anomaly Detection (TSAD) algorithms ![Build Status](https://github.com/benchopt/benchmark_tsad/workflows/Tests/badge.svg) +[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/) -## About Benchopt +This repository is a Benchopt benchmark for unsupervised time series anomaly +detection. It compares classical detectors, reconstruction models, forecasting +models, matrix-profile methods, dictionary-learning methods, and foundation +model baselines. -Benchopt is a package to simplify and make more transparent and reproducible the comparisons of optimization algorithms. +## Quick Start +Install Benchopt and clone the benchmark: -## Objective - -This benchmark evaluates and compares Time Series Anomaly Detection (TSAD) algorithms. The goal is to: +```bash +pip install -U benchopt +git clone https://github.com/Jad-yehya/benchmark_tsad +cd benchmark_tsad +``` -- Provide a standardized framework for comparing different TSAD approaches -- Enable reproducible evaluation of anomaly detection performance +Run a small local experiment: -## Install +```bash +benchopt run . -s RoseCDL -d Simulated +``` -This benchmark can be run using the following command: +Run a specific method on a benchmark dataset: ```bash -$ pip install -U benchopt -$ git clone https://github.com/benchopt/benchmark_tsad -$ benchopt run benchmark_tsad +benchopt run . -s MP -d SMAP ``` -Options for running the benchmark can be passed as command line arguments. -For example, to run a specific solver and a specific dataset, use the following command: + +Show all Benchopt options: ```bash -$ benchopt run benchmark_tsad -s IsolationForest -d SMAP +benchopt run . --help +``` + +## Benchmark Interface + +Datasets return: + +```python +dict( + X_train=X_train, # shape: (n_recordings, n_features, n_samples) + X_test=X_test, # shape: (n_recordings, n_features, n_samples) + y_test=y_test, # shape: (n_recordings, n_samples) +) +``` + +Solvers receive only `X_train` and `X_test`. They return continuous anomaly +scores by default: + +```python +return dict(anomaly_scores=anomaly_scores) +``` + +Larger scores must mean "more anomalous". + +Solvers may optionally return binary predictions: + +```python +return dict( + anomaly_scores=anomaly_scores, + anomaly_predictions=anomaly_predictions, +) ``` -Use ``benchopt run -h`` for more details about these options, or visit https://benchopt.github.io/api.html. +Prediction generation is solver-owned. Solvers that support prediction metrics +expose: + +- `cutoff`: fraction of the highest test scores to label as anomalous. -## Available Solvers +For example, `cutoff=0.05` means the top 5% highest anomaly scores are returned +as `anomaly_predictions = 1`. If `cutoff` is not set, the solver returns only +scores. -- AR (Autoregressive Linear Model) -- ABOD (Angle-Based Outlier Detection) -- CBLOF (Cluster-Based Local Outlier Factor) -- DIF (Deep Isolation Forest) -- Isolation Forest -- LOF (Local Outlier Factor) -- LSTM (Long Short-Term Memory) -- OCSVM (One-Class SVM) -- VAE (Variational Autoencoder) -- Transformer +## Models + +| Solver name | Method family | Main idea | Notes | +| --- | --- | --- | --- | +| `ABOD` | Classical outlier detection | Angle-Based Outlier Detection on windowed features | PyOD; legacy solver | +| `AE` | Neural reconstruction | Feed-forward autoencoder on sliding windows | PyTorch | +| `AnomalyBERT` | Transformer | Anomaly Transformer-style masked replacement objective | Requires local AnomalyBERT checkout | +| `AR` | Forecasting | Autoregressive linear model with reconstruction error | PyTorch | +| `CBLOF` | Classical outlier detection | Cluster-Based Local Outlier Factor on windowed features | PyOD; legacy solver | +| `DAGMM` | Deep density model | Deep Autoencoding Gaussian Mixture Model | Merlion | +| `DIF` | Deep ensemble outlier detection | Deep Isolation Forest | PyOD; GPU recommended | +| `IsolationForest` | Classical outlier detection | Isolation Forest on raw/windowed features | scikit-learn; legacy solver | +| `LocalOutlierFactor` | Classical outlier detection | Local density deviation with novelty mode | scikit-learn; legacy solver | +| `LSTM` | Sequence reconstruction | LSTM encoder-decoder reconstruction error | PyTorch | +| `MP` | Matrix profile | Distance profile over subsequences | TSB-AD; univariate only | +| `OCSVM` | Classical outlier detection | One-Class SVM on windowed features | scikit-learn; legacy solver | +| `RoseCDL` | Dictionary learning | Convolutional dictionary learning reconstruction error | RoseCDL | +| `Transformer` | Sequence reconstruction | Vanilla transformer reconstruction error | PyTorch | +| `TSB-Chronos` | Foundation model / forecasting | Chronos wrapper from TSB-AD | optional backend | +| `TSB-TimesFM` | Foundation model / forecasting | TimesFM wrapper from TSB-AD | optional backend | +| `TSB-TimesNet` | Deep forecasting | TimesNet wrapper from TSB-AD | optional backend | +| `VAE` | Neural reconstruction | Variational Autoencoder on windows | PyOD | + +Some solvers depend on optional or heavy packages. Benchopt installs solver +dependencies in isolated environments when possible. The validation suite marks +optional backends and unavailable local checkouts as expected skips or xfails. ## Datasets -- Soil Moisture Active Passive (SMAP) -- Mars Science Laboratory (MSL) -- Pooled Server Metric (PSM) -- Secure Water Treatment (SWaT) -- Water Distribution (WADI.A2_19) -- Simulated dataset +| Dataset name | Type | +| --- | --- | +| `DAPHNET` | Real sensor/health time series | +| `DODGERS` | Real event-count time series | +| `ECG` | Real ECG time series | +| `GENESIS` | Real anomaly-detection benchmark data | +| `GHL` | Real sensor benchmark data | +| `IOPS` | Real server metric time series | +| `KDD21` | Real anomaly-detection benchmark data | +| `MGAB` | Real anomaly-detection benchmark data | +| `MITDB` | Real ECG time series | +| `MSL` | NASA Mars Science Laboratory telemetry | +| `NAB` | Numenta Anomaly Benchmark data | +| `OCCUPANCY` | Real occupancy/environmental time series | +| `OPPORTUNITY` | Real sensor/activity time series | +| `Pattern` | Synthetic convolutional-pattern anomalies | +| `PSM` | Pooled Server Metrics | +| `SENSORSCOPE` | Real environmental sensor time series | +| `Simulated` | Synthetic regression time series with injected anomalies | +| `SMAP` | NASA Soil Moisture Active Passive telemetry | +| `SMD` | Server Machine Dataset | +| `SVDB` | Real ECG time series | +| `SWaT` | Secure Water Treatment industrial process | +| `Trend` | Synthetic pattern anomalies with low-frequency trend | +| `WADI` | Water Distribution industrial process | +| `YAHOO` | Yahoo anomaly benchmark data | + +`SMAP`, `MSL`, and `PSM` are fetched automatically when missing. `Simulated`, +`Pattern`, and `Trend` are generated at runtime. -The SMAP, MSL and PSM datasets are automatically fetched when running the benchmark. -The simulated dataset is generated at running time. -However, the automatic use of the SWaT and WADI datasets is not possible. In order to use them, you must request access to the owners at the following link : -https://itrust.sutd.edu.sg/itrust-labs_datasets/dataset_info/. +Most remaining loaders expect files under Benchopt's data directory, available +through `benchopt.config.get_data_path()`. + +`SWaT` and `WADI` +cannot be redistributed; request access from the dataset owners: +https://itrust.sutd.edu.sg/itrust-labs_datasets/dataset_info/. ## Metrics -The benchmark uses three categories of metrics to evaluate the performance of unsupervised time series anomaly detection methods: +The objective separates score-based metrics from prediction-based metrics. + +### Score Metrics + +These are computed from `anomaly_scores` and are enabled by default. + +| Metric | Key | Direction | +| --- | --- | --- | +| Area under the precision-recall curve | `auc_pr` | higher is better | +| Area under the ROC curve | `auc_roc` | higher is better | + +Default objective configuration: + +```python +score_metrics = ("auc_pr", "auc_roc") +prediction_metrics = None +``` -1. **Classical Binary Classification Metrics.** These traditional metrics treat anomaly detection as a point-wise binary classification task. - - Precision: The ratio of correctly identified anomalies to all predicted anomalies - - Recall: The ratio of correctly identified anomalies to all actual anomalies - - F1-Score: The harmonic mean of precision and recall +### Prediction Metrics -2. **Time-Forgiving Metrics.** These metrics introduce temporal flexibility by allowing predictions within a certain time window of the actual anomaly to be considered correct. - - Time-Forgiving Precision (soft_precision_s): The ratio of predictions that fall within the acceptable time window of true anomalies of size *s*. - - Time-Forgiving Recall (soft_recall_s): The ratio of true anomalies that have at least one prediction within a certain time window of size *s*. - - Time-Forgiving F1-Score (soft_f1_s): The harmonic mean of the time-forgiving precision and recall. +These are computed from `anomaly_predictions`. They are opt-in and require a +solver `cutoff` or a solver that otherwise returns predictions. - - Range_metrics : Another family of precision, recall and f1-score tailored for time series anomaly detection. (Noted precision_t, recall_t, f1_t). +| Metric | Key | Direction | +| --- | --- | --- | +| Point-wise precision | `precision` | higher is better | +| Point-wise recall | `recall` | higher is better | +| Point-wise F1 | `f1` | higher is better | +| Zero-one loss | `zoloss` | lower is better | +| Range precision | `precision_t` | higher is better | +| Range recall | `recall_t` | higher is better | +| Range F1 | `f1_t` | higher is better | +| Candidate-to-target temporal distance | `ctt` | lower is better | +| Target-to-candidate temporal distance | `ttc` | lower is better | +| Time-forgiving precision | `soft_precision_{1,3,5,10,20}` | higher is better | +| Time-forgiving recall | `soft_recall_{1,3,5,10,20}` | higher is better | +| Time-forgiving F1 | `soft_f1_{1,3,5,10,20}` | higher is better | -3. **Temporal Distance Metrics.** -These metrics quantify the temporal offset between predicted and actual anomalies, providing insights into whether the detection method tends to identify anomalies early or late. (CTT and TTC). +Prediction metric aliases: +- `prediction_metrics="all"` expands to all prediction metrics. +- `soft_precision`, `soft_recall`, and `soft_f1` expand over detection ranges + `1, 3, 5, 10, 20`. +Benchopt expects a scalar `value` in every result row. The benchmark reports all +requested metrics and uses a finite fallback value when no primary optimization +metric is selected. + +## Development + +Run the repository tests: + +```bash +python -m pytest +``` + +Run Benchopt's benchmark validation suite: + +```bash +benchopt test . -- -q +``` + +Run a quick syntax check: + +```bash +python -m py_compile objective.py benchmark_utils/*.py solvers/*.py solvers/legacy/*.py +``` + +Useful repository checks before opening a pull request: + +```bash +git diff --check +python -m pytest +benchopt test . -- -q -x +``` + +## Adding a Solver + +New solvers should follow the score-first convention: + +```python +class Solver(BaseSolver): + name = "MySolver" + + parameters = { + "cutoff": [None], + } + + def set_objective(self, X_train, X_test): + ... + + def run(self, _): + self.anomaly_scores = ... + self.anomaly_predictions = cutoff_scores( + self.anomaly_scores, + cutoff=self.cutoff, + ) + + def get_result(self): + result = dict(anomaly_scores=self.anomaly_scores) + if self.anomaly_predictions is not None: + result["anomaly_predictions"] = self.anomaly_predictions + return result +``` + +Keep scores oriented so larger means more anomalous. Use `np.nan` for unscored +score positions and `-1` for ignored prediction positions. ## Contributing -Contributions are welcome! Please feel free to submit a Pull Request. +Contributions are welcome. Keep changes reproducible and scoped: + +- preserve the canonical `anomaly_scores` and optional `anomaly_predictions` + output format; +- add focused tests for objective behavior, solver output shapes, and metric + regressions; +- avoid committing downloaded datasets, model checkpoints, generated outputs, + caches, or local environment files; +- document optional dependencies and expected skips clearly.