Event Sequence Detector

docs/detectors.md

@@ -88,6 +88,7 @@ List of detectors:
 * [New Value](detectors/new_value.md): Detect new values in the variables in the logs.
 * [Combo Detector](detectors/combo.md): Detect new combination of variables in the logs.
 * [New Event](detectors/new_event.md): Detect new events in the variables in the logs.
+* [Event Sequence](detectors/event_sequence.md): Detect unseen sequences of consecutive events in the logs.
 * [Value Range](detectors/value_range.md) Detect numeric value ranges in variables in the logs.
 * [Rule Based](detectors/rule_based.md): Detect anomalies based in a set of rules.
 * [Bigram Frequency](detectors/bigram_frequency.md): Detect bigram-frequency-based anomalies in the logs.

docs/detectors/event_sequence.md

@@ -0,0 +1,88 @@
+# Event Sequence Detector
+
+The Event Sequence Detector raises alerts when a run of consecutive event IDs appears in an order that was never observed during training. It is useful to detect broken or unexpected workflows in an environment where the individual events are all benign on their own.
+
+|            | Schema                     | Description        |
+|------------|----------------------------|--------------------|
+| **Input**  | [ParserSchema](../schemas.md) | Structured log  |
+| **Output** | [DetectorSchema](../schemas.md) | Alert / finding |
+
+## Description
+
+The detector slides a window of `fixed_window_size` event IDs over the log stream. During training every full window is stored as a known sequence; during detection a window whose exact sequence is not in that set is reported as an anomaly.
+
+Because a novel event stays inside the window for `fixed_window_size` steps, a single unexpected event produces up to `fixed_window_size` consecutive alerts — one per window it invalidates. This is intentional: each of those windows is a distinct sequence that was never trained.
+
+Sequences are stored as fixed-length n-grams, so a persisted model is only meaningful at the length it was trained with. When state is restored via [persistency](../auxiliar/persistency.md) at a different `fixed_window_size`, the detector logs a warning, adopts the persisted length, and skips auto-configuration.
+
+## Auto configuration
+
+With `auto_config: True` the detector spends the configure phase feeding one window per candidate length in `min_window_size .. max_window_size` (inclusive) and tracking how stable the resulting sequences are. The longest candidate whose sequences are classified `STABLE` or `STATIC` is written to `fixed_window_size`, so the resulting configuration can be replayed verbatim with `auto_config: False`.
+
+Candidates whose window never filled during the configure phase are skipped, so a short configure phase simply narrows the choice.
+
+If no candidate is stable, no window length is meaningful for this log stream. Rather than fall back to an arbitrary length and alert on nearly every window, the detector generates an empty configuration: **no instance of the detector is created**, `fixed_window_size` stays `None`, and it neither trains nor alerts for the rest of the run. A warning names the range that was searched. The same applies to `auto_config: False` without a `fixed_window_size` — the detector stays inert.
+
+Longer windows are more specific and therefore alert more readily; if the auto-configured length is too sensitive, narrow the range or set `fixed_window_size` explicitly.
+
+## Configuration example
+
+```yaml
+detectors:
+    EventSequenceDetector:
+        method_type: event_sequence_detector
+        auto_config: False
+        params:
+            fixed_window_size: 3
+```
+
+With auto configuration:
+
+```yaml
+detectors:
+    EventSequenceDetector:
+        method_type: event_sequence_detector
+        auto_config: True
+        data_use_configure: 500
+        params:
+            min_window_size: 2
+            max_window_size: 10
+```
+
+| Parameter | Default | Description |
+|---|---|---|
+| `fixed_window_size` | `None` | Length of the sliding event-ID window. Overrides `min_window_size`/`max_window_size` and skips auto configuration. Auto configuration writes its own choice here. While it is `None` the detector neither trains nor alerts. Must be `>= 1`. |
+| `min_window_size` | `2` | Shortest window length tried during auto configuration. Must be `>= 1`. |
+| `max_window_size` | `10` | Longest window length tried during auto configuration. Must be `>= min_window_size`. |
+
+## Example usage
+
+```python
+from detectmatelibrary.detectors.event_sequence_detector import EventSequenceDetector, \
+    EventSequenceDetectorConfig
+import detectmatelibrary.schemas as schemas
+
+detector = EventSequenceDetector(
+    name="EventSequenceTest",
+    config=EventSequenceDetectorConfig(auto_config=False, fixed_window_size=3),
+)
+
+parser_data = schemas.ParserSchema({
+    "parserType": "test",
+    "EventID": 1,
+    "template": "test template",
+    "variables": ["var1"],
+    "logID": "1",
+    "parsedLogID": "1",
+    "parserID": "test_parser",
+    "log": "test log message",
+    "logFormatVariables": {"timestamp": "123456"}
+})
+
+
+alert = detector.process(parser_data)
+```
+
+The sequences learned so far are available via `detector.get_known_sequences()`, which returns a set of event-ID tuples.
+
+Go back [Index](../index.md)

mkdocs.yml

@@ -27,6 +27,7 @@ nav:
     - New Value: detectors/new_value.md
     - Combo Detector: detectors/combo.md
     - New Event: detectors/new_event.md
+    - Event Sequence: detectors/event_sequence.md
     - Rule Based: detectors/rule_based.md
     - BiGram Frequency: detectors/bigram_frequency.md
     - CharSet: detectors/charset.md

src/detectmatelibrary/detectors/__init__.py

@@ -4,6 +4,7 @@
 from .new_event_detector import NewEventDetector, NewEventDetectorConfig
 from .value_range_detector import ValueRangeDetector, ValueRangeDetectorConfig
 from .charset_detector import CharsetDetector, CharsetDetectorConfig
+from .event_sequence_detector import EventSequenceDetector, EventSequenceDetectorConfig
 
 __all__ = [
     "random_detector",
@@ -18,5 +19,7 @@
     "CharsetDetector",
     "CharsetDetectorConfig",
     "BigramFrequencyDetector",
-    "BigramFrequencyDetectorConfig"
+    "BigramFrequencyDetectorConfig",
+    "EventSequenceDetector",
+    "EventSequenceDetectorConfig"
 ]

src/detectmatelibrary/detectors/event_sequence_detector.py

@@ -0,0 +1,264 @@
+"""Detect EventID sequences that were not observed during training."""
+
+from collections import deque
+from typing import Any, Sequence
+
+from pydantic import Field, model_validator
+
+from detectmatelibrary.common._config._compile import generate_detector_config
+from detectmatelibrary.common.detector import CoreDetectorConfig, CoreDetector
+from detectmatelibrary.tools.logging import logger
+from detectmatelibrary.utils import persistency
+from detectmatelibrary.utils.data_buffer import BufferMode
+from detectmatelibrary.schemas import ParserSchema, DetectorSchema
+
+_SEQUENCE_SEPARATOR = "\x1f"
+
+
+def _encode_sequence(sequence: Sequence[int]) -> str:
+    return _SEQUENCE_SEPARATOR.join(str(event_id) for event_id in sequence)
+
+
+def _decode_sequence(encoded: str) -> tuple[int, ...]:
+    return tuple(int(event_id) for event_id in encoded.split(_SEQUENCE_SEPARATOR))
+
+
+class EventSequenceDetectorConfig(CoreDetectorConfig):
+    """
+    @param fixed_window_size length of the sliding EventID window. A window whose exact
+           EventID sequence was not seen during training is reported as an anomaly. When
+           set it overrides `min_window_size`/`max_window_size` and skips
+           auto-configuration; auto-configuration writes its own choice here. While it is
+           None the detector is unconfigured and neither trains nor alerts.
+    @param min_window_size shortest window length tried during the auto-configuration
+           phase. Only used while `fixed_window_size` is None.
+    @param max_window_size longest window length tried during the auto-configuration
+           phase. The longest length whose sequences are classified STABLE or STATIC wins.
+    """
+    method_type: str = "event_sequence_detector"
+    min_window_size: int = Field(default=2, ge=1)
+    max_window_size: int = Field(default=10, ge=1)
+    fixed_window_size: int | None = Field(default=None, ge=1)
+
+    @model_validator(mode="after")
+    def _validate_window_range(self) -> "EventSequenceDetectorConfig":
+        if self.max_window_size < self.min_window_size:
+            raise ValueError("max_window_size must be >= min_window_size")
+        return self
+
+
+class EventSequenceDetector(CoreDetector):
+    """Detect EventID sequences not encountered in training as anomalies."""
+
+    def __init__(
+            self,
+            name: str = "EventSequenceDetector",
+            config: EventSequenceDetectorConfig = EventSequenceDetectorConfig()
+    ) -> None:
+        if isinstance(config, dict):
+            config = EventSequenceDetectorConfig.from_dict(config, name)
+
+        super().__init__(name=name, buffer_mode=BufferMode.NO_BUF, config=config)
+        self.config: EventSequenceDetectorConfig
+        # CoreComponent.process() calls train() *and* run()->detect() for every
+        # training event, so a single shared window would ingest each event twice.
+        # maxlen is None while unconfigured, but nothing is appended in that state.
+        self._train_window: deque[int] = deque(maxlen=self.config.fixed_window_size)
+        self._detect_window: deque[int] = deque(maxlen=self.config.fixed_window_size)
+        # ponytail: only events_seen is used here — sequences carry no variables.
+        # EventPersistency still requires an event_data_class, and changing it would
+        # change the on-disk format for no gain.
+        self.persistency = persistency.EventPersistency(
+            event_data_class=persistency.EventStabilityTracker,
+        )
+        self._configure_windows: dict[int, deque[int]] = {}
+        self.auto_conf_persistency = persistency.EventPersistency(
+            event_data_class=persistency.EventStabilityTracker
+        )
+        self._register_persistency(self.persistency)  # restores state when auto_load
+        self._restored_length = self._adopt_restored_length()
+        if not self.config.auto_config and self.config.fixed_window_size is None:
+            logger.warning(
+                f"[{self.name}] auto_config=False but no fixed_window_size was given. "
+                "The detector stays unconfigured and will neither train nor alert."
+            )
+
+    def _set_window_length(self, length: int) -> None:
+        """Set the window length and resize both sliding windows to match."""
+        self.config.fixed_window_size = length
+        self._train_window = deque(self._train_window, maxlen=length)
+        self._detect_window = deque(self._detect_window, maxlen=length)
+
+    def _adopt_restored_length(self) -> int | None:
+        """Align `fixed_window_size` with restored state, if any.
+
+        Sequences are stored as fixed-length n-grams, so a model trained at one
+        length cannot be evaluated at another: every restored entry would miss and
+        every detection would become a false positive. The persisted length
+        therefore wins over the configured one.
+
+        Returns the persisted length, or None when nothing was restored.
+        """
+        restored = self.persistency.get_events_seen()
+        if not restored:
+            return None
+        length = len(_decode_sequence(str(next(iter(restored)))))
+        if length != self.config.fixed_window_size:
+            logger.warning(
+                f"[{self.name}] restored state holds sequences of length {length}, but "
+                f"fixed_window_size is {self.config.fixed_window_size}. Using the "
+                "persisted length — the restored model is only valid at that length."
+            )
+            self._set_window_length(length)
+        return length
+
+    def import_state(
+        self, path: str | bytes, storage_options: dict[str, Any] | None = None
+    ) -> None:
+        """Load state, then align the window length with what was restored.
+
+        Unlike `auto_load`, this runs after construction, so the length check in
+        `__init__` has already passed and has to be redone here.
+        """
+        super().import_state(path, storage_options)
+        self._restored_length = self._adopt_restored_length()
+
+    def train(self, input_: ParserSchema) -> None:  # type: ignore
+        """Train the detector by learning EventID sequences from the input
+        data.
+
+        No-op while the window size is unconfigured.
+        """
+        if (length := self.config.fixed_window_size) is None:
+            return
+        self._train_window.append(input_["EventID"])
+        if len(self._train_window) < length:
+            return
+        self.persistency.ingest_event(
+            event_id=_encode_sequence(self._train_window),
+            event_template=input_["template"]
+        )
+
+    def detect(self, input_: ParserSchema, output_: DetectorSchema) -> bool:  # type: ignore
+        """Report EventID windows that were not seen during training.
+
+        A single novel event stays in the window for `fixed_window_size` steps
+        and therefore yields that many alerts — each window is a distinct unseen
+        sequence. No-op while the window size is unconfigured.
+        """
+        if (length := self.config.fixed_window_size) is None:
+            return False
+        self._detect_window.append(input_["EventID"])
+        if len(self._detect_window) < length:
+            return False
+
+        if _encode_sequence(self._detect_window) in self.persistency.get_events_seen():
+            return False
+
+        sequence = tuple(self._detect_window)
+        output_["score"] = 1.0
+        output_["description"] = f"{self.name} detects unknown EventID sequences as anomalies."
+        output_["alertsObtain"].update({
+            f"Sequence {sequence}": (
+                f"EventID sequence of length {len(sequence)} ending at logID "
+                f"{input_['logID']} was not seen during training."
+            )
+        })
+        return True
+
+    def configure(self, input_: ParserSchema) -> None:  # type: ignore
+        """Feed the event into one sliding window per candidate length.
+
+        Nothing to decide once `fixed_window_size` is set, whether by the user
+        or by restored state.
+        """
+        if self.config.fixed_window_size is not None:
+            return
+        for length in range(self.config.min_window_size, self.config.max_window_size + 1):
+            window = self._configure_windows.setdefault(length, deque(maxlen=length))
+            window.append(input_["EventID"])
+            if len(window) == length:
+                self.auto_conf_persistency.ingest_event(
+                    event_id=length,
+                    event_template=input_["template"],
+                    named_variables={"seq": tuple(window)},
+                )
+
+    def set_configuration(self) -> None:
+        """Choose `fixed_window_size` from the configure-phase data.
+
+        Each candidate length is scored by the stability of the
+        sequences it produced; the longest STABLE or STATIC candidate
+        wins. Candidates whose window never filled during the configure
+        phase produced no data and are skipped. When nothing is stable
+        an empty configuration is generated — this detector then holds
+        no instance and stays silent, which beats alerting on every
+        window at an arbitrary length.
+        """
+        if (fixed := self.config.fixed_window_size) is not None:
+            reason = (
+                "persisted state was restored" if self._restored_length is not None
+                else "fixed_window_size is set"
+            )
+            logger.warning(
+                f"[{self.name}] auto_config=True but {reason}. Keeping window size "
+                f"{fixed}."
+            )
+            self._release_configure_state()
+            return
+
+        stable = []
+        for length, event_tracker in self.auto_conf_persistency.get_events_data().items():
+            tracker = event_tracker.get_data()["seq"]
+            if len(tracker.change_series) < tracker.min_samples:
+                continue
+            if tracker.classify().type in ("STABLE", "STATIC"):
+                stable.append(int(length))
+
+        if not stable:
+            logger.warning(
+                f"[{self.name}] auto_config=True found no stable window size in "
+                f"[{self.config.min_window_size}..{self.config.max_window_size}]. "
+                "Generating an empty configuration — no instance of this detector is "
+                "created and it will neither train nor alert."
+            )
+            old_persist = self.config.persist
+            self.config = EventSequenceDetectorConfig.from_dict(
+                generate_detector_config(
+                    variable_selection={},
+                    detector_name=self.name,
+                    method_type=self.config.method_type,
+                ),
+                self.name,
+            )
+            self.config.persist = old_persist
+            self._release_configure_state()
+            return
+
+        chosen = max(stable)
+        logger.debug(
+            f"[{self.name}] auto_config selected fixed_window_size={chosen} "
+            f"from stable candidates {sorted(stable)}."
+        )
+        self._set_window_length(chosen)
+        self._release_configure_state()
+
+    def _release_configure_state(self) -> None:
+        """Drop configure-phase state — nothing reads it after
+        configuration."""
+        self._configure_windows.clear()
+        self.auto_conf_persistency = persistency.EventPersistency(
+            event_data_class=persistency.EventStabilityTracker
+        )
+
+    def reset_window(self) -> None:
+        """Clear the training and detection windows."""
+        self._train_window.clear()
+        self._detect_window.clear()
+
+    def get_known_sequences(self) -> set[tuple[int, ...]]:
+        """Return the EventID sequences learned during training."""
+        return {
+            _decode_sequence(str(encoded))
+            for encoded in self.persistency.get_events_seen()
+        }