feat: enterprise observability platform foundation

Dockerfile

@@ -1,38 +1,26 @@
-# ── Stage 1: build ─────────────────────────────────────────────────────────
+# Stage 1: build
 FROM python:3.11-slim AS builder
 
 WORKDIR /app
-
-# Install build tools
 RUN pip install --upgrade pip build
-
-# Copy project files required for the build
 COPY pyproject.toml requirements.txt README.md ./
 COPY logsight/ logsight/
-
-# Build the wheel
 RUN python -m build --wheel --outdir /dist
 
-# ── Stage 2: runtime ────────────────────────────────────────────────────────
+# Stage 2: runtime
 FROM python:3.11-slim AS runtime
 
+ARG INSTALL_EXTRAS=""
 LABEL maintainer="CoreyLeath-code" \
-      description="LogSight-AI: AI-powered log analysis and anomaly detection" \
-      version="0.1.0"
+      description="LogSight-AI: production log analysis and observability" \
+      version="0.2.0"
 
-# Create a non-root user for security
 RUN useradd --create-home --shell /bin/bash logsight
-
 WORKDIR /app
-
-# Install the built wheel from the builder stage
 COPY --from=builder /dist/*.whl /tmp/
-RUN pip install --no-cache-dir /tmp/*.whl && rm /tmp/*.whl
+RUN WHEEL=$(ls /tmp/*.whl) && if [ -n "$INSTALL_EXTRAS" ]; then pip install --no-cache-dir "$WHEEL[$INSTALL_EXTRAS]"; else pip install --no-cache-dir "$WHEEL"; fi && rm /tmp/*.whl
 
-# Drop privileges
 USER logsight
-
 HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 CMD ["logsight", "health"]
-
 ENTRYPOINT ["logsight"]
 CMD ["--help"]

README.md

@@ -1,133 +1,127 @@
-# LogSight-AI — Explainable Log Analysis Heuristics
+# LogSight-AI — Enterprise Log Monitoring & Anomaly Detection
 
 [![Latest Release](https://img.shields.io/github/v/release/CoreyLeath-code/LogSight-AI?display_name=tag&sort=semver)](https://github.com/CoreyLeath-code/LogSight-AI/releases/latest)
 [![CI](https://github.com/CoreyLeath-code/LogSight-AI/actions/workflows/ci.yml/badge.svg)](https://github.com/CoreyLeath-code/LogSight-AI/actions/workflows/ci.yml)
 [![CodeQL](https://github.com/CoreyLeath-code/LogSight-AI/actions/workflows/codeql.yml/badge.svg)](https://github.com/CoreyLeath-code/LogSight-AI/actions/workflows/codeql.yml)
 [![Python](https://img.shields.io/badge/python-3.10%2B-3776AB.svg)](https://www.python.org/)
-[![Development p50](https://img.shields.io/badge/1k--line_development_p50-10.315_ms-6f42c1)](benchmarks/benchmark_report.md)
-[![Benchmark workload](https://img.shields.io/badge/benchmark_workload-1%2C000_lines-2ea44f)](docs/BENCHMARKING.md)
-[![Local first](https://img.shields.io/badge/data_boundary-local_first-6b7280)](#architecture)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
-## Abstract
+## Overview
 
-LogSight-AI is a local-first Python CLI that parses common log formats, summarizes error patterns, flags message-length outliers, and identifies elevated error-rate windows. Its production package does not transmit logs or require credentials. The detector is an explainable statistical heuristic—not a trained incident classifier, root-cause system, or measured accuracy claim.
+LogSight-AI is a local-first log analysis and anomaly-detection platform that is being extended from a deterministic CLI into a production-oriented observability stack. The original parser/analyzer remains dependency-light and explainable; enterprise integrations are optional and isolated behind adapters.
 
-## Formal detection logic
+## Enterprise architecture
 
-For $N$ parsed entries with message lengths $\ell_i$, the analyzer computes the population mean $\mu$ and population standard deviation $\sigma$. When $\sigma>0$, it flags a length outlier only when
+```mermaid
+flowchart LR
+    A["App / Node Logs"] --> B["Vector / Fluent Bit"]
+    B --> C["Kafka / Redpanda"]
+    C --> D["Parsing + Enrichment"]
+    D --> E["Drain3 / Templates"]
+    D --> F["AI Detection"]
+    F --> G["ClickHouse"]
+    F --> H["Qdrant / Vector Search"]
+    D --> I["OpenTelemetry"]
+    I --> J["Prometheus"]
+    J --> K["Grafana"]
+    F --> L["Alert Webhooks"]
+    G --> M["RAG / Incident Analysis"]
+    H --> M
+    M --> L
+```
 
-\[
-z_i=\frac{|\ell_i-\mu|}{\sigma}>\tau_z,\qquad \tau_z=2.5\ \text{by default}.
-\]
+### Five production layers
 
-Separately, ERROR and CRITICAL entries are flagged by a direct rule. For each complete, non-overlapping window of $W$ entries, it reports an error-rate spike when
+1. **Ingestion & streaming** — bounded async streams plus an optional Kafka/Redpanda adapter. Deploy Vector or Fluent Bit at the edge for collection and metadata enrichment.
+2. **AI & anomaly detection** — deterministic template/frequency signals are available now. The adapter boundary supports Drain3, Sentence-Transformers/LogBERT, ONNX Runtime, and future learned detectors without replacing the core API.
+3. **Storage & search** — ClickHouse event storage and Qdrant vector storage adapters are included; Redis is reserved for caching, deduplication, and rate limiting.
+4. **Observability & MLOps** — OpenTelemetry span helpers and Prometheus-compatible `/metrics` are included. Model/data-drift tooling can be attached to the normalized event stream.
+5. **Delivery & alerting** — `WebhookNotifier` provides a minimal outbound contract for Slack, PagerDuty, Discord, or an internal incident gateway.
 
-\[
-r_j=\frac{e_j}{W}\geq\tau_r,\qquad W=100,\quad \tau_r=0.25\ \text{by default}.
-\]
+## New enterprise interfaces
 
-These definitions map directly to [logsight/analyzer.py](logsight/analyzer.py); zero-variance message lengths receive no z-score, and partial trailing windows are intentionally excluded. Read the complete [mathematical foundations](docs/MATHEMATICAL_FOUNDATIONS.md) and [complexity analysis](docs/COMPLEXITY_ANALYSIS.md).
+- `logsight.enterprise.EnrichedLog` — normalized event schema with service/source/host, trace ID, span ID, template, fingerprint, and attributes.
+- `logsight.enterprise.extract_template()` — dependency-free template baseline; replace with Drain3 for high-cardinality production parsing.
+- `logsight.enterprise.semantic_features()` — deterministic feature baseline; explicitly **not** represented as a learned embedding.
+- `logsight.streaming.InMemoryStream` — bounded local async stream for tests and development.
+- `logsight.streaming.KafkaStream` — optional Kafka/Redpanda producer using `aiokafka`.
+- `logsight.storage.ClickHouseStore` and `QdrantStore` — HTTP adapters for durable analytical and semantic storage.
+- `logsight.api` — optional FastAPI gateway with `/health`, `/metrics`, `/v1/logs`, `/v1/logs/batch`, and `/v1/logs/recent`.
+- `logsight.observability.span()` — no-op-safe OpenTelemetry instrumentation helper.
 
-## Evidence snapshot
+## Enterprise local stack
 
-| Evidence | Value | Scope |
-|---|---:|---|
-| Development benchmark input | 1,000 log lines | Local pipeline microbenchmark, 2026-07-17 |
-| Median / mean latency | 10.315 / 10.394 ms | Development baseline; not a service SLO |
-| Mean throughput | 96.21 pipeline runs/s | Same 1,000-line local workload |
-| Detection threshold | z-score > 2.5 | Fixed default policy, not a calibrated significance level |
-| Spike threshold | error rate >= 0.25 in 100 entries | Fixed default policy, not a learned decision boundary |
+The base project still runs without external services. To launch the reference enterprise stack:
 
-The dated measurements are recorded in [benchmarks/benchmark_report.md](benchmarks/benchmark_report.md). CI generates and retains per-commit benchmark JSON; compare only like-for-like Python, hardware, workload, and warm-up configurations. No labeled incident dataset or precision/recall result is committed.
-
-## Research questions
+```bash
+docker compose -f docker-compose.enterprise.yml up --build
+```
 
-1. What precision, recall, and alert burden do error-level, z-score, and error-rate rules produce on a versioned labeled corpus?
-2. How do non-overlapping, overlapping, and time-based windows trade detection delay against false alerts?
-3. How stable are fixed thresholds across formats, services, and message-length distributions?
-4. How do parsing and analysis latency scale with line count, line length, and unique-message cardinality?
+Services exposed locally:
 
-The [academic audit](docs/ACADEMIC_AUDIT.md) documents the repository's direct algorithmic strengths, evidence boundaries, and next experiments.
+| Service | Port | Purpose |
+|---|---:|---|
+| LogSight API | 8000 | Async ingestion/API gateway |
+| Redpanda | 9092 | Streaming buffer |
+| ClickHouse | 8123 | Log analytics/storage |
+| Qdrant | 6333 | Vector similarity search |
+| Redis | 6379 | Cache/dedup/rate limiting foundation |
+| Prometheus | 9090 | Metrics collection |
+| Grafana | 3000 | Dashboards |
 
-## Architecture
+Install optional Python integrations with:
 
-```mermaid
-flowchart LR
-    A["File or stdin"] --> B["Format parser"]
-    B --> C["Typed LogEntry records"]
-    C --> D["Statistics and anomaly analysis"]
-    D --> E["Rich CLI report"]
+```bash
+pip install -e ".[api,kafka,observability,ai,parsing]"
 ```
 
-Supported formats include ISO-8601 application logs, syslog, nginx access logs, and generic level-prefixed lines. Detection is an explainable statistical heuristic; it is not a trained model and no accuracy claim is made without a labeled evaluation corpus.
-
-## Evidence-backed reasoning
+The optional integrations are deliberately separated so a developer can use the original local-first CLI without downloading heavyweight ML or infrastructure clients.
 
-The optional `--explain` flag turns existing detector output into concise, user-facing evidence statements. Each statement identifies its direct or statistical basis: parsed error level, message-length z-score with its configured threshold, or observed error count/rate in a complete analysis window. Each statement also reports a deterministic support level: `single-signal` for one detector signal and `corroborated` when the same entry meets both error-level and statistical criteria. LogSight does not infer an incident root cause, use an LLM, send logs externally, or report a model-confidence score.
+## Current detection contract
 
-```bash
-logsight analyze application.log --window 200 --spike-threshold 0.20 --explain
-cat application.log | logsight stdin --explain
-```
+The original analyzer uses explainable statistical heuristics: message-length z-score outliers, direct ERROR/CRITICAL rules, and elevated error-rate windows. These are not presented as calibrated incident probabilities. The enterprise layer preserves that evidence-first behavior while adding normalized event context and pluggable AI interfaces.
 
 ## Quick start
 
 ```bash
 python -m venv .venv
-source .venv/bin/activate  # Windows: .venv\Scripts\activate
+source .venv/bin/activate  # Windows: .venv\\Scripts\\activate
 pip install -e .
 logsight health
-logsight analyze application.log
-cat application.log | logsight stdin
+logsight analyze application.log --explain
 ```
 
-Useful controls:
-
-```bash
-logsight analyze application.log --threshold 3.0 --window 200 --spike-threshold 0.20
-```
-
-## Verified metrics
-
-Measured locally on 2026-07-17; CI artifacts are the canonical per-commit record.
-
-| Metric | Value |
-|---|---:|
-| Automated tests | 50 passing |
-| Core package coverage | 95.26% |
-| Benchmark input | 1,000 lines |
-| Median pipeline latency | 10.315 ms |
-| Mean throughput | 96.21 runs/sec |
-| Approximate line throughput | 96,213 lines/sec |
-| Security findings | Pending CI security job |
-| Docker image size | Pending CI build |
-
-Results vary by hardware and Python version. See [Benchmark Guide](docs/BENCHMARKING.md) and [Benchmark Report](benchmarks/benchmark_report.md).
-
 ## Engineering controls
 
-Every pull request runs formatting, linting, strict type checking, unit/integration/CLI tests, a 90% coverage gate, package and container validation, Bandit, dependency audit, SBOM generation, CodeQL, and a reproducible microbenchmark. Checks fail closed.
+Pull requests retain the existing formatting, linting, strict typing, unit/integration/CLI tests, coverage gate, package/container validation, Bandit, dependency audit, SBOM, CodeQL, and reproducible benchmark controls. Enterprise changes should include contract tests for adapters and load tests before production rollout.
+
+## Roadmap
+
+- [x] Normalized enterprise event contract
+- [x] Kafka/Redpanda ingestion adapter
+- [x] ClickHouse/Qdrant storage adapters
+- [x] FastAPI async gateway
+- [x] Prometheus endpoint and OpenTelemetry helper
+- [x] Enterprise Docker Compose reference stack
+- [ ] Drain3 production parser adapter
+- [ ] Sentence-Transformers/LogBERT embedding worker
+- [ ] ONNX Runtime inference worker
+- [ ] Redis-backed deduplication/rate limiting
+- [ ] OTel Collector deployment and trace-log correlation pipeline
+- [ ] Celery/Ray distributed inference workers
+- [ ] Helm chart with HPA/PDB/network policies
+- [ ] RAG incident investigator with versioned runbooks/commits
+- [ ] Labeled benchmark corpus with precision/recall and alert-burden metrics
 
 ## Documentation
 
 - [Production audit](docs/AUDIT.md)
 - [Architecture](docs/architecture.md)
 - [Deployment and rollback checklist](docs/DEPLOYMENT.md)
 - [Benchmark methodology](docs/BENCHMARKING.md)
-- [Runtime metrics](docs/metrics.md)
+- [Mathematical foundations](docs/MATHEMATICAL_FOUNDATIONS.md)
 - [Security policy](SECURITY.md)
 
-The Streamlit and external-LLM files are retained as demonstrations and are not part of the supported package or deployment contract; see the audit for the work required to promote them.
-
-## Development
-
-```bash
-pip install -e ".[dev]"
-ruff format .
-ruff check .
-mypy
-pytest
-```
+## License
 
-Contributions should include tests and documentation for behavioral changes. Report vulnerabilities privately as described in [SECURITY.md](SECURITY.md).
+MIT

deploy/clickhouse/init.sql

@@ -0,0 +1,21 @@
+CREATE DATABASE IF NOT EXISTS logsight;
+
+CREATE TABLE IF NOT EXISTS logsight.events
+(
+    timestamp DateTime64(3, 'UTC'),
+    level LowCardinality(String),
+    service LowCardinality(String),
+    source LowCardinality(String),
+    host LowCardinality(String),
+    message String,
+    raw String,
+    trace_id Nullable(String),
+    span_id Nullable(String),
+    template Nullable(String),
+    fingerprint String,
+    attributes String
+)
+ENGINE = MergeTree
+PARTITION BY toDate(timestamp)
+ORDER BY (service, timestamp, fingerprint)
+TTL toDateTime(timestamp) + INTERVAL 30 DAY;

deploy/prometheus.yml

@@ -0,0 +1,9 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: logsight
+    metrics_path: /metrics
+    static_configs:
+      - targets: ["logsight-api:8000"]

docker-compose.enterprise.yml

@@ -0,0 +1,79 @@
+services:
+  logsight-api:
+    build:
+      context: .
+      args:
+        INSTALL_EXTRAS: api
+    image: logsight-ai:enterprise
+    entrypoint: []
+    command: ["uvicorn", "logsight.api:app", "--host", "0.0.0.0", "--port", "8000"]
+    environment:
+      LOGSIGHT_WINDOW: "500"
+      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
+      CLICKHOUSE_URL: http://clickhouse:8123
+      QDRANT_URL: http://qdrant:6333
+      REDIS_URL: redis://redis:6379/0
+    ports:
+      - "8000:8000"
+    depends_on:
+      - kafka
+      - clickhouse
+      - qdrant
+      - redis
+    read_only: true
+    security_opt:
+      - no-new-privileges:true
+    cap_drop: [ALL]
+    tmpfs:
+      - /tmp:size=32m,mode=1777
+
+  kafka:
+    image: redpandadata/redpanda:v25.2.1
+    command: ["redpanda", "start", "--overprovisioned", "--smp=1", "--memory=1G", "--reserve-memory=0M", "--node-id=0", "--check=false"]
+    ports:
+      - "9092:9092"
+
+  clickhouse:
+    image: clickhouse/clickhouse-server:25.8-alpine
+    ports:
+      - "8123:8123"
+    volumes:
+      - clickhouse_data:/var/lib/clickhouse
+      - ./deploy/clickhouse/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
+
+  qdrant:
+    image: qdrant/qdrant:v1.15.1
+    ports:
+      - "6333:6333"
+    volumes:
+      - qdrant_data:/qdrant/storage
+
+  redis:
+    image: redis:8-alpine
+    command: ["redis-server", "--appendonly", "yes"]
+    ports:
+      - "6379:6379"
+    volumes:
+      - redis_data:/data
+
+  prometheus:
+    image: prom/prometheus:v3.5.0
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./deploy/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+
+  grafana:
+    image: grafana/grafana:12.1.1
+    ports:
+      - "3000:3000"
+    depends_on:
+      - prometheus
+    volumes:
+      - grafana_data:/var/lib/grafana
+
+volumes:
+  clickhouse_data:
+  qdrant_data:
+  redis_data:
+  grafana_data: