feat: enterprise observability platform foundation

[CoreyLeath-code]

Summary

Upgrades LogSight-AI from a local-first analyzer toward an enterprise observability platform while preserving the existing dependency-light CLI.

Included

• Normalized enterprise event contract with service/source/host, trace/span IDs, templates, fingerprints, and attributes.
• Bounded async ingestion with an optional Kafka/Redpanda adapter.
• Dependency-free template/frequency hybrid anomaly baseline with explicit extension points for Drain3, Sentence-Transformers/LogBERT, and ONNX Runtime.
• ClickHouse and Qdrant storage adapters.
• Optional FastAPI async gateway with batch ingestion, recent events, health, and Prometheus-compatible metrics.
• OpenTelemetry span helper.
• Webhook notification contract for Slack/PagerDuty/Discord-style integrations.
• Enterprise Docker Compose reference stack with Redpanda, ClickHouse, Qdrant, Redis, Prometheus, and Grafana.
• ClickHouse schema with partitioning, ordering, compression-friendly column types, and retention TTL.
• Optional dependency groups for API, Kafka, observability, AI, and parsing integrations.
• Tests for the new core enterprise event/detection contracts.
• README roadmap and architecture documentation.

Deliberate boundaries

This PR establishes production architecture and contracts without falsely claiming that the platform already has trained LogBERT models, calibrated precision/recall, a full OTel Collector deployment, distributed Ray/Celery workers, or a complete RAG incident investigator. Those are explicitly tracked in the roadmap.

Next phase

1. Add Drain3 adapter + parser benchmarks.
2. Add Sentence-Transformers/LogBERT inference worker and ONNX export path.
3. Add Redis-backed dedup/rate limiting.
4. Add OTel Collector and trace/log correlation tests.
5. Add Helm chart, HPA/PDB/network policies, and Kubernetes security contexts.
6. Add RAG incident investigator over versioned runbooks and repository commits.
7. Build a labeled incident corpus and publish precision/recall/alert-burden results.

Summary by CodeRabbit

• New Features

  • Added an optional API for single and batch log ingestion, health checks, metrics, and recent-log retrieval.
  • Added enterprise log enrichment, anomaly scoring, template extraction, webhook notifications, and OpenTelemetry support.
  • Added streaming ingestion options, including in-memory and Kafka-based workflows.
  • Added ClickHouse and Qdrant storage integrations.
  • Added a Docker Compose observability stack with Kafka, ClickHouse, Qdrant, Redis, Prometheus, Grafana, and the API.
  • Added ClickHouse event storage with a 30-day retention period.

• Documentation

  • Reworked the README to describe the enterprise observability architecture, interfaces, deployment stack, and roadmap.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 6c95c5cf-9cdf-41fb-99de-1d6889a95593

📥 Commits

Reviewing files that changed from the base of the PR and between 818c16b and 9c2f338.

📒 Files selected for processing (12)

• Dockerfile
• README.md
• deploy/clickhouse/init.sql
• deploy/prometheus.yml
• docker-compose.enterprise.yml
• logsight/api.py
• logsight/enterprise.py
• logsight/observability.py
• logsight/storage.py
• logsight/streaming.py
• pyproject.toml
• tests/test_enterprise.py

---

📝 Walkthrough

Walkthrough

The pull request extends LogSight-AI with optional enterprise ingestion, enrichment, anomaly scoring, streaming, storage, observability, and deployment integrations. It adds a FastAPI gateway, enterprise Docker Compose stack, package extras, ClickHouse schema, Prometheus configuration, and updated documentation.

Changes

Enterprise observability stack

Layer / File(s)	Summary
Event enrichment and anomaly scoring logsight/enterprise.py, tests/test_enterprise.py	Adds EnrichedLog, serialization, enrichment, template extraction, semantic features, hybrid anomaly scoring, webhook notification, and focused tests.
API ingestion and metrics logsight/api.py, logsight/observability.py, deploy/prometheus.yml, docker-compose.enterprise.yml	Adds optional FastAPI endpoints for log ingestion, batch processing, recent events, health, and Prometheus metrics. Adds optional OpenTelemetry spans and wires the API service into Prometheus.
Streaming and storage adapters logsight/streaming.py, logsight/storage.py, deploy/clickhouse/init.sql, docker-compose.enterprise.yml	Adds bounded in-memory and Kafka streams, batching, ClickHouse and Qdrant HTTP adapters, the ClickHouse events table, and supporting Kafka, ClickHouse, Qdrant, and Redis services.
Packaging, deployment, and documentation Dockerfile, pyproject.toml, README.md, docker-compose.enterprise.yml	Adds optional installation extras, conditional Docker wheel installation, enterprise service configuration, persistent volumes, metadata updates, and enterprise architecture documentation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant FastAPI
  participant EnterpriseAnalyzer
  participant Metrics
  Client->>FastAPI: Submit log payload
  FastAPI->>EnterpriseAnalyzer: Parse, enrich, extract template
  EnterpriseAnalyzer-->>FastAPI: Return event and anomaly score
  FastAPI->>Metrics: Increment ingestion and anomaly counters
  FastAPI-->>Client: Return JSON result

Loading

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/enterprise-observability

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}