Skip to content

feat(telemetry): add ingress traffic telemetry and latency anomaly diagnostic - #16

Merged
ATLURI0001 merged 1 commit into
mainfrom
feat/traffic-metrics-telemetry
Nov 21, 2026
Merged

ATLURI0001 merged 1 commit into
mainfrom
feat/traffic-metrics-telemetry

Conversation

@ATLURI0001

Copy link
Copy Markdown
Owner

1. Architecture & Diagnostic Overview

This pull request introduces TrafficTelemetry and AnomalyEvent in scripts/traffic_telemetry.py alongside pytest.ini test path configuration. It establishes an in-flight statistical latency profiling rotary and anomaly diagnostic engine for RouteKernel ingress proxies:

+--------------------------------------------------------------------------------------------------+
|                            ROUTEKERNEL INGRESS TRAFFIC TELEMETRY                                 |
|                                                                                                  |
|   HTTP Traffic Stream           Sliding Window & EWMA Tracking         Statistical Diagnostic    |
|   +-------------------+         +----------------------------+         +----------------------+  |
|   | Target: upstream  | ======> | Latency Deque (N=1000)     | ======> | Compute Mean & Var   |  |
|   | Status: 200 / 5xx |         | EWMA Latency Buffer        |         | z = (lat - u) / s    |  |
|   | Latency: 25.4ms   |         | Status Code Histograms     |         +----------------------+  |
|   +-------------------+         +----------------------------+                    |              |
|                                                                                   v              |
|   +------------------------------------------------------------------------------------------+   |
|   |                          UPSTREAM ANOMALY & EXPOSITION ENGINE                            |   |
|   |                                                                                          |   |
|   |   - If z >= 3.0:  Flag Latency Anomaly (WARNING if z < 4.5, CRITICAL if z >= 4.5)        |   |
|   |   - Percentiles:  p50, p90, p99, and Max across active windows                           |   |
|   |   - Exposition:   Standard Prometheus metric format for Grafana ingestion                |   |
|   +------------------------------------------------------------------------------------------+   |
+--------------------------------------------------------------------------------------------------+

2. Statistical Anomaly Detection Mechanics

  1. Sliding Window Statistics:
    Maintains a bounded circular buffer (default $N = 1,000$ requests per upstream target). Once the minimum sample threshold $N \ge 20$ is satisfied, sample mean and variance are computed:
    $$\mu = \frac{1}{N}\sum_{i=1}^N L_i, \quad \sigma^2 = \frac{1}{N - 1}\sum_{i=1}^N (L_i - \mu)^2$$
  2. Defensive Variance Floor:
    To prevent division-by-zero or micro-jitter false alarms on ultra-consistent synthetic traffic, effective standard deviation is guarded by a dynamic floor:
    $$\sigma_{\text{eff}} = \max\left(\sigma, 0.05 \cdot \mu, 0.1\text{ ms}\right)$$
  3. Z-Score Spike Classification:
    $$z = \frac{L_{\text{observed}} - \mu}{\sigma_{\text{eff}}}$$
    • $z &lt; 3.0$: Normal routing variance.
    • $3.0 \le z &lt; 4.5$: Elevated variance flagged as WARNING.
    • $z \ge 4.5$: Acute service disruption or upstream saturation flagged as CRITICAL.
  4. Exponentially Weighted Moving Average (EWMA):
    Maintains smoothed latency trends $\text{EWMA}t = \alpha \cdot L_t + (1 - \alpha) \cdot \text{EWMA}{t-1}$ with smoothing parameter $\alpha = 0.1$.

3. Prometheus Metric Exposition Schema

  • routekernel_requests_total{service="..."}: Total request throughput counter.
  • routekernel_upstream_errors_total{service="...",upstream="..."}: Upstream 5xx error counter per target.
  • routekernel_latency_ms{service="...",quantile="0.50|0.90|0.99"}: Latency percentile gauges.
  • routekernel_anomalies_total{service="..."}: Cumulative anomaly count.
  • routekernel_http_status_total{service="...",status="<code>"}: Status code breakdown.

4. Verification & Diagnostic Test Coverage

Full unit test suite in tests/test_traffic_telemetry.py:

  • test_traffic_telemetry_recording_and_percentiles: Asserts exact $p50$, $p90$, $p99$, and maximum latencies over a 100-sample distribution.
  • test_traffic_telemetry_anomaly_detection_spike: Verifies $z$-score calculation and critical severity assignment on sudden upstream latency spikes.
  • test_traffic_telemetry_error_tracking: Validates 5xx and connection drop tracking per upstream target.
  • test_traffic_telemetry_prometheus_export: Validates metric syntax against standard Prometheus exposition guidelines.
  • All 46 pytest assertions passing cleanly.

@ATLURI0001
ATLURI0001 merged commit 92c020c into main Nov 21, 2026
0 of 2 checks passed
@ATLURI0001
ATLURI0001 deleted the feat/traffic-metrics-telemetry branch November 21, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant