Skip to content

feat(upstream): RFC 2439 route flap damping & oscillation suppressor - #17

Merged
ATLURI0001 merged 1 commit into
mainfrom
feat/upstream-flap-damper
Nov 21, 2026
Merged

ATLURI0001 merged 1 commit into
mainfrom
feat/upstream-flap-damper

Conversation

@ATLURI0001

Copy link
Copy Markdown
Owner

Technical Deep-Dive: Upstream Route Flap Damping & RFC 2439 Oscillation Suppression

1. Problem Statement & Cascading Failure Modes

In high-throughput edge proxies and layer-7 reverse proxy clusters, intermittent hardware faults, transient network partitions, and thrashing container orchestrators cause upstream servers to rapidly oscillate between healthy and unhealthy states.

Unmitigated route flapping induces catastrophic failure cascades:

  1. Continuous routing table invalidation and connection pool thrashing.
  2. Premature reconnection attempts into degraded backends, amplifying blast radius.
  3. Premature recovery flapping (a server accepting queries before its memory cache/JIT warm-up completes, crashing under immediate traffic bursts).

To defend against route flapping, this pull request implements RouteFlapDamper in scripts/flap_damper.py, bringing deterministic RFC 2439 route flap damping principles to ingress gateway upstreams.

2. Defensive Hardening Architecture

[ Upstream Health Check Probe ]
               |
               v
+-------------------------------------------------------------+
| RouteFlapDamper                                             |
|                                                             |
| 1. Decay historical penalty:                                |
|    P(t) = P_0 * 2^(-dt / T_half)                            |
|                                                             |
| 2. Detect State Flap (True -> False or False -> True):      |
|    P(t) += penalty_per_flap                                 |
|                                                             |
| 3. State Evaluation:                                        |
|    - If P(t) >= suppress_threshold (e.g. 2000):             |
|      -> State: SUPPRESSED (Routable: False)                 |
|    - If Suppressed & P(t) <= reuse_threshold (e.g. 600):    |
|      -> State: HEALTHY / FLAPPING (Routable: True)          |
|    - If elapsed >= max_suppress_seconds (e.g. 60s):         |
|      -> Force unsuppress ceiling                            |
+-------------------------------------------------------------+

Mathematical Formulation

The accumulated penalty decays continuously as an exponential decay function:
$$P(t) = P(t_0) \cdot 2^{-\frac{\Delta t}{T_{\text{half}}}}$$

Where:

  • $T_{\text{half}}$ is the configured half-life duration in seconds (half_life_seconds).
  • Each health transition adds a fixed penalty quantum $\Delta P = \text{penalty_per_flap}$.
  • If $P(t) \ge \text{suppress_threshold}$, the host is isolated from routing tables regardless of whether its latest health check succeeded.
  • Re-admission to the active pool requires the penalty to passively cool down below $\text{reuse_threshold}$.

Defensive Safeguards

  • Hysteresis Banding: Guaranteed separation between suppress_threshold and reuse_threshold prevents boundary fluttering.
  • Maximum Suppression Timeout: Prevents permanent deadlocking of critical upstreams through max_suppress_seconds.
  • Zero-Allocation Tracking: In-memory status tracking with microsecond timestamp granularity.

3. Test Coverage & Verification

  • test_flap_damping_config_validation: Enforces invariant checks on configuration parameters ($suppress &gt; reuse &gt; 0$).
  • test_flap_damping_stable_upstream: Confirms zero penalty accumulation for consistently healthy backends.
  • test_flap_damping_escalation_to_suppressed: Validates threshold breach and immediate traffic isolation under rapid flapping.
  • test_flap_damping_exponential_decay_and_reuse: Verifies accurate mathematical half-life decay and automatic restoration once below reuse threshold.
  • test_flap_damping_max_suppress_timeout: Asserts upper bound timeout release for suppressed nodes.
  • test_flap_damping_reset: Verifies programmatic reset and maintenance overrides.

@ATLURI0001
ATLURI0001 merged commit 985c15f into main Nov 21, 2026
0 of 2 checks passed
@ATLURI0001
ATLURI0001 deleted the feat/upstream-flap-damper branch November 21, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant