diff --git a/pipelines/run/modules/receive/rate/doc.md b/pipelines/run/modules/receive/rate/doc.md index 2ae5dbff..a4fcd806 100644 --- a/pipelines/run/modules/receive/rate/doc.md +++ b/pipelines/run/modules/receive/rate/doc.md @@ -25,7 +25,7 @@ An optional total file bounds what each service passes per window on this node, - **Largest first.** When the service's caps alone let more than its total through in the previous window, every pattern's cap scales by total / that amount for the whole window, or by what they let through so far this window once that is larger. The service's largest patterns reach the scaled cap first and are held, and its small ones keep passing. The scale follows the previous window, so the window after a storm is scaled by the storm for its whole length and recovers the window after. The first window after a quiet one fills its total in arrival order before the scale applies. A service whose first lines of this window and of the last window it sent in are two window lengths or more apart scales by total / its units so far this window instead. - **Hard bound.** Once the lines the service passed reach its total, every further line takes the service's action. -The baseline sample counts toward the total like any line. Once the total is spent, the service passes at most `baselineMaxOverTotal` more sample lines in the window (default 50, counted in lines whatever the `unit`), and the rest of its samples take the service's action, so a service that prints many new patterns at once exceeds its total by that many lines at most. The error-class lines in `keepCounts` and the lines a mute entry passes always pass and count toward the total, so the service's other patterns give way first; an error storm can exceed the total by the keep count per storming pattern, with no limit per service. Each regulator instance holds its own total; the MCP's budget run (`log10x_setup_recurring`) divides a fleet budget across instances, and a node added between runs enforces its own total until the next run. +The baseline sample counts toward the total like any line. Once the total is spent, the service passes at most `baselineMaxOverTotal` more sample lines in the window (default 50, counted in lines whatever the `unit`), and the rest of its samples take the service's action. The error-class lines in `keepCounts` count toward the total the same way, and once it is spent the service passes at most `keepMaxOverTotal` more of them in the window (default 500, counted in lines), so with the shipped settings a burst of new error patterns exceeds the total by at most 550 lines. The lines a mute entry passes always pass and count toward the total, so the service's other patterns give way first. Each regulator instance holds its own total; the MCP's budget run (`log10x_setup_recurring`) divides a fleet budget across instances, and a node added between runs enforces its own total until the next run. **File format**, CSV with a header row, keyed by the `containerField` value, read through `totalLookupFile` (`TOTAL_LOOKUP_FILE`): @@ -157,6 +157,7 @@ rateReceiver: warmupMs: 0 # per-instance grace before capping; 0 caps from the first line baselineCount: 5 baselineMaxOverTotal: 50 # sample lines a service passes per window once over its total; 0 = no bound + keepMaxOverTotal: 500 # keep-count lines a service passes per window once over its total; 0 = no bound capLookup: # file: $=path("data/caps") + "/caps.csv" # optional per-container caps, with a * row retain: $=parseDuration("10m") diff --git a/pipelines/run/modules/receive/rate/module.yaml b/pipelines/run/modules/receive/rate/module.yaml index 8c0edcbd..36a43a9c 100644 --- a/pipelines/run/modules/receive/rate/module.yaml +++ b/pipelines/run/modules/receive/rate/module.yaml @@ -180,8 +180,9 @@ module: sample rate under a mute. A listed level skips its `rateReceiverSeverityFloors` draw. The count is per pattern, per container, per regulator instance (one per node), per window - (`rateReceiverResetIntervalMs`). The lines it passes count toward the container's total, so - an error storm can exceed the total by the count times the storming patterns. + (`rateReceiverResetIntervalMs`). The lines it passes count toward the container's total, and + once the total is spent the container passes at most `rateReceiverKeepMaxOverTotal` more of + them in the window. Keys must match the level vocabulary emitted by the level enrichment. The module's settings set `ERROR=50`, `CRITICAL=50` and `FATAL=50`, and entries merge by level, so those three @@ -458,9 +459,10 @@ module: The `rateReceiverBaselineCount` sample counts toward the total like any line, and once the total is spent the service passes at most `rateReceiverBaselineMaxOverTotal` more sample lines in the - window. The `rateReceiverKeepCounts` lines and the lines an active mute entry passes always pass - and count toward the total, so an error storm or a muted pattern can exceed it by that much: the - keep count is per pattern, with no limit per service. + window. The `rateReceiverKeepCounts` lines count toward the total the same way, and once it is + spent the service passes at most `rateReceiverKeepMaxOverTotal` more of them in the window. The + lines an active mute entry passes always pass and count toward the total, so a muted pattern can + exceed it by that much. **File format** (CSV; header row + one comma-separated entry per row): @@ -588,6 +590,24 @@ module: bound, so every sample line passes. Read by the cap-file regulators, for services with a total. defValue: 50 + - names: + - rateReceiverKeepMaxOverTotal + description: keep-count lines a service passes per window once it has passed its total + type: number + required: false + doc: | + defines how many `rateReceiverKeepCounts` lines a service passes per window on this regulator + instance once it has passed its total (`rateReceiverTotalLookupFile`). Within the total a + keep-count line passes like any line. Past it, keep-count lines pass up to this count, and the + rest take the service's action. This bounds a service that prints many new error patterns in + one window, each of which would otherwise pass its keep count over the total. + + The count is in lines whatever `rateReceiverUnit` is, like the keep counts, so under `bytes` a + service can exceed its total by this many of its own lines. `0` removes the bound, so every + keep-count line passes. The lines an active mute entry passes follow the entry and are not + counted here. Read by the cap-file regulators, for services with a total. + defValue: 500 + - names: - rateReceiverAbsoluteCap description: fleet-wide cap per pattern per container per window; `0` (default) disables the cap unless overridden per-container via `rateReceiverCapLookupFile`