Rate regulator: at most 500 keep-count lines per service per window past its total (E90)

pipelines/run/receive/rate/config.yaml

@@ -78,7 +78,9 @@ rateReceiver:
   # 'keepCounts' is the number of lines per pattern per window a level passes once
   #  the pattern is over its cap or its service over its total, also under a mute
   #  entry, in place of its 'severityFloors' draw. ERROR, CRITICAL and FATAL keep
-  #  a count of 50 even when removed here; -1 gives a level its floor instead.
+  #  a count of 50 even when removed here; -1 gives a level its floor instead. In
+  #  a service with a total, the lines count toward it and 'keepMaxOverTotal'
+  #  bounds them past it.
   keepCounts:
     - ERROR=50
     - CRITICAL=50    # the level enrichment maps fatal/critical -> CRITICAL
@@ -118,6 +120,13 @@ rateReceiver:
   #  bound, so every sample line passes.
   baselineMaxOverTotal: 50
 
+  # 'keepMaxOverTotal' is the most 'keepCounts' lines a service passes per window
+  #  once it has passed its total; the rest take the service's action. Within the
+  #  total they pass like any line. It bounds a service that prints many new
+  #  error patterns in one window. Counted in lines whatever 'unit' is. 0 removes
+  #  the bound, so every keep-count line passes.
+  keepMaxOverTotal: 500
+
   # 'ingestionCostPerGB' is used only to render savings metrics in dollars; it does
   #  not affect the keep/drop decision (the decision is share-based).
   ingestionCostPerGB: 1.5
@@ -262,12 +271,13 @@ rateReceiver:
   # takes the service's action. The 'baselineCount' sample counts toward the
   # total like any line, and once the total is spent the service passes at most
   # 'baselineMaxOverTotal' more sample lines in the window. The 'keepCounts'
-  # lines and the lines a mute entry passes still pass and count toward the
-  # total, so an error storm or a muted pattern can exceed it by that much. Read
-  # by the cap-file regulators, for containers with a cap. A row keyed '*'
-  # applies to every service without a row of its own; a row of 0 exempts its
-  # service, and a '*' row of 0 gives every unlisted service no total. A missing
-  # file loads as an empty table.
+  # lines count toward the total the same way, and once it is spent the service
+  # passes at most 'keepMaxOverTotal' more of them in the window. The lines a
+  # mute entry passes still pass and count toward the total, so a muted pattern
+  # can exceed it by that much. Read by the cap-file regulators, for containers
+  # with a cap. A row keyed '*' applies to every service without a row of its
+  # own; a row of 0 exempts its service, and a '*' row of 0 gives every unlisted
+  # service no total. A missing file loads as an empty table.
   #
   # File format (CSV; header row + one comma-separated entry per row):
   #   container,total

pipelines/run/receive/rate/rate-object-cap.js

@@ -266,10 +266,21 @@ export class rateReceiverCapObject extends TenXObject {
         if (!keep && inSample && sampleOver > 0 && TenXCounter.getAndInc("rg_sx_" + container, 1, windowMs) < sampleOver) keep = true;
 
         // Error-class levels with a keep count: the first K lines of a pattern
-        // held by its cap or by the service total pass each window.
+        // held by its cap or by the service total pass each window, within the
+        // service total like any line, and past it up to
+        // rateReceiverKeepMaxOverTotal keep lines per service per window.
         var keptBefore = -1;
         if (!keep && keepCount >= 0) keptBefore = TenXCounter.getAndInc("rg_kc_" + key, 1, windowMs);
-        if (keptBefore >= 0 && keptBefore < keepCount) keep = true;
+        var inKeep = false;
+        if (keptBefore >= 0 && keptBefore < keepCount) inKeep = true;
+        var keepOver = TenXEnv.get("rateReceiverKeepMaxOverTotal", 500);
+        if (inKeep && serviceTotal <= 0) keep = true;
+        if (inKeep && keepOver <= 0) keep = true;
+        if (inKeep && !keep) passedBefore = TenXCounter.getAndInc("rg_pas_" + container, units, windowMs);
+        if (inKeep && !keep && (passedBefore + units) <= serviceTotal) counted = true;
+        if (inKeep && !keep && !counted) TenXCounter.getAndInc("rg_pas_" + container, 0 - units, windowMs);
+        if (inKeep && counted) keep = true;
+        if (inKeep && !keep && TenXCounter.getAndInc("rg_kx_" + container, 1, windowMs) < keepOver) keep = true;
 
         if (keep && serviceTotal > 0 && !counted) TenXCounter.getAndInc("rg_pas_" + container, units, windowMs);
         if (keep) return true;

pipelines/run/receive/rate/rate-object-lookup-cap.js

@@ -242,10 +242,21 @@ export class rateReceiverLookupCapObject extends TenXObject {
         if (!keep && inSample && sampleOver > 0 && TenXCounter.getAndInc("rg_sx_" + container, 1, windowMs) < sampleOver) keep = true;
 
         // Error-class levels with a keep count: the first K lines of a pattern
-        // held by its cap or by the service total pass each window.
+        // held by its cap or by the service total pass each window, within the
+        // service total like any line, and past it up to
+        // rateReceiverKeepMaxOverTotal keep lines per service per window.
         var keptBefore = -1;
         if (!keep && keepCount >= 0) keptBefore = TenXCounter.getAndInc("rg_kc_" + key, 1, windowMs);
-        if (keptBefore >= 0 && keptBefore < keepCount) keep = true;
+        var inKeep = false;
+        if (keptBefore >= 0 && keptBefore < keepCount) inKeep = true;
+        var keepOver = TenXEnv.get("rateReceiverKeepMaxOverTotal", 500);
+        if (inKeep && serviceTotal <= 0) keep = true;
+        if (inKeep && keepOver <= 0) keep = true;
+        if (inKeep && !keep) passedBefore = TenXCounter.getAndInc("rg_pas_" + container, units, windowMs);
+        if (inKeep && !keep && (passedBefore + units) <= serviceTotal) counted = true;
+        if (inKeep && !keep && !counted) TenXCounter.getAndInc("rg_pas_" + container, 0 - units, windowMs);
+        if (inKeep && counted) keep = true;
+        if (inKeep && !keep && TenXCounter.getAndInc("rg_kx_" + container, 1, windowMs) < keepOver) keep = true;
 
         if (keep && serviceTotal > 0 && !counted) TenXCounter.getAndInc("rg_pas_" + container, units, windowMs);
         if (keep) return true;

pipelines/run/receive/rate/schema.json

@@ -76,7 +76,7 @@
             "string",
             "null"
           ],
-          "markdownDescription" : "Lines per pattern per window that pass for a level once the pattern is held\n\nDefines a map of [severity levels](https://doc.log10x.com/run/initialize/level/) to a count: the number of lines of each pattern of that level that pass per window once the pattern is over its cap or its container is over its total (`rateReceiverTotalLookupFile`), and under an active mute-file entry. Lines past the count take the container's action, or the mute's sample rate under a mute. A listed level skips its `rateReceiverSeverityFloors` draw.  The count is per pattern, per container, per regulator instance (one per node), per window (`rateReceiverResetIntervalMs`). The lines it passes count toward the container's total, so an error storm can exceed the total by the count times the storming patterns.  Keys must match the level vocabulary emitted by the level enrichment. The module's settings set `ERROR=50`, `CRITICAL=50` and `FATAL=50`, and entries merge by level, so those three levels always keep a count (50 unless set); an entry for one of them in the config or on the command line comes later and wins, and `-1` gives it its floor instead. Any other level not listed follows its floor.  For example:  ``` yaml keepCounts:   - ERROR=50   - CRITICAL=50   - FATAL=50 ```",
+          "markdownDescription" : "Lines per pattern per window that pass for a level once the pattern is held\n\nDefines a map of [severity levels](https://doc.log10x.com/run/initialize/level/) to a count: the number of lines of each pattern of that level that pass per window once the pattern is over its cap or its container is over its total (`rateReceiverTotalLookupFile`), and under an active mute-file entry. Lines past the count take the container's action, or the mute's sample rate under a mute. A listed level skips its `rateReceiverSeverityFloors` draw.  The count is per pattern, per container, per regulator instance (one per node), per window (`rateReceiverResetIntervalMs`). The lines it passes count toward the container's total, and once the total is spent the container passes at most `rateReceiverKeepMaxOverTotal` more of them in the window.  Keys must match the level vocabulary emitted by the level enrichment. The module's settings set `ERROR=50`, `CRITICAL=50` and `FATAL=50`, and entries merge by level, so those three levels always keep a count (50 unless set); an entry for one of them in the config or on the command line comes later and wins, and `-1` gives it its floor instead. Any other level not listed follows its floor.  For example:  ``` yaml keepCounts:   - ERROR=50   - CRITICAL=50   - FATAL=50 ```",
           "items" : {
             "type" : "string"
           }
@@ -147,7 +147,7 @@
             "string",
             "null"
           ],
-          "markdownDescription" : "Declarative per-service total file, keyed by the `rateReceiverContainerField` value; bounds what each capped service passes per window\n\nDefines the path to a declarative **per-service total file** (sibling to the cap file). Each entry bounds what a service passes per window (`rateReceiverResetIntervalMs`) on this regulator instance, in `rateReceiverUnit`. Read by the cap-file regulator classes, for containers with a resolved cap.  Two rules apply once a service has a total:  - **Largest first.** When the service's caps alone let more than its total through in the   previous window, every pattern's cap scales by total / that amount for the whole window,   or by what they let through so far this window once that is larger, so the service's   largest patterns are held first and its small ones keep passing. The scale follows the   previous window, so the window after a storm is scaled by the storm for its whole length   and recovers the window after. The first window after a quiet one fills its total in   arrival order before the scale applies. A service whose first lines of this window and   of the last window it sent in are two window lengths or more apart scales by total / its   units so far this window instead. - **Hard bound.** Once the lines the service passed reach its total, every further line takes   the service's action (`rateReceiverActionLookupFile`).  The `rateReceiverBaselineCount` sample counts toward the total like any line, and once the total is spent the service passes at most `rateReceiverBaselineMaxOverTotal` more sample lines in the window. The `rateReceiverKeepCounts` lines and the lines an active mute entry passes always pass and count toward the total, so an error storm or a muted pattern can exceed it by that much: the keep count is per pattern, with no limit per service.  **File format** (CSV; header row + one comma-separated entry per row):  ``` container,total <container>,<total>[:<untilEpochSec>][:<reason>] *,<total>[:<untilEpochSec>][:<reason>] ```  - `container`, value of `rateReceiverContainerField` (the k8s container = the service), or   `*` for every service without a row of its own. - `total`, the service's total per window on this instance, in `rateReceiverUnit`. `0`   exempts the service; a `*` row of `0` gives every service without a row of its own no   total. - `untilEpochSec`, optional Unix-epoch (seconds) expiry; past it the entry is a no-op. - `reason`, optional free-text for audit. Must not contain commas (would break CSV parsing).  A missing file loads as an empty table, so no service has a total until a row names it. Each regulator instance holds its own total; the MCP's budget run (`log10x_setup_recurring`) divides a fleet total across instances.  **From a policy repo.** With `GH_ENABLED=true`, the default is the GitHub policy lane's mirrored copy of the repo's `log10x/policy/totals.csv`: `${GH_DEST}/log10x/policy/totals.csv`, with `GH_DEST` defaulting to `${java.io.tmpdir}/tenx/github` (`pipelines/gitops/config.yaml`). Each sync rewrites the copy in place, so the lookup hot-reloads. `GH_ENABLED` turns on with `true`, `True` or `TRUE`, as the lane reads it, and with the Kubernetes ConfigMap lane also on the GitHub copy wins: enable one lane. (Default: )",
+          "markdownDescription" : "Declarative per-service total file, keyed by the `rateReceiverContainerField` value; bounds what each capped service passes per window\n\nDefines the path to a declarative **per-service total file** (sibling to the cap file). Each entry bounds what a service passes per window (`rateReceiverResetIntervalMs`) on this regulator instance, in `rateReceiverUnit`. Read by the cap-file regulator classes, for containers with a resolved cap.  Two rules apply once a service has a total:  - **Largest first.** When the service's caps alone let more than its total through in the   previous window, every pattern's cap scales by total / that amount for the whole window,   or by what they let through so far this window once that is larger, so the service's   largest patterns are held first and its small ones keep passing. The scale follows the   previous window, so the window after a storm is scaled by the storm for its whole length   and recovers the window after. The first window after a quiet one fills its total in   arrival order before the scale applies. A service whose first lines of this window and   of the last window it sent in are two window lengths or more apart scales by total / its   units so far this window instead. - **Hard bound.** Once the lines the service passed reach its total, every further line takes   the service's action (`rateReceiverActionLookupFile`).  The `rateReceiverBaselineCount` sample counts toward the total like any line, and once the total is spent the service passes at most `rateReceiverBaselineMaxOverTotal` more sample lines in the window. The `rateReceiverKeepCounts` lines count toward the total the same way, and once it is spent the service passes at most `rateReceiverKeepMaxOverTotal` more of them in the window. The lines an active mute entry passes always pass and count toward the total, so a muted pattern can exceed it by that much.  **File format** (CSV; header row + one comma-separated entry per row):  ``` container,total <container>,<total>[:<untilEpochSec>][:<reason>] *,<total>[:<untilEpochSec>][:<reason>] ```  - `container`, value of `rateReceiverContainerField` (the k8s container = the service), or   `*` for every service without a row of its own. - `total`, the service's total per window on this instance, in `rateReceiverUnit`. `0`   exempts the service; a `*` row of `0` gives every service without a row of its own no   total. - `untilEpochSec`, optional Unix-epoch (seconds) expiry; past it the entry is a no-op. - `reason`, optional free-text for audit. Must not contain commas (would break CSV parsing).  A missing file loads as an empty table, so no service has a total until a row names it. Each regulator instance holds its own total; the MCP's budget run (`log10x_setup_recurring`) divides a fleet total across instances.  **From a policy repo.** With `GH_ENABLED=true`, the default is the GitHub policy lane's mirrored copy of the repo's `log10x/policy/totals.csv`: `${GH_DEST}/log10x/policy/totals.csv`, with `GH_DEST` defaulting to `${java.io.tmpdir}/tenx/github` (`pipelines/gitops/config.yaml`). Each sync rewrites the copy in place, so the lookup hot-reloads. `GH_ENABLED` turns on with `true`, `True` or `TRUE`, as the lane reads it, and with the Kubernetes ConfigMap lane also on the GitHub copy wins: enable one lane. (Default: )",
           "default" : ""
         },
         "configGenerationFile" : {
@@ -185,6 +185,15 @@
           "markdownDescription" : "Sample lines a service passes per window once it has passed its total\n\nDefines how many `rateReceiverBaselineCount` sample lines a service passes per window on this regulator instance once it has passed its total (`rateReceiverTotalLookupFile`). Within the total the sample passes like any line. Past it, sample lines pass up to this count, and the rest take the regular path: the keep count for the levels in `rateReceiverKeepCounts`, then the service's action. This bounds a service that prints many new patterns in one window, each of which would otherwise pass its sample over the total.  The count is in lines whatever `rateReceiverUnit` is, like the sample and the keep counts, so under `bytes` a service can exceed its total by this many of its own lines. `0` removes the bound, so every sample line passes. Read by the cap-file regulators, for services with a total. (Accepts number or string with $= prefix for runtime evaluation) (Default: 50)",
           "default" : 50
         },
+        "keepMaxOverTotal" : {
+          "type" : [
+            "number",
+            "string",
+            "null"
+          ],
+          "markdownDescription" : "Keep-count lines a service passes per window once it has passed its total\n\nDefines how many `rateReceiverKeepCounts` lines a service passes per window on this regulator instance once it has passed its total (`rateReceiverTotalLookupFile`). Within the total a keep-count line passes like any line. Past it, keep-count lines pass up to this count, and the rest take the service's action. This bounds a service that prints many new error patterns in one window, each of which would otherwise pass its keep count over the total.  The count is in lines whatever `rateReceiverUnit` is, like the keep counts, so under `bytes` a service can exceed its total by this many of its own lines. `0` removes the bound, so every keep-count line passes. The lines an active mute entry passes follow the entry and are not counted here. Read by the cap-file regulators, for services with a total. (Accepts number or string with $= prefix for runtime evaluation) (Default: 500)",
+          "default" : 500
+        },
         "absoluteCap" : {
           "type" : [
             "number",