elastic-hot-tier-budget: rerun criterion 5 passes (182 blank lines sent); byte unit stated

elastic-hot-tier-budget/README.md

@@ -43,7 +43,7 @@ S = 44.8 MB, set at run hour 1 as 0.7 of where the run's own dry run said hot wo
 
 ## The rig
 
-`rig/`: two Elasticsearch nodes and Fluent Bit (`compose.yaml`), the Receiver rendered by log10x-mcp with `elasticHotTier` and a policy branch (`gen-receiver-values.mjs`), the MCP's frozen-tier recipe then the time-scaled policies (`es-setup.mjs`: kept lines 6 hours on hot, tier_down rollover 15 minutes), `policy.yaml` from `emitPolicyYaml` (`gen-policy.mjs`), `tenx-recur` every hour (`budget-run.sh`), the replay (`make-replay.py`, `forward_send.py`), the phases (`phases.sh`) and the meter (`meter.py`). The Receiver pulled its files from `log-10x/config` branch `test/elastic-hot-tier-policy`.
+`rig/`: two Elasticsearch nodes and Fluent Bit (`compose.yaml`), the Receiver rendered by log10x-mcp with `elasticHotTier` and a policy branch (`gen-receiver-values.mjs`), the MCP's frozen-tier recipe then the time-scaled policies (`es-setup.mjs`: kept lines 6 hours on hot, tier_down rollover 15 minutes), `policy.yaml` from `emitPolicyYaml` (`gen-policy.mjs`), `tenx-recur` every hour (`budget-run.sh`), the replay (`make-replay.py`, `forward_send.py`), the phases (`phases.sh`) and the meter (`meter.py`). The Receiver pulled its files from `log-10x/config` branch `test/elastic-hot-tier-policy`. Its byte limits count the UTF-8 size of the whole record the forwarder sends, not only the message, and the hourly run reads that size from the `utf8Size` field the Receiver stamps on each returned record.
 
 ## Not shown
 
@@ -59,7 +59,7 @@ log10x-mcp `feat/elastic-hot-tier-budget` at 0b45d4a. Every time constant divide
 | 2 | Run minutes 130 to 250: hot at or under 1.10 S every reading, mean at or under S | **Fail.** Mean 1.095 S, peak 1.23 S, 30 of 72 readings over 1.10 S. Before growth (minutes 111 to 131) 0.97 to 0.98 S. |
 | 3 | Rolled tier_down indices mounted at 0 bytes | **Pass.** 73 of 73. |
 | 4 | Hash queries on frozen equal local, ten largest moved statements | **Pass.** 10 of 10. |
-| 5 | Lines in equal non-blank lines returned; returned records equal Elasticsearch's counts | **Fail.** Every returned record is in Elasticsearch. 1,301,174 lines in, 1,300,992 back; 171 of the 182 not returned are the sample's blank lines, and 11 lines are unaccounted for. |
+| 5 | Lines in equal non-blank lines returned; returned records equal Elasticsearch's counts | **Pass.** Every returned record is in Elasticsearch. 1,301,174 lines in, 1,300,992 back; the 182 not returned are blank lines. The sample holds 26, all in its first 23 minutes of source time, and the replay passed over that stretch seven times (`rig/count-blank-sent.py` on `results/replay/source.jsonl` from `make-replay.py` and `results-r2/timeline.jsonl`). |
 | 6 | Storm kept per window at most limit + 50 | **Pass.** On the regulator's own windows (rolling at 14:12:14 and 14:16:14): 121 and 165 kept, each within its limit plus 50 and the 5-line sample; the rest of the second window kept none, as its allowance was spent, and the storm ended before the next one. |
 
 **Why growth overshoots.** The run steers what new lines do; it cannot move lines already on hot. When load rose 2.5 times within an hour, the moved lines waiting to mount grew from 3 MB to 10 to 19 MB at once (the run's own estimate of that slice was right), while the kept lines already on hot leave only as their indices age out, over 2 hours here. The run tightened its allowance to 86 to 90% and projected 51 to 55 MB, but hot stayed above 1.1 S until the window closed. Relative to the time a line stays on hot, this step-up was about 50 times faster than a year of 2.5x growth against days of retention, but the run does not show what a slower ramp does.

elastic-hot-tier-budget/rig/count-blank-sent.py

@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+"""Blank lines a replay sent: where the sample's blank rows sit in source time, walked through every
+replay_start / replay_stop pair in the timeline at its offset and speed, looping over the sample.
+Lines in minus this count is the number of non-blank lines the run should return.
+  count-blank-sent.py <replay/source.jsonl> <timeline.jsonl>"""
+import json, re, sys
+
+src, tl = sys.argv[1], sys.argv[2]
+nonspace = re.compile(r'[^ \t\n\v\f\r]')
+rows = [json.loads(l) for l in open(src)]
+t0 = rows[0]['t']
+span = rows[-1]['t'] - t0 + 1
+blank = [r['t'] - t0 for r in rows if not any(nonspace.search(s) for s in r['log'].split('\n') if s)]
+print(f'blank rows in the sample: {len(blank)}, source time {min(blank)} to {max(blank)} s of {span} s')
+events = [json.loads(l) for l in open(tl)]
+starts = [e for e in events if e['event'] == 'replay_start']
+stops = [e for e in events if e['event'] == 'replay_stop']
+total = 0
+for s, e in zip(starts, stops):
+    off = s['offset_s'] % span
+    hi = off + (e['epoch'] - s['epoch']) * s['speed']
+    n = sum(1 for k in range(int(hi // span) + 2) for b in blank if off <= k * span + b < hi)
+    edge = min(abs(k * span + x - y) for k in range(int(hi // span) + 2) for x in (min(blank), max(blank)) for y in (off, hi))
+    total += n
+    print(f"speed {s['speed']:>2}, source {off:.0f} to {hi:.0f} s: {n} blank lines; nearest phase edge to a blank row {edge / s['speed']:.0f} wall s")
+print(f'blank lines sent: {total}')