elastic-hot-tier-budget: E103 fix on the two-arm smoke, arm A, 0 cross-container records

elastic-hot-tier-budget/README.md

@@ -160,3 +160,28 @@ Warm peaked at 20.7 MB in arm A (08:34:51) and 9.1 MB in arm B (08:33:11). Each
 **Rig additions.** `compose.warm.yaml`, `phases-two-arm-warm.sh` (the 2026-10-24 schedule with warm placement; containers and volumes removed before and after), `es-setup.mjs` and `es-setup-b.mjs` (`TD_PLACEMENT=warm` renders the recipe with the option and checks each bootstrap index sits on a warm node), `meter.py` (each shard's node from `_cat/shards`: per stream, store on hot and on warm; per arm, the warm node's store), `score-two-arm.py` (the warm lines beside the unchanged criteria), `run.sh clean`.
 
 Not shown: the cost of a warm tier, a measured Elastic Cloud bill, 300 nodes, compact, autoscaling, or a hold longer than the hot retention.
+
+## Grouping across containers on the E103 fix, arm A, 2026-10-26
+
+The question: on the rig where the defect was found, does the E103 fix return any record holding lines from more than one container? The two-arm smoke's schedule (every phase at a tenth of its length, the dry run at 240 seconds) ran for arm A alone from 00:10 to 00:36 UTC. Results are in `results-e103-smoke/`.
+
+The engine is commit `0cd458f0f` (log-10x/engine, branch `fix/e103-sequence-key`) with modules commit `6c69d90a` (log-10x/modules, same branch). They ran in the image `local/edge-10x:e103` (sha256:8f74b8bbec2e…), which runs the engine jar on a JVM, so the Receiver had `TENX_JAVA_HEAP=320M` inside its rendered 512 MB limit. Everything else is the 2026-10-24 smoke: the same feed (sha256 24e02233a797…), log10x-mcp 1.30.84 (d6fa2b1), the same ILM policies and the same policy branch.
+
+**Result: no cross-container record, and delivery is exact.**
+
+| Count | Value |
+|---|---:|
+| Lines fed, by seq | 101,663 |
+| Non-blank lines fed | 101,637 |
+| Records returned | 79,415, plus the Receiver's connect message |
+| Records holding lines from more than one container | 0 |
+| Non-blank lines missing (sent, in no record) | 0 |
+| Lines duplicated or unmatched (returned, no unused sent line) | 0 |
+| Records with lines out of feed order | 0 |
+| Elasticsearch documents (`tenx-app`, `tenx-tierdown`) | 79,416, with line counts summing to 101,637 |
+
+**How the count is made.** Fluent Bit keeps every returned record whole in this run. `score-e103.py` matches each record's lines to the lines sent and counts the records whose lines came from more than one container. The 2026-10-24 count (71 of 848,190, and 11 of 78,852 in that day's smoke) treated each record as the run of seqs up to the next record's seq. That holds only while lines are grouped in arrival order. Here it reads 0, with 2 records whose runs do not match their line counts. The scorer's self-test groups the run's feed with one head rule both ways. In arrival order it finds 4,333 cross-container records by lines and the same 4,333 by the run-based count. Per container it finds none by lines, while the run-based count reports 4,322.
+
+In this run, kafka's four startup lines (seq 2751 to 2754) came back as four kafka records, one line each. The jaeger record at seq 2750 holds only its own line.
+
+**Rig additions.** `ARM_A_ONLY=1` in `phases-two-arm.sh` runs arm A alone: no cluster B, no copy to it, and every returned record written whole to `data/returned.jsonl`. `gen-receiver-values.mjs` passes `TENX_JAVA_HEAP` to the Receiver. `score-e103.py` makes the count above.

elastic-hot-tier-budget/results-e103-smoke/arm-b-rule.json

@@ -0,0 +1,232 @@
+{
+ "source": "results-two-arm/replay/source.jsonl",
+ "speed": 8.0,
+ "minutes": 20.0,
+ "source_seconds": 9600.0,
+ "lines": 67200,
+ "bytes": 13282731,
+ "top_half": [
+  "opentelemetry-collector"
+ ],
+ "top_half_bytes": 9149163,
+ "top_half_share": 0.6888,
+ "regex": "^(opentelemetry-collector)$",
+ "services": [
+  {
+   "service": "opentelemetry-collector",
+   "bytes": 9149163,
+   "lines": 32427,
+   "share": 0.6888
+  },
+  {
+   "service": "kafka",
+   "bytes": 1891441,
+   "lines": 10216,
+   "share": 0.1424
+  },
+  {
+   "service": "cart",
+   "bytes": 718696,
+   "lines": 12130,
+   "share": 0.0541
+  },
+  {
+   "service": "ad",
+   "bytes": 353362,
+   "lines": 2180,
+   "share": 0.0266
+  },
+  {
+   "service": "recommendation",
+   "bytes": 317242,
+   "lines": 1034,
+   "share": 0.0239
+  },
+  {
+   "service": "frontend",
+   "bytes": 292949,
+   "lines": 4934,
+   "share": 0.0221
+  },
+  {
+   "service": "grafana",
+   "bytes": 292610,
+   "lines": 1861,
+   "share": 0.022
+  },
+  {
+   "service": "fluentd-10x",
+   "bytes": 85930,
+   "lines": 641,
+   "share": 0.0065
+  },
+  {
+   "service": "opensearch",
+   "bytes": 49164,
+   "lines": 363,
+   "share": 0.0037
+  },
+  {
+   "service": "kube-proxy",
+   "bytes": 28157,
+   "lines": 245,
+   "share": 0.0021
+  },
+  {
+   "service": "frontend-proxy",
+   "bytes": 27512,
+   "lines": 121,
+   "share": 0.0021
+  },
+  {
+   "service": "jaeger",
+   "bytes": 17627,
+   "lines": 87,
+   "share": 0.0013
+  },
+  {
+   "service": "valkey-cart",
+   "bytes": 13212,
+   "lines": 192,
+   "share": 0.001
+  },
+  {
+   "service": "prometheus-server",
+   "bytes": 10711,
+   "lines": 67,
+   "share": 0.0008
+  },
+  {
+   "service": "metrics-server",
+   "bytes": 5148,
+   "lines": 35,
+   "share": 0.0004
+  },
+  {
+   "service": "coredns",
+   "bytes": 4099,
+   "lines": 49,
+   "share": 0.0003
+  },
+  {
+   "service": "accounting",
+   "bytes": 3653,
+   "lines": 76,
+   "share": 0.0003
+  },
+  {
+   "service": "postgresql",
+   "bytes": 3543,
+   "lines": 64,
+   "share": 0.0003
+  },
+  {
+   "service": "load-generator",
+   "bytes": 2640,
+   "lines": 45,
+   "share": 0.0002
+  },
+  {
+   "service": "wait-for-kafka",
+   "bytes": 2592,
+   "lines": 150,
+   "share": 0.0002
+  },
+  {
+   "service": "init-config",
+   "bytes": 2565,
+   "lines": 110,
+   "share": 0.0002
+  },
+  {
+   "service": "grafana-sc-dashboard",
+   "bytes": 1862,
+   "lines": 14,
+   "share": 0.0001
+  },
+  {
+   "service": "grafana-sc-alerts",
+   "bytes": 1327,
+   "lines": 10,
+   "share": 0.0001
+  },
+  {
+   "service": "flagd",
+   "bytes": 1322,
+   "lines": 16,
+   "share": 0.0001
+  },
+  {
+   "service": "grafana-sc-datasources",
+   "bytes": 1322,
+   "lines": 10,
+   "share": 0.0001
+  },
+  {
+   "service": "payment",
+   "bytes": 1104,
+   "lines": 34,
+   "share": 0.0001
+  },
+  {
+   "service": "email",
+   "bytes": 766,
+   "lines": 13,
+   "share": 0.0001
+  },
+  {
+   "service": "image-provider",
+   "bytes": 650,
+   "lines": 40,
+   "share": 0.0
+  },
+  {
+   "service": "aws-vpc-cni-init",
+   "bytes": 556,
+   "lines": 7,
+   "share": 0.0
+  },
+  {
+   "service": "fraud-detection",
+   "bytes": 543,
+   "lines": 6,
+   "share": 0.0
+  },
+  {
+   "service": "aws-node",
+   "bytes": 388,
+   "lines": 6,
+   "share": 0.0
+  },
+  {
+   "service": "wait-for-valkey-cart",
+   "bytes": 268,
+   "lines": 11,
+   "share": 0.0
+  },
+  {
+   "service": "currency",
+   "bytes": 265,
+   "lines": 1,
+   "share": 0.0
+  },
+  {
+   "service": "flagd-ui",
+   "bytes": 194,
+   "lines": 3,
+   "share": 0.0
+  },
+  {
+   "service": "aws-eks-nodeagent",
+   "bytes": 122,
+   "lines": 1,
+   "share": 0.0
+  },
+  {
+   "service": "quote",
+   "bytes": 26,
+   "lines": 1,
+   "share": 0.0
+  }
+ ]
+}

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/actions.csv

@@ -0,0 +1,2 @@
+container,action
+tenx-seed,pass::install seed

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/caps.csv

@@ -0,0 +1,2 @@
+container,cap
+tenx-seed,1::install seed

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/commit.json

@@ -0,0 +1 @@
+{"before":"78c5eea4ed9fd3c0b1fef9757f4904e9284207d1","after":"78c5eea4ed9fd3c0b1fef9757f4904e9284207d1","exit":0}

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/config-generation.csv

@@ -0,0 +1,2 @@
+key,value
+generation,seed

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/exit

@@ -0,0 +1 @@
+0

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/report.txt

@@ -0,0 +1,54 @@
+[tenx-recur] policy loaded: target=30% services=all lookback=24h min_delta=2pp [DRY RUN]
+[tenx-recur] Hot tier: 1 MB of 1000 GB (1000 GB set in the policy) on 1 hot node; at these limits it settles at about 58 MB within 2 hours, with 0 MB a day of counted lines moved to the frozen tier.
+[tenx-recur] Outside the Receiver: 0 MB on hot. The Receiver's kept lines: 1 MB in 1 indices; its tier_down lines not yet moved: 0 MB in 1.
+[tenx-recur] A kept line stays on hot 2 hours (ILM policy tenx-keep (rollover max_age 10m + frozen min_age 110m)); a tier_down index 2 hours (default (no tier_down index has been moved yet)).
+[tenx-recur] Each counted byte takes 0.88 bytes of store kept, 0.88 as tier_down (measured on the indices on hot).
+[tenx-recur] The Receiver's statements send 784 MB a day, under the 13589 GB a day that fits the size; every statement keeps its lines, with room to grow 2.0 times.
+[tenx-recur] The always-pass sample (5 lines a statement a window on every node, 50 more for an error statement over its limit) keeps about 597 MB a day on hot whatever the limits; the size holds down to that floor.
+# Hot-tier run: 2026-10-26T00:14:47.000Z
+
+Hot tier: 1 MB of 1000 GB (1000 GB set in the policy) on 1 hot node; at these limits it settles at about 58 MB within 2 hours, with 0 MB a day of counted lines moved to the frozen tier.
+
+- Outside the Receiver: 0 MB on hot. The Receiver's kept lines: 1 MB in 1 indices; its tier_down lines not yet moved: 0 MB in 1.
+- A kept line stays on hot 2 hours (ILM policy tenx-keep (rollover max_age 10m + frozen min_age 110m)); a tier_down index 2 hours (default (no tier_down index has been moved yet)).
+- Each counted byte takes 0.88 bytes of store kept, 0.88 as tier_down (measured on the indices on hot).
+- The Receiver's statements send 784 MB a day, under the 13589 GB a day that fits the size; every statement keeps its lines, with room to grow 2.0 times.
+- The always-pass sample (5 lines a statement a window on every node, 50 more for an error statement over its limit) keeps about 597 MB a day on hot whatever the limits; the size holds down to that floor.
+- Warning: 1 documents on hot carry no utf8Size, so their statements count as 0 bytes. The Receiver writes it when installed for the hot-tier run.
+
+| Service | Destination | Nodes | Sent a day | Kept a day | Limit per statement | Total |
+|---|---|---|---|---|---|---|
+| accounting | the frozen tier | 1 | 8 MB | 16 MB | 4359 | 43432 |
+| ad | the frozen tier | 1 | 10 MB | 20 MB | 25377 | 56529 |
+| aws-eks-nodeagent | the frozen tier | 1 | 0 MB | 0 MB | 832 | 832 |
+| aws-node | the frozen tier | 1 | 1 MB | 1 MB | 748 | 3948 |
+| aws-vpc-cni-init | the frozen tier | 1 | 1 MB | 2 MB | 761 | 5026 |
+| cart | the frozen tier | 1 | 19 MB | 37 MB | 63511 | 104041 |
+| coredns | the frozen tier | 1 | 6 MB | 11 MB | 12287 | 31658 |
+| currency | the frozen tier | 1 | 0 MB | 0 MB | 1186 | 1186 |
+| email | the frozen tier | 1 | 1 MB | 1 MB | 1705 | 3458 |
+| flagd | the frozen tier | 1 | 2 MB | 4 MB | 959 | 11159 |
+| flagd-ui | the frozen tier | 1 | 0 MB | 1 MB | 706 | 1974 |
+| fraud-detection | the frozen tier | 1 | 1 MB | 1 MB | 1386 | 2949 |
+| frontend | the frozen tier | 1 | 5 MB | 10 MB | 20593 | 27902 |
+| frontend-proxy | the frozen tier | 1 | 23 MB | 46 MB | 6613 | 129136 |
+| grafana | the frozen tier | 1 | 251 MB | 502 MB | 6064 | 1395753 |
+| grafana-sc-alerts | the frozen tier | 1 | 2 MB | 3 MB | 1056 | 8637 |
+| grafana-sc-dashboard | the frozen tier | 1 | 2 MB | 4 MB | 2150 | 12214 |
+| grafana-sc-datasources | the frozen tier | 1 | 2 MB | 3 MB | 1085 | 8756 |
+| image-provider | the frozen tier | 1 | 4 MB | 8 MB | 4554 | 21955 |
+| init-config | the frozen tier | 1 | 1 MB | 3 MB | 7256 | 7256 |
+| jaeger | the frozen tier | 1 | 14 MB | 27 MB | 4992 | 76313 |
+| kafka | the frozen tier | 1 | 125 MB | 251 MB | 59246 | 696146 |
+| kube-proxy | the frozen tier | 1 | 33 MB | 65 MB | 44101 | 180621 |
+| load-generator | the frozen tier | 1 | 2 MB | 4 MB | 9159 | 9994 |
+| metrics-server | the frozen tier | 1 | 6 MB | 11 MB | 4501 | 31281 |
+| opensearch | the frozen tier | 1 | 43 MB | 86 MB | 11605 | 237669 |
+| opentelemetry-collector | the frozen tier | 1 | 209 MB | 417 MB | 524270 | 1159227 |
+| wait-for-kafka | the frozen tier | 1 | 15 MB | 30 MB | 80530 | 82286 |
+| wait-for-valkey-cart | the frozen tier | 1 | 1 MB | 2 MB | 5798 | 6418 |
+| `*` (any other service) | the frozen tier | 1 | | 13588 GB | 37744131243 | 37744131243 |
+
+Limits and totals are bytes per 4-minute window per node, counted the way the Receiver counts them (utf8Size). Error lines pass 50 over a limit per statement per node per window.
+
+[tenx-recur] actions: 

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/run.json

@@ -0,0 +1 @@
+{"ts":"2026-10-26T00:14:51.625Z","status":"dry_run","projected_savings_pct":0,"delta_patterns":0,"delta_pp":0,"message":"dry-run: Hot tier: 1 MB of 1000 GB (1000 GB set in the policy) on 1 hot node; at these limits it settles at about 58 MB within 2 hours, with 0 MB a day of counted lines moved to the frozen tier."}

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001446Z/totals.csv

@@ -0,0 +1,2 @@
+container,total
+tenx-seed,1::install seed

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001452Z/actions.csv

@@ -0,0 +1,31 @@
+container,action
+accounting,tier_down::budget run
+ad,tier_down::budget run
+aws-eks-nodeagent,tier_down::budget run
+aws-node,tier_down::budget run
+aws-vpc-cni-init,tier_down::budget run
+cart,tier_down::budget run
+coredns,tier_down::budget run
+currency,tier_down::budget run
+email,tier_down::budget run
+flagd,tier_down::budget run
+flagd-ui,tier_down::budget run
+fraud-detection,tier_down::budget run
+frontend,tier_down::budget run
+frontend-proxy,tier_down::budget run
+grafana,tier_down::budget run
+grafana-sc-alerts,tier_down::budget run
+grafana-sc-dashboard,tier_down::budget run
+grafana-sc-datasources,tier_down::budget run
+image-provider,tier_down::budget run
+init-config,tier_down::budget run
+jaeger,tier_down::budget run
+kafka,tier_down::budget run
+kube-proxy,tier_down::budget run
+load-generator,tier_down::budget run
+metrics-server,tier_down::budget run
+opensearch,tier_down::budget run
+opentelemetry-collector,tier_down::budget run
+wait-for-kafka,tier_down::budget run
+wait-for-valkey-cart,tier_down::budget run
+*,tier_down::budget run

elastic-hot-tier-budget/results-e103-smoke/budget-runs/20261026T001452Z/caps.csv

@@ -0,0 +1,31 @@
+container,cap
+accounting,1::budget run
+ad,1::budget run
+aws-eks-nodeagent,1::budget run
+aws-node,1::budget run
+aws-vpc-cni-init,1::budget run
+cart,1::budget run
+coredns,1::budget run
+currency,1::budget run
+email,1::budget run
+flagd,1::budget run
+flagd-ui,1::budget run
+fraud-detection,1::budget run
+frontend,1::budget run
+frontend-proxy,1::budget run
+grafana,1::budget run
+grafana-sc-alerts,1::budget run
+grafana-sc-dashboard,1::budget run
+grafana-sc-datasources,1::budget run
+image-provider,1::budget run
+init-config,1::budget run
+jaeger,1::budget run
+kafka,1::budget run
+kube-proxy,1::budget run
+load-generator,1::budget run
+metrics-server,1::budget run
+opensearch,1::budget run
+opentelemetry-collector,1::budget run
+wait-for-kafka,1::budget run
+wait-for-valkey-cart,1::budget run
+*,1::budget run