e103-grouping: cross-container grouping on 1.1.150 and on the E103 fix

e103-grouping/README.md

@@ -0,0 +1,100 @@
+# E103: multi-line grouping across containers
+
+A minimal reproduction of engine ask E103 on the shipped engine image. In the `elastic-hot-tier-budget` run, 71 of 848,190 returned records held lines from more than one container: kafka's four startup lines (seq 2751 to 2754) joined the jaeger record at seq 2750 and were filed under jaeger. This rig sends the same record shape over the same path and counts such records.
+
+## What runs
+
+The path is the hot-tier rig's, with Elasticsearch dropped. A sender writes Fluent Forward records to Fluent Bit on 24224. Fluent Bit moves the container into a `kubernetes` map (`fluent-bit/prep.lua`, the rig's `prep`), copies every record to `in.jsonl`, and forwards to the Receiver on `tenx:24224` with one worker and one connection. The Receiver returns records to Fluent Bit's second forward input on 24225, which writes them to `returned.jsonl`.
+
+The Receiver is started the way `gen-receiver-values.mjs` rendered it for the two-arm runs (`elastic-hot-tier-budget/results-two-arm*/values/compose.receiver.yaml`):
+
+- image `ghcr.io/log-10x/edge-10x:1.1.150@sha256:3e755bbc14780a27e643e09b76dfd7ec9616cab67833f70daf9420eff646d350`
+- arguments `@run/input/forwarder/fluentbit @apps/receiver symbolMessageHashField tenx_hash outputEncodeField fullText("tenx_hash","routeState","message_pattern","severity_level","utf8Size")`
+- environment `TENX_AIRGAPPED=true` and `TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit`
+- 0.5 CPU, 512 MB, tmpfs on `/tmp`
+
+The rendered `GH_*` variables (the policy branch) are left out, so no cap is set and nothing is held. The image keeps its config at `/etc/tenx/config` (`TENX_CONFIG`) and its modules at `/opt/tenx-edge/lib/app/modules` (`TENX_MODULES`). `E103_CONFIG_DIR` and `E103_MODULES_DIR` mount host directories read-only over them, so a candidate fix can be tested on the same image. `run.json` records the image ID and each directory's git commit.
+
+In the shipped modules, the Fluent Bit input takes the event's source from the Forward tag (`pipelines/run/modules/input/forwarder/fluentbit/stream.yaml`, `sourcePattern: \"tag\":\"(.*?)\"`). The group step's flush timeout counts per source (`pipelines/run/units/transform/group/unit.yaml`). The hot-tier sender gave every container the tag `app.replay`.
+
+## Cases
+
+`send.py` uses `pack` and `Sender` from `elastic-hot-tier-budget/rig/forward_send.py`. Every record carries `container_name`, `container_id`, `source`, `log` and `seq`. Every line ends with the token `e103id=<seq>`, which sits inside the JSON on the jaeger line. Each case has its own container names, and cases are 10 s apart. Cases a to d send every container under the tag `app.replay`, the way the hot-tier sender did.
+
+- **a.** orders logs an ERROR head. Then billing's timestamped INFO lines (1 to 3) arrive, and then orders' stack trace (3 to 8 lines of `\tat ...`, `Caused by: ...` and `\t... N more`).
+- **b.** jaeger logs its JSON line. kafka's four startup lines from the hot-tier run follow (no timestamp, no level), then kafka's own timestamped INFO line.
+- **c.** orders and payments are both partway through a stack trace. orders' head and two frames go out, then payments' head, then the remaining frames of both, alternating.
+- **d.** Control with one container. orders sends a head and its stack trace, then 0 to 2 INFO lines.
+- **e.** Cases a, b and c in rotation, with one tag per container (`docker.<id12>`, the fluentd driver's default tag).
+
+Each case sends 500 episodes back to back. Lines within an episode go out with no sleep between them, and episodes are 1 ms apart. Twelve paused episodes are spread through each case, with pauses of 1, 2, 3, 4, 5 and 6 s, twice each. The pause comes right after the first container's head, and in case c after orders' second frame. The full run sends 22,557 lines in 6,461 events. `groupFlushTimeout` is 5 s, and on a forwarder input a batch reaches the group step up to about 2 s after its line.
+
+## Scoring
+
+`score.py <results>/data` splits each returned record's `log` on `"\n"` and matches every line to the line sent by its token. It reports the following, per case and in total:
+
+- records returned and lines returned
+- cross-container records: records whose lines came from more than one sender container
+- lines missing and lines duplicated
+- stack-trace events returned whole: the head and every continuation line in one record that holds nothing else
+- cross-container records by the pause in their episode
+- records returned under the wrong tag: Fluent Bit stores the tag a record came back under as `_fb_tag`, and the scorer checks it equals `tenx.` followed by the tag its first line was sent under
+
+It exits 1 when there is any cross-container record, missing line, duplicated line or wrong tag, and 0 otherwise.
+
+`score.py --self-test` builds three inputs. A clean input must pass. The README's shape (a jaeger record holding kafka's banner lines) must fail, and so must a record returned under the wrong tag. All three verdicts came out as expected. On the full manifest, a simulated grouping in arrival order per tag gives 1,536 cross-container records and exit 1. A simulated grouping per container gives 0 and exit 0.
+
+## Running
+
+Docker must have no other rig running: `run.sh up` refuses to start when any container is up other than `e103-*` and `census-ch`. The rig publishes no host port. A run takes about 5 minutes.
+
+```
+./run.sh all                       # 1.1.150 as shipped, results in results/shipped-1.1.150/
+./run.sh extract /path/to/e103-fix # the image's own config and modules, to patch
+E103_CONFIG_DIR=/path/to/e103-fix/config E103_MODULES_DIR=/path/to/e103-fix/modules E103_LABEL=fix ./run.sh all
+E103_IMAGE=local/edge-10x:e103 E103_JAVA_HEAP=320M E103_LABEL=fix-1.1.151 ./run.sh all   # another image
+```
+
+`E103_JAVA_HEAP` sets `TENX_JAVA_HEAP` for an image that runs the engine jar on a JVM, so the heap fits the 512 MB limit.
+
+`run.sh all` starts the containers with `docker run -d --name e103-*`, sends, waits for the returned records, saves logs and Fluent Bit's counters, scores, gzips the data and removes the containers and the network (`docker rm -f -v`).
+
+## Result on 1.1.150
+
+Run 2026-10-25 from 22:50 to 22:55 UTC. Fluent Bit did not yet store the returned tag, so this run has no tag check. Results are in `results/shipped-1.1.150/`: `run.json` holds the image ID and launch, and `data/` holds the lines sent, the records Fluent Bit received and returned, and `score.json`.
+
+**Verdict: FAIL, with 1,533 cross-container records.** Fluent Bit took in 22,557 records, forwarded 22,557 to the Receiver and wrote 5,836 returned records. One of those is the Receiver's own connect message.
+
+| Case | Lines sent | Records | Lines returned | Cross-container records | Lines missing | Lines duplicated | Stack traces whole |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| a | 4,428 | 1,554 | 4,428 | 512 | 0 | 0 | 0 of 512 |
+| b | 3,072 | 1,036 | 3,072 | 509 | 0 | 0 | none sent |
+| c | 6,602 | 1,024 | 6,602 | 512 | 0 | 0 | 0 of 1,024 |
+| d | 3,781 | 1,027 | 3,781 | 0 | 0 | 0 | 509 of 512 |
+| e | 4,674 | 1,194 | 4,674 | 0 | 0 | 0 | 508 of 511 |
+| Total | 22,557 | 5,836 | 22,557 | 1,533 | 0 | 0 | |
+
+In cases a and c, every episode produced one cross-container record, paused or not. In case b, 509 of 512 episodes did. The three that did not were the two 6 s pauses and one 5 s pause, where the jaeger record was sealed before kafka's lines arrived. Case e sends the same three patterns with one tag per container and returns no cross-container record, so the cross-container records follow the shared tag.
+
+**Example from case b, the hot-tier shape.** The record with seq 4429 is filed under b-jaeger. It holds seq 4429 (b-jaeger, the JSON line) and seq 4430 to 4433 (b-kafka: `===> User`, `uid=1000(appuser) ...`, `===> Setting default values ...` and `===> Configuring ...`). All five lines were sent under the tag `app.replay`.
+
+Six stack traces in cases d and e came back split, three in each case. All six were in episodes paused 5 or 6 s after the head, and each came back as 4 to 7 records.
+
+## Result with the E103 fix
+
+Run 2026-10-26 from 00:00 to 00:04 UTC. Results are in `results/fix2-1.1.151/`.
+
+The fix is engine commit `0cd458f0f133daab281bfc9d067e38c1410ce050` (log-10x/engine, branch `fix/e103-sequence-key`: "Key a line's sequence values in any field order"). It ships with modules commit `6c69d90a015adaffc2163e927682d5f6304e504d` (log-10x/modules, branch `fix/e103-sequence-key`: "Sequence patterns capture the field name with its value"). The image `local/edge-10x:e103` (`sha256:8f74b8bbec2ef115fb83eb365172872afa5aec2cea4ae5fd065b49c1d77fe361`) is the 1.1.150 image with the engine jar from that commit and those modules at `/opt/tenx-edge/lib/app/modules`. Its modules match the modules commit file for file. The jar runs on a JVM, so the Receiver had `TENX_JAVA_HEAP=320M`. The arguments, other environment and limits are the same as the 1.1.150 run.
+
+**Verdict: PASS.** Fluent Bit took in 22,557 records, forwarded 22,557 and wrote 5,869 returned records, one of them the Receiver's connect message. All 5,868 records holding sent lines came back under `tenx.` followed by their sent tag: 4,674 under `tenx.app.replay` and 1,194 under `tenx.docker.<id12>`.
+
+| Case | Lines sent | Records (1.1.150) | Records (fix) | Lines returned | Cross-container records | Lines missing | Lines duplicated | Wrong tag | Stack traces whole (1.1.150) | Stack traces whole (fix) |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| a | 4,428 | 1,554 | 1,570 | 4,428 | 0 | 0 | 0 | 0 | 0 of 512 | 509 of 512 |
+| b | 3,072 | 1,036 | 1,040 | 3,072 | 0 | 0 | 0 | 0 | none sent | none sent |
+| c | 6,602 | 1,024 | 1,037 | 6,602 | 0 | 0 | 0 | 0 | 0 of 1,024 | 1,021 of 1,024 |
+| d | 3,781 | 1,027 | 1,027 | 3,781 | 0 | 0 | 0 | 0 | 509 of 512 | 509 of 512 |
+| e | 4,674 | 1,194 | 1,194 | 4,674 | 0 | 0 | 0 | 0 | 508 of 511 | 508 of 511 |
+| Total | 22,557 | 5,836 | 5,869 | 22,557 | 0 | 0 | 0 | 0 | 1,017 | 2,547 |
+
+Every stack trace returned whole on 1.1.150 is returned whole with the fix. All twelve traces that came back split, three in each of cases a, c, d and e, were in episodes paused 5 or 6 s. Cases d and e split the same six on both runs. No returned record holds stack trace lines from more than one episode.

e103-grouping/fluent-bit/fluent-bit.conf

@@ -0,0 +1,71 @@
+# E103 grouping rig: the hot-tier rig's Fluent Bit path with Elasticsearch dropped.
+#   sender (Forward, 24224) -> prep -> [in.jsonl] -> Receiver (tenx:24224, one worker, one connection)
+#   Receiver (24225, Tag_Prefix tenx.) -> _fb_tag (the tag it came back under) -> [returned.jsonl]
+# The forward output block is the hot-tier rig's, setting for setting.
+
+[SERVICE]
+    Flush        1
+    Log_Level    info
+    Grace        10
+    HTTP_Server  On
+    HTTP_Listen  0.0.0.0
+    HTTP_Port    2020
+
+[INPUT]
+    Name        forward
+    Listen      0.0.0.0
+    Port        24224
+    Buffer_Max_Size   16M
+    Buffer_Chunk_Size 1M
+
+[FILTER]
+    Name    lua
+    Match   app.*
+    script  /fluent-bit/etc/rig/prep.lua
+    call    prep
+
+[FILTER]
+    Name    lua
+    Match   docker.*
+    script  /fluent-bit/etc/rig/prep.lua
+    call    prep
+
+[OUTPUT]
+    Name    file
+    Match_Regex ^(app|docker)\.
+    Path    /data
+    File    in.jsonl
+    Format  plain
+
+[OUTPUT]
+    Name                        forward
+    Match_Regex                 ^(app|docker)\.
+    Host                        tenx
+    Port                        24224
+    Retry_Limit                 False
+    Workers                     1
+    net.max_worker_connections  1
+    net.keepalive               on
+    net.keepalive_max_recycle   100000000
+    net.keepalive_idle_timeout  300
+
+[INPUT]
+    Name        forward
+    Listen      0.0.0.0
+    Port        24225
+    Tag_Prefix  tenx.
+    Buffer_Max_Size   16M
+    Buffer_Chunk_Size 1M
+
+[FILTER]
+    Name    lua
+    Match   tenx.*
+    script  /fluent-bit/etc/rig/prep.lua
+    call    return_tag
+
+[OUTPUT]
+    Name    file
+    Match   tenx.*
+    Path    /data
+    File    returned.jsonl
+    Format  plain

e103-grouping/fluent-bit/prep.lua

@@ -0,0 +1,19 @@
+-- prep from elastic-hot-tier-budget/rig/fluent-bit/rig.lua, unchanged: the container moves into a
+-- Kubernetes-style `kubernetes` map (container_name, docker_id); fb_ts is the event time.
+function prep(tag, ts, rec)
+  local c = rec["container_name"]
+  if c then c = (string.gsub(c, "^/", "")) end
+  local k = { container_name = c }
+  if rec["container_id"] then k["docker_id"] = rec["container_id"] end
+  rec["container_name"] = nil
+  rec["container_id"] = nil
+  rec["kubernetes"] = k
+  rec["fb_ts"] = string.format("%.3f", ts)
+  return 2, ts, rec
+end
+
+-- returned records: the Forward tag they came back under (the Receiver's tag with the tenx. prefix)
+function return_tag(tag, ts, rec)
+  rec["_fb_tag"] = tag
+  return 2, ts, rec
+end

e103-grouping/results/fix2-1.1.151/data/score.json

@@ -0,0 +1,55 @@
+{
+ "verdict": "PASS",
+ "records_in_returned_file": 5869,
+ "fluent_bit_in_records": 22557,
+ "cases": {
+  "a": {
+   "records": 1570,
+   "lines_returned": 4428,
+   "lines_sent": 4428,
+   "trace_events": 512,
+   "trace_events_whole": 509,
+   "cross_container_by_pause_s": {}
+  },
+  "b": {
+   "records": 1040,
+   "lines_returned": 3072,
+   "lines_sent": 3072,
+   "cross_container_by_pause_s": {}
+  },
+  "c": {
+   "records": 1037,
+   "lines_returned": 6602,
+   "lines_sent": 6602,
+   "trace_events": 1024,
+   "trace_events_whole": 1021,
+   "cross_container_by_pause_s": {}
+  },
+  "d": {
+   "records": 1027,
+   "lines_returned": 3781,
+   "lines_sent": 3781,
+   "trace_events": 512,
+   "trace_events_whole": 509,
+   "cross_container_by_pause_s": {}
+  },
+  "e": {
+   "records": 1194,
+   "lines_returned": 4674,
+   "lines_sent": 4674,
+   "trace_events": 511,
+   "trace_events_whole": 508,
+   "cross_container_by_pause_s": {}
+  }
+ },
+ "total": {
+  "records": 5869,
+  "lines_returned": 22557,
+  "lines_sent": 22557,
+  "trace_events": 2559,
+  "trace_events_whole": 2547,
+  "records_without_log": 1,
+  "tag_checked": 5868
+ },
+ "cross_container_examples": []
+}

e103-grouping/results/fix2-1.1.151/fluent-bit-metrics.json

@@ -0,0 +1 @@
+{"input":{"forward.0":{"records":22557,"bytes":5724518},"forward.1":{"records":5869,"bytes":4257869}},"filter":{"lua.0":{"drop_records":0,"add_records":0,"records":17883,"bytes":4987203,"drop_bytes":0},"lua.1":{"drop_records":0,"add_records":0,"records":4674,"bytes":1301240,"drop_bytes":0},"lua.2":{"drop_records":0,"add_records":0,"records":5869,"bytes":4468167,"drop_bytes":0}},"output":{"file.0":{"proc_records":22557,"proc_bytes":6288443,"errors":0,"retries":0,"retries_failed":0,"dropped_records":0,"retried_records":0},"forward.1":{"proc_records":22557,"proc_bytes":6288443,"errors":0,"retries":0,"retries_failed":0,"dropped_records":0,"retried_records":0},"file.2":{"proc_records":5869,"proc_bytes":4468167,"errors":0,"retries":0,"retries_failed":0,"dropped_records":0,"retried_records":0}}}

e103-grouping/results/fix2-1.1.151/fluent-bit.log

@@ -0,0 +1,32 @@
+Fluent Bit v4.2.8
+* Copyright (C) 2015-2026 The Fluent Bit Authors
+* Fluent Bit is a CNCF graduated project under the Fluent organization
+* https://fluentbit.io
+
+______ _                  _    ______ _ _             ___   _____ 
+|  ___| |                | |   | ___ (_) |           /   | / __  \
+| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| | `' / /'
+|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| |   / /  
+| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |_./ /___
+\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)_____/
+
+             Fluent Bit v4.2 – Direct Routes Ahead
+         Celebrating 10 Years of Open, Fluent Innovation!
+
+[2026/10/25 23:59:45.279046242] [ info] [fluent bit] version=4.2.8, commit=419540c428, pid=1
+[2026/10/25 23:59:45.279256912] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128
+[2026/10/25 23:59:45.279274178] [ info] [simd    ] SSE2
+[2026/10/25 23:59:45.279278183] [ info] [cmetrics] version=1.0.6
+[2026/10/25 23:59:45.279281967] [ info] [ctraces ] version=0.6.6
+[2026/10/25 23:59:45.279355303] [ info] [input:forward:forward.0] initializing
+[2026/10/25 23:59:45.279363817] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
+[2026/10/25 23:59:45.279490244] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
+[2026/10/25 23:59:45.279532687] [ info] [input:forward:forward.1] initializing
+[2026/10/25 23:59:45.279537698] [ info] [input:forward:forward.1] storage_strategy='memory' (memory only)
+[2026/10/25 23:59:45.279569595] [ info] [input:forward:forward.1] listening on 0.0.0.0:24225
+[2026/10/25 23:59:45.309599825] [ info] [output:forward:forward.1] worker #0 started
+[2026/10/25 23:59:45.309601028] [ info] [output:file:file.2] worker #0 started
+[2026/10/25 23:59:45.309632131] [ info] [output:file:file.0] worker #0 started
+[2026/10/25 23:59:45.311057193] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
+[2026/10/25 23:59:45.311101039] [ info] [sp] stream processor started
+[2026/10/25 23:59:45.311140443] [ info] [engine] Shutdown Grace Period=10, Shutdown Input Grace Period=5

e103-grouping/results/fix2-1.1.151/run.json

@@ -0,0 +1,26 @@
+{
+ "started_utc": "2026-10-26T00:00:14Z",
+ "image": "local/edge-10x:e103",
+ "image_id": "sha256:8f74b8bbec2ef115fb83eb365172872afa5aec2cea4ae5fd065b49c1d77fe361",
+ "args": [
+  "@run/input/forwarder/fluentbit",
+  "@apps/receiver",
+  "symbolMessageHashField",
+  "tenx_hash",
+  "outputEncodeField",
+  "fullText(\"tenx_hash\",\"routeState\",\"message_pattern\",\"severity_level\",\"utf8Size\")"
+ ],
+ "env": [
+  "TENX_AIRGAPPED=true",
+  "TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit",
+  "TENX_JAVA_HEAP=320M"
+ ],
+ "cpus": 0.5,
+ "memory": "512m",
+ "config_dir": null,
+ "config_commit": null,
+ "modules_dir": null,
+ "modules_commit": null,
+ "fluent_bit_image": "fluent/fluent-bit:4.2",
+ "fluent_bit_image_id": "sha256:6806769c10acf11b41d14083b3101a24433b4ca113a1ab7a6b1a1b7cad86a7e8"
+}

e103-grouping/results/fix2-1.1.151/score.md

@@ -0,0 +1,9 @@
+| case | lines_sent | records | lines_returned | cross_container_records | lines_in_cross_container_records | lines_missing | lines_duplicated | records_wrong_tag | trace_events | trace_events_whole | cross by pause (s) |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| a | 4428 | 1570 | 4428 | 0 | 0 | 0 | 0 | 0 | 512 | 509 | {} |
+| b | 3072 | 1040 | 3072 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | {} |
+| c | 6602 | 1037 | 6602 | 0 | 0 | 0 | 0 | 0 | 1024 | 1021 | {} |
+| d | 3781 | 1027 | 3781 | 0 | 0 | 0 | 0 | 0 | 512 | 509 | {} |
+| e | 4674 | 1194 | 4674 | 0 | 0 | 0 | 0 | 0 | 511 | 508 | {} |
+| total | 22557 | 5869 | 22557 | 0 | 0 | 0 | 0 | 0 | 2559 | 2547 | |
+{"verdict": "PASS", "records_in_returned_file": 5869, "fluent_bit_in_records": 22557, "records_without_sent_line": 0, "records_without_log": 1, "nonblank_lines_without_token": 0, "tag_checked": 5868}

e103-grouping/results/fix2-1.1.151/send.log

@@ -0,0 +1,6 @@
+{"case": "a", "episodes": 512, "lines": 4428, "seconds": 44.0}
+{"case": "b", "episodes": 512, "lines": 3072, "seconds": 43.6}
+{"case": "c", "episodes": 512, "lines": 6602, "seconds": 44.2}
+{"case": "d", "episodes": 512, "lines": 3781, "seconds": 43.8}
+{"case": "e", "episodes": 512, "lines": 4674, "seconds": 43.9}
+{"final_seq": 22557}