diff --git a/README.md b/README.md index ad3396b..17d2881 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ rerun it and check every figure a post cites. | [`datadog-containment/`](datadog-containment/) | When a service floods its logs, does the released Receiver, installed with the values the MCP writes for Datadog, hold the flood before Datadog, including in a service with no rule, and what does it hold that it should not? The OpenTelemetry demo's own seeded failures on engine 1.1.142, and the Receiver's memory and CPU on a replay of the run's capture. | Current | | [`datadog-ceiling/`](datadog-ceiling/) | With a monthly budget X for Datadog Standard indexing, does the released Receiver, driven by the released MCP's hourly budget run, hold Standard at X while log volume grows 2 to 3x, as Datadog's own usage metric counts it? The OpenTelemetry demo into a real Datadog org on engine 1.1.146 and log10x-mcp 1.30.71, with a new service, an error storm and one service held to its own number. | Current. Not behind a post. | | [`elastic-hot-tier/`](elastic-hot-tier/) | With the Receiver's per-statement byte budget and log10x-mcp's Elasticsearch frozen-tier recipe, how long do the lines it moves stay on the hot tier, can they still be found by pattern hash on frozen, and how much of the volume does a fixed budget keep off hot as load grows, as the cluster's own APIs count it? Self-managed Elasticsearch 9.5.5 with hot and frozen nodes, engine 1.1.147, log10x-mcp 1.30.77, the published OpenTelemetry demo sample replayed for 11 hours. | Current. Not behind a post. | +| [`elastic-store-four-ways/`](elastic-store-four-ways/) | On Elasticsearch 9.5.5, how much index store does the engine's compact form save over a raw index, over logsdb, and over logsdb with `message` as `pattern_text` sorted on its template id, on the same sample? Primary store after a force merge to one segment, `best_compression` everywhere, trial licence, engine 1.1.150, the published OpenTelemetry demo sample, the compact form in standard, logsdb and template-sorted logsdb mode. Logsdb with `pattern_text` holds the sample in 34.7% of the raw index's store, the compact form on logsdb in 53.3% and sorted alike in 44.1%: on Elasticsearch 9.x the compact form is not the smaller store. | Current. Not behind a post. | | [`elastic-hot-tier-budget/`](elastic-hot-tier-budget/) | Does log10x-mcp's hourly hot-tier run, driving the Receiver, hold an Elasticsearch hot tier at a size S while load grows 2.5 times, as the hot node's own stats count it? Self-managed Elasticsearch 9.5.5 with hot and frozen nodes and the published OpenTelemetry demo sample replayed, in four runs, the last two beside a native reroute by service. With the tier-down index on hot the hold fails its bound of 1.10 S, peaking at 1.17 and 1.23 S in the two single-arm runs and at 1.43 S against the reroute's 1.36 S; with it on warm the hold keeps hot at 0.94 to 1.04 S from 20 minutes after the step, 0 of 25 readings over, while the reroute reaches 1.25 S and is still rising. | Current. Not behind a post. | | [`cloudwatch-ia-accounting/`](cloudwatch-ia-accounting/) | CloudWatch bills ingestion per GB, and an Infrequent Access log group ingests at half the Standard rate. When the 10x Receiver moves events to Infrequent Access or offloads them to S3 under the plan the 10x MCP writes, how many GB move, and what do they cost at AWS's list prices for us-east-1? Computed from the engine's counts at list price; no AWS account needed. | Current. Not behind a post. | | [`intent/`](intent/) | Does each pattern hold one logging statement? Every event of two OpenTelemetry-demo captures and a LogHub ZooKeeper capture is labelled with the source statement that wrote it, from the programs' own source, and the 10x engine (1.1.142, the current release, on all three; 1.1.132 and 1.1.133 on the demo captures; 1.1.136 on ZooKeeper) and Drain3 are scored against those labels: events merged with another statement, events split from their statement's main pattern, and exact grouping. | Current. Linked from [*How a Drain pattern group changes as more lines arrive*](https://www.log10x.com/blog/drain-pattern-groups-change/). | @@ -67,6 +68,7 @@ each resolves its paths relative to itself, so a benchmark can be run from anywh | `datadog-service-control/` | `./run.sh` | the same as `datadog-flex-accounting/` | `results/` | the same asset, and `rec1155.log.gz` from `log-10x/benchmarks` tag `intent-data-v1`, both checked by sha256 | | `datadog-containment/` | `rig/go.sh`; `rig/memory/run.sh` for the memory replay | engine and Fluent Bit images by digest, log10x-mcp from npm at a pinned version, OpenTelemetry demo 3.1.0 | `results/` | the demo's images, pulled by tag | | `elastic-hot-tier/` | `rig/make-replay.py`, then `rig/phases.sh` | Elasticsearch 9.5.5 and Fluent Bit 4.2 images by tag, the engine image by digest, a log10x-mcp checkout (the recipe and Receiver values are rendered from it) | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` | +| `elastic-store-four-ways/` | `./run.sh` | the engine and Elasticsearch 9.5.5 images by digest | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1`, checked by sha256 | | `cloudwatch-ia-accounting/` | `./run.sh` | the same as `datadog-flex-accounting/` | `results/` | the same asset | | `intent/` | `./run.sh` | engine image digests (1.1.132, 1.1.133, 1.1.136, 1.1.142) and drain3 0.9.11 from `pattern-identity/requirements.txt` | `results/` | the `otel-sample-200mb.log` asset from `log-10x/config` tag `otel-sample-v1` and `rec1155.log.gz` from this repository's `intent-data-v1` release, fetched into `cache/` on first run | | `naming-stability/` | `./run.sh` | engine image digests (1.1.140, 1.1.141, 1.1.142) | `results/` | the same two captures as `intent/`, checked by sha256 | diff --git a/elastic-store-four-ways/.gitignore b/elastic-store-four-ways/.gitignore new file mode 100644 index 0000000..8fce603 --- /dev/null +++ b/elastic-store-four-ways/.gitignore @@ -0,0 +1 @@ +data/ diff --git a/elastic-store-four-ways/METHOD.md b/elastic-store-four-ways/METHOD.md new file mode 100644 index 0000000..61f40a7 --- /dev/null +++ b/elastic-store-four-ways/METHOD.md @@ -0,0 +1,167 @@ +# Elasticsearch store, four ways: method (written before the run, 2026-10-25) + +One question: on Elasticsearch 9.5.5, how much index store does the engine's compact form save over +(a) a raw index, (b) logsdb index mode and (c) logsdb with `message` mapped as `pattern_text`, on +the same sample? The compact form is indexed twice, (d) in standard mode as on 8.17.0 and (e) in +logsdb mode, the 9.x default for logs, so the run has five variants. The comparison that decides is +(e) against (c): compact on logsdb against logsdb with `pattern_text`. + +The earlier figure this replaces as the Elastic comparison: on 8.17.0 the compact form (encoded +index plus template index, 13.8 MB) took 51.7% less store than a raw index holding each whole line +in one `message` text field (28.6 MB). That raw index had neither logsdb nor `pattern_text`; +`pattern_text`'s doc page marks it preview in 9.2 and generally available since 9.3. + +## Sample + +`otel-sample-200mb.log` from release `otel-sample-v1` of `log-10x/config` (byte-identical to the raw +asset of `otel-sample-v2`): 215,039,161 bytes, 197,430 lines, sha256 +`aa79b9349a4123d88936fd064f5eccd9fd84f45a1fd03cff3f64c259715c2432`. Container output of the +OpenTelemetry demo as Fluentd ships it: every line is a JSON record with the message under `log` +and `stream`, `docker.container_id`, `kubernetes.*` (16 label keys, one host) and `tenx_tag` +around it. The records carry no timestamp field. + +## The compact form + +Engine `ghcr.io/log-10x/edge-10x:1.1.150` by digest, in file mode, with the published round-trip +configs `../drain3-vs-log10x/tenx-encode.config.yaml` and `tenx-decode.config.yaml` unchanged +(the configs `otel-denominators/` and `loghub-compaction/` measure the compact form with). The +encode reads each whole line and writes `encoded.log` (one encoded event per line; multi-line +events become one) and `templates.json` (one template per line). Run twice; both outputs are +hashed and the two runs compared. The template dictionary followed by the encoded events is decoded +and compared with the sample byte for byte. The first run's output is what is indexed. + +## Elasticsearch + +One node, `docker.elastic.co/elasticsearch/elasticsearch:9.5.5` by digest, `discovery.type: +single-node`, security and ML off, heap 1 GB, container limit 2 GB, the disk allocation threshold +off (the measurement is not about watermarks). Before the trial, a logsdb index with a +`pattern_text` field is created on the default licence and the response recorded. Then +`POST /_license/start_trial?acknowledge=true`; the run stops unless `GET /_license` reads type +`trial`, status `active`. + +Elastic's docs, read 2026-10-25: + +- `pattern_text` (https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/pattern-text): + "Elastic Stack: Generally available since 9.3"; "This feature requires a [subscription]"; + "The `pattern_text` field type is a variant of `text` with improved space efficiency for log + data."; "The compression provided by `pattern_text` can be significantly improved if the index + is sorted by the `template_id` field."; "This index sorting is not applied by default, but can + be enabled for the `message` field of LogsDB indices (assuming it is of type `pattern_text`) by + setting the index setting `index.logsdb.default_sort_on_message_template` to `true`. This will + cause the index to be sorted by `host.name` (if present), then `message.template_id`, and + finally by `@timestamp`."; the mapping in the doc's example is `"notice": { "type": + "pattern_text" }`; `index_options` "Defaults to `docs`". +- Subscriptions (https://www.elastic.co/subscriptions): "Pattern text field type for logsdb + indices" and "Synthetic _source" are ticked under Enterprise only. +- Start trial API: "Start a 30-day trial, which gives access to all subscription features." +- Logsdb (https://www.elastic.co/docs/manage-data/data-store/data-streams/logs-data-stream-configure): + "If you have the required subscription, logsdb index mode uses synthetic `_source`"; "In logsdb + index mode, indices are sorted by the fields `host.name` and `@timestamp` by default"; "By + default, logsdb index mode uses the `best_compression` codec". + +## The five variants + +Every index: one primary shard, no replica, `index.codec: best_compression` set explicitly, +refresh off while loading, documents sent in file order in bulk requests of 2,000 with +generated `_id`s, from one client. + +(a), (b) and (c) index the same document per line: the parsed record as captured, with `log` +renamed `message`, plus `@timestamp` (line i at 2025-10-02T00:00:00Z + i/7 s, the pace the Elastic +rigs replay this capture at). All three map every other string as `keyword` (`ignore_above: 1024`) +by one dynamic template. (a) is therefore not the 8.17.0 raw index, which held each whole line as one +`message` text field; it is the record a shipper sends, parsed into fields, in standard mode. Each +step changes one thing: + +| | Index | Index settings beyond the common ones | `message` | +|---|---|---|---| +| (a) raw | `es4-raw` | none (standard mode) | `text` | +| (b) logsdb | `es4-logsdb` | `index.mode: logsdb` | `text` | +| (c) logsdb + `pattern_text` | `es4-pattern` | `index.mode: logsdb`, `index.logsdb.default_sort_on_message_template: true` | `pattern_text`, defaults | +| (d) compact | `es4-compact` + `es4-compact-dict` | none (standard mode) | the encoded event, `text` | +| (e) compact on logsdb | `es4-compact-logsdb` + `es4-compact-dict` | `index.mode: logsdb` | the encoded event, `text` | + +(d) is the shape the compact form was measured in on 8.17.0 and that the l1es plugin documents: +`es4-compact` holds one document per encoded event, `{"@timestamp", "message": , +"l1x_tid":