elastic-hot-tier-budget: install proof from the tools alone (recipe, advise_install elasticsearch, setup_recurring hot_tier_gb)

elastic-hot-tier-budget/install-e2e/.gitignore

@@ -0,0 +1,3 @@
+results/data/
+results/replay/
+results/policy-git/

elastic-hot-tier-budget/install-e2e/README.md

@@ -0,0 +1,30 @@
+# Install proof: the hot-tier run from the tools alone
+
+Does `log10x_setup_recurring hot_tier_gb` run against a cluster set up only by the frozen-tier recipe from `log10x_advise_retriever` and a Receiver installed only from `log10x_advise_install` with `log_destination: "elasticsearch"`, with no hand-set engine option and no hand-added ILM phase?
+
+**Result: yes.** Four `tenx-recur` runs, all exit 0 at their reads: a dry run and an applied run with only kept lines on hot, then, once the applied limits sent lines to `tenx-tierdown`, a dry run and an applied run with documents in both aliases. Each run read the kept lines' hours on hot from the recipe's own policy: `A kept line stays on hot 168 hours (ILM policy tenx-keep (rollover max_age 1d + frozen min_age 6d))`.
+
+log10x-mcp 1.30.90, engine image `ghcr.io/log-10x/edge-10x:1.1.150` (the values render `log10x/edge-10x:1.1.147@sha256:a043ccd828742c4eccee09abb654ce6a7357b6123f73300ed04f8605e874b8c4`; `launch.json` records both), Elasticsearch 9.5.5, one hot node and one warm and frozen node, 2026-10-25 23:46 to 23:53 UTC. No time scaling.
+
+## What ran
+
+`MCP=<log10x-mcp build> ./run.sh <step>`, in this order:
+
+1. `render 0.002`: `render.mjs` calls the three tools' handlers from the build. `advise_retriever` with `destination: "elasticsearch"`: the frozen-tier block is `results/recipe.txt`, its six `PUT` requests with the comment lines dropped are `results/recipe-requests.json`. `advise_install` with app `receiver`, forwarder `fluentbit`, `log_destination: "elasticsearch"`, builtin licence, airgapped, metrics backend `elastic`: the plan is `results/install-plan.md`, the sidecar and the policy seed are `results/receiver.json`. `setup_recurring` with `hot_tier_gb: 0.002` on `github_actions`, every replay container in `nodes_by_container`: `results/policy.yaml` and `results/apply-instructions.md`. The size is 2 MB so that the run holds lines on the first pass.
+2. `policy-init`: a local bare repo stands in for the policy repo: `policy.yaml` on `main`, the seed files from advise_install's ConfigMap (`caps.csv`, `actions.csv`, `totals.csv`, `regulator-unit: bytes`) under `log10x/policy/` on `log10x-policy`.
+3. `up`: the two Elasticsearch nodes, then `setup-es.mjs`: the trial licence and a filesystem repository registered as `found-snapshots` (cluster preparation, so that the recipe applies as rendered), then the six requests, all 200; `tenx-tierdown-000001` on the warm node, `tenx-app-000001` on hot. Then Fluent Bit (`../rig/fluent-bit`, which sends `tier_down` lines to `tenx-tierdown` and the rest to `tenx-app`) and the Receiver from `results/compose.receiver.yaml`: the args exactly as the values render them, including `symbolMessageHashField tenx_hash` and `outputEncodeField fullText("tenx_hash","routeState","message_pattern","severity_level","utf8Size")`.
+4. `feed start 8`: the OTel demo sample at 56 lines a second (`../rig/forward_send.py`, the first 12,000 rows of the replay source).
+5. `tick --dry-run`, then `tick`: exit 0 both; the applied run committed the four files to `log10x-policy`.
+6. `sync`: the committed `caps.csv`, `actions.csv` and `totals.csv` copied into the Receiver's policy mount, Receiver restarted.
+7. About three minutes later, `tenx-app` held 7,440 documents and `tenx-tierdown` 2,340, every `tier_down` document with `utf8Size`. `tick --dry-run`, then `tick`: exit 0 both.
+8. `feed stop`, `down`: every e100 container, volume and network removed.
+
+Summary line of the last run (`results/ticks/20261025T235320Z/run.json`): `applied: Hot tier: 4 MB of 2 MB (0.002 GB set in the policy) on 1 hot node; at these limits it settles at about 4.6 GB within 168 hours, with 442 MB a day of counted lines moved to the frozen tier.` It settles above the size because the always-pass sample alone keeps about 592 MB a day, which the report states.
+
+## Rig substitutions (no cluster)
+
+The values read the policy files through the Kubernetes pull lane (`K8S_ENABLED`, `K8S_CONFIGMAP`, `K8S_NAMESPACE`); here `K8S_ENABLED=false` and the three lookup files are mounted at `/policy`. `CONFIG_GENERATION_FILE` is left out, since the seed has no `config-generation.csv`. `TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit`, because Fluent Bit is a separate container. The applied files reach the Receiver by `sync` and a restart, in place of the ConfigMap or git pull. No engine option was added or changed. `launch.json` lists each substitution.
+
+## Not shown
+
+Kept lines reaching frozen after 6 days; a tier_down mount; the GitHub Actions workflow or a Kubernetes CronJob; the git pull lane (`policy_git`); an Elastic Cloud deployment.

elastic-hot-tier-budget/install-e2e/compose.yaml

@@ -0,0 +1,74 @@
+# E100/E101 install proof: one hot node and one warm+frozen node (Elasticsearch 9.5.5, the
+# same images, heaps and limits as ../rig/compose.yaml with ../rig/compose.warm.yaml), Fluent Bit
+# with ../rig/fluent-bit, and the Receiver from compose.receiver.yaml (written by render.mjs from
+# log10x_advise_install's elasticsearch values). Compose project e100, containers e100-*.
+#   docker compose -p e100 -f compose.yaml -f results/compose.receiver.yaml ...
+
+volumes:
+  es-hot-data: {}
+  es-wf-data: {}
+  es-snapshots: {}
+
+x-es-common: &es-common
+  image: docker.elastic.co/elasticsearch/elasticsearch:9.5.5
+  ulimits:
+    memlock: { soft: -1, hard: -1 }
+    nofile: { soft: 65536, hard: 65536 }
+
+services:
+
+  es-hot:
+    <<: *es-common
+    container_name: e100-es-hot
+    environment:
+      - node.name=es-hot
+      - cluster.name=e100
+      - node.roles=master,data_hot,data_content,ingest
+      - cluster.initial_master_nodes=es-hot
+      - discovery.seed_hosts=es-hot
+      - path.repo=/snapshots
+      - xpack.security.enabled=false
+      - xpack.ml.enabled=false
+      - ES_JAVA_OPTS=-Xms1g -Xmx1g
+    volumes:
+      - es-hot-data:/usr/share/elasticsearch/data
+      - es-snapshots:/snapshots
+    ports:
+      - "127.0.0.1:9200:9200"
+    deploy:
+      resources:
+        limits:
+          memory: 2G
+
+  es-warm-frozen:
+    <<: *es-common
+    container_name: e100-es-warm-frozen
+    environment:
+      - node.name=es-warm-frozen
+      - cluster.name=e100
+      - node.roles=data_warm,data_frozen
+      - discovery.seed_hosts=es-hot
+      - path.repo=/snapshots
+      - xpack.security.enabled=false
+      - xpack.ml.enabled=false
+      - xpack.searchable.snapshot.shared_cache.size=512mb
+      - ES_JAVA_OPTS=-Xms512m -Xmx512m
+    volumes:
+      - es-wf-data:/usr/share/elasticsearch/data
+      - es-snapshots:/snapshots
+    deploy:
+      resources:
+        limits:
+          memory: 1200M
+
+  fluent-bit:
+    image: fluent/fluent-bit:4.2
+    container_name: e100-fluent-bit
+    command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/etc/rig/fluent-bit.conf"]
+    volumes:
+      - ${RIG_DIR}/fluent-bit:/fluent-bit/etc/rig:ro
+      - ${RIG_DATA}:/data
+    deploy:
+      resources:
+        limits:
+          memory: 300M

elastic-hot-tier-budget/install-e2e/render.mjs

@@ -0,0 +1,172 @@
+// Renders what the E100/E101 install proof applies, from a log10x-mcp build, through the three
+// tools' own handlers, and adds nothing to what they return:
+//   MCP=<log10x-mcp package> node render.mjs <out dir> <hot_tier_gb> <policy checkout> <containers,...>
+// 1. log10x_advise_retriever (destination elasticsearch): the plan's Elasticsearch frozen-tier
+//    block, verbatim to recipe.txt; its requests with the comment lines dropped to recipe-requests.json.
+// 2. log10x_advise_install (app receiver, forwarder fluentbit, log_destination elasticsearch,
+//    builtin licence, airgapped, metrics backend elastic): the plan to install-plan.md; from its values file the sidecar's
+//    image, args, env and limits, and from its seed step the policy ConfigMap's data, to receiver.json.
+// 3. log10x_setup_recurring (hot_tier_gb on github_actions, the checkout as the config plane):
+//    policy.yaml and apply-instructions.md.
+import { writeFileSync, mkdirSync, readFileSync } from 'fs';
+import { join, resolve } from 'path';
+import { createRequire } from 'module';
+
+const MCP = process.env.MCP;
+if (!MCP) throw new Error('set MCP to a log10x-mcp package');
+const [outArg, gbArg, checkout, containersArg] = process.argv.slice(2);
+if (!outArg || !gbArg || !checkout || !containersArg) throw new Error('usage: render.mjs <out dir> <hot_tier_gb> <policy checkout> <containers,...>');
+const out = resolve(outArg);
+mkdirSync(out, { recursive: true });
+const YAML = createRequire(`${MCP}/package.json`)('yaml');
+const mcpVersion = JSON.parse(readFileSync(`${MCP}/package.json`, 'utf8')).version;
+
+const { putSnapshot } = await import(`${MCP}/build/lib/discovery/snapshot-store.js`);
+const { SNAPSHOT_SCHEMA_VERSION } = await import(`${MCP}/build/lib/discovery/types.js`);
+const { executeAdviseRetriever } = await import(`${MCP}/build/tools/advise-retriever.js`);
+const { executeAdviseInstall } = await import(`${MCP}/build/tools/advise-install.js`);
+const { executeSetupRecurring } = await import(`${MCP}/build/tools/setup-recurring.js`);
+
+// A discovery snapshot of a cluster with Fluent Bit in `logging`; the AWS fields give the
+// Retriever plan a bucket and region so it renders its offload section.
+const snapshotId = `e100-${Date.now()}`;
+putSnapshot({
+  schemaVersion: SNAPSHOT_SCHEMA_VERSION,
+  snapshotId,
+  startedAt: new Date().toISOString(),
+  finishedAt: new Date().toISOString(),
+  kubectl: {
+    available: true,
+    namespaces: ['logging'],
+    probedNamespaces: ['logging'],
+    forwarders: [{ kind: 'fluentbit', namespace: 'logging', workloadKind: 'DaemonSet', workloadName: 'fluent-bit', image: 'cr.fluentbit.io/fluent/fluent-bit:4.2', containerName: 'fluent-bit', labels: {}, readyReplicas: 1 }],
+    helmReleases: [],
+    log10xApps: [],
+    storageClasses: [],
+    ingressClasses: [],
+    serviceAccountIrsa: [],
+    backendAgents: [],
+  },
+  aws: { available: true, region: 'us-east-1', s3Buckets: [], sqsQueues: [], cwLogGroups: [] },
+  recommendations: { suggestedNamespace: 'logging', alreadyInstalled: {} },
+  probeLog: [],
+});
+
+// 1. advise_retriever
+const q = (n) => `https://sqs.us-east-1.amazonaws.com/000000000000/e100-${n}`;
+const ret = await executeAdviseRetriever({
+  snapshot_id: snapshotId,
+  infra_mode: 'existing',
+  index_source_bucket: 'e100-retriever-input',
+  iam_role_arn: 'arn:aws:iam::000000000000:role/e100-retriever',
+  index_queue_url: q('index'),
+  query_queue_url: q('query'),
+  subquery_queue_url: q('subquery'),
+  stream_queue_url: q('stream'),
+  license_source: 'builtin',
+  destination: 'elasticsearch',
+});
+const retMd = String(ret.data?.markdown ?? JSON.stringify(ret.data));
+writeFileSync(join(out, 'retriever-plan.md'), retMd);
+const at = retMd.indexOf('_Elasticsearch frozen tier_');
+if (at < 0) throw new Error('advise_retriever rendered no Elasticsearch frozen-tier block; see retriever-plan.md');
+const fence = retMd.indexOf('```text\n', at);
+const recipe = retMd.slice(fence + '```text\n'.length, retMd.indexOf('\n```', fence + 8));
+writeFileSync(join(out, 'recipe.txt'), recipe + '\n');
+const requests = [];
+const lines = recipe.split('\n').filter((l) => !l.startsWith('#'));
+for (let i = 0; i < lines.length; i++) {
+  const m = /^(PUT|POST) (\S+)\s*$/.exec(lines[i]);
+  if (!m) continue;
+  const buf = [];
+  let depth = 0;
+  for (let j = i + 1; j < lines.length; j++) {
+    buf.push(lines[j]);
+    for (const ch of lines[j]) depth += ch === '{' ? 1 : ch === '}' ? -1 : 0;
+    if (depth === 0) { i = j; break; }
+  }
+  requests.push({ method: m[1], path: '/' + m[2], body: JSON.parse(buf.join('\n')) });
+}
+writeFileSync(join(out, 'recipe-requests.json'), JSON.stringify(requests, null, 1) + '\n');
+
+// 2. advise_install
+const inst = await executeAdviseInstall({
+  snapshot_id: snapshotId,
+  app: 'receiver',
+  forwarder: 'fluentbit',
+  backends: ['elastic'],
+  backend_credentials: { elastic: { secretName: 'elastic-credentials', plainValues: { ELASTICSEARCH_HOST: 'http://es-hot:9200' } } },
+  airgapped: true,
+  license_source: 'builtin',
+  log_destination: 'elasticsearch',
+}, {});
+const md = String(inst.data?.markdown ?? '');
+writeFileSync(join(out, 'install-plan.md'), md);
+const vAt = md.indexOf('Write `my-receiver-values.yaml`');
+if (vAt < 0) throw new Error(`advise_install rendered no values file (mode ${inst.data?.mode}); see install-plan.md`);
+const vStart = md.indexOf('```yaml\n', vAt) + '```yaml\n'.length;
+const values = YAML.parse(md.slice(vStart, md.indexOf('\n```', vStart)));
+const sidecar = values.extraContainers.find((c) => c.name === 'log10x');
+const seedAt = md.indexOf("cat <<'POLICY_EOF' | kubectl apply -f -\n");
+const seedText = md.slice(seedAt).split('\n').slice(1).join('\n').split('\nPOLICY_EOF')[0];
+const cm = YAML.parseAllDocuments(seedText).map((d) => d.toJS()).find((d) => d?.kind === 'ConfigMap');
+writeFileSync(join(out, 'receiver.json'), JSON.stringify({ mcp_version: mcpVersion, image: sidecar.image, args: sidecar.args, env: sidecar.env, resources: sidecar.resources, policy_configmap: { name: cm.metadata.name, data: cm.data } }, null, 1) + '\n');
+
+// 3. setup_recurring
+const rec = await executeSetupRecurring({
+  target_services: [],
+  hot_tier_gb: gbArg === 'auto' ? 'auto' : Number(gbArg),
+  scheduler: 'github_actions',
+  config_plane: checkout,
+  nodes_by_container: Object.fromEntries(containersArg.split(',').filter(Boolean).map((c) => [c, 1])),
+  confirm: true,
+});
+if (rec.data?.mode !== 'emit') throw new Error(`setup_recurring did not emit: ${rec.data?.markdown}`);
+writeFileSync(join(out, 'policy.yaml'), String(rec.data.policy_yaml));
+writeFileSync(join(out, 'apply-instructions.md'), String(rec.data.apply_instructions) + '\n');
+
+// The Receiver as a compose service: the image (ENGINE_IMAGE replaces the rendered one, both
+// recorded), every arg in order, every env var with a plain value, the limits. Substituted,
+// because there is no cluster: the values read the policy files through the Kubernetes pull
+// lane (K8S_ENABLED, K8S_CONFIGMAP, K8S_NAMESPACE); here the seed files are mounted at /policy.
+// CONFIG_GENERATION_FILE is left out: the seed carries no config-generation.csv.
+// Rig plumbing: TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit (Fluent Bit is a separate container).
+const SUBSTITUTE = { K8S_ENABLED: 'false', CAP_LOOKUP_FILE: '/policy/caps.csv', ACTION_LOOKUP_FILE: '/policy/actions.csv', TOTAL_LOOKUP_FILE: '/policy/totals.csv' };
+const LEAVE_OUT = new Set(['K8S_NAMESPACE', 'K8S_CONFIGMAP', 'CONFIG_GENERATION_FILE']);
+const applied = [];
+const substituted = [];
+const left = [];
+for (const e of sidecar.env) {
+  if (LEAVE_OUT.has(e.name)) left.push(e.name);
+  else if (e.name in SUBSTITUTE) { substituted.push({ name: e.name, rendered: e.value, rig: SUBSTITUTE[e.name] }); applied.push([e.name, SUBSTITUTE[e.name]]); }
+  else if (e.value !== undefined) applied.push([e.name, String(e.value)]);
+  else left.push(e.name);
+}
+const image = process.env.ENGINE_IMAGE || sidecar.image;
+const lim = sidecar.resources?.limits ?? {};
+const cpus = lim.cpu ? (String(lim.cpu).endsWith('m') ? String(Number(String(lim.cpu).slice(0, -1)) / 1000) : String(lim.cpu)) : undefined;
+const mem = lim.memory ? String(lim.memory).replace(/Mi$/, 'M').replace(/Gi$/, 'G') : undefined;
+const jq = (v) => JSON.stringify(String(v)).replace(/\$/g, '$$$$');
+writeFileSync(join(out, 'compose.receiver.yaml'), [
+  '# Generated by install-e2e/render.mjs from log10x_advise_install\'s values (receiver.json). Do not edit.',
+  'services:',
+  '  tenx:',
+  `    image: ${jq(image)}`,
+  '    container_name: e100-tenx',
+  '    environment:',
+  ...applied.map(([k, v]) => `      - ${jq(`${k}=${v}`)}`),
+  `      - ${jq('TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit')}`,
+  '    tmpfs:',
+  '      - /tmp',
+  '    volumes:',
+  '      - ${RESULTS}/policy:/policy:ro',
+  '    command:',
+  ...sidecar.args.map((a) => `      - ${jq(a)}`),
+  ...(cpus || mem ? ['    deploy:', '      resources:', '        limits:', ...(cpus ? [`          cpus: ${jq(cpus)}`] : []), ...(mem ? [`          memory: ${jq(mem)}`] : [])] : []),
+  '',
+].join('\n'));
+mkdirSync(join(out, 'policy'), { recursive: true });
+for (const f of ['caps.csv', 'actions.csv', 'totals.csv']) writeFileSync(join(out, 'policy', f), cm.data[f]);
+writeFileSync(join(out, 'launch.json'), JSON.stringify({ mcp_version: mcpVersion, image, image_rendered: sidecar.image, args_from_values: sidecar.args, env_applied: applied.map(([k]) => k), env_substituted: substituted, env_left_out: left, rig_env: ['TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit'], resources_limits: { cpus, memory: mem }, policy_seed_from_configmap: Object.keys(cm.data) }, null, 1) + '\n');
+
+console.log(JSON.stringify({ mcp_version: mcpVersion, recipe_requests: requests.map((r) => `${r.method} ${r.path}`), image: sidecar.image, args: sidecar.args, env: sidecar.env.map((e) => e.name), seed_keys: Object.keys(cm.data) }, null, 1));

elastic-hot-tier-budget/install-e2e/results/apply-instructions.md

@@ -0,0 +1,9 @@
+### Apply instructions: hot-tier run, GitHub Actions (gitops pull)
+
+1. Install (or re-install) the Receiver with `log10x_advise_install` and `log_destination: "elasticsearch"`: it counts bytes, the default, stamps `utf8Size` (the size the limits count) and `tenx_hash` on every record it returns, and records `bytes` in the policy files (`regulator-unit`). The run writes limits in bytes and refuses, writing nothing, a Receiver recorded as counting events. Pass `policy_git: { repo: "<owner/name>" }` so the Receiver pulls `log10x/policy/*.csv` from the `log10x-policy` branch (GH_ENABLED, GH_REPO, GH_BRANCH, GH_TOKEN).
+2. Apply the Elasticsearch frozen-tier recipe (`log10x_advise_retriever` with `destination: "elasticsearch"`): kept lines go to the `tenx-app` alias and `routeState: tier_down` lines to `tenx-tierdown`, which ILM mounts on the frozen tier within hours. Its `tenx-keep` policy rolls kept lines over daily and mounts them on frozen 6 days later, 7 days on hot at most, and the run reads that from the ILM policy on `tenx-app`: its rollover `max_age` plus the `min_age` of the first phase after hot. A policy of the account's own on `tenx-app` needs a phase after hot at the age a kept line leaves hot (`warm`, `cold`, `frozen` or `delete`), or `hot_retention_hours` in policy.yaml.
+3. **Commit `policy.yaml`** to the root of the repo, with `nodes_by_container` set for every service.
+4. **Commit `log10x-recurring.yml`** to `.github/workflows/`.
+5. **Add the secrets** `LOG10X_ES_URL` and `LOG10X_ES_API_KEY` (an encoded API key). The run reads the cluster (node stats, ILM and the Receiver's indices) and writes nothing to it.
+6. Each run commits `log10x/policy/{caps,actions,totals,config-generation}.csv` and `reports/latest.md` directly to the `log10x-policy` branch (schedule `0 * * * *` UTC).
+7. Hourly is about 720 runs a month on the repo's Actions minutes; for fewer, pass `schedule: "every-6h"` or a cron expression such as `"0 */3 * * *"`.

elastic-hot-tier-budget/install-e2e/results/compose.receiver.yaml

@@ -0,0 +1,28 @@
+# Generated by install-e2e/render.mjs from log10x_advise_install's values (receiver.json). Do not edit.
+services:
+  tenx:
+    image: "ghcr.io/log-10x/edge-10x:1.1.150"
+    container_name: e100-tenx
+    environment:
+      - "TENX_AIRGAPPED=true"
+      - "K8S_ENABLED=false"
+      - "CAP_LOOKUP_FILE=/policy/caps.csv"
+      - "ACTION_LOOKUP_FILE=/policy/actions.csv"
+      - "TOTAL_LOOKUP_FILE=/policy/totals.csv"
+      - "TENX_FLUENTBIT_OUTPUT_HOST=fluent-bit"
+    tmpfs:
+      - /tmp
+    volumes:
+      - ${RESULTS}/policy:/policy:ro
+    command:
+      - "@run/input/forwarder/fluentbit"
+      - "@apps/receiver"
+      - "symbolMessageHashField"
+      - "tenx_hash"
+      - "outputEncodeField"
+      - "fullText(\"tenx_hash\",\"routeState\",\"message_pattern\",\"severity_level\",\"utf8Size\")"
+    deploy:
+      resources:
+        limits:
+          cpus: "0.5"
+          memory: "512M"