From 7d1358ae0e43d3f388e2ef31641e73d003ed4a2d Mon Sep 17 00:00:00 2001 From: Tal Weiss <73443663+talwgx@users.noreply.github.com> Date: Sun, 25 Oct 2026 19:21:56 -0400 Subject: [PATCH] intent, naming-stability: reconcile the two pattern counts intent counts the patterns of labelled events (2,277 on otel215, 406 on rec1155); naming-stability counts every event (2,687 and 582), both on the 1.1.142 image. bench/pattern_counts.py checks one run of each benchmark's engine config against the labelled capture, event for event: the two runs cut the same events and give every event the same name, and the difference is the patterns only unlabelled events take (410 and 176). Both READMEs state this. METHOD.md's unlabelled-by-container lines now match the labels file (cart 10 on rec1155; 11 smaller containers on otel215). Co-Authored-By: Claude Opus 5.5 --- intent/METHOD.md | 4 +- intent/README.md | 5 +- intent/bench/pattern_counts.py | 124 ++++++++++++++ .../pattern-counts-1.1.142_otel215.json | 151 ++++++++++++++++++ .../pattern-counts-1.1.142_rec1155.json | 59 +++++++ naming-stability/README.md | 16 +- 6 files changed, 352 insertions(+), 7 deletions(-) create mode 100644 intent/bench/pattern_counts.py create mode 100644 intent/results/pattern-counts-1.1.142_otel215.json create mode 100644 intent/results/pattern-counts-1.1.142_rec1155.json diff --git a/intent/METHOD.md b/intent/METHOD.md index 577dbc2..9c44e04 100644 --- a/intent/METHOD.md +++ b/intent/METHOD.md @@ -99,8 +99,8 @@ events. | otel215 | 157,228 | 156,563 (99.58%) | 665 (0.42%) | 299 | 289 / 10 | Unlabelled events, by container: -- **rec1155:** frontend-proxy 115, postgresql 83, flagd 14, image-provider 9, flagd-ui 5, init-config 2, and single events in cart, accounting and email. -- **otel215:** prometheus-server 152, frontend-proxy 114, opensearch 102 (single events outside its 20 labelled statements), jaeger 87, coredns 43, metrics-server 35, Grafana sidecars 34, postgresql 33, init-config 14, and 13 smaller containers. +- **rec1155:** frontend-proxy 115, postgresql 83, flagd 14, cart 10, image-provider 9, flagd-ui 5, init-config 2, and single events in accounting and email. +- **otel215:** prometheus-server 152, frontend-proxy 114, opensearch 102 (single events outside its 20 labelled statements), jaeger 87, coredns 43, metrics-server 35, Grafana sidecars 34, postgresql 33, init-config 14, and 11 smaller containers. Labelled events and statements, by container: - **rec1155:** diff --git a/intent/README.md b/intent/README.md index 9969898..b7a0de7 100644 --- a/intent/README.md +++ b/intent/README.md @@ -47,7 +47,7 @@ Two tools are scored on the same events: | Drain3, factory defaults | 3.34% (1,381) | 10.34% | 26.19% | 57.75% of 213 | 393 | | Drain3, strong arm, per record | 1.78% (734) | 10.35% | 79.79% | 65.26% of 213 | 377 | -The patterns column counts the patterns of labelled events only (`distinct_patterns` in `bench/score.py`): 2,277 on otel215 and 406 on rec1155 for 1.1.142. [`../naming-stability`](../naming-stability) counts the patterns of every event, with the same 1.1.142 image on the same captures: 2,687 on otel215 and 582 on rec1155. The counts differ by 410 on otel215 and 176 on rec1155; the difference is not reconciled here. +The patterns column counts the patterns of labelled events only (`distinct_patterns` in `bench/score.py`): 2,277 on otel215 and 406 on rec1155 for 1.1.142. [`../naming-stability`](../naming-stability) counts the patterns of every event, with the same 1.1.142 image on the same captures: 2,687 on otel215 and 582 on rec1155. The counts differ by 410 on otel215 and 176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs cut each capture into the same events and give every event the same pattern name. On otel215 the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. [`bench/pattern_counts.py`](bench/pattern_counts.py) computes these numbers from the two runs' events ([otel215](results/pattern-counts-1.1.142_otel215.json), [rec1155](results/pattern-counts-1.1.142_rec1155.json)). Without the opentelemetry-collector container (57% of otel215's events, 46% of rec1155's): @@ -194,4 +194,5 @@ python3 bench/score.py otel215 out/events.jsonl - `bench/drain_arm.py`: the Drain3 arms. - `bench/score.py`: the scorer. - `bench/summarize.py`: writes `results/results.md`. -- `results/`: the committed output of `run.sh`. +- `bench/pattern_counts.py`: compares this benchmark's pattern count with `../naming-stability`'s, from one run of each on the same capture, and writes `results/pattern-counts-1.1.142_.json`. +- `results/`: the committed output of `run.sh` and of `bench/pattern_counts.py`. diff --git a/intent/bench/pattern_counts.py b/intent/bench/pattern_counts.py new file mode 100644 index 0000000..768a4fb --- /dev/null +++ b/intent/bench/pattern_counts.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +"""Reconcile the pattern count of this benchmark with ../naming-stability's. + + python3 bench/pattern_counts.py + [--out file.json] + + is rec1155 or otel215. The intent events file is what bench/engine.sh +writes; the naming-stability one is what ../naming-stability/bench/engine.sh +writes, for the same image on the same capture. Both are checked against the +labelled capture, event for event, as bench/score.py does. + +Reports, per capture: + whether the two runs gave every event the same pattern name; + the patterns of every event (the naming-stability count); + the patterns of labelled events (`distinct_patterns` in bench/score.py); + the patterns that occur only in unlabelled events, by container. +""" + +import argparse +import collections +import gzip +import hashlib +import json +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +LABELS = os.path.join(os.path.dirname(HERE), 'labels') + + +def opener(path): + return gzip.open(path, 'rt', encoding='utf-8') if path.endswith('.gz') else open(path, encoding='utf-8') + + +def container_of(text): + try: + d = json.loads(text.split('\n', 1)[0]) + except ValueError: + return None + return (d.get('kubernetes') or {}).get('container_name', '') + + +def load_events(path, labels, hashes): + names = [] + with opener(path) as f: + for k, line in enumerate(f): + d = json.loads(line) + t = d['text'] + if k >= len(labels) or hashlib.sha1(t.encode()).hexdigest()[:16] != hashes[k] \ + or container_of(t) != labels[k]['container']: + sys.exit(f'ALIGNMENT: {path}: event {k} differs from the labelled capture') + names.append(d.get('message_pattern') or '') + if len(names) != len(labels): + sys.exit(f'ALIGNMENT: {path}: {len(names)} events, {len(labels)} labels') + return names + + +def reconcile(capture, intent_path, ns_path): + with opener(os.path.join(LABELS, f'labels_{capture}.jsonl.gz')) as f: + labels = [json.loads(l) for l in f] + with opener(os.path.join(LABELS, f'texthash_{capture}.txt.gz')) as f: + hashes = f.read().split() + a = load_events(intent_path, labels, hashes) + b = load_events(ns_path, labels, hashes) + differ = sum(x != y for x, y in zip(a, b)) + + names = b + labelled = [k for k, r in enumerate(labels) if r['statement']] + unlabelled = [k for k, r in enumerate(labels) if not r['statement']] + p_all = set(names) + p_lab = {names[k] for k in labelled} + p_unl = {names[k] for k in unlabelled} + only = p_unl - p_lab + + # Per container: its unlabelled events and the unlabelled-only patterns + # they take. A pattern seen in several containers counts in each. + by_pat = collections.defaultdict(collections.Counter) + for k in unlabelled: + if names[k] in only: + by_pat[names[k]][labels[k]['container']] += 1 + by_container = collections.defaultdict(lambda: dict(unlabelled_events=0, patterns=0)) + for k in unlabelled: + by_container[labels[k]['container']]['unlabelled_events'] += 1 + for c in by_pat.values(): + for cont in c: + by_container[cont]['patterns'] += 1 + multi = {p: dict(c) for p, c in sorted(by_pat.items()) if len(c) > 1} + + return dict( + capture=capture, + events=len(labels), + labelled_events=len(labelled), + unlabelled_events=len(unlabelled), + unlabelled_multiline_events=sum(1 for k in unlabelled if labels[k]['multiline']), + events_named_differently_by_the_two_runs=differ, + patterns_every_event=len(p_all), + patterns_labelled_events=len(p_lab), + patterns_unlabelled_events=len(p_unl), + patterns_in_both=len(p_lab & p_unl), + patterns_only_in_unlabelled_events=len(only), + events_in_unlabelled_only_patterns=sum(sum(c.values()) for c in by_pat.values()), + unlabelled_only_patterns_by_container=dict( + sorted(by_container.items(), key=lambda x: (-x[1]['patterns'], -x[1]['unlabelled_events'], x[0]))), + unlabelled_only_patterns_in_more_than_one_container=multi, + ) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('capture', choices=('rec1155', 'otel215')) + ap.add_argument('intent_events') + ap.add_argument('naming_stability_events') + ap.add_argument('--out') + a = ap.parse_args() + res = reconcile(a.capture, a.intent_events, a.naming_stability_events) + if a.out: + with open(a.out, 'w', encoding='utf-8') as f: + json.dump(res, f, indent=1) + f.write('\n') + print(json.dumps(res, indent=1)) + + +if __name__ == '__main__': + main() diff --git a/intent/results/pattern-counts-1.1.142_otel215.json b/intent/results/pattern-counts-1.1.142_otel215.json new file mode 100644 index 0000000..94729f1 --- /dev/null +++ b/intent/results/pattern-counts-1.1.142_otel215.json @@ -0,0 +1,151 @@ +{ + "capture": "otel215", + "events": 157228, + "labelled_events": 156563, + "unlabelled_events": 665, + "unlabelled_multiline_events": 39, + "events_named_differently_by_the_two_runs": 0, + "patterns_every_event": 2687, + "patterns_labelled_events": 2277, + "patterns_unlabelled_events": 411, + "patterns_in_both": 1, + "patterns_only_in_unlabelled_events": 410, + "events_in_unlabelled_only_patterns": 663, + "unlabelled_only_patterns_by_container": { + "frontend-proxy": { + "unlabelled_events": 114, + "patterns": 114 + }, + "opensearch": { + "unlabelled_events": 102, + "patterns": 83 + }, + "jaeger": { + "unlabelled_events": 87, + "patterns": 58 + }, + "prometheus-server": { + "unlabelled_events": 152, + "patterns": 38 + }, + "postgresql": { + "unlabelled_events": 33, + "patterns": 24 + }, + "metrics-server": { + "unlabelled_events": 35, + "patterns": 15 + }, + "grafana-sc-dashboard": { + "unlabelled_events": 14, + "patterns": 13 + }, + "init-config": { + "unlabelled_events": 14, + "patterns": 10 + }, + "grafana-sc-alerts": { + "unlabelled_events": 10, + "patterns": 10 + }, + "grafana-sc-datasources": { + "unlabelled_events": 10, + "patterns": 10 + }, + "flagd": { + "unlabelled_events": 9, + "patterns": 9 + }, + "image-provider": { + "unlabelled_events": 9, + "patterns": 9 + }, + "aws-vpc-cni-init": { + "unlabelled_events": 7, + "patterns": 7 + }, + "cart": { + "unlabelled_events": 7, + "patterns": 7 + }, + "coredns": { + "unlabelled_events": 43, + "patterns": 6 + }, + "aws-node": { + "unlabelled_events": 4, + "patterns": 4 + }, + "email": { + "unlabelled_events": 3, + "patterns": 3 + }, + "flagd-ui": { + "unlabelled_events": 3, + "patterns": 3 + }, + "fluentd-10x": { + "unlabelled_events": 2, + "patterns": 2 + }, + "load-generator": { + "unlabelled_events": 5, + "patterns": 1 + }, + "aws-eks-nodeagent": { + "unlabelled_events": 1, + "patterns": 1 + }, + "currency": { + "unlabelled_events": 1, + "patterns": 1 + } + }, + "unlabelled_only_patterns_in_more_than_one_container": { + "Config_for_cluster_api_at_https_loaded": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 1, + "grafana-sc-datasources": 1 + }, + "Loading_incluster_config": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 1, + "grafana-sc-datasources": 1 + }, + "No_folder_annotation_was_provided_defaulting_to": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 1, + "grafana-sc-datasources": 1 + }, + "None_sent_to_http_localhost_Response_OK": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 2, + "grafana-sc-datasources": 1 + }, + "Starting_collector": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 1, + "grafana-sc-datasources": 1 + }, + "Unique_filenames_will_not_be_enforced": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 1, + "grafana-sc-datasources": 1 + }, + "response_content_will_not_be_enabled": { + "grafana-sc-alerts": 1, + "grafana-sc-dashboard": 1, + "grafana-sc-datasources": 1 + }, + "stream_stdout_log_docker": { + "flagd": 1, + "image-provider": 1 + }, + "template_9ae16a3b2f90404f": { + "flagd": 1, + "image-provider": 1, + "opensearch": 1, + "postgresql": 6 + } + } +} diff --git a/intent/results/pattern-counts-1.1.142_rec1155.json b/intent/results/pattern-counts-1.1.142_rec1155.json new file mode 100644 index 0000000..b30cb88 --- /dev/null +++ b/intent/results/pattern-counts-1.1.142_rec1155.json @@ -0,0 +1,59 @@ +{ + "capture": "rec1155", + "events": 41573, + "labelled_events": 41333, + "unlabelled_events": 240, + "unlabelled_multiline_events": 29, + "events_named_differently_by_the_two_runs": 0, + "patterns_every_event": 582, + "patterns_labelled_events": 406, + "patterns_unlabelled_events": 176, + "patterns_in_both": 0, + "patterns_only_in_unlabelled_events": 176, + "events_in_unlabelled_only_patterns": 240, + "unlabelled_only_patterns_by_container": { + "frontend-proxy": { + "unlabelled_events": 115, + "patterns": 115 + }, + "postgresql": { + "unlabelled_events": 83, + "patterns": 23 + }, + "flagd": { + "unlabelled_events": 14, + "patterns": 13 + }, + "cart": { + "unlabelled_events": 10, + "patterns": 10 + }, + "image-provider": { + "unlabelled_events": 9, + "patterns": 9 + }, + "flagd-ui": { + "unlabelled_events": 5, + "patterns": 5 + }, + "init-config": { + "unlabelled_events": 2, + "patterns": 1 + }, + "accounting": { + "unlabelled_events": 1, + "patterns": 1 + }, + "email": { + "unlabelled_events": 1, + "patterns": 1 + } + }, + "unlabelled_only_patterns_in_more_than_one_container": { + "stream_stdout_docker": { + "flagd": 2, + "image-provider": 1, + "postgresql": 2 + } + } +} diff --git a/naming-stability/README.md b/naming-stability/README.md index 6812ba0..cc62a92 100644 --- a/naming-stability/README.md +++ b/naming-stability/README.md @@ -26,9 +26,19 @@ changed its pattern name, its pattern hash (`tenx_hash`, the hash of the name) o hash, so no pattern is listed as changed ([`results/results.md`](results/results.md)). These pattern counts take every event: 2,687 on otel215 and 582 on rec1155 for 1.1.142. -[`../intent`](../intent) counts the patterns of labelled events only and reports 2,277 on otel215 -and 406 on rec1155 for the same 1.1.142 image. The counts differ by 410 on otel215 and 176 on -rec1155; the difference is not reconciled here. +[`../intent`](../intent) counts the patterns of labelled events only and reports 2,277 on +otel215 and 406 on rec1155 for the same 1.1.142 image. The counts differ by 410 on otel215 and +176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs +cut each capture into the same events and give every event the same pattern name. On otel215 +the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled +event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled +event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled +event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of +Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and +prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. +[`../intent/bench/pattern_counts.py`](../intent/bench/pattern_counts.py) computes these numbers +from the two runs' events ([otel215](../intent/results/pattern-counts-1.1.142_otel215.json), +[rec1155](../intent/results/pattern-counts-1.1.142_rec1155.json)). ## What changed in each release, and what these captures exercise