intent, naming-stability: reconcile the two pattern counts

intent/METHOD.md

@@ -99,8 +99,8 @@ events.
 | otel215 | 157,228 | 156,563 (99.58%) | 665 (0.42%) | 299 | 289 / 10 |
 
 Unlabelled events, by container:
-- **rec1155:** frontend-proxy 115, postgresql 83, flagd 14, image-provider 9, flagd-ui 5, init-config 2, and single events in cart, accounting and email.
-- **otel215:** prometheus-server 152, frontend-proxy 114, opensearch 102 (single events outside its 20 labelled statements), jaeger 87, coredns 43, metrics-server 35, Grafana sidecars 34, postgresql 33, init-config 14, and 13 smaller containers.
+- **rec1155:** frontend-proxy 115, postgresql 83, flagd 14, cart 10, image-provider 9, flagd-ui 5, init-config 2, and single events in accounting and email.
+- **otel215:** prometheus-server 152, frontend-proxy 114, opensearch 102 (single events outside its 20 labelled statements), jaeger 87, coredns 43, metrics-server 35, Grafana sidecars 34, postgresql 33, init-config 14, and 11 smaller containers.
 
 Labelled events and statements, by container:
 - **rec1155:**

intent/README.md

@@ -47,7 +47,7 @@ Two tools are scored on the same events:
 | Drain3, factory defaults | 3.34% (1,381) | 10.34% | 26.19% | 57.75% of 213 | 393 |
 | Drain3, strong arm, per record | 1.78% (734) | 10.35% | 79.79% | 65.26% of 213 | 377 |
 
-The patterns column counts the patterns of labelled events only (`distinct_patterns` in `bench/score.py`): 2,277 on otel215 and 406 on rec1155 for 1.1.142. [`../naming-stability`](../naming-stability) counts the patterns of every event, with the same 1.1.142 image on the same captures: 2,687 on otel215 and 582 on rec1155. The counts differ by 410 on otel215 and 176 on rec1155; the difference is not reconciled here.
+The patterns column counts the patterns of labelled events only (`distinct_patterns` in `bench/score.py`): 2,277 on otel215 and 406 on rec1155 for 1.1.142. [`../naming-stability`](../naming-stability) counts the patterns of every event, with the same 1.1.142 image on the same captures: 2,687 on otel215 and 582 on rec1155. The counts differ by 410 on otel215 and 176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs cut each capture into the same events and give every event the same pattern name. On otel215 the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155. [`bench/pattern_counts.py`](bench/pattern_counts.py) computes these numbers from the two runs' events ([otel215](results/pattern-counts-1.1.142_otel215.json), [rec1155](results/pattern-counts-1.1.142_rec1155.json)).
 
 Without the opentelemetry-collector container (57% of otel215's events, 46% of rec1155's):
 
@@ -194,4 +194,5 @@ python3 bench/score.py otel215 out/events.jsonl
 - `bench/drain_arm.py`: the Drain3 arms.
 - `bench/score.py`: the scorer.
 - `bench/summarize.py`: writes `results/results.md`.
-- `results/`: the committed output of `run.sh`.
+- `bench/pattern_counts.py`: compares this benchmark's pattern count with `../naming-stability`'s, from one run of each on the same capture, and writes `results/pattern-counts-1.1.142_<capture>.json`.
+- `results/`: the committed output of `run.sh` and of `bench/pattern_counts.py`.

intent/bench/pattern_counts.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""Reconcile the pattern count of this benchmark with ../naming-stability's.
+
+    python3 bench/pattern_counts.py <capture> <intent events> <naming-stability events>
+                                    [--out file.json]
+
+<capture> is rec1155 or otel215. The intent events file is what bench/engine.sh
+writes; the naming-stability one is what ../naming-stability/bench/engine.sh
+writes, for the same image on the same capture. Both are checked against the
+labelled capture, event for event, as bench/score.py does.
+
+Reports, per capture:
+  whether the two runs gave every event the same pattern name;
+  the patterns of every event (the naming-stability count);
+  the patterns of labelled events (`distinct_patterns` in bench/score.py);
+  the patterns that occur only in unlabelled events, by container.
+"""
+
+import argparse
+import collections
+import gzip
+import hashlib
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+LABELS = os.path.join(os.path.dirname(HERE), 'labels')
+
+
+def opener(path):
+    return gzip.open(path, 'rt', encoding='utf-8') if path.endswith('.gz') else open(path, encoding='utf-8')
+
+
+def container_of(text):
+    try:
+        d = json.loads(text.split('\n', 1)[0])
+    except ValueError:
+        return None
+    return (d.get('kubernetes') or {}).get('container_name', '<nok8s>')
+
+
+def load_events(path, labels, hashes):
+    names = []
+    with opener(path) as f:
+        for k, line in enumerate(f):
+            d = json.loads(line)
+            t = d['text']
+            if k >= len(labels) or hashlib.sha1(t.encode()).hexdigest()[:16] != hashes[k] \
+                    or container_of(t) != labels[k]['container']:
+                sys.exit(f'ALIGNMENT: {path}: event {k} differs from the labelled capture')
+            names.append(d.get('message_pattern') or '')
+    if len(names) != len(labels):
+        sys.exit(f'ALIGNMENT: {path}: {len(names)} events, {len(labels)} labels')
+    return names
+
+
+def reconcile(capture, intent_path, ns_path):
+    with opener(os.path.join(LABELS, f'labels_{capture}.jsonl.gz')) as f:
+        labels = [json.loads(l) for l in f]
+    with opener(os.path.join(LABELS, f'texthash_{capture}.txt.gz')) as f:
+        hashes = f.read().split()
+    a = load_events(intent_path, labels, hashes)
+    b = load_events(ns_path, labels, hashes)
+    differ = sum(x != y for x, y in zip(a, b))
+
+    names = b
+    labelled = [k for k, r in enumerate(labels) if r['statement']]
+    unlabelled = [k for k, r in enumerate(labels) if not r['statement']]
+    p_all = set(names)
+    p_lab = {names[k] for k in labelled}
+    p_unl = {names[k] for k in unlabelled}
+    only = p_unl - p_lab
+
+    # Per container: its unlabelled events and the unlabelled-only patterns
+    # they take. A pattern seen in several containers counts in each.
+    by_pat = collections.defaultdict(collections.Counter)
+    for k in unlabelled:
+        if names[k] in only:
+            by_pat[names[k]][labels[k]['container']] += 1
+    by_container = collections.defaultdict(lambda: dict(unlabelled_events=0, patterns=0))
+    for k in unlabelled:
+        by_container[labels[k]['container']]['unlabelled_events'] += 1
+    for c in by_pat.values():
+        for cont in c:
+            by_container[cont]['patterns'] += 1
+    multi = {p: dict(c) for p, c in sorted(by_pat.items()) if len(c) > 1}
+
+    return dict(
+        capture=capture,
+        events=len(labels),
+        labelled_events=len(labelled),
+        unlabelled_events=len(unlabelled),
+        unlabelled_multiline_events=sum(1 for k in unlabelled if labels[k]['multiline']),
+        events_named_differently_by_the_two_runs=differ,
+        patterns_every_event=len(p_all),
+        patterns_labelled_events=len(p_lab),
+        patterns_unlabelled_events=len(p_unl),
+        patterns_in_both=len(p_lab & p_unl),
+        patterns_only_in_unlabelled_events=len(only),
+        events_in_unlabelled_only_patterns=sum(sum(c.values()) for c in by_pat.values()),
+        unlabelled_only_patterns_by_container=dict(
+            sorted(by_container.items(), key=lambda x: (-x[1]['patterns'], -x[1]['unlabelled_events'], x[0]))),
+        unlabelled_only_patterns_in_more_than_one_container=multi,
+    )
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument('capture', choices=('rec1155', 'otel215'))
+    ap.add_argument('intent_events')
+    ap.add_argument('naming_stability_events')
+    ap.add_argument('--out')
+    a = ap.parse_args()
+    res = reconcile(a.capture, a.intent_events, a.naming_stability_events)
+    if a.out:
+        with open(a.out, 'w', encoding='utf-8') as f:
+            json.dump(res, f, indent=1)
+            f.write('\n')
+    print(json.dumps(res, indent=1))
+
+
+if __name__ == '__main__':
+    main()

intent/results/pattern-counts-1.1.142_otel215.json

@@ -0,0 +1,151 @@
+{
+ "capture": "otel215",
+ "events": 157228,
+ "labelled_events": 156563,
+ "unlabelled_events": 665,
+ "unlabelled_multiline_events": 39,
+ "events_named_differently_by_the_two_runs": 0,
+ "patterns_every_event": 2687,
+ "patterns_labelled_events": 2277,
+ "patterns_unlabelled_events": 411,
+ "patterns_in_both": 1,
+ "patterns_only_in_unlabelled_events": 410,
+ "events_in_unlabelled_only_patterns": 663,
+ "unlabelled_only_patterns_by_container": {
+  "frontend-proxy": {
+   "unlabelled_events": 114,
+   "patterns": 114
+  },
+  "opensearch": {
+   "unlabelled_events": 102,
+   "patterns": 83
+  },
+  "jaeger": {
+   "unlabelled_events": 87,
+   "patterns": 58
+  },
+  "prometheus-server": {
+   "unlabelled_events": 152,
+   "patterns": 38
+  },
+  "postgresql": {
+   "unlabelled_events": 33,
+   "patterns": 24
+  },
+  "metrics-server": {
+   "unlabelled_events": 35,
+   "patterns": 15
+  },
+  "grafana-sc-dashboard": {
+   "unlabelled_events": 14,
+   "patterns": 13
+  },
+  "init-config": {
+   "unlabelled_events": 14,
+   "patterns": 10
+  },
+  "grafana-sc-alerts": {
+   "unlabelled_events": 10,
+   "patterns": 10
+  },
+  "grafana-sc-datasources": {
+   "unlabelled_events": 10,
+   "patterns": 10
+  },
+  "flagd": {
+   "unlabelled_events": 9,
+   "patterns": 9
+  },
+  "image-provider": {
+   "unlabelled_events": 9,
+   "patterns": 9
+  },
+  "aws-vpc-cni-init": {
+   "unlabelled_events": 7,
+   "patterns": 7
+  },
+  "cart": {
+   "unlabelled_events": 7,
+   "patterns": 7
+  },
+  "coredns": {
+   "unlabelled_events": 43,
+   "patterns": 6
+  },
+  "aws-node": {
+   "unlabelled_events": 4,
+   "patterns": 4
+  },
+  "email": {
+   "unlabelled_events": 3,
+   "patterns": 3
+  },
+  "flagd-ui": {
+   "unlabelled_events": 3,
+   "patterns": 3
+  },
+  "fluentd-10x": {
+   "unlabelled_events": 2,
+   "patterns": 2
+  },
+  "load-generator": {
+   "unlabelled_events": 5,
+   "patterns": 1
+  },
+  "aws-eks-nodeagent": {
+   "unlabelled_events": 1,
+   "patterns": 1
+  },
+  "currency": {
+   "unlabelled_events": 1,
+   "patterns": 1
+  }
+ },
+ "unlabelled_only_patterns_in_more_than_one_container": {
+  "Config_for_cluster_api_at_https_loaded": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 1,
+   "grafana-sc-datasources": 1
+  },
+  "Loading_incluster_config": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 1,
+   "grafana-sc-datasources": 1
+  },
+  "No_folder_annotation_was_provided_defaulting_to": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 1,
+   "grafana-sc-datasources": 1
+  },
+  "None_sent_to_http_localhost_Response_OK": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 2,
+   "grafana-sc-datasources": 1
+  },
+  "Starting_collector": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 1,
+   "grafana-sc-datasources": 1
+  },
+  "Unique_filenames_will_not_be_enforced": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 1,
+   "grafana-sc-datasources": 1
+  },
+  "response_content_will_not_be_enabled": {
+   "grafana-sc-alerts": 1,
+   "grafana-sc-dashboard": 1,
+   "grafana-sc-datasources": 1
+  },
+  "stream_stdout_log_docker": {
+   "flagd": 1,
+   "image-provider": 1
+  },
+  "template_9ae16a3b2f90404f": {
+   "flagd": 1,
+   "image-provider": 1,
+   "opensearch": 1,
+   "postgresql": 6
+  }
+ }
+}

intent/results/pattern-counts-1.1.142_rec1155.json

@@ -0,0 +1,59 @@
+{
+ "capture": "rec1155",
+ "events": 41573,
+ "labelled_events": 41333,
+ "unlabelled_events": 240,
+ "unlabelled_multiline_events": 29,
+ "events_named_differently_by_the_two_runs": 0,
+ "patterns_every_event": 582,
+ "patterns_labelled_events": 406,
+ "patterns_unlabelled_events": 176,
+ "patterns_in_both": 0,
+ "patterns_only_in_unlabelled_events": 176,
+ "events_in_unlabelled_only_patterns": 240,
+ "unlabelled_only_patterns_by_container": {
+  "frontend-proxy": {
+   "unlabelled_events": 115,
+   "patterns": 115
+  },
+  "postgresql": {
+   "unlabelled_events": 83,
+   "patterns": 23
+  },
+  "flagd": {
+   "unlabelled_events": 14,
+   "patterns": 13
+  },
+  "cart": {
+   "unlabelled_events": 10,
+   "patterns": 10
+  },
+  "image-provider": {
+   "unlabelled_events": 9,
+   "patterns": 9
+  },
+  "flagd-ui": {
+   "unlabelled_events": 5,
+   "patterns": 5
+  },
+  "init-config": {
+   "unlabelled_events": 2,
+   "patterns": 1
+  },
+  "accounting": {
+   "unlabelled_events": 1,
+   "patterns": 1
+  },
+  "email": {
+   "unlabelled_events": 1,
+   "patterns": 1
+  }
+ },
+ "unlabelled_only_patterns_in_more_than_one_container": {
+  "stream_stdout_docker": {
+   "flagd": 2,
+   "image-provider": 1,
+   "postgresql": 2
+  }
+ }
+}

naming-stability/README.md

@@ -26,9 +26,19 @@ changed its pattern name, its pattern hash (`tenx_hash`, the hash of the name) o
 hash, so no pattern is listed as changed ([`results/results.md`](results/results.md)).
 
 These pattern counts take every event: 2,687 on otel215 and 582 on rec1155 for 1.1.142.
-[`../intent`](../intent) counts the patterns of labelled events only and reports 2,277 on otel215
-and 406 on rec1155 for the same 1.1.142 image. The counts differ by 410 on otel215 and 176 on
-rec1155; the difference is not reconciled here.
+[`../intent`](../intent) counts the patterns of labelled events only and reports 2,277 on
+otel215 and 406 on rec1155 for the same 1.1.142 image. The counts differ by 410 on otel215 and
+176 on rec1155. The difference is the patterns that only unlabelled events take. The two runs
+cut each capture into the same events and give every event the same pattern name. On otel215
+the 665 unlabelled events take 411 patterns, and one of them, `Starting`, also names a labelled
+event, so they add 410. On rec1155 the 240 unlabelled events take 176 patterns that no labelled
+event shares, so they add 176. In frontend-proxy, which has no labelled events, each unlabelled
+event takes a pattern of its own: 114 on otel215 and 115 on rec1155, most of them lines of
+Envoy's startup listing. The next largest are opensearch with 83 patterns, jaeger 58 and
+prometheus-server 38 on otel215, and postgresql 23 and flagd 13 on rec1155.
+[`../intent/bench/pattern_counts.py`](../intent/bench/pattern_counts.py) computes these numbers
+from the two runs' events ([otel215](../intent/results/pattern-counts-1.1.142_otel215.json),
+[rec1155](../intent/results/pattern-counts-1.1.142_rec1155.json)).
 
 ## What changed in each release, and what these captures exercise