From be8af1e243a9a52ea674e758d2ded87f9d4747a7 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Tue, 13 Oct 2026 15:22:01 -0400 Subject: [PATCH 1/2] lev.program: classifiers as data, fitted on labelled cases Regex facts, typed questions and ordered rules, run deterministically: facts first, the model only for the questions the undecided rules read. fit sets the rules' cutoffs from labelled cases. POST /v1/program runs one over the wire. bench/program.clj and bench/compile.clj measure it on BGL log lines and civil_comments, with programs written by hand and by Qwen3.8-27B; results in bench/README.md. --- README.md | 31 ++++ bench/README.md | 80 +++++++++++ bench/compile.clj | 86 ++++++++++++ bench/program.clj | 93 ++++++++++++ bench/program_data.py | 64 +++++++++ bench/programs/bgl-direct.json | 125 +++++++++++++++++ bench/programs/bgl-qwen27b.json | 1 + bench/programs/bgl-regex.json | 106 ++++++++++++++ bench/programs/toxic-direct.json | 5 + bench/programs/toxic-facets.json | 13 ++ bench/programs/toxic-qwen27b.json | 1 + src/lev/program.clj | 226 ++++++++++++++++++++++++++++++ src/lev/server.clj | 31 ++++ test/lev/program_test.clj | 141 +++++++++++++++++++ test/lev/server_test.clj | 34 +++++ test/lev/test_runner.clj | 1 + 16 files changed, 1038 insertions(+) create mode 100644 bench/compile.clj create mode 100644 bench/program.clj create mode 100644 bench/program_data.py create mode 100644 bench/programs/bgl-direct.json create mode 100644 bench/programs/bgl-qwen27b.json create mode 100644 bench/programs/bgl-regex.json create mode 100644 bench/programs/toxic-direct.json create mode 100644 bench/programs/toxic-facets.json create mode 100644 bench/programs/toxic-qwen27b.json create mode 100644 src/lev/program.clj create mode 100644 test/lev/program_test.clj diff --git a/README.md b/README.md index 50d7857..5255136 100644 --- a/README.md +++ b/README.md @@ -667,6 +667,9 @@ POST /v1/route same body, questions optional -> the routing decision POST /v1/embed {"state"} or {"states": [1-256]}, "model"?/"lang"? -> {"model", "embeddings": [[unit vector] ...], "dimensions", "usage", "routing"}: an encoder's mean-pooled final states, for state similarity (rank by cosine; bench/README.md has how good it is) +POST /v1/program {"program": {...}, "state"} or {.. "states": [1-256]}, "model"?/"lang"? + -> {"label", "rule", "facts", ("answers", "model", "usage" when the model was asked)}, + or {"results": [...]}: a classifier written as data (lev.program, below) POST /v1/workflows/ {"input": , "options"?: {...}, "constraints"?: [...], "on_infeasible"?: ..., "model"?/"lang"?/"task"?} -> the systemone answer + "workflow" + the built "state"; the request's constraints @@ -817,6 +820,34 @@ The one-question conveniences and the patterns, as a library (`lev.api`, (pat/utility-choice rt incident (api/choice "Remedy?" ["restart" "rollback" "page-human"]) {:value 10 :costs {"page-human" 4}}) ``` +A classifier can also be written as data, by a person or by a model +from a sample of the inputs, and run deterministically (`lev.program`). +Regex facts are matched first. A rule decided by facts alone never calls +the model. Otherwise the encoder answers, in one call, the questions the +remaining rules read. The first rule that holds names the label. +`fit` sets the numeric cutoffs from labelled cases without asking the +model again: + +```clojure +(require '[lev.program :as prog]) +(def triage + {"labels" ["incident" "refund" "other"] + "extract" {"oom" "(?i)oom-killed|out of memory"} + "questions" {"refund" (api/noul "Does the customer ask for money back?")} + "rules" [{"when" {"fact" "oom"} "then" "incident"} + {"when" {"noul" "refund" ">=" 0.5} "then" "refund"}] + "default" "other"}) +(prog/run rt "pods OOM-killed on payments" triage) ; {"label" "incident" "rule" 0 "facts" {"oom" true}}, no model call +(prog/fit triage (map #(assoc (prog/observe rt (:text %) triage) "label" (:label %)) labelled)) +;; => {"program" "accuracy" .. "before" ..} +``` + +A condition is `{"fact" f}`, `{"noul" q ">=" p}` (or `>` `<=` `<`), +`{"score" q ">=" x}`, `{"choice" q "is" o}` / `{"choice" q "in" [..]}`, +or `all` / `any` / `not` over conditions. `"escalate" {"model" thinker}` +re-asks unsure answers on a thinker. `bench/README.md` (programs) has +what this buys on log lines and on comments. + ### As a binary `jolt binary` runs `jolt build -m lev.server -o lev-server` with the C diff --git a/bench/README.md b/bench/README.md index 758094b..6f48106 100644 --- a/bench/README.md +++ b/bench/README.md @@ -621,3 +621,83 @@ the nearest neighbour here and needs a corpus mean. The signal is real anisotropic: any two states sit between ~0.82 and ~0.95. Rank by cosine; a fixed similarity cutoff is not portable between domains. Nothing here touches an answer's numbers. + +## Programs: classifiers written as data (`lev.program`, 2026-10-13) + +Can a model write a classifier once (regex facts, typed questions, ordered +rules) that lev then runs deterministically, and does that beat asking the +model the target question directly? `bench/program.clj` scores a program +on a labelled split. `--fit` first answers every question on the train +split and fits the rules' cutoffs there (`lev.program/fit`). Build the +data with `uv run --with pyarrow python bench/program_data.py`: + +- `bgl-*` is LogHub's BGL_2k: Blue Gene/L RAS log lines, 12 hand-labelled + alert categories plus normal, alternated into halves (1,004 train, 996 + test). +- `toxic-*` is google/civil_comments: 200 toxic (≥ 0.5) and 200 clean + (< 0.1) comments a half. Train comes from the validation split, test + from the test split. + +The programs are in `bench/programs/`: + +- `*-direct` asks the target as one question. +- `bgl-regex` and `toxic-facets` were written by hand from the train + half. `toxic-facets` has five facet questions (insult, identity, + threat, obscene and the direct one) plus three word-list facts. +- `*-qwen27b` were written by Qwen3.8-27B through `bench/compile.clj` + (greedy, thinking off). It saw the format, the task and up to 8 + distinct train inputs a label, and got the validator's error back: one + bad regex on toxicity, then fixed. Each program took about 4 minutes + of generation. + +Test half; macro F1 counts every label that is either gold or predicted: + +| BGL (996 lines) | accuracy | macro F1 | model calls | ms / line | +|---|---|---|---|---| +| `english`, direct 13-way choice | 0.286 | 0.309 | 996 | 227 | +| Qwen3.5-4B, direct 13-way choice | 0.741 | 0.341 | 996 | 1,858 | +| `bgl-qwen27b` (20 regex facts, no questions) | 0.990 | 0.846 | 0 | 0.1 | +| `bgl-regex` (by hand) | **1.000** | **1.000** | 0 | 0.1 | + +| toxicity (400 comments) | accuracy | macro F1 | model calls | +|---|---|---|---| +| Qwen3.5-4B, direct, cutoff 0.5 | 0.695 | 0.693 | 400 | +| Qwen3.5-4B, direct, cutoff fitted | 0.708 | 0.707 | 400 | +| `english`, direct, cutoff 0.5 | 0.755 | 0.740 | 400 | +| `toxic-qwen27b`, as written | 0.733 | 0.714 | 332 | +| `toxic-qwen27b`, cutoff fitted (0.6 → 0.34) | 0.905 | 0.905 | 332 | +| `toxic-facets`, cutoffs fitted | 0.920 | 0.920 | 400 | +| `english`, direct, cutoff fitted (0.5 → 0.33) | **0.933** | **0.932** | 400 | + +What this says: + +- **Log lines.** BGL's alerts are fixed message templates, so regex + facts decide them outright. That is perfect accuracy, no model, and + about 2,000× faster than the encoder. The encoder's direct choice is + near useless on raw log lines. The 4B thinker is better but slow, and + it misses the rare classes. Qwen3.8-27B wrote a near-perfect program + unaided. Its only misses are ten `LOGIN chdir` lines labelled normal: + its rule left out the `Input/output error` that marks the one APPOUT + alert. +- **Free text.** The cutoff matters more than anything else. The + encoder's yes/no for "toxic" sits low (it says yes to fewer than half + the toxic comments at 0.5), and fitting the one cutoff on 400 labelled + comments takes it from 0.755 to 0.933. Splitting the judgement into + facet questions and word lists fit train better (0.953 against 0.938) + but lost on test, at five questions a comment instead of one. +- **A model writing free-text programs overfits.** Qwen3.8-27B's word + list is mostly whole train phrases ("happy clients", "tennis", "okay"). + It fires on 17% of test comments and is wrong on many of them. Fitting + rescues the question's cutoff (0.905) but not the list. +- **Escalation does not help here.** Qwen3.5-4B is worse than the fitted + encoder on toxicity (0.708), so nothing was escalated to it. + +So the practical shape is: a model writes the program, it is validated +and its cutoffs are fitted on a labelled sample, and only then is it +trusted. Regex facts carry structured text such as logs. On free text, +one well-fitted question beats an elaborate decomposition. A program +should not be deployed on the model's say-so: on this data the fit and +the held-out score are what separated 0.733 from 0.905. + +Timing was taken with other benches sharing the machine, so it is +indicative only. diff --git a/bench/compile.clj b/bench/compile.clj new file mode 100644 index 0000000..152d2cb --- /dev/null +++ b/bench/compile.clj @@ -0,0 +1,86 @@ +(ns bench.compile + "A local model writes the lev.program: it sees the task, the program + format, and a sample of the train split (up to --per-label inputs a + label, distinct after digits are masked), answers greedily with JSON, + and gets lev.program/validate's error back up to twice. The program + is written to --save for bench/program.clj to fit and score. + + jolt -M bench/compile.clj --split bgl-train --task \"...\" --save FILE + [--gguf PATH | --model NAME] [--per-label 8] [--max-tokens 3000]" + (:require [clojure.string :as str] + [lev.config :as cfg] + [lev.json :as json] + [lev.llm :as llm] + [lev.program :as prog] + [lev.sequence :as seq])) + +(def format-spec + "A program is one JSON object: +{\"labels\": [label, ...], + \"extract\": {fact: regex}, a fact holds when its (Java) regex finds a match in the input + \"questions\": {qid: question}, asked to a small text encoder when a rule needs them + \"rules\": [{\"when\": condition, \"then\": label}, ...], the first rule that holds names the label + \"default\": label} when no rule holds +A question is {\"type\": \"noul\", \"instructions\": \"a yes/no question about the input\"}, +{\"type\": \"choice\", \"instructions\": \"...\", \"criteria\": {option: description}} or +{\"type\": \"score\", \"instructions\": \"...\", \"criteria\": [level, ...]}. +A condition is {\"fact\": f}, {\"noul\": qid, \">=\": p} (p the probability of yes; also > <= <), +{\"score\": qid, \">=\": x}, {\"choice\": qid, \"is\": option}, {\"choice\": qid, \"in\": [option, ...]}, +{\"all\": [condition, ...]}, {\"any\": [condition, ...]} or {\"not\": condition}. +Facts cost nothing; questions cost about 50 ms each and the encoder is weak on long or technical text, +so use a regex wherever the input's wording decides the label, and a question where meaning does. +Numeric cutoffs are refitted on labelled data afterwards.") + +(defn- sample [cases per-label] + (mapcat (fn [[label cs]] + (->> cs + (map #(get % "text")) + (reduce (fn [[seen out] t] + (let [k (str/replace t #"[0-9a-fA-F]{2,}|\d+" "N")] + (if (contains? seen k) [seen out] [(conj seen k) (conj out t)]))) + [#{} []]) + second + (take per-label) + (map (fn [t] [label t])))) + (group-by #(get % "label") cases))) + +(defn- prompt-for [task cases per-label] + (str "Write a classifier program for this task: " task "\n\n" format-spec + "\n\nLabelled examples (labelinput), from the training data:\n" + (str/join "\n" (map (fn [[l t]] (str l "\t" (subs t 0 (min 300 (count t))))) (sample cases per-label))) + "\n\nAnswer with the JSON program only.")) + +(defn- json-in [text] + (let [a (str/index-of text "{") b (str/last-index-of text "}")] + (when (and a b (< a b)) (json/read-str (subs text a (inc b)))))) + +(defn -main [& args] + (let [opts (cfg/parse-args args) + ctx (cfg/context opts) + gguf (or (get opts "--gguf") (:model (get (cfg/thinkers ctx) (get opts "--model" "qwen3.8-27b")))) + cases (map json/read-str (remove str/blank? (str/split-lines (slurp (str "bench/data/" (get opts "--split") ".jsonl"))))) + m (llm/load gguf {:n-ctx 16384 :n-seq-max 1}) + max-tokens (Long/parseLong (get opts "--max-tokens" "3000")) + ask (fn [messages] + (let [t0 (System/nanoTime) + r (llm/generate m (llm/chat-prompt m messages {:thinking false}) {:max-tokens max-tokens :temperature 0.0 :stop "<|im_end|>"})] + (println (format " %d tokens in %.1f s" (:tokens r) (/ (- (System/nanoTime) t0) 1e9))) + (:text r))) + first-msg {:role "user" :content (prompt-for (get opts "--task") cases (Long/parseLong (get opts "--per-label" "8")))}] + (println "compiling with" gguf) + (loop [messages [first-msg] tries 0] + (let [text (ask messages) + [program err] (try [(prog/validate (json-in text)) nil] + (catch Exception e [nil (ex-message e)]))] + (cond + program (do (spit (get opts "--save") (str (seq/json-str program) "\n")) + (println "valid program:" (count (get program "extract")) "facts," + (count (get program "questions")) "questions," (count (get program "rules")) "rules") + (println (seq/json-str program))) + (< tries 2) (do (println "invalid:" err) + (recur (conj messages {:role "assistant" :content text} + {:role "user" :content (str "That program is invalid: " err ". Answer with the corrected JSON program only.")}) + (inc tries))) + :else (do (println "gave up:" err) (println text) (System/exit 1))))))) + +(apply -main *command-line-args*) diff --git a/bench/program.clj b/bench/program.clj new file mode 100644 index 0000000..50251c0 --- /dev/null +++ b/bench/program.clj @@ -0,0 +1,93 @@ +(ns bench.program + "Scores a lev.program on a labelled split (bench/program_data.py builds + them): accuracy, macro F1 over the labels, how many inputs needed the + model, how many answers escalated, and the time per input. The + baselines are programs too: the target asked directly as one question + and mapped to the label by one rule per answer. + + jolt -M bench/program.clj --program bench/programs/toxic-direct.json --split toxic-test + [--model NAME] [--fit SPLIT [--save FILE]] [--limit N] [--out FILE] [--quiet] + + --fit answers every question of the program on another split (the + train half) and fits its cutoffs there (lev.program/fit) before + scoring; --save writes the fitted program. --out writes one line per + input, {id label predicted rule answers}." + (:require [lev.json :as json] + [clojure.string :as str] + [lev.config :as cfg] + [lev.program :as prog] + [lev.router :as router] + [lev.sequence :as seq])) + +(defn- read-jsonl [path] + (map json/read-str (remove str/blank? (str/split-lines (slurp path))))) + +(defn- f1s [pairs labels] + (into (sorted-map) + (map (fn [l] + (let [tp (count (filter (fn [[g p]] (and (= g l) (= p l))) pairs)) + fp (count (filter (fn [[g p]] (and (not= g l) (= p l))) pairs)) + fn* (count (filter (fn [[g p]] (and (= g l) (not= p l))) pairs)) + prec (if (pos? (+ tp fp)) (/ tp (double (+ tp fp))) 0.0) + rec (if (pos? (+ tp fn*)) (/ tp (double (+ tp fn*))) 0.0)] + [l {:f1 (if (pos? (+ prec rec)) (/ (* 2 prec rec) (+ prec rec)) 0.0) :n (+ tp fn*)}]))) + labels)) + +(defn -main [& args] + (let [opts (cfg/parse-args args) + ctx (cfg/context opts) + program (prog/validate (json/read-str (slurp (get opts "--program")))) + cases (cond->> (read-jsonl (str "bench/data/" (get opts "--split") ".jsonl")) + (get opts "--limit") (take (Long/parseLong (get opts "--limit")))) + model (get opts "--model") + rt (router/make-router {:data (cfg/setting ctx "--data" "LEV_DATA" :data "data") + :checkpoints (into {} (map (fn [n] [n (cfg/limits ctx n)])) router/names) + :calibrations (cfg/calibrations ctx) + :thinkers (cfg/thinkers ctx)}) + agent (if model (router/load-model rt model) rt) + agent (if (get program "escalate") rt agent) + program (if-let [fs (get opts "--fit")] + (let [fit-agent (if model agent (router/load-model rt "english")) + observed (mapv (fn [c] (assoc (prog/observe fit-agent (get c "text") program) "label" (get c "label"))) + (read-jsonl (str "bench/data/" fs ".jsonl"))) + r (prog/fit program observed)] + (println (format "fitted on %s (%d): accuracy %.3f -> %.3f" fs (count observed) (get r "before") (get r "accuracy"))) + (when-let [o (get opts "--save")] (spit o (str (seq/json-str (get r "program")) "\n"))) + (get r "program")) + program) + _ (prog/run agent (get (first cases) "text") program) ; warm + rows (vec (for [c cases] + (let [t0 (System/nanoTime) + r (prog/run agent (get c "text") program) + ms (/ (- (System/nanoTime) t0) 1e6)] + (when (and (not (contains? opts "--quiet")) (not= (get r "label") (get c "label"))) + (println (format " miss %s: %s -> %s %s" (get c "id") (get c "label") (get r "label") + (subs (get c "text") 0 (min 90 (count (get c "text"))))))) + {:id (get c "id") :answers (get r "answers") :rule (get r "rule") + :gold (get c "label") :pred (get r "label") :ms ms + :asked (some? (get r "answers")) + :escalated (count (get-in r ["escalation" "escalated"]))}))) + _ (when-let [o (get opts "--out")] + (spit o (apply str (map (fn [{:keys [id gold pred rule answers]}] + (str (seq/json-str (seq/ordered-map [["id" id] ["label" gold] ["predicted" pred] + ["rule" rule] ["answers" answers]])) + "\n")) + rows)))) + pairs (map (juxt :gold :pred) rows) + labels (distinct (concat (map :gold rows) (map :pred rows))) + per (f1s pairs labels) + n (count rows)] + (println (format "%s on %s (%d inputs), model %s" + (get opts "--program") (get opts "--split") n + (or model (get-in program ["escalate" "model"]) "routed"))) + (println (format " accuracy %.3f macro F1 %.3f model asked %d/%d answers escalated %d %.1f ms/input (p50 %.1f)" + (/ (count (filter (fn [[g p]] (= g p)) pairs)) (double n)) + (/ (reduce + (map :f1 (vals per))) (double (count per))) + (count (filter :asked rows)) n + (reduce + (map :escalated rows)) + (/ (reduce + (map :ms rows)) n) + (nth (vec (sort (map :ms rows))) (quot n 2)))) + (doseq [[l {:keys [f1 n]}] per] + (println (format " %-10s F1 %.3f (n %d)" l f1 n))))) + +(apply -main *command-line-args*) diff --git a/bench/program_data.py b/bench/program_data.py new file mode 100644 index 0000000..4676374 --- /dev/null +++ b/bench/program_data.py @@ -0,0 +1,64 @@ +"""Build the labelled sets bench/program.clj scores lev.program against: +real noisy inputs, each split into a train half (what a program's author +may look at) and a test half (what it is scored on). + + bench/data/bgl-{train,test}.jsonl LogHub's BGL_2k sample (Blue Gene/L + RAS log lines, hand-labelled alert + categories, "-" -> "normal"); the + lines of each label alternate between + the halves, in log order + bench/data/toxic-{train,test}.jsonl google/civil_comments: 200 toxic + (toxicity >= 0.5) and 200 non-toxic + (< 0.1) comments a half, train from + the validation split, test from the + test split, seeded + +Each line is {id text label}. + + uv run --with pyarrow python bench/program_data.py +""" +import csv, io, json, os, random, urllib.request + +import pyarrow.parquet as pq + +OUT = os.path.join(os.path.dirname(__file__), "data") +BGL = "https://raw.githubusercontent.com/logpai/loghub/master/BGL/BGL_2k.log_structured.csv" +CC = "https://huggingface.co/datasets/google/civil_comments/resolve/refs%2Fconvert%2Fparquet/default/{}/0000.parquet" + + +def write(name, rows): + with open(os.path.join(OUT, name), "w") as f: + for r in rows: + f.write(json.dumps(r, ensure_ascii=False) + "\n") + print(name, len(rows)) + + +def bgl(): + text = urllib.request.urlopen(BGL).read().decode("utf-8") + seen, halves = {}, ([], []) + for r in csv.DictReader(io.StringIO(text)): + label = "normal" if r["Label"] == "-" else r["Label"] + i = seen.get(label, 0) + seen[label] = i + 1 + line = " ".join([r["Date"], r["Node"], r["Time"], r["Type"], r["Component"], r["Level"], r["Content"]]) + halves[i % 2].append({"id": "bgl-" + r["LineId"], "text": line, "label": label}) + write("bgl-train.jsonl", halves[0]) + write("bgl-test.jsonl", halves[1]) + + +def toxic(split, out, seed): + t = pq.read_table(io.BytesIO(urllib.request.urlopen(CC.format(split)).read()), columns=["text", "toxicity"]) + rows = [r for r in t.to_pylist() if 20 <= len(r["text"]) <= 600] + rng = random.Random(seed) + pos = rng.sample([r for r in rows if r["toxicity"] >= 0.5], 200) + neg = rng.sample([r for r in rows if r["toxicity"] < 0.1], 200) + cases = [{"text": r["text"], "label": "toxic"} for r in pos] + [{"text": r["text"], "label": "ok"} for r in neg] + rng.shuffle(cases) + write(out, [dict(id=f"{split}-{i}", **c) for i, c in enumerate(cases)]) + + +if __name__ == "__main__": + os.makedirs(OUT, exist_ok=True) + bgl() + toxic("validation", "toxic-train.jsonl", 1) + toxic("test", "toxic-test.jsonl", 2) diff --git a/bench/programs/bgl-direct.json b/bench/programs/bgl-direct.json new file mode 100644 index 0000000..d38b131 --- /dev/null +++ b/bench/programs/bgl-direct.json @@ -0,0 +1,125 @@ +{ + "labels": [ + "normal", + "KERNDTLB", + "KERNSTOR", + "KERNMNTF", + "KERNTERM", + "KERNRTSP", + "KERNREC", + "APPSEV", + "APPRES", + "APPTO", + "APPREAD", + "APPCHILD", + "APPOUT" + ], + "questions": { + "category": { + "type": "choice", + "instructions": "Which alert category does this Blue Gene/L RAS log line belong to?", + "criteria": { + "normal": "routine or corrected event, no alert: core dumps being generated, alignment exceptions, corrected cache parity or ddr errors, interrupt statistics, register dumps", + "KERNDTLB": "kernel data TLB error interrupt", + "KERNSTOR": "kernel data storage interrupt", + "KERNMNTF": "Lustre file system mount failed on an I/O node", + "KERNTERM": "rts: the compute kernel terminated", + "KERNRTSP": "rts panic, execution stopped", + "KERNREC": "error receiving a packet on the tree network", + "APPSEV": "ciod: the CioStream socket link has been severed", + "APPRES": "ciod: the CioStream socket connection was reset by peer", + "APPTO": "ciod: the CioStream socket connection timed out", + "APPREAD": "ciod: failed to read the message prefix on the control stream", + "APPCHILD": "ciod: error creating the node map, no child processes", + "APPOUT": "ciod: login chdir failed with an input/output error" + } + } + }, + "rules": [ + { + "when": { + "choice": "category", + "is": "KERNDTLB" + }, + "then": "KERNDTLB" + }, + { + "when": { + "choice": "category", + "is": "KERNSTOR" + }, + "then": "KERNSTOR" + }, + { + "when": { + "choice": "category", + "is": "KERNMNTF" + }, + "then": "KERNMNTF" + }, + { + "when": { + "choice": "category", + "is": "KERNTERM" + }, + "then": "KERNTERM" + }, + { + "when": { + "choice": "category", + "is": "KERNRTSP" + }, + "then": "KERNRTSP" + }, + { + "when": { + "choice": "category", + "is": "KERNREC" + }, + "then": "KERNREC" + }, + { + "when": { + "choice": "category", + "is": "APPSEV" + }, + "then": "APPSEV" + }, + { + "when": { + "choice": "category", + "is": "APPRES" + }, + "then": "APPRES" + }, + { + "when": { + "choice": "category", + "is": "APPTO" + }, + "then": "APPTO" + }, + { + "when": { + "choice": "category", + "is": "APPREAD" + }, + "then": "APPREAD" + }, + { + "when": { + "choice": "category", + "is": "APPCHILD" + }, + "then": "APPCHILD" + }, + { + "when": { + "choice": "category", + "is": "APPOUT" + }, + "then": "APPOUT" + } + ], + "default": "normal" +} \ No newline at end of file diff --git a/bench/programs/bgl-qwen27b.json b/bench/programs/bgl-qwen27b.json new file mode 100644 index 0000000..9db254e --- /dev/null +++ b/bench/programs/bgl-qwen27b.json @@ -0,0 +1 @@ +{"labels": ["APPTO", "KERNSTOR", "KERNDTLB", "KERNTERM", "KERNREC", "APPREAD", "normal", "APPOUT", "APPRES", "KERNRTSP", "APPCHILD", "KERNMNTF", "APPSEV"], "extract": {"is_fatal": "RAS (KERNEL|APP) FATAL", "is_info": "RAS (KERNEL|APP) INFO", "is_kernel": "RAS KERNEL", "is_app": "RAS APP", "msg_ciod": "ciod:", "msg_rts": "rts:", "msg_lustre": "Lustre mount FAILED", "msg_data_storage": "data storage interrupt", "msg_data_tlb": "data TLB error interrupt", "msg_kernel_terminated": "kernel terminated", "msg_rts_panic": "rts panic!", "msg_error_receiving": "Error receiving packet", "msg_lustre_mount": "Lustre mount FAILED", "msg_conn_timeout": "Connection timed out", "msg_conn_reset": "Connection reset by peer", "msg_link_severed": "Link has been severed", "msg_failed_read_prefix": "failed to read message prefix", "msg_error_read_prefix": "Error reading message prefix", "msg_error_creating_node": "Error creating node map", "msg_login_chdir": "LOGIN chdir"}, "questions": {}, "rules": [{"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_conn_timeout"}]}, "then": "APPTO"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_conn_reset"}]}, "then": "APPRES"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_link_severed"}]}, "then": "APPSEV"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_failed_read_prefix"}]}, "then": "APPREAD"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_error_creating_node"}]}, "then": "APPCHILD"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_login_chdir"}]}, "then": "APPOUT"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_data_storage"}]}, "then": "KERNSTOR"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_data_tlb"}]}, "then": "KERNDTLB"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_rts"}, {"fact": "msg_kernel_terminated"}]}, "then": "KERNTERM"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_rts_panic"}]}, "then": "KERNRTSP"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_error_receiving"}]}, "then": "KERNREC"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_lustre_mount"}]}, "then": "KERNMNTF"}], "default": "normal"} diff --git a/bench/programs/bgl-regex.json b/bench/programs/bgl-regex.json new file mode 100644 index 0000000..08f4698 --- /dev/null +++ b/bench/programs/bgl-regex.json @@ -0,0 +1,106 @@ +{ + "labels": [ + "normal", + "KERNDTLB", + "KERNSTOR", + "KERNMNTF", + "KERNTERM", + "KERNRTSP", + "KERNREC", + "APPSEV", + "APPRES", + "APPTO", + "APPREAD", + "APPCHILD", + "APPOUT" + ], + "extract": { + "dtlb": "data TLB error interrupt", + "dstor": "data storage interrupt", + "mount": "Lustre mount FAILED", + "term": "rts: kernel terminated", + "panic": "rts panic", + "tree": "Error receiving packet on tree network", + "severed": "ciod: Error reading message prefix.*Link has been severed", + "reset": "ciod: Error reading message prefix.*Connection reset by peer", + "timeout": "ciod: Error reading message prefix.*Connection timed out", + "prefix": "ciod: failed to read message prefix on control stream", + "child": "ciod: Error creating node map.*No child processes", + "chdir": "ciod: LOGIN chdir.*failed: Input/output error" + }, + "rules": [ + { + "when": { + "fact": "dtlb" + }, + "then": "KERNDTLB" + }, + { + "when": { + "fact": "dstor" + }, + "then": "KERNSTOR" + }, + { + "when": { + "fact": "mount" + }, + "then": "KERNMNTF" + }, + { + "when": { + "fact": "term" + }, + "then": "KERNTERM" + }, + { + "when": { + "fact": "panic" + }, + "then": "KERNRTSP" + }, + { + "when": { + "fact": "tree" + }, + "then": "KERNREC" + }, + { + "when": { + "fact": "severed" + }, + "then": "APPSEV" + }, + { + "when": { + "fact": "reset" + }, + "then": "APPRES" + }, + { + "when": { + "fact": "timeout" + }, + "then": "APPTO" + }, + { + "when": { + "fact": "prefix" + }, + "then": "APPREAD" + }, + { + "when": { + "fact": "child" + }, + "then": "APPCHILD" + }, + { + "when": { + "fact": "chdir" + }, + "then": "APPOUT" + } + ], + "default": "normal" +} \ No newline at end of file diff --git a/bench/programs/toxic-direct.json b/bench/programs/toxic-direct.json new file mode 100644 index 0000000..3c37bbb --- /dev/null +++ b/bench/programs/toxic-direct.json @@ -0,0 +1,5 @@ +{"labels": ["toxic", "ok"], + "questions": {"toxic": {"type": "noul", + "instructions": "Is this comment toxic: rude, disrespectful or unreasonable enough to make someone leave a discussion?"}}, + "rules": [{"when": {"noul": "toxic", ">=": 0.5}, "then": "toxic"}], + "default": "ok"} diff --git a/bench/programs/toxic-facets.json b/bench/programs/toxic-facets.json new file mode 100644 index 0000000..1fdab16 --- /dev/null +++ b/bench/programs/toxic-facets.json @@ -0,0 +1,13 @@ +{"labels": ["toxic", "ok"], + "extract": {"insult_words": "(?i)\\b(idiot\\w*|i-diot|stupid\\w*|moron\\w*|dumb\\w*|fools?|foolish|clowns?|trolls?|liars?|hypocrit\\w*|pathetic|ignorant|ignorance|imbecile\\w*|rubbish|garbage|trash|losers?|scum\\w*|gullible|incompetent|flakes?|lunatic\\w*|nuts?|crazy|insane|sick|disgusting|vile|silliness|nonsense|bigot\\w*|racists?|sexist|whiners?|sheep|cowards?)\\b", + "profanity": "(?i)(\\bf+u+c+k\\w*|\\bsh[i1]t\\w*|\\bcrap\\w*|\\bdamn\\w*|\\bhell\\b|\\bass(es|hole\\w*)?\\b|\\bbutt\\b|\\bbitch\\w*|\\bbastard\\w*|\\bpiss\\w*|\\bb\\.\\.\\.s|\\bfreaking\\b)", + "violence_words": "(?i)\\b(kill\\w*|die|dead|shoot\\w*|bombs?|wipe \\w+ out|rot in|hang\\w*|trigger|violent)\\b"}, + "questions": {"toxic": {"type": "noul", "instructions": "Is this comment toxic: rude, disrespectful or unreasonable enough to make someone leave a discussion?"}, + "insult": {"type": "noul", "instructions": "Does the comment insult, mock or call names at a person or a group?"}, + "identity": {"type": "noul", "instructions": "Does the comment demean people for their race, religion, gender, sexuality, nationality or politics?"}, + "threat": {"type": "noul", "instructions": "Does the comment threaten or wish violence or death on someone?"}, + "obscene": {"type": "noul", "instructions": "Does the comment use profanity or obscene language, even partly censored?"}}, + "rules": [{"when": {"any": [{"noul": "toxic", ">=": 0.5}, {"noul": "insult", ">=": 0.5}, {"noul": "identity", ">=": 0.5}, + {"noul": "threat", ">=": 0.5}, {"noul": "obscene", ">=": 0.5}]}, "then": "toxic"}, + {"when": {"any": [{"fact": "insult_words"}, {"fact": "profanity"}, {"fact": "violence_words"}]}, "then": "toxic"}], + "default": "ok"} diff --git a/bench/programs/toxic-qwen27b.json b/bench/programs/toxic-qwen27b.json new file mode 100644 index 0000000..b0958a1 --- /dev/null +++ b/bench/programs/toxic-qwen27b.json @@ -0,0 +1 @@ +{"labels": ["ok", "toxic"], "extract": {"insult_regex": "(?i)(\\b(idiot|stupid|dumb|fool|moron|retard|retarded|bozo|punk|loser|dork|dorks|sheep|clown|clowns|puppet|puppets|degenerate|rabid|crazy|brain washed|brainwashed|ignorant|full of himself|full of shit|full of crap|absolute rubbish|save the stupidity|get some help|give it up|exit now|who cares|who cares!?|never get laid|malicious|malignant|bozos|punk|dorks|sheep of left wing dumb|clown show|puppets controlled|dancing in place|renouncing violence|violence instigatores|planted at those events|get over it|you and the left|left wing dumb|trump didnt encourage|expell the violence|nominate|coup regime|nazi elements|blow us all sky high|doubling down|support for the coup|anti-russian views|shares the same rabid|lied about her grandfather|successful business|happy clients|referral keep me rather overly busy|selling the herb|plain old hard work|manufacturing a wante|pot head|consider me a|your comment is|absolute rubbish|stronger defensive team|zairsev alone|better defensive team|lack the|josée chouinard|tennis|andy murray|nova sc|slainte mhath|arrest has been made|dailyworld|donkey creek|fatal hit and run|first nations|thousands of years|without our interference|wise parents|graduates|sorry bud|no matter how you spin it|stretch of imagination|liberal disaster|ontario has endured|improvement|perspective from his native province|motsditssansdessein|ce tu wla gwassideurw|okay|had not heard that|military court martial|felony conviction|gap|dv conviction|dv incident|court martial|specifically for dv|legal deta|npc|defensive team|rookies|running around less|own zone|better positioned|heck|bozak|lack the|bernstein is a degenerate bozo|who cares what this full of himself punk thinks|bernstein and 'his crowd' are the malignant ones|bb, your comment is absolute rubbish|if you consider me a 'pot head'|then how is it that i have a successful business|long list of happy clients|by referral keep me rather overly busy|and no, it has nothing to do with selling the herb|but just plain old hard work manufacturing a wante|but idiots with a more diverse and healthier economy|she's lied about her grandfather|because she shares the same rabid, anti-russian views|it matters because canada is doubling down on its support for the coup regime in the ukraine|that contains nazi elements and is spoiling to draw nato into a conflict with russia|that could blow us all sky high|note to \"chopper bob\"|i would state that what the gop lead legislature has done over the past several years is also (very) insignificant|actually it's been a clown show of puppets controlled by their corporate puppeteers-dancing in place !|oh save the stupidity and lies again sheep of left wing dumb|trump didnt encourage anything other than to expell the violence instigatores planted at those events|get over it...|you and the left have to start renouncing violence since it is the left that has been doing it since trump was nominate|you are absolutely crazy..|trump is informed but you are brain washed ignorant,,|please get some help for your stupidity see loser,|who cares!?|stark trek and star wars fans are all dorks who never get laid))"}, "questions": {"q_toxic_intent": {"type": "noul", "instructions": "Does the comment contain language that is rude, disrespectful, or unreasonable enough to make someone leave a discussion? This includes insults, personal attacks, hate speech, or aggressive dismissal of others' views."}}, "rules": [{"when": {"fact": "insult_regex"}, "then": "toxic"}, {"when": {"noul": "q_toxic_intent", ">=": 0.6}, "then": "toxic"}], "default": "ok"} diff --git a/src/lev/program.clj b/src/lev/program.clj new file mode 100644 index 0000000..4a3d3fe --- /dev/null +++ b/src/lev/program.clj @@ -0,0 +1,226 @@ +(ns lev.program + "A classifier written as data, to be authored once (by a person or a + large model, from a sample of the inputs) and run deterministically on + every input: + + {\"labels\" [label ...] optional; rules are checked against it + \"extract\" {fact regex} a fact holds when its regex finds a match + \"questions\" {qid question} typed questions (choice / score / noul) + \"rules\" [{\"when\" condition \"then\" label} ...] + \"default\" label when no rule fires + \"escalate\" {\"model\" thinker \"threshold\" t}} optional; needs a router + + A condition is one of + {\"fact\" f} + {\"noul\" qid op x} op one of >= > <= <, on the probability of yes + {\"score\" qid op x} on the expected level index + {\"choice\" qid \"is\" option} / {\"choice\" qid \"in\" [option ...]} + {\"all\" [c ...]} / {\"any\" [c ...]} / {\"not\" c} + + `run` matches the facts first and evaluates the rules in order with the + answers unknown: when the first rule that is not false is decided by the + facts, no model is called. Otherwise the agent answers, in one call, + the questions the rules from there on read, and the first true rule + names the label. With \"escalate\" the answers below the threshold are + re-asked on the thinker (lev.patterns/escalate)." + (:require [lev.agent :as ag] + [lev.api :as api] + [lev.patterns :as pat] + [lev.sequence :as seq])) + +(def ^:private ops {">=" >= ">" > "<=" <= "<" <}) + +(defn- bad [field msg] + (throw (ex-info (str field ": " msg) {:type :invalid-request :field field}))) + +(defn- qtype [program qid] (get-in program ["questions" qid "type"])) + +(defn- options [program qid] + (set (map (fn [k] (if (keyword? k) (name k) (str k))) + (keys (:crit (ag/to-internal (get-in program ["questions" qid]))))))) + +(defn- check-condition [program field c] + (let [fail #(bad field (str % ": " (pr-str c))) + compare-ok (fn [t] + (let [qid (get c t) + op (first (filter #(contains? c %) (keys ops)))] + (when-not (= t (qtype program qid)) (fail (str "no " t " question " (pr-str qid)))) + (when-not (and op (number? (get c op))) (fail "needs one of >= > <= < with a number"))))] + (when-not (map? c) (fail "a condition is an object")) + (cond + (contains? c "fact") (when-not (contains? (get program "extract") (get c "fact")) + (fail (str "no extracted fact " (pr-str (get c "fact"))))) + (contains? c "noul") (compare-ok "noul") + (contains? c "score") (compare-ok "score") + (contains? c "choice") (let [qid (get c "choice")] + (when-not (= "choice" (qtype program qid)) (fail (str "no choice question " (pr-str qid)))) + (let [os (cond (contains? c "is") [(get c "is")] + (sequential? (get c "in")) (get c "in") + :else (fail "a choice condition needs \"is\" or \"in\""))] + (doseq [o os] + (when-not (contains? (options program qid) o) (fail (str "unknown option " (pr-str o))))))) + (contains? c "all") (if (sequential? (get c "all")) (doseq [x (get c "all")] (check-condition program field x)) (fail "\"all\" takes a list")) + (contains? c "any") (if (sequential? (get c "any")) (doseq [x (get c "any")] (check-condition program field x)) (fail "\"any\" takes a list")) + (contains? c "not") (check-condition program field (get c "not")) + :else (fail "unknown condition")))) + +(defn validate + "The program, or {:type :invalid-request :field path} naming the first + thing wrong with it: a regex that does not compile, a question lev + would refuse, a rule reading a fact or question that is not there (or + a question of another type, or an option it does not have), a label + outside \"labels\", a missing \"default\"." + [program] + (when-not (map? program) (bad "program" "must be an object")) + (doseq [[f re] (get program "extract")] + (try (re-pattern re) (catch Exception e (bad (str "extract." f) (str "bad regex: " (ex-message e)))))) + (doseq [[qid qdef] (get program "questions")] + (try (ag/validate-question qid qdef) + (catch Exception e (bad (str "questions." qid) (ex-message e))))) + (let [labels (get program "labels") + label-ok (fn [field l] + (when-not (string? l) (bad field "a label is a string")) + (when (and labels (not (some #{l} labels))) (bad field (str (pr-str l) " is not in labels"))))] + (when-not (sequential? (get program "rules" [])) (bad "rules" "must be a list")) + (doseq [[i r] (map-indexed vector (get program "rules"))] + (check-condition program (str "rules[" i "].when") (get r "when")) + (label-ok (str "rules[" i "].then") (get r "then"))) + (when-not (contains? program "default") (bad "default" "is required")) + (label-ok "default" (get program "default"))) + (when-let [e (get program "escalate")] + (when-not (string? (get e "model")) (bad "escalate.model" "names the thinker"))) + program) + +;; --- evaluation: true, false or ::unknown (an answer not asked yet) ---------- + +(defn- holds [c facts answers] + (let [answer (fn [qid] (get answers qid))] + (cond + (contains? c "fact") (boolean (get facts (get c "fact"))) + (contains? c "all") (let [vs (map #(holds % facts answers) (get c "all"))] + (cond (some false? vs) false (some #{::unknown} vs) ::unknown :else true)) + (contains? c "any") (let [vs (map #(holds % facts answers) (get c "any"))] + (cond (some true? vs) true (some #{::unknown} vs) ::unknown :else false)) + (contains? c "not") (let [v (holds (get c "not") facts answers)] (if (= ::unknown v) v (not v))) + :else + (let [t (first (filter #(contains? c %) ["noul" "score" "choice"]))] + (if-let [a (answer (get c t))] + (if (= t "choice") + (contains? (set (if (contains? c "is") [(get c "is")] (get c "in"))) (get a "choice")) + (let [op (first (filter #(contains? c %) (keys ops)))] + ((ops op) (double (get a t)) (double (get c op))))) + ::unknown))))) + +(defn- first-open + "[index value] of the first rule that is not false, or nil." + [rules facts answers] + (first (keep-indexed (fn [i r] (let [v (holds (get r "when") facts answers)] (when-not (false? v) [i v]))) rules))) + +(defn- reads [c] + (cond + (contains? c "all") (mapcat reads (get c "all")) + (contains? c "any") (mapcat reads (get c "any")) + (contains? c "not") (reads (get c "not")) + :else (keep #(get c %) ["noul" "score" "choice"]))) + +(defn facts + "{fact matched?} for the input: a string as it is, anything else as JSON." + [program input] + (let [text (if (string? input) input (seq/json-str input))] + (into {} (map (fn [[f re]] [f (some? (re-find (re-pattern re) text))])) (get program "extract")))) + +(defn label + "[label rule] for the facts and answers (nil answers: none asked): the + first rule that holds, or [default nil]. Pure; what `run` decides + with, and what `fit` replays." + [program facts answers] + (let [rules (get program "rules" [])] + (or (first (keep-indexed (fn [i r] (when (true? (holds (get r "when") facts answers)) [(get r "then") i])) rules)) + [(get program "default") nil]))) + +(defn run + "The program's label for one input on an agent or router: an ordered + map {label rule facts}, rule the index of the rule that fired (nil: + the default); when the model was asked, its answers, model and usage + follow (and the escalation report when there was one). Validates the + program first. opts go to lev.api/system-one (:model :lang on a + router; with escalation, :model is the fast one)." + ([agent input program] (run agent input program nil)) + ([agent input program opts] + (let [program (validate program) + rules (get program "rules" []) + fs (facts program input) + [i v] (first-open rules fs nil) + done (fn [answers tail] + (let [[l r] (label program fs answers)] + (seq/ordered-map (concat [["label" l] ["rule" r] ["facts" fs]] tail))))] + (if (not= ::unknown v) + (done nil nil) + (let [needed (set (mapcat #(reads (get % "when")) (subvec (vec rules) i))) + questions (seq/ordered-map (filter (fn [[qid _]] (contains? needed qid)) (get program "questions"))) + esc (get program "escalate") + resp (if esc + (pat/escalate agent input questions {:model (get esc "model") :threshold (get esc "threshold") + :fast-model (:model opts) :lang (:lang opts)}) + (api/system-one agent input questions opts)) + answers (get resp "answers")] + (done answers + (concat [["answers" answers] ["model" (get resp "model")] ["usage" (get resp "usage")]] + (when esc [["escalation" (get resp "escalation")]])))))))) + +(defn observe + "{facts answers} for one input with every question of the program + answered (no escalation): what `fit` replays." + [agent input program] + (let [program (validate program)] + (seq/ordered-map [["facts" (facts program input)] + ["answers" (when (seq (get program "questions")) + (get (api/system-one agent input (get program "questions")) "answers"))]]))) + +;; --- fitting the cutoffs on labelled cases ------------------------------------ + +(defn- cutoffs + "[path-to-the-number candidates] for every noul / score comparison in + the rules: a noul's cutoffs 0.01..0.99, a score's 0..(levels - 1) by 0.05." + [program] + (letfn [(walk [path c] + (cond + (contains? c "all") (mapcat #(walk (conj path "all" %1) %2) (range) (get c "all")) + (contains? c "any") (mapcat #(walk (conj path "any" %1) %2) (range) (get c "any")) + (contains? c "not") (walk (conj path "not") (get c "not")) + (or (contains? c "noul") (contains? c "score")) + (let [op (first (filter #(contains? c %) (keys ops))) + cands (if (contains? c "noul") + (map #(ag/round4 (/ % 100.0)) (range 1 100)) + (let [top (dec (count (get-in program ["questions" (get c "score") "criteria"])))] + (map #(ag/round4 (* % 0.05)) (range 0 (inc (* 20 top))))))] + [[(conj path op) cands]]) + :else nil))] + (vec (mapcat (fn [i r] (walk ["rules" i "when"] (get r "when"))) (range) (get program "rules"))))) + +(defn- accuracy [program cases] + (/ (count (filter (fn [c] (= (get c "label") (first (label program (get c "facts") (get c "answers"))))) cases)) + (double (max 1 (count cases))))) + +(defn fit + "The program's numeric cutoffs fitted to labelled cases ({facts answers + label}, as `observe` answers plus the gold label): coordinate ascent, + three passes over every noul / score comparison, each moved to the + median of the cutoffs that label the most cases right (kept where it + already is one of them). Deterministic; no model is asked. Answers + {program accuracy before}." + [program cases] + (let [program (validate program) + cases (vec cases) + points (cutoffs program) + step (fn [p [path cands]] + (let [scored (mapv (fn [x] [x (accuracy (assoc-in p path x) cases)]) cands) + top (reduce max (map second scored)) + best (mapv first (filter #(= top (second %)) scored))] + (if (some #{(get-in p path)} best) + p + (assoc-in p path (nth best (quot (dec (count best)) 2)))))) + fitted (reduce (fn [p _] (reduce step p points)) program (range 3))] + (seq/ordered-map [["program" fitted] + ["accuracy" (ag/round4 (accuracy fitted cases))] + ["before" (ag/round4 (accuracy program cases))]]))) diff --git a/src/lev/server.clj b/src/lev/server.clj index 397e3d5..2662e7e 100644 --- a/src/lev/server.clj +++ b/src/lev/server.clj @@ -23,6 +23,11 @@ \"embeddings\" [[unit vector] ...] \"dimensions\" \"usage\" \"routing\"}: the encoder's mean-pooled final states (lev.agent/embed; bench/README.md, embeddings) + POST /v1/program {\"program\" (lev.program), \"state\"} or {.. \"states\" [...]}, + \"model\"? \"lang\"? -> {\"label\" \"rule\" \"facts\" (\"answers\" + \"model\" \"usage\" when the model was asked)}, or + {\"results\" [...]} for states: a classifier written + as data (regex facts, questions, ordered rules) POST /v1/route same body, questions optional -> the routing decision alone, nothing loaded or run POST /v1/patterns/confidence-gate systemone body + \"threshold\"? -> {\"automatic\" \"escalate\" \"response\"} @@ -83,6 +88,7 @@ [lev.llm] [lev.mcp :as mcp] [lev.patterns :as pat] + [lev.program :as prog] [lev.router :as router] [lev.sequence :as seq] [lev.tokenizer :as tk] @@ -362,6 +368,29 @@ (json-response 200 (update out "embeddings" (fn [es] (mapv (fn [e] (mapv #(/ (Math/round (* 1e6 (double %))) 1e6) e)) es)))))))) +(defn run-program + "POST /v1/program: {\"program\" {...}} and {\"state\"} or {\"states\" + [...]} (1-256), routed like /v1/systemone -> lev.program/run's map, or + {\"results\" [...]} in order for states. A bad program is a 422 at + [\"body\" \"program.\"]." + [rt lock body] + (let [details (if-not (map? body) + [(detail ["body"] "request body must be a JSON object" "type_error")] + (vec (concat (when-not (map? (get body "program")) + [(detail ["body" "program"] "program is required: an object (lev.program)" "value_error")]) + (if (contains? body "states") (check-states body) (check-state body)) + (check-routing-fields rt body))))] + (if (seq details) + (unprocessable details) + (let [p (try (prog/validate (get body "program")) + (catch Exception e + (throw (ex-info (ex-message e) (update (ex-data e) :field #(str "program." %)))))) + {:keys [model lang]} (routing-opts body) + one (fn [x] (prog/run rt x p {:model model :lang lang}))] + (json-response 200 (if (contains? body "states") + {"results" (locking lock (mapv one (get body "states")))} + (locking lock (one (get body "state"))))))))) + (defn route-only "POST /v1/route: the decision for a body, without loading or running." [rt body] @@ -542,6 +571,8 @@ :response (guard (json-in #(systemone-batch rt lock %) false))} {:path "/v1/embed" :method :post :response (guard (json-in #(embed rt lock %) false))} + {:path "/v1/program" :method :post + :response (guard (json-in #(run-program rt lock %) false))} {:path "/v1/route" :method :post :response (guard (json-in #(route-only rt %) false))} {:path "/v1/models" :method :get :response (guard (fn [_] (models rt)))} diff --git a/test/lev/program_test.clj b/test/lev/program_test.clj new file mode 100644 index 0000000..76ffa0e --- /dev/null +++ b/test/lev/program_test.clj @@ -0,0 +1,141 @@ +(ns lev.program-test + "lev.program: a classifier written as data (regex facts, typed + questions, ordered rules) and run deterministically. Canned agents, no + model loaded." + (:require [clojure.test :refer [deftest is testing]] + [lev.agent :as ag] + [lev.api :as api] + [lev.program :as prog] + [lev.router :as router] + [lev.sequence :as seq])) + +(def calls (atom [])) + +(defmethod ag/system-one* :canned-program + [{:keys [name table]} _state questions _opts] + (swap! calls conj [name (vec (map (fn [[k _]] k) questions))]) + (seq/ordered-map + [["model" name] + ["answers" (seq/ordered-map + (map (fn [[qid qdef]] + (let [q (ag/to-internal (ag/validate-question qid qdef)) + p (or (get table qid) (throw (ex-info (str "no answer for " qid) {})))] + [qid (ag/typed-answer q p (count p) nil nil)])) + questions))] + ["usage" (array-map "input_tokens" (* 10 (count questions)) "output_tokens" 0)]])) + +(defn- canned [name table] {:kind :canned-program :name name :table table}) + +(def program + {"labels" ["oom" "abuse" "refund" "ok"] + "extract" {"oom" "(?i)out of memory|oom-killed"} + "questions" (array-map + "insult" (api/noul "Does the text insult someone?") + "intent" (api/choice "What does the writer want?" (array-map "refund" nil "help" nil "other" nil)) + "anger" (api/score "How angry is the writer?" ["calm" "annoyed" "furious"])) + "rules" [{"when" {"fact" "oom"} "then" "oom"} + {"when" {"any" [{"noul" "insult" ">=" 0.7} + {"score" "anger" ">=" 1.5}]} "then" "abuse"} + {"when" {"all" [{"choice" "intent" "is" "refund"} + {"not" {"noul" "insult" ">=" 0.5}}]} "then" "refund"}] + "default" "ok"}) + +(def calm-refund {"insult" [0.9 0.1] "intent" [0.8 0.1 0.1] "anger" [0.7 0.2 0.1]}) + +(deftest a-fact-rule-decides-without-the-model + (reset! calls []) + (let [r (prog/run (canned "fast" calm-refund) "pod oom-killed at 3am" program)] + (is (= "oom" (get r "label"))) + (is (= 0 (get r "rule"))) + (is (= {"oom" true} (get r "facts"))) + (is (nil? (get r "answers"))) + (is (= [] @calls)))) + +(deftest the-model-answers-what-the-rules-still-need + (reset! calls []) + (let [r (prog/run (canned "fast" calm-refund) "I want my money back" program)] + (is (= "refund" (get r "label"))) + (is (= 2 (get r "rule"))) + (is (= ["label" "rule" "facts" "answers" "model" "usage"] (vec (keys r)))) + (is (= 0.8 (get-in r ["answers" "intent" "probabilities" "refund"]))) + (testing "one call, only the questions rules from the first undecided one on read" + (is (= [["fast" ["insult" "intent" "anger"]]] @calls)))) + (testing "a later rule's questions only" + (reset! calls []) + (let [p (assoc program "rules" [{"when" {"fact" "oom"} "then" "oom"} + {"when" {"choice" "intent" "in" ["refund" "help"]} "then" "refund"}])] + (prog/run (canned "fast" calm-refund) "hello" p) + (is (= [["fast" ["intent"]]] @calls))))) + +(deftest rules-fire-in-order-and-the-default-closes + (let [angry {"insult" [0.2 0.8] "intent" [0.8 0.1 0.1] "anger" [0.1 0.2 0.7]} + calm-other {"insult" [0.9 0.1] "intent" [0.1 0.1 0.8] "anger" [0.9 0.05 0.05]}] + (is (= ["abuse" 1] ((juxt #(get % "label") #(get % "rule")) (prog/run (canned "f" angry) "x" program)))) + (let [r (prog/run (canned "f" calm-other) "x" program)] + (is (= "ok" (get r "label"))) + (is (nil? (get r "rule")))))) + +(deftest a-program-without-questions-never-calls + (reset! calls []) + (let [p {"extract" {"tlb" "data TLB error"} "rules" [{"when" {"fact" "tlb"} "then" "KERNDTLB"}] "default" "normal"}] + (is (= "KERNDTLB" (get (prog/run (canned "f" {}) "data TLB error interrupt" p) "label"))) + (is (= "normal" (get (prog/run (canned "f" {}) "all quiet" p) "label"))) + (is (= [] @calls)))) + +(deftest a-bad-program-is-refused-before-anything-runs + (let [field (fn [p] (try (prog/validate p) nil + (catch Exception e + (is (= :invalid-request (:type (ex-data e)))) + (:field (ex-data e)))))] + (is (= program (prog/validate program))) + (is (= "extract.oom" (field (assoc-in program ["extract" "oom"] "(unclosed")))) + (is (= "rules[0].when" (field (assoc-in program ["rules" 0 "when"] {"fact" "nope"})))) + (is (= "rules[1].when" (field (assoc-in program ["rules" 1 "when"] {"noul" "intent" ">=" 0.5})))) + (is (= "rules[2].when" (field (assoc-in program ["rules" 2 "when"] {"choice" "intent" "is" "nope"})))) + (is (= "rules[1].when" (field (assoc-in program ["rules" 1 "when"] {"noul" "insult" ">=" "high"})))) + (is (= "rules[1].when" (field (assoc-in program ["rules" 1 "when"] {"what" 1})))) + (is (= "rules[0].then" (field (assoc-in program ["rules" 0 "then"] "unlisted")))) + (is (= "default" (field (dissoc program "default")))) + (is (= "questions.insult" (field (assoc-in program ["questions" "insult"] {"type" "noul"})))))) + +(deftest unsure-answers-escalate-to-a-thinker + (reset! calls []) + (let [fast (canned "fast" {"insult" [0.4 0.6] "intent" [0.98 0.01 0.01] "anger" [0.98 0.01 0.01]}) + slow (canned "slow" {"insult" [0.05 0.95]}) + rt (router/make-router {:loader (fn [name & _] (if (= name "english") fast (throw (ex-info "no" {})))) + :thinkers {"slow" {:model "x.gguf"}} + :thinker-loader (fn [_ _] slow)}) + r (prog/run rt "x" (assoc program "escalate" {"model" "slow" "threshold" 0.8}))] + (is (= "abuse" (get r "label"))) + (is (= ["insult"] (get-in r ["escalation" "escalated"]))) + (is (= 0.95 (get-in r ["answers" "insult" "noul"]))) + (is (= [["fast" ["insult" "intent" "anger"]] ["slow" ["insult"]]] @calls)))) + +(deftest label-is-the-rules-over-facts-and-answers + (let [a (fn [table] (get (prog/run (canned "f" table) "x" program) "answers")) + angry (a {"insult" [0.2 0.8] "intent" [0.8 0.1 0.1] "anger" [0.1 0.2 0.7]})] + (is (= ["abuse" 1] (prog/label program {"oom" false} angry))) + (is (= ["oom" 0] (prog/label program {"oom" true} angry))) + (is (= ["ok" nil] (prog/label {"rules" [] "default" "ok"} {} nil))))) + +(deftest observe-answers-every-question + (reset! calls []) + (let [o (prog/observe (canned "fast" calm-refund) "pod oom-killed" program)] + (is (= {"oom" true} (get o "facts"))) + (is (= ["insult" "intent" "anger"] (vec (keys (get o "answers"))))) + (is (= [["fast" ["insult" "intent" "anger"]]] @calls)))) + +(deftest fit-moves-each-cutoff-to-the-middle-of-its-best-range + (let [p {"questions" {"toxic" (api/noul "Toxic?")} + "rules" [{"when" {"any" [{"noul" "toxic" ">=" 0.5}]} "then" "toxic"}] + "default" "ok"} + case* (fn [prob label] {"facts" {} "answers" {"toxic" {"type" "noul" "noul" prob}} "label" label}) + cases [(case* 0.6 "toxic") (case* 0.35 "toxic") (case* 0.25 "ok") (case* 0.1 "ok")] + r (prog/fit p cases)] + (is (= 0.75 (get r "before"))) + (is (= 1.0 (get r "accuracy"))) + (testing "the best cutoffs are 0.26..0.35; the median of that run is kept" + (is (= 0.3 (get-in r ["program" "rules" 0 "when" "any" 0 ">="])))) + (is (= (prog/validate (get r "program")) (get r "program"))) + (testing "nothing to gain: the program is unchanged" + (is (= p (get (prog/fit p [(case* 0.9 "toxic") (case* 0.1 "ok")]) "program")))))) diff --git a/test/lev/server_test.clj b/test/lev/server_test.clj index ca160be..755813b 100644 --- a/test/lev/server_test.clj +++ b/test/lev/server_test.clj @@ -525,6 +525,40 @@ (is (= ["body" "model"] (loc (second (post {"state" "a" "model" "nope"}))))) (is (= 405 (first (call h (req :get "/v1/embed")))))))) +(deftest programs-on-the-wire + (let [rt (router/preloaded @agent "english" {}) + h (srv/handler rt {:workflows @workflows}) + post (fn [body] (call h (req :post "/v1/program" :body (seq/json-str body)))) + loc (fn [b] (get-in b ["detail" 0 "loc"])) + p {"extract" {"oom" "(?i)oom-killed|out of memory"} + "questions" {"refund" {"type" "noul" "instructions" "Does the customer ask for money back?"}} + "rules" [{"when" {"fact" "oom"} "then" "incident"} + {"when" {"noul" "refund" ">=" 0.5} "then" "refund"}] + "default" "other"}] + (testing "a fact decides without the model" + (let [[st b] (post {"program" p "state" "pods OOM-killed on payments"})] + (is (= 200 st) (pr-str b)) + (is (= ["label" "rule" "facts"] (keys b))) + (is (= "incident" (get b "label"))))) + (testing "otherwise the encoder answers the questions the rules read" + (let [[st b] (post {"program" p "state" "I was charged twice, please refund the duplicate charge"})] + (is (= 200 st)) + (is (= "refund" (get b "label"))) + (is (= "english" (get b "model"))))) + (testing "several states" + (let [[st b] (post {"program" p "states" ["out of memory again" "hello there"]})] + (is (= 200 st)) + (is (= ["incident" 0] ((juxt #(get % "label") #(get % "rule")) (first (get b "results"))))) + (is (= 2 (count (get b "results")))))) + (testing "validation" + (is (= ["body" "program"] (loc (second (post {"state" "a"}))))) + (is (= ["body" "state"] (loc (second (post {"program" p}))))) + (is (= ["body" "program.rules[0].when"] + (loc (second (post {"program" (assoc-in p ["rules" 0 "when"] {"fact" "nope"}) "state" "a"}))))) + (is (= ["body" "program.extract.oom"] + (loc (second (post {"program" (assoc-in p ["extract" "oom"] "(") "state" "a"}))))) + (is (= 405 (first (call h (req :get "/v1/program")))))))) + (deftest an-engine-thinker-is-listed-as-not-thinking ;; Clef and TinyDecide generate nothing: there is no thought to switch on (let [rt (router/make-router {:thinkers {"clef" {:engine "clef" :model "target/no-such.gguf" :head "target/nope"} diff --git a/test/lev/test_runner.clj b/test/lev/test_runner.clj index 0d1ea11..a9ea194 100644 --- a/test/lev/test_runner.clj +++ b/test/lev/test_runner.clj @@ -13,6 +13,7 @@ lev.clef-test lev.tinydecide-test lev.patterns-test + lev.program-test lev.calibrate-test workflows.email-test workflows.presets-test From 043b2c2cd738e98eb9ef7b29a4372b560ccc58d7 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Tue, 13 Oct 2026 15:22:22 -0400 Subject: [PATCH 2/2] bench/README: correct the programs section against the data --- bench/README.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/bench/README.md b/bench/README.md index 6f48106..02b80b1 100644 --- a/bench/README.md +++ b/bench/README.md @@ -680,15 +680,18 @@ What this says: its rule left out the `Input/output error` that marks the one APPOUT alert. - **Free text.** The cutoff matters more than anything else. The - encoder's yes/no for "toxic" sits low (it says yes to fewer than half - the toxic comments at 0.5), and fitting the one cutoff on 400 labelled + encoder's yes/no for "toxic" sits low (at 0.5 it says yes to only 103 + of the 200 toxic comments), and fitting the one cutoff on 400 labelled comments takes it from 0.755 to 0.933. Splitting the judgement into facet questions and word lists fit train better (0.953 against 0.938) but lost on test, at five questions a comment instead of one. -- **A model writing free-text programs overfits.** Qwen3.8-27B's word +- **A model writing free-text programs memorises.** Qwen3.8-27B's word list is mostly whole train phrases ("happy clients", "tennis", "okay"). - It fires on 17% of test comments and is wrong on many of them. Fitting - rescues the question's cutoff (0.905) but not the list. + On train it labels clean comments toxic: the program scores 0.705 + there, below the bare question. On test it fires on 68 comments, 65 of + them toxic, so it is precise but covers little. The as-written program + loses on the cutoff it picked (0.6). Fitted to 0.34 it reaches 0.905, + still below the single fitted question. - **Escalation does not help here.** Qwen3.5-4B is worse than the fitted encoder on toxicity (0.708), so nothing was escalated to it.