diff --git a/README.md b/README.md index 50d7857..5255136 100644 --- a/README.md +++ b/README.md @@ -667,6 +667,9 @@ POST /v1/route same body, questions optional -> the routing decision POST /v1/embed {"state"} or {"states": [1-256]}, "model"?/"lang"? -> {"model", "embeddings": [[unit vector] ...], "dimensions", "usage", "routing"}: an encoder's mean-pooled final states, for state similarity (rank by cosine; bench/README.md has how good it is) +POST /v1/program {"program": {...}, "state"} or {.. "states": [1-256]}, "model"?/"lang"? + -> {"label", "rule", "facts", ("answers", "model", "usage" when the model was asked)}, + or {"results": [...]}: a classifier written as data (lev.program, below) POST /v1/workflows/ {"input": , "options"?: {...}, "constraints"?: [...], "on_infeasible"?: ..., "model"?/"lang"?/"task"?} -> the systemone answer + "workflow" + the built "state"; the request's constraints @@ -817,6 +820,34 @@ The one-question conveniences and the patterns, as a library (`lev.api`, (pat/utility-choice rt incident (api/choice "Remedy?" ["restart" "rollback" "page-human"]) {:value 10 :costs {"page-human" 4}}) ``` +A classifier can also be written as data, by a person or by a model +from a sample of the inputs, and run deterministically (`lev.program`). +Regex facts are matched first. A rule decided by facts alone never calls +the model. Otherwise the encoder answers, in one call, the questions the +remaining rules read. The first rule that holds names the label. +`fit` sets the numeric cutoffs from labelled cases without asking the +model again: + +```clojure +(require '[lev.program :as prog]) +(def triage + {"labels" ["incident" "refund" "other"] + "extract" {"oom" "(?i)oom-killed|out of memory"} + "questions" {"refund" (api/noul "Does the customer ask for money back?")} + "rules" [{"when" {"fact" "oom"} "then" "incident"} + {"when" {"noul" "refund" ">=" 0.5} "then" "refund"}] + "default" "other"}) +(prog/run rt "pods OOM-killed on payments" triage) ; {"label" "incident" "rule" 0 "facts" {"oom" true}}, no model call +(prog/fit triage (map #(assoc (prog/observe rt (:text %) triage) "label" (:label %)) labelled)) +;; => {"program" "accuracy" .. "before" ..} +``` + +A condition is `{"fact" f}`, `{"noul" q ">=" p}` (or `>` `<=` `<`), +`{"score" q ">=" x}`, `{"choice" q "is" o}` / `{"choice" q "in" [..]}`, +or `all` / `any` / `not` over conditions. `"escalate" {"model" thinker}` +re-asks unsure answers on a thinker. `bench/README.md` (programs) has +what this buys on log lines and on comments. + ### As a binary `jolt binary` runs `jolt build -m lev.server -o lev-server` with the C diff --git a/bench/README.md b/bench/README.md index 758094b..02b80b1 100644 --- a/bench/README.md +++ b/bench/README.md @@ -621,3 +621,86 @@ the nearest neighbour here and needs a corpus mean. The signal is real anisotropic: any two states sit between ~0.82 and ~0.95. Rank by cosine; a fixed similarity cutoff is not portable between domains. Nothing here touches an answer's numbers. + +## Programs: classifiers written as data (`lev.program`, 2026-10-13) + +Can a model write a classifier once (regex facts, typed questions, ordered +rules) that lev then runs deterministically, and does that beat asking the +model the target question directly? `bench/program.clj` scores a program +on a labelled split. `--fit` first answers every question on the train +split and fits the rules' cutoffs there (`lev.program/fit`). Build the +data with `uv run --with pyarrow python bench/program_data.py`: + +- `bgl-*` is LogHub's BGL_2k: Blue Gene/L RAS log lines, 12 hand-labelled + alert categories plus normal, alternated into halves (1,004 train, 996 + test). +- `toxic-*` is google/civil_comments: 200 toxic (≥ 0.5) and 200 clean + (< 0.1) comments a half. Train comes from the validation split, test + from the test split. + +The programs are in `bench/programs/`: + +- `*-direct` asks the target as one question. +- `bgl-regex` and `toxic-facets` were written by hand from the train + half. `toxic-facets` has five facet questions (insult, identity, + threat, obscene and the direct one) plus three word-list facts. +- `*-qwen27b` were written by Qwen3.8-27B through `bench/compile.clj` + (greedy, thinking off). It saw the format, the task and up to 8 + distinct train inputs a label, and got the validator's error back: one + bad regex on toxicity, then fixed. Each program took about 4 minutes + of generation. + +Test half; macro F1 counts every label that is either gold or predicted: + +| BGL (996 lines) | accuracy | macro F1 | model calls | ms / line | +|---|---|---|---|---| +| `english`, direct 13-way choice | 0.286 | 0.309 | 996 | 227 | +| Qwen3.5-4B, direct 13-way choice | 0.741 | 0.341 | 996 | 1,858 | +| `bgl-qwen27b` (20 regex facts, no questions) | 0.990 | 0.846 | 0 | 0.1 | +| `bgl-regex` (by hand) | **1.000** | **1.000** | 0 | 0.1 | + +| toxicity (400 comments) | accuracy | macro F1 | model calls | +|---|---|---|---| +| Qwen3.5-4B, direct, cutoff 0.5 | 0.695 | 0.693 | 400 | +| Qwen3.5-4B, direct, cutoff fitted | 0.708 | 0.707 | 400 | +| `english`, direct, cutoff 0.5 | 0.755 | 0.740 | 400 | +| `toxic-qwen27b`, as written | 0.733 | 0.714 | 332 | +| `toxic-qwen27b`, cutoff fitted (0.6 → 0.34) | 0.905 | 0.905 | 332 | +| `toxic-facets`, cutoffs fitted | 0.920 | 0.920 | 400 | +| `english`, direct, cutoff fitted (0.5 → 0.33) | **0.933** | **0.932** | 400 | + +What this says: + +- **Log lines.** BGL's alerts are fixed message templates, so regex + facts decide them outright. That is perfect accuracy, no model, and + about 2,000× faster than the encoder. The encoder's direct choice is + near useless on raw log lines. The 4B thinker is better but slow, and + it misses the rare classes. Qwen3.8-27B wrote a near-perfect program + unaided. Its only misses are ten `LOGIN chdir` lines labelled normal: + its rule left out the `Input/output error` that marks the one APPOUT + alert. +- **Free text.** The cutoff matters more than anything else. The + encoder's yes/no for "toxic" sits low (at 0.5 it says yes to only 103 + of the 200 toxic comments), and fitting the one cutoff on 400 labelled + comments takes it from 0.755 to 0.933. Splitting the judgement into + facet questions and word lists fit train better (0.953 against 0.938) + but lost on test, at five questions a comment instead of one. +- **A model writing free-text programs memorises.** Qwen3.8-27B's word + list is mostly whole train phrases ("happy clients", "tennis", "okay"). + On train it labels clean comments toxic: the program scores 0.705 + there, below the bare question. On test it fires on 68 comments, 65 of + them toxic, so it is precise but covers little. The as-written program + loses on the cutoff it picked (0.6). Fitted to 0.34 it reaches 0.905, + still below the single fitted question. +- **Escalation does not help here.** Qwen3.5-4B is worse than the fitted + encoder on toxicity (0.708), so nothing was escalated to it. + +So the practical shape is: a model writes the program, it is validated +and its cutoffs are fitted on a labelled sample, and only then is it +trusted. Regex facts carry structured text such as logs. On free text, +one well-fitted question beats an elaborate decomposition. A program +should not be deployed on the model's say-so: on this data the fit and +the held-out score are what separated 0.733 from 0.905. + +Timing was taken with other benches sharing the machine, so it is +indicative only. diff --git a/bench/compile.clj b/bench/compile.clj new file mode 100644 index 0000000..152d2cb --- /dev/null +++ b/bench/compile.clj @@ -0,0 +1,86 @@ +(ns bench.compile + "A local model writes the lev.program: it sees the task, the program + format, and a sample of the train split (up to --per-label inputs a + label, distinct after digits are masked), answers greedily with JSON, + and gets lev.program/validate's error back up to twice. The program + is written to --save for bench/program.clj to fit and score. + + jolt -M bench/compile.clj --split bgl-train --task \"...\" --save FILE + [--gguf PATH | --model NAME] [--per-label 8] [--max-tokens 3000]" + (:require [clojure.string :as str] + [lev.config :as cfg] + [lev.json :as json] + [lev.llm :as llm] + [lev.program :as prog] + [lev.sequence :as seq])) + +(def format-spec + "A program is one JSON object: +{\"labels\": [label, ...], + \"extract\": {fact: regex}, a fact holds when its (Java) regex finds a match in the input + \"questions\": {qid: question}, asked to a small text encoder when a rule needs them + \"rules\": [{\"when\": condition, \"then\": label}, ...], the first rule that holds names the label + \"default\": label} when no rule holds +A question is {\"type\": \"noul\", \"instructions\": \"a yes/no question about the input\"}, +{\"type\": \"choice\", \"instructions\": \"...\", \"criteria\": {option: description}} or +{\"type\": \"score\", \"instructions\": \"...\", \"criteria\": [level, ...]}. +A condition is {\"fact\": f}, {\"noul\": qid, \">=\": p} (p the probability of yes; also > <= <), +{\"score\": qid, \">=\": x}, {\"choice\": qid, \"is\": option}, {\"choice\": qid, \"in\": [option, ...]}, +{\"all\": [condition, ...]}, {\"any\": [condition, ...]} or {\"not\": condition}. +Facts cost nothing; questions cost about 50 ms each and the encoder is weak on long or technical text, +so use a regex wherever the input's wording decides the label, and a question where meaning does. +Numeric cutoffs are refitted on labelled data afterwards.") + +(defn- sample [cases per-label] + (mapcat (fn [[label cs]] + (->> cs + (map #(get % "text")) + (reduce (fn [[seen out] t] + (let [k (str/replace t #"[0-9a-fA-F]{2,}|\d+" "N")] + (if (contains? seen k) [seen out] [(conj seen k) (conj out t)]))) + [#{} []]) + second + (take per-label) + (map (fn [t] [label t])))) + (group-by #(get % "label") cases))) + +(defn- prompt-for [task cases per-label] + (str "Write a classifier program for this task: " task "\n\n" format-spec + "\n\nLabelled examples (labelinput), from the training data:\n" + (str/join "\n" (map (fn [[l t]] (str l "\t" (subs t 0 (min 300 (count t))))) (sample cases per-label))) + "\n\nAnswer with the JSON program only.")) + +(defn- json-in [text] + (let [a (str/index-of text "{") b (str/last-index-of text "}")] + (when (and a b (< a b)) (json/read-str (subs text a (inc b)))))) + +(defn -main [& args] + (let [opts (cfg/parse-args args) + ctx (cfg/context opts) + gguf (or (get opts "--gguf") (:model (get (cfg/thinkers ctx) (get opts "--model" "qwen3.8-27b")))) + cases (map json/read-str (remove str/blank? (str/split-lines (slurp (str "bench/data/" (get opts "--split") ".jsonl"))))) + m (llm/load gguf {:n-ctx 16384 :n-seq-max 1}) + max-tokens (Long/parseLong (get opts "--max-tokens" "3000")) + ask (fn [messages] + (let [t0 (System/nanoTime) + r (llm/generate m (llm/chat-prompt m messages {:thinking false}) {:max-tokens max-tokens :temperature 0.0 :stop "<|im_end|>"})] + (println (format " %d tokens in %.1f s" (:tokens r) (/ (- (System/nanoTime) t0) 1e9))) + (:text r))) + first-msg {:role "user" :content (prompt-for (get opts "--task") cases (Long/parseLong (get opts "--per-label" "8")))}] + (println "compiling with" gguf) + (loop [messages [first-msg] tries 0] + (let [text (ask messages) + [program err] (try [(prog/validate (json-in text)) nil] + (catch Exception e [nil (ex-message e)]))] + (cond + program (do (spit (get opts "--save") (str (seq/json-str program) "\n")) + (println "valid program:" (count (get program "extract")) "facts," + (count (get program "questions")) "questions," (count (get program "rules")) "rules") + (println (seq/json-str program))) + (< tries 2) (do (println "invalid:" err) + (recur (conj messages {:role "assistant" :content text} + {:role "user" :content (str "That program is invalid: " err ". Answer with the corrected JSON program only.")}) + (inc tries))) + :else (do (println "gave up:" err) (println text) (System/exit 1))))))) + +(apply -main *command-line-args*) diff --git a/bench/program.clj b/bench/program.clj new file mode 100644 index 0000000..50251c0 --- /dev/null +++ b/bench/program.clj @@ -0,0 +1,93 @@ +(ns bench.program + "Scores a lev.program on a labelled split (bench/program_data.py builds + them): accuracy, macro F1 over the labels, how many inputs needed the + model, how many answers escalated, and the time per input. The + baselines are programs too: the target asked directly as one question + and mapped to the label by one rule per answer. + + jolt -M bench/program.clj --program bench/programs/toxic-direct.json --split toxic-test + [--model NAME] [--fit SPLIT [--save FILE]] [--limit N] [--out FILE] [--quiet] + + --fit answers every question of the program on another split (the + train half) and fits its cutoffs there (lev.program/fit) before + scoring; --save writes the fitted program. --out writes one line per + input, {id label predicted rule answers}." + (:require [lev.json :as json] + [clojure.string :as str] + [lev.config :as cfg] + [lev.program :as prog] + [lev.router :as router] + [lev.sequence :as seq])) + +(defn- read-jsonl [path] + (map json/read-str (remove str/blank? (str/split-lines (slurp path))))) + +(defn- f1s [pairs labels] + (into (sorted-map) + (map (fn [l] + (let [tp (count (filter (fn [[g p]] (and (= g l) (= p l))) pairs)) + fp (count (filter (fn [[g p]] (and (not= g l) (= p l))) pairs)) + fn* (count (filter (fn [[g p]] (and (= g l) (not= p l))) pairs)) + prec (if (pos? (+ tp fp)) (/ tp (double (+ tp fp))) 0.0) + rec (if (pos? (+ tp fn*)) (/ tp (double (+ tp fn*))) 0.0)] + [l {:f1 (if (pos? (+ prec rec)) (/ (* 2 prec rec) (+ prec rec)) 0.0) :n (+ tp fn*)}]))) + labels)) + +(defn -main [& args] + (let [opts (cfg/parse-args args) + ctx (cfg/context opts) + program (prog/validate (json/read-str (slurp (get opts "--program")))) + cases (cond->> (read-jsonl (str "bench/data/" (get opts "--split") ".jsonl")) + (get opts "--limit") (take (Long/parseLong (get opts "--limit")))) + model (get opts "--model") + rt (router/make-router {:data (cfg/setting ctx "--data" "LEV_DATA" :data "data") + :checkpoints (into {} (map (fn [n] [n (cfg/limits ctx n)])) router/names) + :calibrations (cfg/calibrations ctx) + :thinkers (cfg/thinkers ctx)}) + agent (if model (router/load-model rt model) rt) + agent (if (get program "escalate") rt agent) + program (if-let [fs (get opts "--fit")] + (let [fit-agent (if model agent (router/load-model rt "english")) + observed (mapv (fn [c] (assoc (prog/observe fit-agent (get c "text") program) "label" (get c "label"))) + (read-jsonl (str "bench/data/" fs ".jsonl"))) + r (prog/fit program observed)] + (println (format "fitted on %s (%d): accuracy %.3f -> %.3f" fs (count observed) (get r "before") (get r "accuracy"))) + (when-let [o (get opts "--save")] (spit o (str (seq/json-str (get r "program")) "\n"))) + (get r "program")) + program) + _ (prog/run agent (get (first cases) "text") program) ; warm + rows (vec (for [c cases] + (let [t0 (System/nanoTime) + r (prog/run agent (get c "text") program) + ms (/ (- (System/nanoTime) t0) 1e6)] + (when (and (not (contains? opts "--quiet")) (not= (get r "label") (get c "label"))) + (println (format " miss %s: %s -> %s %s" (get c "id") (get c "label") (get r "label") + (subs (get c "text") 0 (min 90 (count (get c "text"))))))) + {:id (get c "id") :answers (get r "answers") :rule (get r "rule") + :gold (get c "label") :pred (get r "label") :ms ms + :asked (some? (get r "answers")) + :escalated (count (get-in r ["escalation" "escalated"]))}))) + _ (when-let [o (get opts "--out")] + (spit o (apply str (map (fn [{:keys [id gold pred rule answers]}] + (str (seq/json-str (seq/ordered-map [["id" id] ["label" gold] ["predicted" pred] + ["rule" rule] ["answers" answers]])) + "\n")) + rows)))) + pairs (map (juxt :gold :pred) rows) + labels (distinct (concat (map :gold rows) (map :pred rows))) + per (f1s pairs labels) + n (count rows)] + (println (format "%s on %s (%d inputs), model %s" + (get opts "--program") (get opts "--split") n + (or model (get-in program ["escalate" "model"]) "routed"))) + (println (format " accuracy %.3f macro F1 %.3f model asked %d/%d answers escalated %d %.1f ms/input (p50 %.1f)" + (/ (count (filter (fn [[g p]] (= g p)) pairs)) (double n)) + (/ (reduce + (map :f1 (vals per))) (double (count per))) + (count (filter :asked rows)) n + (reduce + (map :escalated rows)) + (/ (reduce + (map :ms rows)) n) + (nth (vec (sort (map :ms rows))) (quot n 2)))) + (doseq [[l {:keys [f1 n]}] per] + (println (format " %-10s F1 %.3f (n %d)" l f1 n))))) + +(apply -main *command-line-args*) diff --git a/bench/program_data.py b/bench/program_data.py new file mode 100644 index 0000000..4676374 --- /dev/null +++ b/bench/program_data.py @@ -0,0 +1,64 @@ +"""Build the labelled sets bench/program.clj scores lev.program against: +real noisy inputs, each split into a train half (what a program's author +may look at) and a test half (what it is scored on). + + bench/data/bgl-{train,test}.jsonl LogHub's BGL_2k sample (Blue Gene/L + RAS log lines, hand-labelled alert + categories, "-" -> "normal"); the + lines of each label alternate between + the halves, in log order + bench/data/toxic-{train,test}.jsonl google/civil_comments: 200 toxic + (toxicity >= 0.5) and 200 non-toxic + (< 0.1) comments a half, train from + the validation split, test from the + test split, seeded + +Each line is {id text label}. + + uv run --with pyarrow python bench/program_data.py +""" +import csv, io, json, os, random, urllib.request + +import pyarrow.parquet as pq + +OUT = os.path.join(os.path.dirname(__file__), "data") +BGL = "https://raw.githubusercontent.com/logpai/loghub/master/BGL/BGL_2k.log_structured.csv" +CC = "https://huggingface.co/datasets/google/civil_comments/resolve/refs%2Fconvert%2Fparquet/default/{}/0000.parquet" + + +def write(name, rows): + with open(os.path.join(OUT, name), "w") as f: + for r in rows: + f.write(json.dumps(r, ensure_ascii=False) + "\n") + print(name, len(rows)) + + +def bgl(): + text = urllib.request.urlopen(BGL).read().decode("utf-8") + seen, halves = {}, ([], []) + for r in csv.DictReader(io.StringIO(text)): + label = "normal" if r["Label"] == "-" else r["Label"] + i = seen.get(label, 0) + seen[label] = i + 1 + line = " ".join([r["Date"], r["Node"], r["Time"], r["Type"], r["Component"], r["Level"], r["Content"]]) + halves[i % 2].append({"id": "bgl-" + r["LineId"], "text": line, "label": label}) + write("bgl-train.jsonl", halves[0]) + write("bgl-test.jsonl", halves[1]) + + +def toxic(split, out, seed): + t = pq.read_table(io.BytesIO(urllib.request.urlopen(CC.format(split)).read()), columns=["text", "toxicity"]) + rows = [r for r in t.to_pylist() if 20 <= len(r["text"]) <= 600] + rng = random.Random(seed) + pos = rng.sample([r for r in rows if r["toxicity"] >= 0.5], 200) + neg = rng.sample([r for r in rows if r["toxicity"] < 0.1], 200) + cases = [{"text": r["text"], "label": "toxic"} for r in pos] + [{"text": r["text"], "label": "ok"} for r in neg] + rng.shuffle(cases) + write(out, [dict(id=f"{split}-{i}", **c) for i, c in enumerate(cases)]) + + +if __name__ == "__main__": + os.makedirs(OUT, exist_ok=True) + bgl() + toxic("validation", "toxic-train.jsonl", 1) + toxic("test", "toxic-test.jsonl", 2) diff --git a/bench/programs/bgl-direct.json b/bench/programs/bgl-direct.json new file mode 100644 index 0000000..d38b131 --- /dev/null +++ b/bench/programs/bgl-direct.json @@ -0,0 +1,125 @@ +{ + "labels": [ + "normal", + "KERNDTLB", + "KERNSTOR", + "KERNMNTF", + "KERNTERM", + "KERNRTSP", + "KERNREC", + "APPSEV", + "APPRES", + "APPTO", + "APPREAD", + "APPCHILD", + "APPOUT" + ], + "questions": { + "category": { + "type": "choice", + "instructions": "Which alert category does this Blue Gene/L RAS log line belong to?", + "criteria": { + "normal": "routine or corrected event, no alert: core dumps being generated, alignment exceptions, corrected cache parity or ddr errors, interrupt statistics, register dumps", + "KERNDTLB": "kernel data TLB error interrupt", + "KERNSTOR": "kernel data storage interrupt", + "KERNMNTF": "Lustre file system mount failed on an I/O node", + "KERNTERM": "rts: the compute kernel terminated", + "KERNRTSP": "rts panic, execution stopped", + "KERNREC": "error receiving a packet on the tree network", + "APPSEV": "ciod: the CioStream socket link has been severed", + "APPRES": "ciod: the CioStream socket connection was reset by peer", + "APPTO": "ciod: the CioStream socket connection timed out", + "APPREAD": "ciod: failed to read the message prefix on the control stream", + "APPCHILD": "ciod: error creating the node map, no child processes", + "APPOUT": "ciod: login chdir failed with an input/output error" + } + } + }, + "rules": [ + { + "when": { + "choice": "category", + "is": "KERNDTLB" + }, + "then": "KERNDTLB" + }, + { + "when": { + "choice": "category", + "is": "KERNSTOR" + }, + "then": "KERNSTOR" + }, + { + "when": { + "choice": "category", + "is": "KERNMNTF" + }, + "then": "KERNMNTF" + }, + { + "when": { + "choice": "category", + "is": "KERNTERM" + }, + "then": "KERNTERM" + }, + { + "when": { + "choice": "category", + "is": "KERNRTSP" + }, + "then": "KERNRTSP" + }, + { + "when": { + "choice": "category", + "is": "KERNREC" + }, + "then": "KERNREC" + }, + { + "when": { + "choice": "category", + "is": "APPSEV" + }, + "then": "APPSEV" + }, + { + "when": { + "choice": "category", + "is": "APPRES" + }, + "then": "APPRES" + }, + { + "when": { + "choice": "category", + "is": "APPTO" + }, + "then": "APPTO" + }, + { + "when": { + "choice": "category", + "is": "APPREAD" + }, + "then": "APPREAD" + }, + { + "when": { + "choice": "category", + "is": "APPCHILD" + }, + "then": "APPCHILD" + }, + { + "when": { + "choice": "category", + "is": "APPOUT" + }, + "then": "APPOUT" + } + ], + "default": "normal" +} \ No newline at end of file diff --git a/bench/programs/bgl-qwen27b.json b/bench/programs/bgl-qwen27b.json new file mode 100644 index 0000000..9db254e --- /dev/null +++ b/bench/programs/bgl-qwen27b.json @@ -0,0 +1 @@ +{"labels": ["APPTO", "KERNSTOR", "KERNDTLB", "KERNTERM", "KERNREC", "APPREAD", "normal", "APPOUT", "APPRES", "KERNRTSP", "APPCHILD", "KERNMNTF", "APPSEV"], "extract": {"is_fatal": "RAS (KERNEL|APP) FATAL", "is_info": "RAS (KERNEL|APP) INFO", "is_kernel": "RAS KERNEL", "is_app": "RAS APP", "msg_ciod": "ciod:", "msg_rts": "rts:", "msg_lustre": "Lustre mount FAILED", "msg_data_storage": "data storage interrupt", "msg_data_tlb": "data TLB error interrupt", "msg_kernel_terminated": "kernel terminated", "msg_rts_panic": "rts panic!", "msg_error_receiving": "Error receiving packet", "msg_lustre_mount": "Lustre mount FAILED", "msg_conn_timeout": "Connection timed out", "msg_conn_reset": "Connection reset by peer", "msg_link_severed": "Link has been severed", "msg_failed_read_prefix": "failed to read message prefix", "msg_error_read_prefix": "Error reading message prefix", "msg_error_creating_node": "Error creating node map", "msg_login_chdir": "LOGIN chdir"}, "questions": {}, "rules": [{"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_conn_timeout"}]}, "then": "APPTO"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_conn_reset"}]}, "then": "APPRES"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_link_severed"}]}, "then": "APPSEV"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_failed_read_prefix"}]}, "then": "APPREAD"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_error_creating_node"}]}, "then": "APPCHILD"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_app"}, {"fact": "msg_ciod"}, {"fact": "msg_login_chdir"}]}, "then": "APPOUT"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_data_storage"}]}, "then": "KERNSTOR"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_data_tlb"}]}, "then": "KERNDTLB"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_rts"}, {"fact": "msg_kernel_terminated"}]}, "then": "KERNTERM"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_rts_panic"}]}, "then": "KERNRTSP"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_error_receiving"}]}, "then": "KERNREC"}, {"when": {"all": [{"fact": "is_fatal"}, {"fact": "is_kernel"}, {"fact": "msg_lustre_mount"}]}, "then": "KERNMNTF"}], "default": "normal"} diff --git a/bench/programs/bgl-regex.json b/bench/programs/bgl-regex.json new file mode 100644 index 0000000..08f4698 --- /dev/null +++ b/bench/programs/bgl-regex.json @@ -0,0 +1,106 @@ +{ + "labels": [ + "normal", + "KERNDTLB", + "KERNSTOR", + "KERNMNTF", + "KERNTERM", + "KERNRTSP", + "KERNREC", + "APPSEV", + "APPRES", + "APPTO", + "APPREAD", + "APPCHILD", + "APPOUT" + ], + "extract": { + "dtlb": "data TLB error interrupt", + "dstor": "data storage interrupt", + "mount": "Lustre mount FAILED", + "term": "rts: kernel terminated", + "panic": "rts panic", + "tree": "Error receiving packet on tree network", + "severed": "ciod: Error reading message prefix.*Link has been severed", + "reset": "ciod: Error reading message prefix.*Connection reset by peer", + "timeout": "ciod: Error reading message prefix.*Connection timed out", + "prefix": "ciod: failed to read message prefix on control stream", + "child": "ciod: Error creating node map.*No child processes", + "chdir": "ciod: LOGIN chdir.*failed: Input/output error" + }, + "rules": [ + { + "when": { + "fact": "dtlb" + }, + "then": "KERNDTLB" + }, + { + "when": { + "fact": "dstor" + }, + "then": "KERNSTOR" + }, + { + "when": { + "fact": "mount" + }, + "then": "KERNMNTF" + }, + { + "when": { + "fact": "term" + }, + "then": "KERNTERM" + }, + { + "when": { + "fact": "panic" + }, + "then": "KERNRTSP" + }, + { + "when": { + "fact": "tree" + }, + "then": "KERNREC" + }, + { + "when": { + "fact": "severed" + }, + "then": "APPSEV" + }, + { + "when": { + "fact": "reset" + }, + "then": "APPRES" + }, + { + "when": { + "fact": "timeout" + }, + "then": "APPTO" + }, + { + "when": { + "fact": "prefix" + }, + "then": "APPREAD" + }, + { + "when": { + "fact": "child" + }, + "then": "APPCHILD" + }, + { + "when": { + "fact": "chdir" + }, + "then": "APPOUT" + } + ], + "default": "normal" +} \ No newline at end of file diff --git a/bench/programs/toxic-direct.json b/bench/programs/toxic-direct.json new file mode 100644 index 0000000..3c37bbb --- /dev/null +++ b/bench/programs/toxic-direct.json @@ -0,0 +1,5 @@ +{"labels": ["toxic", "ok"], + "questions": {"toxic": {"type": "noul", + "instructions": "Is this comment toxic: rude, disrespectful or unreasonable enough to make someone leave a discussion?"}}, + "rules": [{"when": {"noul": "toxic", ">=": 0.5}, "then": "toxic"}], + "default": "ok"} diff --git a/bench/programs/toxic-facets.json b/bench/programs/toxic-facets.json new file mode 100644 index 0000000..1fdab16 --- /dev/null +++ b/bench/programs/toxic-facets.json @@ -0,0 +1,13 @@ +{"labels": ["toxic", "ok"], + "extract": {"insult_words": "(?i)\\b(idiot\\w*|i-diot|stupid\\w*|moron\\w*|dumb\\w*|fools?|foolish|clowns?|trolls?|liars?|hypocrit\\w*|pathetic|ignorant|ignorance|imbecile\\w*|rubbish|garbage|trash|losers?|scum\\w*|gullible|incompetent|flakes?|lunatic\\w*|nuts?|crazy|insane|sick|disgusting|vile|silliness|nonsense|bigot\\w*|racists?|sexist|whiners?|sheep|cowards?)\\b", + "profanity": "(?i)(\\bf+u+c+k\\w*|\\bsh[i1]t\\w*|\\bcrap\\w*|\\bdamn\\w*|\\bhell\\b|\\bass(es|hole\\w*)?\\b|\\bbutt\\b|\\bbitch\\w*|\\bbastard\\w*|\\bpiss\\w*|\\bb\\.\\.\\.s|\\bfreaking\\b)", + "violence_words": "(?i)\\b(kill\\w*|die|dead|shoot\\w*|bombs?|wipe \\w+ out|rot in|hang\\w*|trigger|violent)\\b"}, + "questions": {"toxic": {"type": "noul", "instructions": "Is this comment toxic: rude, disrespectful or unreasonable enough to make someone leave a discussion?"}, + "insult": {"type": "noul", "instructions": "Does the comment insult, mock or call names at a person or a group?"}, + "identity": {"type": "noul", "instructions": "Does the comment demean people for their race, religion, gender, sexuality, nationality or politics?"}, + "threat": {"type": "noul", "instructions": "Does the comment threaten or wish violence or death on someone?"}, + "obscene": {"type": "noul", "instructions": "Does the comment use profanity or obscene language, even partly censored?"}}, + "rules": [{"when": {"any": [{"noul": "toxic", ">=": 0.5}, {"noul": "insult", ">=": 0.5}, {"noul": "identity", ">=": 0.5}, + {"noul": "threat", ">=": 0.5}, {"noul": "obscene", ">=": 0.5}]}, "then": "toxic"}, + {"when": {"any": [{"fact": "insult_words"}, {"fact": "profanity"}, {"fact": "violence_words"}]}, "then": "toxic"}], + "default": "ok"} diff --git a/bench/programs/toxic-qwen27b.json b/bench/programs/toxic-qwen27b.json new file mode 100644 index 0000000..b0958a1 --- /dev/null +++ b/bench/programs/toxic-qwen27b.json @@ -0,0 +1 @@ +{"labels": ["ok", "toxic"], "extract": {"insult_regex": "(?i)(\\b(idiot|stupid|dumb|fool|moron|retard|retarded|bozo|punk|loser|dork|dorks|sheep|clown|clowns|puppet|puppets|degenerate|rabid|crazy|brain washed|brainwashed|ignorant|full of himself|full of shit|full of crap|absolute rubbish|save the stupidity|get some help|give it up|exit now|who cares|who cares!?|never get laid|malicious|malignant|bozos|punk|dorks|sheep of left wing dumb|clown show|puppets controlled|dancing in place|renouncing violence|violence instigatores|planted at those events|get over it|you and the left|left wing dumb|trump didnt encourage|expell the violence|nominate|coup regime|nazi elements|blow us all sky high|doubling down|support for the coup|anti-russian views|shares the same rabid|lied about her grandfather|successful business|happy clients|referral keep me rather overly busy|selling the herb|plain old hard work|manufacturing a wante|pot head|consider me a|your comment is|absolute rubbish|stronger defensive team|zairsev alone|better defensive team|lack the|josée chouinard|tennis|andy murray|nova sc|slainte mhath|arrest has been made|dailyworld|donkey creek|fatal hit and run|first nations|thousands of years|without our interference|wise parents|graduates|sorry bud|no matter how you spin it|stretch of imagination|liberal disaster|ontario has endured|improvement|perspective from his native province|motsditssansdessein|ce tu wla gwassideurw|okay|had not heard that|military court martial|felony conviction|gap|dv conviction|dv incident|court martial|specifically for dv|legal deta|npc|defensive team|rookies|running around less|own zone|better positioned|heck|bozak|lack the|bernstein is a degenerate bozo|who cares what this full of himself punk thinks|bernstein and 'his crowd' are the malignant ones|bb, your comment is absolute rubbish|if you consider me a 'pot head'|then how is it that i have a successful business|long list of happy clients|by referral keep me rather overly busy|and no, it has nothing to do with selling the herb|but just plain old hard work manufacturing a wante|but idiots with a more diverse and healthier economy|she's lied about her grandfather|because she shares the same rabid, anti-russian views|it matters because canada is doubling down on its support for the coup regime in the ukraine|that contains nazi elements and is spoiling to draw nato into a conflict with russia|that could blow us all sky high|note to \"chopper bob\"|i would state that what the gop lead legislature has done over the past several years is also (very) insignificant|actually it's been a clown show of puppets controlled by their corporate puppeteers-dancing in place !|oh save the stupidity and lies again sheep of left wing dumb|trump didnt encourage anything other than to expell the violence instigatores planted at those events|get over it...|you and the left have to start renouncing violence since it is the left that has been doing it since trump was nominate|you are absolutely crazy..|trump is informed but you are brain washed ignorant,,|please get some help for your stupidity see loser,|who cares!?|stark trek and star wars fans are all dorks who never get laid))"}, "questions": {"q_toxic_intent": {"type": "noul", "instructions": "Does the comment contain language that is rude, disrespectful, or unreasonable enough to make someone leave a discussion? This includes insults, personal attacks, hate speech, or aggressive dismissal of others' views."}}, "rules": [{"when": {"fact": "insult_regex"}, "then": "toxic"}, {"when": {"noul": "q_toxic_intent", ">=": 0.6}, "then": "toxic"}], "default": "ok"} diff --git a/src/lev/program.clj b/src/lev/program.clj new file mode 100644 index 0000000..4a3d3fe --- /dev/null +++ b/src/lev/program.clj @@ -0,0 +1,226 @@ +(ns lev.program + "A classifier written as data, to be authored once (by a person or a + large model, from a sample of the inputs) and run deterministically on + every input: + + {\"labels\" [label ...] optional; rules are checked against it + \"extract\" {fact regex} a fact holds when its regex finds a match + \"questions\" {qid question} typed questions (choice / score / noul) + \"rules\" [{\"when\" condition \"then\" label} ...] + \"default\" label when no rule fires + \"escalate\" {\"model\" thinker \"threshold\" t}} optional; needs a router + + A condition is one of + {\"fact\" f} + {\"noul\" qid op x} op one of >= > <= <, on the probability of yes + {\"score\" qid op x} on the expected level index + {\"choice\" qid \"is\" option} / {\"choice\" qid \"in\" [option ...]} + {\"all\" [c ...]} / {\"any\" [c ...]} / {\"not\" c} + + `run` matches the facts first and evaluates the rules in order with the + answers unknown: when the first rule that is not false is decided by the + facts, no model is called. Otherwise the agent answers, in one call, + the questions the rules from there on read, and the first true rule + names the label. With \"escalate\" the answers below the threshold are + re-asked on the thinker (lev.patterns/escalate)." + (:require [lev.agent :as ag] + [lev.api :as api] + [lev.patterns :as pat] + [lev.sequence :as seq])) + +(def ^:private ops {">=" >= ">" > "<=" <= "<" <}) + +(defn- bad [field msg] + (throw (ex-info (str field ": " msg) {:type :invalid-request :field field}))) + +(defn- qtype [program qid] (get-in program ["questions" qid "type"])) + +(defn- options [program qid] + (set (map (fn [k] (if (keyword? k) (name k) (str k))) + (keys (:crit (ag/to-internal (get-in program ["questions" qid]))))))) + +(defn- check-condition [program field c] + (let [fail #(bad field (str % ": " (pr-str c))) + compare-ok (fn [t] + (let [qid (get c t) + op (first (filter #(contains? c %) (keys ops)))] + (when-not (= t (qtype program qid)) (fail (str "no " t " question " (pr-str qid)))) + (when-not (and op (number? (get c op))) (fail "needs one of >= > <= < with a number"))))] + (when-not (map? c) (fail "a condition is an object")) + (cond + (contains? c "fact") (when-not (contains? (get program "extract") (get c "fact")) + (fail (str "no extracted fact " (pr-str (get c "fact"))))) + (contains? c "noul") (compare-ok "noul") + (contains? c "score") (compare-ok "score") + (contains? c "choice") (let [qid (get c "choice")] + (when-not (= "choice" (qtype program qid)) (fail (str "no choice question " (pr-str qid)))) + (let [os (cond (contains? c "is") [(get c "is")] + (sequential? (get c "in")) (get c "in") + :else (fail "a choice condition needs \"is\" or \"in\""))] + (doseq [o os] + (when-not (contains? (options program qid) o) (fail (str "unknown option " (pr-str o))))))) + (contains? c "all") (if (sequential? (get c "all")) (doseq [x (get c "all")] (check-condition program field x)) (fail "\"all\" takes a list")) + (contains? c "any") (if (sequential? (get c "any")) (doseq [x (get c "any")] (check-condition program field x)) (fail "\"any\" takes a list")) + (contains? c "not") (check-condition program field (get c "not")) + :else (fail "unknown condition")))) + +(defn validate + "The program, or {:type :invalid-request :field path} naming the first + thing wrong with it: a regex that does not compile, a question lev + would refuse, a rule reading a fact or question that is not there (or + a question of another type, or an option it does not have), a label + outside \"labels\", a missing \"default\"." + [program] + (when-not (map? program) (bad "program" "must be an object")) + (doseq [[f re] (get program "extract")] + (try (re-pattern re) (catch Exception e (bad (str "extract." f) (str "bad regex: " (ex-message e)))))) + (doseq [[qid qdef] (get program "questions")] + (try (ag/validate-question qid qdef) + (catch Exception e (bad (str "questions." qid) (ex-message e))))) + (let [labels (get program "labels") + label-ok (fn [field l] + (when-not (string? l) (bad field "a label is a string")) + (when (and labels (not (some #{l} labels))) (bad field (str (pr-str l) " is not in labels"))))] + (when-not (sequential? (get program "rules" [])) (bad "rules" "must be a list")) + (doseq [[i r] (map-indexed vector (get program "rules"))] + (check-condition program (str "rules[" i "].when") (get r "when")) + (label-ok (str "rules[" i "].then") (get r "then"))) + (when-not (contains? program "default") (bad "default" "is required")) + (label-ok "default" (get program "default"))) + (when-let [e (get program "escalate")] + (when-not (string? (get e "model")) (bad "escalate.model" "names the thinker"))) + program) + +;; --- evaluation: true, false or ::unknown (an answer not asked yet) ---------- + +(defn- holds [c facts answers] + (let [answer (fn [qid] (get answers qid))] + (cond + (contains? c "fact") (boolean (get facts (get c "fact"))) + (contains? c "all") (let [vs (map #(holds % facts answers) (get c "all"))] + (cond (some false? vs) false (some #{::unknown} vs) ::unknown :else true)) + (contains? c "any") (let [vs (map #(holds % facts answers) (get c "any"))] + (cond (some true? vs) true (some #{::unknown} vs) ::unknown :else false)) + (contains? c "not") (let [v (holds (get c "not") facts answers)] (if (= ::unknown v) v (not v))) + :else + (let [t (first (filter #(contains? c %) ["noul" "score" "choice"]))] + (if-let [a (answer (get c t))] + (if (= t "choice") + (contains? (set (if (contains? c "is") [(get c "is")] (get c "in"))) (get a "choice")) + (let [op (first (filter #(contains? c %) (keys ops)))] + ((ops op) (double (get a t)) (double (get c op))))) + ::unknown))))) + +(defn- first-open + "[index value] of the first rule that is not false, or nil." + [rules facts answers] + (first (keep-indexed (fn [i r] (let [v (holds (get r "when") facts answers)] (when-not (false? v) [i v]))) rules))) + +(defn- reads [c] + (cond + (contains? c "all") (mapcat reads (get c "all")) + (contains? c "any") (mapcat reads (get c "any")) + (contains? c "not") (reads (get c "not")) + :else (keep #(get c %) ["noul" "score" "choice"]))) + +(defn facts + "{fact matched?} for the input: a string as it is, anything else as JSON." + [program input] + (let [text (if (string? input) input (seq/json-str input))] + (into {} (map (fn [[f re]] [f (some? (re-find (re-pattern re) text))])) (get program "extract")))) + +(defn label + "[label rule] for the facts and answers (nil answers: none asked): the + first rule that holds, or [default nil]. Pure; what `run` decides + with, and what `fit` replays." + [program facts answers] + (let [rules (get program "rules" [])] + (or (first (keep-indexed (fn [i r] (when (true? (holds (get r "when") facts answers)) [(get r "then") i])) rules)) + [(get program "default") nil]))) + +(defn run + "The program's label for one input on an agent or router: an ordered + map {label rule facts}, rule the index of the rule that fired (nil: + the default); when the model was asked, its answers, model and usage + follow (and the escalation report when there was one). Validates the + program first. opts go to lev.api/system-one (:model :lang on a + router; with escalation, :model is the fast one)." + ([agent input program] (run agent input program nil)) + ([agent input program opts] + (let [program (validate program) + rules (get program "rules" []) + fs (facts program input) + [i v] (first-open rules fs nil) + done (fn [answers tail] + (let [[l r] (label program fs answers)] + (seq/ordered-map (concat [["label" l] ["rule" r] ["facts" fs]] tail))))] + (if (not= ::unknown v) + (done nil nil) + (let [needed (set (mapcat #(reads (get % "when")) (subvec (vec rules) i))) + questions (seq/ordered-map (filter (fn [[qid _]] (contains? needed qid)) (get program "questions"))) + esc (get program "escalate") + resp (if esc + (pat/escalate agent input questions {:model (get esc "model") :threshold (get esc "threshold") + :fast-model (:model opts) :lang (:lang opts)}) + (api/system-one agent input questions opts)) + answers (get resp "answers")] + (done answers + (concat [["answers" answers] ["model" (get resp "model")] ["usage" (get resp "usage")]] + (when esc [["escalation" (get resp "escalation")]])))))))) + +(defn observe + "{facts answers} for one input with every question of the program + answered (no escalation): what `fit` replays." + [agent input program] + (let [program (validate program)] + (seq/ordered-map [["facts" (facts program input)] + ["answers" (when (seq (get program "questions")) + (get (api/system-one agent input (get program "questions")) "answers"))]]))) + +;; --- fitting the cutoffs on labelled cases ------------------------------------ + +(defn- cutoffs + "[path-to-the-number candidates] for every noul / score comparison in + the rules: a noul's cutoffs 0.01..0.99, a score's 0..(levels - 1) by 0.05." + [program] + (letfn [(walk [path c] + (cond + (contains? c "all") (mapcat #(walk (conj path "all" %1) %2) (range) (get c "all")) + (contains? c "any") (mapcat #(walk (conj path "any" %1) %2) (range) (get c "any")) + (contains? c "not") (walk (conj path "not") (get c "not")) + (or (contains? c "noul") (contains? c "score")) + (let [op (first (filter #(contains? c %) (keys ops))) + cands (if (contains? c "noul") + (map #(ag/round4 (/ % 100.0)) (range 1 100)) + (let [top (dec (count (get-in program ["questions" (get c "score") "criteria"])))] + (map #(ag/round4 (* % 0.05)) (range 0 (inc (* 20 top))))))] + [[(conj path op) cands]]) + :else nil))] + (vec (mapcat (fn [i r] (walk ["rules" i "when"] (get r "when"))) (range) (get program "rules"))))) + +(defn- accuracy [program cases] + (/ (count (filter (fn [c] (= (get c "label") (first (label program (get c "facts") (get c "answers"))))) cases)) + (double (max 1 (count cases))))) + +(defn fit + "The program's numeric cutoffs fitted to labelled cases ({facts answers + label}, as `observe` answers plus the gold label): coordinate ascent, + three passes over every noul / score comparison, each moved to the + median of the cutoffs that label the most cases right (kept where it + already is one of them). Deterministic; no model is asked. Answers + {program accuracy before}." + [program cases] + (let [program (validate program) + cases (vec cases) + points (cutoffs program) + step (fn [p [path cands]] + (let [scored (mapv (fn [x] [x (accuracy (assoc-in p path x) cases)]) cands) + top (reduce max (map second scored)) + best (mapv first (filter #(= top (second %)) scored))] + (if (some #{(get-in p path)} best) + p + (assoc-in p path (nth best (quot (dec (count best)) 2)))))) + fitted (reduce (fn [p _] (reduce step p points)) program (range 3))] + (seq/ordered-map [["program" fitted] + ["accuracy" (ag/round4 (accuracy fitted cases))] + ["before" (ag/round4 (accuracy program cases))]]))) diff --git a/src/lev/server.clj b/src/lev/server.clj index 397e3d5..2662e7e 100644 --- a/src/lev/server.clj +++ b/src/lev/server.clj @@ -23,6 +23,11 @@ \"embeddings\" [[unit vector] ...] \"dimensions\" \"usage\" \"routing\"}: the encoder's mean-pooled final states (lev.agent/embed; bench/README.md, embeddings) + POST /v1/program {\"program\" (lev.program), \"state\"} or {.. \"states\" [...]}, + \"model\"? \"lang\"? -> {\"label\" \"rule\" \"facts\" (\"answers\" + \"model\" \"usage\" when the model was asked)}, or + {\"results\" [...]} for states: a classifier written + as data (regex facts, questions, ordered rules) POST /v1/route same body, questions optional -> the routing decision alone, nothing loaded or run POST /v1/patterns/confidence-gate systemone body + \"threshold\"? -> {\"automatic\" \"escalate\" \"response\"} @@ -83,6 +88,7 @@ [lev.llm] [lev.mcp :as mcp] [lev.patterns :as pat] + [lev.program :as prog] [lev.router :as router] [lev.sequence :as seq] [lev.tokenizer :as tk] @@ -362,6 +368,29 @@ (json-response 200 (update out "embeddings" (fn [es] (mapv (fn [e] (mapv #(/ (Math/round (* 1e6 (double %))) 1e6) e)) es)))))))) +(defn run-program + "POST /v1/program: {\"program\" {...}} and {\"state\"} or {\"states\" + [...]} (1-256), routed like /v1/systemone -> lev.program/run's map, or + {\"results\" [...]} in order for states. A bad program is a 422 at + [\"body\" \"program.\"]." + [rt lock body] + (let [details (if-not (map? body) + [(detail ["body"] "request body must be a JSON object" "type_error")] + (vec (concat (when-not (map? (get body "program")) + [(detail ["body" "program"] "program is required: an object (lev.program)" "value_error")]) + (if (contains? body "states") (check-states body) (check-state body)) + (check-routing-fields rt body))))] + (if (seq details) + (unprocessable details) + (let [p (try (prog/validate (get body "program")) + (catch Exception e + (throw (ex-info (ex-message e) (update (ex-data e) :field #(str "program." %)))))) + {:keys [model lang]} (routing-opts body) + one (fn [x] (prog/run rt x p {:model model :lang lang}))] + (json-response 200 (if (contains? body "states") + {"results" (locking lock (mapv one (get body "states")))} + (locking lock (one (get body "state"))))))))) + (defn route-only "POST /v1/route: the decision for a body, without loading or running." [rt body] @@ -542,6 +571,8 @@ :response (guard (json-in #(systemone-batch rt lock %) false))} {:path "/v1/embed" :method :post :response (guard (json-in #(embed rt lock %) false))} + {:path "/v1/program" :method :post + :response (guard (json-in #(run-program rt lock %) false))} {:path "/v1/route" :method :post :response (guard (json-in #(route-only rt %) false))} {:path "/v1/models" :method :get :response (guard (fn [_] (models rt)))} diff --git a/test/lev/program_test.clj b/test/lev/program_test.clj new file mode 100644 index 0000000..76ffa0e --- /dev/null +++ b/test/lev/program_test.clj @@ -0,0 +1,141 @@ +(ns lev.program-test + "lev.program: a classifier written as data (regex facts, typed + questions, ordered rules) and run deterministically. Canned agents, no + model loaded." + (:require [clojure.test :refer [deftest is testing]] + [lev.agent :as ag] + [lev.api :as api] + [lev.program :as prog] + [lev.router :as router] + [lev.sequence :as seq])) + +(def calls (atom [])) + +(defmethod ag/system-one* :canned-program + [{:keys [name table]} _state questions _opts] + (swap! calls conj [name (vec (map (fn [[k _]] k) questions))]) + (seq/ordered-map + [["model" name] + ["answers" (seq/ordered-map + (map (fn [[qid qdef]] + (let [q (ag/to-internal (ag/validate-question qid qdef)) + p (or (get table qid) (throw (ex-info (str "no answer for " qid) {})))] + [qid (ag/typed-answer q p (count p) nil nil)])) + questions))] + ["usage" (array-map "input_tokens" (* 10 (count questions)) "output_tokens" 0)]])) + +(defn- canned [name table] {:kind :canned-program :name name :table table}) + +(def program + {"labels" ["oom" "abuse" "refund" "ok"] + "extract" {"oom" "(?i)out of memory|oom-killed"} + "questions" (array-map + "insult" (api/noul "Does the text insult someone?") + "intent" (api/choice "What does the writer want?" (array-map "refund" nil "help" nil "other" nil)) + "anger" (api/score "How angry is the writer?" ["calm" "annoyed" "furious"])) + "rules" [{"when" {"fact" "oom"} "then" "oom"} + {"when" {"any" [{"noul" "insult" ">=" 0.7} + {"score" "anger" ">=" 1.5}]} "then" "abuse"} + {"when" {"all" [{"choice" "intent" "is" "refund"} + {"not" {"noul" "insult" ">=" 0.5}}]} "then" "refund"}] + "default" "ok"}) + +(def calm-refund {"insult" [0.9 0.1] "intent" [0.8 0.1 0.1] "anger" [0.7 0.2 0.1]}) + +(deftest a-fact-rule-decides-without-the-model + (reset! calls []) + (let [r (prog/run (canned "fast" calm-refund) "pod oom-killed at 3am" program)] + (is (= "oom" (get r "label"))) + (is (= 0 (get r "rule"))) + (is (= {"oom" true} (get r "facts"))) + (is (nil? (get r "answers"))) + (is (= [] @calls)))) + +(deftest the-model-answers-what-the-rules-still-need + (reset! calls []) + (let [r (prog/run (canned "fast" calm-refund) "I want my money back" program)] + (is (= "refund" (get r "label"))) + (is (= 2 (get r "rule"))) + (is (= ["label" "rule" "facts" "answers" "model" "usage"] (vec (keys r)))) + (is (= 0.8 (get-in r ["answers" "intent" "probabilities" "refund"]))) + (testing "one call, only the questions rules from the first undecided one on read" + (is (= [["fast" ["insult" "intent" "anger"]]] @calls)))) + (testing "a later rule's questions only" + (reset! calls []) + (let [p (assoc program "rules" [{"when" {"fact" "oom"} "then" "oom"} + {"when" {"choice" "intent" "in" ["refund" "help"]} "then" "refund"}])] + (prog/run (canned "fast" calm-refund) "hello" p) + (is (= [["fast" ["intent"]]] @calls))))) + +(deftest rules-fire-in-order-and-the-default-closes + (let [angry {"insult" [0.2 0.8] "intent" [0.8 0.1 0.1] "anger" [0.1 0.2 0.7]} + calm-other {"insult" [0.9 0.1] "intent" [0.1 0.1 0.8] "anger" [0.9 0.05 0.05]}] + (is (= ["abuse" 1] ((juxt #(get % "label") #(get % "rule")) (prog/run (canned "f" angry) "x" program)))) + (let [r (prog/run (canned "f" calm-other) "x" program)] + (is (= "ok" (get r "label"))) + (is (nil? (get r "rule")))))) + +(deftest a-program-without-questions-never-calls + (reset! calls []) + (let [p {"extract" {"tlb" "data TLB error"} "rules" [{"when" {"fact" "tlb"} "then" "KERNDTLB"}] "default" "normal"}] + (is (= "KERNDTLB" (get (prog/run (canned "f" {}) "data TLB error interrupt" p) "label"))) + (is (= "normal" (get (prog/run (canned "f" {}) "all quiet" p) "label"))) + (is (= [] @calls)))) + +(deftest a-bad-program-is-refused-before-anything-runs + (let [field (fn [p] (try (prog/validate p) nil + (catch Exception e + (is (= :invalid-request (:type (ex-data e)))) + (:field (ex-data e)))))] + (is (= program (prog/validate program))) + (is (= "extract.oom" (field (assoc-in program ["extract" "oom"] "(unclosed")))) + (is (= "rules[0].when" (field (assoc-in program ["rules" 0 "when"] {"fact" "nope"})))) + (is (= "rules[1].when" (field (assoc-in program ["rules" 1 "when"] {"noul" "intent" ">=" 0.5})))) + (is (= "rules[2].when" (field (assoc-in program ["rules" 2 "when"] {"choice" "intent" "is" "nope"})))) + (is (= "rules[1].when" (field (assoc-in program ["rules" 1 "when"] {"noul" "insult" ">=" "high"})))) + (is (= "rules[1].when" (field (assoc-in program ["rules" 1 "when"] {"what" 1})))) + (is (= "rules[0].then" (field (assoc-in program ["rules" 0 "then"] "unlisted")))) + (is (= "default" (field (dissoc program "default")))) + (is (= "questions.insult" (field (assoc-in program ["questions" "insult"] {"type" "noul"})))))) + +(deftest unsure-answers-escalate-to-a-thinker + (reset! calls []) + (let [fast (canned "fast" {"insult" [0.4 0.6] "intent" [0.98 0.01 0.01] "anger" [0.98 0.01 0.01]}) + slow (canned "slow" {"insult" [0.05 0.95]}) + rt (router/make-router {:loader (fn [name & _] (if (= name "english") fast (throw (ex-info "no" {})))) + :thinkers {"slow" {:model "x.gguf"}} + :thinker-loader (fn [_ _] slow)}) + r (prog/run rt "x" (assoc program "escalate" {"model" "slow" "threshold" 0.8}))] + (is (= "abuse" (get r "label"))) + (is (= ["insult"] (get-in r ["escalation" "escalated"]))) + (is (= 0.95 (get-in r ["answers" "insult" "noul"]))) + (is (= [["fast" ["insult" "intent" "anger"]] ["slow" ["insult"]]] @calls)))) + +(deftest label-is-the-rules-over-facts-and-answers + (let [a (fn [table] (get (prog/run (canned "f" table) "x" program) "answers")) + angry (a {"insult" [0.2 0.8] "intent" [0.8 0.1 0.1] "anger" [0.1 0.2 0.7]})] + (is (= ["abuse" 1] (prog/label program {"oom" false} angry))) + (is (= ["oom" 0] (prog/label program {"oom" true} angry))) + (is (= ["ok" nil] (prog/label {"rules" [] "default" "ok"} {} nil))))) + +(deftest observe-answers-every-question + (reset! calls []) + (let [o (prog/observe (canned "fast" calm-refund) "pod oom-killed" program)] + (is (= {"oom" true} (get o "facts"))) + (is (= ["insult" "intent" "anger"] (vec (keys (get o "answers"))))) + (is (= [["fast" ["insult" "intent" "anger"]]] @calls)))) + +(deftest fit-moves-each-cutoff-to-the-middle-of-its-best-range + (let [p {"questions" {"toxic" (api/noul "Toxic?")} + "rules" [{"when" {"any" [{"noul" "toxic" ">=" 0.5}]} "then" "toxic"}] + "default" "ok"} + case* (fn [prob label] {"facts" {} "answers" {"toxic" {"type" "noul" "noul" prob}} "label" label}) + cases [(case* 0.6 "toxic") (case* 0.35 "toxic") (case* 0.25 "ok") (case* 0.1 "ok")] + r (prog/fit p cases)] + (is (= 0.75 (get r "before"))) + (is (= 1.0 (get r "accuracy"))) + (testing "the best cutoffs are 0.26..0.35; the median of that run is kept" + (is (= 0.3 (get-in r ["program" "rules" 0 "when" "any" 0 ">="])))) + (is (= (prog/validate (get r "program")) (get r "program"))) + (testing "nothing to gain: the program is unchanged" + (is (= p (get (prog/fit p [(case* 0.9 "toxic") (case* 0.1 "ok")]) "program")))))) diff --git a/test/lev/server_test.clj b/test/lev/server_test.clj index ca160be..755813b 100644 --- a/test/lev/server_test.clj +++ b/test/lev/server_test.clj @@ -525,6 +525,40 @@ (is (= ["body" "model"] (loc (second (post {"state" "a" "model" "nope"}))))) (is (= 405 (first (call h (req :get "/v1/embed")))))))) +(deftest programs-on-the-wire + (let [rt (router/preloaded @agent "english" {}) + h (srv/handler rt {:workflows @workflows}) + post (fn [body] (call h (req :post "/v1/program" :body (seq/json-str body)))) + loc (fn [b] (get-in b ["detail" 0 "loc"])) + p {"extract" {"oom" "(?i)oom-killed|out of memory"} + "questions" {"refund" {"type" "noul" "instructions" "Does the customer ask for money back?"}} + "rules" [{"when" {"fact" "oom"} "then" "incident"} + {"when" {"noul" "refund" ">=" 0.5} "then" "refund"}] + "default" "other"}] + (testing "a fact decides without the model" + (let [[st b] (post {"program" p "state" "pods OOM-killed on payments"})] + (is (= 200 st) (pr-str b)) + (is (= ["label" "rule" "facts"] (keys b))) + (is (= "incident" (get b "label"))))) + (testing "otherwise the encoder answers the questions the rules read" + (let [[st b] (post {"program" p "state" "I was charged twice, please refund the duplicate charge"})] + (is (= 200 st)) + (is (= "refund" (get b "label"))) + (is (= "english" (get b "model"))))) + (testing "several states" + (let [[st b] (post {"program" p "states" ["out of memory again" "hello there"]})] + (is (= 200 st)) + (is (= ["incident" 0] ((juxt #(get % "label") #(get % "rule")) (first (get b "results"))))) + (is (= 2 (count (get b "results")))))) + (testing "validation" + (is (= ["body" "program"] (loc (second (post {"state" "a"}))))) + (is (= ["body" "state"] (loc (second (post {"program" p}))))) + (is (= ["body" "program.rules[0].when"] + (loc (second (post {"program" (assoc-in p ["rules" 0 "when"] {"fact" "nope"}) "state" "a"}))))) + (is (= ["body" "program.extract.oom"] + (loc (second (post {"program" (assoc-in p ["extract" "oom"] "(") "state" "a"}))))) + (is (= 405 (first (call h (req :get "/v1/program")))))))) + (deftest an-engine-thinker-is-listed-as-not-thinking ;; Clef and TinyDecide generate nothing: there is no thought to switch on (let [rt (router/make-router {:thinkers {"clef" {:engine "clef" :model "target/no-such.gguf" :head "target/nope"} diff --git a/test/lev/test_runner.clj b/test/lev/test_runner.clj index 0d1ea11..a9ea194 100644 --- a/test/lev/test_runner.clj +++ b/test/lev/test_runner.clj @@ -13,6 +13,7 @@ lev.clef-test lev.tinydecide-test lev.patterns-test + lev.program-test lev.calibrate-test workflows.email-test workflows.presets-test