Skip to content

lev.program: classifiers as data, fitted on labelled cases #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yogthos merged 2 commits into from
Oct 13, 2026
Merged

lev.program: classifiers as data, fitted on labelled cases #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be8af1e
lev.program: classifiers as data, fitted on labelled cases
yogthos Oct 13, 2026
043b2c2
bench/README: correct the programs section against the data
yogthos Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -667,6 +667,9 @@ POST /v1/route same body, questions optional -> the routing decision
POST /v1/embed {"state"} or {"states": [1-256]}, "model"?/"lang"? -> {"model", "embeddings": [[unit vector] ...],
"dimensions", "usage", "routing"}: an encoder's mean-pooled final states, for state
similarity (rank by cosine; bench/README.md has how good it is)
POST /v1/program {"program": {...}, "state"} or {.. "states": [1-256]}, "model"?/"lang"?
-> {"label", "rule", "facts", ("answers", "model", "usage" when the model was asked)},
or {"results": [...]}: a classifier written as data (lev.program, below)
POST /v1/workflows/<name> {"input": <anything the workflow's state fn takes>, "options"?: {...}, "constraints"?: [...],
"on_infeasible"?: ..., "model"?/"lang"?/"task"?}
-> the systemone answer + "workflow" + the built "state"; the request's constraints
Expand Down Expand Up @@ -817,6 +820,34 @@ The one-question conveniences and the patterns, as a library (`lev.api`,
(pat/utility-choice rt incident (api/choice "Remedy?" ["restart" "rollback" "page-human"]) {:value 10 :costs {"page-human" 4}})
```

A classifier can also be written as data, by a person or by a model
from a sample of the inputs, and run deterministically (`lev.program`).
Regex facts are matched first. A rule decided by facts alone never calls
the model. Otherwise the encoder answers, in one call, the questions the
remaining rules read. The first rule that holds names the label.
`fit` sets the numeric cutoffs from labelled cases without asking the
model again:

```clojure
(require '[lev.program :as prog])
(def triage
{"labels" ["incident" "refund" "other"]
"extract" {"oom" "(?i)oom-killed|out of memory"}
"questions" {"refund" (api/noul "Does the customer ask for money back?")}
"rules" [{"when" {"fact" "oom"} "then" "incident"}
{"when" {"noul" "refund" ">=" 0.5} "then" "refund"}]
"default" "other"})
(prog/run rt "pods OOM-killed on payments" triage) ; {"label" "incident" "rule" 0 "facts" {"oom" true}}, no model call
(prog/fit triage (map #(assoc (prog/observe rt (:text %) triage) "label" (:label %)) labelled))
;; => {"program" <cutoffs fitted> "accuracy" .. "before" ..}
```

A condition is `{"fact" f}`, `{"noul" q ">=" p}` (or `>` `<=` `<`),
`{"score" q ">=" x}`, `{"choice" q "is" o}` / `{"choice" q "in" [..]}`,
or `all` / `any` / `not` over conditions. `"escalate" {"model" thinker}`
re-asks unsure answers on a thinker. `bench/README.md` (programs) has
what this buys on log lines and on comments.

### As a binary

`jolt binary` runs `jolt build -m lev.server -o lev-server` with the C
Expand Down
83 changes: 83 additions & 0 deletions bench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -621,3 +621,86 @@ the nearest neighbour here and needs a corpus mean. The signal is real
anisotropic: any two states sit between ~0.82 and ~0.95. Rank by cosine;
a fixed similarity cutoff is not portable between domains. Nothing here
touches an answer's numbers.

## Programs: classifiers written as data (`lev.program`, 2026-10-13)

Can a model write a classifier once (regex facts, typed questions, ordered
rules) that lev then runs deterministically, and does that beat asking the
model the target question directly? `bench/program.clj` scores a program
on a labelled split. `--fit` first answers every question on the train
split and fits the rules' cutoffs there (`lev.program/fit`). Build the
data with `uv run --with pyarrow python bench/program_data.py`:

- `bgl-*` is LogHub's BGL_2k: Blue Gene/L RAS log lines, 12 hand-labelled
alert categories plus normal, alternated into halves (1,004 train, 996
test).
- `toxic-*` is google/civil_comments: 200 toxic (≥ 0.5) and 200 clean
(< 0.1) comments a half. Train comes from the validation split, test
from the test split.

The programs are in `bench/programs/`:

- `*-direct` asks the target as one question.
- `bgl-regex` and `toxic-facets` were written by hand from the train
half. `toxic-facets` has five facet questions (insult, identity,
threat, obscene and the direct one) plus three word-list facts.
- `*-qwen27b` were written by Qwen3.8-27B through `bench/compile.clj`
(greedy, thinking off). It saw the format, the task and up to 8
distinct train inputs a label, and got the validator's error back: one
bad regex on toxicity, then fixed. Each program took about 4 minutes
of generation.

Test half; macro F1 counts every label that is either gold or predicted:

| BGL (996 lines) | accuracy | macro F1 | model calls | ms / line |
|---|---|---|---|---|
| `english`, direct 13-way choice | 0.286 | 0.309 | 996 | 227 |
| Qwen3.5-4B, direct 13-way choice | 0.741 | 0.341 | 996 | 1,858 |
| `bgl-qwen27b` (20 regex facts, no questions) | 0.990 | 0.846 | 0 | 0.1 |
| `bgl-regex` (by hand) | **1.000** | **1.000** | 0 | 0.1 |

| toxicity (400 comments) | accuracy | macro F1 | model calls |
|---|---|---|---|
| Qwen3.5-4B, direct, cutoff 0.5 | 0.695 | 0.693 | 400 |
| Qwen3.5-4B, direct, cutoff fitted | 0.708 | 0.707 | 400 |
| `english`, direct, cutoff 0.5 | 0.755 | 0.740 | 400 |
| `toxic-qwen27b`, as written | 0.733 | 0.714 | 332 |
| `toxic-qwen27b`, cutoff fitted (0.6 → 0.34) | 0.905 | 0.905 | 332 |
| `toxic-facets`, cutoffs fitted | 0.920 | 0.920 | 400 |
| `english`, direct, cutoff fitted (0.5 → 0.33) | **0.933** | **0.932** | 400 |

What this says:

- **Log lines.** BGL's alerts are fixed message templates, so regex
facts decide them outright. That is perfect accuracy, no model, and
about 2,000× faster than the encoder. The encoder's direct choice is
near useless on raw log lines. The 4B thinker is better but slow, and
it misses the rare classes. Qwen3.8-27B wrote a near-perfect program
unaided. Its only misses are ten `LOGIN chdir` lines labelled normal:
its rule left out the `Input/output error` that marks the one APPOUT
alert.
- **Free text.** The cutoff matters more than anything else. The
encoder's yes/no for "toxic" sits low (at 0.5 it says yes to only 103
of the 200 toxic comments), and fitting the one cutoff on 400 labelled
comments takes it from 0.755 to 0.933. Splitting the judgement into
facet questions and word lists fit train better (0.953 against 0.938)
but lost on test, at five questions a comment instead of one.
- **A model writing free-text programs memorises.** Qwen3.8-27B's word
list is mostly whole train phrases ("happy clients", "tennis", "okay").
On train it labels clean comments toxic: the program scores 0.705
there, below the bare question. On test it fires on 68 comments, 65 of
them toxic, so it is precise but covers little. The as-written program
loses on the cutoff it picked (0.6). Fitted to 0.34 it reaches 0.905,
still below the single fitted question.
- **Escalation does not help here.** Qwen3.5-4B is worse than the fitted
encoder on toxicity (0.708), so nothing was escalated to it.

So the practical shape is: a model writes the program, it is validated
and its cutoffs are fitted on a labelled sample, and only then is it
trusted. Regex facts carry structured text such as logs. On free text,
one well-fitted question beats an elaborate decomposition. A program
should not be deployed on the model's say-so: on this data the fit and
the held-out score are what separated 0.733 from 0.905.

Timing was taken with other benches sharing the machine, so it is
indicative only.
86 changes: 86 additions & 0 deletions bench/compile.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
(ns bench.compile
"A local model writes the lev.program: it sees the task, the program
format, and a sample of the train split (up to --per-label inputs a
label, distinct after digits are masked), answers greedily with JSON,
and gets lev.program/validate's error back up to twice. The program
is written to --save for bench/program.clj to fit and score.

jolt -M bench/compile.clj --split bgl-train --task \"...\" --save FILE
[--gguf PATH | --model NAME] [--per-label 8] [--max-tokens 3000]"
(:require [clojure.string :as str]
[lev.config :as cfg]
[lev.json :as json]
[lev.llm :as llm]
[lev.program :as prog]
[lev.sequence :as seq]))

(def format-spec
"A program is one JSON object:
{\"labels\": [label, ...],
\"extract\": {fact: regex}, a fact holds when its (Java) regex finds a match in the input
\"questions\": {qid: question}, asked to a small text encoder when a rule needs them
\"rules\": [{\"when\": condition, \"then\": label}, ...], the first rule that holds names the label
\"default\": label} when no rule holds
A question is {\"type\": \"noul\", \"instructions\": \"a yes/no question about the input\"},
{\"type\": \"choice\", \"instructions\": \"...\", \"criteria\": {option: description}} or
{\"type\": \"score\", \"instructions\": \"...\", \"criteria\": [level, ...]}.
A condition is {\"fact\": f}, {\"noul\": qid, \">=\": p} (p the probability of yes; also > <= <),
{\"score\": qid, \">=\": x}, {\"choice\": qid, \"is\": option}, {\"choice\": qid, \"in\": [option, ...]},
{\"all\": [condition, ...]}, {\"any\": [condition, ...]} or {\"not\": condition}.
Facts cost nothing; questions cost about 50 ms each and the encoder is weak on long or technical text,
so use a regex wherever the input's wording decides the label, and a question where meaning does.
Numeric cutoffs are refitted on labelled data afterwards.")

(defn- sample [cases per-label]
(mapcat (fn [[label cs]]
(->> cs
(map #(get % "text"))
(reduce (fn [[seen out] t]
(let [k (str/replace t #"[0-9a-fA-F]{2,}|\d+" "N")]
(if (contains? seen k) [seen out] [(conj seen k) (conj out t)])))
[#{} []])
second
(take per-label)
(map (fn [t] [label t]))))
(group-by #(get % "label") cases)))

(defn- prompt-for [task cases per-label]
(str "Write a classifier program for this task: " task "\n\n" format-spec
"\n\nLabelled examples (label<TAB>input), from the training data:\n"
(str/join "\n" (map (fn [[l t]] (str l "\t" (subs t 0 (min 300 (count t))))) (sample cases per-label)))
"\n\nAnswer with the JSON program only."))

(defn- json-in [text]
(let [a (str/index-of text "{") b (str/last-index-of text "}")]
(when (and a b (< a b)) (json/read-str (subs text a (inc b))))))

(defn -main [& args]
(let [opts (cfg/parse-args args)
ctx (cfg/context opts)
gguf (or (get opts "--gguf") (:model (get (cfg/thinkers ctx) (get opts "--model" "qwen3.8-27b"))))
cases (map json/read-str (remove str/blank? (str/split-lines (slurp (str "bench/data/" (get opts "--split") ".jsonl")))))
m (llm/load gguf {:n-ctx 16384 :n-seq-max 1})
max-tokens (Long/parseLong (get opts "--max-tokens" "3000"))
ask (fn [messages]
(let [t0 (System/nanoTime)
r (llm/generate m (llm/chat-prompt m messages {:thinking false}) {:max-tokens max-tokens :temperature 0.0 :stop "<|im_end|>"})]
(println (format " %d tokens in %.1f s" (:tokens r) (/ (- (System/nanoTime) t0) 1e9)))
(:text r)))
first-msg {:role "user" :content (prompt-for (get opts "--task") cases (Long/parseLong (get opts "--per-label" "8")))}]
(println "compiling with" gguf)
(loop [messages [first-msg] tries 0]
(let [text (ask messages)
[program err] (try [(prog/validate (json-in text)) nil]
(catch Exception e [nil (ex-message e)]))]
(cond
program (do (spit (get opts "--save") (str (seq/json-str program) "\n"))
(println "valid program:" (count (get program "extract")) "facts,"
(count (get program "questions")) "questions," (count (get program "rules")) "rules")
(println (seq/json-str program)))
(< tries 2) (do (println "invalid:" err)
(recur (conj messages {:role "assistant" :content text}
{:role "user" :content (str "That program is invalid: " err ". Answer with the corrected JSON program only.")})
(inc tries)))
:else (do (println "gave up:" err) (println text) (System/exit 1)))))))

(apply -main *command-line-args*)
93 changes: 93 additions & 0 deletions bench/program.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
(ns bench.program
"Scores a lev.program on a labelled split (bench/program_data.py builds
them): accuracy, macro F1 over the labels, how many inputs needed the
model, how many answers escalated, and the time per input. The
baselines are programs too: the target asked directly as one question
and mapped to the label by one rule per answer.

jolt -M bench/program.clj --program bench/programs/toxic-direct.json --split toxic-test
[--model NAME] [--fit SPLIT [--save FILE]] [--limit N] [--out FILE] [--quiet]

--fit answers every question of the program on another split (the
train half) and fits its cutoffs there (lev.program/fit) before
scoring; --save writes the fitted program. --out writes one line per
input, {id label predicted rule answers}."
(:require [lev.json :as json]
[clojure.string :as str]
[lev.config :as cfg]
[lev.program :as prog]
[lev.router :as router]
[lev.sequence :as seq]))

(defn- read-jsonl [path]
(map json/read-str (remove str/blank? (str/split-lines (slurp path)))))

(defn- f1s [pairs labels]
(into (sorted-map)
(map (fn [l]
(let [tp (count (filter (fn [[g p]] (and (= g l) (= p l))) pairs))
fp (count (filter (fn [[g p]] (and (not= g l) (= p l))) pairs))
fn* (count (filter (fn [[g p]] (and (= g l) (not= p l))) pairs))
prec (if (pos? (+ tp fp)) (/ tp (double (+ tp fp))) 0.0)
rec (if (pos? (+ tp fn*)) (/ tp (double (+ tp fn*))) 0.0)]
[l {:f1 (if (pos? (+ prec rec)) (/ (* 2 prec rec) (+ prec rec)) 0.0) :n (+ tp fn*)}])))
labels))

(defn -main [& args]
(let [opts (cfg/parse-args args)
ctx (cfg/context opts)
program (prog/validate (json/read-str (slurp (get opts "--program"))))
cases (cond->> (read-jsonl (str "bench/data/" (get opts "--split") ".jsonl"))
(get opts "--limit") (take (Long/parseLong (get opts "--limit"))))
model (get opts "--model")
rt (router/make-router {:data (cfg/setting ctx "--data" "LEV_DATA" :data "data")
:checkpoints (into {} (map (fn [n] [n (cfg/limits ctx n)])) router/names)
:calibrations (cfg/calibrations ctx)
:thinkers (cfg/thinkers ctx)})
agent (if model (router/load-model rt model) rt)
agent (if (get program "escalate") rt agent)
program (if-let [fs (get opts "--fit")]
(let [fit-agent (if model agent (router/load-model rt "english"))
observed (mapv (fn [c] (assoc (prog/observe fit-agent (get c "text") program) "label" (get c "label")))
(read-jsonl (str "bench/data/" fs ".jsonl")))
r (prog/fit program observed)]
(println (format "fitted on %s (%d): accuracy %.3f -> %.3f" fs (count observed) (get r "before") (get r "accuracy")))
(when-let [o (get opts "--save")] (spit o (str (seq/json-str (get r "program")) "\n")))
(get r "program"))
program)
_ (prog/run agent (get (first cases) "text") program) ; warm
rows (vec (for [c cases]
(let [t0 (System/nanoTime)
r (prog/run agent (get c "text") program)
ms (/ (- (System/nanoTime) t0) 1e6)]
(when (and (not (contains? opts "--quiet")) (not= (get r "label") (get c "label")))
(println (format " miss %s: %s -> %s %s" (get c "id") (get c "label") (get r "label")
(subs (get c "text") 0 (min 90 (count (get c "text")))))))
{:id (get c "id") :answers (get r "answers") :rule (get r "rule")
:gold (get c "label") :pred (get r "label") :ms ms
:asked (some? (get r "answers"))
:escalated (count (get-in r ["escalation" "escalated"]))})))
_ (when-let [o (get opts "--out")]
(spit o (apply str (map (fn [{:keys [id gold pred rule answers]}]
(str (seq/json-str (seq/ordered-map [["id" id] ["label" gold] ["predicted" pred]
["rule" rule] ["answers" answers]]))
"\n"))
rows))))
pairs (map (juxt :gold :pred) rows)
labels (distinct (concat (map :gold rows) (map :pred rows)))
per (f1s pairs labels)
n (count rows)]
(println (format "%s on %s (%d inputs), model %s"
(get opts "--program") (get opts "--split") n
(or model (get-in program ["escalate" "model"]) "routed")))
(println (format " accuracy %.3f macro F1 %.3f model asked %d/%d answers escalated %d %.1f ms/input (p50 %.1f)"
(/ (count (filter (fn [[g p]] (= g p)) pairs)) (double n))
(/ (reduce + (map :f1 (vals per))) (double (count per)))
(count (filter :asked rows)) n
(reduce + (map :escalated rows))
(/ (reduce + (map :ms rows)) n)
(nth (vec (sort (map :ms rows))) (quot n 2))))
(doseq [[l {:keys [f1 n]}] per]
(println (format " %-10s F1 %.3f (n %d)" l f1 n)))))

(apply -main *command-line-args*)
Loading
Loading