From 0c24e501b5085382a8919a37b4b565484cd66c95 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Wed, 14 Oct 2026 23:21:23 -0400 Subject: [PATCH 1/3] Build on official llama.cpp; vendor the decision engine The parallel-decision fork changed nothing in libllama: its engine runs on the stock API. native/decision-engine.{h,cpp} is that engine (the class lev_decision.cpp drives, without the fork's schema compiler and server glue), with the two common/ batch helpers upstream dropped kept locally. llama.cpp is pinned at release b11630, 589 commits past the fork's base, until ggml-org/llama.cpp#29752 lands. --- .github/workflows/release.yml | 4 +- README.md | 18 +- deps.edn | 15 +- native/build_llm.sh | 15 +- native/decision-engine.cpp | 507 ++++++++++++++++++++++++++++++++++ native/decision-engine.h | 115 ++++++++ native/lev_decision.cpp | 6 +- 7 files changed, 652 insertions(+), 28 deletions(-) create mode 100644 native/decision-engine.cpp create mode 100644 native/decision-engine.h diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 85b785d..abeb2a2 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -56,8 +56,8 @@ jobs: - name: Test multilingual run: LEV_TEST_CHECKPOINTS=multilingual jolt -M:test lev.checkpoints-test - # jolt binary first runs the llama task: llama.cpp (thecodacus's - # parallel-decision fork) is fetched at its pinned commit and built static (cmake; Metal on mac, CPU elsewhere), so + # jolt binary first runs the llama task: llama.cpp is fetched at its + # pinned release and built static (cmake; Metal on mac, CPU elsewhere), so # the binary carries the thinker - name: Build and self-test the binary run: jolt binary diff --git a/README.md b/README.md index 5255136..4bfd152 100644 --- a/README.md +++ b/README.md @@ -144,7 +144,7 @@ models' templates use. Another model family needs its own template, which lives ``` jolt kernels # compile native/lev_kernels.c -jolt llama # fetch + build llama.cpp (the parallel-decision fork, pinned commit) into native/liblev_llm.*, the thinker's native +jolt llama # fetch + build llama.cpp (pinned release) with the vendored decision engine into native/liblev_llm.*, the thinker's native jolt mlx # mac: clone + build mlx-c (pinned tag) into native/liblev_mlx.*, the encoders' GPU backend jolt prepare # every checkpoint under ../laya -> data/, data/typed-decisions, ... jolt -M:test # parity suites vs golden/ @@ -333,8 +333,10 @@ a 503. ### Jev mode With thinking off, a thinker answers every question of a call in one -pass. `native/llama.cpp` is [thecodacus/llama.cpp's](https://github.com/thecodacus/llama.cpp/tree/parallel-decision) `parallel-decision` -branch, and its decision engine does the work. The chat up to the state +pass. The decision engine of [thecodacus/llama.cpp's](https://github.com/thecodacus/llama.cpp/tree/parallel-decision) +`parallel-decision` branch does the work. It is vendored in +`native/decision-engine.{h,cpp}` and built against official llama.cpp +until upstream takes it ([#29752](https://github.com/ggml-org/llama.cpp/pull/29752)). The chat up to the state is decoded once and kept for later calls. The state is decoded once per call. Each question is then a branch forked from the state in the KV cache, and every branch is decoded in the same batch. Each branch holds @@ -967,13 +969,13 @@ entries, and the build task branches on OS. `LEV_THREADS`. Attention calls `cblas_sgemm` through a pointer the Clojure side hands it, so the library links against no BLAS. - **llama.cpp**: `native/liblev_llm.dylib` / `.so` plus `liblev_llm.a`, - built by `jolt llama`. It fetches thecodacus/llama.cpp's - `parallel-decision` branch at its pinned commit (upstream b10435 plus - the decision engine) into `native/llama.cpp` and builds `llama` and + built by `jolt llama`. It fetches official llama.cpp at its pinned + release into `native/llama.cpp` and builds `llama` and `llama-common` static with cmake, with no https or subprocess support, Metal with the shader library embedded on mac and CPU elsewhere. They - sit behind `native/lev_llm.c` and `native/lev_decision.cpp` (the - engine's C face), which jolt.ffi binds as `lev.llm`. A checkout at + sit behind `native/lev_llm.c` and `native/lev_decision.cpp` (the C + face of the decision engine vendored in `native/decision-engine.cpp`), + which jolt.ffi binds as `lev.llm`. A checkout at another commit is replaced. `LLAMA_CPP_SRC=path` fetches from a local clone that has the commit. It is optional, since without it the encoders run and thinkers are diff --git a/deps.edn b/deps.edn index e44c3be..c4e01cc 100644 --- a/deps.edn +++ b/deps.edn @@ -109,14 +109,15 @@ (println "built:" kern-lib kern-archive)))} llama - {:doc "fetch llama.cpp at the pinned commit of thecodacus/llama.cpp's parallel-decision branch (upstream b10435+617 plus its decision engine) into native/llama.cpp, build it static, wrap it and the engine in native/liblev_llm.* (the thinker's native; LLAMA_CPP_SRC=path fetches from a local checkout holding that commit)" - :task (let [rev "14d04e755fa28653e87b9a07072892265bdc0fad" - src (or (System/getenv "LLAMA_CPP_SRC") "https://github.com/thecodacus/llama.cpp") + {:doc "fetch llama.cpp at the pinned release into native/llama.cpp, build it static, wrap it and the decision engine (native/decision-engine.cpp, vendored until llama.cpp takes it) in native/liblev_llm.* (the thinker's native; LLAMA_CPP_SRC=path fetches from a local checkout holding that commit)" + :task (let [release "b11630" + rev "6868a52cb337450658006ad2f9281c2eb63f1666" + src (or (System/getenv "LLAMA_CPP_SRC") "https://github.com/ggml-org/llama.cpp") dir "native/llama.cpp" head (str dir "/.git/HEAD") lib (str "native/liblev_llm." (if mac? "dylib" "so"))] - ;; a checkout of any other commit (the upstream tag this used to - ;; pin) is replaced, build and all + ;; a checkout of any other commit (the parallel-decision fork this + ;; used to pin) is replaced, build and all (when-not (and (fs/exists? head) (= rev (.trim (slurp head)))) (fs/delete-tree dir) (fs/create-dirs dir) @@ -139,9 +140,9 @@ (shell "cmake" "--build" (str dir "/build") "--target" "llama" "llama-common" "--parallel" (str (.availableProcessors (Runtime/getRuntime))))) (if (and (fs/exists? lib) (fs/exists? "native/liblev_llm.a") - (empty? (fs/modified-since lib ["native/lev_llm.c" "native/lev_llm.h" "native/lev_decision.cpp" "native/lev_clef.cpp" "native/build_llm.sh" head]))) + (empty? (fs/modified-since lib ["native/lev_llm.c" "native/lev_llm.h" "native/lev_decision.cpp" "native/decision-engine.cpp" "native/decision-engine.h" "native/lev_clef.cpp" "native/build_llm.sh" head]))) (println "llm: up to date -" lib "native/liblev_llm.a") - (shell "sh" "native/build_llm.sh" rev)))} + (shell "sh" "native/build_llm.sh" release)))} mlx {:doc "clone mlx-c at the pinned tag into native/mlx-c (it fetches mlx itself), build both static, wrap them in native/liblev_mlx.* with the Metal kernels beside it (the encoders' optional GPU backend, mac only; MLX_C_SRC=path clones from a local checkout)" diff --git a/native/build_llm.sh b/native/build_llm.sh index 618d804..46db3b0 100755 --- a/native/build_llm.sh +++ b/native/build_llm.sh @@ -1,16 +1,15 @@ #!/bin/sh # Build native/lev_llm.c and native/lev_decision.cpp (with the decision -# engine of the vendored fork, tools/parallel-decision) against llama.cpp -# (native/llama.cpp, fetched by the `jolt llama` task at the pinned commit) +# engine vendored in native/decision-engine.cpp) against llama.cpp +# (native/llama.cpp, fetched by the `jolt llama` task at the pinned release) # into liblev_llm.{dylib,so} for jolt run/test and liblev_llm.a for jolt -# build. Run from the project root; $1 is the commit, for lev_llm_version. +# build. Run from the project root; $1 is the release tag, for lev_llm_version. set -e L=native/llama.cpp -REV=${1:-$(cd "$L" && git rev-parse HEAD 2>/dev/null || echo unknown)} -TAG="thecodacus/parallel-decision@$(echo "$REV" | cut -c1-9)" +TAG="${1:-$(cd "$L" && git rev-parse --short HEAD 2>/dev/null || echo unknown)}" LIBS="$L/build/common/libllama-common.a $L/build/common/libllama-common-base.a $L/build/vendor/cpp-httplib/libcpp-httplib.a $L/build/src/libllama.a $L/build/ggml/src/libggml.a $L/build/ggml/src/libggml-cpu.a $L/build/ggml/src/libggml-base.a" -INC="-I$L/include -I$L/ggml/include -I$L/common -I$L/vendor -I$L/tools/parallel-decision -Inative" +INC="-I$L/include -I$L/ggml/include -I$L/common -I$L/vendor -Inative" CFLAGS="-O2 -std=c11 -Wall -Wextra -DLEV_LLAMA_BUILD=\"$TAG\" $INC" CXXFLAGS="-O2 -std=c++17 -Wall $INC" # lev_clef.cpp reads hidden states through the staging header in src/ @@ -23,7 +22,7 @@ case "$(uname -s)" in cc $CFLAGS -c native/lev_llm.c -o native/lev_llm.o c++ $CXXFLAGS -c native/lev_decision.cpp -o native/lev_decision.o c++ $CXXFLAGS $CLEF_INC -c native/lev_clef.cpp -o native/lev_clef.o - c++ $CXXFLAGS -w -c $L/tools/parallel-decision/decision-engine.cpp -o native/decision-engine.o + c++ $CXXFLAGS -c native/decision-engine.cpp -o native/decision-engine.o c++ -dynamiclib $OBJS $LIBS $FRAMEWORKS -o native/liblev_llm.dylib # one archive with every llama.cpp member inside, for jolt build's force-load libtool -static -o native/liblev_llm.a $OBJS $LIBS @@ -39,7 +38,7 @@ case "$(uname -s)" in cc $CFLAGS -fPIC -c native/lev_llm.c -o native/lev_llm.o c++ $CXXFLAGS -fPIC -c native/lev_decision.cpp -o native/lev_decision.o c++ $CXXFLAGS $CLEF_INC -fPIC -c native/lev_clef.cpp -o native/lev_clef.o - c++ $CXXFLAGS -w -fPIC -c $L/tools/parallel-decision/decision-engine.cpp -o native/decision-engine.o + c++ $CXXFLAGS -fPIC -c native/decision-engine.cpp -o native/decision-engine.o c++ -shared $OBJS -Wl,--whole-archive $LIBS -Wl,--no-whole-archive -lm -lpthread -o native/liblev_llm.so printf 'create native/liblev_llm.a\n' > native/llm.mri for o in $OBJS; do printf 'addmod %s\n' "$o" >> native/llm.mri; done diff --git a/native/decision-engine.cpp b/native/decision-engine.cpp new file mode 100644 index 0000000..51640a7 --- /dev/null +++ b/native/decision-engine.cpp @@ -0,0 +1,507 @@ +// Vendored from thecodacus/llama.cpp, branch parallel-decision, commit 14d04e755 +// (tools/parallel-decision/decision-engine.{h,cpp}; MIT, Copyright (c) 2023-2026 +// The ggml authors), until llama.cpp takes it (ggml-org/llama.cpp#29752). +// Kept here: the engine lev_decision.cpp drives. Left out: the schema compiler, +// the prompt renderer and assemble, the fork's CLI and server use them. +// Changed: common_batch_add / common_batch_clear, which left llama.cpp's +// common/ after the fork's base, are local (de_batch_add / de_batch_clear). + +#include "decision-engine.h" + +#include "common.h" + +#include +#include +#include +#include +#include +#include + +namespace llama_decision { + +namespace { + +void de_batch_clear(llama_batch & batch) { batch.n_tokens = 0; } + +void de_batch_add(llama_batch & batch, llama_token id, llama_pos pos, + const std::vector & seq_ids, bool logits) { + batch.token [batch.n_tokens] = id; + batch.pos [batch.n_tokens] = pos; + batch.n_seq_id[batch.n_tokens] = (int32_t) seq_ids.size(); + for (size_t i = 0; i < seq_ids.size(); ++i) { + batch.seq_id[batch.n_tokens][i] = seq_ids[i]; + } + batch.logits [batch.n_tokens] = logits; + batch.n_tokens++; +} + +} // namespace + +namespace { + +double ms_since(std::chrono::steady_clock::time_point t) { + return std::chrono::duration(std::chrono::steady_clock::now() - t).count(); +} + +struct decision_field { + tokens_t suffix; + std::vector paths; + + std::vector active; // greedy walk + tokens_t chosen; + + std::vector node_prefix; // tree: trie nodes with more than one allowed next token + std::vector node_options; + std::vector probs; + + bool use_tree = false; + int winner = -1; + float path_score = 1.0f; + int scored_nodes = 0; + + decision_field(tokens_t s, std::vector p) : suffix(std::move(s)), paths(std::move(p)) { + for (int i = 0; i < (int) paths.size(); ++i) { + active.push_back(i); + } + } + + // Greedy: follow single-path edges for free; return the options at the next divergence. + tokens_t options() { + while (active.size() > 1) { + const size_t depth = chosen.size(); + tokens_t opts; + for (int i : active) { + if (depth >= paths[i].size()) { + throw std::runtime_error("candidate paths must be distinct and terminated"); + } + if (std::find(opts.begin(), opts.end(), paths[i][depth]) == opts.end()) { + opts.push_back(paths[i][depth]); + } + } + if (opts.size() > 1) { + return opts; + } + chosen.push_back(opts[0]); + } + winner = active.empty() ? -1 : active[0]; + return {}; + } + + void select(llama_token tok, float p) { + const size_t depth = chosen.size(); + std::vector remaining; + for (int i : active) { + if (paths[i][depth] == tok) { + remaining.push_back(i); + } + } + active = remaining; + chosen.push_back(tok); + path_score *= p; + scored_nodes += 1; + } + + void build_nodes() { + std::vector, size_t>> stack; + std::vector root(paths.size()); + for (int i = 0; i < (int) paths.size(); ++i) { + root[i] = i; + } + stack.push_back({ root, 0 }); + while (!stack.empty()) { + auto [act, depth] = stack.back(); + stack.pop_back(); + if (act.size() <= 1) { + continue; + } + tokens_t opts; + for (int i : act) { + if (depth >= paths[i].size()) { + throw std::runtime_error("candidate paths must be distinct and terminated"); + } + if (std::find(opts.begin(), opts.end(), paths[i][depth]) == opts.end()) { + opts.push_back(paths[i][depth]); + } + } + if (opts.size() > 1) { + node_prefix.emplace_back(paths[act[0]].begin(), paths[act[0]].begin() + depth); + node_options.push_back(opts); + } + for (llama_token tok : opts) { + std::vector sub; + for (int i : act) { + if (paths[i][depth] == tok) { + sub.push_back(i); + } + } + stack.push_back({ sub, depth + 1 }); + } + } + } + + int tree_rows() const { + int n = 0; + for (const auto & p : node_prefix) { + n += (int) (suffix.size() + p.size()); + } + return n; + } + + // Exact constrained distribution: log-softmax at each node over its allowed tokens, + // summed along every candidate path, normalised over candidates. + void finish_tree(const std::vector> & node_scores) { + std::vector> node_logp; + for (const auto & s : node_scores) { + const float mx = *std::max_element(s.begin(), s.end()); + double z = 0; + for (float x : s) { + z += std::exp(x - mx); + } + const float lz = mx + (float) std::log(z); + std::vector lp; + for (float x : s) { + lp.push_back(x - lz); + } + node_logp.push_back(lp); + } + std::vector path_lp(paths.size(), 0.0f); + for (size_t i = 0; i < paths.size(); ++i) { + for (size_t n = 0; n < node_prefix.size(); ++n) { + const auto & pre = node_prefix[n]; + if (pre.size() >= paths[i].size() || !std::equal(pre.begin(), pre.end(), paths[i].begin())) { + continue; + } + const auto & opts = node_options[n]; + const auto it = std::find(opts.begin(), opts.end(), paths[i][pre.size()]); + if (it == opts.end()) { + throw std::runtime_error("candidate token missing from its trie node"); + } + path_lp[i] += node_logp[n][it - opts.begin()]; + } + } + const int best = (int) (std::max_element(path_lp.begin(), path_lp.end()) - path_lp.begin()); + double z = 0; + for (float x : path_lp) { + z += std::exp(x - path_lp[best]); + } + probs.clear(); + for (float x : path_lp) { + probs.push_back((float) (std::exp(x - path_lp[best]) / z)); + } + winner = best; + path_score = probs[best]; + scored_nodes = (int) node_prefix.size(); + } +}; + +} // namespace + +// ---------------------------------------------------------------- engine + +engine::engine(llama_context * ctx, llama_seq_id seq_base, int n_seqs) + : ctx(ctx), vocab(llama_model_get_vocab(llama_get_model(ctx))), mem(llama_get_memory(ctx)), + seq_snap(seq_base), seq_pool(seq_base + 1), n_pool(n_seqs - 1) { + if (n_seqs < 3) { + throw std::invalid_argument("a decision engine needs at least 3 sequences"); + } +} + +tokens_t engine::tokenize(const std::string & text, bool add_special) const { + tokens_t toks = common_tokenize(vocab, text, add_special, /*parse_special=*/ true); + // a chat template may already start with the BOS text; keep a single BOS + const llama_token bos = llama_vocab_bos(vocab); + if (toks.size() >= 2 && toks[0] == bos && toks[1] == bos) { + toks.erase(toks.begin()); + } + return toks; +} + +// Decode several prompts, each on its own sequence, packed into as few batches as n_batch allows. +void engine::decode_parts(const std::vector & parts) { + const int n_batch = (int) llama_n_batch(ctx); + llama_batch batch = llama_batch_init(n_batch, 0, 1); + auto flush = [&]() { + const int rc = batch.n_tokens > 0 ? llama_decode(ctx, batch) : 0; + de_batch_clear(batch); + if (rc != 0) { + llama_batch_free(batch); + throw std::runtime_error(rc == 1 ? "no free KV cache space for the decision prompt" + : "llama_decode failed on the decision prompt (" + std::to_string(rc) + ")"); + } + }; + for (const auto & p : parts) { + for (size_t i = 0; i < p.toks->size(); ++i) { + if (batch.n_tokens == n_batch) { + flush(); + } + de_batch_add(batch, (*p.toks)[i], p.pos0 + (llama_pos) i, { p.seq }, false); + } + } + flush(); + llama_batch_free(batch); +} + +// Restore (or build) the cached static prefix on seq_snap. Only this engine's own sequences are +// touched, so it can share a context with other users (e.g. server slots). +bool engine::prepare_prefix(const tokens_t & shared, bool allow_cache) { + if (allow_cache && !shared.empty() && shared == cached && + llama_memory_seq_pos_max(mem, seq_snap) == (llama_pos) cached.size() - 1) { + return true; + } + for (llama_seq_id s = seq_snap; s < seq_pool + n_pool; ++s) { + llama_memory_seq_rm(mem, s, -1, -1); + } + cached.clear(); + if (!shared.empty()) { + decode_parts({ { &shared, 0, seq_snap } }); + cached = shared; + } + return false; +} + +// Score each branch as its own sequence forked from its trunk; return each branch's last-token +// logits restricted to its candidate tokens. Groups are bounded by free sequences and batch rows. +std::vector> engine::score_branches(const std::vector & branches, llama_seq_id first, int n_free) { + std::vector> result(branches.size()); + const int max_rows = (int) llama_n_batch(ctx); + size_t start = 0; + while (start < branches.size()) { + size_t end = start; + int rows = 0; + while (end < branches.size() && (int) (end - start) < n_free && rows + (int) branches[end].toks.size() <= max_rows) { + rows += (int) branches[end].toks.size(); + ++end; + } + if (end == start) { + throw std::runtime_error("a decision suffix exceeds the batch size"); + } + llama_batch batch = llama_batch_init(rows, 0, 1); + std::vector out_idx; + for (size_t b = start; b < end; ++b) { + const llama_seq_id seq = first + (llama_seq_id) (b - start); + llama_memory_seq_rm(mem, seq, -1, -1); + llama_memory_seq_cp(mem, branches[b].trunk, seq, -1, -1); + const auto & toks = branches[b].toks; + for (size_t i = 0; i < toks.size(); ++i) { + const bool last = i + 1 == toks.size(); + if (last) { + out_idx.push_back(batch.n_tokens); + } + de_batch_add(batch, toks[i], branches[b].pos0 + (llama_pos) i, { seq }, last); + } + } + const int rc = llama_decode(ctx, batch); + llama_batch_free(batch); + if (rc != 0) { + throw std::runtime_error(rc == 1 ? "no free KV cache space for the decision branches" + : "llama_decode failed on the decision branches (" + std::to_string(rc) + ")"); + } + for (size_t b = start; b < end; ++b) { + const float * logits = llama_get_logits_ith(ctx, out_idx[b - start]); + for (llama_token t : branches[b].cands) { + result[b].push_back(logits[t]); + } + llama_memory_seq_rm(mem, first + (llama_seq_id) (b - start), -1, -1); + } + start = end; + } + return result; +} + +result engine::decide(const std::string & shared_text, const std::string & context_text, + const std::vector & inputs, const options & opt) { + batch_result b = decide_batch(shared_text, { context_text }, inputs, opt); + result r = std::move(b.items[0]); + r.cache_hit = b.cache_hit; + r.shared_tokens = b.shared_tokens; + r.rounds = b.rounds; + r.prefill_ms = b.prefill_ms; + r.scoring_ms = b.scoring_ms; + return r; +} + +batch_result engine::decide_batch(const std::string & shared_text, const std::vector & contexts, + const std::vector & inputs, const options & opt) { + if (opt.mode != "auto" && opt.mode != "tree" && opt.mode != "greedy") { + throw std::invalid_argument("mode must be auto, tree or greedy"); + } + if (contexts.empty()) { + throw std::invalid_argument("a decision needs at least one context"); + } + const tokens_t shared = tokenize(shared_text, true); + std::vector prefixes; + for (const auto & text : contexts) { + prefixes.push_back(tokenize(text, shared.empty())); + if (prefixes.back().empty()) { + throw std::invalid_argument("the decision context must not be empty"); + } + } + + std::vector fields; + int total = 0; + int branches = 0; // round-1 branches of one context + for (const auto & in : inputs) { + const size_t n = in.candidates.size(); + if (n < 1 || n > 255) { + throw std::invalid_argument("each field needs 1-255 allowed values"); + } + tokens_t suffix; + std::vector paths; + if (opt.split_boundary) { + for (const auto & c : in.candidates) { + paths.push_back(tokenize(c + "\n", false)); + } + suffix = tokenize(in.suffix, false); + } else { + // Tokenise each complete "suffix + value + terminator" and split at the longest token + // prefix shared by every candidate: the value's first token is exactly what the model + // would write itself (e.g. `":` then ` true`, not `": ` then `true`). + std::vector seqs; + for (const auto & c : in.candidates) { + seqs.push_back(tokenize(in.suffix + c + "\n", false)); + } + size_t common = seqs[0].size(); + for (const auto & s : seqs) { + size_t c = 0; + while (c < std::min(common, s.size()) && s[c] == seqs[0][c]) { + ++c; + } + common = std::min({ common, c, s.size() - 1 }); + } + suffix.assign(seqs[0].begin(), seqs[0].begin() + common); + for (const auto & s : seqs) { + paths.emplace_back(s.begin() + common, s.end()); + } + } + if (suffix.empty()) { + throw std::invalid_argument("a field suffix must not be empty"); + } + size_t max_path = 0; + for (size_t a = 0; a < paths.size(); ++a) { + max_path = std::max(max_path, paths[a].size()); + for (size_t b = 0; b < a; ++b) { + const size_t m = std::min(paths[a].size(), paths[b].size()); + if (std::equal(paths[a].begin(), paths[a].begin() + m, paths[b].begin())) { + throw std::invalid_argument("two allowed values tokenise to colliding paths"); + } + } + } + decision_field field(suffix, paths); + field.build_nodes(); + field.use_tree = opt.mode == "tree" ? true : opt.mode == "greedy" ? false : n <= opt.tree_max; + total += field.use_tree ? field.tree_rows() : (int) (suffix.size() + max_path); + branches += field.use_tree ? (int) field.node_prefix.size() : 1; + fields.push_back(std::move(field)); + } + + batch_result out; + out.shared_tokens = shared.size(); + out.rows = total * (int) contexts.size(); + out.items.resize(contexts.size()); + + const auto t0 = std::chrono::steady_clock::now(); + out.cache_hit = prepare_prefix(shared, opt.allow_cache); + out.prefill_ms += ms_since(t0); + + // each context in a group holds one trunk sequence; the rest of the pool scores branches + const size_t per_group = std::clamp(n_pool / (1 + branches), 1, contexts.size()); + for (size_t g0 = 0; g0 < contexts.size(); g0 += per_group) { + const size_t n_group = std::min(per_group, contexts.size() - g0); + + const auto tp = std::chrono::steady_clock::now(); + std::vector parts; + for (size_t i = 0; i < n_group; ++i) { + const llama_seq_id trunk = seq_pool + (llama_seq_id) i; + llama_memory_seq_rm(mem, trunk, -1, -1); + if (!shared.empty()) { + llama_memory_seq_cp(mem, seq_snap, trunk, -1, -1); + } + parts.push_back({ &prefixes[g0 + i], (llama_pos) shared.size(), trunk }); + } + decode_parts(parts); + llama_synchronize(ctx); // llama_decode is asynchronous: wait for the prefill so its time isn't billed to scoring + out.prefill_ms += ms_since(tp); + + // round 1 carries every tree node and each greedy field's first step; later rounds only + // continue greedy fields that are still open + const auto ts = std::chrono::steady_clock::now(); + std::vector> state(n_group, fields); + bool first = true; + while (true) { + std::vector todo; + std::vector> owner; // (context in group, field) + for (size_t i = 0; i < n_group; ++i) { + const llama_seq_id trunk = seq_pool + (llama_seq_id) i; + const llama_pos pos0 = (llama_pos) (shared.size() + prefixes[g0 + i].size()); + for (size_t f = 0; f < state[i].size(); ++f) { + auto & fd = state[i][f]; + if (fd.use_tree) { + if (first) { + for (size_t n = 0; n < fd.node_prefix.size(); ++n) { + tokens_t ids = fd.suffix; + ids.insert(ids.end(), fd.node_prefix[n].begin(), fd.node_prefix[n].end()); + todo.push_back({ trunk, pos0, ids, fd.node_options[n] }); + owner.push_back({ i, f }); + } + } + } else { + tokens_t opts = fd.options(); + if (!opts.empty()) { + tokens_t ids = fd.suffix; + ids.insert(ids.end(), fd.chosen.begin(), fd.chosen.end()); + todo.push_back({ trunk, pos0, ids, opts }); + owner.push_back({ i, f }); + } + } + } + } + if (todo.empty()) { + break; + } + const auto scores = score_branches(todo, seq_pool + (llama_seq_id) n_group, n_pool - (int) n_group); + out.rounds += 1; + std::vector>>> tree_scores(n_group, std::vector>>(fields.size())); + for (size_t row = 0; row < owner.size(); ++row) { + const auto [i, f] = owner[row]; + auto & fd = state[i][f]; + if (fd.use_tree) { + tree_scores[i][f].push_back(scores[row]); + } else { + const auto & s = scores[row]; + const int best = (int) (std::max_element(s.begin(), s.end()) - s.begin()); + double denom = 0; + for (float x : s) { + denom += std::exp(x - s[best]); + } + fd.select(todo[row].cands[best], (float) (1.0 / denom)); + } + } + if (first) { + for (size_t i = 0; i < n_group; ++i) { + for (size_t f = 0; f < fields.size(); ++f) { + if (state[i][f].use_tree) { + state[i][f].finish_tree(tree_scores[i][f]); + } + } + } + } + first = false; + } + for (size_t i = 0; i < n_group; ++i) { + llama_memory_seq_rm(mem, seq_pool + (llama_seq_id) i, -1, -1); + result & r = out.items[g0 + i]; + r.context_tokens = prefixes[g0 + i].size(); + r.rows = total; + for (auto & fd : state[i]) { + if (fd.use_tree && fd.probs.empty()) { + fd.finish_tree({}); + } + r.fields.push_back({ fd.winner, fd.path_score, fd.scored_nodes, fd.use_tree, fd.probs }); + } + } + out.scoring_ms += ms_since(ts); + } + return out; +} + +} // namespace llama_decision diff --git a/native/decision-engine.h b/native/decision-engine.h new file mode 100644 index 0000000..7a669c2 --- /dev/null +++ b/native/decision-engine.h @@ -0,0 +1,115 @@ +#pragma once + +// Vendored from thecodacus/llama.cpp, branch parallel-decision, commit 14d04e755 +// (tools/parallel-decision/decision-engine.{h,cpp}; MIT, Copyright (c) 2023-2026 +// The ggml authors), until llama.cpp takes it (ggml-org/llama.cpp#29752). +// Kept here: the engine lev_decision.cpp drives. Left out: the schema compiler, +// the prompt renderer and assemble, the fork's CLI and server use them. +// Changed: common_batch_add / common_batch_clear, which left llama.cpp's +// common/ after the fork's base, are local (de_batch_add / de_batch_clear). + +// Parallel constrained decisions: every field of a call scored in one batched decode. +// +// Every field of a schema has a finite set of allowed values. After a shared context, each +// field's value is scored as token paths following that field's own suffix; every scored path +// runs as its own sequence forked from the context (llama_memory_seq_cp), so all fields are +// evaluated in one batched llama_decode and cannot see each other. Small fields score every +// divergence node of their token trie at once and return the exact constrained distribution; +// larger fields walk the trie greedily. + +#include "llama.h" + +#include +#include +#include + +namespace llama_decision { + +using tokens_t = std::vector; + +// One field as the scorer sees it: the text before its value and the allowed value texts. +struct field_input { + std::string suffix; // e.g. ' "fire": ' + std::vector candidates; // allowed values, with the suffix's shared prefix removed +}; + +struct options { + std::string mode = "auto"; // auto: tree up to tree_max values, else greedy; tree; greedy + size_t tree_max = 128; + bool split_boundary = false; // legacy: tokenise suffix and values separately + bool allow_cache = true; // reuse the cached static prefix when it matches +}; + +struct field_result { + int winner = -1; + float path_score = 1.0f; + int scored_nodes = 0; + bool tree = false; + std::vector probs; // tree fields: probability of every allowed value +}; + +struct result { + std::vector fields; + bool cache_hit = false; + size_t shared_tokens = 0; + size_t context_tokens = 0; + int rows = 0; + int rounds = 0; + double prefill_ms = 0; + double scoring_ms = 0; +}; + +// Several contexts decided against one schema and one cached prefix. Items carry fields, +// context_tokens and rows; timings and cache state cover the whole batch. +struct batch_result { + std::vector items; + bool cache_hit = false; + size_t shared_tokens = 0; + int rows = 0; + int rounds = 0; + double prefill_ms = 0; + double scoring_ms = 0; +}; + +// Scores decisions on an existing context with the sequence ids [seq_base, seq_base + n_seqs): +// one keeps the cached static prefix; the rest hold one trunk (prefix + context) per context in +// flight, then branches. The context needs a unified KV cache so branches share the trunk's cells. +class engine { + public: + engine(llama_context * ctx, llama_seq_id seq_base, int n_seqs); + + result decide(const std::string & shared_text, const std::string & context_text, + const std::vector & fields, const options & opt); + + // Contexts are prefilled together and their branches scored together, in groups sized to fit + // the sequence budget; results keep the order of the contexts. + batch_result decide_batch(const std::string & shared_text, const std::vector & contexts, + const std::vector & fields, const options & opt); + + private: + struct prompt_part { + const tokens_t * toks; + llama_pos pos0; + llama_seq_id seq; + }; + struct branch { + llama_seq_id trunk; + llama_pos pos0; + tokens_t toks; + tokens_t cands; + }; + + llama_context * ctx; + const llama_vocab * vocab; + llama_memory_t mem; + llama_seq_id seq_snap, seq_pool; + int n_pool; + tokens_t cached; + + tokens_t tokenize(const std::string & text, bool add_special) const; + void decode_parts(const std::vector & parts); + bool prepare_prefix(const tokens_t & shared, bool allow_cache); + std::vector> score_branches(const std::vector & branches, llama_seq_id first, int n_free); +}; + +} // namespace llama_decision diff --git a/native/lev_decision.cpp b/native/lev_decision.cpp index aa0540d..b114056 100644 --- a/native/lev_decision.cpp +++ b/native/lev_decision.cpp @@ -1,6 +1,6 @@ -/* lev_decision.cpp — Jev mode: the decision engine of thecodacus/llama.cpp's - * parallel-decision branch (tools/parallel-decision/decision-engine.cpp) - * behind one flat C entry point for jolt.ffi. +/* lev_decision.cpp — Jev mode: the decision engine (native/decision-engine.cpp, + * vendored from thecodacus/llama.cpp's parallel-decision branch) behind one + * flat C entry point for jolt.ffi. * * A call answers a set of fields for one or more contexts in one pass: the * shared text (the part of the prompt every call repeats) is decoded once From a242c9916da387d7e0303c8d444f940a4473aca2 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Wed, 14 Oct 2026 23:21:23 -0400 Subject: [PATCH 2/3] prepare_test: find the checkpoints the way jolt prepare does It read LEV_CHECKPOINTS or ../laya and ignored config.edn's :checkpoints-home, which jolt prepare honours. --- test/lev/prepare_test.clj | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/test/lev/prepare_test.clj b/test/lev/prepare_test.clj index 0ef9d0d..a34a888 100644 --- a/test/lev/prepare_test.clj +++ b/test/lev/prepare_test.clj @@ -2,18 +2,20 @@ "jolt prepare must reproduce the reference Python conversion byte for byte: golden/prepare.edn pins the size and zlib CRC-32 of every file the Python converter wrote for this checkpoint. Needs the checkpoint - (LEV_CHECKPOINTS, default ../laya); writes to target/prepare-test and removes it." + (LEV_CHECKPOINTS, config.edn :checkpoints-home, else ../laya); writes to target/prepare-test and removes it." (:require [clojure.edn :as edn] [clojure.java.io :as io] [clojure.string :as str] [clojure.test :refer [deftest is testing]] + [lev.config :as cfg] [lev.prepare :as prep])) (def golden-dir (or (System/getenv "LEV_GOLDEN") "golden")) (def home - (or (System/getenv "LEV_CHECKPOINTS") "../laya")) + "Where jolt prepare looks: LEV_CHECKPOINTS, config.edn :checkpoints-home, ../laya." + (cfg/setting (cfg/context {}) "--checkpoints" "LEV_CHECKPOINTS" :checkpoints-home "../laya")) (deftest jolt-prepare-reproduces-python-conversion (is (.exists (io/file home "model.safetensors")) From 7b885926cce19942b01c6f3901206057f780f6f9 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Wed, 14 Oct 2026 23:43:44 -0400 Subject: [PATCH 3/3] bench/README: official llama.cpp against the fork, measured --- bench/README.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/bench/README.md b/bench/README.md index 02b80b1..08f893c 100644 --- a/bench/README.md +++ b/bench/README.md @@ -704,3 +704,34 @@ the held-out score are what separated 0.733 from 0.905. Timing was taken with other benches sharing the machine, so it is indicative only. + +## Official llama.cpp, the decision engine vendored (2026-10-15) + +The parallel-decision fork changed nothing in libllama: its decision +engine runs on the stock API (`llama_memory_seq_cp`, `seq_rm`, +`llama_decode`, `llama_get_logits_ith`). The engine is now vendored in +`native/decision-engine.{h,cpp}` and built against official llama.cpp +b11630, 589 commits past the fork's base. Only the class lev drives is +kept, and the two `common/` batch helpers upstream dropped are local. + +Same machine, back to back: upstream, the fork's library, upstream again. +The workflow rows use Qwen3.5-4B Q8 in Jev mode (thinking off), p50 over +20 iterations. The authored144 rows use MiniCPM5-2B Q8, thinking off. + +| | upstream b11630 | fork | upstream b11630 again | +|---|---|---|---| +| 4 questions, short state | 689 ms | 746 ms | 653 ms | +| 4 questions, long state (1.6k tokens) | 3,165 ms | 3,561 ms | 3,439 ms | +| 1 question, long state | 2,786 ms | 3,010 ms | 2,923 ms | +| 8 short states, one call each | 5,218 ms | 6,134 ms | 5,234 ms | +| 8 short states, one batch | 5,260 ms | 5,403 ms | 4,672 ms | +| authored144 (MiniCPM5) | 108/144, ECE 0.174, 116 ms | 108/144, ECE 0.174, 125 ms | 108/144, ECE 0.174, 112 ms | + +The answers do not move: authored144 is the same 108 cases with the same +calibration, and llm-test, think-test and clef-test pass against their +goldens. Upstream is 3–15% faster on every row. + +Qwen3.5-4B pays more per call than MiniCPM5 did in the Jev-mode section. +It is a hybrid model with recurrent layers, and llama.cpp splits its +batches by sequence, so its branches take several passes, not one. On a +long state the prefill dominates either way.