From 0bb9b25f67ebdb79aa2ea80ad3149af5cd5e17b2 Mon Sep 17 00:00:00 2001 From: Yogthos Date: Thu, 15 Oct 2026 01:26:52 -0400 Subject: [PATCH] Drop ICU: the tokenizers' Unicode comes from jolt's own tables NFC, NFD, lowercasing, the character categories and White_Space came from ICU through FFI. jolt carries the Unicode tables itself (Character/getType, java.text.Normalizer), so lev.tokenizer reads those, and the icu native is gone from deps.edn. That also fixes two portability problems. deficu picked the versioned symbol (u_charType_74) at expansion time, so a built binary ran only against the ICU major it was built with. And the candidate list missed ICU 70 (Ubuntu 22.04), where lev refused to start at all. char-type keeps ICU's numbering: the JVM's constants are the same but for the gap it leaves at 17. Checked over every code point against ICU 70: White_Space and NFC agree everywhere; categories, NFD and lowercase differ only on code points assigned or recategorized after Unicode 14, which ICU 70 predates. The suite gives the same results as with ICU. --- .github/actions/setup/action.yml | 12 +-- .github/workflows/release.yml | 2 +- README.md | 17 ++-- deps.edn | 6 -- src/lev/lang.clj | 6 +- src/lev/tokenizer.clj | 158 +++++++------------------------ 6 files changed, 48 insertions(+), 153 deletions(-) diff --git a/.github/actions/setup/action.yml b/.github/actions/setup/action.yml index 1d174da..0c03104 100644 --- a/.github/actions/setup/action.yml +++ b/.github/actions/setup/action.yml @@ -24,18 +24,16 @@ inputs: runs: using: composite steps: - # The tokenizer dlopens ICU and the matmul dlopens OpenBLAS at runtime - # (see :jolt/native in deps.edn). macOS has both in the system: libicucore - # and the Accelerate framework. The -dev packages also install the - # unversioned .so symlinks, which are the last fallback in deps.edn, so - # this keeps working when ubuntu-latest moves to a newer ICU. uuid-dev is - # for `jolt build`: the Chez launcher links -luuid on linux. + # The matmul dlopens OpenBLAS at runtime (see :jolt/native in + # deps.edn); macOS has the Accelerate framework. The -dev package also + # installs the unversioned .so symlink, the last fallback in deps.edn. + # uuid-dev is for `jolt build`: the Chez launcher links -luuid on linux. - name: System deps (Linux) if: runner.os == 'Linux' shell: bash run: | sudo apt-get update - sudo apt-get install -y --no-install-recommends libicu-dev libopenblas-dev uuid-dev + sudo apt-get install -y --no-install-recommends libopenblas-dev uuid-dev # Latest release by default; the JOLT_PIN repository variable overrides # without a commit. Echoed so the log records which compiler built what. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index abeb2a2..88ed8e8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -89,7 +89,7 @@ jobs: The binary does not carry the weights. Download the \`convaiinnovations/laya\` checkpoints you want (README, Getting the checkpoints) and convert them once with \`jolt prepare\`, then run \`./lev-server --data data --port 8080 --workflows workflows\`. \`./lev-server --self-test --data data --golden golden\` checks the pair. Requests route between the english, multilingual and typed-decisions checkpoints like the Python package's Router; unprepared ones answer 503. - Runtime deps on Linux: ICU and OpenBLAS (\`apt install libicu-dev libopenblas-dev\`, or the runtime packages for your release). macOS uses the system ICU and Accelerate. + Runtime deps on Linux: OpenBLAS (\`apt install libopenblas-dev\`, or the runtime package for your release). macOS uses Accelerate. The macOS binary is unsigned; clear the quarantine flag before first run: \`xattr -d com.apple.quarantine ./lev-server\`." \ || echo "release already exists, uploading asset" gh release upload "${GITHUB_REF_NAME}" "${PKG}" --clobber diff --git a/README.md b/README.md index 4bfd152..baea3d3 100644 --- a/README.md +++ b/README.md @@ -865,8 +865,8 @@ The binary still needs the prepared data root next to it, or `--data DIR`, or `config.edn`. It loads the workflow files from source at startup, so they need no rebuild, from `./workflows` relative to where it runs, `--workflows`, `LEV_WORKFLOWS` or `config.edn`, plus -`~/.config/lev/workflows`. From the OS it needs ICU and BLAS, and -libssl and libcrypto for the adapter. +`~/.config/lev/workflows`. From the OS it needs BLAS, and libssl and +libcrypto for the adapter. ``` ./lev-server --data data --workflows workflows --port 8080 --api-key s3cret @@ -997,11 +997,10 @@ entries, and the build task branches on OS. static link adds the QuartzCore framework to llama's set, and the binary needs `mlx.metallib` (the precompiled Metal kernels, 100 MB) beside it at runtime. -- **ICU**: `libicucore.dylib` on mac, whose system dylib leaves the - symbols unguarded, and `libicuuc.so.` on linux. The tokenizers - call `unorm2` for NFC and the `u_charType` / `u_isUWhiteSpace` - classifiers. Linux ICU builds append the major version to every symbol, like `u_charType_76`, and the bindings resolve the first spelling that exists, for versions - 60 to 90. +- **Unicode**: the tokenizers' NFC and NFD, lowercasing and character + classes come from jolt's own Unicode tables (`java.text.Normalizer`, + `Character/getType`), so no ICU is loaded and every host answers + alike. - **JSON**: `org.clojure/data.json` from Maven, plus `jolt-lang/time`, which provides the `java.time` classes data.json needs to load. Only `prepare` uses them. Requests are read by `lev.json`, which keeps key @@ -1011,9 +1010,7 @@ entries, and the build task branches on OS. - **BLAS**: `cblas_sgemm` from the Accelerate framework on mac, OpenBLAS on linux. -On linux, install the ICU and OpenBLAS runtime packages. Adjust the -version suffixes in `deps.edn` if your distro's `libicuuc.so` version is -not listed. +On linux, install the OpenBLAS runtime package. ## Status diff --git a/deps.edn b/deps.edn index c4e01cc..1fcd916 100644 --- a/deps.edn +++ b/deps.edn @@ -21,8 +21,6 @@ :git/sha "ff99a9d42761482851e40876c880ac4443cc4d96"}} ;; kernels: built from native/lev_kernels.c by the `kernels` task. - ;; icu: NFC for the tokenizer. mac ships libicucore in the shared cache; - ;; linux distros name it libicuuc.so.. ;; blas: Accelerate framework on mac, OpenBLAS elsewhere. ;; `jolt build` links the kernels statically from the archive so the binary ;; is self-contained; run/repl/test load the shared object. @@ -76,10 +74,6 @@ :darwin ["/System/Library/Frameworks/MetalKit.framework/MetalKit"] :linux [] :static {:darwin {:lib "MetalKit" :libdir "native/frameworks"}}} - {:name "icu" - :darwin ["libicucore.dylib"] - :linux ["libicuuc.so.76" "libicuuc.so.75" "libicuuc.so.74" - "libicuuc.so.72" "libicuuc.so.71" "libicuuc.so"]} ;; on mac the static llm archive (ggml-cpu) also needs Accelerate ;; at link time, through its stub like the frameworks above {:name "blas" diff --git a/src/lev/lang.clj b/src/lev/lang.clj index bd38c92..8206723 100644 --- a/src/lev/lang.clj +++ b/src/lev/lang.clj @@ -9,8 +9,8 @@ primary signal; whether Latin text is English is a best-effort stopword/diacritic guess. Pass an explicit model or lang when you know. - Python's str.isalpha / \\w classes come from ICU u_charType, as in the - tokenizer." + Python's str.isalpha / \\w classes come from the Unicode categories, + as in the tokenizer (lev.tokenizer/char-type)." (:require [clojure.string :as str] [lev.sequence :as seq] [lev.tokenizer :as tk])) @@ -76,7 +76,7 @@ "The class [^\\W\\d_] of Python's re: \\w (isalnum or _) minus decimal digits (Nd) minus underscore, i.e. letters plus Nl/No numerics." [c] - (let [t (tk/u-char-type* (int c))] + (let [t (tk/char-type c)] (or (and (>= t 1) (<= t 5)) (= t 10) (= t 11)))) (defn- words diff --git a/src/lev/tokenizer.clj b/src/lev/tokenizer.clj index e23a9e4..5f32698 100644 --- a/src/lev/tokenizer.clj +++ b/src/lev/tokenizer.clj @@ -2,8 +2,8 @@ "The two tokenizers in the checkpoint bundle, faithful to their tokenizer.json: :byte-level (english, typed-decisions) - GPT-2 byte-level BPE: NFC - normalize (ICU unorm2 via FFI), GPT-2 pre-tokenization (hand-written - scanner with exact \\p{L}/\\p{N} classes via ICU u_charType, since the + normalize, GPT-2 pre-tokenization (hand-written scanner with exact + \\p{L}/\\p{N} classes from the Unicode categories, since the pattern needs lookahead that irregex lacks), byte-to-unicode mapping, and lowest-rank-first BPE merges. @@ -17,147 +17,53 @@ (input_ids from the real transformers fast tokenizers) and the scanner by the ByteLevel pre-tokenizer boundaries in golden/cases.edn." (:require [clojure.edn :as edn] - [clojure.string :as str] - [jolt.ffi :as ffi])) - -;; --- ICU: NFC + character classes --------------------------------------------- - -(defmacro ^:private deficu - "defcfn for an ICU symbol. mac's libicucore exports plain names; linux ICU - builds append the major version (u_charType_76), so resolve the first name - that exists at expansion time. Extend the version list if your ICU is newer." - [name csym argtypes rettype] - (let [candidates (cons csym (map #(str csym "_" %) (range 90 59 -1))) - found (some #(when (ffi/find-symbol %) %) candidates)] - (when-not found - (throw (ex-info (str "ICU symbol not found: " csym - " (is libicuuc / libicucore declared in :jolt/native?)") - {:symbol csym}))) - `(def ~name (ffi/foreign-fn ~found ~argtypes ~rettype)))) - -;; takes UErrorCode* and reads it before doing anything: passing nothing -;; "worked" on mac by luck and segfaulted on linux -(deficu unorm2-get-instance* "unorm2_getNFCInstance" [:pointer] :pointer) -(deficu unorm2-normalize* "unorm2_normalize" - [:pointer :pointer :int32 :pointer :int32 :pointer] :int32) -(deficu unorm2-get-nfd-instance* "unorm2_getNFDInstance" [:pointer] :pointer) -(deficu u-str-to-lower* "u_strToLower" - [:pointer :int32 :pointer :int32 :pointer :pointer] :int32) -(deficu u-char-type* "u_charType" [:int32] :int32) -;; White_Space property == regex \\s. NOT u_isspace, which also admits the -;; U+001C..U+001F separators that the GPT-2 pattern treats as ordinary chars. -(deficu u-is-uwhitespace* "u_isUWhiteSpace" [:int32] :int8) - -(defn- normalizer [get-instance what] - (delay (with-open [a (ffi/confined-arena)] - (let [err (ffi/alloc a 4)] - (ffi/write err :int32 0 0) - (let [p (get-instance err) - code (ffi/read err :int32 0)] - (when (pos? code) - (throw (ex-info (str what " failed") {:code code}))) - p))))) - -(def nfc-instance (normalizer unorm2-get-instance* "unorm2_getNFCInstance")) -(def nfd-instance (normalizer unorm2-get-nfd-instance* "unorm2_getNFDInstance")) - -(def ^:private U_BUFFER_OVERFLOW_ERROR 15) - -(defn utf8->u16 - "Codepoint string -> vector of UTF-16 code units." - [s] - (loop [chs (seq s) out (transient [])] - (if (empty? chs) - (persistent! out) - (let [cp (long (first chs))] - (if (< cp 0x10000) - (recur (next chs) (conj! out cp)) - (let [v (- cp 0x10000)] - (recur (next chs) - (-> out - (conj! (bit-or 0xD800 (bit-shift-right v 10))) - (conj! (bit-or 0xDC00 (bit-and v 0x3FF))))))))))) - -(defn u16->string - "Vector of UTF-16 code units -> codepoint string (jolt strings are - codepoint-indexed, so surrogates collapse to single chars)." - [units] - (let [cps (loop [us (seq units) out ()] - (if (empty? us) - (reverse out) - (let [u (first us)] - (if (and (>= u 0xD800) (< u 0xDC00) - (seq (rest us)) - (>= (second us) 0xDC00) (< (second us) 0xE000)) - (recur (rest (rest us)) - (conj out (+ 0x10000 - (bit-shift-left (bit-and (- u 0xD800) 0x3FF) 10) - (bit-and (- (second us) 0xDC00) 0x3FF)))) - (recur (rest us) (conj out u))))))] - (apply str (map char cps)))) - -(defn- icu-transform - "s through an ICU UTF-16 transform (f src n dst cap err) -> length, - retried once with the length ICU reports on overflow, so no input fails - on buffer size." - [s f what] - (let [units (utf8->u16 s) - n (count units) - run (fn [cap] - (with-open [a (ffi/confined-arena)] - (let [src (ffi/alloc a (* 2 (max n 1))) - dst (ffi/alloc a (* 2 cap)) - err (ffi/alloc a 4)] - (dotimes [i n] - (ffi/write src :uint16 (bit-and 0xFFFF (nth units i)) (* 2 i))) - ;; ICU returns early on a pre-set error code - (ffi/write err :int32 0 0) - (let [ret (f a src (int n) dst (int cap) err) - code (ffi/read err :int32 0)] - (cond - (zero? code) (u16->string (mapv #(ffi/read dst :uint16 (* 2 %)) (range ret))) - (= code U_BUFFER_OVERFLOW_ERROR) [:retry ret] - :else (throw (ex-info (str what " failed") {:code code}))))))) - r (run (+ (* 3 n) 16))] - (if (vector? r) (run (inc (second r))) r))) + [clojure.string :as str])) + +;; --- Unicode: normalization + character classes ------------------------------ +;; From the runtime's own tables (jolt: Chez's, the same Unicode data the +;; JVM's Character and Normalizer read), so no ICU is loaded and every host +;; answers alike, whatever ICU it has. They track a newer Unicode than an +;; older ICU: the code points they differ on are the ones assigned since. + +(defn- ascii? [s] (every? #(< (long %) 128) s)) (defn nfd - "NFD-normalize via ICU. ASCII-only input is identity (fast path)." + "NFD-normalize. ASCII-only input is identity (fast path)." [s] - (if (every? #(< (long %) 128) s) - s - (icu-transform s (fn [_ src n dst cap err] (unorm2-normalize* @nfd-instance src n dst cap err)) - "unorm2_normalize"))) + (if (ascii? s) s (java.text.Normalizer/normalize s java.text.Normalizer$Form/NFD))) (defn lower "Full Unicode lowercase in the root locale (JavaScript's toLowerCase: İ is i + U+0307, never a Turkish dotless i)." [s] - (if (every? #(< (long %) 128) s) - (str/lower-case s) - (icu-transform s (fn [a src n dst cap err] (u-str-to-lower* dst cap src n (ffi/string->ptr a "") err)) - "u_strToLower"))) + (str/lower-case s)) (defn char-type "ICU's UCharCategory of a code point (1 Lu .. 6 Mn, 12 Zs, 15 Cc, 16 Cf, - 19..23 and 28..29 the punctuation)." + 19..23 and 28..29 the punctuation). The JVM's Character.getType numbers + them alike but skips 17, so its private use (18) to final punctuation + (30) are ICU's 17 to 29." [c] - (u-char-type* (int c))) + (let [t (Character/getType (int c))] + (if (> t 17) (dec t) t))) (defn nfc - "NFC-normalize via ICU. ASCII-only input is identity (fast path). NFC - can grow a string (composition exclusions decompose without - recomposing), by at most 3x: icu-transform starts there." + "NFC-normalize. ASCII-only input is identity (fast path)." [s] - (if (every? #(< (long %) 128) s) - s - (icu-transform s (fn [_ src n dst cap err] (unorm2-normalize* @nfc-instance src n dst cap err)) - "unorm2_normalize"))) + (if (ascii? s) s (java.text.Normalizer/normalize s java.text.Normalizer$Form/NFC))) + +(def ^:private white-space + "The White_Space property (PropList.txt), regex \\s: unchanged since + Unicode 6.3. Not Character/isWhitespace, which leaves out the no-break + spaces and takes the U+001C..U+001F separators the GPT-2 pattern treats + as ordinary chars." + (into #{0x20 0x85 0xA0 0x1680 0x2028 0x2029 0x202F 0x205F 0x3000} + (concat (range 0x09 0x0E) (range 0x2000 0x200B)))) ;; \\p{L} = UCharType 1..5; \\p{N} = 9..11 (Nd, Nl, No) -(defn letter? [c] (let [t (u-char-type* (int c))] (and (>= t 1) (<= t 5)))) -(defn number? [c] (let [t (u-char-type* (int c))] (and (>= t 9) (<= t 11)))) -(defn space? [c] (not (zero? (u-is-uwhitespace* (int c))))) +(defn letter? [c] (let [t (char-type c)] (and (>= t 1) (<= t 5)))) +(defn number? [c] (let [t (char-type c)] (and (>= t 9) (<= t 11)))) +(defn space? [c] (contains? white-space (long c))) ;; --- byte-level BPE -----------------------------------------------------------