diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..35e201f --- /dev/null +++ b/.gitattributes @@ -0,0 +1,6 @@ +# Files a Unix shell runs keep LF line endings on every checkout. Git for Windows checks text out +# with CRLF by default, and the WSL engine build runs scripts/*.sh straight from that checkout +# (BUILDING-ENGINES.md: bash "/mnt/d/.../scripts/build-static-engine.sh"), where bash stops at the +# first \r ("set: pipefail: invalid option name"). gradlew is the same for WSL, Linux and macOS. +*.sh text eol=lf +gradlew text eol=lf diff --git a/.github/workflows/pr-alpha.yml b/.github/workflows/pr-alpha.yml index 6847934..a9fad22 100644 --- a/.github/workflows/pr-alpha.yml +++ b/.github/workflows/pr-alpha.yml @@ -92,13 +92,15 @@ jobs: - name: Fetch libnode (nodejs-mobile ${{ env.NODEJS_MOBILE_VERSION }}, digidem 16 KB build) run: node scripts/fetch-libnode.mjs - # Daemon + web UI. stage-daemon-assets.mjs rebuilds the Android daemon bundle itself but - # deliberately not the web UI, so that is built here first (into turbollm/src/webdist). - - name: Build the TurboLLM web UI and install daemon deps + # Daemon + web UI packages. stage-daemon-assets.mjs builds both itself: the Android daemon + # bundle every time, and the web UI (into turbollm/src/webdist) whenever web/ is not what it + # was last built from there, which on this fresh checkout is always. A web build here would + # leave no record of its sources, so staging would build it a second time. + - name: Install the TurboLLM daemon and web UI packages working-directory: turbollm-src/turbollm run: | npm ci - (cd web && npm ci && npm run build) + (cd web && npm ci) - name: Materialise signing + Play credentials env: diff --git a/BUILDING-ENGINES.md b/BUILDING-ENGINES.md index 2f8a7a3..79dad28 100644 --- a/BUILDING-ENGINES.md +++ b/BUILDING-ENGINES.md @@ -16,11 +16,12 @@ paste into a search box in three months. ## What you actually need to do - **Just building the app?** Nothing special — `./gradlew assembleDebug` stages the Node - daemon into the APK for you (`stageDaemonAssets`). It only needs the `turbollm` checkout - beside this repo, or `-Pturbollm.dir=`. See - [the daemon payload](#the-daemon-payload--what-makes-the-apk-self-sufficient) — read that - before touching anything about how the daemon is packaged, because the failure mode is an - APK that builds and installs cleanly and then dies at first launch. + daemon into the APK for you (`stageDaemonAssets`), and builds the web UI again first + whenever `turbollm/web` has changed since its last build. It only needs the `turbollm` + checkout beside this repo, or `-Pturbollm.dir=`, with `npm ci` run in it and in its + `web/`. See [the daemon payload](#the-daemon-payload--what-makes-the-apk-self-sufficient) + — read that before touching anything about how the daemon is packaged, because the failure + mode is an APK that builds and installs cleanly and then dies at first launch. - **Shipping a new CPU engine?** You don't build it by hand. Bump `LLAMA_BUILD` in `turbollm/src/engines/download.ts`, push to `main`, and the [`android-llama-build.yml`](../.github/workflows/android-llama-build.yml) workflow @@ -37,6 +38,9 @@ paste into a search box in three months. build its server with the NDK AGP already uses (`stageLitertLmEngine`, needs network on the first build). `-PturbollmLitertLmEngine=false` leaves it out. See [the LiteRT-LM engine](#the-litert-lm-engine). +- **The engine tether?** Nothing to do — every build compiles `libtether_exec.so`, the helper + that makes the kernel end each engine with the app, with the same NDK (`stageTetherExec`). See + [the engine tether](#the-engine-tether). - **Before you change anything about packaging**, read [the platform constraint](#the-platform-constraint-that-shapes-everything). It is the reason the binaries are bundled rather than downloaded, the reason they are named @@ -64,8 +68,17 @@ The script expects the `turbollm` checkout as a sibling of this repo. If it live ``` It rebuilds the daemon bundle every time (~200 ms) rather than trusting what is on disk, so the -APK can never ship a daemon older than the source. It does **not** rebuild the web UI — that is -slower and changes less often, so run `npm run build` in `turbollm/web` when the UI changes. +APK can never ship a daemon older than the source. The web UI (`turbollm/src/webdist`, which git +does not hold) takes 40-55 s to build, so the script builds it only when `turbollm/web` is not +what it was last built from: it hashes every file under `web/` but `node_modules`, +dot-directories and test output (under 50 ms) and compares that with `build-inputs.json`, the +stamp it writes into `webdist` after each web build it runs. When they differ (after a pull or a +branch switch, or when the web UI was built by hand, which leaves no stamp) it runs +`npm run build` in `turbollm/web` before staging, so an APK never carries a web UI older than its +checkout. **You never need to build the web UI yourself for the app.** It does need +`turbollm/web/node_modules` as `turbollm/web/package-lock.json` has it: when that is missing or +out of date the build stops and names the command to run, `npm --prefix "/web" ci`. +`node --test scripts/stage-daemon-assets.test.mjs` covers the check on a stand-in checkout. ### Why the payload is built specially instead of just copying `dist/` @@ -87,14 +100,15 @@ about 220 MB of which is `@earendil-works/pi-*` plus its `@mistralai`/`@google`/ `turbollm/tsup.android.config.ts` is a **separate build pass** that bundles the dependencies in. Total payload: **14.5 MB**. -Four things in that config are load-bearing, each one a crash that was hit and fixed: +Four things in that build are load-bearing, each one a crash that was hit and fixed. The first +three are set in that config; the last is written by `stage-daemon-assets.mjs`: | Setting | Without it | |---|---| | `target: 'node18'` | nodejs-mobile embeds Node **18.20.4**, not the 22 the desktop package targets — syntax the phone cannot parse | | `banner` defines `require` | bundled CommonJS keeps esbuild's `__require` shim, which throws: `Dynamic require of "assert" is not supported` (undici) | | `banner` defines `__dirname` | `ReferenceError: __dirname is not defined in ES module scope` — sql.js locates its own `.wasm` that way | -| `dist/package.json` = `{"type":"module"}` | Node decides ESM by walking up for the nearest `package.json`; without one above the extracted bundle it reads the ESM entrypoint as CommonJS and dies on `import` | +| `daemon/package.json` = `{"type":"module","version":…}` | Node decides ESM by walking up for the nearest `package.json`; without one above the extracted bundle it reads the ESM entrypoint as CommonJS and dies on `import`. The daemon also reads its version from the file's `version` field (`dist/../package.json`) and reports 0.1.1 when that field is missing | Two files cannot be bundled and are staged beside the bundle instead: @@ -126,11 +140,38 @@ adb install composeApp/build/outputs/apk/debug/composeApp-debug.apk adb shell am start -n com.turbollm/com.turbollm.app.MainActivity # then, with NOTHING pushed: adb logcat -d | grep -E "E nodejs|TurboLLM:" -adb forward tcp:7799 tcp:7000 && curl -s http://127.0.0.1:7799/api/v1/engines +adb forward tcp:7799 tcp:7000 && curl -s http://127.0.0.1:7799/healthz # {"status":"ok","version":…} +adb shell "run-as com.turbollm cat files/daemon.log" | grep "registered bundled engine" ``` -Last run of exactly that: daemon booted, UI served HTTP 200, and both engines registered -(`llama-server-android` and `llama-server-android-vulkan`). +Last recorded run (before the app secret, when the check still read `/api/v1/engines`): daemon +booted, UI served HTTP 200, and both engines registered (`llama-server-android` and +`llama-server-android-vulkan`). + +`/api/v1/engines` over the adb forward now answers 401, and that is not a failed boot. The app +starts the daemon with a per-launch secret (`TURBOLLM_APP_SECRET`) that only the app's own +WebView and client carry, so adb, curl and Termux count as another app on the phone and need an +API key like any LAN client. `/healthz` stays open, and on the first launch after a clean +install, as in the procedure above, the daemon logs one `android: registered bundled engine …` +line per engine to `files/daemon.log`. Only that launch registers them: later boots log +`android: refreshed bundled engine …` or `android: repaired stale engine path …` for an engine +whose path or version changed and nothing for the others, and `daemon.log` is truncated each +time Node starts. So on a phone that already had the app, an empty grep does not mean the +engines are missing. Uninstalling first does not always make the next launch a first one +either: Android restores an app's Auto Backup whenever it is installed, `adb install` included, +if the backup was made by an APK signed with the same key (for the debug APK above, one of your +own earlier debug builds). On such a phone the restored `.turbollm/config.json` already lists +the engines, at the old install's library path. That launch then logs the `refreshed …` or +`repaired …` line for an engine whose path or version changed and nothing for the others, and +the grep above finds nothing. To keep the check a first registration, turn install-time restore +off around it: note what `adb shell settings get secure backup_auto_restore` prints (`1`, or +`null` if it was never changed, means on), run `adb shell bmgr autorestore false` before +`adb install`, and once done run `adb shell bmgr autorestore true` only if it was on, since it +is the phone's own Automatic restore setting. Or read the engines from the API. To query the +API itself, create a key in the app (Customize → Connect → API keys) and send it as +`X-TurboLLM-Auth: `. Only a debug build lets an adb host skip the key: its WebView accepts +Chrome DevTools over adb, and script run there acts as the app. Release builds turn WebView +debugging off. --- @@ -199,19 +240,77 @@ Termux users get their best speed from LiteRT-LM, which TurboLLM runs there as `python -m litert_lm_cli.main serve`. The app can't: it has no Python, and it may only execute files that shipped in its APK. But that Python server is a thin layer — every inference call goes through ctypes into **one** native library, `liblitert-lm.so`, the only binary in Google's -`litert-lm-api` wheel. So the app ships that same library unchanged and replaces only the layer -above it: +`litert-lm-api` wheel. So the app ships that same library, with one dependency renamed (see +[libandroid.so and its shim](#libandroidso-and-its-shim)), and replaces only the layer above it: | File (jniLibs/<abi>/) | What it is | Where it comes from | |---|---|---| -| `liblitert-lm.so` | Google's LiteRT-LM runtime (CPU + GPU) | Extracted from the `litert-lm-api` 0.18.0 Android wheel on PyPI, **sha256-pinned** in `scripts/stage-litert-lm-engine.mjs`. Byte-identical to what Termux runs. Apache-2.0. | +| `liblitert-lm.so` | Google's LiteRT-LM runtime (CPU + GPU) | Extracted from the `litert-lm-api` 0.18.0 Android wheel on PyPI, **sha256-pinned** in `scripts/stage-litert-lm-engine.mjs`. The build renames its `libandroid.so` dependency to `liblrtandr.so` (7 bytes of `.dynstr`) and checks the result against a second pinned hash; otherwise byte-identical to what Termux runs. Apache-2.0. | | `liblitertlm_server.so` | Native port of `litert-lm serve` — an executable named like a library so Android installs it | Built from `engines/litert-lm-server/` with the NDK on every build | +| `liblrtandr.so` | Stands in for `libandroid.so`, which the engine process cannot load | Generated from the library's own imports and built with the NDK on every build | -`stageLitertLmEngine` (`composeApp/build.gradle.kts`) does both, then checks each file's ELF +`stageLitertLmEngine` (`composeApp/build.gradle.kts`) does all three, then checks each file's ELF architecture and that every LOAD segment is 16 KB-aligned (the wheel's library already is). Output goes to `composeApp/build/generated/litert-lm/jniLibs/`, never into `src/`. Cost: about 14 MB compressed per ABI. +### libandroid.so and its shim + +`liblitert-lm.so` is built for apps, which load it into their own process, and it lists +`libandroid.so` among its dependencies. The server is not an app process: the daemon executes it +from `nativeLibraryDir`, and in a freshly executed process the linker has to find all of +`libandroid.so`'s dependencies itself (`libandroid_runtime`, `libhwui`, the media stack), which an +app, forked from the zygote, already has loaded. On the owner's phone that failed and no +`.litertlm` ever loaded: + +``` +cannot load liblitert-lm.so: + .../lib/arm64/liblitert-lm.so: dlopen failed: library "libstagefright.so" not found: needed by /system/lib64/libmediandk.so in namespace (default) +``` + +The library uses `libandroid.so` only to read APK assets (`AAssetManager_*`, `AAsset_*`), and the +server loads models from file paths. So `scripts/stage-litert-lm-engine.mjs` replaces that +dependency, working from the library it ships rather than from a list written for one version: + +1. It reads the library's dynamic section: its `DT_NEEDED` entries and every undefined dynamic + symbol with its binding and type. `LOADABLE_LIBS` names the libraries a standalone process can + load (bionic's, `liblog`, `libz`, EGL/GLES/Vulkan/`libnativewindow`); every other `DT_NEEDED` + entry is replaced. Today that is `libandroid.so` alone. +2. Against the NDK's stub libraries, at every API level from the app's minSdk (24) to the newest + the NDK has, it takes the GLOBAL imports only a replaced library provides (today 8 functions: + `AAssetManager_fromJava`, `AAssetManager_open`, `AAssetManager_openDir`, + `AAssetDir_getNextFileName`, `AAssetDir_close`, `AAsset_close`, `AAsset_getBuffer`, + `AAsset_getLength`). The build fails, naming the symbols, if one of them is data rather than a + function, if a GLOBAL import has no provider at all, if a GLOBAL import comes from both a + replaced and a loadable library, if a WEAK one comes only from a replaced library, or if a + replaced library's imports carry symbol versions, at any of those levels: a WEAK import that + `libandroid.so` gained some levels before `libnativewindow.so` did would be unresolved on the + devices in between, even though the minSdk and the newest stubs agree. +3. It writes a C shim per replaced library into the build directory and builds it with the NDK's + clang: one function per import, each of which, if it is ever called, writes + `liblitert-lm.so called (), which libandroid.so provides, but libandroid.so cannot be + loaded in the engine process (liblrtandr.so stands in for it); stopping.` to stderr (which the + daemon shows when the engine exits) and to logcat, then calls `abort()`. None returns a value. + The shim's file name has the replaced name's length, fits `lib*.so` and starts with `liblr`; + the build checks that it exports exactly those functions and needs only loadable libraries. A + shim an earlier run staged and this one does not is removed; nothing else in the output + directory is touched, so pointing `--out` at `composeApp/src/androidMain/jniLibs` by hand + leaves the llama.cpp engines there alone. +4. It renames the `DT_NEEDED` string in place (refusing if it is the tail of a longer string or + if any symbol, version entry or other dynamic tag shares its bytes; the second copy of + `libandroid.so` in `.rodata`, next to TFLite's NNAPI strings, is left alone) and checks the + file against `patchedSha256`. +5. It links a probe executable against the staged library, the shim and the NDK stubs of the + library's other dependencies with `-Wl,--no-allow-shlib-undefined`, so a GLOBAL import nothing + in the library's dependency list provides fails the build with the linker's own message. + +WEAK imports get no shim function: the library allows them to be absent. `AHardwareBuffer_*` is +the case that matters: `libandroid.so` defines those too from API 26, and with it gone they bind to +`libnativewindow.so`'s, which `libEGL.so` needs (checked on the Android 15 emulator image). + +`node --test scripts/stage-litert-lm-engine.test.mjs` covers the reading, the checks, the rename, +the naming and the removal of stale shims on synthetic ELF files. + ### What the server is, and why it is shaped this way `engines/litert-lm-server/src/main.cc` is a translation of the Python serve modules @@ -232,6 +331,140 @@ for thinking-channel text, `finish_reason: "length"` when the token budget runs cancels a generation when its client disconnects. Not ported: `/v1/responses`, `/v1/embeddings` and http(s) image URLs (TurboLLM uses none of them with this engine). +**Constrained decoding is on only for a request with tools**, which is all it is for: the data +processors that support it (Gemma 3's, Gemma 4's and FunctionGemma's) then constrain the model's +tool calls to the tools given; the others, Qwen3's for one, never do. The Python server turns it +on for every request (0.18.0 and upstream's main), and some data processors refuse it whatever +the request: LFM2's always, and Gemma 3's, Gemma 4's and FunctionGemma's when the FST constraint +library rejects the model's vocabulary (gemma-3-270m-it's does). Every request to such a model +fails with `litert-lm serve`, and did here. A request with tools to such a model is now answered +without the constraint (the model's own tool-call parsing still applies, and a call it writes in +a form that parsing cannot read fails the request, as for any model whose calls are not +constrained), and the server logs once per load that the model cannot constrain tool calls. A +`response_format` has its own constraint (llguidance) and works on these models too; one the +model cannot follow, or whose regex or JSON schema llguidance rejects, gets a 400 that names the +reason instead of a 500. `node --test engines/litert-lm-server/constrained-decoding.test.mjs` +checks this against a stand-in `liblitert-lm.so` that answers as 0.18.0 does for those models, +and on real models when `LITERT_LM_TEST_LIBRARY` and `LITERT_LM_TEST_MODELS` are set (see the +file). + +**A model whose decoder keeps a recurrent state gets a new engine for each conversation after +the first.** LiteRT-LM 0.18.0 starts every new conversation on an engine at step 0 with the +executor's buffers as the last one left them: right for a KV cache, whose entries past the step +are masked, but not for LFM2's short-convolution cache, which is read as it is, so each chat +turn (each is a new request, and the daemon's title, memory and compaction requests come in +between) began from the end of the one before (LiteRT-LM #3165). The server tells such a model +by its own header: LlmMetadata's model type is LFM2, or ExecutorMetadata declares a +linear-attention state or LFM2's `kv_cache_c_*` caches (files packed as `generic_model` declare +those too). The C API cannot reset that state, so the engine is created again with the same +settings: 0.13 to 0.17 s per conversation for LFM2.5-230M on a 4-core x86-64 CPU with the +compiled-model cache, 0.9 s without it, and about 1.5 s for LFM2.5-VL-450M with the cache (its +decoder, not its vision encoder, takes that long); GPU unmeasured. The daemon's title, memory +and compaction requests are conversations too, and each pays it. Sending a request that only +extends the previous conversation to that conversation instead is not done: LiteRT-LM's own +multi-turn continuation does not prefill the tokens a fresh render of the same history does +(other token counts, other replies, for Gemma models too), so it would change what the model +answers. `node --test engines/litert-lm-server/model-settings.test.mjs` checks which models get +a new engine against the stand-in library, and on real models (see the file) that a +conversation's reply does not depend on the ones before it. + +**A model that takes images or audio gets its encoders**, on the backend its file prefers (the +encoder section's first `backend_constraint`, else CPU, as `litert-lm serve` chooses) or on the +config's `vision_backend` or `audio_backend`. Without a backend named for it LiteRT-LM creates no +vision or audio executor, and every image sent to LFM2.5-VL-450M got a 500. The encoders come +with the engine, so a text-only chat with such a model holds them too: about 0.2 GB more +resident memory for LFM2.5-VL-450M on CPU without the compiled-model cache; GPU unmeasured. An +image or audio part sent to a model without that input is a 400 that says so, before anything +loads. Which inputs a file takes, on which backends, the server reads from the file's header +itself, as LiteRT-LM does. The C API's `litert_lm_loaded_file_*` answer the same, but read whole +model sections into memory to do it (every vision section, and the main model when its metadata +gives no context size): 285 MB for gemma-3-270m-it, before any engine loads. + +**Sampling starts from the model's own parameters.** A session's sampler replaces the model's +(LlmMetadata's `sampler_params`) entirely, so every conversation is created with the model's own +top_k, top_p and temperature (read from the file's LlmMetadata), each field the request sets put +in its place, and a random seed unless the request gives one. A model with no parameters of its +own (LFM2.5-230M, gemma-3-270m-it) gets LiteRT-LM's defaults, which are greedy (top_k 1), so a +temperature or top_p sent without a top_k there samples with no top-k limit (on CPU the largest +k, which its sampler clamps to the vocabulary; on GPU the model's vocabulary size, its number of +logits), and the server logs it once per model. The CPU sampler implements only TOP_P, so a +model whose own type is TOP_K gets TOP_P with p 1. On NPU, and for an Artisan text decoder on GPU +(which LiteRT-LM runs as GPU_ARTISAN; Gemma 4's `-gpu` bundles have one), nothing is set unless +the request sets a field, as before: LiteRT-LM leaves sampling to the model there. A `top_k` of 0 +or below means no top-k limit, as in llama.cpp (TurboLLM's Top K 0 means disabled), and one above +0 is passed as set, on GPU too, where LiteRT-LM 0.18.0 sets no limit of its own; a `top_p` +outside 0 to 1, a negative `temperature`, and a `top_k` or `seed` with a fraction are a 400 that +names the field instead of the runtime's 500. + +**Thinking is asked for the ways clients ask for it.** In order, the first that says: the +top-level `reasoning_effort` (what the Python server reads, and what the daemon sends for a +LiteRT-LM model), `chat_template_kwargs.enable_thinking`, +`chat_template_kwargs.reasoning_effort`, then `thinking_budget_tokens` (0 is off, n > 0 a budget +of n tokens, which also applies when another field turned thinking on), then the config's +`thinking` keys. `chat_template_kwargs` is passed to the model's template as its extra context, +with the `enable_thinking` decided above so that the template and the decoder's budget agree, +and its other keys (`preserve_thinking`, which LFM2's template reads) as sent. A model that does +not think (its metadata says so, or it has neither the template's switch nor a thinking channel: +gemma-3-270m-it) gets a 400 for thinking on, and thinking off changes nothing there. A model +whose file does not say and whose template has no switch, but that has a thinking channel +(LFM2.5-230M, which does not think, and LFM2.5-1.2B-Thinking, which always does; LFM2-700M's and +LFM2.5-350M's say they do not), gets the request's thinking as asked: LiteRT-LM 0.18.0 cannot +stop such a model from thinking (a budget of 0 sets no constraint), and nothing in the file tells +the two apart, so a 400 for one would refuse the other. The server logs that once, and every such +reply carries `X-Thinking: unverified`, since a client sees nothing of the server's log. + +**OpenAI's `stop` is applied by the server**, which the C API cannot do (it knows only the +model's stop tokens): the reply's text ends before the first stop string, streamed or not, and +the generation is cancelled there, with `finish_reason` "stop". Text that could be the start of +a stop string is held back until the next chunk tells, so no part of one is ever sent. It +applies to the reply's text only, not to its thinking or tool calls, as OpenAI's does. + +**Repetition penalties go through LiteRT-LM's own** (`RepetitionPenaltyConfig`, applied per +message while it decodes): `presence_penalty` and `frequency_penalty` as OpenAI defines them, +and `repetition_penalty`, or llama.cpp's `repeat_penalty`, which the daemon sends +(multiplicative, 1 is none). The window is `repeat_last_n` when the request sets it, as +llama.cpp reads it (0 turns the penalties off, -1 is the whole reply); else the last 64 tokens +for `repeat_penalty`, llama.cpp's default; else the whole reply. LiteRT-LM counts the reply's +tokens only. A repetition penalty below 1, which LiteRT-LM would clamp to 1 without a word, is a +400. `min_p`, which it has no sampler for, is not applied, and a reply to a request with one +above 0 says so in its `X-Unapplied-Parameters: min_p` header: a 400 would fail every chat whose +saved settings have a Min P, chats that worked when it was dropped unread. + +**`tool_choice` "none" leaves the tools out of the conversation** (earlier calls and results +stay in the history), and "auto" or none at all keeps them. LiteRT-LM 0.18.0's C API cannot make +a model call a tool (constrained decoding keeps a call well formed; Gemma's "function call only" +mode comes from the model file), so `tool_choice` "required" or a named function is a 400 that +says so. + +**A context larger than the model takes is cut to the model's, and said.** The runtime used the +model's own limit with only a warning in its log (LFM2.5-230M has a fixed 4096), and "length" +was judged against the size asked for. Now the engine is given the model's limit (its +LlmMetadata's `max_num_tokens`, or where that is missing, as in gemma-3-270m-it, the size of its +prefill signature's mask, which is where LiteRT-LM reads it), the server logs that once per +model and size asked, and it reports the window in each chat reply's `X-Context-Window` header +and as `max_model_len` on `/v1/models`, with `X-Context-Window-Requested` and +`max_model_len_requested` when more was asked. That window is the one LiteRT-LM gives the +engine, worked out as it does: with no size asked for, the model's own, but 4096 for a file that +gives none and on GPU no more than 4096; for a model with a dynamic context (a prime above 10, +its magic number), the size asked when it is below that number, and otherwise LiteRT-LM's +default for it, the largest multiple of 256 below it (below 256, the largest power of two), also +with none asked. A prompt that does not fit the window is a 400 that names its size; in a +stream, already answered 200, it is the `response.error` event, with the same words. + +**With no backend in the model reference or the config, the model's own is used**: the first its +header lists for its text (a GPU-only bundle, Gemma 4's `-gpu` one among them, lists gpu alone; a +file without a backend constraint, LFM2.5-230M for one, lists cpu first; an Artisan text decoder +puts gpu first), as `litert-lm serve` picks it. The server picked CPU for every model, so a +GPU-only bundle sent without a backend failed to load. When the model's own backend does not +start, the engine falls back to the next one the model lists, CPU for most, and never to one it +does not list, and the server logs why; when none starts, the error gives each one's reason. A +backend the user set is used as set, failing as before. + +**A conversation that ends with the assistant's own words is a 400.** A client that prefills the +start of a reply asks for it to be continued, which LiteRT-LM 0.18.0 can do only from C++ +(`has_pending_message`); through the C API the partial reply was sent as a finished turn and the +model began a new one, which the client then appended to its prefill. + **The library is loaded with `dlopen(RTLD_LOCAL)`, not linked**, and the server's own C++ runtime is static and unexported. `liblitert-lm.so` carries its own statically linked C++ runtime and *exports* it; when the server was linked against the library, the linker @@ -239,6 +472,16 @@ satisfied the server's `std::runtime_error`/`__cxa_*` references from those expo AddressSanitizer caught an exception being destroyed by the other runtime. See `src/litert_lm_api.h`. +**It ends with whatever started it.** The first thing `main()` does is arm +`PR_SET_PDEATHSIG(SIGKILL)`, so the kernel kills the server the moment the thread that started it +ends, and it exits 1 without starting when its parent is already init (the starter died before +the `prctl`). The daemon's [engine tether](#the-engine-tether) does the same before it execs the +server; this covers a server started any other way: by a daemon without the tether, through the +daemon's `/system/bin/sh` fallback, or by hand. `node --test +engines/litert-lm-server/parent-death.test.mjs` builds the server for the machine (Linux, `cc` +and `c++`) against a stand-in `liblitert-lm.so` whose functions do nothing, and checks that a +stand-in daemon's SIGKILL takes the listening server with it and that an orphaned start exits. + ### GPU The app reads the GPU's name from a throwaway GL context (`MainActivity.gpuEnv`) and passes it @@ -251,7 +494,9 @@ system Vulkan/OpenCL/GL libraries. ### Bumping LiteRT-LM Change `LITERT_LM_VERSION` and both wheel hashes in `scripts/stage-litert-lm-engine.mjs` -(PyPI publishes them at `https://pypi.org/pypi/litert-lm-api//json`), refresh +(PyPI publishes them at `https://pypi.org/pypi/litert-lm-api//json`), then both +`patchedSha256` values: the build stops with the new hash once the shim, its checks and the link +probe have passed, and a changed list of shimmed functions is worth a look. Refresh `engines/litert-lm-server/third_party/litert_lm/c/*.h` from the matching `google-ai-edge/LiteRT-LM` tag, and check that every function listed in `src/litert_lm_api.h` still exists — the server refuses to start, naming the missing ones, if @@ -259,11 +504,129 @@ it doesn't. ### Not yet verified -- **No device has run it.** The NDK cross-compile itself was not run while writing it (no NDK - was available); the first `assembleDebug` is that check. The code uses only POSIX/bionic - APIs available at API 24. +- **No phone has answered a chat yet.** On the owner's phone the wheel's library failed to load, + which is why it now has [a shim](#libandroidso-and-its-shim). With the patched one the same + phone (Mali-G715) loaded LFM2.5-230M, and then the daemon's one-token warm-up request failed + because constrained decoding was on for every request (see above), so the model never finished + loading. The patched library was run in a chroot of the x86_64 Android 15 emulator image's + system and vendor partitions, with that image's linker and its generated `/linkerconfig`: it + loads and answers a chat request on CPU, and still does with `libstagefright.so` removed from + the image, where the unpatched one fails to load as it did on the owner's phone. In that chroot + (image unmodified) it also answers LFM2.5-230M, gemma-3-270m-it, gemma-3-1b-it, a FunctionGemma + 270M and Qwen3-0.6B on CPU, streamed or not, with tools and with a `response_format`. The code + uses only POSIX/bionic APIs available at API 24. - GPU speed versus Termux on the same phone, and whether every GPU path (OpenCL, WebGPU/Vulkan) - loads from an app-spawned process the way it does from Termux. + loads from an app-spawned process the way it does from Termux. The linker namespace of a + `/data` executable searches `/system` and `/system_ext` only, so a vendor `libOpenCL.so` may + not be found. A GPU that cannot be initialised fails the load with the runtime's error (there + is no automatic fallback); the app then suggests switching the model's Backend to CPU. + +--- + +## The engine tether + +Every engine the daemon starts (the llama.cpp ones and the LiteRT-LM server) is meant to end the +moment the app process does, however it ends. The app ships a helper for that, +`libtether_exec.so`, and the daemon starts each engine as +`libtether_exec.so `, and each probe the same way (a short run of +a program whose output it reads, such as an engine's version check). Only those: the helper ties +the one program it execs, not what that program starts, so every other process the daemon ties +to its life (an MCP stdio server, a tunnel) keeps the daemon's `/system/bin/sh` lifeline: a +process group of its own, which a stop ends as a whole, and a watcher that kills that group +when the daemon dies. The helper: + +1. arms `PR_SET_PDEATHSIG(SIGKILL)`, so the kernel kills the process the moment the thread that + started it ends; +2. exits 127 without starting the engine if its parent is no longer that pid (the daemon + process ended between the fork and the `prctl`, so no signal would ever come). This sees the + process end, not the thread: had only the forking thread ended, the kernel would have moved + the helper to another thread of the daemon, and the signal would come when that one ends; + the daemon's thread ends only with the daemon; +3. `exec`s the engine in place: the engine keeps the helper's pid, stdio and exit status, and no + extra process is left behind. + +libuv forks on the thread that calls `spawn`, the one `MainActivity` starts `node::Start` on, +which lives exactly as long as the daemon. So whichever way the app process ends (Quit, which +is `Process.killProcess`, a crash, Android killing the app), the kernel ends the engines with +it at once. Behind that, system_server still kills whatever is left in the +app's cgroup once the app process has died (AOSP 7 to 15); before the helper, that was an +engine's only tie to the app. `liblitertlm_server.so` also arms the signal itself, at the top of +its `main()` ([what the server is](#what-the-server-is-and-why-it-is-shaped-this-way)). + +| | | +|---|---| +| Source | `engines/tether-exec/tether_exec.c` | +| Built by | `scripts/stage-tether-exec.mjs` (`stageTetherExec` in `composeApp/build.gradle.kts`, hooked to `merge*JniLibFolders`), with the NDK's clang for API 24, as a PIE with 16 KB-aligned LOAD segments, stripped, checked with the LiteRT-LM staging's `checkElf` | +| Output | `composeApp/build/generated/tether-exec/jniLibs//libtether_exec.so`, about 6 KB per ABI | +| In the APK | Always. `assemble*`/`bundle*` fail if it is missing for a shipped ABI | + +**The daemon's side** is TurboLLM's `util/tether.ts` (`fix(daemon): a process the daemon starts can +be tied to its life`). It looks for the helper in `TURBOLLM_ANDROID_NATIVE_LIB_DIR` and, at each +start, before it spawns anything, checks it by watching it work, while the app lives (a force-stop +cannot test it: the cgroup kill would hide a helper that does nothing). +It runs `/system/bin/sh -c '"$0" "$$" /system/bin/sleep 30 & echo "$!"; read -r _' ` +and must see `/system/bin/sleep 30` running with that `sh` as its parent, then gone within 1 s +of ending the `sh`'s stdin. It logs `tether: libtether_exec.so self-test passed (… ms)` to +`files/daemon.log`, or `… self-test failed: ; tethered spawns use the +/system/bin/sh lifeline instead`, or, when the APK has no helper, `tether: libtether_exec.so not +found in ; tethered spawns use the /system/bin/sh lifeline instead`. Either way it then ties each engine +with a `/system/bin/sh` watcher instead (one more process per engine, counted against +Android 12+'s phantom process limit). A daemon without that commit starts engines directly: +then the cgroup kill alone ties the llama.cpp engines, and the LiteRT-LM server is also tied +by its own signal. + +`node --test scripts/stage-tether-exec.test.mjs` builds the helper for the machine with `cc` +(Linux) and runs it: a stand-in daemon's SIGKILL takes its engine; a parent that is not the +given pid, or a pid that is not a number above 1, exits 127 without running anything; the pid, +stdin, stdout, exit status and signals pass through; it passes the daemon's self-test under each +shell found (`/bin/sh`, bash, mksh), while a helper that execs without the signal and one that +exits 127 fail it. With an NDK (`ANDROID_NDK_HOME`, `ANDROID_NDK_ROOT`, or 27.0.12077973 under +`ANDROID_HOME` or `ANDROID_SDK_ROOT`) it also builds both ABIs and checks the files. + +### Checking it on a device + +With a model loaded: + +```bash +adb shell run-as com.turbollm cat files/daemon.log | grep 'tether:' # "self-test passed" +adb shell ps -A -o PID,PPID,NAME | grep -E 'llama_server|litertlm|tether' +# the engine's PPID is the app's pid, and no libtether_exec.so is listed (it exec'd the engine) +adb shell am force-stop com.turbollm +adb shell ps -A | grep -E 'llama_server|litertlm' # nothing +``` + +Then the same `ps` after each other way the app ends with a model loaded, loading it again +each time: the notification's Quit (the app ends about 3 seconds after the tap), and +`adb shell run-as com.turbollm kill -9 ` (a debug build). No engine is left after +either. After Quit, a reply that finished just before is there when the app is opened again: +the daemon wrote its chat database before the kill. + +Back from the app's main screen, or a swipe from Recents, with a model loaded does not end the +app, by design: a loaded model counts as running, so the app and its engine stay (`ps` still +lists the engine under the app, and the notification reads "Idle · " and the model's name) +until Quit, or until the daemon's idle timer has unloaded the model and the app is closed again +with nothing running (opened and closed, or the card a Back left in Recents swiped away). + +With nothing loaded or running, Back or a swipe from Recents ends the app about 3 seconds +later: `adb shell pidof com.turbollm` prints nothing, and the `ps` above lists no engine. A +reply that finished just before is there when the app is opened again. + +On the first launch after an install or an update, close the app only once it shows its own +splash: the animated bolt (an orange badge that pops in, then pulses) with "Starting TurboLLM…" +under it, not the still launcher icon Android shows before it from Android 12. Until then the app +is still unpacking the daemon and has not started the service that hears a close, so whether a +swipe made then ends the app is up to Android. + +Only the `tether:` line tests the helper itself. Every ending above would also pass on the +cgroup kill alone. + +### Not yet verified + +No device has run it. The x86_64 NDK build was run in a chroot of the x86_64 Android 15 +emulator image (its own linker, `/system/bin/sh`, which is mksh, and toybox's `sleep`): it +passed the self-test above, and a stand-in daemon's SIGKILL took its engine within 20 ms. The +arm64 build has only been checked as a file (architecture, PIE, `/system/bin/linker64`, +alignment). --- @@ -421,8 +784,19 @@ wsl -d Ubuntu-24.04 -u root -- bash "/mnt/d/llama-turbo/TurboLLM Android/scripts DRY_RUN=1 bash scripts/build-static-engine.sh # resolved paths + the commands, builds nothing ``` -It is the WSL original plus the four things that original did not do, each of which exists -because of something that went wrong: +A Windows checkout that had the scripts before `.gitattributes` pinned them and `gradlew` to LF +still has the CRLF copies Git for Windows wrote by default (`core.autocrlf=true`), and bash stops +at the first `\r`: "set: pipefail: invalid option name". Pulling does not rewrite them, since +their contents did not change, and `git status` stays clean. Write them again once, with Git for +Windows in that checkout (this discards uncommitted edits to those four files, nothing else): + +```bash +git rm -q --cached -- "scripts/*.sh" gradlew +git checkout HEAD -- "scripts/*.sh" gradlew +``` + +The script is the WSL original plus the four things that original did not do, each of which +exists because of something that went wrong: - **Builds ONE static binary** — `-DBUILD_SHARED_LIBS=OFF` with `-DGGML_BACKEND_DL=OFF`, since the runtime backend loader is a shared-library mechanism and the two are incompatible. Same diff --git a/PLAY-FGS-DECLARATION.md b/PLAY-FGS-DECLARATION.md index d39a241..c57bdb7 100644 --- a/PLAY-FGS-DECLARATION.md +++ b/PLAY-FGS-DECLARATION.md @@ -19,14 +19,36 @@ No standard type describes this work: - `shortService` is limited to about 3 minutes. - A WorkManager long-running worker still runs as a typed foreground service. +The same service also keeps a model import or an in-app model download going until it is done. +Taken alone, those transfers are `dataSync` work: Android's list for that type includes upload or +download and import or export. They are not why the service runs, though. It starts when the app +is opened, not for them, so it is already up for the inference server by the time the user starts +an import or a download from the app's own screens, and they only bring in the models that server +loads. Declaring the service `dataSync` instead would limit the inference server to 6 hours in 24 +while the app is in the background (this app targets Android 15+). The declaration below says so +to Play. If Play asks for the transfers to be declared on their own: it allows more than one type +per service, but adding `dataSync` is a code change (`EngineService`'s `startForeground` type and +the manifest) that this page does not make. + The manifest's `PROPERTY_SPECIAL_USE_FGS_SUBTYPE` already states the use. The text below expands it for the form. ## Declaration text The answers describe what the code does (`EngineService.kt`). The service starts when the app is opened and stays -in the foreground, also while idle, until the user quits, so the text says so rather than suggesting it runs only -during generation. +in the foreground, also while idle, until the user quits or closes the app with nothing running, so the text says so +rather than suggesting it runs only during generation. + +LAN access, remote access and Turbo Link are read from the daemon the app bundles: the `serving` field of its GET +/api/v1/status, from TurboLLM's feat(api): /api/v1/status says which server features are on. A daemon without that +field reports them all off (the app reads a missing or malformed `serving` as nothing served), so a close with only +one of them on would quit: the text below holds only for an APK built with a TurboLLM that has it. + +A close made while the app is still starting, before its daemon answers (a swipe from Recents during the splash +screen, and before Android 12 Back there too), waits for the daemon, for up to 30 seconds, and is then decided as the +text says. So the text's daemon that gives "no answer" is one that has not answered by then, or one that stops +answering once it is up. Such a close with nothing running ends the app about 3 seconds after the close, or, when the +daemon takes longer to answer, right after it does. **Which feature uses the foreground service?** @@ -38,34 +60,64 @@ during generation. > The service keeps that local server and the loaded model available while the user is in another app. Without it, > Android freezes the app as soon as it leaves the screen: a reply being written stops mid-sentence and is lost, and > the model is reclaimed, so the next message waits for it to load again. Replies on a phone take from seconds to -> minutes, and users switch apps while they wait. The service also keeps a model import (a multi-GB file copy) from -> being frozen halfway. +> minutes, and users switch apps while they wait. The service also keeps a model import (a multi-GB file copy) or an +> in-app model download from being frozen halfway. **What does the user see, and how do they stop it?** > From the moment the app is opened, a low-importance ongoing notification titled "TurboLLM", with a **Quit** action -> that stops the service, unloads the model and closes the app. Its text says what is happening: "Running in the -> background" while idle, "Generating" while a reply is being written, "Importing models" during an import, and, -> when the phone is thermally throttled, "Running. The phone is hot, so generation will be slower" or "Generating. -> The phone is hot, so it is throttling speed". If the user has turned notifications off, so Quit cannot be reached, -> leaving the app (Back, or swiping it away from Recents) unloads the model and stops the service as soon as nothing -> is generating or importing. A partial wake lock is held only while a reply is being generated or a model imported, -> never while idle. +> that stops the service, unloads the model and closes the app, about 3 seconds after the tap. On Android 14 and +> later, where the user can swipe this notification away, it is posted again at once, so Quit stays reachable. Its +> text says what is happening: "Idle" while no work is in progress, or "Idle · " and the model's name while a model is +> loaded (the chat model's name, or a text classification model's while no chat model is loaded; an embedding model +> loaded alone is not named), "Generating" while a reply is being written, "Loading a model", "Auto-tuning" or +> "Writing a skill" while the app does one of those, "Waiting for your approval" while a reply waits for the user to +> allow a tool call, "Importing models" during an import, the progress while a model download is running or queued +> (the file's name as the title, or "Downloading models" for several, how much has arrived and how fast, and a +> progress bar), and, when the phone is thermally throttled, "Idle. The phone is hot, so generation will be slower" or +> what is running followed by "The phone is hot, so it is throttling speed" (for example "Generating. The phone is +> hot, so it is throttling speed"). Leaving the app never unloads the model by itself: the user does, in the app or +> with Quit, or the app's idle timer does once the model has gone unused for a while (60 minutes by default; the user +> can change it or turn it off in Settings). Closing the app (Back from its main screen, or swiping it away from +> Recents; pressing Home does not close it) with nothing running works as Quit does, whether notifications are on or +> off: about 3 seconds later the service stops and the app's process ends. Opening the app again within those 3 +> seconds, after Quit or such a close, keeps it running, as does the app coming back on screen after Quit (for example +> the phone unlocked onto it). Nothing is running when no model is loaded or loading (any model, an embedding model +> included), nothing is being generated, imported or downloaded (a paused or failed download does not count), no reply +> waits for the user's approval, no routine is running, and no access from other devices is on: LAN access, remote +> access (a tunnel, Tailscale Serve or Funnel), or Turbo Link letting other devices use this phone's models; when the +> app gets no answer from the daemon, or one it cannot read, it counts as running. Closed with something running, the +> app keeps the service, also after that work has ended, after the idle timer has unloaded the model and after that +> access has been turned off, until the user taps Quit or closes the app again with nothing running (after opening it, +> or by swiping away the card it left in Recents). A routine only scheduled for later does not count as running: such +> a close stops the routine scheduler with the app, until the app is opened again. If the user has turned +> notifications off (or the notification's channel), there is no Quit to tap: on Android 13 and later the system's +> list of active apps can still stop the app, and closing it again with nothing running quits. A partial wake lock is +> held only while a reply is being generated, a model being loaded, auto-tuned, imported or downloaded, or a skill +> written, never while idle or while a reply waits for the user's approval. **Why does no other foreground service type fit?** -> The work is a local, user-started AI inference server. It is not data sync, media playback or processing, -> location, a phone call, health or connected-device work, and it lasts as long as the user keeps the app in use, -> longer than shortService allows. The other candidate types are capped at 6 hours on Android 15 and describe -> different work. +> The work is a local, user-started AI inference server, kept running for as long as the user keeps the app in use. +> It is not media playback or processing, location, a phone call, health or connected-device work, and it lasts +> longer than shortService allows. The model imports and downloads the service also keeps going are data transfers, +> but they are not why it runs: it starts when the app is opened, not for them, they run inside it while it is +> already up for the inference server, and they only bring in the models that server loads. dataSync fits those +> transfers alone, and like mediaProcessing it is capped at 6 hours in 24 on Android 15, while the inference server +> runs for as long as the app is in use. ## Video script (about 45 seconds, screen recording on a phone) Record with the system screen recorder, with touches shown (Developer options → Show taps). Use a small model, so the first load is quick and the reply visibly continues. -1. **0:00** Open TurboLLM. On the Models screen, load a small model, for example a - `.litertlm` such as LFM2.5-230M. +1. **0:00** Open TurboLLM. On the Models screen, load a small GGUF model, for example + Qwen2.5-0.5B-Instruct (Q4_K_M). It runs on llama.cpp, the engine the app selects until you + pick another. A `.litertlm` model such as LFM2.5-230M runs only on LiteRT-LM, which the app + never selects by itself: under llama.cpp the Models screen hides it and a load started + anyway fails. To record with one, select LiteRT-LM in Engines before you start recording + (tap Use on its card under "Built into the app", or pick "LiteRT-LM" in the engine selector + at the top of that screen). 2. **0:08** In Workspace, ask for something long: *"Write a 500-word story about a lighthouse keeper."* Let a line or two of the reply appear. 3. **0:15** Pull down the notification shade. Show the TurboLLM notification reading @@ -73,9 +125,10 @@ a small model, so the first load is quick and the reply visibly continues. 4. **0:20** Press Home and open another app (for example Settings) for about 10 seconds. 5. **0:30** Return to TurboLLM. The reply has kept growing while the app was in the background, and is complete or nearly so. -6. **0:38** Pull down the shade again. The notification now reads "Running in the background": - the model stays loaded, ready for the next message, which is what the service is for. - Tap **Quit**. The notification disappears and the app closes, freeing the model. +6. **0:38** Pull down the shade again. The notification now reads "Idle · " followed by the + model's name: the reply is done, and the model stays loaded, ready for the next message, + which is what the service is for. Tap **Quit**. About 3 seconds later the notification + disappears and the app closes, freeing the model. Upload the video unlisted (for example on YouTube) and paste the link into the declaration. diff --git a/PROVENANCE.md b/PROVENANCE.md index b2b4d7d..ea3186f 100644 --- a/PROVENANCE.md +++ b/PROVENANCE.md @@ -44,6 +44,18 @@ Not vendored: `scripts/stage-litert-lm-engine.mjs` fetches it at build time from The extracted arm64 `liblitert-lm.so` (39.9 MB, sha256 `3d591b09…dfd9`) is byte-identical to the one the Termux path installs. License: Apache-2.0 (google-ai-edge/LiteRT-LM). +The APK carries it with one change. The engine process cannot load `libandroid.so`, so the script +renames that `DT_NEEDED` entry in place to `liblrtandr.so` (7 bytes of `.dynstr`; nothing else +differs, the `readelf -d -V --dyn-syms` output only in that line) and ships `liblrtandr.so`, a shim +it generates from the library's imports and builds with the NDK: each of its functions aborts with +a message if called (BUILDING-ENGINES.md, "libandroid.so and its shim"). The patched file is +pinned as well, and the build refuses any other result: + +| ABI | SHA-256 (`liblitert-lm.so` from the wheel) | SHA-256 (as shipped) | +|---|---|---| +| arm64-v8a | `3d591b09116e7be3ab6b2b5b1a27bb3079dda0eb66550271533e6aee19e2dfd9` | `ae7e50538d9298e113686bc2ac07ebf33aac1325e6e3cbff299a1024a1b16d6a` | +| x86_64 | `cc8fdef43d1900234d3bebecd29b3610a7b09c62e0a3b3cdb08499e4b8e8e4c7` | `681f770323d7021ca47a0481ae49744699d80a62c79f759cd1ce443e21becb22` | + Vendored in `engines/litert-lm-server/third_party/`: LiteRT-LM's C API headers from tag `v0.18.0` (`c/engine.h`, `c/conversation.h`, `c/error_reporter.h`, `c/api_export.h`; Apache-2.0, LICENSE alongside) and nlohmann/json 3.11.3 (`json.hpp`, sha256 `9bea4c80…3ea6`; MIT, diff --git a/RELEASING.md b/RELEASING.md index 1f511dd..44c2069 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -45,8 +45,9 @@ The script refuses to overwrite an existing keystore for the same reason. ``` That single command also stages the Node daemon into the APK (`stageDaemonAssets`), which -rebuilds the daemon bundle from the `turbollm` checkout first. If turbollm is not a sibling -directory, pass `-Pturbollm.dir=`. +rebuilds the daemon bundle from the `turbollm` checkout first, and its web UI too whenever +`turbollm/web` has changed since the web UI was last built there (BUILDING-ENGINES.md, "The +daemon payload"). If turbollm is not a sibling directory, pass `-Pturbollm.dir=`. Use `:composeApp:assembleRelease` instead if you want an installable `.apk` (for sideloading a beta build); Play takes the `.aab`. @@ -108,6 +109,11 @@ not change `minSdk`, which stays at 24 — only the Vulkan binary itself needs A wsl -d Ubuntu-24.04 -u root -- bash "/mnt/d/llama-turbo/TurboLLM Android/scripts/build-static-engine.sh" ``` +On a Windows checkout that had the scripts before `.gitattributes` pinned them to LF, this stops +at once with "set: pipefail: invalid option name" until they are written again; the one-time +commands for that are in +[BUILDING-ENGINES.md](BUILDING-ENGINES.md#the-build-script--scriptsbuild-static-enginesh). + That script verifies the result (AArch64, system-only `NEEDED`, embedded commit == `$SRC` HEAD, size, sha256) and installs it into `composeApp/src/androidMain/jniLibs/arm64-v8a/` itself. See [BUILDING-ENGINES.md](BUILDING-ENGINES.md#build-b--the-vulkan-gpu-engine). @@ -180,12 +186,12 @@ still on Gradle 8.x. Revisit the version once the toolchain itself moves to Grad `.github/workflows/pr-alpha.yml` builds a signed release bundle and runs `publishReleaseBundle` against the **alpha** track on every PR open / push / reopen (markdown-only changes and fork PRs are skipped; a newer push cancels an in-flight build for the same PR). It fetches the TurboLLM -daemon from `mohitsoni48/TurboLLM@main`, builds its web UI, fetches `libnode.so` with -`scripts/fetch-libnode.mjs` (the digidem 16 KB-aligned nodejs-mobile build, verified against the -SHA-256s in the script and PROVENANCE.md), stages the LiteRT-LM engine from PyPI, and uses the -tracked CPU llama.cpp engine in `jniLibs/`. The Vulkan engine is not built in CI. Because the daemon -comes from TurboLLM `main`, a PR here that needs daemon changes builds a working alpha only once -those daemon changes are merged. +daemon from `mohitsoni48/TurboLLM@main` and installs its packages (staging builds its web UI), +fetches `libnode.so` with `scripts/fetch-libnode.mjs` (the digidem 16 KB-aligned nodejs-mobile build, +verified against the SHA-256s in the script and PROVENANCE.md), stages the LiteRT-LM engine from +PyPI, and uses the tracked CPU llama.cpp engine in `jniLibs/`. The Vulkan engine is not built in CI. +Because the daemon comes from TurboLLM `main`, a PR here that needs daemon changes builds a working +alpha only once those daemon changes are merged. Secrets live in the **`play-alpha` environment** (Settings → Environments), not as repo secrets. That environment requires the owner's approval for every run, `main` requires an owner-approved PR (`.github/CODEOWNERS`), and fork PRs never get secrets. Already created: diff --git a/composeApp/build.gradle.kts b/composeApp/build.gradle.kts index 89b385c..45e78f3 100644 --- a/composeApp/build.gradle.kts +++ b/composeApp/build.gradle.kts @@ -96,9 +96,14 @@ val shipLitertLmEngine: Boolean = // never disagree about which engine builds exist. val shippedAbis = listOf("arm64-v8a", "x86_64") -// Where stageLitertLmEngine puts /{liblitert-lm.so, liblitertlm_server.so}. +// Where stageLitertLmEngine puts /{liblitert-lm.so, liblitertlm_server.so} and the shim that +// stands in for libandroid.so (liblrtandr.so). jniLibs packages everything in it. val litertLmJniLibs = layout.buildDirectory.dir("generated/litert-lm/jniLibs") +// Where stageTetherExec puts /libtether_exec.so, the helper the daemon starts every engine through (see "The +// engine tether" below). jniLibs packages everything in it. +val tetherJniLibs = layout.buildDirectory.dir("generated/tether-exec/jniLibs") + val uploadStoreFile: File? = localProp("TURBOLLM_UPLOAD_STORE_FILE") ?.let { rootProject.projectDir.resolve(it).normalize() } ?.takeIf { it.exists() } @@ -231,6 +236,9 @@ android { // The LiteRT-LM engine, staged into the build dir by stageLitertLmEngine (below) rather // than src/: it is fetched and compiled on every machine, never committed. if (shipLitertLmEngine) jniLibs.srcDir(litertLmJniLibs) + // The engine tether, staged into the build dir by stageTetherExec (below): compiled on every machine, like the + // LiteRT-LM server, and in every build. + jniLibs.srcDir(tetherJniLibs) } // Upload signing, read from local.properties (gitignored) — never from the repo. Created by @@ -387,7 +395,9 @@ tasks.matching { it.name == "generateReleasePlayResources" }.configureEach { // through whenever Gradle skipped straight to the merge. // // The staged directory is gitignored build output; the script rebuilds the daemon bundle each -// time (~200 ms) so the APK can never ship a daemon older than the source. +// time (~200 ms) so the APK can never ship a daemon older than the source, and the web UI +// whenever turbollm/web is not what it was last built from, so it never ships an older web UI +// either (a web build takes 40-55 s; checking costs a hash of web/, under 50 ms). val stagedDaemonAssets = layout.projectDirectory.dir("src/androidMain/assets/daemon") val stageDaemonAssets by tasks.registering(Exec::class) { @@ -472,11 +482,15 @@ if (shipVulkanEngine) { // ── The LiteRT-LM engine ────────────────────────────────────────────────────────────────── // -// Two files per ABI: liblitert-lm.so — Google's runtime, the same file the litert-lm-api wheel -// installs under Termux, fetched from PyPI against a pinned sha256 — and liblitertlm_server.so, +// Three files per ABI: liblitert-lm.so — Google's runtime, the file the litert-lm-api wheel +// installs under Termux, fetched from PyPI against a pinned sha256, with its libandroid.so +// dependency renamed because the engine process cannot load that library —, liblitertlm_server.so, // the native OpenAI-compatible server in engines/litert-lm-server that replaces the Python -// `litert-lm serve` the app cannot run. scripts/stage-litert-lm-engine.mjs does both, then checks -// each file's ELF architecture and 16 KB alignment. See BUILDING-ENGINES.md, "The LiteRT-LM engine". +// `litert-lm serve` the app cannot run, and liblrtandr.so, the shim generated from the library's +// imports to stand in for libandroid.so. scripts/stage-litert-lm-engine.mjs does all three, checks +// each file's ELF architecture and 16 KB alignment and the patched library's pinned sha256, and +// links a probe against the result so an import nothing provides fails the build. See +// BUILDING-ENGINES.md, "The LiteRT-LM engine". // // The NDK is the one AGP itself resolves for native-lib, and CMake is the SDK copy that // externalNativeBuild above pins, so this needs nothing a plain `assembleDebug` didn't already. @@ -485,7 +499,7 @@ if (shipVulkanEngine) { // incrementally, so a repeat run costs about a second. val stageLitertLmEngine by tasks.registering(Exec::class) { group = "build" - description = "Fetches liblitert-lm.so and builds liblitertlm_server.so into the APK's jniLibs." + description = "Fetches liblitert-lm.so and builds liblitertlm_server.so and its libandroid.so shim into the APK's jniLibs." workingDir = rootProject.projectDir // BaseExtension resolves both from local.properties / ANDROID_HOME and the pinned ndkVersion. (The // androidComponents.sdkComponents providers carry no value at execution time in AGP 8.7.) @@ -514,6 +528,8 @@ if (shipLitertLmEngine) { // Same reasoning as the daemon payload check: an APK that silently lacks the engine it was // built for is worse than a failed build. Only the APK/AAB tasks themselves: AGP also has // bundle*ClassesToJar and assemble*UnitTest, which run without merging native libs at all. + // The shim is not listed: its name comes from the library, and the staging run that writes + // liblitert-lm.so fails unless the shim it links against is staged beside it. val packagingTask = Regex("^(assemble|bundle)(Debug|Release)?$") tasks.matching { packagingTask.matches(it.name) }.configureEach { doLast { @@ -532,6 +548,54 @@ if (shipLitertLmEngine) { } } +// ── The engine tether ───────────────────────────────────────────────────────────────────── +// +// libtether_exec.so, built from engines/tether-exec/tether_exec.c by scripts/stage-tether-exec.mjs with the NDK AGP +// resolves, as for the LiteRT-LM server. The daemon starts every engine through it: it arms PR_SET_PDEATHSIG(SIGKILL) +// and execs the engine in place, so the kernel kills the engine the moment the daemon's thread ends, however the app +// process ends. Before, an engine's only tie to the app was system_server killing the app's cgroup after the process +// had died. The daemon checks the helper at boot by watching it work, and without it (missing, or failing that check) +// it ties each engine with a /system/bin/sh watcher instead: one more process per engine, counted against Android +// 12+'s phantom process limit. So it is in every build, whatever the engine flags say, and its absence fails the build +// like a missing LiteRT-LM engine does. See BUILDING-ENGINES.md, "The engine tether". Hooked to merge*JniLibFolders, +// as stageLitertLmEngine is; two clang runs of one small file, so it runs every time. +val stageTetherExec by tasks.registering(Exec::class) { + group = "build" + description = "Builds libtether_exec.so, the helper that ties each engine to the app process, into the APK's jniLibs." + workingDir = rootProject.projectDir + val androidExt = project.extensions.getByType(com.android.build.gradle.BaseExtension::class.java) + val outDir = tetherJniLibs + outputs.upToDateWhen { false } + doFirst { + commandLine( + "node", "scripts/stage-tether-exec.mjs", + "--ndk", androidExt.ndkDirectory.absolutePath, + "--out", outDir.get().asFile.absolutePath, + "--abis", shippedAbis.joinToString(","), + ) + } +} + +tasks.matching { it.name.startsWith("merge") && it.name.endsWith("JniLibFolders") }.configureEach { + dependsOn(stageTetherExec) +} +// The APK/AAB tasks only, as for the LiteRT-LM engine above. +val tetherPackagingTask = Regex("^(assemble|bundle)(Debug|Release)?$") +tasks.matching { tetherPackagingTask.matches(it.name) }.configureEach { + doLast { + for (abi in shippedAbis) { + val f = tetherJniLibs.get().file("$abi/libtether_exec.so").asFile + if (!f.exists()) { + throw GradleException( + "Engine tether missing: ${f.absolutePath}\n" + + "The daemon would tie each engine with an extra sh process instead. " + + "It is built by stageTetherExec (scripts/stage-tether-exec.mjs).", + ) + } + } + } +} + // ── The embedded Node runtime ───────────────────────────────────────────────────────────── // // cpp/libnode is gitignored build input that native-lib's CMake links and jniLibs packages, so it diff --git a/composeApp/src/androidMain/AndroidManifest.xml b/composeApp/src/androidMain/AndroidManifest.xml index c30a407..68919e2 100644 --- a/composeApp/src/androidMain/AndroidManifest.xml +++ b/composeApp/src/androidMain/AndroidManifest.xml @@ -2,7 +2,6 @@ - @@ -10,8 +9,9 @@ + app holds none, so in shared storage it reads only the files it created itself (downloads into a primary + model folder under Download/ or Documents/); models the user or another app put there are imported with the + system file picker instead, which needs no permission (ModelImport.kt, StorageAccess.kt). --> @@ -27,10 +27,12 @@ android:theme="@style/Theme.TurboLLM" android:usesCleartextTraffic="true"> + android:value="On-device LLM inference server: keeps a loaded model and in-progress generations alive while the user is in another app, along with the model imports and downloads the user started for it" /> diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonAuth.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonAuth.kt new file mode 100644 index 0000000..d5838d5 --- /dev/null +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonAuth.kt @@ -0,0 +1,133 @@ +package com.turbollm.app + +import java.io.IOException +import java.io.InputStream +import java.net.HttpURLConnection +import java.net.InetSocketAddress +import java.net.Proxy +import java.net.Socket +import java.net.URL +import java.security.MessageDigest +import java.security.SecureRandom +import java.util.concurrent.ScheduledThreadPoolExecutor +import java.util.concurrent.TimeUnit +import javax.crypto.Mac +import javax.crypto.spec.SecretKeySpec + +/** The only way native code reaches the daemon with DAEMON_APP_SECRET, so the secret goes nowhere but to the daemon + * this process started. Native code must not open any other connection to the daemon's port (see the last paragraph). + * + * Any app on the phone can listen on 127.0.0.1:ANDROID_DAEMON_PORT while our daemon does not: before it binds at + * launch (EngineService polls from the start, and the daemon retries a busy port for about 10 s), or for a moment + * while a LAN toggle rebinds it. A secret sent there would let that app use the whole API for as long as this process + * lives. So before each request the port's holder must answer a fresh random nonce with HMAC-SHA256 under the secret + * (GET /api/v1/app-proof, daemon auth.ts appSecretProof), which only our daemon can do and which is no use as the + * secret itself. A proof per request rather than one kept until a request fails: an app that took the port over + * between two requests would answer the second one, not fail it. + * + * A proof vouches only for the listener, so the proof and the request each go out on a new socket, never on a pooled + * one: the proof on a plain socket of its own (see PROOF_DEADLINE_MS), the request with Connection: close. Android's + * HttpURLConnection keeps keep-alive sockets in one pool for the process and reuses them even after disconnect(), and + * a pooled socket stays connected to whoever accepted it: an app holding the port at launch could let our daemon have + * it, keep the socket we pooled, pass each nonce on to the daemon, and receive the request that follows on that + * socket. With every socket to the port closed after its one request, nothing to it is ever in the pool. What this + * guarantees: the secret goes out only on a connection opened just after our daemon, on another new connection, proved + * it held the port. It reaches another app only if the daemon gave the port up in between, i.e. a LAN-toggle rebind; + * an app that is still on the port when the daemon binds again makes the daemon exit (cli.ts), so what is left is an + * app that binds and lets go inside that gap of a few ms, just as a request is made. + * + * Both connections also go straight to the port, never through the system HTTP proxy (Proxy.NO_PROXY). Android + * exempts 127.0.0.1 from that proxy only when the proxy's bypass list is not empty, and it often is empty: a Wi-Fi + * proxy's "Bypass proxy for" starts empty, and so does the list of the proxy a VPN app sets with + * VpnService.Builder.setHttpProxy. A proxy app on the phone would then carry both connections: it could pass the + * nonce on to our daemon, which proves itself, and read the request that follows, secret and all. */ +object DaemonAuth { + private const val HOST = "127.0.0.1" + private const val BASE = "http://$HOST:$ANDROID_DAEMON_PORT" + private val random = SecureRandom() + + // Until its answer checks out, the port's holder may be another app, and nothing it sends may stall or crash us. + // Socket timeouts are per read, so an answer trickled a byte at a time would keep the splash and the EngineService + // poll waiting as long as the sender liked: hence one deadline for the whole request. An endless or deeply nested + // answer would run a plain read or a recursive JSON parser out of memory or stack, an Error: hence a cap on what + // is read, a regex rather than a parse, and catching Throwable. The cap covers the whole answer, status line and + // headers included, so the request is written by hand on a plain socket: HttpURLConnection reads the status line + // and headers whole before it hands over the body, with no limit, and header lines sent as fast as loopback goes + // filled a 96 MiB heap in under 2 s, well inside the deadline. The daemon's answer is about 230 bytes, its body + // {"proof":"<64 hex>"} 76. + private const val PROOF_DEADLINE_MS = 3000L + private const val PROOF_MAX_BYTES = 4096 + private val PROOF = Regex("\"proof\"\\s*:\\s*\"([0-9a-f]{64})\"") + private val STATUS_OK = Regex("^HTTP/1\\.[01] 200 ") + + /** Ends a proof request still running at PROOF_DEADLINE_MS: close() from another thread closes its socket, which + * ends the connect or read it is blocked in. */ + private val deadlines = ScheduledThreadPoolExecutor(1) { r -> + Thread(r, "DaemonAuth deadline").apply { isDaemon = true } + }.apply { removeOnCancelPolicy = true } + + /** True only when our daemon answered a fresh nonce while it held the port (another app may have passed the nonce + * on, see the class comment), so a new connection made right after reaches it. Blocking network I/O: never on + * the main thread. False on any failure, a daemon still starting included, and within about PROOF_DEADLINE_MS + * whatever the port's holder sends. */ + fun daemonProven(): Boolean = try { + val nonce = hex(ByteArray(32).also { random.nextBytes(it) }) + // Proxy.NO_PROXY: to the port itself, not to a proxy that could relay the nonce (class comment). A socket of + // its own, never pooled, and closed after this one answer. + Socket(Proxy.NO_PROXY).use { socket -> + val deadline = deadlines.schedule(Runnable { socket.close() }, PROOF_DEADLINE_MS, TimeUnit.MILLISECONDS) + try { + socket.connect(InetSocketAddress(HOST, ANDROID_DAEMON_PORT), 1000) + socket.soTimeout = 2000 + // Connection: close, so the daemon closes the socket after its answer and that ends the read below. + val request = "GET /api/v1/app-proof?nonce=$nonce HTTP/1.1\r\n" + + "Host: $HOST:$ANDROID_DAEMON_PORT\r\nConnection: close\r\n\r\n" + socket.getOutputStream().apply { write(request.toByteArray(Charsets.US_ASCII)); flush() } + // Only the first PROOF_MAX_BYTES of the answer, headers included, and matched as text rather than + // parsed: nothing the port's holder sends can grow the heap or the stack (see PROOF_DEADLINE_MS). + val answer = readAtMost(socket.getInputStream(), PROOF_MAX_BYTES) + val bodyAt = answer.indexOf("\r\n\r\n") + // A 200 only: a redirect is not followed, the answer must come from this port. The proof is taken from + // the body, after the blank line that ends the headers. + STATUS_OK.containsMatchIn(answer) && bodyAt >= 0 && run { + val proof = PROOF.find(answer, bodyAt + 4)?.groupValues?.get(1).orEmpty() + // The daemon's message (appSecretProof): the address and port it was asked on, then the nonce. + val mac = Mac.getInstance("HmacSHA256") + mac.init(SecretKeySpec(DAEMON_APP_SECRET.toByteArray(), "HmacSHA256")) + val want = hex(mac.doFinal("turbollm-app-proof\n$HOST:$ANDROID_DAEMON_PORT\n$nonce".toByteArray())) + MessageDigest.isEqual(want.toByteArray(), proof.toByteArray()) // constant time + } + } finally { + deadline.cancel(false) + } + } + } catch (_: Throwable) { + false // an Error too: no answer from the port may crash the app (comment at PROOF_DEADLINE_MS) + } + + /** A connection to [path] on the daemon carrying the secret, opened only once [daemonProven]. Throws otherwise, + * which every caller already handles as an unreachable daemon. Callers must not override its Connection header. */ + fun open(path: String): HttpURLConnection { + if (!daemonProven()) throw IOException("no proof from port $ANDROID_DAEMON_PORT that it is our daemon; secret withheld") + // Proxy.NO_PROXY: the secret goes to the port itself, never through a proxy (class comment). + return (URL("$BASE$path").openConnection(Proxy.NO_PROXY) as HttpURLConnection).apply { + // Its own new socket, and none left in the pool for the next proof or request to pick up (class comment). + setRequestProperty("Connection", "close") + setRequestProperty("X-TurboLLM-App", DAEMON_APP_SECRET) + } + } + + /** Up to [max] bytes of [input] as text; whatever follows is never read. */ + private fun readAtMost(input: InputStream, max: Int): String { + val buf = ByteArray(max) + var n = 0 + while (n < max) { + val got = input.read(buf, n, max - n) + if (got < 0) break + n += got + } + return String(buf, 0, n, Charsets.UTF_8) + } + + private fun hex(bytes: ByteArray) = bytes.joinToString("") { (it.toInt() and 0xff).toString(16).padStart(2, '0') } +} diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonWebView.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonWebView.kt index 8d5d865..b7497a8 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonWebView.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonWebView.kt @@ -4,16 +4,24 @@ import android.annotation.SuppressLint import android.app.Activity import android.content.ActivityNotFoundException import android.content.Intent +import android.content.pm.ApplicationInfo import android.net.Uri import android.os.Build +import android.os.SystemClock +import android.util.Log +import android.view.ViewGroup +import android.webkit.CookieManager +import android.webkit.RenderProcessGoneDetail import android.webkit.ValueCallback import android.webkit.WebChromeClient import android.webkit.WebResourceRequest +import android.webkit.WebResourceResponse import android.webkit.WebView import android.webkit.WebViewClient import androidx.activity.compose.BackHandler import androidx.activity.compose.rememberLauncherForActivityResult import androidx.activity.result.contract.ActivityResultContracts +import androidx.annotation.RequiresApi import androidx.compose.foundation.layout.Arrangement import androidx.compose.foundation.layout.WindowInsets import androidx.compose.foundation.layout.Column @@ -22,12 +30,15 @@ import androidx.compose.foundation.layout.padding import androidx.compose.foundation.layout.safeDrawing import androidx.compose.foundation.layout.size import androidx.compose.foundation.layout.windowInsetsPadding +import androidx.compose.material3.Button import androidx.compose.material3.MaterialTheme import androidx.compose.material3.Surface import androidx.compose.material3.Text import androidx.compose.runtime.Composable import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.getValue +import androidx.compose.runtime.key +import androidx.compose.runtime.mutableIntStateOf import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember import androidx.compose.runtime.setValue @@ -42,42 +53,71 @@ import com.airbnb.lottie.compose.LottieCompositionSpec import com.airbnb.lottie.compose.LottieConstants import com.airbnb.lottie.compose.animateLottieCompositionAsState import com.airbnb.lottie.compose.rememberLottieComposition +import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.delay +import kotlinx.coroutines.withContext +import java.io.ByteArrayInputStream + +private const val TAG = "TurboLLM" private val DAEMON_URL = "http://127.0.0.1:$ANDROID_DAEMON_PORT/" /** The daemon UI's origin, with the trailing slash so a look-alike host cannot match. */ private val DAEMON_ORIGIN = DAEMON_URL -/** Confirmed live (2026-09-03) via Chrome DevTools attached over the WebView's own devtools - * socket: this WebView (Android System WebView, Chromium 145 on this emulator) resolves ANY - * raw `vh`-family CSS length (`vh`/`dvh`/`svh`/`lvh`) to a literal 0px — not just on - * html/body/#root (the first occurrence found, fixed below by setting an explicit pixel - * height from `window.innerHeight`), but on ANY element anywhere in the app that uses one, - * including ones that only exist inside dynamically-rendered content (confirmed on a second, - * unrelated occurrence: the Models → Discover results list, `max-h-[45vh]`, which only - * renders after an async fetch completes — so a page-load-time-only fix can't reach it. DOM - * and CSS both load correctly (zero console errors, real data in `innerText`) but the - * affected subtree paints zero pixels regardless. - * - * Fixed generally: htmlFix pins html/body/#root's own height in pixels (unchanged from the - * original fix). vhClassFix separately scans element classNames for Tailwind's arbitrary-value - * bracket syntax (`h-[45vh]`, `max-h-[100dvh]`, `min-h-[50svh]`, …) — which is a literal - * substring of the class attribute, not something requiring stylesheet introspection — and - * overrides the corresponding CSS property with an `!important` inline pixel value computed - * from `window.innerHeight`. A MutationObserver re-runs it on every newly-added subtree so - * content rendered after page load (any async-fetched list, not just Discover) is covered - * too, not just what exists at `onPageFinished` time. +/** Schemes the WebView loads itself: web pages, and what the UI's own frames use. Navigations to any other scheme are + * never loaded (shouldOverrideUrlLoading); turbollm-app is handled before that check. */ +private val WEBVIEW_SCHEMES = setOf("http", "https", "about", "data", "blob", "javascript") + +/** Contact links handed to the app that handles them (tel only opens the dialer). The web UI's markdown and model-card + * sanitizers allow mailto, xmpp, irc and ircs links, and GFM turns a bare e-mail address in a reply into a mailto + * link. */ +private val HANDED_OFF_SCHEMES = setOf("mailto", "xmpp", "irc", "ircs", "tel") + +/** A link handed to another app goes out as a browser sends one: ACTION_VIEW with CATEGORY_BROWSABLE, so only an + * activity that declares it, i.e. accepts links from web pages, can be started. A bare ACTION_VIEW also matched + * exported activities that never declared it: ones their apps open to other apps' code, not to any link on a page. */ +private fun linkIntent(url: Uri) = Intent(Intent.ACTION_VIEW, url).addCategory(Intent.CATEGORY_BROWSABLE) + +/** The page is laid out at the WebView's own height because the factory gives the WebView MATCH_PARENT layout + * params (TurboLLMWebView). It had none, so AndroidView's holder added it with ViewGroup's default, WRAP_CONTENT, and + * Chromium lays out the page of a WebView whose layout params are WRAP_CONTENT high at height 0 (android_webview's + * AwLayoutSizer turns Blink's forceZeroLayoutHeight on for it), while the view itself was measured full size and + * window.innerHeight was right. Every vh, dvh, svh, lvh and vmin length, and every percentage height resolved against + * the viewport (html, body and #root at 100%), was 0px. That is the "0px vh" first seen live over DevTools on + * 2026-09-03 and put down to the System WebView. It was worked around one class shape at a time (html/body/#root, the + * Discover list's `max-h-[45vh]`, QA_BUGS.md BUG-05's `min-h-dvh`) until the model picker's + * `max-h-[min(60vh,…)]` (TurboLLM #270), a shape the workaround did not cover, opened as an empty 280x10 strip. * - * QA_BUGS.md BUG-05 (2026-09-04): the same bug also breaks Tailwind's NAMED vh-family - * utilities (`min-h-dvh`, `h-screen`, …), not just the bracket arbitrary-value form above — - * `.app-shell`'s `min-h-dvh` (Shell.tsx, document-scroll mode) collapsing to 0 left the - * bottom tab bar rendering right after short content instead of pinned to the real bottom of - * the screen. fixNamedVhClasses covers this with one small stylesheet (updated on resize) - * instead of per-element scanning — a CSS class selector already applies to elements added - * after it exists, so no MutationObserver entry is needed for this half of the fix. */ + * HEIGHT_FIX_JS stays as a fallback for a WebView that still lays the page out at height 0, which no device check has + * ruled out yet. It measures a `height:100vh` element once per page (at the first resize that gives the window a + * height, on a page that finished loading before it had one) and does nothing unless that is under 1px while + * window.innerHeight is not. Then: htmlFix pins html, body and #root to window.innerHeight; every `h-[…]`, + * `min-h-[…]` and `max-h-[…]` class (Tailwind's `!` included) whose value has a viewport unit, alone or anywhere + * inside min()/max()/calc()/clamp(), gets that value as an `!important` inline style of the same property, each such + * unit computed in px and every other term (var() ones included) left as it is, on elements added or given the class + * later too (a MutationObserver). The value is the one in the class's rule, as Tailwind wrote it (it spaces out a + * calc()'s operators: `h-[calc(100vh-2rem)]` is `calc(100vh - 2rem)`), or the class name's, `_` read as a space, + * when no rule is found. Tailwind's named utilities (`h-dvh`, `min-h-screen`, …) get one stylesheet. All of it + * follows resize. It answers whether it did this, and onPageFinished logs a warning when it did, so logcat shows a + * WebView still laying pages out at height 0. scripts/webview-height-fix.test.mjs runs this string in Chromium under + * both kinds of host. */ private const val HEIGHT_FIX_JS = """ (function() { + // onPageFinished can come more than once for the same page: the first run decided (or waits to), and its listeners + // are in place. + if (window.__turbollmHeightFix !== undefined) return null; + + // Only a WebView that lays the page out at height 0 measures 100vh as nothing while the window has a height. + function zeroVh() { + var probe = document.createElement('div'); + probe.style.cssText = 'position:absolute;top:0;left:0;width:0;height:100vh;visibility:hidden'; + document.documentElement.appendChild(probe); + var zero = probe.getBoundingClientRect().height < 1; + document.documentElement.removeChild(probe); + return zero; + } + function htmlFix() { var px = window.innerHeight + 'px'; document.documentElement.style.height = px; @@ -86,37 +126,89 @@ private const val HEIGHT_FIX_JS = """ if (root) root.style.height = px; } - var VH_CLASS_RE = /^(min-h|max-h|h)-\[(\d+(?:\.\d+)?)(?:d|s|l)?vh\]${'$'}/; + // Tailwind's arbitrary-value height classes, whatever the value, with or without its `!` (important) before or after: + // `max-h-[60vh]`, `h-[calc(100vh-2rem)]`, `h-[50vh]!`, the model picker's + // `max-h-[min(60vh,var(--radix-dropdown-menu-content-available-height))]`. + var VH_CLASS_RE = /^!?(min-h|max-h|h)-\[(.+)\]!?${'$'}/; var PROP_BY_PREFIX = { 'h': 'height', 'min-h': 'min-height', 'max-h': 'max-height' }; + // A number with a unit that is 0px or wrong here (vh and vb, which are 0, vmin, 0 as well, and vmax, the width: each + // also d/s/l-prefixed), as a term of its own: after the start, a space, a comma, a bracket or an operator, so the + // digits inside a custom property's name (var(--a-10vh)) are not one. + var VP_UNIT_RE = /(^|[\s,(*\/+])(-?(?:\d+\.?\d*|\.\d+))(?:d|s|l)?(vh|vb|vmin|vmax)(?![\w-])/g; + + // The value of [prop] in the rule for class [cls], as Tailwind wrote it, or '' when there is none. Tailwind does not + // only read '_' as a space: it spaces out a calc()'s operators too (`h-[calc(100vh-2rem)]` is `calc(100vh - 2rem)`), + // so the class name alone is not always the CSS. Rules at the top level and in @layer blocks, where Tailwind puts its + // utilities, are collected again when a stylesheet comes or goes (a screen's own CSS loaded later). + var rules = null, ruleSheets = -1; + function declared(cls, prop) { + var sheets = document.styleSheets; + if (ruleSheets !== sheets.length) { + rules = {}; + ruleSheets = sheets.length; + var collect = function(list) { + for (var i = 0; i < list.length; i++) { + var rule = list[i]; + if (rule.selectorText !== undefined) rules[rule.selectorText] = rule.style; + else if (window.CSSLayerBlockRule && rule instanceof CSSLayerBlockRule) collect(rule.cssRules); + } + }; + for (var s = 0; s < sheets.length; s++) { + try { collect(sheets[s].cssRules); } catch (e) {} // another origin's stylesheet cannot be read + } + } + var style = rules['.' + CSS.escape(cls)]; + return style ? style.getPropertyValue(prop) : ''; + } + + // A value as a class name spells it, for a class no rule was found for: '_' is a space, '\_' an underscore, and a + // custom property's name keeps its underscores. + function decoded(raw) { + return raw.replace(/\\_|--[\w-]+|_/g, function(t) { return t === '_' ? ' ' : t === '\\_' ? '_' : t; }); + } + + // [value] with each viewport-unit term in px, or null when it has none. + function pxValue(value) { + var h = window.innerHeight, w = window.innerWidth; + var px = value.replace(VP_UNIT_RE, function(term, before, n, unit) { + var base = unit === 'vmin' ? Math.min(w, h) : unit === 'vmax' ? Math.max(w, h) : h; + return before + (parseFloat(n) / 100 * base) + 'px'; + }); + return px === value ? null : px; + } function fixVhElement(el) { if (!el.classList) return; + var next = {}, any = false; for (var i = 0; i < el.classList.length; i++) { - var m = VH_CLASS_RE.exec(el.classList[i]); + var cls = el.classList[i], m = VH_CLASS_RE.exec(cls); if (!m) continue; var prop = PROP_BY_PREFIX[m[1]]; - var px = (parseFloat(m[2]) / 100 * window.innerHeight) + 'px'; - el.style.setProperty(prop, px, 'important'); + var px = pxValue(declared(cls, prop) || decoded(m[2])); + if (px) { next[prop] = px; any = true; } } + var prev = el.__turbollmVh; + if (!prev && !any) return; + // What was set for a class the element no longer has goes too, so the class's height leaves with it. + for (var p in prev) if (!(p in next)) el.style.removeProperty(p); + for (var q in next) el.style.setProperty(q, next[q], 'important'); + el.__turbollmVh = any ? next : undefined; } function fixVhSubtree(node) { if (node.nodeType !== 1) return; fixVhElement(node); - var descendants = node.querySelectorAll ? node.querySelectorAll('[class*="vh]"]') : []; + var descendants = node.querySelectorAll ? node.querySelectorAll('[class*="h-["]') : []; for (var i = 0; i < descendants.length; i++) fixVhElement(descendants[i]); } - // QA_BUGS.md BUG-05: the same 0px-vh bug also breaks Tailwind's NAMED viewport-height - // utilities (`min-h-dvh`, `h-dvh`, `h-screen`, ...) — not just the bracket arbitrary-value - // form vhClassFix (above) already covers. `.app-shell` uses `min-h-dvh` in document-scroll - // mode (Shell.tsx) to let MobileNav stick to the true viewport bottom; with min-height - // collapsing to 0 the shell only grew to its own (often short) content, so the tab bar - // rendered right after that content instead of pinned to the screen — reported as "the - // tab bar floats mid-screen with a dead white gap below it" on tabs with little content. - // A single stylesheet, updated on resize, covers every element with these classes at - // once — no MutationObserver needed here, unlike vhClassFix's inline-style scan, since - // CSS class selectors apply to future elements automatically. + // QA_BUGS.md BUG-05: Tailwind's NAMED viewport-height utilities (`min-h-dvh`, `h-dvh`, `h-screen`, ...) are 0px + // too. `.app-shell` uses `min-h-dvh` in document-scroll mode (Shell.tsx) to let MobileNav stick to the true viewport + // bottom; with min-height collapsing to 0 the shell only grew to its own (often short) content, so the tab bar + // rendered right after that content instead of pinned to the screen — reported as "the tab bar floats mid-screen + // with a dead white gap below it" on tabs with little content. A single stylesheet, updated on resize, covers every + // element with these classes at once — no MutationObserver needed here, unlike the class scan above, since CSS + // class selectors apply to future elements automatically. var namedVhStyleEl = null; function fixNamedVhClasses() { var px = window.innerHeight + 'px'; @@ -136,24 +228,52 @@ private const val HEIGHT_FIX_JS = """ fixNamedVhClasses(); } - fixAll(); - window.addEventListener('resize', fixAll); + function activate() { + fixAll(); + window.addEventListener('resize', fixAll); - var observer = new MutationObserver(function(mutations) { - for (var i = 0; i < mutations.length; i++) { - var added = mutations[i].addedNodes; - for (var j = 0; j < added.length; j++) fixVhSubtree(added[j]); - } - }); - observer.observe(document.documentElement, { childList: true, subtree: true }); + // Content rendered after this (an async-fetched list, a Radix portal) and a class set on an existing element later. + var observer = new MutationObserver(function(mutations) { + for (var i = 0; i < mutations.length; i++) { + if (mutations[i].type === 'attributes') { fixVhElement(mutations[i].target); continue; } + var added = mutations[i].addedNodes; + for (var j = 0; j < added.length; j++) fixVhSubtree(added[j]); + } + }); + observer.observe(document.documentElement, { + childList: true, subtree: true, attributes: true, attributeFilter: ['class'], + }); + } + + // A page that finished loading before the WebView was laid out has no height yet, so 100vh is 0 whatever the WebView: + // it decides at the first resize that gives the window one. onPageFinished has had its answer by then, so that + // warning goes to the page's console. + if (!(window.innerHeight > 0)) { + window.__turbollmHeightFix = null; + window.addEventListener('resize', function sized() { + if (!(window.innerHeight > 0)) return; + window.removeEventListener('resize', sized); + window.__turbollmHeightFix = zeroVh(); + if (!window.__turbollmHeightFix) return; + activate(); + console.warn("100vh is 0px in this WebView (the page is laid out at height 0); HEIGHT_FIX_JS pins the UI's " + + 'viewport heights to window.innerHeight'); + }); + return null; + } + // One probe per page: on a WebView that lays the page out at its own height, that is all the script does. + window.__turbollmHeightFix = zeroVh(); + if (!window.__turbollmHeightFix) return false; + activate(); + return true; })(); """ -/** Injects the native window insets as CSS custom properties so the web UI's - * `var(--tllm-safe-top)` / `var(--tllm-safe-bottom)` resolve to real pixel values. - * Android's System WebView resolves `env(safe-area-inset-*)` to 0px on many builds, - * so we read the insets natively and patch them in via JS (same pattern as HEIGHT_FIX_JS). - * Called after HEIGHT_FIX_JS so the CSS vars are ready before the web layout reads them. +/** Sets the web UI's safe-area padding, the `--tllm-safe-top` / `--tllm-safe-bottom` CSS custom properties, on the + * page's root element, where it wins over index.css (which, below 768px wide, defaults them to + * `env(safe-area-inset-*, 24px)` for a page drawn under the system bars). onPageFinished passes 0: the WebView is + * already inset away from every system bar and the display cutout natively (TurboLLMWebView), so the page must not + * pad for them again, whatever env() reports inside it. * * [topPx]/[bottomPx] are PHYSICAL pixels (that's what WindowInsets reports) — CSS `px` * inside the WebView is device-independent and gets multiplied by devicePixelRatio at @@ -182,11 +302,21 @@ private fun WebView.injectSafeAreaInsets(topPx: Int, bottomPx: Int) { @Composable fun DaemonWebViewScreen() { var ready by remember { mutableStateOf(false) } - val client = remember { DaemonClient() } + // Counts the WebView renderers that have died (TurboLLMWebView's onRenderProcessGone). Each death gives the screen + // a new key, so the whole TurboLLMWebView is built again: the dead WebView leaves the composition (its onRelease + // destroys it) and a fresh one is made with the same setup as the first. + var renderer by remember { mutableIntStateOf(0) } + // Unless the renderer keeps crashing: a page that crashes it each time it loads was built and crashed again in a + // loop. After RENDERER_CRASHES crashes within a minute (RendererCrashes; a kill for memory does not count) the + // screen shows RendererCrashScreen in its place, until the user taps Reload. + val crashes = remember { RendererCrashes() } + var crashLoop by remember { mutableStateOf(false) } + // Up AND ours: the WebView below gets the app-secret cookie the moment it is created, so it waits until the + // daemon has proved it holds the port (DaemonAuth), not merely until something answers there. LaunchedEffect(Unit) { while (!ready) { - ready = client.isDaemonReachable(ANDROID_DAEMON_PORT) + ready = withContext(Dispatchers.IO) { DaemonAuth.daemonProven() } if (!ready) delay(500) } } @@ -247,16 +377,51 @@ fun DaemonWebViewScreen() { modifier = Modifier.padding(top = 20.dp), ) } + } else if (crashLoop) { + RendererCrashScreen(onReload = { + Log.i(TAG, "Reload tapped after the WebView renderer kept crashing; loading the UI again") + crashLoop = false + }) } else { - TurboLLMWebView() + key(renderer) { + TurboLLMWebView(onRendererGone = { didCrash -> + val again = crashes.rebuild(didCrash, SystemClock.elapsedRealtime()) + if (again) renderer++ else crashLoop = true + again + }) + } } } } } +/** Shown in place of the WebView once its renderer has crashed RENDERER_CRASHES times within a minute: plain native + * text, and a Reload button that builds the WebView again. The daemon, the loaded model and any download keep + * running behind it; only the page is gone. On the splash's pinned dark Surface, hence the explicit light text color + * (see the splash's own Text). */ +@Composable +private fun RendererCrashScreen(onReload: () -> Unit) { + // Back leaves the app as from the UI's root screen (TurboLLMWebView's BackHandler), finishing the activity, so it + // goes through the same close (EngineService.activityFinished) whichever screen is up. + val activity = LocalContext.current as? Activity + BackHandler(enabled = true) { activity?.finish() } + Column( + modifier = Modifier.fillMaxSize().windowInsetsPadding(WindowInsets.safeDrawing).padding(24.dp), + verticalArrangement = Arrangement.Center, + horizontalAlignment = Alignment.CenterHorizontally, + ) { + Text("The page keeps crashing.", color = Color(0xFFEDEDEC)) + Button(onClick = onReload, modifier = Modifier.padding(top = 20.dp)) { + Text("Reload") + } + } +} + +/** [onRendererGone] is told of each renderer death, and whether it was a crash; it answers whether a fresh WebView + * is built (DaemonWebViewScreen). */ @SuppressLint("SetJavaScriptEnabled") @Composable -private fun TurboLLMWebView() { +private fun TurboLLMWebView(onRendererGone: (didCrash: Boolean) -> Boolean) { var webViewRef by remember { mutableStateOf(null) } // Hardware back (QA_BUGS.md BUG-02, then QA_UX_REPORT.md F-01). Two earlier shapes of this // both failed: `enabled = webViewRef?.canGoBack() == true` was a snapshot Compose could never @@ -298,7 +463,14 @@ private fun TurboLLMWebView() { // (adjustResize no longer resizes an edge-to-edge window), so the web UI needs no insets of its own. modifier = Modifier.fillMaxSize().windowInsetsPadding(WindowInsets.safeDrawing), factory = { context -> - WebView.setWebContentsDebuggingEnabled(true) + // Debuggable builds only. DevTools attached over adb runs script in the daemon's origin, where every + // request carries the HttpOnly app-secret cookie, and can read that cookie too: the whole keyless API, + // creating an API key included, which a release build must not give an adb host (BUILDING-ENGINES.md). + // Other apps were never let in (the devtools socket admits only the shell, root and this app). QA over + // CDP (QA_BUGS.md) uses a debug build; -PturbollmApplicationIdSuffix installs one next to the Play app. + WebView.setWebContentsDebuggingEnabled( + (context.applicationInfo.flags and ApplicationInfo.FLAG_DEBUGGABLE) != 0, + ) // QA_BUGS.md BUG-13: by default the Android System WebView renders as a single // opaque NAF="true" node in the accessibility tree — invisible to TalkBack and // to UI-Automator/Maestro element selectors. Chrome's accessibility bridge for @@ -306,6 +478,16 @@ private fun TurboLLMWebView() { // not enough. Flagging the view as important for accessibility helps the // platform expose at least the WebView container itself to TalkBack. WebView(context).apply { + // MATCH_PARENT both ways, set before AndroidView adds the WebView (it does so right after this + // factory returns, here and for the WebView built again after a renderer crash). With no layout params + // it got ViewGroup's default WRAP_CONTENT, and Chromium lays out the page of a WebView whose height is + // WRAP_CONTENT at height 0: every vh unit and viewport-relative height in the UI was 0px (see + // HEIGHT_FIX_JS). The view's own size does not change: AndroidView measures it EXACTLY at the size + // its modifier gives either way. + layoutParams = ViewGroup.LayoutParams( + ViewGroup.LayoutParams.MATCH_PARENT, + ViewGroup.LayoutParams.MATCH_PARENT, + ) settings.javaScriptEnabled = true settings.domStorageEnabled = true webViewClient = object : WebViewClient() { @@ -322,25 +504,113 @@ private fun TurboLLMWebView() { } return true // never let the WebView try to load this scheme } - // A link off the daemon (Hugging Face, docs, a model card) opens in the browser rather than - // replacing the app's UI inside its own WebView, where Back would be the only way out. - val external = (url.scheme == "http" || url.scheme == "https") && - url.host != "127.0.0.1" && url.host != "localhost" + // The daemon under the name localhost (a link in a reply or a docs snippet): opened as is, it is a + // host the app-secret cookie (scoped to 127.0.0.1) never reaches, so the page asked for an API key. + // Load the same page from 127.0.0.1 instead; it stays in the app, like any other daemon link. An + // https link to the daemon's port, under either name, is loaded the same way, over http: the + // daemon speaks plain http, so loading it as is put a TLS error page (ERR_SSL_PROTOCOL_ERROR) + // over the UI. + val onDaemonPort = (url.host == "localhost" || url.host == "127.0.0.1") && + url.port == ANDROID_DAEMON_PORT + if (request.isForMainFrame && onDaemonPort && + (url.scheme == "https" || (url.scheme == "http" && url.host == "localhost")) + ) { + val http = url.buildUpon().scheme("http").encodedAuthority("127.0.0.1:$ANDROID_DAEMON_PORT") + view.loadUrl(http.build().toString()) + return true + } + // A link off the daemon (Hugging Face, docs, a model card) opens outside the app rather than + // replacing the app's UI inside its own WebView, where Back would be the only way out. So does + // localhost on any port but the daemon's: that is some other app's server, not ours. It goes + // where a browser would send it (linkIntent): the browser, or an app that has claimed that web + // address (an App Link), but never an activity that does not take links from web pages. + val external = (url.scheme == "http" || url.scheme == "https") && url.host != "127.0.0.1" && + !(url.host == "localhost" && url.port == ANDROID_DAEMON_PORT) if (request.isForMainFrame && external) { - runCatching { view.context.startActivity(Intent(Intent.ACTION_VIEW, url)) } + runCatching { view.context.startActivity(linkIntent(url)) } + return true + } + // A scheme the WebView cannot load: navigated to, it replaced the app's UI with an + // ERR_UNKNOWN_URL_SCHEME page, and Back then reloaded the UI from scratch, losing its state. A + // contact link (mailto: and the like) opens in the app that handles it, sent the same way and + // main frame only, like the links above. Anything else (intent:, market:, file:, a deep link in + // another app's own scheme) does nothing, since what the page shows (a reply, a model card) is + // not to be trusted to open apps. An https deep link is a web link like any other: it still + // opens the app that claimed its address, as it would from a browser. + if (url.scheme !in WEBVIEW_SCHEMES) { + if (request.isForMainFrame && url.scheme in HANDED_OFF_SCHEMES) { + runCatching { view.context.startActivity(linkIntent(url)) } + } return true } + // Another 127.0.0.1 port would be sent the app-secret cookie (see shouldInterceptRequest + // below). Unlike that callback, this one runs again for each redirect hop of a main-frame + // navigation, so a link that 302s to 127.0.0.1: is stopped here, before that + // request (and the cookie) goes out. + if (url.host == "127.0.0.1" && url.port != ANDROID_DAEMON_PORT) return true return false } + // The app-secret cookie (below) is scoped to 127.0.0.1, and cookies ignore ports: anything on the + // page pointing at another loopback port (an image in a model card or a reply, a link) would hand + // it to whichever app listens there. Nothing the UI needs is on another port, so a request that + // starts there is refused here. This only sees a request's FIRST URL, not its redirect hops: an + // https image that 302s to 127.0.0.1: gets past it. Those hops are covered elsewhere: + // for images and media by the daemon's CSP on the UI's HTML (APP_SECRET_CSP, daemon server.ts), + // which is checked on every redirect; for main-frame navigations by shouldOverrideUrlLoading above. + // Runs on a WebView thread, so it touches nothing but the request. + override fun shouldInterceptRequest(view: WebView, request: WebResourceRequest): WebResourceResponse? { + val url = request.url + if (url.host != "127.0.0.1" || url.port == ANDROID_DAEMON_PORT) return null + return WebResourceResponse( + "text/plain", "utf-8", 403, "Forbidden", emptyMap(), ByteArrayInputStream(ByteArray(0)), + ) + } + override fun onPageFinished(view: WebView, url: String?) { super.onPageFinished(view, url) - view.evaluateJavascript(HEIGHT_FIX_JS, null) + view.evaluateJavascript(HEIGHT_FIX_JS) { activated -> + // "true" only from the page's first run, and only when it found 100vh at 0px: the WebView + // still lays the page out at height 0 despite its MATCH_PARENT layout params. A page that + // finished loading before the WebView had a height answers null, decides at its first + // resize, and warns in its own console then. + if (activated == "true") { + Log.w( + TAG, + "100vh is 0px in this WebView (the page is laid out at height 0); HEIGHT_FIX_JS " + + "pins the UI's viewport heights to window.innerHeight", + ) + } + } // The WebView is already inset away from every system bar (the modifier above), so the // web UI's own safe-area padding (Shell.tsx --tllm-safe-top/bottom) must be zero — // anything else would pad twice. view.injectSafeAreaInsets(0, 0) } + + // The renderer process is gone: Android killed it for memory (most likely with the app in the + // background) or it crashed. Without this override Android's default then kills the app's own + // process as well, and that process holds the daemon, the loaded model and any download. Handled + // instead: this WebView can never be used again, so the screen drops it and builds a fresh one + // (DaemonWebViewScreen) with this same factory, set up as on first load: the app-secret cookie + // (below), then the UI. After RENDERER_CRASHES crashes within a minute it shows its Reload screen + // instead (RendererCrashes). Called on Android 8+ only, where the callback exists. + @RequiresApi(Build.VERSION_CODES.O) + override fun onRenderProcessGone(view: WebView, detail: RenderProcessGoneDetail): Boolean { + if (webViewRef === view) webViewRef = null + val next = if (onRendererGone(detail.didCrash())) { + "loading the UI again" + } else { + "it crashed $RENDERER_CRASHES times within ${RENDERER_CRASH_WINDOW_MS / 1000} s, " + + "so the UI is not loaded again until Reload" + } + Log.w( + TAG, + "WebView renderer gone (didCrash=${detail.didCrash()}, " + + "rendererPriorityAtExit=${detail.rendererPriorityAtExit()}); $next", + ) + return true + } } webChromeClient = object : WebChromeClient() { override fun onShowFileChooser( @@ -363,9 +633,27 @@ private fun TurboLLMWebView() { } } } + // The daemon trusts only loopback requests that carry the app secret (DAEMON_APP_SECRET); this cookie is + // how every request the page makes carries it, WebSocket handshakes included. HttpOnly: page scripts + // (a chat artifact included) cannot read it. Set before the first load, every launch, since the secret + // changes with each process, and only once the daemon has proved it holds the port (DaemonWebViewScreen + // creates this WebView after DaemonAuth.daemonProven). + CookieManager.getInstance().apply { + setCookie(DAEMON_URL, "turbollm_app=$DAEMON_APP_SECRET; Path=/; HttpOnly; SameSite=Strict") + flush() + } loadUrl(DAEMON_URL) webViewRef = this } }, + // The WebView goes with its screen (Back finishes the activity, which disposes this composition). Nothing used + // to destroy it, so the WebView, its page and the renderer process behind it lived on until the garbage + // collector reached them. The page going away does not touch the daemon's work, so this only frees the memory + // and the renderer sooner. Loading is stopped first, then the WebView is destroyed: it is never used again. + onRelease = { webView -> + if (webViewRef === webView) webViewRef = null + webView.stopLoading() + webView.destroy() + }, ) } diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt new file mode 100644 index 0000000..dda5027 --- /dev/null +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt @@ -0,0 +1,87 @@ +package com.turbollm.app + +import org.json.JSONObject +import java.util.Locale +import kotlin.math.roundToLong + +/** The engine notification's progress bar is out of this many steps: a multi-GB byte count does not fit the Int + * setProgress takes, and a thousand steps is finer than the bar can draw. */ +internal const val DOWNLOAD_PROGRESS_MAX = 1000 + +/** One row of the daemon's GET /api/v1/downloads that is running or queued, the two statuses that keep the engine + * service at work (a paused or failed download transfers nothing). [total] is 0 while the size is unknown: a raw + * URL until its reply names a length. [bytesPerSec] is the daemon's own measure, 0 before the first one. */ +internal data class LiveDownload( + val name: String, + val queued: Boolean, + val received: Long, + val total: Long, + val bytesPerSec: Double, +) + +/** What the engine notification shows while downloads run or wait (EngineService.buildNotification). [progress] is + * out of [DOWNLOAD_PROGRESS_MAX] and means nothing when [indeterminate]. */ +internal data class DownloadNotice(val title: String, val text: String, val progress: Int, val indeterminate: Boolean) + +/** The running and queued rows of a GET /api/v1/downloads reply, or null when the reply is not the list the daemon + * sends (the poll then counts as failed, and as idle, as before). A row with a field missing or of another type + * reads that field as 0 instead of failing the whole list. */ +internal fun parseLiveDownloads(body: String): List? { + val list = runCatching { JSONObject(body).optJSONArray("downloads") }.getOrNull() ?: return null + return (0 until list.length()).mapNotNull { i -> + val row = list.optJSONObject(i) ?: return@mapNotNull null + val status = row.optString("status") + if (status != "downloading" && status != "queued") return@mapNotNull null + val speed = row.optDouble("bytesPerSec", 0.0) + LiveDownload( + name = row.optString("name"), + queued = status == "queued", + received = row.optLong("received").coerceAtLeast(0), + total = row.optLong("total").coerceAtLeast(0), + bytesPerSec = if (speed.isFinite() && speed > 0) speed else 0.0, + ) + } +} + +/** The notification for [rows], or null when there are none. One running file is named in the title (the system + * ellipsizes a long name), with " of · " under it; several are summed under "Downloading + * models", as "N downloads · ...". Queued rows are counted as "N waiting", not summed: nothing of theirs is moving. + * The bar runs over the summed totals of the running files, and is indeterminate while any of their sizes is + * unknown or nothing is running yet. A speed of 0 (none measured yet) is left out, as the web's Downloads panel + * leaves it out. */ +internal fun downloadNotice(rows: List): DownloadNotice? { + if (rows.isEmpty()) return null + val running = rows.filter { !it.queued } + val waiting = rows.size - running.size + val received = running.sumOf { it.received } + val total = running.sumOf { it.total } + val sized = running.isNotEmpty() && running.all { it.total > 0 } + val speed = running.sumOf { it.bytesPerSec } + val amount = buildString { + append(formatSize(received)) + if (sized) append(" of ").append(formatSize(total)) + if (speed > 0) append(" · ").append(formatSpeed(speed)) + } + val text = when (running.size) { + 0 -> if (waiting == 1) "1 download waiting" else "$waiting downloads waiting" + 1 -> amount + else -> "${running.size} downloads · $amount" + } + return DownloadNotice( + title = running.singleOrNull()?.name?.takeIf { it.isNotBlank() } ?: "Downloading models", + text = if (running.isNotEmpty() && waiting > 0) "$text · $waiting waiting" else text, + progress = if (sized) (received.coerceAtMost(total) * DOWNLOAD_PROGRESS_MAX / total).toInt() else 0, + indeterminate = !sized, + ) +} + +/** A size as the web's Downloads panel writes it (DownloadsPanel.tsx fmtSize), so the shade and the app agree: + * decimal units, "1.23 GB" from 1 GB up and whole megabytes below. Locale.ROOT, like the panel's toFixed. */ +internal fun formatSize(bytes: Long): String = + if (bytes >= 1_000_000_000L) String.format(Locale.ROOT, "%.2f GB", bytes / 1e9) + else "${(bytes / 1e6).roundToLong()} MB" + +/** A speed as the same panel writes it (fmtSpeed): "12.3 MB/s" from 1 MB/s up, whole kilobytes below. */ +internal fun formatSpeed(bytesPerSec: Double): String = + if (bytesPerSec >= 1e6) String.format(Locale.ROOT, "%.1f MB/s", bytesPerSec / 1e6) + else "${(bytesPerSec / 1e3).roundToLong()} KB/s" diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt index 07292ac..d5b7e94 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt @@ -12,22 +12,41 @@ import android.os.Build import android.os.IBinder import android.os.PowerManager import android.os.Process +import android.os.SystemClock import android.util.Log import androidx.core.app.NotificationCompat -import androidx.core.app.NotificationManagerCompat import androidx.core.app.ServiceCompat import androidx.core.content.ContextCompat import org.json.JSONObject -import java.util.concurrent.atomic.AtomicBoolean -import java.net.HttpURLConnection -import java.net.URL private const val TAG = "TurboLLM" private const val CHANNEL_ID = "engine" private const val NOTIFICATION_ID = 1 private const val ACTION_QUIT = "com.turbollm.app.QUIT" private const val ACTION_ACTIVITY_FINISHED = "com.turbollm.app.ACTIVITY_FINISHED" +private const val ACTION_REPOST = "com.turbollm.app.REPOST" +private const val EXTRA_ACTIVITY = "com.turbollm.app.ACTIVITY" private const val POLL_MS = 3000L +/** How many polls in a row may fail to read the download list before the notification drops its last progress. */ +private const val DOWNLOAD_MISSES = 3 +/** How many polls in a row may fail to read the activity before the notification drops its last work or approval + * line. */ +private const val ACTIVITY_MISSES = 3 +/** The longest the daemon is given to write its database before the process ends (flushDatabase). */ +private const val FLUSH_WAIT_MS = 2000L +/** How long a close made while the daemon gives no proof (still starting, mostly) waits for it before it is decided + * (appClosed), give or take one proof: one that hangs to its 3 s deadline as this runs out stretches the wait to + * about 33.5 s. A daemon that has not proved itself by then counts as something running. */ +private const val BOOT_WAIT_MS = 30_000L +/** How often that close asks the daemon for its proof again, as the splash does (DaemonWebView). */ +private const val BOOT_POLL_MS = 500L +/** The shortest time from a close (appClosed) or a Quit (quit) to the process kill. The WebView writes the page's + * localStorage (the theme, the font size, the chat to open again, the onboarding step) to disk only 1 s after a + * change: WebView turns on Chromium's aggressive DOM storage flushing, whose commit delay is 1 s, and a change made + * while a commit is under way waits for that one first. Nothing writes it when the process is killed. */ +private const val CLOSE_HOLD_MS = 3000L +/** The primary engine states in which it holds a model (the daemon's manager.ts State): see statusReading. */ +private val LOADED_STATES = setOf("starting", "running", "stopping") /** Keeps the daemon and its engine processes running while the app is not on screen. * @@ -40,29 +59,61 @@ private const val POLL_MS = 3000L * Type `specialUse` (Android 14+): a local inference server fits none of the fixed types, and * `dataSync` would be wrong and is capped at 6 hours on Android 15. * - * It also holds a partial wake lock, but only while something is generating, so a long reply - * finishes with the screen off without keeping the CPU awake while idle. "Generating" comes - * from the daemon's own /api/v1/activity (the gate the UI uses), polled over loopback. */ + * It also holds a partial wake lock, but only while there is work: a reply being generated, or a + * model load, an auto-tune or a skill being written (the daemon's own /api/v1/activity, the gate + * the UI uses), a model download (/api/v1/downloads), both polled over loopback, or a model + * import (ModelImport). So a long reply or a multi-GB transfer finishes with the screen off + * without keeping the CPU awake while idle, or while a chat waits for the user to answer a tool + * approval. */ class EngineService : Service() { @Volatile private var running = false private var wakeLock: PowerManager.WakeLock? = null private var thermalListener: PowerManager.OnThermalStatusChangedListener? = null @Volatile private var busy = false + /** The notification's line for what the daemon is running (DaemonActivity.work), null while nothing is. Written by + * the poll, read wherever the notification is built, like the fields below. */ + @Volatile private var work: String? = null + /** A chat waits for the user to answer a tool approval (DaemonActivity.waiting): named in the notification, not + * work, so no wake lock is held for it. */ + @Volatile private var waiting = false + /** The loaded model's name (StatusReading.name), which the notification names while nothing runs; null while no + * model is loaded or the loaded one has no name in the status. Read by the poll only while nothing runs. */ + @Volatile private var model: String? = null @Volatile private var importing = false + /** What the notification shows while a model download is running or queued (downloadNotice), null while none + * is. Written by the poll, read wherever the notification is built. */ + @Volatile private var download: DownloadNotice? = null + /** Polls in a row that could not read the download list (see pollActivity). Poll thread only. */ + private var downloadMisses = 0 + /** Polls in a row that could not read the activity (see pollActivity). Poll thread only. */ + private var activityMisses = 0 @Volatile private var thermal = 0 - /** Set when the app was closed with notifications off: stop as soon as nothing is generating (see onStartCommand). */ - @Volatile private var stopWhenIdle = false - /** The newest start request, so a stop only takes effect if no start arrived after it (stopSelfResult). Without - * this, Back then a quick reopen could stop the service under a pending startForegroundService, which Android - * answers by crashing the app ("did not then call Service.startForeground()"). */ + /** The newest plain start request (EngineService.start, from MainActivity.onCreate), so a close quits only if no + * start arrived after it (see appClosed). A close, a repost and Quit do not count: a repost (the notification + * swiped away) or a second close while a close is being decided must not cancel the quit. */ @Volatile private var lastStartId = 0 - private val stopping = AtomicBoolean(false) + /** The close being decided (appClosed), as the session it was made in (session), or null while none is. A close + * made in the same one (Back, then the card it left in Recents swiped away) shares its verdict instead of + * reading and flushing again. Main thread only. */ + private var deciding: Triple? = null + /** The Quit under way (quit), as the session it was tapped in, or null while none is. It ends the app unless the + * app is opened again during its hold, so a close made in that session decides nothing, and one being decided + * there is called off. Set on the main thread, read by the close's thread too. */ + @Volatile private var quitting: Triple? = null + /** The number of the newest Quit (quit), so a Quit tapped again during the hold of the last one takes over from + * it. Set on the main thread. */ + @Volatile private var quits = 0 /** Guards `running` against the poll thread: with it held, a check of `running` and the wake-lock or * notification call after it are one step, so onDestroy cannot slip between them. */ private val stateLock = Any() override fun onBind(intent: Intent?): IBinder? = null + /** The app's session, as the service sees it: the newest activity (newestActivity), the newest plain start + * (lastStartId) and how many times the app came on screen (appShown). A close or a Quit keeps the one it was + * made in: any change since means the app was opened again. */ + private fun session() = Triple(newestActivity, lastStartId, appShown) + override fun onCreate() { super.onCreate() val manager = getSystemService(NotificationManager::class.java) @@ -86,7 +137,6 @@ class EngineService : Service() { } override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int { - lastStartId = startId if (intent?.action == ACTION_QUIT) { quit() return START_NOT_STICKY @@ -96,10 +146,22 @@ class EngineService : Service() { stopSelf(startId) // this intent created the service; it was not running in the foreground return START_NOT_STICKY } + // Only the newest activity's close counts. Android destroys a finished activity once the next one on + // screen is idle, so after Back and a quick reopen the old activity's close can come after the new one's + // start, and acting on it unloaded the model under the open app (and with notifications off stopped the + // service too); now it could quit the open app. + if (intent.getIntExtra(EXTRA_ACTIVITY, 0) != newestActivity) return START_NOT_STICKY appClosed() return START_NOT_STICKY } - stopWhenIdle = false + if (intent?.action == ACTION_REPOST) { + // Android 14+ lets the user swipe the engine notification away (setOngoing no longer prevents it), and its + // Quit with it, while a service the app was closed under keeps running until Quit (appClosed). Put it + // straight back. MainActivity sends this too whenever it comes on screen (repostNotification). + if (running) updateNotification() else stopSelf(startId) + return START_NOT_STICKY + } + lastStartId = startId ServiceCompat.startForeground( this, NOTIFICATION_ID, buildNotification(), if (Build.VERSION.SDK_INT >= 34) ServiceInfo.FOREGROUND_SERVICE_TYPE_SPECIAL_USE else 0, @@ -123,79 +185,196 @@ class EngineService : Service() { super.onDestroy() } - /** "Quit" in the notification: the only way to free a loaded model's memory short of force-stop, since the - * service otherwise outlives the activity. Killing the process takes the engine children with it (Android kills - * the app's whole process group). */ + /** "Quit" in the notification: how the user frees a loaded model's memory without going back into the app, since + * the service otherwise outlives the activity (with notifications off there is no Quit, see appClosed). Killing + * the process takes the engines with it: the kernel kills each one when the daemon's thread ends, since the + * daemon starts it through libtether_exec.so (BUILDING-ENGINES.md, "The engine tether"), and system_server then + * kills whatever is left in the app's cgroup. A cgroup, not a process group: an engine may lead its own group. + * + * The daemon writes its chat database to disk first (flushDatabase). The kill used to come at once, and the + * daemon writes the database only on its own schedule, so a reply that had just finished, or a chat started just + * before Quit, was gone when the app was opened again. The flush runs off the main thread, where this is called, + * and takes at most FLUSH_WAIT_MS; the service stays in the foreground until then, so Android cannot freeze the + * process under the write. The kill follows whatever the flush answered. + * + * The flush and the kill wait until CLOSE_HOLD_MS after the tap, as a close's do (appClosed): the kill came right + * after the flush, so a setting the page had stored less than a second before Quit (the theme, say) was lost + * with it. The app opened again during the hold or the flush (session: a new activity, a newer start, or the app + * brought back on screen after Home, which the kill would otherwise end under the user) calls the Quit off, and + * the service goes on as before the tap. A Quit tapped again meanwhile takes over, with a hold of its own. A close + * made before the app is opened again decides nothing: the Quit ends the app (appClosed). */ private fun quit() { Log.i(TAG, "quit from the notification") - synchronized(stateLock) { - running = false - releaseWakeLock() - } + val session = session() + quitting = session + val quit = ++quits + val tappedAt = SystemClock.uptimeMillis() + Thread({ + // Not called off: no activity created, no newer start and the app not back on screen since, no newer Quit. + fun current() = quits == quit && session() == session + val hold = CLOSE_HOLD_MS - (SystemClock.uptimeMillis() - tappedAt) + if (hold > 0) Thread.sleep(hold) + if (current()) flushDatabase() + ContextCompat.getMainExecutor(this).execute { + // On the main thread, where MainActivity.onCreate takes its number: the app is either open again by + // now, or it is not opened before the kill. + if (quits == quit) quitting = null + if (current()) { + synchronized(stateLock) { + running = false + releaseWakeLock() + } + exit() + } + } + }, "turbollm-quit").start() + } + + /** How Quit ends, and a close with nothing running (appClosed): out of the foreground, stopped, and the process + * killed, the daemon and its engine children with it. Main thread, after flushDatabase. */ + private fun exit() { ServiceCompat.stopForeground(this, ServiceCompat.STOP_FOREGROUND_REMOVE) stopSelf() Process.killProcess(Process.myPid()) } - /** Swiped away from Recents: the same as backing out of the app. */ + /** Swiped away from Recents: the same as backing out of the app. Also the card that Back left there, with no + * activity left in it. A swipe sends only this: the activity it finishes sends no close of its own + * (MainActivity.onDestroy), so one swipe is one close. */ override fun onTaskRemoved(rootIntent: Intent?) { if (running) appClosed() super.onTaskRemoved(rootIntent) } - /** The user left the app (Back, or swiped from Recents). With the notification visible, its Quit is how they free - * the model, so the service stays. Without it there is nothing to reach, so the service unloads the model and - * stops once nothing is generating, and the process becomes an ordinary closed app Android can reclaim. */ + /** The user closed the app: Back from its root screen, or a swipe from Recents (Home is not a close). With nothing + * running, that is a Quit, whatever the notification settings: the database is written (flushDatabase) and the + * process ends (exit), so nothing of the app is left in memory. With anything running (anythingRuns: a model + * loaded or loading, work in progress, a reply waiting for a tool approval, a download or an import, or a server + * feature on: LAN access, remote access, Turbo Link peers hosted), the service stays in the foreground and keeps + * the daemon up, also after that work ends or that feature is turned off: only Quit, or a later close with + * nothing running (opening the app again and closing it, or swiping away the card a Back left in Recents), ends + * it. Leaving never unloads the model: only the user does (in the app, or with Quit), or the daemon's own idle + * timer (Settings, daemon.idleTtlMinutes), which leaves the service up. A routine counts while it runs (an + * activity item); one only scheduled for later does not, and the routine scheduler ends with the app. + * + * It used to quit nothing: with the notification visible the service stayed until Quit, even with no model + * loaded, and with notifications off it stopped once nothing was loaded or running, leaving the process to + * Android, which kept it as a cached app. With notifications off there is no Quit: on Android 13+ the system's + * active-apps list can still stop the service, and a later close with nothing running quits. + * + * The poll's view (`busy`) is up to 3 s old: a reply sent or a model load started just before the close would not + * be in it yet. So the daemon is asked now, and again after the flush, right before the kill. A query that fails + * counts as something running: the daemon is never quit on a guess. Nor does the kill go ahead once an activity + * was created since the close (the app opened again: MainActivity.onCreate takes its number at once, while the + * start it sends waits on the main thread behind that onCreate) or a newer start arrived. The kill comes no + * sooner than CLOSE_HOLD_MS after the close, so the WebView has written what the page last stored. A Quit tapped + * meanwhile calls the decision off and ends the app itself (quit). + * + * A close made while the daemon gives no proof (DaemonAuth.daemonProven, which the splash waits on too) first + * waits for it: it asks again every BOOT_POLL_MS, for about BOOT_WAIT_MS (a proof that hangs to its 3 s deadline + * as that runs out stretches it to about 33.5 s) and only while the close is still the app's last, and is then + * decided on what the daemon says. Mostly that is a close during start-up, before the daemon listens (a swipe + * from Recents during the splash, and before Android 12 Back there too), but it is also one made in the gap + * while a LAN toggle rebinds the daemon's port, or while the daemon is up but does not answer in time. Read at + * once, the daemon not up yet failed the query, which counted as something running, so a close during the + * splash kept the app until Quit with nothing ever run. A daemon that has not proved itself by the end of the + * wait, or that proves itself and then answers badly, still counts as something running. The hold still counts + * from the close. With the model's auto-load at start on (Settings), that load begins only after the daemon's + * boot scan, so a close read right after the daemon first answers can quit before it begins: nothing ran at the + * close. + * + * A second close in the same session (session) while one is being decided (Back, then the card it left swiped + * away) shares that decision's verdict: decided afresh once the first had ended, it would read the daemon + * seconds after that close, so a close made while a reply ran would quit the app if the reply had ended by then. + * A close after the app was opened again is decided on its own, at once, as a first close is: the earlier + * decision is called off by that reopening, and its reading may predate it. Queued behind that decision, a close + * made before the reopening would be decided once the decision ended, in the open app's session, and kill the + * app on screen. */ private fun appClosed() { - if (quitReachable()) return - stopWhenIdle = true - if (!busy) stopNow() - } - - /** Whether the notification, and its Quit, can be seen: notifications on for the app (Android 13+ asks) and the - * Engine channel not turned off on its own. */ - private fun quitReachable(): Boolean { - if (!NotificationManagerCompat.from(this).areNotificationsEnabled()) return false - if (Build.VERSION.SDK_INT >= 26) { - val channel = getSystemService(NotificationManager::class.java).getNotificationChannel(CHANNEL_ID) - if (channel?.importance == NotificationManager.IMPORTANCE_NONE) return false - } - return true - } - - /** Unload the model, then stop, unless the app came back or a newer start request arrived in the meantime. - * Unloading first matters: a stopped service alone leaves the model resident until Android reclaims the process, - * and with no notification there would be no other way to free it. */ - private fun stopNow() { - if (!stopping.compareAndSet(false, true)) return - val startId = lastStartId + // Post it again first: a post made while notifications were off is dropped for good (on Android 13+ + // startForeground posts before the user can answer the permission prompt), and a later grant, at that prompt + // or in Settings with the process still alive, re-posts nothing. Without this, Back after such a grant could + // leave the service running with no notification and no Quit. + updateNotification() + val session = session() + // A Quit tapped in this session ends the app itself; a close in it already being decided shares that one. + if (quitting == session || deciding == session) return + deciding = session + // uptimeMillis, as the WebView's commit timer, does not count deep sleep. + val closedAt = SystemClock.uptimeMillis() Thread({ - // `busy` is the poll's view, up to 3 s old: a reply sent just before Back would not be in it yet. Ask the - // daemon now, and again right before unloading, so a generation that has just started is never aborted. - val idle = !queryBusy() && !ModelImport.isRunning() - if (idle) unloadModel { stopWhenIdle && !queryBusy() && !ModelImport.isRunning() } + // Still the app's last close: no activity created and no newer start since (the app opened again), and no + // Quit tapped in its session. + fun current() = running && quitting != session && session() == session + // No proof from the daemon (still starting, its port rebound for a LAN toggle, or not answering in time): + // wait for it, so the close is decided on what it runs, not on its silence. + val bootUntil = SystemClock.uptimeMillis() + BOOT_WAIT_MS + while (current() && SystemClock.uptimeMillis() < bootUntil && !DaemonAuth.daemonProven()) { + Thread.sleep(BOOT_POLL_MS) + } + // true: quit. false: something runs, so the service stays. null: called off, the app was opened again or + // Quit tapped. + val verdict: Boolean? = if (!current()) null else if (anythingRuns()) false else { + val hold = CLOSE_HOLD_MS - (SystemClock.uptimeMillis() - closedAt) + if (hold > 0) Thread.sleep(hold) + if (!current()) { + null + } else { + flushDatabase() + if (current()) !anythingRuns() else null + } + } ContextCompat.getMainExecutor(this).execute { - stopping.set(false) - // onDestroy releases the wake lock and clears `running` once the stop takes effect. - if (idle && stopWhenIdle && !busy) stopSelfResult(startId) + if (deciding == session) deciding = null + if (verdict == true && current()) { + Log.i(TAG, "closed with nothing running: quitting") + synchronized(stateLock) { + running = false + releaseWakeLock() + } + exit() + } } - }, "turbollm-stop").start() + }, "turbollm-close").start() } - /** Stop the primary model through the daemon (POST /api/v1/engine/stop), freeing its memory. Best-effort. */ - private fun unloadModel(stillWanted: () -> Boolean) { - runCatching { - if (!stillWanted()) return - val conn = URL("http://127.0.0.1:$ANDROID_DAEMON_PORT/api/v1/engine/stop").openConnection() as HttpURLConnection - conn.requestMethod = "POST" - conn.doOutput = true - conn.setRequestProperty("Content-Type", "application/json") - conn.connectTimeout = 1000 - conn.readTimeout = 5000 - conn.outputStream.use { it.write("{}".toByteArray()) } - conn.responseCode - conn.disconnect() - }.onFailure { Log.w(TAG, "could not unload the model before stopping", it) } + /** Whether anything runs, asked of the daemon now (see appClosed): a model import, work in progress or a reply + * waiting for a tool approval (GET /api/v1/activity's items, engineGenerating or waiting), a model download + * running or queued (a paused or failed one does not count), or a model loaded or loading or a server feature on + * (queryStatus). A query that fails, or a reply it does not recognise, counts as something running. */ + private fun anythingRuns(): Boolean { + if (ModelImport.isRunning()) return true + val activity = queryActivity() ?: return true + if (activity.work != null || activity.waiting) return true + if (queryDownloads()?.isEmpty() != true) return true + return queryStatus()?.runs != false + } + + /** Ask the daemon to write its chat database to disk now (POST /api/v1/daemon/flush), waiting at most + * FLUSH_WAIT_MS. On Android the database is kept in memory (sql.js) and written out on the daemon's own schedule, + * so what it had not written yet when the process went was lost. Best-effort: a daemon without the route (404), + * one that does not answer, or a failed proof only ends the wait. The wait has a deadline of its own because + * DaemonAuth.open's proof alone may take up to 3 s against a port that does not answer. */ + private fun flushDatabase() { + val flush = Thread({ + runCatching { + val conn = DaemonAuth.open("/api/v1/daemon/flush") + try { + conn.requestMethod = "POST" + conn.doOutput = true + conn.setRequestProperty("Content-Type", "application/json") + conn.connectTimeout = 500 + conn.readTimeout = 1500 + conn.outputStream.use { it.write("{}".toByteArray()) } + val code = conn.responseCode + if (code != 200) Log.w(TAG, "the daemon did not write its database (HTTP $code)") + } finally { + conn.disconnect() + } + }.onFailure { Log.w(TAG, "could not ask the daemon to write its database", it) } + }, "turbollm-flush").apply { isDaemon = true } + flush.start() + runCatching { flush.join(FLUSH_WAIT_MS) } } private fun pollActivity() { @@ -203,26 +382,64 @@ class EngineService : Service() { while (running) { // A model import counts as work too: the service must neither stop nor let the CPU sleep under the copy. val importing = ModelImport.isRunning() - if (importing != this.importing) { - this.importing = importing - updateNotification() + // So does a model download (Discover): a stopped service or a sleeping CPU would stall a multi-GB transfer. + val downloads = queryDownloads() + val downloading = !downloads.isNullOrEmpty() + val activity = queryActivity() + // A poll that could not read the activity keeps its last work or approval line up for up to + // ACTIVITY_MISSES polls in a row, as the download lines do below: the daemon can be slow to answer for a + // moment mid-reply, and the line read "Idle" for that poll, while a close then counts the unreadable + // reply as something running. It still counts as idle for the wake lock, as before, so a daemon that + // stops answering never pins it. Only this thread writes `work` and `waiting`, so they are read here + // without the lock. + val shown = when { + activity != null -> activity.also { activityMisses = 0 } + ++activityMisses <= ACTIVITY_MISSES -> DaemonActivity(this.work, this.waiting) + else -> null } - val nowBusy = queryBusy() || importing + val work = shown?.work + val waiting = shown?.waiting == true + val nowBusy = activity?.work != null || importing || downloading + // The loaded model's name, for the line shown while nothing runs: asked only then, and kept from the last + // reading when the status cannot be read, so a slow answer does not drop the name for 3 s. + val status = if (work == null && !importing && !downloading && !waiting) queryStatus() else null + val model = if (status != null) status.name else this.model // The query can take seconds. Under the lock, so the service being destroyed meanwhile is seen before the // wake lock is re-taken or a removed notification is posted again. synchronized(stateLock) { if (!running) return + // A download's progress changes from one poll to the next, so the notification is posted again + // whenever its download lines would read differently, at the poll's pace (3 s) and quietly + // (setOnlyAlertOnce, setSilent). A poll that could not read the list keeps the last lines up for up + // to DOWNLOAD_MISSES polls in a row (the daemon slow to answer for a moment) instead of flashing + // another state for 3 s. It still counts as idle above, as before, so a daemon that stops answering + // never pins the wake lock. + val download = when { + downloads != null -> downloadNotice(downloads).also { downloadMisses = 0 } + ++downloadMisses <= DOWNLOAD_MISSES -> this.download + else -> null + } + // Everything the notification shows is updated first, then it is posted once, and only if any of it + // changed. Posted after each change, it showed the state in between: "Generating" for a moment at the + // end of every download or import, with the download gone and the busy state not yet updated. + val changed = importing != this.importing || download != this.download || work != this.work || + waiting != this.waiting || model != this.model + this.importing = importing + this.download = download + this.work = work + this.waiting = waiting + this.model = model if (nowBusy != busy) { busy = nowBusy if (nowBusy) wakeLock?.acquire(30 * 60 * 1000L) else releaseWakeLock() - updateNotification() } else if (nowBusy) { - wakeLock?.acquire(30 * 60 * 1000L) // renew the timeout for a generation that is still going + wakeLock?.acquire(30 * 60 * 1000L) // renew the timeout for work that is still going } + if (changed) updateNotification() } - if (stopWhenIdle && !nowBusy) { - ContextCompat.getMainExecutor(this).execute { if (stopWhenIdle && !busy) stopNow() } - } + // Nothing here stops the service: once the app was closed with something running, it stays until Quit or + // the next close with nothing running (appClosed), also once that work has ended, or the daemon's idle + // timer has unloaded the model. if (++polls % 20 == 0) DaemonLog.rotateIfLarge(filesDir) // about once a minute try { Thread.sleep(POLL_MS) @@ -232,25 +449,67 @@ class EngineService : Service() { } } - /** True while a chat, Code run, routine or API generation is in flight. Any failure (daemon still booting) = idle. */ - private fun queryBusy(): Boolean = try { - val conn = URL("http://127.0.0.1:$ANDROID_DAEMON_PORT/api/v1/activity").openConnection() as HttpURLConnection + /** What the daemon is doing (GET /api/v1/activity, see DaemonActivity), or null when it could not be read: the poll + * reads that as idle for the wake lock and keeps its last line for a few polls, a close (anythingRuns) counts it + * as something running. */ + private fun queryActivity(): DaemonActivity? = try { + val conn = DaemonAuth.open("/api/v1/activity") conn.connectTimeout = 1000 conn.readTimeout = 2000 try { if (conn.responseCode != 200) { - false + null } else { // Parsed, not string-matched: any shape this does not recognise reads as idle, so a format change - // can never pin the wake lock on with the screen off. - val json = JSONObject(conn.inputStream.bufferedReader().use { it.readText() }) - json.optBoolean("engineGenerating", false) || (json.optJSONArray("items")?.length() ?: 0) > 0 + // can never pin the wake lock on with the screen off. A reply with no `items` list is not the + // daemon's: unreadable, so a close does not take it for nothing running. + JSONObject(conn.inputStream.bufferedReader().use { it.readText() }) + .takeIf { it.optJSONArray("items") != null } + ?.let(::daemonActivity) + } + } finally { + conn.disconnect() + } + } catch (_: Exception) { + null + } + + /** What GET /api/v1/status says (see statusReading), or null when the status could not be read, which a close + * (anythingRuns) counts as running: it never quits the daemon on a guess. */ + private fun queryStatus(): StatusReading? = try { + val conn = DaemonAuth.open("/api/v1/status") + conn.connectTimeout = 1000 + conn.readTimeout = 2000 + try { + if (conn.responseCode != 200) { + null + } else { + statusReading(JSONObject(conn.inputStream.bufferedReader().use { it.readText() })) + } + } finally { + conn.disconnect() + } + } catch (_: Exception) { + null + } + + /** The daemon's running and queued model downloads (parseLiveDownloads), or null when the list could not be read + * (the daemon still booting, a reply this does not recognise). */ + private fun queryDownloads(): List? = try { + val conn = DaemonAuth.open("/api/v1/downloads") + conn.connectTimeout = 1000 + conn.readTimeout = 2000 + try { + if (conn.responseCode != 200) { + null + } else { + parseLiveDownloads(conn.inputStream.bufferedReader().use { it.readText() }) } } finally { conn.disconnect() } } catch (_: Exception) { - false + null } private fun releaseWakeLock() { @@ -272,17 +531,34 @@ class EngineService : Service() { Intent(this, EngineService::class.java).setAction(ACTION_QUIT), PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT, ) + val repost = PendingIntent.getService( + this, 2, + Intent(this, EngineService::class.java).setAction(ACTION_REPOST), + PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT, + ) val hot = thermal >= PowerManager.THERMAL_STATUS_SEVERE + // A download names its file (or the count) and shows its progress, unless an import, named first, is running. + val download = this.download?.takeUnless { importing } + val work = this.work + val model = this.model val text = when { importing -> "Importing models" - hot && busy -> "Generating. The phone is hot, so it is throttling speed" - hot -> "Running. The phone is hot, so generation will be slower" - busy -> "Generating" - else -> "Running in the background" + download != null -> download.text + hot && work != null -> "$work. The phone is hot, so it is throttling speed" + work != null -> work + // A reply parked on a tool approval waits for the user, in the app, not for the phone: say so. It holds + // no wake lock (see DaemonActivity). + waiting -> "Waiting for your approval" + // Nothing runs: "Idle", with the model still loaded, if any, by name (the system ellipsizes a long one). + // It read "Running in the background", which did not say whether anything was. Hot, the warning takes + // the line, as a long name would push it out of the one line the shade shows. + hot -> "Idle. The phone is hot, so generation will be slower" + model != null -> "Idle · $model" + else -> "Idle" } return NotificationCompat.Builder(this, CHANNEL_ID) .setSmallIcon(R.drawable.ic_stat_engine) - .setContentTitle("TurboLLM") + .setContentTitle(download?.title ?: "TurboLLM") .setContentText(text) .setOngoing(true) .setOnlyAlertOnce(true) @@ -290,15 +566,42 @@ class EngineService : Service() { .setCategory(NotificationCompat.CATEGORY_SERVICE) .setForegroundServiceBehavior(NotificationCompat.FOREGROUND_SERVICE_IMMEDIATE) .setContentIntent(open) + .setDeleteIntent(repost) .addAction(0, "Quit", quit) + .apply { + if (download != null) setProgress(DOWNLOAD_PROGRESS_MAX, download.progress, download.indeterminate) + } .build() } companion object { - /** The activity is finishing (Back): see ACTION_ACTIVITY_FINISHED. */ - fun activityFinished(context: Context) { + /** The number of the newest MainActivity instance (activityOpened), so a close from an older one is ignored + * (see ACTION_ACTIVITY_FINISHED). Set on the main thread, read by the close thread too. */ + @Volatile private var newestActivity = 0 + + /** MainActivity.onCreate: this activity's number, to hand back to activityFinished. */ + fun activityOpened(): Int = ++newestActivity + + /** The activity numbered [activity] is finishing (Back): see ACTION_ACTIVITY_FINISHED. */ + fun activityFinished(context: Context, activity: Int) { + runCatching { + val intent = Intent(context, EngineService::class.java).setAction(ACTION_ACTIVITY_FINISHED) + context.startService(intent.putExtra(EXTRA_ACTIVITY, activity)) + }.onFailure { Log.w(TAG, "could not reach the engine service", it) } + } + + /** How many times the app came on screen (repostNotification), so a Quit is called off when the app is brought + * back during its hold, also without a new activity (left with Home, not closed): see quit. Set on the main + * thread, read by the Quit's thread too. */ + @Volatile private var appShown = 0 + + /** The app came on screen: post the notification again (ACTION_REPOST). One dropped while notifications were + * off then shows as soon as the user allows them, at the first-launch prompt or in Settings, since Android + * re-posts nothing on a grant and returning to the app (singleTask, no recreation) does not run start(). */ + fun repostNotification(context: Context) { + appShown++ runCatching { - context.startService(Intent(context, EngineService::class.java).setAction(ACTION_ACTIVITY_FINISHED)) + context.startService(Intent(context, EngineService::class.java).setAction(ACTION_REPOST)) }.onFailure { Log.w(TAG, "could not reach the engine service", it) } } @@ -309,3 +612,63 @@ class EngineService : Service() { } } } + +/** What GET /api/v1/activity says the daemon is doing. [work] is the notification's line for what is in progress, null + * while nothing is; [waiting] is true while a chat waits for the user to answer a tool approval, which is not work + * (the daemon lists those chats under `waiting`, not `items`). */ +private class DaemonActivity(val work: String?, val waiting: Boolean) + +/** What GET /api/v1/status says. [runs]: something runs there, a model held or a server feature on (statusReading). + * [name]: the model the notification names while nothing runs, null when none is loaded or the loaded one is only + * an embedding model in a pool slot, which the status does not name. */ +private class StatusReading(val runs: Boolean, val name: String?) + +/** What GET /api/v1/status counts as running (see anythingRuns). A model is loaded while the primary engine is + * loading, running or still stopping (`engine.state` starting, running or stopping), or any model is alive in a pool + * slot beside it, such as an embedding model or a Jev or Laya model. `poolModels` counts those slots (running, + * starting or stopping); a daemon from before TurboLLM#287 leaves it out, which reads as 0, and names only a Jev or + * Laya model (`jev`, `laya`), so those are still read. An engine in `error` holds nothing. A server feature on counts + * too (daemonServes). The name is the primary's model while it holds one, else the Jev or Laya model's. Null for a + * reply with no `engine`, which is not the daemon's status: unreadable. */ +private fun statusReading(json: JSONObject): StatusReading? { + val engine = json.optJSONObject("engine") ?: return null + val primary = engine.optString("state") in LOADED_STATES + val jev = json.optJSONObject("jev") + val laya = json.optJSONObject("laya") + val loaded = primary || json.optInt("poolModels") > 0 || jev != null || laya != null + // opt, not optString: optString reads a JSON null as the text "null". + val name = listOfNotNull(json.optJSONObject("model")?.takeIf { primary }, jev, laya) + .firstNotNullOfOrNull { (it.opt("name") as? String)?.takeIf(String::isNotBlank) } + return StatusReading(loaded || daemonServes(json), name) +} + +/** A server feature is on (`serving` in GET /api/v1/status, TurboLLM's serving-status.ts): the daemon listens beyond + * loopback (`lan`, LAN access), remote access is up or coming up (`remote`: a tunnel, Tailscale Serve or Funnel), or + * it hosts Turbo Link peers (`linkPeers`, the link tokens it minted for other machines). Quitting would cut off the + * devices using it, so a close keeps the daemon up. A daemon from before that field leaves it out, which reads as + * nothing served, as does a `serving` that is not an object, a `lan` or `remote` that is not a boolean and a + * `linkPeers` that is not a number: only the daemon's own true or count says something is served. */ +private fun daemonServes(json: JSONObject): Boolean { + val serving = json.optJSONObject("serving") ?: return false + return serving.opt("lan") == true || serving.opt("remote") == true || + ((serving.opt("linkPeers") as? Number)?.toDouble() ?: 0.0) > 0 +} + +/** The work line comes from the items' `kind`. A model load is named first: nothing generates on that model until it + * is loaded, and a chat waiting for it is listed too. Then an auto-tune, which runs the engine itself. A chat, Code + * run or routine is "Generating", and so is a kind this does not know, or an item that is not an object, so new work + * still reads as work. A skill being written is named only when nothing else runs. An API client's generation has no + * item, only `engineGenerating`. */ +private fun daemonActivity(json: JSONObject): DaemonActivity { + val items = json.optJSONArray("items") + val kinds = (0 until (items?.length() ?: 0)).map { items?.optJSONObject(it)?.optString("kind").orEmpty() } + val work = when { + "load" in kinds -> "Loading a model" + "tune" in kinds -> "Auto-tuning" + kinds.any { it != "skill" } -> "Generating" + kinds.isNotEmpty() -> "Writing a skill" + json.optBoolean("engineGenerating", false) -> "Generating" + else -> null + } + return DaemonActivity(work, (json.optJSONArray("waiting")?.length() ?: 0) > 0) +} diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/MainActivity.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/MainActivity.kt index 6a38366..ec7542c 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/MainActivity.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/MainActivity.kt @@ -17,6 +17,7 @@ import androidx.activity.result.contract.ActivityResultContracts import java.io.File import java.io.FileInputStream import java.io.FileOutputStream +import java.security.SecureRandom import java.util.concurrent.atomic.AtomicBoolean private const val TAG = "TurboLLM" @@ -28,12 +29,25 @@ private const val TAG = "TurboLLM" * (below) and for the WebView's own URL (DaemonWebView.kt). */ const val ANDROID_DAEMON_PORT = 7000 +/** Every app on the phone shares 127.0.0.1, and the daemon trusts loopback callers, so any installed app could use + * its whole API. The daemon is started with this random secret (TURBOLLM_APP_SECRET, see the daemon's auth.ts) and + * then trusts only loopback requests carrying it: the WebView as the `turbollm_app` cookie (DaemonWebView.kt), native + * calls as the X-TurboLLM-App header. Neither is sent before the daemon proves it holds the port (DaemonAuth.kt). One + * per process, like Node itself (nodeStarted below), so every component in this process holds the value the running + * daemon was given. */ +val DAEMON_APP_SECRET: String = ByteArray(32).also { SecureRandom().nextBytes(it) } + .joinToString("") { (it.toInt() and 0xff).toString(16).padStart(2, '0') } + /** Node can start once per process: a second node::Start aborts it. The process outlives the activity (EngineService * keeps it running after Back), so a flag on the activity instance let a reopened app start Node again and crash. */ private val nodeStarted = AtomicBoolean(false) class MainActivity : ComponentActivity() { private lateinit var storageAccess: StorageAccess + /** From EngineService.activityOpened: the service ignores this activity's close once a newer one is open. */ + private var activityNumber = 0 + /** Set by finish(), so onDestroy tells Back from a swipe from Recents. */ + private var finishedByBack = false /** The system file picker for importing models (ModelImport). Any type: .gguf and .litertlm have no MIME type * providers agree on, so ModelImport checks the name instead. */ @@ -57,6 +71,9 @@ class MainActivity : ComponentActivity() { ) super.onCreate(savedInstanceState) recordCrashes() + // First, so a close the service is deciding on the last Back sees at once that the app is open again and + // quits nothing (EngineService.appClosed). + activityNumber = EngineService.activityOpened() // The daemon ships inside the APK and is unpacked here, so a Play Store install needs // nothing pushed to the device. Synchronous on purpose: the Node thread started below @@ -66,7 +83,8 @@ class MainActivity : ComponentActivity() { EngineService.start(this) requestNotificationPermission() storageAccess = StorageAccess(this) - // A model import interrupted by the process dying leaves a multi-GB .part behind; nothing else removes it. + // A model import interrupted by the process dying leaves a multi-GB .import-part behind; nothing else + // removes it. Only that suffix: a daemon download into the same folder keeps its .part there. Thread { ModelImport.sweepPartialFiles(this) }.start() if (nodeStarted.compareAndSet(false, true)) { @@ -97,6 +115,8 @@ class MainActivity : ComponentActivity() { "TURBOLLM_ANDROID_NATIVE_LIB_DIR" to applicationInfo.nativeLibraryDir, // Node's stdout/stderr, otherwise lost in an app process (native-lib.cpp). "TURBOLLM_DAEMON_LOG" to File(filesDir, "daemon.log").absolutePath, + // Loopback callers must present it (DAEMON_APP_SECRET). + "TURBOLLM_APP_SECRET" to DAEMON_APP_SECRET, ) + gpuEnv() + socEnv(), args = arrayOf("--no-telemetry", "--port", ANDROID_DAEMON_PORT.toString()), preloadScripts = arrayOf("${filesDir.absolutePath}/daemon/dist/android-preload.cjs"), @@ -120,6 +140,11 @@ class MainActivity : ComponentActivity() { override fun onResume() { super.onResume() + // Post the engine notification (and its Quit) again whenever the app comes on screen, the permission prompt + // closing and a return from Settings included. Android drops a post made without the permission for good and + // re-posts nothing on a grant, and the first post (startForeground, from onCreate) goes out before the user + // can answer the prompt. + EngineService.repostNotification(this) storageAccess.onResume() DaemonLog.rotateIfLarge(filesDir) } @@ -129,9 +154,22 @@ class MainActivity : ComponentActivity() { super.onPause() } + /** Called only for Back: DaemonWebView's BackHandlers call it, and before Android 12 so does the system's own Back + * while none is composed (the splash), through onBackPressed and finishAfterTransition; from Android 12 that Back + * moves the task to the back instead, which is no close. A swipe from Recents finishes the activity too, but from + * the system's side, without calling this. */ + override fun finish() { + finishedByBack = true + super.finish() + } + override fun onDestroy() { - // Back, not a configuration change (those no longer recreate the activity, see the manifest). - if (isFinishing) EngineService.activityFinished(this) + // Back, not a configuration change (those no longer recreate the activity, see the manifest), and not a swipe + // from Recents: the service hears that one as onTaskRemoved. Sent for a swipe too, it was a second close of + // the same swipe, decided afresh a moment later, so a swipe with a reply running quit the app if the reply + // ended in between (EngineService.appClosed). + // Android may run this only after the app was opened again, a new MainActivity included (activityNumber). + if (isFinishing && finishedByBack) EngineService.activityFinished(this, activityNumber) super.onDestroy() } @@ -145,8 +183,9 @@ class MainActivity : ComponentActivity() { return mapOf("TURBOLLM_ANDROID_SOC" to name) } - /** The engine notification carries the Quit action and the "phone is hot" warning. Asked once; Android stops - * prompting on its own after the user declines, and the service runs either way. */ + /** The engine notification carries the Quit action and the "phone is hot" warning. Asked each time the activity + * is created while it is not granted, with nothing kept to stop asking; Android stops showing the prompt on its + * own once the user has declined it twice. The service runs either way. */ private fun requestNotificationPermission() { if (Build.VERSION.SDK_INT < 33) return if (checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS) == PackageManager.PERMISSION_GRANTED) return @@ -157,12 +196,14 @@ class MainActivity : ComponentActivity() { * install works with nothing ever pushed to the device. * * This replaces a `/sdcard` copy that was only ever a dev helper. That approach cannot work - * for a real user twice over: the app declares no storage permission, and nothing would have - * put the files there anyway. It also hid a second, larger problem — the daemon bundle the - * npm build produces leaves `dependencies` external (npm installs them on a desktop), so a - * genuinely fresh install died at boot with `Cannot find package '@hono/node-server'`. The - * payload staged into assets is therefore built by a separate Android pass with its - * dependencies bundled in (turbollm/tsup.android.config.ts + scripts/stage-daemon-assets.mjs). + * for a real user twice over: the app holds no storage permission to read it with unless, on + * Android 10 or older, the user allowed one for a model folder in shared storage (none on 11+), + * and nothing would have put the files there anyway. It also hid a second, larger problem — + * the daemon bundle the npm build produces leaves `dependencies` external (npm installs them + * on a desktop), so a genuinely fresh install died at boot with `Cannot find package + * '@hono/node-server'`. The payload staged into assets is therefore built by a separate + * Android pass with its dependencies bundled in (turbollm/tsup.android.config.ts + + * scripts/stage-daemon-assets.mjs). * * Re-extracted whenever the APK changes, keyed on `lastUpdateTime`: an app update ships a new * daemon, and a stale copy in filesDir would silently win over it. Everything else in diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/ModelImport.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/ModelImport.kt index 4222c70..98397ba 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/ModelImport.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/ModelImport.kt @@ -16,8 +16,6 @@ import android.widget.TextView import android.widget.Toast import org.json.JSONObject import java.io.File -import java.net.HttpURLConnection -import java.net.URL import java.util.concurrent.atomic.AtomicBoolean private const val TAG = "TurboLLM" @@ -30,8 +28,11 @@ private const val TAG = "TurboLLM" * that folder itself in the app, see TurboLLM engines/seed.ts) and the engines can open by path. The cost is the * file's size again until the user deletes the original. * - * - Each file is written to `.part` and renamed only when complete, so an interrupted copy never shows up in the - * library. A `.part` left by a process that died mid-copy is swept at the next app start or import. + * - Each file is written to `.import-part` and renamed only when complete, so an interrupted copy never shows up + * in the library. One left by a process that died mid-copy is swept at the next app start or import. The suffix is + * the import's own because [modelsDir] is also one of the daemon's model folders: a download the daemon writes there + * (a non-Hugging Face Import from URL once the user makes this folder the primary one) keeps its partial file as + * `.part` at the top level, which may be live or paused when a sweep runs and must never be swept. * - Free space is checked first; a name already in use by a different file gets a numbered suffix, never overwritten. * - A partial wake lock is held for the copy, and EngineService counts a running import as busy, so leaving the app * neither freezes the copy nor stops the service under it. @@ -39,20 +40,23 @@ private const val TAG = "TurboLLM" object ModelImport { private val MODEL_FILE = Regex("""\.(gguf|litertlm)$""", RegexOption.IGNORE_CASE) private val SPLIT_PART = Regex("""^(.*)-(\d{5})-of-(\d{5})\.gguf$""", RegexOption.IGNORE_CASE) + /** Not `.part`: that is the daemon's download suffix, in the same folder (see the class comment). */ + private const val IMPORT_PART = ".import-part" private val running = AtomicBoolean(false) fun isRunning(): Boolean = running.get() fun modelsDir(context: Context) = File(context.filesDir, "models") - /** Delete `*.part` files: only ever left behind by a copy whose process died, since a live import renames or - * deletes its own. Safe whenever no import is running. */ + /** Delete `*.import-part` files: only ever left behind by a copy whose process died, since a live import renames or + * deletes its own. Safe whenever no import is running. A daemon download's `*.part` here is left alone: deleting + * it under a running download made its final rename fail with ENOENT, and a paused one resume from byte 0. */ fun sweepPartialFiles(context: Context) { if (!running.get()) sweep(modelsDir(context)) } private fun sweep(dir: File) { - dir.listFiles { f -> f.isFile && f.name.endsWith(".part") }?.forEach { f -> + dir.listFiles { f -> f.isFile && f.name.endsWith(IMPORT_PART) }?.forEach { f -> if (f.delete()) Log.i(TAG, "removed an interrupted import: ${f.name}") } } @@ -90,7 +94,7 @@ object ModelImport { try { wakeLock.acquire(3 * 60 * 60 * 1000L) val dest = modelsDir(app).apply { mkdirs() } - sweep(dest) // this import has written nothing yet, so any .part here is a dead one + sweep(dest) // this import has written nothing yet, so any .import-part here is a dead one for ((i, uri) in uris.withIndex()) { if (cancelled.get()) break var label = uri.lastPathSegment ?: "file ${i + 1}" @@ -114,7 +118,7 @@ object ModelImport { } val target = freeName(dest, name) main.post { status.text = "${i + 1} of ${uris.size}: $name"; bar.progress = 0 } - part = File(dest, "${target.name}.part") + part = File(dest, "${target.name}$IMPORT_PART") copyStream(app, uri, part, size, cancelled) { p -> main.post { bar.progress = p } } if (!part.renameTo(target)) throw IllegalStateException("could not finish writing the file") part = null @@ -222,7 +226,7 @@ object ModelImport { /** Add the folder to the daemon's model folders (a no-op when it is there already); that request also rescans. * False when the daemon did not accept it, so the caller says so instead of reporting a silent success. */ private fun registerWithDaemon(dir: File): Boolean = runCatching { - val conn = URL("http://127.0.0.1:$ANDROID_DAEMON_PORT/api/v1/modeldirs").openConnection() as HttpURLConnection + val conn = DaemonAuth.open("/api/v1/modeldirs") try { conn.requestMethod = "POST" conn.doOutput = true diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/RendererCrashes.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/RendererCrashes.kt new file mode 100644 index 0000000..41ea5f9 --- /dev/null +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/RendererCrashes.kt @@ -0,0 +1,27 @@ +package com.turbollm.app + +/** Renderer crashes within RENDERER_CRASH_WINDOW_MS after which the WebView is not built again by itself + * (RendererCrashes). */ +internal const val RENDERER_CRASHES = 3 + +internal const val RENDERER_CRASH_WINDOW_MS = 60_000L + +/** Decides, at each death of the WebView's renderer (DaemonWebView.kt, onRenderProcessGone), whether the screen builds + * a fresh WebView. A renderer killed for memory (didCrash false: Android reclaiming it, most often with the app in the + * background) is always followed by a fresh WebView, and is not counted: it says nothing about the page. One that + * crashed is too, unless it is the RENDERER_CRASHES-th crash within RENDERER_CRASH_WINDOW_MS. A page that crashes its + * renderer each time it loads was otherwise built, loaded and crashed again in a loop for as long as the app was on + * screen, a new renderer process each round; the screen shows its Reload screen instead. Reload clears nothing: one + * more crash inside the window brings that screen back at once. Main thread only. */ +internal class RendererCrashes { + /** When the crashes still inside the window happened (SystemClock.elapsedRealtime), oldest first. */ + private val times = ArrayDeque() + + /** One renderer death at [now] (ms): true to build a fresh WebView, false to show the Reload screen. */ + fun rebuild(didCrash: Boolean, now: Long): Boolean { + if (!didCrash) return true + while (times.isNotEmpty() && now - times.first() >= RENDERER_CRASH_WINDOW_MS) times.removeFirst() + times.addLast(now) + return times.size < RENDERER_CRASHES + } +} diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/StorageAccess.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/StorageAccess.kt index 82282f0..f043632 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/StorageAccess.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/StorageAccess.kt @@ -11,8 +11,6 @@ import android.os.Looper import android.util.Log import org.json.JSONObject import java.io.File -import java.net.HttpURLConnection -import java.net.URL private const val TAG = "TurboLLM" private const val CHECK_MS = 2000L @@ -135,7 +133,7 @@ class StorageAccess(private val activity: Activity) { private fun rescan() { runCatching { - val conn = URL("http://127.0.0.1:$ANDROID_DAEMON_PORT/api/v1/models/rescan").openConnection() as HttpURLConnection + val conn = DaemonAuth.open("/api/v1/models/rescan") conn.requestMethod = "POST" conn.connectTimeout = 2000 conn.readTimeout = 5000 diff --git a/composeApp/src/androidMain/res/xml/backup_rules.xml b/composeApp/src/androidMain/res/xml/backup_rules.xml index 18b819d..2e8fb82 100644 --- a/composeApp/src/androidMain/res/xml/backup_rules.xml +++ b/composeApp/src/androidMain/res/xml/backup_rules.xml @@ -1,5 +1,6 @@ - + @@ -7,4 +8,17 @@ + + + + + + + + + + diff --git a/composeApp/src/androidMain/res/xml/data_extraction_rules.xml b/composeApp/src/androidMain/res/xml/data_extraction_rules.xml index 208d6ac..863d428 100644 --- a/composeApp/src/androidMain/res/xml/data_extraction_rules.xml +++ b/composeApp/src/androidMain/res/xml/data_extraction_rules.xml @@ -1,5 +1,8 @@ - + @@ -8,13 +11,36 @@ + + + + + + + + + + - + + + + + + + + + + diff --git a/composeApp/src/commonMain/kotlin/com/turbollm/app/DaemonClient.kt b/composeApp/src/commonMain/kotlin/com/turbollm/app/DaemonClient.kt index 4cec0ee..9268b4a 100644 --- a/composeApp/src/commonMain/kotlin/com/turbollm/app/DaemonClient.kt +++ b/composeApp/src/commonMain/kotlin/com/turbollm/app/DaemonClient.kt @@ -5,8 +5,6 @@ import io.ktor.client.call.body import io.ktor.client.plugins.HttpTimeout import io.ktor.client.plugins.contentnegotiation.ContentNegotiation import io.ktor.client.request.get -import io.ktor.client.statement.HttpResponse -import io.ktor.http.isSuccess import io.ktor.serialization.kotlinx.json.json import kotlinx.serialization.Serializable import kotlinx.serialization.json.Json @@ -42,18 +40,4 @@ class DaemonClient { } catch (e: Exception) { null } - - /** Plain reachability check against the REAL daemon's own status endpoint — deliberately - * ignores the response body/shape (unlike `status()`, which expects Spike A's toy JSON - * shape at `/`) since all this needs to know is "has the HTTP server come up yet," to - * gate showing the WebView pointed at the daemon's real web UI. `port` is caller-supplied - * (not hardcoded like `status()` above) since Android runs the daemon on its own port — - * see ANDROID_DAEMON_PORT. */ - suspend fun isDaemonReachable(port: Int): Boolean = - try { - val response: HttpResponse = client.get("http://127.0.0.1:$port/api/v1/status") - response.status.isSuccess() - } catch (e: Exception) { - false - } } diff --git a/engines/litert-lm-server/constrained-decoding.test.mjs b/engines/litert-lm-server/constrained-decoding.test.mjs new file mode 100644 index 0000000..fd8a2f1 --- /dev/null +++ b/engines/litert-lm-server/constrained-decoding.test.mjs @@ -0,0 +1,180 @@ +// Tests which conversations the LiteRT-LM server asks the runtime for, and what it answers when one cannot be made +// (src/main.cc, HandleChatCompletions): +// node --test engines/litert-lm-server/constrained-decoding.test.mjs +// The server is built for this machine with c++, as in parent-death.test.mjs (testdata/harness.mjs), and runs against +// testdata/fake_litert_lm.cc, a stand-in liblitert-lm.so that answers the conversation calls as LiteRT-LM 0.18.0 does +// for an LFM2 model, for Gemma models and for a few failures, and whose reply says how its conversation was created. +// Linux only, like that test, and skipped without a C and a C++ compiler. Building takes a few seconds. +// +// Given a liblitert-lm.so that runs on this machine (the one in the manylinux litert-lm-api wheel of the version +// scripts/stage-litert-lm-engine.mjs pins) and .litertlm models, the last tests also run the server on each model: +// LITERT_LM_TEST_LIBRARY= LITERT_LM_TEST_MODELS=[,...] \ +// node --test engines/litert-lm-server/constrained-decoding.test.mjs +// (an LFM2 one: litert-community/LFM2.5-230M on Hugging Face; a Gemma one: litert-community/gemma-3-270m-it). +// Every server a test starts is killed by pid when that test ends, or at the latest when the tests end. Every wait is +// bounded (the compiler check, the build, a server's start, each request, each test), so a server that hangs fails the +// test by name instead of holding the run. +import assert from 'node:assert/strict' +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { after, before, test } from 'node:test' +import { buildServer, chat, killServers, logged, realLibrary, realModels, skip, skipReal, startServer, stopServer } from './testdata/harness.mjs' + +const work = mkdtempSync(join(tmpdir(), 'litert-lm-server-cd-test-')) +// What before() is still doing. With a --test-name-pattern that leaves no test in this file, node runs after() while +// before() is starting the server, and removing `work` under it failed the file (spawn ... ENOENT); so after() waits. +let ready = Promise.resolve() +after(async () => { + await ready.catch(() => {}) + killServers() + rmSync(work, { recursive: true, force: true }) +}) + +// One file per kind of model the stand-in knows (see testdata/fake_litert_lm.cc); it reads only the name. None starts +// with "LITERTLM", so the server leaves it to the runtime. +const model = (kind) => { + const path = join(work, `${kind}.litertlm`) + writeFileSync(path, 'stand-in model') + return path +} + +let binary +let fake + +before(() => (ready = (async () => { + if (skip) return + binary = buildServer(work) + // --verbose for logged(). + fake = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: '' }, ['--verbose']) +})()), { timeout: 300_000 }) + +// The stand-in's reply: how the conversation was created and sent, and how many conversations the process has created. +function created(reply) { + assert.equal(reply.status, 200, reply.error) + return Object.fromEntries(reply.content.split(' ').map((kv) => kv.split('=')).map(([k, v]) => [k, Number(v)])) +} +// The same without the count. +function how(reply) { + const { creates, ...rest } = created(reply) + return rest +} + +const hello = [{ role: 'user', content: 'Hello' }] +const tools = [{ + type: 'function', + function: { name: 'get_weather', description: 'Weather in a city', parameters: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'] } }, +}] +const jsonSchema = { + type: 'json_schema', + json_schema: { name: 'city', schema: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'] } }, +} +const notSupported = 'constrained decoding is not available for' + +test('a chat with an LFM2 model is answered, streamed or not, without constrained decoding', { skip, timeout: 60_000 }, async () => { + const lfm2 = model('lfm2-a') + assert.deepEqual(how(await chat(fake, { model: lfm2, messages: hello })), { constrained: 0, tools: 0, provider: 0, constraint: 0 }) + const streamed = await chat(fake, { model: lfm2, messages: hello, stream: true }) + assert.equal(streamed.status, 200) + assert.equal(streamed.finish, 'stop') + assert.match(streamed.content, /^constrained=0 tools=0 provider=0 constraint=0 /) +}) + +test('constrained decoding is turned on for a request with tools, and only for one', { skip, timeout: 60_000 }, async () => { + const gemma = model('gemma') + assert.equal(created(await chat(fake, { model: gemma, messages: hello, tools })).constrained, 1) + assert.equal(created(await chat(fake, { model: gemma, messages: hello })).constrained, 0) + assert.equal(created(await chat(fake, { model: gemma, messages: hello, response_format: jsonSchema })).constrained, 0) + assert.equal((await logged(fake)).some((l) => l.startsWith(notSupported)), false) +}) + +test('a request with tools to a model that refuses constrained decoding is answered without it, found out once per load', { skip, timeout: 60_000 }, async () => { + for (const kind of ['lfm2-b', 'fst-unsupported']) { + const path = model(kind) + const before = created(await chat(fake, { model: path, messages: hello })).creates + const first = created(await chat(fake, { model: path, messages: hello, tools })) + assert.deepEqual({ ...first, creates: first.creates - before }, + { constrained: 0, tools: 1, provider: 0, constraint: 0, creates: 2 }, 'refused, then created without it') + const second = created(await chat(fake, { model: path, messages: hello, tools })) + assert.deepEqual({ ...second, creates: second.creates - first.creates }, + { constrained: 0, tools: 1, provider: 0, constraint: 0, creates: 1 }, 'created without it at once') + const lines = (await logged(fake)).filter((l) => l.startsWith(`${notSupported} ${kind}.litertlm`)) + assert.equal(lines.length, 1, fake.stderr) + } + const lines = await logged(fake) + assert.ok(lines.includes('constrained decoding is not available for lfm2-b.litertlm (FAILED_PRECONDITION: Constrained ' + + 'decoding is not supported for Lfm2DataProcessor.); its tool calls are not constrained'), fake.stderr) + assert.ok(lines.includes('constrained decoding is not available for fst-unsupported.litertlm (INTERNAL: Failed to ' + + 'create GemmaModelConstraintProvider.); its tool calls are not constrained'), fake.stderr) + // Found out again once the model is loaded again (the file may since hold another model). + const lfm2 = model('lfm2-b') + await chat(fake, { model: model('gemma'), messages: hello }) + assert.equal(created(await chat(fake, { model: lfm2, messages: hello, tools })).constrained, 0) + assert.equal((await logged(fake)).filter((l) => l.startsWith(`${notSupported} lfm2-b.litertlm`)).length, 2) +}) + +test('a response_format is followed by a model that refuses constrained decoding', { skip, timeout: 60_000 }, async () => { + const lfm2 = model('lfm2-c') + assert.deepEqual(how(await chat(fake, { model: lfm2, messages: hello, response_format: jsonSchema })), + { constrained: 0, tools: 0, provider: 1, constraint: 2 }) + assert.deepEqual(how(await chat(fake, { model: lfm2, messages: hello, tools, response_format: { type: 'regex', regex: 'yes|no' } })), + { constrained: 0, tools: 1, provider: 1, constraint: 1 }) +}) + +test('a response_format the model cannot follow is a 400 that says why', { skip, timeout: 60_000 }, async () => { + const noLlg = model('no-llg') + for (const extra of [{}, { tools }]) { + const reply = await chat(fake, { model: noLlg, messages: hello, response_format: { type: 'regex', regex: 'yes|no' }, ...extra }) + assert.equal(reply.status, 400) + assert.equal(reply.error, 'Invalid response_format parameters: this model cannot follow a response_format: ' + + "INVALID_ARGUMENT: LlGuidanceConfig::eos_id wasn't set and no valid stop token was found in SessionConfig.") + } + assert.equal(created(await chat(fake, { model: noLlg, messages: hello, tools })).constrained, 1) +}) + +test('a response_format whose regex does not compile is a 400 that says why', { skip, timeout: 60_000 }, async () => { + for (const kind of ['gemma', 'lfm2-d']) { + const reply = await chat(fake, { model: model(kind), messages: hello, response_format: { type: 'regex', regex: '(' } }) + assert.equal(reply.status, 400) + assert.equal(reply.error, 'Invalid response_format parameters: INTERNAL: Failed to create LLGuidance constraint: ' + + 'invalid regex "(": unclosed group') + } +}) + +test('a conversation that cannot be created at all is still a 500 with the runtime\'s reason', { skip, timeout: 60_000 }, async () => { + const broken = model('broken') + for (const extra of [{}, { tools }, { response_format: jsonSchema }, { tools, response_format: jsonSchema }]) { + const reply = await chat(fake, { model: broken, messages: hello, ...extra }) + assert.deepEqual(reply, { status: 500, error: 'RuntimeError: Failed to create conversation: INTERNAL: cannot create a session' }) + } + assert.equal((await logged(fake)).some((l) => l.startsWith(`${notSupported} broken.litertlm`)), false) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model answers a chat, a stream, tools and a response_format: ${path}`, { skip: skipReal, timeout: 600_000 }, async () => { + // No compiled-model cache: by default the runtime writes one beside the model, as large as the model. + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const real = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ms = 300_000 // per request: the first one loads the model + const ref = `${path},cpu` + const plain = await chat(real, { model: ref, messages: [{ role: 'user', content: 'Say hello.' }], max_completion_tokens: 16 }, ms) + assert.equal(plain.status, 200, plain.error) + assert.ok(plain.content || plain.reasoning, 'no text') // a thinking model (Qwen3) may only think in 16 tokens + const streamed = await chat(real, { model: ref, messages: [{ role: 'user', content: 'Count to three.' }], max_completion_tokens: 16, stream: true }, ms) + assert.equal(streamed.status, 200, streamed.error) + assert.ok(streamed.content || streamed.reasoning, 'no text streamed') + const withTools = await chat(real, { model: ref, messages: [{ role: 'user', content: 'What is the weather in Paris?' }], tools, max_completion_tokens: 48 }, ms) + assert.equal(withTools.status, 200, withTools.error) + const yesNo = await chat(real, { model: ref, messages: [{ role: 'user', content: 'Is the sky blue? Answer yes or no.' }], response_format: { type: 'regex', regex: '(yes|no)' }, max_completion_tokens: 8 }, ms) + assert.equal(yesNo.status, 200, yesNo.error) + assert.match(yesNo.content, /^(yes|no)$/) + const badRegex = await chat(real, { model: ref, messages: hello, response_format: { type: 'regex', regex: '(' }, max_completion_tokens: 8 }, ms) + assert.equal(badRegex.status, 400) + assert.match(badRegex.error, /^Invalid response_format parameters: INTERNAL: Failed to create LLGuidance constraint: /) + } finally { + stopServer(real) // now, not when the file ends: each one holds its model in memory + } + }) +} diff --git a/engines/litert-lm-server/model-settings.test.mjs b/engines/litert-lm-server/model-settings.test.mjs new file mode 100644 index 0000000..4f75b63 --- /dev/null +++ b/engines/litert-lm-server/model-settings.test.mjs @@ -0,0 +1,1085 @@ +// Tests what the LiteRT-LM server asks the runtime for: what each model itself needs and supports, and the settings a +// request makes (src/main.cc): +// node --test engines/litert-lm-server/model-settings.test.mjs +// The server is built for this machine with c++ (testdata/harness.mjs) and runs against testdata/fake_litert_lm.cc, a +// stand-in liblitert-lm.so that logs what it is asked for (FAKE_LITERT_LM_CALLS). Linux only, and skipped without a C +// and a C++ compiler. Building takes a few seconds. +// +// Given a liblitert-lm.so that runs on this machine (the one in the manylinux litert-lm-api wheel of the version +// scripts/stage-litert-lm-engine.mjs pins) and .litertlm models, the tests named "a real model ..." also run the +// server on each model (see constrained-decoding.test.mjs): +// LITERT_LM_TEST_LIBRARY= LITERT_LM_TEST_MODELS=[,...] \ +// node --test engines/litert-lm-server/model-settings.test.mjs +// (an LFM2 one: litert-community/LFM2.5-230M on Hugging Face; a Gemma one: litert-community/gemma-3-270m-it). +// LITERT_LM_TEST_VISION_MODELS (the same form) names models that take images (litert-community/LFM2.5-VL-450M), and +// LITERT_LM_TEST_THINKING_MODELS models whose template switches thinking on and off (litert-community/Qwen3-0.6B). +// Every server a test starts is killed by pid when that test ends, or at the latest when the tests end. Every wait is +// bounded. +import assert from 'node:assert/strict' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { basename, join } from 'node:path' +import { deflateSync } from 'node:zlib' +import { after, before, test } from 'node:test' +import { + buildServer, calls, chat, fb, flatbuffer, killServers, litertlmFile, logged, proto, realLibrary, realModels, request, + skip, skipReal, startServer, stopServer, +} from './testdata/harness.mjs' + +const visionModels = (process.env.LITERT_LM_TEST_VISION_MODELS ?? '').split(',').filter(Boolean) +const thinkingModels = (process.env.LITERT_LM_TEST_THINKING_MODELS ?? '').split(',').filter(Boolean) +const skipThinking = skip || (!realLibrary || !thinkingModels.length + ? 'set LITERT_LM_TEST_LIBRARY and LITERT_LM_TEST_THINKING_MODELS to run the server on models that think' : false) +const skipVision = skip || (!realLibrary || !visionModels.length + ? 'set LITERT_LM_TEST_LIBRARY and LITERT_LM_TEST_VISION_MODELS to run the server on models that take images' : false) + +const work = mkdtempSync(join(tmpdir(), 'litert-lm-server-ms-test-')) +// What before() is still doing. With a --test-name-pattern that leaves no test in this file, node runs after() while +// before() is starting the server, and removing `work` under it failed the file (spawn ... ENOENT); so after() waits. +let ready = Promise.resolve() +after(async () => { + await ready.catch(() => {}) + killServers() + rmSync(work, { recursive: true, force: true }) +}) + +const callLog = join(work, 'calls.jsonl') +let binary +let fake + +before(() => (ready = (async () => { + if (skip) return + binary = buildServer(work) + fake = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: '', FAKE_LITERT_LM_CALLS: callLog }, ['--verbose']) +})()), { timeout: 300_000 }) + +// The calls the stand-in logged while `fn` ran. +async function callsDuring(fn) { + const before = calls(callLog).length + await fn() + return calls(callLog).slice(before) +} + +const hello = [{ role: 'user', content: 'Hello' }] + +// The header sections of the stand-in files (runtime/proto/llm_metadata.proto, executor_metadata.proto). +const llmMetadata = (modelType) => ({ type: 5, bytes: proto.field(6, proto.field(modelType, [])) }) // llm_model_type +const stateBuffer = (name, type) => proto.field(2, [proto.field(1, name), proto.varint(6, type)]) +const executorMetadata = (...buffers) => ({ type: 9, bytes: proto.field(1, buffers) }) +const GENERIC = 1 +const LFM2 = 11 + +test('a model whose decoder keeps a recurrent state gets a new engine for each conversation, and only such a model', { skip, timeout: 60_000 }, async () => { + const models = { + // llm_model_type lfm2, as litert-community's LFM2 files have since 2026-10-14 + 'lfm2-typed': [llmMetadata(LFM2)], + // generic_model, with LFM2's conv caches declared (litert-community's LFM2.5-230M before 2026-10-14) + 'conv-generic': [llmMetadata(GENERIC), executorMetadata(stateBuffer('kv_cache_c_0', 5), stateBuffer('kv_cache_k_1', 1), stateBuffer('kv_cache_v_1', 2))], + // a linear-attention state under another name + 'linear-attention': [executorMetadata(stateBuffer('state_0', 5), stateBuffer('kv_cache_k_1', 1))], + // a KV cache only, global and local + 'kv-only': [llmMetadata(GENERIC), executorMetadata(stateBuffer('kv_cache_k_0', 1), stateBuffer('kv_cache_v_0', 2), stateBuffer('kv_cache_k_1', 3))], + // no header the server can read: left to the runtime, as before + 'no-header': null, + } + for (const [name, sections] of Object.entries(models)) { + const path = join(work, `${name}.litertlm`) + if (sections) litertlmFile(path, sections) + else writeFileSync(path, 'stand-in model') + const engines = [] + for (let i = 0; i < 3; i++) { + const log = await callsDuring(async () => assert.equal((await chat(fake, { model: path, messages: hello })).status, 200)) + engines.push(log.filter((c) => c.call === 'engine_create').length) + } + const recurrent = !['kv-only', 'no-header'].includes(name) + assert.deepEqual(engines, recurrent ? [1, 1, 1] : [1, 0, 0], name) + const said = (await logged(fake)).filter((l) => l.startsWith(`${name}.litertlm keeps a recurrent state`)) + assert.equal(said.length, recurrent ? 1 : 0, fake.stderr) + } + // The reason is given once per model, not once per load. + await chat(fake, { model: join(work, 'kv-only.litertlm'), messages: hello }) + for (let i = 0; i < 2; i++) await chat(fake, { model: join(work, 'lfm2-typed.litertlm'), messages: hello }) + assert.equal((await logged(fake)).filter((l) => l.startsWith('lfm2-typed.litertlm keeps a recurrent state')).length, 1) +}) + +test('a model whose decoder keeps a recurrent state still finds out once per load that it refuses constrained decoding', { skip, timeout: 60_000 }, async () => { + const path = litertlmFile(join(work, 'lfm2-tools.litertlm'), [llmMetadata(LFM2)]) + const tools = [{ type: 'function', function: { name: 'f', parameters: { type: 'object', properties: {} } } }] + for (let i = 0; i < 3; i++) assert.equal((await chat(fake, { model: path, messages: hello, tools })).status, 200) + const lines = (await logged(fake)).filter((l) => l.startsWith('constrained decoding is not available for lfm2-tools.litertlm')) + assert.equal(lines.length, 1, fake.stderr) +}) + +// The probe: a pirate persona, a question, a title request, then the question's second turn. +const pirate = [{ role: 'system', content: 'You are a pirate. Always talk like a pirate.' }, { role: 'user', content: 'Tell me about the sea.' }] +const t1 = [{ role: 'system', content: 'Answer in one word.' }, { role: 'user', content: 'What is the capital of France?' }] +const title = [ + { role: 'system', content: 'Write a short title for this conversation. Reply with the title only.' }, + { role: 'user', content: 'User: What is the capital of France?\nAssistant: Paris' }, +] +const t2 = [...t1, { role: 'assistant', content: 'Paris' }, { role: 'user', content: 'And of Italy?' }] + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model answers a conversation as it would on a fresh engine, whatever came before: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') // no compiled-model cache beside the model + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const env = { ...process.env, LITERT_LM_LIBRARY: realLibrary } + const ms = 300_000 + // A seed, so that a model that samples answers the same twice (each conversation seeds its own generator). + const ask = (server, messages) => chat(server, { model: `${path},cpu`, messages, seed: 7, max_completion_tokens: 32 }, ms) + const fresh = {} + for (const [name, messages] of Object.entries({ t1, t2 })) { + const server = await startServer(binary, env, ['--config', config]) + try { + fresh[name] = await ask(server, messages) + assert.equal(fresh[name].status, 200, fresh[name].error) + } finally { + stopServer(server) + } + } + const server = await startServer(binary, env, ['--config', config]) + try { + for (const messages of [pirate, t1]) assert.equal((await ask(server, messages)).status, 200) + const again = await ask(server, t1) + assert.equal(again.content, fresh.t1.content, `${basename(path)}: the same question asked again`) + assert.equal((await ask(server, title)).status, 200) + const second = await ask(server, t2) + assert.equal(second.content, fresh.t2.content, `${basename(path)}: the second turn after a title request`) + } finally { + stopServer(server) + } + }) +} + +// A 64x64 red PNG. +const redPng = (() => { + const crcTable = Array.from({ length: 256 }, (_, n) => { + let c = n + for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1 + return c >>> 0 + }) + const crc = (b) => { let c = 0xffffffff; for (const x of b) c = crcTable[(c ^ x) & 255] ^ (c >>> 8); return (c ^ 0xffffffff) >>> 0 } + const chunk = (type, data) => { + const len = Buffer.alloc(4); len.writeUInt32BE(data.length) + const body = Buffer.concat([Buffer.from(type), data]) + const sum = Buffer.alloc(4); sum.writeUInt32BE(crc(body)) + return Buffer.concat([len, body, sum]) + } + const ihdr = Buffer.alloc(13); ihdr.writeUInt32BE(64, 0); ihdr.writeUInt32BE(64, 4); ihdr.set([8, 2, 0, 0, 0], 8) + const rows = Buffer.concat(Array.from({ length: 64 }, () => Buffer.concat([Buffer.from([0]), Buffer.alloc(192, Buffer.from([255, 0, 0]))]))) + return Buffer.concat([Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]), chunk('IHDR', ihdr), chunk('IDAT', deflateSync(rows)), chunk('IEND', Buffer.alloc(0))]) +})() +const imageMessage = (text) => [{ role: 'user', content: [ + { type: 'image_url', image_url: { url: `data:image/png;base64,${redPng.toString('base64')}` } }, + { type: 'text', text }, +] }] +const audioMessage = [{ role: 'user', content: [{ type: 'input_audio', input_audio: { data: 'AAAA', format: 'wav' } }, { type: 'text', text: 'What is this?' }] }] + +// A stand-in file whose header lists `sections`: by default those `kinds` gives for its name (below), else a text +// decoder alone. TFLite sections have their model_type and, when given, their backend_constraint, as LiteRT-LM's +// builder writes them. The server reads the header; the stand-in library reads only the file's name. +const tflite = (model_type, backend_constraint) => ({ type: 3, items: { model_type, ...(backend_constraint ? { backend_constraint } : {}) } }) +const decoder = tflite('tf_lite_prefill_decode') +const readable = (name, sections = kinds[name] ?? [decoder]) => litertlmFile(join(work, `${name}.litertlm`), sections) +// One with no header the server can read, which is left to the runtime. +const unreadable = (name) => { + const path = join(work, `${name}.litertlm`) + writeFileSync(path, 'stand-in model') + return path +} +const kinds = { + text: [decoder], + vl: [decoder, tflite('tf_lite_embedder'), tflite('tf_lite_vision_encoder'), tflite('tf_lite_vision_adapter')], // LFM2.5-VL-450M's + 'gpu-vision': [decoder, tflite('tf_lite_vision_encoder', 'gpu')], + audio: [decoder, tflite('tf_lite_audio_encoder_hw', 'cpu')], + omni: [decoder, tflite('tf_lite_vision_encoder', 'cpu, GPU'), tflite('tf_lite_audio_adapter', 'cpu')], +} +const settingsOf = (log) => log.filter((c) => c.call === 'engine_settings_create').map(({ backend, vision_backend, audio_backend }) => ({ backend, vision_backend, audio_backend })) + +test('a model that takes images or audio gets its encoders, on the backend it prefers, and a model that does not gets none', { skip, timeout: 60_000 }, async () => { + const expected = { + text: { backend: 'cpu', vision_backend: null, audio_backend: null }, + vl: { backend: 'cpu', vision_backend: 'cpu', audio_backend: null }, + 'gpu-vision': { backend: 'cpu', vision_backend: 'gpu', audio_backend: null }, + audio: { backend: 'cpu', vision_backend: null, audio_backend: 'cpu' }, + omni: { backend: 'cpu', vision_backend: 'cpu', audio_backend: 'cpu' }, + unreadable: { backend: 'cpu', vision_backend: null, audio_backend: null }, + } + for (const [name, settings] of Object.entries(expected)) { + const path = name === 'unreadable' ? unreadable(name) : readable(name) + const log = await callsDuring(async () => assert.equal((await chat(fake, { model: `${path},cpu`, messages: hello })).status, 200)) + assert.deepEqual(settingsOf(log), [settings], name) + } +}) + +test('the config\'s vision_backend and audio_backend are used for a model that takes that input, and said to be unused otherwise', { skip, timeout: 60_000 }, async () => { + const config = join(work, 'encoders.json') + writeFileSync(config, JSON.stringify({ + default: { audio_backend: 'gpu' }, + models: { 'vl-configured.litertlm': { vision_backend: 'GPU' }, 'text-configured.litertlm': { vision_backend: 'gpu' } }, + })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: '', FAKE_LITERT_LM_CALLS: callLog }, ['--config', config, '--verbose']) + try { + const settings = async (name) => settingsOf(await callsDuring(async () => { + assert.equal((await chat(server, { model: readable(name, kinds[name.split('-')[0]]), messages: hello })).status, 200) + })) + assert.deepEqual(await settings('vl-configured'), [{ backend: 'cpu', vision_backend: 'gpu', audio_backend: null }]) + assert.deepEqual(await settings('text-configured'), [{ backend: 'cpu', vision_backend: null, audio_backend: null }]) + assert.deepEqual(await settings('omni-configured'), [{ backend: 'cpu', vision_backend: 'cpu', audio_backend: 'gpu' }]) + const lines = await logged(server) + assert.ok(lines.includes('vision_backend in the config is not used for text-configured.litertlm: it takes no vision input'), server.stderr) + assert.ok(lines.includes('audio_backend in the config is not used for vl-configured.litertlm: it takes no audio input'), server.stderr) + } finally { + stopServer(server) + } +}) + +test('an image or audio part sent to a model without that input is a 400 that says so, before the model loads', { skip, timeout: 60_000 }, async () => { + const text = readable('text-only') + for (const [messages, input] of [[imageMessage('What color is this?'), 'image'], [audioMessage, 'audio']]) { + for (const stream of [false, true]) { + const log = await callsDuring(async () => { + const reply = await chat(fake, { model: text, messages, stream }) + assert.deepEqual(reply, { status: 400, error: `Invalid messages: text-only.litertlm takes no ${input} input` }) + }) + assert.deepEqual(settingsOf(log), [], 'nothing loaded') + } + } + // An image earlier in the conversation counts too. + const reply = await chat(fake, { model: text, messages: [...imageMessage('What color is this?'), { role: 'assistant', content: 'Red.' }, ...hello] }) + assert.equal(reply.status, 400) + // A model that takes it, and one whose header the server cannot read (left to the runtime, as before), get it. + assert.equal((await chat(fake, { model: readable('vl'), messages: imageMessage('What color is this?') })).status, 200) + assert.equal((await chat(fake, { model: readable('omni'), messages: audioMessage })).status, 200) + assert.equal((await chat(fake, { model: unreadable('unreadable'), messages: imageMessage('What color is this?') })).status, 200) +}) + +// The server's peak resident memory, from /proc (Linux, as the tests are). +const peakMb = (server) => Number(/^VmHWM:\s+(\d+) kB/m.exec(readFileSync(`/proc/${server.child.pid}/status`, 'utf8'))[1]) / 1024 + +for (const path of [...new Set([...realModels, ...visionModels])].length ? [...new Set([...realModels, ...visionModels])] : ['']) { + test(`a real model's file is read for what it takes without reading its model into memory: ${path}`, { skip: skipReal && skipVision, timeout: 300_000 }, async (t) => { + // No compiled-model cache, as in every real-model test: a model that is loaded after all (one that takes audio, or + // a refusal that broke) would otherwise get one beside it, as large as the model. + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const idle = peakMb(server) + // Audio, which none of the litert-community files named above takes: refused once the file is read, before any + // engine is created, so the peak is what reading the file cost. + const reply = await chat(server, { model: `${path},cpu`, messages: audioMessage, max_completion_tokens: 1 }, 300_000) + if (reply.status === 200) return t.skip(`${basename(path)} takes audio, so it was loaded`) + assert.equal(reply.error, `Invalid messages: ${basename(path)} takes no audio input`) + assert.ok(peakMb(server) - idle < 32, `${basename(path)}: peak ${idle.toFixed(0)} MB idle, ${peakMb(server).toFixed(0)} MB after its file was read`) + } finally { + stopServer(server) + } + }) +} + +for (const path of visionModels.length ? visionModels : ['']) { + test(`a real model that takes images sees one: ${path}`, { skip: skipVision, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const reply = await chat(server, { model: `${path},cpu`, messages: imageMessage('What color is this image? Answer in one word.'), max_completion_tokens: 16 }, 300_000) + assert.equal(reply.status, 200, reply.error) + assert.match(reply.content, /red/i) + const plain = await chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'Say hello.' }], max_completion_tokens: 16 }, 300_000) + assert.equal(plain.status, 200, plain.error) + } finally { + stopServer(server) + } + }) +} + +// The sampler the stand-in logged for the conversation a request was answered with (the last one created). +async function samplerFor(server, body) { + const log = await callsDuring(async () => { + const reply = await chat(server, body) + assert.equal(reply.status, 200, reply.error) + }) + const created = log.filter((c) => c.call === 'conversation_create') + assert.ok(created.length > 0, 'no conversation created') + return created.at(-1).config.sampler ?? null +} +const f = Math.fround +const NO_LIMIT = 2147483647 +const TOP_P = 2 +// LlmMetadata with the model's own sampler_params (field 4, a SamplerParameters: type 1, k 2, p 3 and temperature 4, +// the last two floats). +const float = (n, v) => { const b = Buffer.alloc(5); b[0] = (n << 3) | 5; b.writeFloatLE(v, 1); return b } +const samplerMetadata = ({ type, k, p, temperature }) => ({ type: 5, bytes: proto.field(4, [ + proto.varint(1, type), ...(k === undefined ? [] : [proto.varint(2, k)]), ...(p === undefined ? [] : [float(3, p)]), + ...(temperature === undefined ? [] : [float(4, temperature)]), +]) }) +// A main model section whose decode signature outputs "logits" of [1, 1, vocabulary], as a tflite::Model lays them +// out (subgraphs 2, signature_defs 7; SignatureDef outputs 1, signature_key 2, subgraph_index 4; TensorMap name 0, +// tensor_index 1; SubGraph tensors 0; Tensor shape 0). +const decoderWith = (vocabulary, model_type = 'tf_lite_prefill_decode', constraint) => ({ + ...tflite(model_type, constraint), + bytes: flatbuffer(fb.table([undefined, undefined, fb.vector([fb.table([fb.vector([fb.table([fb.ints([1, 1, vocabulary])])])])]), + undefined, undefined, undefined, undefined, + fb.vector([fb.table([undefined, fb.vector([fb.table(['logits', fb.u32(0)])]), 'decode', undefined, fb.u32(0)])])])), +}) +Object.assign(kinds, { + 'sampler-topp': [decoder, samplerMetadata({ type: TOP_P, k: 50, p: 0.9, temperature: 0.6 })], + 'sampler-topk': [decoder, samplerMetadata({ type: 1, k: 40, temperature: 0.8 })], + 'sampler-greedy': [decoder, samplerMetadata({ type: 3 })], + 'sampler-k0': [decoder, samplerMetadata({ type: TOP_P, k: 0, p: 0.95, temperature: 0.7 })], + 'plain-vocabulary': [decoderWith(32000)], + // An Artisan text decoder, which LiteRT-LM runs on GPU as GPU_ARTISAN (gemma-4-E2B-it-gpu has one), with a sampler. + artisan: [decoderWith(262144, 'tf_lite_artisan_text_decoder', 'gpu_artisan'), samplerMetadata({ type: TOP_P, k: 64, p: 0.95, temperature: 1 })], +}) + +test('the fields a request leaves out of its sampler come from the model\'s own parameters, and the ones it sets are used as set', { skip, timeout: 60_000 }, async () => { + const model = (name, backend = 'cpu') => `${readable(name)},${backend}` + const check = async (name, request, expected, backend) => { + const sampler = await samplerFor(fake, { model: model(name, backend), messages: hello, ...request }) + assert.ok(sampler, `no sampler set: ${name} ${JSON.stringify(request)}`) + const { seed, ...rest } = sampler + assert.deepEqual(rest, { type: TOP_P, ...expected }, `${name} ${JSON.stringify(request)}`) + assert.ok(Number.isInteger(seed)) + } + // The model's own: TOP_P as given, TOP_K as TOP_P with p 1 (the CPU sampler has only TOP_P), GREEDY as k 1, and no k + // as no limit. + await check('sampler-topp', {}, { top_k: 50, top_p: f(0.9), temperature: f(0.6) }) + await check('sampler-topk', {}, { top_k: 40, top_p: 1, temperature: f(0.8) }) + await check('sampler-greedy', {}, { top_k: 1, top_p: 1, temperature: 1 }) + await check('sampler-k0', {}, { top_k: NO_LIMIT, top_p: f(0.95), temperature: f(0.7) }) + // Each field the request sets replaces only that one. + await check('sampler-topp', { temperature: 1.2 }, { top_k: 50, top_p: f(0.9), temperature: f(1.2) }) + await check('sampler-topp', { top_p: 0.5 }, { top_k: 50, top_p: 0.5, temperature: f(0.6) }) + await check('sampler-topp', { top_k: 10 }, { top_k: 10, top_p: f(0.9), temperature: f(0.6) }) + // A model with none: LiteRT-LM's own defaults, which are greedy (k 1), so a temperature or top_p the request sets + // alone gets no top-k limit, or it would do nothing; an explicit top_k, and temperature 0, stay greedy. + await check('plain', {}, { top_k: 1, top_p: f(0.95), temperature: 1 }) + await check('plain', { temperature: 1.5 }, { top_k: NO_LIMIT, top_p: f(0.95), temperature: 1.5 }) + await check('plain', { top_p: 0.8 }, { top_k: NO_LIMIT, top_p: f(0.8), temperature: 1 }) + await check('plain', { temperature: 0 }, { top_k: 1, top_p: f(0.95), temperature: 0 }) + await check('plain', { temperature: 1.5, top_k: 1 }, { top_k: 1, top_p: f(0.95), temperature: 1.5 }) + // On the GPU backend, no limit is the model's vocabulary, all its tokens; where that is unknown, as on CPU. + await check('plain-vocabulary', { temperature: 1.5 }, { top_k: 32000, top_p: f(0.95), temperature: 1.5 }, 'gpu') + await check('plain-vocabulary', { temperature: 1.5 }, { top_k: NO_LIMIT, top_p: f(0.95), temperature: 1.5 }) + await check('plain', { temperature: 1.5 }, { top_k: NO_LIMIT, top_p: f(0.95), temperature: 1.5 }, 'gpu') + const lines = await logged(fake) + // Said once per model and backend, not on every request. + assert.deepEqual(lines.filter((l) => l.startsWith("top_k is not set and plain.litertlm's")), [ + "top_k is not set and plain.litertlm's default is 1 (greedy), under which the request's temperature would have no " + + 'effect; sampling on cpu with top_k 2147483647 (no limit)', + "top_k is not set and plain.litertlm's default is 1 (greedy), under which the request's temperature would have no " + + 'effect; sampling on gpu with top_k 2147483647 (no limit)', + ], fake.stderr) + assert.ok(lines.includes("top_k is not set and plain-vocabulary.litertlm's default is 1 (greedy), under which the " + + "request's temperature would have no effect; sampling on gpu with top_k 32000 (its vocabulary: no limit)"), fake.stderr) + // On NPU, and for an Artisan text decoder on GPU, where LiteRT-LM leaves sampling to the model, nothing is set + // unless the request sets something. + for (const name of ['sampler-topp,npu', 'artisan,gpu']) { + const [kind, backend] = name.split(',') + assert.equal(await samplerFor(fake, { model: model(kind, backend), messages: hello }), null, name) + } + const npu = await samplerFor(fake, { model: model('sampler-topp', 'npu'), messages: hello, temperature: 0.3 }) + assert.deepEqual({ ...npu, seed: 0 }, { type: TOP_P, top_k: 50, top_p: f(0.9), temperature: f(0.3), seed: 0 }) + const artisan = await samplerFor(fake, { model: model('artisan', 'gpu'), messages: hello, temperature: 0.3 }) + assert.deepEqual({ ...artisan, seed: 0 }, { type: TOP_P, top_k: 64, top_p: f(0.95), temperature: f(0.3), seed: 0 }) +}) + +test('a request without a seed is sampled with a new random seed, and one with a seed with that seed', { skip, timeout: 60_000 }, async () => { + const seeds = [] + for (let i = 0; i < 6; i++) seeds.push((await samplerFor(fake, { model: readable('sampler-topp'), messages: hello }))?.seed) + for (let i = 0; i < 2; i++) seeds.push((await samplerFor(fake, { model: readable('sampler-topp'), messages: hello, temperature: 1 })).seed) + assert.equal(new Set(seeds).size, seeds.length, `seeds repeat: ${seeds}`) + // Not 0 or 1, which seed the CPU sampler's generator alike. + assert.ok(seeds.every((s) => s > 1), `${seeds}`) + assert.equal((await samplerFor(fake, { model: readable('sampler-topp'), messages: hello, seed: 7 })).seed, 7) + assert.equal((await samplerFor(fake, { model: readable('plain'), messages: hello, seed: 0 })).seed, 0) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model samples with a temperature sent alone, differently each time without a seed: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ask = async (extra) => { + const reply = await chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'Write one sentence about the sea.' }], max_completion_tokens: 24, ...extra }, 300_000) + assert.equal(reply.status, 200, reply.error) + return reply.reasoning + reply.content // a thinking model (Qwen3) may only think in 24 tokens + } + const unseeded = [await ask({ temperature: 1.5 }), await ask({ temperature: 1.5 }), await ask({ temperature: 1.5 })] + assert.ok(new Set(unseeded).size > 1, `the same reply three times: ${unseeded[0]}`) + assert.equal(await ask({ temperature: 1.5, seed: 7 }), await ask({ temperature: 1.5, seed: 7 })) + } finally { + stopServer(server) + } + }) +} + +test('top_k 0 or below is no top-k limit, and a sampler value the runtime cannot take is a 400 that names it', { skip, timeout: 60_000 }, async () => { + const k = async (request, backend = 'cpu') => (await samplerFor(fake, { model: `${readable('sampler-topp')},${backend}`, messages: hello, ...request }))?.top_k + assert.equal(await k({ top_k: 0 }), NO_LIMIT) + assert.equal(await k({ top_k: -1, temperature: 0.7 }), NO_LIMIT) + assert.equal(await k({ top_k: 40.0 }), 40) + // On GPU no limit is the model's vocabulary, and a top_k above 0 is used as set: LiteRT-LM sets no limit there. + const gpu = async (request) => (await samplerFor(fake, { model: `${readable('plain-vocabulary')},gpu`, messages: hello, ...request }))?.top_k + assert.equal(await gpu({ top_k: 0 }), 32000) + assert.equal(await gpu({ top_k: 65 }), 65) + assert.equal(await gpu({ top_k: 100000 }), 100000) + const lines = await logged(fake) + assert.equal(lines.filter((l) => l === 'top_k 0 is no top-k limit: top_k 2147483647 for sampler-topp.litertlm (the CPU sampler takes the whole vocabulary)').length, 1, fake.stderr) + assert.ok(lines.includes('top_k 0 is no top-k limit: top_k 32000 for plain-vocabulary.litertlm (its vocabulary)'), fake.stderr) + const refused = async (request, backend = 'cpu') => { + const reply = await chat(fake, { model: `${readable('sampler-topp')},${backend}`, messages: hello, ...request }) + assert.equal(reply.status, 400, JSON.stringify(request)) + return reply.error + } + assert.equal(await refused({ top_p: 1.5 }), 'Invalid sampler parameters: ValueError: top_p must be between 0 and 1, got 1.5') + assert.equal(await refused({ top_p: -0.1 }), 'Invalid sampler parameters: ValueError: top_p must be between 0 and 1, got -0.1') + assert.equal(await refused({ temperature: -1 }), 'Invalid sampler parameters: ValueError: temperature must be at least 0, got -1') + assert.equal(await refused({ top_k: 2.5 }), 'Invalid sampler parameters: ValueError: top_k must be an integer, got 2.5') + assert.equal(await refused({ seed: 1.5 }), 'Invalid sampler parameters: ValueError: seed must be an integer, got 1.5') + assert.equal(await refused({ seed: 1e12 }), 'Invalid sampler parameters: ValueError: seed must be an integer, got 1000000000000') +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model takes top_k 0 and -1 as no limit, and refuses a top_p it cannot take with a 400: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ask = (extra) => chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'Say hello.' }], max_completion_tokens: 8, ...extra }, 300_000) + for (const extra of [{ top_k: 0 }, { top_k: -1, temperature: 0.7 }]) { + const reply = await ask(extra) + assert.equal(reply.status, 200, `${JSON.stringify(extra)}: ${reply.error}`) + } + for (const extra of [{ top_p: 1.5 }, { temperature: -1 }]) assert.equal((await ask(extra)).status, 400, JSON.stringify(extra)) + } finally { + stopServer(server) + } + }) +} + +// LlmMetadata for a stand-in that thinks or not: supports_thinking (field 11), a Jinja template (7) and a channel (8). +const thinkingMetadata = ({ flag, template, channel }) => ({ + type: 5, + bytes: Buffer.concat([ + ...(flag === undefined ? [] : [proto.varint(11, flag ? 1 : 0)]), + ...(template ? [proto.field(7, template)] : []), + ...(channel ? [proto.field(8, [proto.field(1, 'thought'), proto.field(2, ''), proto.field(3, '')])] : []), + ]), +}) +const switchTemplate = "{%- if enable_thinking is defined and not enable_thinking -%}{%- endif -%}" +// What the conversation and its message were created with: its thinking config and the template's extra context. +async function thinkingFor(server, body) { + const log = await callsDuring(async () => { + const reply = await chat(server, body) + assert.equal(reply.status, 200, reply.error) + }) + const config = log.filter((c) => c.call === 'conversation_create').at(-1).config + const sent = log.filter((c) => c.call === 'send_message').at(-1) + return { thinking: config.thinking ?? null, preface: config.extra_context ?? null, message: sent.extra_context } +} + +test('every way a request asks for thinking reaches the model, and the template is told what the decoder is', { skip, timeout: 60_000 }, async () => { + const model = litertlmFile(join(work, 'switch.litertlm'), [thinkingMetadata({ template: switchTemplate, channel: true })]) + const ask = (request) => thinkingFor(fake, { model, messages: hello, ...request }) + const on = { enable: true, budget: -1 } + const off = { enable: false, budget: 0 } + assert.deepEqual(await ask({}), { thinking: null, preface: null, message: {} }) + assert.deepEqual((await ask({ reasoning_effort: 'none' })).thinking, off) + for (const level of ['minimal', 'low', 'medium', 'high', 'xhigh']) assert.deepEqual((await ask({ reasoning_effort: level })).thinking, on) + assert.deepEqual(await ask({ chat_template_kwargs: { enable_thinking: false } }), + { thinking: off, preface: { enable_thinking: false }, message: { enable_thinking: false } }) + assert.deepEqual(await ask({ chat_template_kwargs: { enable_thinking: true, preserve_thinking: true } }), + { thinking: on, preface: { enable_thinking: true, preserve_thinking: true }, message: { enable_thinking: true, preserve_thinking: true } }) + assert.deepEqual((await ask({ chat_template_kwargs: { reasoning_effort: 'low' } })).thinking, on) + assert.deepEqual((await ask({ chat_template_kwargs: { reasoning_effort: 'none' } })).thinking, off) + assert.deepEqual((await ask({ thinking_budget_tokens: 0 })).thinking, off) + assert.deepEqual((await ask({ thinking_budget_tokens: 256 })).thinking, { enable: true, budget: 256 }) + assert.deepEqual((await ask({ reasoning_effort: 'medium', thinking_budget_tokens: 128 })).thinking, { enable: true, budget: 128 }) + // The top-level reasoning_effort comes first, and the template is told the same. + assert.deepEqual(await ask({ reasoning_effort: 'none', chat_template_kwargs: { enable_thinking: true } }), + { thinking: off, preface: { enable_thinking: false }, message: { enable_thinking: false } }) + const refused = async (request) => { + const reply = await chat(fake, { model, messages: hello, ...request }) + assert.equal(reply.status, 400, JSON.stringify(request)) + return reply.error + } + assert.equal(await refused({ chat_template_kwargs: 'enable_thinking=false' }), 'Invalid thinking parameters: ValueError: chat_template_kwargs must be an object') + assert.equal(await refused({ chat_template_kwargs: { enable_thinking: 'no' } }), 'Invalid thinking parameters: ValueError: chat_template_kwargs.enable_thinking must be a boolean') + assert.equal(await refused({ thinking_budget_tokens: 1.5 }), 'Invalid thinking parameters: ValueError: thinking_budget_tokens must be an integer') + assert.match(await refused({ chat_template_kwargs: { reasoning_effort: 'max' } }), /^Invalid thinking parameters: ValueError: Invalid chat_template_kwargs.reasoning_effort value: 'max'/) +}) + +test('thinking on for a model that does not think is a 400, and off changes nothing there', { skip, timeout: 60_000 }, async () => { + const none = litertlmFile(join(work, 'no-thinking.litertlm'), [thinkingMetadata({ template: '{{ messages }}' })]) + const saysNo = litertlmFile(join(work, 'says-no.litertlm'), [thinkingMetadata({ flag: false, template: switchTemplate, channel: true })]) + const saysYes = litertlmFile(join(work, 'says-yes.litertlm'), [thinkingMetadata({ flag: true })]) + const unknown = litertlmFile(join(work, 'channel-only.litertlm'), [thinkingMetadata({ template: '{{ messages }}', channel: true })]) + for (const [path, why] of [[none, 'no-thinking.litertlm does not think'], [saysNo, 'says-no.litertlm does not think (its metadata says so)']]) { + // Refused before it loads: the model loaded before stays loaded. + assert.equal((await chat(fake, { model: saysYes, messages: hello })).status, 200) + for (const [request, from] of [[{ reasoning_effort: 'high' }, 'reasoning_effort'], [{ chat_template_kwargs: { enable_thinking: true } }, 'chat_template_kwargs.enable_thinking'], [{ thinking_budget_tokens: 64 }, 'thinking_budget_tokens']]) { + const log = await callsDuring(async () => { + const reply = await chat(fake, { model: path, messages: hello, ...request }) + assert.deepEqual(reply, { status: 400, error: `Invalid thinking parameters: ${from} turns thinking on, but ${why}` }) + }) + assert.deepEqual(settingsOf(log), [], `${basename(path)} was loaded for ${JSON.stringify(request)}`) + } + assert.deepEqual(settingsOf(await callsDuring(async () => { + assert.equal((await chat(fake, { model: saysYes, messages: hello })).status, 200) + })), [], 'the model loaded before was released') + for (const request of [{ reasoning_effort: 'none' }, { thinking_budget_tokens: 0 }]) assert.equal((await thinkingFor(fake, { model: path, messages: hello, ...request })).thinking, null) + } + assert.deepEqual((await thinkingFor(fake, { model: saysYes, messages: hello, reasoning_effort: 'high' })).thinking, { enable: true, budget: -1 }) + for (let i = 0; i < 2; i++) assert.deepEqual((await thinkingFor(fake, { model: unknown, messages: hello, reasoning_effort: 'high' })).thinking, { enable: true, budget: -1 }) + const lines = await logged(fake) + assert.equal(lines.filter((l) => l.startsWith('channel-only.litertlm does not say whether it thinks')).length, 1, fake.stderr) + assert.ok(lines.includes('channel-only.litertlm does not say whether it thinks, and its template has no switch for it; thinking on (reasoning_effort) is passed to it as asked'), fake.stderr) + assert.equal(lines.some((l) => l.startsWith('says-yes.litertlm does not say')), false) +}) + +test('a thinking request to a model whose file does not say whether it thinks is answered with X-Thinking: unverified', { skip, timeout: 60_000 }, async () => { + const unknown = litertlmFile(join(work, 'channel-only-note.litertlm'), [thinkingMetadata({ template: '{{ messages }}', channel: true })]) + const saysYes = litertlmFile(join(work, 'says-yes-note.litertlm'), [thinkingMetadata({ flag: true })]) + const switched = litertlmFile(join(work, 'switch-note.litertlm'), [thinkingMetadata({ template: switchTemplate, channel: true })]) + const note = async (model, request, stream = false) => { + const reply = await chat(fake, { model, messages: hello, stream, ...request }) + assert.equal(reply.status, 200, reply.error) + return reply.headers.get('x-thinking') + } + // Thinking on or off, streamed or not: the server could not tell whether the model does it. + for (const stream of [false, true]) { + assert.equal(await note(unknown, { reasoning_effort: 'high' }, stream), 'unverified') + assert.equal(await note(unknown, { chat_template_kwargs: { enable_thinking: false } }, stream), 'unverified') + } + // Not without a thinking request, nor for a model that says it thinks or whose template switches it. + assert.equal(await note(unknown, {}), null) + assert.equal(await note(saysYes, { reasoning_effort: 'high' }), null) + assert.equal(await note(switched, { reasoning_effort: 'none' }), null) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model does not think when the request turns thinking off: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + for (const off of [{ chat_template_kwargs: { enable_thinking: false } }, { thinking_budget_tokens: 0 }, { reasoning_effort: 'none' }]) { + const reply = await chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'What is 2+3? Answer briefly.' }], max_completion_tokens: 64, seed: 1, ...off }, 300_000) + assert.equal(reply.status, 200, `${JSON.stringify(off)}: ${reply.error}`) + assert.equal(reply.reasoning, '', `${JSON.stringify(off)} thought: ${reply.reasoning}`) + assert.ok(reply.content, `${JSON.stringify(off)}: no answer`) + } + } finally { + stopServer(server) + } + }) +} + +for (const path of thinkingModels.length ? thinkingModels : ['']) { + test(`a real model thinks when asked, within the budget it is given: ${path}`, { skip: skipThinking, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ask = (extra) => chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'What is 17 times 23?' }], max_completion_tokens: 400, seed: 1, ...extra }, 300_000) + const on = await ask({ chat_template_kwargs: { enable_thinking: true } }) + assert.equal(on.status, 200, on.error) + assert.ok(on.reasoning.length > 0, 'no thinking') + const budgeted = await ask({ thinking_budget_tokens: 16 }) + assert.equal(budgeted.status, 200, budgeted.error) + assert.ok(budgeted.reasoning.length > 0, 'no thinking') + assert.ok(budgeted.usage.completion_tokens_details.reasoning_tokens <= 24, `thought ${budgeted.usage.completion_tokens_details.reasoning_tokens} chunks for a budget of 16`) + assert.ok(budgeted.usage.completion_tokens_details.reasoning_tokens < on.usage.completion_tokens_details.reasoning_tokens) + } finally { + stopServer(server) + } + }) +} + +test('a stop string ends the reply before it, streamed or not, and stops the generation', { skip, timeout: 60_000 }, async () => { + const model = readable('plain') + const say = (text) => [{ role: 'user', content: `say:${text}` }] // the stand-in streams it back, three bytes a chunk + for (const stream of [false, true]) { + const ask = async (text, stop) => { + const reply = await chat(fake, { model, messages: say(text), stop, stream }) + assert.equal(reply.status, 200, reply.error) + return reply + } + let stopped + const log = await callsDuring(async () => { stopped = await ask('The sea is a vast ocean that never ends.', ['ocean']) }) + assert.equal(stopped.content, 'The sea is a vast ', `stream ${stream}`) + assert.equal(stopped.finish, 'stop') + assert.ok(log.some((c) => c.call === 'cancel_process'), 'the generation was not cancelled') + if (stream) { + // Nothing of the stop string went out, not even the part of it that came in one chunk ("oce"). + const sent = stopped.chunks.map((c) => c.choices[0]?.delta?.content ?? '') + assert.ok(sent.every((t) => !t.includes('oc')), JSON.stringify(sent)) + } + assert.equal((await ask('The sea is calm.', ['xyz'])).content, 'The sea is calm.') + // A reply that only looked as if it might become one ends whole. + assert.equal((await ask('It ends with oc', ['ocean'])).content, 'It ends with oc') + assert.equal((await ask('The sea is calm.', 'is')).content, 'The sea ') + assert.equal((await ask('The sea is a vast ocean.', ['vast', 'sea'])).content, 'The ') + assert.equal((await ask('Hello', ['', 'xyz'])).content, 'Hello') + } + for (const stop of [5, [1], [{}]]) { + const reply = await chat(fake, { model, messages: say('x'), stop }) + assert.deepEqual(reply, { status: 400, error: 'stop must be a string or an array of strings' }) + } +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model's reply ends before a stop string, streamed or not: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + // Seeded, and without thinking, so the reply is text and the same each time. + const ask = (extra) => chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'Write one sentence about the sea.' }], max_completion_tokens: 48, seed: 1, reasoning_effort: 'none', ...extra }, 300_000) + const whole = await ask({}) + assert.equal(whole.status, 200, whole.error) + const words = whole.content.split(/\s+/).filter((w) => /^[a-z]{3,}$/.test(w)) + assert.ok(words.length >= 2, `too short: ${whole.content}`) + const stop = words[1] + const expected = whole.content.slice(0, whole.content.indexOf(stop)) + for (const stream of [false, true]) { + const reply = await ask({ stop: [stop], stream }) + assert.equal(reply.status, 200, reply.error) + assert.equal(reply.content, expected, `stop ${JSON.stringify(stop)}, stream ${stream}`) + assert.equal(reply.finish, 'stop') + } + } finally { + stopServer(server) + } + }) +} + +test('repetition penalties reach the model with llama.cpp\'s and OpenAI\'s meanings, and what LiteRT-LM cannot do is said', { skip, timeout: 60_000 }, async () => { + const model = readable('plain') + const penalties = async (request) => { + const log = await callsDuring(async () => { + const reply = await chat(fake, { model, messages: hello, ...request }) + assert.equal(reply.status, 200, reply.error) + }) + return log.filter((c) => c.call === 'send_message').at(-1).args.repetition_penalty_config ?? null + } + const config = (repetition_penalty, presence_penalty, frequency_penalty, window_size) => ({ repetition_penalty, presence_penalty, frequency_penalty, window_size }) + assert.equal(await penalties({}), null) + // llama.cpp's repeat_penalty, over llama.cpp's default 64 tokens unless repeat_last_n says otherwise (0: off). + assert.deepEqual(await penalties({ repeat_penalty: 1.3 }), config(f(1.3), 0, 0, 64)) + assert.deepEqual(await penalties({ repeat_penalty: 1.1, presence_penalty: 0.6 }), config(f(1.1), f(0.6), 0, 64)) + assert.deepEqual(await penalties({ repeat_penalty: 1.3, repeat_last_n: 128 }), config(f(1.3), 0, 0, 128)) + assert.deepEqual(await penalties({ repeat_penalty: 1.3, repeat_last_n: -1 }), config(f(1.3), 0, 0, 0)) + assert.equal(await penalties({ repeat_penalty: 1.3, repeat_last_n: 0 }), null) + assert.equal(await penalties({ repeat_penalty: 1 }), null) + // OpenAI's and HuggingFace's names, over the whole reply. + assert.deepEqual(await penalties({ repetition_penalty: 1.2 }), config(f(1.2), 0, 0, 0)) + assert.deepEqual(await penalties({ presence_penalty: 0.5, frequency_penalty: 1 }), config(1, 0.5, 1, 0)) + assert.deepEqual(await penalties({ frequency_penalty: -0.5 }), config(1, 0, -0.5, 0)) + assert.equal(await penalties({ min_p: 0 }), null) + const refused = async (request) => { + const reply = await chat(fake, { model, messages: hello, ...request }) + assert.equal(reply.status, 400, JSON.stringify(request)) + return reply.error + } + assert.equal(await refused({ repeat_penalty: 0.9 }), 'Invalid sampler parameters: repeat_penalty 0.9 is below 1, which LiteRT-LM does not take (1 is no penalty)') + assert.equal(await refused({ presence_penalty: 'high' }), 'Invalid sampler parameters: ValueError: presence_penalty must be a number') + assert.equal(await refused({ repeat_penalty: 1.2, repeat_last_n: -2 }), 'Invalid sampler parameters: ValueError: repeat_last_n must be an integer of -1 or more') + // min_p, which LiteRT-LM has no sampler for, is not applied, and the reply says so, streamed or not (a 400 would fail + // every chat whose saved settings have a Min P). + for (const stream of [false, true]) { + const reply = await chat(fake, { model, messages: hello, min_p: 0.05, stream }) + assert.equal(reply.status, 200, reply.error) + assert.equal(reply.headers.get('x-unapplied-parameters'), 'min_p') + } + assert.equal((await chat(fake, { model, messages: hello, min_p: 0 })).headers.get('x-unapplied-parameters'), null) + assert.equal(await refused({ min_p: 'low' }), 'Invalid sampler parameters: ValueError: min_p must be a number') + assert.equal((await logged(fake)).filter((l) => l === 'min_p is not applied to plain.litertlm: LiteRT-LM has no min-p sampling').length, 1, fake.stderr) +}) + +test('min_p is said once per model, by a request that is answered, also when requests with it arrive at once', { skip, timeout: 60_000 }, async () => { + // Each connection has its own thread. The note is taken under the engine lock, so requests refused before it (here + // for their response_format) never touch it, and answered ones take turns. + const models = Array.from({ length: 32 }, (_, i) => readable(`min-p-${i}`)) + const replies = await Promise.all(models.map((model) => chat(fake, { model, messages: hello, min_p: 0.05, response_format: { type: 'bogus' } }))) + assert.deepEqual(replies.map((r) => r.status), models.map(() => 400)) + const note = (i) => `min_p is not applied to min-p-${i}.litertlm: LiteRT-LM has no min-p sampling` + assert.equal((await logged(fake)).filter((l) => l.startsWith('min_p is not applied to min-p-')).length, 0, fake.stderr) + const answered = await Promise.all([...models.slice(0, 8), ...models.slice(0, 8)].map((model) => chat(fake, { model, messages: hello, min_p: 0.05 }))) + assert.deepEqual(answered.map((r) => [r.status, r.headers.get('x-unapplied-parameters')]), answered.map(() => [200, 'min_p'])) + const lines = await logged(fake) + for (let i = 0; i < 8; i++) assert.equal(lines.filter((l) => l === note(i)).length, 1, `${note(i)}\n${fake.stderr}`) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model's reply changes under a repetition penalty: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + // Greedy and without thinking, so only the penalty can change the reply. + const ask = async (extra) => { + const reply = await chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'Write two sentences about the sea.' }], max_completion_tokens: 64, temperature: 0, reasoning_effort: 'none', ...extra }, 300_000) + assert.equal(reply.status, 200, reply.error) + return reply.content + } + const plain = await ask({}) + assert.notEqual(await ask({ frequency_penalty: 2 }), plain, 'frequency_penalty 2 changed nothing') + assert.notEqual(await ask({ repeat_penalty: 1.5 }), plain, 'repeat_penalty 1.5 changed nothing') + } finally { + stopServer(server) + } + }) +} + +const weatherTool = [{ + type: 'function', + function: { name: 'get_weather', description: 'Weather in a city', parameters: { type: 'object', properties: { city: { type: 'string' } }, required: ['city'] } }, +}] + +test('tool_choice "none" leaves the tools out, "auto" keeps them, and one LiteRT-LM cannot force is a 400', { skip, timeout: 60_000 }, async () => { + // The stand-in's reply says whether the conversation was given tools ("tools=1") and constrained ("constrained=1"). + const created = async (request) => { + const reply = await chat(fake, { model: readable('gemma-tools'), messages: hello, tools: weatherTool, ...request }) + assert.equal(reply.status, 200, reply.error) + return Object.fromEntries(reply.content.split(' ').map((kv) => kv.split('=')).filter(([k]) => k === 'tools' || k === 'constrained').map(([k, v]) => [k, Number(v)])) + } + assert.deepEqual(await created({}), { constrained: 1, tools: 1 }) + assert.deepEqual(await created({ tool_choice: 'auto' }), { constrained: 1, tools: 1 }) + assert.deepEqual(await created({ tool_choice: null }), { constrained: 1, tools: 1 }) + assert.deepEqual(await created({ tool_choice: 'none' }), { constrained: 0, tools: 0 }) + // With the earlier calls and results still in the history. + const history = [ + { role: 'user', content: 'Weather in Paris?' }, + { role: 'assistant', content: null, tool_calls: [{ id: 'c1', type: 'function', function: { name: 'get_weather', arguments: '{"city":"Paris"}' } }] }, + { role: 'tool', tool_call_id: 'c1', content: 'Sunny, 21 C' }, + ] + assert.deepEqual(await created({ tool_choice: 'none', messages: history }), { constrained: 0, tools: 0 }) + for (const [tool_choice, said] of [['required', '"required"'], [{ type: 'function', function: { name: 'get_weather' } }, '{"function":{"name":"get_weather"},"type":"function"}']]) { + const reply = await chat(fake, { model: readable('gemma-tools'), messages: hello, tools: weatherTool, tool_choice }) + assert.deepEqual(reply, { status: 400, error: `Invalid tool_choice: this server cannot make the model call a tool (${said}); LiteRT-LM can only offer it the tools, as tool_choice "auto" does` }) + } + const odd = await chat(fake, { model: readable('gemma-tools'), messages: hello, tools: weatherTool, tool_choice: 'sometimes' }) + assert.deepEqual(odd, { status: 400, error: 'Invalid tool_choice: "sometimes" (none, auto, required or a function)' }) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model calls no tool with tool_choice "none", with or without earlier calls: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ask = (messages, extra) => chat(server, { model: `${path},cpu`, messages, tools: weatherTool, max_completion_tokens: 96, temperature: 0, reasoning_effort: 'none', ...extra }, 300_000) + const question = [{ role: 'user', content: 'What is the weather in Paris?' }] + const history = [ + ...question, + { role: 'assistant', content: null, tool_calls: [{ id: 'c1', type: 'function', function: { name: 'get_weather', arguments: '{"city":"Paris"}' } }] }, + { role: 'tool', tool_call_id: 'c1', content: 'Sunny, 21 C' }, + { role: 'user', content: 'And in Rome?' }, + ] + // A template that cannot render tool turns at all (gemma-3-270m-it's: "Conversation roles must alternate") + // cannot render them with tool_choice "none" either; that is not this test's business. + const rendersHistory = !/Failed to apply template/.test((await ask(history, {})).error ?? '') + for (const messages of rendersHistory ? [question, history] : [question]) { + const reply = await ask(messages, { tool_choice: 'none' }) + assert.equal(reply.status, 200, reply.error) + assert.equal(reply.tool_calls, undefined, `called ${JSON.stringify(reply.tool_calls)}`) + assert.ok(reply.content, 'no answer') + } + } finally { + stopServer(server) + } + }) +} + +// The loaded model's entry on /v1/models. +async function modelsEntry(server) { + const res = await request(`http://127.0.0.1:${server.port}/v1/models`, undefined, 5000) + assert.equal(res.status, 200) + return JSON.parse(res.text).data[0] +} + +// LlmMetadata's max_num_tokens (field 5), the model's context, which a prime above 10 marks as dynamic; and a main +// model whose LlmMetadata gives none, from whose prefill signature's "mask" input, [1, 1, 32, context], LiteRT-LM +// reads it instead (gemma-3-270m-it's is read so). +const contextMetadata = (n, ...more) => ({ type: 5, bytes: Buffer.concat([proto.varint(5, n), ...more]) }) +const decoderWithMask = (context) => ({ + ...decoder, + bytes: flatbuffer(fb.table([undefined, undefined, fb.vector([fb.table([fb.vector([fb.table([fb.ints([1, 32])]), fb.table([fb.ints([1, 1, 32, context])])])])]), + undefined, undefined, undefined, undefined, + fb.vector([fb.table([fb.vector([fb.table(['tokens', fb.u32(0)]), fb.table(['mask', fb.u32(1)])]), undefined, 'prefill_32', undefined, fb.u32(0)])])])), +}) +Object.assign(kinds, { + ctx4096: [decoder, contextMetadata(4096)], + ctx8192: [decoder, contextMetadata(8192)], + dyn32k: [decoder, contextMetadata(32771)], + dyn251: [decoder, contextMetadata(251)], + 'ctx-prefill': [decoderWithMask(2048)], + 'ctx-prefill-8k': [decoderWithMask(8192)], + 'ctx4096-lfm2': [decoder, contextMetadata(4096, proto.field(6, proto.field(LFM2, [])))], +}) + +test('a context larger than the model takes is cut to the model\'s, logged and reported, never silently', { skip, timeout: 60_000 }, async () => { + const fixed = readable('ctx4096') + const window = async (ref) => { + let reply + const log = await callsDuring(async () => { + reply = await chat(fake, { model: ref, messages: hello }) + assert.equal(reply.status, 200, reply.error) + }) + const set = log.filter((c) => c.call === 'engine_settings_set_max_num_tokens').map((c) => c.max_num_tokens) + const entry = await modelsEntry(fake) + return { + set, + header: reply.headers.get('x-context-window'), + asked: reply.headers.get('x-context-window-requested'), + max_model_len: entry.max_model_len, + max_model_len_requested: entry.max_model_len_requested, + } + } + assert.deepEqual(await window(`${fixed},cpu,8192`), { set: [4096], header: '4096', asked: '8192', max_model_len: 4096, max_model_len_requested: 8192 }) + assert.deepEqual(await window(`${fixed},cpu,2048`), { set: [2048], header: '2048', asked: null, max_model_len: 2048, max_model_len_requested: undefined }) + assert.deepEqual(await window(`${fixed},cpu`), { set: [], header: '4096', asked: null, max_model_len: 4096, max_model_len_requested: undefined }) + // A dynamic context (a magic number, 32771) is made for the size asked when it is less, and for LiteRT-LM's default + // for it otherwise, the largest multiple of 256 below it (32768), with none asked too; below 256, a power of two. + const dynamic = readable('dyn32k') + assert.deepEqual(await window(`${dynamic},cpu,65536`), { set: [32768], header: '32768', asked: '65536', max_model_len: 32768, max_model_len_requested: 65536 }) + assert.deepEqual(await window(`${dynamic},cpu,32771`), { set: [32768], header: '32768', asked: '32771', max_model_len: 32768, max_model_len_requested: 32771 }) + assert.deepEqual(await window(`${dynamic},cpu,32770`), { set: [32770], header: '32770', asked: null, max_model_len: 32770, max_model_len_requested: undefined }) + assert.deepEqual(await window(`${dynamic},cpu`), { set: [], header: '32768', asked: null, max_model_len: 32768, max_model_len_requested: undefined }) + assert.equal((await window(`${readable('dyn251')},cpu`)).header, '128') + // With none asked, on GPU no more than 4096, as LiteRT-LM sets it there. + assert.deepEqual(await window(`${readable('ctx8192')},cpu`), { set: [], header: '8192', asked: null, max_model_len: 8192, max_model_len_requested: undefined }) + assert.deepEqual(await window(`${readable('ctx8192')},gpu`), { set: [], header: '4096', asked: null, max_model_len: 4096, max_model_len_requested: undefined }) + assert.equal((await window(`${readable('ctx8192')},gpu,8192`)).header, '8192') + assert.equal((await window(`${dynamic},gpu`)).header, '4096') + // From the prefill signature, for a file whose LlmMetadata gives none; with none asked, LiteRT-LM gives such a file + // 4096 at most. + assert.deepEqual(await window(`${readable('ctx-prefill')},cpu,8192`), { set: [2048], header: '2048', asked: '8192', max_model_len: 2048, max_model_len_requested: 8192 }) + assert.equal((await window(`${readable('ctx-prefill')},cpu`)).header, '2048') + assert.equal((await window(`${readable('ctx-prefill-8k')},cpu`)).header, '4096') + assert.equal((await window(`${readable('ctx-prefill-8k')},cpu,8192`)).header, '8192') + // A model that gets a new engine for each conversation (an LFM2 one) is told about once, not once per engine. + for (let i = 0; i < 3; i++) assert.equal((await window(`${readable('ctx4096-lfm2')},cpu,8192`)).header, '4096') + const lines = await logged(fake) + assert.ok(lines.includes('max_num_tokens 8192 is more than ctx4096.litertlm takes (its fixed context is 4096 tokens); its context window is 4096'), fake.stderr) + assert.ok(lines.includes('max_num_tokens 65536 is more than dyn32k.litertlm takes (its dynamic context takes fewer than 32771 tokens); its context window is 32768'), fake.stderr) + assert.equal(lines.filter((l) => l.startsWith('max_num_tokens 8192 is more than ctx4096-lfm2.litertlm takes')).length, 1, fake.stderr) + // A stream says it in its headers too. + const streamed = await chat(fake, { model: `${fixed},cpu,8192`, messages: hello, stream: true }) + assert.equal(streamed.headers.get('x-context-window'), '4096') + assert.equal(streamed.headers.get('x-context-window-requested'), '8192') +}) + +test('a prompt that does not fit the context window is a 400 that says the window', { skip, timeout: 60_000 }, async () => { + const reply = await chat(fake, { model: `${readable('ctx4096')},cpu`, messages: [{ role: 'user', content: 'too long' }] }) + assert.deepEqual(reply, { status: 400, error: "Invalid messages: the prompt does not fit the model's context window of 4096 tokens " + + '(INVALID_ARGUMENT: Input token ids are too long. Exceeding the maximum number of tokens allowed: 4209 >= 4096)' }) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model's context window is its own when more is asked, and a reply that fills it ends with "length": ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + // Ask for 64k, more than the static files here take, and a reply long enough to fill what they do. + const numbers = Array.from({ length: 700 }, (_, i) => i + 1).join(', ') + const reply = await chat(server, { model: `${path},cpu,65536`, messages: [{ role: 'user', content: `Continue this list forever: ${numbers}` }], max_completion_tokens: 60000, temperature: 0, reasoning_effort: 'none' }, 600_000) + assert.equal(reply.status, 200, reply.error) + const window = Number(reply.headers.get('x-context-window')) + assert.ok(window > 0 && window < 65536, `window ${window}`) + assert.equal(reply.headers.get('x-context-window-requested'), '65536') + const entry = await modelsEntry(server) + assert.equal(entry.max_model_len, window) + assert.equal(entry.max_model_len_requested, 65536) + // A reply cut off by the window reads "length" (a model that stops on its own first reads "stop"). + assert.ok(!(reply.finish === 'stop' && reply.usage.total_tokens >= window), `${reply.usage.total_tokens} tokens of ${window}, "stop"`) + const long = Array.from({ length: window + 100 }, () => 'word').join(' ') + const tooLong = await chat(server, { model: `${path},cpu,65536`, messages: [{ role: 'user', content: long }], max_completion_tokens: 8 }, 300_000) + assert.equal(tooLong.status, 400, tooLong.error) + assert.match(tooLong.error, new RegExp(`^Invalid messages: the prompt does not fit the model's context window of ${window} tokens`)) + } finally { + stopServer(server) + } + }) +} + +Object.assign(kinds, { + 'plain-backend': [decoder], + 'gpu-first': [tflite('tf_lite_prefill_decode', 'gpu,cpu')], + 'gpu-only': [tflite('tf_lite_prefill_decode', 'gpu')], + 'npu-first': [tflite('tf_lite_prefill_decode', 'npu,cpu')], + // An Artisan text decoder without a constraint: GPU first, as `litert-lm serve` picks it. + 'artisan-backend': [tflite('tf_lite_artisan_text_decoder')], + // The stand-in cannot create an engine on gpu for these (gpu-broken), or on any backend (no-engine). + 'gpu-broken': [decoderWith(32000, 'tf_lite_prefill_decode', 'gpu,cpu')], + 'gpu-broken-only': [tflite('tf_lite_prefill_decode', 'gpu_artisan')], + 'no-engine': [tflite('tf_lite_prefill_decode', 'gpu,cpu')], + // With an 8192-token context, which LiteRT-LM gives in full on CPU and no more than 4096 of on GPU. + 'gpu-first-ctx8192': [tflite('tf_lite_prefill_decode', 'gpu,cpu'), contextMetadata(8192)], + 'gpu-broken-ctx8192': [tflite('tf_lite_prefill_decode', 'gpu,cpu'), contextMetadata(8192)], +}) + +test('with no backend set, the model\'s own is used, and one that does not start falls back to the next it lists, logged', { skip, timeout: 60_000 }, async () => { + const engines = async (ref) => { + let reply + const log = await callsDuring(async () => { reply = await chat(fake, { model: ref, messages: hello }) }) + return { status: reply.status, error: reply.error, created: log.filter((c) => c.call === 'engine_create').map((c) => c.backend) } + } + assert.deepEqual(await engines(readable('plain-backend')), { status: 200, error: undefined, created: ['cpu'] }) + assert.deepEqual(await engines(readable('gpu-first')), { status: 200, error: undefined, created: ['gpu'] }) + assert.deepEqual(await engines(readable('gpu-only')), { status: 200, error: undefined, created: ['gpu'] }) + assert.deepEqual(await engines(readable('npu-first')), { status: 200, error: undefined, created: ['npu'] }) + assert.deepEqual(await engines(readable('artisan-backend')), { status: 200, error: undefined, created: ['gpu'] }) + assert.deepEqual(await engines(unreadable('unreadable-backend')), { status: 200, error: undefined, created: ['cpu'] }) + // The model's own backend does not start: the next it lists, once; the next request keeps that engine. + assert.deepEqual(await engines(readable('gpu-broken')), { status: 200, error: undefined, created: ['gpu', 'cpu'] }) + assert.deepEqual(await engines(readable('gpu-broken')), { status: 200, error: undefined, created: [] }) + // Only to one it lists: a GPU-only bundle is not tried on CPU, and the error is the GPU's. + const path = (name) => join(work, `${name}.litertlm`) + assert.deepEqual(await engines(readable('gpu-broken-only')), { status: 500, created: ['gpu'], + error: `Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for ${path('gpu-broken-only')}: UNAVAILABLE: no usable OpenCL device')` }) + // When none starts, each one's reason. + assert.deepEqual(await engines(readable('no-engine')), { status: 500, created: ['gpu', 'cpu'], + error: `Failed to load engine: RuntimeError('on gpu: Failed to create LiteRT-LM engine for ${path('no-engine')}: INTERNAL: cannot load the model on gpu; ` + + `on cpu: Failed to create LiteRT-LM engine for ${path('no-engine')}: INTERNAL: cannot load the model on cpu')` }) + // A backend the request or the config sets is used as set, and not replaced when it fails. + assert.deepEqual(await engines(`${readable('gpu-first')},cpu`), { status: 200, error: undefined, created: ['cpu'] }) + const failed = await engines(`${readable('gpu-broken')},gpu`) + assert.deepEqual(failed, { status: 500, error: `Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for ${path('gpu-broken')}: UNAVAILABLE: no usable OpenCL device')`, created: ['gpu'] }) + const lines = await logged(fake) + assert.ok(lines.includes("Using gpu-first.litertlm's own backend: gpu"), fake.stderr) + assert.ok(lines.includes("gpu-broken.litertlm's own backend, gpu, did not start (Failed to create LiteRT-LM engine for " + + `${path('gpu-broken')}: UNAVAILABLE: no usable OpenCL device); falling back to cpu`), fake.stderr) + // The sampler follows the backend the engine runs on: CPU's no-limit top_k, not the GPU's (the vocabulary). + assert.equal((await samplerFor(fake, { model: readable('gpu-broken'), messages: hello, top_k: 0 })).top_k, NO_LIMIT) + // So does the context window with no size asked: the CPU's 8192 for a model that fell back to it, not the GPU's 4096. + const windowOf = async (ref) => { + const reply = await chat(fake, { model: ref, messages: hello }) + assert.equal(reply.status, 200, reply.error) + return reply.headers.get('x-context-window') + } + assert.equal(await windowOf(readable('gpu-broken-ctx8192')), '8192') + assert.equal(await windowOf(readable('gpu-first-ctx8192')), '4096') + // A model that gets a new engine for each conversation (an LFM2 one) tries its own backend once, not every time. + const recurrent = litertlmFile(join(work, 'gpu-broken-lfm2.litertlm'), [tflite('tf_lite_prefill_decode', 'gpu,cpu'), llmMetadata(LFM2)]) + assert.deepEqual(await engines(recurrent), { status: 200, error: undefined, created: ['gpu', 'cpu'] }) + assert.deepEqual(await engines(recurrent), { status: 200, error: undefined, created: ['cpu'] }) + assert.deepEqual(await engines(recurrent), { status: 200, error: undefined, created: ['cpu'] }) +}) + +test('a developer message is a system message', { skip, timeout: 60_000 }, async () => { + let log = await callsDuring(async () => { + const reply = await chat(fake, { model: readable('plain'), messages: [{ role: 'developer', content: 'Answer in one word.' }, ...hello] }) + assert.equal(reply.status, 200, reply.error) + }) + assert.deepEqual(log.filter((c) => c.call === 'conversation_create').at(-1).config.messages, [{ role: 'system', content: 'Answer in one word.' }]) + log = await callsDuring(async () => { + const reply = await chat(fake, { model: readable('plain'), messages: [{ role: 'developer', content: [{ type: 'text', text: 'Be brief.' }] }] }) + assert.equal(reply.status, 200, reply.error) + }) + assert.deepEqual(log.filter((c) => c.call === 'send_message').at(-1).message, { role: 'system', content: [{ type: 'text', text: 'Be brief.' }] }) + assert.equal((await logged(fake)).filter((l) => l === 'the developer role is read as system, its older name').length, 1) +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model follows a developer message as it follows a system one: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ask = async (role) => { + const reply = await chat(server, { model: `${path},cpu`, messages: [{ role, content: 'Answer in one word.' }, { role: 'user', content: 'What is the capital of France?' }], max_completion_tokens: 24, temperature: 0, reasoning_effort: 'none' }, 300_000) + assert.equal(reply.status, 200, reply.error) + return { content: reply.content, prompt: reply.usage.prompt_tokens } + } + assert.deepEqual(await ask('developer'), await ask('system')) + } finally { + stopServer(server) + } + }) +} + +test('max_completion_tokens or max_tokens caps the reply, and below 1 is a 400', { skip, timeout: 60_000 }, async () => { + const cap = async (request) => { + const log = await callsDuring(async () => { + const reply = await chat(fake, { model: readable('plain'), messages: hello, ...request }) + assert.equal(reply.status, 200, reply.error) + }) + return log.filter((c) => c.call === 'send_message').at(-1).args.max_output_tokens ?? null + } + assert.equal(await cap({}), null) + assert.equal(await cap({ max_completion_tokens: 7 }), 7) + assert.equal(await cap({ max_tokens: 5 }), 5) + assert.equal(await cap({ max_completion_tokens: 7, max_tokens: 5 }), 7) + assert.equal(await cap({ max_completion_tokens: null, max_tokens: 5 }), 5) + for (const [request, error] of [ + [{ max_completion_tokens: 0 }, 'max_completion_tokens must be an integer of at least 1, got 0'], + [{ max_completion_tokens: -5 }, 'max_completion_tokens must be an integer of at least 1, got -5'], + [{ max_completion_tokens: 1.5 }, 'max_completion_tokens must be an integer of at least 1, got 1.5'], + [{ max_tokens: 0 }, 'max_tokens must be an integer of at least 1, got 0'], + ]) { + assert.deepEqual(await chat(fake, { model: readable('plain'), messages: hello, ...request }), { status: 400, error }) + } +}) + +for (const path of realModels.length ? realModels : ['']) { + test(`a real model's reply is capped by max_tokens, and max_completion_tokens 0 is a 400: ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { + const config = join(work, 'no-cache.json') + writeFileSync(config, JSON.stringify({ default: { cache: 'no' } })) + const server = await startServer(binary, { ...process.env, LITERT_LM_LIBRARY: realLibrary }, ['--config', config]) + try { + const ask = (extra) => chat(server, { model: `${path},cpu`, messages: [{ role: 'user', content: 'Write two sentences about the sea.' }], temperature: 0, reasoning_effort: 'none', ...extra }, 300_000) + const capped = await ask({ max_tokens: 5 }) + assert.equal(capped.status, 200, capped.error) + assert.ok(capped.usage.completion_tokens <= 5, `${capped.usage.completion_tokens} tokens`) + assert.equal(capped.finish, 'length') + assert.equal((await ask({ max_completion_tokens: 0 })).status, 400) + } finally { + stopServer(server) + } + }) +} + +test('n above 1, logprobs and top_logprobs, which LiteRT-LM cannot give, are 400s that name the field', { skip, timeout: 60_000 }, async () => { + const model = readable('plain') + for (const ok of [{ n: 1 }, { n: null }, { logprobs: false }, { top_logprobs: 0 }, { logprobs: null }]) { + assert.equal((await chat(fake, { model, messages: hello, ...ok })).status, 200, JSON.stringify(ok)) + } + for (const [request, error] of [ + [{ n: 2 }, 'Unsupported parameter: n 2: this server gives one choice per request'], + [{ n: 0 }, 'Unsupported parameter: n 0: this server gives one choice per request'], + [{ logprobs: true }, 'Unsupported parameter: logprobs true: LiteRT-LM gives no token log-probabilities'], + [{ logprobs: true, top_logprobs: 3 }, 'Unsupported parameter: logprobs true: LiteRT-LM gives no token log-probabilities'], + [{ top_logprobs: 3 }, 'Unsupported parameter: top_logprobs 3: LiteRT-LM gives no token log-probabilities'], + ]) { + assert.deepEqual(await chat(fake, { model, messages: hello, ...request }), { status: 400, error }, JSON.stringify(request)) + } +}) + +test('a conversation that ends with the assistant\'s words is a 400 that says the reply cannot be continued', { skip, timeout: 60_000 }, async () => { + const model = readable('plain') + const error = "Invalid messages: the last message is the assistant's, and this server cannot continue a reply " + + "(LiteRT-LM's C API has no way to); end with a user or tool message" + for (const stream of [false, true]) { + for (const last of [ + { role: 'assistant', content: 'The capital of France is' }, + { role: 'assistant', content: [{ type: 'text', text: 'The' }] }, + // with no calls, as clients that write out every optional field send a plain reply + { role: 'assistant', content: 'The capital of France is', tool_calls: null }, + { role: 'assistant', content: 'The capital of France is', tool_calls: [] }, + ]) { + const reply = await chat(fake, { model, messages: [{ role: 'user', content: 'What is the capital of France?' }, last], stream }) + assert.deepEqual(reply, { status: 400, error }) + } + } + // An assistant turn of tool calls, and the history's own assistant turns, are as before. + const calls = { role: 'assistant', content: null, tool_calls: [{ id: 'c1', type: 'function', function: { name: 'get_weather', arguments: '{}' } }] } + assert.equal((await chat(fake, { model, messages: [...hello, calls] })).status, 200) + assert.equal((await chat(fake, { model, messages: [...hello, { role: 'assistant', content: 'Hi.' }, ...hello] })).status, 200) +}) diff --git a/engines/litert-lm-server/parent-death.test.mjs b/engines/litert-lm-server/parent-death.test.mjs new file mode 100644 index 0000000..b5c3069 --- /dev/null +++ b/engines/litert-lm-server/parent-death.test.mjs @@ -0,0 +1,133 @@ +// Tests that the LiteRT-LM server ends with the process that started it (src/main.cc, the top of main): +// node --test engines/litert-lm-server/parent-death.test.mjs +// The server is built for this machine with c++ and runs against a stand-in liblitert-lm.so whose C API functions do +// nothing, which is enough for it to start listening: no model is loaded. Linux only (PR_SET_PDEATHSIG and /proc), and +// skipped without a C and a C++ compiler. Building takes a few seconds. Every process a test starts is killed by pid +// when it ends. +import assert from 'node:assert/strict' +import { spawn, spawnSync } from 'node:child_process' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { after, before, test } from 'node:test' +import { fileURLToPath } from 'node:url' + +const root = fileURLToPath(new URL('.', import.meta.url)) +const work = mkdtempSync(join(tmpdir(), 'litert-lm-server-test-')) +after(() => rmSync(work, { recursive: true, force: true })) + +const missing = ['cc', 'c++'].filter((c) => spawnSync(c, ['--version'], { stdio: 'ignore' }).error) +const skip = process.platform !== 'linux' ? 'PR_SET_PDEATHSIG and /proc are Linux only' + : missing.length ? `no ${missing.join(' or ')} on this machine` : false +const server = join(work, 'litertlm_server') + +before(() => { + if (skip) return + // The stand-in library: every function the server resolves (src/litert_lm_api.h), each doing nothing. Beside the + // server, where LoadLiteRtLm looks first after LITERT_LM_LIBRARY, as in the app's nativeLibraryDir. + const header = readFileSync(join(root, 'src', 'litert_lm_api.h'), 'utf8') + const names = [...header.matchAll(/^\s*X\((\w+)\)/gm)].map((m) => m[1]) + assert.ok(names.length > 40, 'the C API list in litert_lm_api.h was not found') + writeFileSync(join(work, 'stub.c'), names.map((n) => `void ${n}(void) {}\n`).join('')) + const lib = spawnSync('cc', ['-shared', '-fPIC', '-o', join(work, 'liblitert-lm.so'), join(work, 'stub.c')], { encoding: 'utf8' }) + assert.equal(lib.status, 0, lib.stderr) + const built = spawnSync('c++', [ + '-std=c++17', '-O0', '-w', '-I', join(root, 'third_party', 'litert_lm'), '-I', join(root, 'third_party'), + '-DLITERT_LM_VERSION="test"', join(root, 'src', 'main.cc'), join(root, 'src', 'litert_lm_api.cc'), + '-ldl', '-pthread', '-o', server, + ], { encoding: 'utf8' }) + assert.equal(built.status, 0, built.stderr) +}) + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)) +function stat(pid) { + try { + const s = readFileSync(`/proc/${pid}/stat`, 'utf8') + const f = s.slice(s.lastIndexOf(')') + 2).split(' ') + return { state: f[0], ppid: Number(f[1]) } + } catch { + return null + } +} +// A zombie counts as gone: it holds no model and no port. +const gone = (pid) => { + const s = stat(pid) + return !s || s.state === 'Z' +} +async function waitFor(check, ms) { + for (const end = Date.now() + ms; Date.now() < end; await sleep(20)) if (check()) return true + return check() +} +function killAll(pids) { + for (const pid of pids) { + try { + process.kill(pid, 'SIGKILL') + } catch {} + } +} +function firstLine(stream) { + return new Promise((resolve, reject) => { + let buf = '' + stream.setEncoding('utf8') + stream.on('data', (d) => { + buf += d + if (buf.includes('\n')) resolve(buf.slice(0, buf.indexOf('\n'))) + }) + stream.on('end', () => reject(new Error(`no line before the end (got ${JSON.stringify(buf)})`))) + }) +} + +test('a daemon killed with SIGKILL takes the server it started with it', { skip }, async () => { + const pids = [] + try { + // The stand-in daemon starts the server directly, as a daemon without libtether_exec.so does, and prints its pid + // once the server says it is listening. + const daemon = spawn(process.execPath, ['-e', ` + const c = require('child_process').spawn(process.argv[1], ['serve', '--host', '127.0.0.1', '--port', '0'], { stdio: ['ignore', 'ignore', 'pipe'] }) + let err = '' + c.stderr.on('data', (d) => { err += d; if (err.includes('Starting OpenAI-compatible API server')) console.log(c.pid) }) + c.on('exit', (code) => { console.error('server exited ' + code + ': ' + err); process.exit(1) }) + setInterval(() => {}, 1000)`, server], { stdio: ['ignore', 'pipe', 'inherit'] }) + pids.push(daemon.pid) + const pid = Number(await firstLine(daemon.stdout)) + pids.push(pid) + assert.equal(stat(pid)?.ppid, daemon.pid) + process.kill(daemon.pid, 'SIGKILL') + assert.ok(await waitFor(() => gone(pid), 1000), 'the server outlived the daemon') + } finally { + killAll(pids) + } +}) + +test('a server whose parent exited before it started does not start', { skip }, async (t) => { + const pids = [] + try { + const errFile = join(work, 'orphan.err') + // The outer sh starts an inner one in the background, prints its pid and exits, so the inner sh is reparented + // before it execs the server, as a server is whose daemon dies between the fork and the prctl. The inner sh waits + // on fd 3 until this test has seen who adopted it, so it can neither exec the server nor be gone before then. + const outer = spawn('sh', ['-c', 'sh -c "$2" "$0" >/dev/null 2>"$1" & echo $!', server, errFile, + 'read -r _ <&3; exec "$0" serve --host 127.0.0.1 --port 0 3<&-'], { stdio: ['ignore', 'pipe', 'ignore', 'pipe'] }) + pids.push(outer.pid) + const pid = Number(await firstLine(outer.stdout)) + pids.push(pid) + assert.ok(await waitFor(() => gone(outer.pid), 1000)) + const ppid = stat(pid)?.ppid + assert.ok(ppid !== undefined, 'the orphaned sh was gone before it was let go') + if (ppid !== 1) { + t.skip(`an orphan is adopted by pid ${ppid} here, not init`) + return + } + outer.stdio[3].end('go\n') + assert.ok(await waitFor(() => gone(pid), 3000), 'the server started') + assert.match(readFileSync(errFile, 'utf8'), /the process that started turbollm-litertlm-server has already exited; not starting/) + } finally { + killAll(pids) + } +}) + +test('--version still answers', { skip }, () => { + const r = spawnSync(server, ['--version'], { encoding: 'utf8' }) + assert.equal(r.status, 0, r.stderr) + assert.match(r.stdout, /^turbollm-litertlm-server .* \(LiteRT-LM test\)/) +}) diff --git a/engines/litert-lm-server/src/litert_lm_api.h b/engines/litert-lm-server/src/litert_lm_api.h index 992bcf2..89abf2d 100644 --- a/engines/litert-lm-server/src/litert_lm_api.h +++ b/engines/litert-lm-server/src/litert_lm_api.h @@ -30,6 +30,7 @@ X(litert_lm_conversation_config_delete) \ X(litert_lm_conversation_config_set_constraint_provider) \ X(litert_lm_conversation_config_set_enable_constrained_decoding) \ + X(litert_lm_conversation_config_set_extra_context) \ X(litert_lm_conversation_config_set_messages) \ X(litert_lm_conversation_config_set_session_config) \ X(litert_lm_conversation_config_set_thinking_config) \ @@ -41,6 +42,7 @@ X(litert_lm_conversation_optional_args_delete) \ X(litert_lm_conversation_optional_args_set_constraint) \ X(litert_lm_conversation_optional_args_set_max_output_tokens) \ + X(litert_lm_conversation_optional_args_set_repetition_penalty_config) \ X(litert_lm_conversation_send_message_stream) \ X(litert_lm_engine_create) \ X(litert_lm_engine_delete) \ @@ -54,6 +56,12 @@ X(litert_lm_engine_settings_set_num_threads) \ X(litert_lm_engine_settings_set_use_ringbuffers_local_attention) \ X(litert_lm_get_last_error_message) \ + X(litert_lm_repetition_penalty_config_create) \ + X(litert_lm_repetition_penalty_config_delete) \ + X(litert_lm_repetition_penalty_config_set_frequency_penalty) \ + X(litert_lm_repetition_penalty_config_set_presence_penalty) \ + X(litert_lm_repetition_penalty_config_set_repetition_penalty) \ + X(litert_lm_repetition_penalty_config_set_window_size) \ X(litert_lm_sampler_params_create) \ X(litert_lm_sampler_params_delete) \ X(litert_lm_sampler_params_set_seed) \ diff --git a/engines/litert-lm-server/src/main.cc b/engines/litert-lm-server/src/main.cc index ac8b24c..d779ad7 100644 --- a/engines/litert-lm-server/src/main.cc +++ b/engines/litert-lm-server/src/main.cc @@ -13,8 +13,8 @@ // The model loads lazily on the first chat request and stays loaded while requests keep // naming the same model and settings (util.get_or_initialize_server_engine). // - --config is the CLI's config file: {"default": {...}, "models": {"": {...}}}, of -// which the serve path reads backend, cpu_thread_count, max_num_tokens, cache, -// speculative_decoding, gpu_decode_steps_per_sync, thinking and thinking_budget. +// which the serve path reads backend, vision_backend, audio_backend, cpu_thread_count, +// max_num_tokens, cache, speculative_decoding, gpu_decode_steps_per_sync, thinking and thinking_budget. // - A failed request reports the runtime's own message in the HTTP status line, as // BaseHTTPRequestHandler.send_error does, because that is where the daemon's warm-up reads it. // @@ -23,10 +23,18 @@ // - `timings` (llama.cpp's field names) on the final response, taken from the runtime's own // benchmark counters, so speed is measured rather than derived from time-to-first-token. // - Thinking-channel text streamed as `reasoning_content` instead of being dropped. -// - "length" as the finish reason when the token budget runs out. +// - "length" as the finish reason when the reply is cut off: by max_completion_tokens, or by the +// context window LiteRT-LM gives the engine, which each reply's X-Context-Window header names. // - A client that disconnects cancels its generation instead of letting it run on. // - Requests are served concurrently; only inference itself is serialized (the engine is one // shared resource, exactly as in the Python server, which serialized everything). +// - Constrained decoding only for requests with tools, and none for a model that refuses it, so +// LFM2 models answer at all (the Python server fails every request to one). A response_format +// the model cannot follow, or whose regex or schema cannot be compiled, is a 400, not a 500. +// - A new engine for each conversation of a model whose decoder keeps a recurrent state (LFM2), which the runtime +// would otherwise carry from one conversation into the next (see GetOrCreateEngine). +// - An image or audio part sent to a model that takes no such input is a 400 that says so, not the runtime's 500. +// - The developer role, OpenAI's newer name for system, is read as system. // // Not translated: /v1/responses, /v1/embeddings and the Gemini API (TurboLLM uses none of them // with this engine), and image URLs other than base64 data: URLs (remote URLs, and local paths, which the Python @@ -43,10 +51,15 @@ #include #include #include +#ifdef __linux__ +#include +#endif +#include #include #include #include +#include #include #include #include @@ -55,10 +68,13 @@ #include #include #include +#include #include #include #include #include +#include +#include #include #include #include @@ -139,7 +155,7 @@ std::string AbsPath(const std::string& p) { // ── Config (litert_lm_cli/config.py) ───────────────────────────────────────────────────── struct ModelConfig { - std::optional backend; + std::optional backend, vision_backend, audio_backend; std::optional cpu_thread_count; std::optional max_num_tokens; std::optional cache; @@ -150,6 +166,8 @@ struct ModelConfig { void MergeFrom(const ModelConfig& o) { if (o.backend) backend = o.backend; + if (o.vision_backend) vision_backend = o.vision_backend; + if (o.audio_backend) audio_backend = o.audio_backend; if (o.cpu_thread_count) cpu_thread_count = o.cpu_thread_count; if (o.max_num_tokens) max_num_tokens = o.max_num_tokens; if (o.cache) cache = o.cache; @@ -176,6 +194,8 @@ ModelConfig ParseModelConfig(const json& j) { return std::nullopt; }; c.backend = str("backend"); + c.vision_backend = str("vision_backend"); + c.audio_backend = str("audio_backend"); c.cpu_thread_count = num("cpu_thread_count"); c.max_num_tokens = num("max_num_tokens"); c.cache = str("cache"); @@ -267,14 +287,421 @@ ModelRef ParseModelParameter(const std::string& raw) { return ref; } +// ── What the model file says about itself ───────────────────────────────────────────────── +// After the 32-byte preamble a .litertlm has a FlatBuffer header (schema/core/litertlm_header_schema.fbs in LiteRT-LM +// 0.18.0) that lists its sections with their byte ranges in the file. Two are small protobufs the runtime reads at load: +// LlmMetadata (runtime/proto/llm_metadata.proto) and ExecutorMetadata (executor_metadata.proto). + +struct ModelFileInfo { + // The decoder keeps a recurrent state besides its KV cache: LFM2's short-convolution cache (the kv_cache_c_* inputs), + // or any linear-attention state. See GetOrCreateEngine for why that matters. + bool recurrent_state = false; + // What the header's TFLite sections say, when the header could be read: for text, image and audio input, the + // backends the file's decoder or encoder supports, the one it prefers first; none when it takes no such input. + bool capabilities_known = false; + std::vector text_backends, vision_backends, audio_backends; + // The model's own sampler (LlmMetadata's sampler_params): type UNSPECIFIED and zeros for a file that has none, as + // LFM2.5-230M and gemma-3-270m-it are, as the C API's litert_lm_loaded_file_sampler_* also answer. + LiteRtLmSamplerType sampler_type = kLiteRtLmSamplerTypeUnspecified; + int top_k = 0; + float top_p = 0, temperature = 0; + // Its text decoder is an Artisan one (a tf_lite_artisan_text_decoder section), which LiteRT-LM runs on GPU as + // GPU_ARTISAN (engine_settings.cc), a backend whose sampling is the model's own. + bool artisan = false; + // The number of logits its decode signature outputs: its vocabulary. 0 when unknown. + int vocabulary = 0; + // What LlmMetadata says about thinking: its supports_thinking (field 11, absent from most files: LFM2.5-230M, + // Qwen3-0.6B, Gemma 4), whether its Jinja template has the enable_thinking switch (Qwen3's and Gemma 4's do), and + // whether it has a channel, the thinking one being the first (Conversation::CreateDecodeConfig). + std::optional supports_thinking; + bool thinking_switch = false, channel = false; + // The context the file takes, as the C API's litert_lm_loaded_file_max_context_tokens and _is_dynamic_context read + // it: a static model's fixed size, or a dynamic one's magic number (see RuntimeWindow); 0 when unknown. And + // LlmMetadata's own max_num_tokens, which the runtime starts from when it is not given one: 0 when it has none. + int max_context = 0, metadata_context = 0; + bool dynamic_context = false; +}; + +// Bounds-checked reads of a FlatBuffer of `size` bytes, through `read`: just what the server needs of the header's +// section list. Little-endian, as are the FlatBuffer format and every ABI the server is built for. +struct FbReader { + std::function read; // called only within the bounds + uint64_t size; + template + std::optional Get(uint64_t pos) const { + T v; + if (pos > size || sizeof(T) > size - pos || !read(pos, &v, sizeof v)) return std::nullopt; + return v; + } + // Where field i of the table at `table` is, or 0 when the table does not have it. + uint64_t Field(uint64_t table, int i) const { + auto back = Get(table); + if (!back || static_cast(table) - *back < 0) return 0; + uint64_t vtable = static_cast(static_cast(table) - *back); + auto vsize = Get(vtable); + if (!vsize || 4u + 2u * i + 2u > *vsize) return 0; + auto at = Get(vtable + 4 + 2 * i); + return at && *at ? table + *at : 0; + } + // The table or vector a uoffset at `pos` points to, or 0. + uint64_t Follow(uint64_t pos) const { + auto off = pos ? Get(pos) : std::nullopt; + return off && *off ? pos + *off : 0; + } + // The number of elements of the vector at `vec`, 0 when there is none. + uint32_t Length(uint64_t vec) const { return vec ? Get(vec).value_or(0) : 0; } + // The scalar at `pos` (a field's place, from Field), or `missing` when the table does not have the field. + template + T Scalar(uint64_t pos, T missing) const { + return pos ? Get(pos).value_or(missing) : missing; + } + // The string a uoffset at `pos` points to (its first 256 bytes), empty when there is none. + std::string String(uint64_t pos) const { + const uint64_t at = Follow(pos); + std::string out(std::min(Length(at), 256), '\0'); + if (!at || at + 4 > size || out.size() > size - at - 4 || !read(at + 4, &out[0], out.size())) return ""; + return out; + } +}; + +FbReader MemoryReader(const std::string& b) { + return {[&b](uint64_t pos, void* out, size_t n) { + std::memcpy(out, b.data() + pos, n); + return true; + }, + b.size()}; +} + +// Calls f(field number, value, bytes) for each top-level field of a protobuf message: the varint for wire type 0, the +// 32 bits for wire type 5 (a float's), the bytes for wire type 2; 64-bit fields are skipped. False when the message is +// malformed. +template +bool ForEachProtoField(const std::string& m, F f) { + size_t i = 0; + auto varint = [&](uint64_t& v) { + v = 0; + for (int shift = 0; shift < 64; shift += 7) { + if (i >= m.size()) return false; + unsigned char c = static_cast(m[i++]); + v |= static_cast(c & 0x7f) << shift; + if (c < 0x80) return true; + } + return false; + }; + while (i < m.size()) { + uint64_t key, v; + if (!varint(key)) return false; + const int field = static_cast(key >> 3); + switch (key & 7) { + case 0: + if (!varint(v)) return false; + f(field, v, std::string()); + break; + case 2: + if (!varint(v) || v > m.size() - i) return false; + f(field, 0, m.substr(i, v)); + i += v; + break; + case 1: + i += 8; + break; + case 5: + if (m.size() - i < 4) return false; + v = 0; + for (int b = 3; b >= 0; --b) v = (v << 8) | static_cast(m[i + b]); + f(field, v, std::string()); + i += 4; + break; + default: + return false; + } + if (i > m.size()) return false; + } + return true; +} + +// LlmMetadata names the model type LFM2 (llm_model_type, field 6, with its lfm2 member, field 11), or ExecutorMetadata +// declares a state buffer that is a linear-attention state (StateBuffer.type 5) or LFM2's conv cache (kv_cache_c_*, +// which the runtime keeps among its KV buffers: GetKVCacheRootNames in litert_compiled_model_executor_utils.cc). The +// type alone misses files packed with generic_model, as litert-community's LFM2.5-230M was before 2026-10-14 and a +// third-party LFM2.5-VL-3B is; their ExecutorMetadata still declares the conv caches. +bool HasRecurrentState(int section_type, const std::string& proto) { + bool found = false; + if (section_type == 5) { // LlmMetadataProto + ForEachProtoField(proto, [&](int field, uint64_t, const std::string& type) { + if (field == 6) ForEachProtoField(type, [&](int member, uint64_t, const std::string&) { found |= member == 11; }); + }); + } else if (section_type == 9) { // ExecutorMetadataProto + ForEachProtoField(proto, [&](int field, uint64_t, const std::string& llm) { + if (field != 1) return; + ForEachProtoField(llm, [&](int field, uint64_t, const std::string& buffer) { + if (field != 2) return; // state_buffers + ForEachProtoField(buffer, [&](int field, uint64_t v, const std::string& name) { + found |= (field == 6 && v == 5) || ((field == 1 || field == 3) && name.rfind("kv_cache_c_", 0) == 0); + }); + }); + }); + } + return found; +} + +// LlmMetadata's sampler_params (field 4, a SamplerParameters: type 1, k 2, p 3, temperature 4), when it has them. +void ReadSampler(const std::string& llm_metadata, ModelFileInfo& info) { + auto as_float = [](uint64_t bits) { + const uint32_t b = static_cast(bits); + float f; + std::memcpy(&f, &b, sizeof f); + return f; + }; + ForEachProtoField(llm_metadata, [&](int field, uint64_t, const std::string& params) { + if (field != 4) return; + ForEachProtoField(params, [&](int field, uint64_t v, const std::string&) { + if (field == 1) info.sampler_type = static_cast(v); + if (field == 2) info.top_k = static_cast(v); + if (field == 3) info.top_p = as_float(v); + if (field == 4) info.temperature = as_float(v); + }); + }); +} + +// LlmMetadata's max_num_tokens (field 5): the model's context, which a prime number above 10 marks as dynamic, as +// LiteRT-LM reads it (IsMagicNumber in schema/model_info/model_info.cc). +void ReadContext(const std::string& llm_metadata, ModelFileInfo& info) { + ForEachProtoField(llm_metadata, [&](int field, uint64_t v, const std::string&) { + if (field == 5) info.max_context = info.metadata_context = static_cast(std::min(v, INT32_MAX)); + }); +} + +bool IsMagicNumber(int64_t n) { + if (n < 11 || n % 2 == 0) return false; + for (int64_t i = 3; i * i <= n; i += 2) { + if (n % i == 0) return false; + } + return true; +} + +void ReadThinking(const std::string& llm_metadata, ModelFileInfo& info) { + ForEachProtoField(llm_metadata, [&](int field, uint64_t v, const std::string& bytes) { + if (field == 11) info.supports_thinking = v != 0; + if (field == 7) info.thinking_switch = bytes.find("enable_thinking") != std::string::npos; + if (field == 8) info.channel = true; + }); +} + +// What the server needs of the model's main TFLite graph (a tflite::Model FlatBuffer: subgraphs 2, signature_defs 7; +// SignatureDef inputs 0, outputs 1, signature_key 2, subgraph_index 4; TensorMap name 0, tensor_index 1; SubGraph +// tensors 0; Tensor shape 0), read through `m` where it lies in the file, a few tables of it and never the whole: the +// number of logits the decode signature outputs (the last dimension of its "logits", which is what the runtime sizes +// its sampler by: GetVocabSize in llm_litert_compiled_model_executor.cc); and, for a file whose LlmMetadata gives no +// max_num_tokens, the context: the last dimension of the first mask input of the first prefill signature that has one +// (ExtractContextTokensFromTfliteBuffer in schema/model_info/model_info.cc, which the C API reads it with). +void ReadTflite(const FbReader& m, ModelFileInfo& info) { + const uint64_t model = m.Get(0).value_or(0); // the root table's uoffset, from 0 + const uint64_t subgraphs = m.Follow(m.Field(model, 2)), signatures = m.Follow(m.Field(model, 7)); + // The last dimension of the tensor that `entry`, one of the signature's inputs or outputs, names; 0 if none. + auto last_dimension = [&](uint64_t signature, uint64_t entry) { + const uint32_t subgraph = m.Scalar(m.Field(signature, 4), 0); + if (subgraph >= m.Length(subgraphs)) return 0; + const uint64_t tensors = m.Follow(m.Field(m.Follow(subgraphs + 4 + 4ull * subgraph), 0)); + const uint32_t tensor = m.Scalar(m.Field(entry, 1), 0); + if (tensor >= m.Length(tensors)) return 0; + const uint64_t shape = m.Follow(m.Field(m.Follow(tensors + 4 + 4ull * tensor), 0)); + const uint32_t rank = m.Length(shape); + return rank ? m.Get(shape + 4 + 4ull * (rank - 1)).value_or(0) : 0; + }; + const bool want_context = info.max_context == 0; + for (uint32_t k = 0; k < m.Length(signatures) && k < 256; ++k) { + const uint64_t signature = m.Follow(signatures + 4 + 4ull * k); + const std::string key = signature ? m.String(m.Field(signature, 2)) : ""; + if (want_context && info.max_context == 0 && key.rfind("prefill", 0) == 0) { + const uint64_t inputs = m.Follow(m.Field(signature, 0)); + for (uint32_t j = 0; j < m.Length(inputs) && j < 4096 && info.max_context == 0; ++j) { + const uint64_t input = m.Follow(inputs + 4 + 4ull * j); + if (input && m.String(m.Field(input, 0)).find("mask") != std::string::npos) { + info.max_context = std::max(0, last_dimension(signature, input)); + } + } + } + if (key != "decode") continue; + const uint64_t outputs = m.Follow(m.Field(signature, 1)); + for (uint32_t j = 0; j < m.Length(outputs) && j < 4096; ++j) { + const uint64_t output = m.Follow(outputs + 4 + 4ull * j); + if (output && m.String(m.Field(output, 0)) == "logits") { + info.vocabulary = std::max(0, last_dimension(signature, output)); + } + } + } +} + +// The backends a section's backend_constraint lists, in its order, as LiteRT-LM 0.18.0 reads it +// (ParseBackendConstraint in schema/model_info/model_info.cc, what its C API's litert_lm_loaded_file_* answer): cpu +// or cpu_artisan, gpu or gpu_artisan, npu or google_tensor_artisan; a section without one takes cpu, then gpu. +std::vector ConstraintBackends(const std::string& constraint) { + if (constraint.empty()) return {"cpu", "gpu"}; + std::vector out; + std::stringstream ss(constraint); + for (std::string b; std::getline(ss, b, ',');) { + b = Trim(b); + std::string name = b == "cpu" || b == "cpu_artisan" ? "cpu" + : b == "gpu" || b == "gpu_artisan" ? "gpu" + : b == "npu" || b == "google_tensor_artisan" ? "npu" + : ""; + if (!name.empty() && std::find(out.begin(), out.end(), name) == out.end()) out.push_back(name); + } + return out; +} + +// `header` is the file's bytes from 32 to the header's end, `in` the file. What the file takes and on which backends is +// what its header lists, as LiteRT-LM 0.18.0 reads it (DiscoverSections in schema/model_info/model_info.cc): each +// TFLite section's model_type and backend_constraint items, a vision or audio encoder or adapter being that input, +// the last one listed deciding its backends. The C API's litert_lm_loaded_file_create reads the same, but also reads +// whole sections into memory: every vision section, and the main model when LlmMetadata has no max_num_tokens +// (GetModelInfo), so asking it which inputs gemma-3-270m-it takes raised the server's peak memory by 285 MB, the size +// of its model, before any engine loaded, and by up to 2 GB for a larger bundle. Here nothing but the header, the small +// protobuf sections and a few tables of the main model (ReadTflite) is read. A header or section this cannot read says +// nothing. +ModelFileInfo ReadModelFileInfo(std::ifstream& in, const std::string& header) { + ModelFileInfo info; + const FbReader fb = MemoryReader(header); + const uint64_t root = fb.Get(0).value_or(0); // the root table's uoffset, from 0 + const uint64_t sections = fb.Follow(fb.Field(fb.Follow(fb.Field(root, 1)), 0)); // section_metadata.objects + const uint32_t n = fb.Length(sections); + info.capabilities_known = n > 0; + std::optional vision, audio; // the encoder's backend_constraint + std::string text; // the decoder's, or the text encoder's + bool npu_aux = false; // an NPU auxiliary model, which makes NPU a text backend too + uint64_t main_begin = 0, main_end = 0; // the main TFLite model, as DiscoverSections picks it + bool main_found = false, other_found = false; + for (uint32_t k = 0; k < n && k < 4096; ++k) { + const uint64_t s = fb.Follow(sections + 4 + 4ull * k); + if (!s) continue; + const int type = fb.Scalar(fb.Field(s, 3), 0); + if (type == 3) { // TFLiteModel + std::string model_type, constraint; + const uint64_t items = fb.Follow(fb.Field(s, 0)); + for (uint32_t j = 0; j < fb.Length(items) && j < 256; ++j) { + const uint64_t item = fb.Follow(items + 4 + 4ull * j); // a KeyValuePair, whose value is a StringValue (9) + if (!item || fb.Scalar(fb.Field(item, 1), 0) != 9) continue; + const std::string key = fb.String(fb.Field(item, 0)); + const std::string value = Lower(Trim(fb.String(fb.Field(fb.Follow(fb.Field(item, 2)), 0)))); + if (key == "model_type") model_type = value; + if (key == "backend_constraint") constraint = value; + } + // ClassifyModelType's kinds: the main model is the last main section, or else the first that is no other kind. + const uint64_t begin = fb.Scalar(fb.Field(s, 1), 0), end = fb.Scalar(fb.Field(s, 2), 0); + const bool main = model_type == "tf_lite_prefill_decode" || model_type == "tf_lite_prefill_decode_hw" || + model_type == "tf_lite_prefill" || model_type == "tf_lite_decode" || + model_type == "tf_lite_artisan_text_decoder"; + const bool other = !main && model_type != "tf_lite_text_encoder" && model_type != "tf_lite_embedder" && + model_type != "tf_lite_vision_adapter" && model_type != "tf_lite_vision_encoder" && + model_type != "tf_lite_audio_adapter" && model_type != "tf_lite_audio_encoder_hw" && + model_type != "tf_lite_audio_frontend" && model_type != "tf_lite_video_adapter" && + model_type != "tf_lite_video_encoder" && model_type != "tf_lite_mtp_drafter" && + model_type != "tf_lite_aux"; + if (main || (other && !main_found && !other_found)) { + main_begin = begin; + main_end = end; + } + main_found |= main; + other_found |= other; + info.artisan |= model_type == "tf_lite_artisan_text_decoder"; + if (main || model_type == "tf_lite_text_encoder" || model_type == "tf_lite_embedder") text = constraint; + npu_aux |= model_type == "tf_lite_aux"; + if (model_type == "tf_lite_vision_adapter" || model_type == "tf_lite_vision_encoder") vision = constraint; + if (model_type == "tf_lite_audio_adapter" || model_type == "tf_lite_audio_encoder_hw" || + model_type == "tf_lite_audio_frontend") { + audio = constraint; + } + continue; + } + const uint64_t begin = fb.Scalar(fb.Field(s, 1), 0), end = fb.Scalar(fb.Field(s, 2), 0); + if ((type != 5 && type != 9) || end <= begin || end - begin > (1u << 20)) continue; + std::string proto(end - begin, '\0'); + in.clear(); + if (!in.seekg(static_cast(begin)) || !in.read(&proto[0], static_cast(proto.size()))) { + continue; + } + info.recurrent_state |= HasRecurrentState(type, proto); + if (type == 5) { + ReadSampler(proto, info); + ReadThinking(proto, info); + ReadContext(proto, info); + } + } + if (main_end > main_begin) { + const FbReader model{[&in, main_begin](uint64_t pos, void* out, size_t n) { + in.clear(); + return static_cast(in.seekg(static_cast(main_begin + pos)) && + in.read(static_cast(out), static_cast(n))); + }, + main_end - main_begin}; + ReadTflite(model, info); + } + info.dynamic_context = IsMagicNumber(info.max_context); + // The text's backends as the C API lists them (ConfigureSupportedBackendsAndNpu), except that an Artisan text decoder + // puts GPU first, as `litert-lm serve` does (model.model_default_backend): LiteRT-LM runs it on GPU only, switching + // GPU to GPU_ARTISAN for it (engine_settings.cc). + info.text_backends = ConstraintBackends(text); + if (npu_aux && std::find(info.text_backends.begin(), info.text_backends.end(), "npu") == info.text_backends.end()) { + info.text_backends.push_back("npu"); + } + if (info.artisan) { + std::vector& text_backends = info.text_backends; + text_backends.erase(std::remove(text_backends.begin(), text_backends.end(), "gpu"), text_backends.end()); + text_backends.insert(text_backends.begin(), "gpu"); + } + if (vision) info.vision_backends = ConstraintBackends(*vision); + if (audio) info.audio_backends = ConstraintBackends(*audio); + return info; +} + +// ModelFileInfo for the model file, read once per version of it (its size and modification time). A .litertlm starts +// with "LITERTLM", three uint32 versions, 4 bytes of padding and a little-endian uint64 header-end offset (the same 32 +// bytes the daemon's models/litertlm.ts reads). The runtime allocates and zeroes header_end - 32 bytes before reading +// the header, and the std::bad_alloc a bogus offset raises crosses the C API into std::terminate, killing the whole +// server, so the offset is bounded here first. The bound is the runtime's own: it reads the header from the file's +// first 16 KiB only (kLitertLmHeaderMaxSize, LiteRT-LM 0.18.0's litert_lm_loader.h), so a larger offset can never load, +// yet would still cost an allocation up to the file's size (hundreds of MB on a phone). Checked before the loaded +// engine is released, which a bad file then leaves in place. A file that can't be read or lacks the magic is left to +// the runtime, which reports it cleanly and more precisely (e.g. "Permission denied"), and says nothing here. Caller +// holds g_engine_mu. +const ModelFileInfo& FileInfo(const ModelRef& ref) { + struct Cached { + long long size = -1, mtime = -1; + ModelFileInfo info; + }; + static std::map cache; + struct stat st {}; + unsigned char h[32] = {}; + std::ifstream in(ref.path, std::ios::binary); + bool ok = ::stat(ref.path.c_str(), &st) == 0 && in.read(reinterpret_cast(h), sizeof h) && + std::memcmp(h, "LITERTLM", 8) == 0; + uint64_t header_end = 0; + for (int i = 31; i >= 24; --i) header_end = (header_end << 8) | h[i]; + constexpr uint64_t kRuntimeHeaderMaxSize = 16 * 1024; + if (ok && (header_end <= 32 || header_end > std::min(st.st_size, kRuntimeHeaderMaxSize))) { + throw HttpError(500, "Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for " + ref.path + + ": invalid .litertlm header')"); + } + Cached& c = cache[ref.path]; + if (c.size == static_cast(st.st_size) && c.mtime == static_cast(st.st_mtime)) return c.info; + c = Cached{static_cast(st.st_size), static_cast(st.st_mtime), ModelFileInfo()}; + if (!ok) return c.info; + std::string header(header_end - 32, '\0'); + if (in.read(&header[0], static_cast(header.size()))) c.info = ReadModelFileInfo(in, header); + return c.info; +} + // ── Engine lifetime (util.get_or_initialize_server_engine) ───────────────────────────────── struct EngineKey { std::string path, backend, cache; + std::string vision_backend, audio_backend; // empty: no vision or audio executor int max_num_tokens = -1, threads = -1, gpu_steps = -1, spec = -1; bool operator==(const EngineKey& o) const { - return path == o.path && backend == o.backend && cache == o.cache && max_num_tokens == o.max_num_tokens && - threads == o.threads && gpu_steps == o.gpu_steps && spec == o.spec; + return path == o.path && backend == o.backend && cache == o.cache && vision_backend == o.vision_backend && + audio_backend == o.audio_backend && max_num_tokens == o.max_num_tokens && threads == o.threads && + gpu_steps == o.gpu_steps && spec == o.spec; } }; @@ -282,6 +709,19 @@ struct EngineHolder { LiteRtLmEngine* engine = nullptr; EngineKey key; double init_seconds = 0; + // Set once a conversation with constrained decoding failed to be created for this engine's model and one without + // it was created: its data processor cannot constrain tool calls (see HandleChatCompletions), so its later requests + // with tools are created without it at once. Cleared when another engine is loaded. + bool tools_unconstrained = false; + // What the model file says about itself (ModelFileInfo), as it was when this engine was created. + ModelFileInfo info; + // The backend the engine runs on: key.backend, or CPU when the model's own backend did not start. + std::string backend; + // The context window this engine has, in tokens (-1: unknown), and the larger one that was asked for, if any. + int window = -1, window_asked = -1; + // The model's decoder keeps a recurrent state (info.recurrent_state), and a message has been sent on this engine + // since it was created, so that state is no longer the initial one. + bool state_used = false; }; std::mutex g_engine_mu; // held for the whole of every inference: the engine is one shared resource @@ -327,53 +767,202 @@ void ReleaseEngine() { } } +// The backend for a model's vision or audio encoder: the config's vision_backend or audio_backend when it sets one, +// else the one the file prefers, as `litert-lm serve` chooses them (util.get_or_initialize_server_engine and +// model.parse_backend: the config key, else the encoder section's first backend_constraint, else cpu, which is also +// what ReadModelFileInfo lists first). Empty for a model without that input: LiteRT-LM then creates no executor for it. +std::string EncoderBackend(const ModelRef& ref, const std::vector& supported, + const std::optional& configured, const char* input) { + if (supported.empty()) { + if (configured) { + static std::set told; + if (told.insert(ref.path + input).second) { + Log(std::string(input) + "_backend in the config is not used for " + ref.id + ": it takes no " + input + + " input"); + } + } + return ""; + } + if (!configured) return supported.front(); + std::string b = Lower(*configured); + if (b != "cpu" && b != "gpu" && b != "npu") { + throw HttpError(500, "Failed to load engine: ValueError(\"Unsupported " + std::string(input) + " backend: '" + b + + "'\")"); + } + return b; +} + +// What a request's messages hold besides text, which only a model with that input can take. +struct InputNeeds { + bool image = false, audio = false; +}; + +// The context window LiteRT-LM 0.18.0 gives an engine of this model on `backend` when `asked` tokens are asked for +// (0 or less: none), worked out as it does, so that the window the server reports and judges "length" by is the +// engine's; 0 when the file does not say. With none asked, the runtime takes LlmMetadata's max_num_tokens, or 4096 +// for a file without one, and on GPU no more than 4096 (EngineSettings::MaybeUpdateAndValidate, engine_settings.cc; +// an Artisan decoder runs as GPU_ARTISAN, which that cap leaves out). A static model's window is then cut to the +// model's own (ClampMaxNumTokens, llm_litert_compiled_model_executor.cc). A dynamic one, whose context is a magic +// number M, is made for that many tokens when they are fewer than M, and otherwise for LiteRT-LM's default for M: the +// largest multiple of 256 below it, or below 256 the largest power of two (GetTargetNumber and GetDefaultTargetNumber, +// magic_number_configs_helper.cc). So M itself, which the server reported, is never what such a model gets. +int RuntimeWindow(const ModelFileInfo& info, int asked, const std::string& backend) { + if (info.max_context <= 0) return std::max(asked, 0); + int n = asked; + if (n <= 0) { + n = info.metadata_context > 0 ? info.metadata_context : 4096; + if (backend == "gpu" && !info.artisan && n > 4096) n = 4096; + } + if (!info.dynamic_context) return std::min(n, info.max_context); + const int m = info.max_context; + if (n < m) return n; + if (m > 256) return m / 256 * 256; + int t = 256; + while (t > m) t /= 2; + return t; +} + // Caller holds g_engine_mu. -LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc) { +LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc, const InputNeeds& needs) { EngineKey key; key.path = ref.path; - key.backend = ref.backend ? *ref.backend : (mc.backend ? Lower(*mc.backend) : "cpu"); + key.backend = ref.backend ? *ref.backend : (mc.backend ? Lower(*mc.backend) : ""); key.max_num_tokens = ref.max_num_tokens ? *ref.max_num_tokens : (mc.max_num_tokens ? *mc.max_num_tokens : -1); key.threads = mc.cpu_thread_count ? *mc.cpu_thread_count : -1; key.gpu_steps = mc.gpu_decode_steps_per_sync ? *mc.gpu_decode_steps_per_sync : -1; key.spec = mc.speculative_decoding ? (*mc.speculative_decoding ? 1 : 0) : -1; key.cache = CacheDirValue(mc.cache); if (key.cache.empty()) key.cache = FallbackCacheDir(ref.path); - if (key.backend != "cpu" && key.backend != "gpu" && key.backend != "npu") { + if (!key.backend.empty() && key.backend != "cpu" && key.backend != "gpu" && key.backend != "npu") { throw HttpError(500, "Failed to load engine: ValueError(\"Unsupported backend: '" + key.backend + "'\")"); } + const ModelFileInfo& info = FileInfo(ref); + // With no backend in the model reference or the config, the model's own: the first its header lists for its text + // (ReadModelFileInfo: a GPU-only bundle lists gpu alone, one with no constraint [cpu, gpu]), as `litert-lm serve` + // picks it (model.model_default_backend: an Artisan text decoder means gpu, else the decoder section's first + // backend_constraint, else cpu). The server picked CPU for every model, so a GPU-only bundle sent without a backend + // failed to load ("Main backend constraint mismatch", engine_settings.cc). A backend the model picked that does not + // start falls back to the next one it lists (below); one the user set does not. + const bool backend_set = !key.backend.empty(); + if (!backend_set) key.backend = info.text_backends.empty() ? "cpu" : info.text_backends.front(); + // A vision or audio executor is created only when the engine settings name a backend for it (c/engine.cc, + // CreateEngineSettingsHelper), and none was named, so an image sent to LFM2.5-VL-450M got a 500 "Vision executor + // should not be null, please TryLoadingVisionExecutor() first." although the model takes images. + key.vision_backend = EncoderBackend(ref, info.vision_backends, mc.vision_backend, "vision"); + key.audio_backend = EncoderBackend(ref, info.audio_backends, mc.audio_backend, "audio"); + // An image or audio part for a model that has no such input is the request's mistake, said before anything loads. + // When the server could not read the file's header, the runtime is left to say it. + if (info.capabilities_known && needs.image && key.vision_backend.empty()) { + throw HttpError(400, "Invalid messages: " + ref.id + " takes no image input"); + } + if (info.capabilities_known && needs.audio && key.audio_backend.empty()) { + throw HttpError(400, "Invalid messages: " + ref.id + " takes no audio input"); + } - if (g_engine.engine && g_engine.key == key) return g_engine.engine; - if (g_engine.engine) { + // A model whose decoder keeps a recurrent state gets a new engine for every conversation after the first. LiteRT-LM + // 0.18.0 starts each new conversation on an engine at step 0 and keeps the executor's buffers as they are there + // (RestoreContext in llm_litert_compiled_model_executor.cc), which is right for a KV cache, whose entries past the + // step are masked, but not for LFM2's short-convolution cache: it is read as it is, so every conversation after the + // first began from the end of the one before. On one engine LFM2.5-230M answered [system "Answer in one word.", + // user "What is the capital of France?", assistant "Paris", user "And of Italy?"] with "Paris is the capital of + // France." after that same question's first turn and a title request, and with "The capital of Italy is **Rome**." + // on a fresh one (LiteRT-LM #3165; the model card says to create a new Engine for each conversation). The C API has + // no way to reset that state, so the engine is created again, with the same settings. + const bool fresh_state = g_engine.engine && g_engine.key == key && g_engine.info.recurrent_state && g_engine.state_used; + if (g_engine.engine && g_engine.key == key && !fresh_state) return g_engine.engine; + if (fresh_state) { + static std::set told; // once per model file, not per conversation + if (told.insert(ref.path).second) { + Log(ref.id + " keeps a recurrent state that LiteRT-LM carries from one conversation to the next on an engine; " + "its engine is created again for each conversation"); + } + ReleaseEngine(); + } else if (g_engine.engine) { Log("Re-initializing engine (model: " + ref.id + ", backend: " + key.backend + ")"); ReleaseEngine(); } - Log("Initializing engine for model: " + ref.path + " (backend: " + key.backend + ")"); + // A context larger than the model takes was cut to the model's by the runtime with only a WARNING in its log + // ("Passed in max_num_tokens (8192) is larger than what the static model supports (4096). Using model limit.", + // ClampMaxNumTokens in llm_litert_compiled_model_executor.cc), so nobody was told, and "length" was judged against + // the size asked for: a reply cut off at the model's 4096 read "stop". Now the server gives the model its own + // limit, says so in the log, and reports both on /v1/models and in each reply's headers. With no size asked for, + // the window is the one the runtime picks (RuntimeWindow, once the backend is known). + int window = key.max_num_tokens > 0 ? RuntimeWindow(info, key.max_num_tokens, key.backend) : -1, window_asked = -1; + if (window > 0 && window < key.max_num_tokens) { + window_asked = key.max_num_tokens; + // Once per model and size: a model that gets a new engine for each conversation (LFM2) would log it every time. + static std::set told; + if (told.insert(ref.path + "," + std::to_string(window_asked)).second) { + Log("max_num_tokens " + std::to_string(window_asked) + " is more than " + ref.id + " takes (" + + (info.dynamic_context ? "its dynamic context takes fewer than " + std::to_string(info.max_context) + : "its fixed context is " + std::to_string(info.max_context)) + + " tokens); its context window is " + std::to_string(window)); + } + } auto t0 = std::chrono::steady_clock::now(); - L.litert_lm_clear_last_error(); - LiteRtLmEngineSettings* s = L.litert_lm_engine_settings_create(ref.path.c_str(), key.backend.c_str(), nullptr, nullptr); - if (!s) { - std::string why = LastRuntimeError(); - throw HttpError(500, "Failed to load engine: RuntimeError('Failed to create engine settings for " + ref.path + - (why.empty() ? "" : ": " + why) + "')"); - } - L.litert_lm_engine_settings_enable_benchmark(s); - L.litert_lm_engine_settings_set_use_ringbuffers_local_attention(s, true); - if (key.backend == "cpu" && key.threads > 0) L.litert_lm_engine_settings_set_num_threads(s, key.threads); - if (key.max_num_tokens > 0) L.litert_lm_engine_settings_set_max_num_tokens(s, key.max_num_tokens); - L.litert_lm_engine_settings_set_cache_dir(s, key.cache.c_str()); - if (key.spec >= 0) L.litert_lm_engine_settings_set_enable_speculative_decoding(s, key.spec == 1); - if (key.backend == "gpu" && key.gpu_steps > 0) L.litert_lm_engine_settings_set_gpu_decode_steps_per_sync(s, key.gpu_steps); - - LiteRtLmEngine* e = L.litert_lm_engine_create(s); - L.litert_lm_engine_settings_delete(s); - if (!e) { - std::string why = LastRuntimeError(); - throw HttpError(500, "Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for " + ref.path + - (why.empty() ? "" : ": " + why) + "')"); + // The engine on `backend`, or null with the runtime's reason in `why`. + std::string why; + auto create = [&](const std::string& backend) -> LiteRtLmEngine* { + Log("Initializing engine for model: " + ref.path + " (backend: " + backend + + (key.vision_backend.empty() ? "" : ", vision: " + key.vision_backend) + + (key.audio_backend.empty() ? "" : ", audio: " + key.audio_backend) + ")"); + L.litert_lm_clear_last_error(); + LiteRtLmEngineSettings* s = L.litert_lm_engine_settings_create( + ref.path.c_str(), backend.c_str(), key.vision_backend.empty() ? nullptr : key.vision_backend.c_str(), + key.audio_backend.empty() ? nullptr : key.audio_backend.c_str()); + if (!s) { + why = "Failed to create engine settings for " + ref.path; + if (std::string runtime = LastRuntimeError(); !runtime.empty()) why += ": " + runtime; + return nullptr; + } + L.litert_lm_engine_settings_enable_benchmark(s); + L.litert_lm_engine_settings_set_use_ringbuffers_local_attention(s, true); + if (backend == "cpu" && key.threads > 0) L.litert_lm_engine_settings_set_num_threads(s, key.threads); + if (window > 0) L.litert_lm_engine_settings_set_max_num_tokens(s, window); + L.litert_lm_engine_settings_set_cache_dir(s, key.cache.c_str()); + if (key.spec >= 0) L.litert_lm_engine_settings_set_enable_speculative_decoding(s, key.spec == 1); + if (backend == "gpu" && key.gpu_steps > 0) L.litert_lm_engine_settings_set_gpu_decode_steps_per_sync(s, key.gpu_steps); + LiteRtLmEngine* e = L.litert_lm_engine_create(s); + L.litert_lm_engine_settings_delete(s); + if (!e) { + why = "Failed to create LiteRT-LM engine for " + ref.path; + if (std::string runtime = LastRuntimeError(); !runtime.empty()) why += ": " + runtime; + } + return e; + }; + // A new engine for a fresh state (above) runs where the one before it did, so a model whose own backend fell back is + // not tried on that backend again for every conversation. A backend other than CPU that the model picked and that + // does not start falls back to the next one the model lists, and only to one it lists: a GPU-only bundle was tried + // on CPU, which cannot run it, and the error then gave the CPU's reason instead of the GPU's. When none starts, the + // error gives each one's reason. + std::vector backends = {fresh_state ? g_engine.backend : key.backend}; + if (!backend_set && backends[0] != "cpu") { + auto at = std::find(info.text_backends.begin(), info.text_backends.end(), backends[0]); + if (at != info.text_backends.end()) backends.insert(backends.end(), at + 1, info.text_backends.end()); + } + if (!backend_set && !fresh_state && backends[0] != "cpu") Log("Using " + ref.id + "'s own backend: " + backends[0]); + LiteRtLmEngine* e = nullptr; + std::string backend, reasons; + for (size_t i = 0; i < backends.size() && !e; ++i) { + backend = backends[i]; + if ((e = create(backend))) break; + reasons += (reasons.empty() ? "" : "; ") + (backends.size() > 1 ? "on " + backend + ": " : std::string()) + why; + if (i + 1 < backends.size()) { + Log(ref.id + "'s own backend, " + backend + ", did not start (" + why + "); falling back to " + backends[i + 1]); + } } + if (!e) throw HttpError(500, "Failed to load engine: RuntimeError('" + reasons + "')"); + if (window <= 0) window = RuntimeWindow(info, -1, backend); g_engine.engine = e; g_engine.key = key; + g_engine.backend = backend; + // The same model and settings again: what was found out about the model still holds. + if (!fresh_state) g_engine.tools_unconstrained = false; + g_engine.info = info; + g_engine.window = window > 0 ? window : -1; + g_engine.window_asked = window_asked; + g_engine.state_used = false; g_engine.init_seconds = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); Log("Engine ready in " + std::to_string(g_engine.init_seconds) + " s"); return e; @@ -397,11 +986,59 @@ std::map BuildNameByToolCallId(const json& messages) { return out; } +// Arrays and objects nested deeper than this are refused in what a request sends. The parser keeps its own stack, but +// copying a value and dump() recurse once per level on the connection thread's stack (8 MB on glibc, about 1 MB on +// Android), so a 20 KB body of ten thousand nested brackets overflowed it and killed the server with SIGSEGV. Nothing +// a chat request carries comes near 128 levels, serde_json's default limit too. +constexpr int kMaxJsonDepth = 128; + +struct JsonTooDeep : std::runtime_error { + JsonTooDeep() : std::runtime_error("nested deeper than " + std::to_string(kMaxJsonDepth) + " levels") {} +}; + +// The DOM builder json::parse itself uses, counting the arrays and objects open: it throws JsonTooDeep at one that +// kMaxJsonDepth others enclose. Not json::parse with a parser callback, which would be simpler: that parser's +// end_object scans the whole enclosing array or object for a value the callback discarded, each time an object closes, +// so a body holding N objects in one array or object took O(N^2). A normal chat request with an extra field of 50,000 +// empty objects (575 KB) kept a core busy for 16 s, on the connection thread and before any lock, and nothing stops +// that once the client has gone. This builder's start and end are what json::parse does, so it stays linear. +struct BoundedDomParser : nlohmann::detail::json_sax_dom_parser { + using json_sax_dom_parser::json_sax_dom_parser; + int depth = 0; + bool start_object(std::size_t n) { + if (++depth > kMaxJsonDepth) throw JsonTooDeep(); + return json_sax_dom_parser::start_object(n); + } + bool end_object() { + --depth; + return json_sax_dom_parser::end_object(); + } + bool start_array(std::size_t n) { + if (++depth > kMaxJsonDepth) throw JsonTooDeep(); + return json_sax_dom_parser::start_array(n); + } + bool end_array() { + --depth; + return json_sax_dom_parser::end_array(); + } +}; + +// json::parse, throwing JsonTooDeep at an array or object that kMaxJsonDepth others enclose. A syntax error throws +// json::parse_error, as with json::parse (the builder is made with allow_exceptions). +json ParseBounded(const std::string& s) { + json result; + BoundedDomParser builder(result, true); + json::sax_parse(s, &builder); + return result; +} + json ParseToolArguments(const json& args) { if (args.is_object()) return args; if (args.is_string()) { try { - json j = json::parse(args.get()); + // Bounded too: a tool call's arguments arrive as a string the body's own depth check cannot see into. Arguments + // nested too deeply become {}, as unparseable ones do. + json j = ParseBounded(args.get()); return j; } catch (...) { } @@ -413,6 +1050,16 @@ json TranslateMessage(const json& msg, const std::map& if (!msg.is_object()) throw HttpError(400, "Invalid messages: Message must be an object"); std::string role = msg.value("role", ""); json content = msg.contains("content") ? msg["content"] : json(); + // OpenAI's newer name for the system role. Model templates know only "system", so a developer message was rendered + // as an unknown role (LFM2.5-230M showed the word in its prompt) and its instructions were not followed: with + // developer "Answer in one word." it answered "The capital of France is Paris." where the system role gets "Paris". + if (role == "developer") { + static std::once_flag told; + std::call_once(told, [] { Log("the developer role is read as system, its older name"); }); + json out = msg; + out["role"] = "system"; + return TranslateMessage(out, names); + } if (role == "tool") { std::string id = msg.value("tool_call_id", ""); @@ -482,6 +1129,8 @@ struct Request { std::vector g_cors_origins; +// False once the connection is gone, or once the client has stopped reading: every accepted socket has a 30 s +// SO_SNDTIMEO (see main), and a send() that times out with nothing sent fails with EAGAIN. bool SendAll(int fd, const std::string& data) { size_t off = 0; while (off < data.size()) { @@ -554,28 +1203,73 @@ std::string CorsHeaders(const Request& req) { return h; } +// Why a request that a web page may have sent is refused, or empty. This port takes no key (only the daemon is meant +// to use it), and a browser that lets pages reach loopback (Firefox for Android does) let any site the user opened +// POST here: one cross-site request naming any file released the user's model to load that file instead. +// - Host: a page on a domain the attacker points at 127.0.0.1 (DNS rebinding) is same-origin with this port, so it +// can send anything and read the reply. Its requests name that domain in Host; the daemon's name 127.0.0.1. So +// Host must be localhost or an IP address: a page whose origin is an IP address and this port came from whatever +// listens there, this server, which serves no pages. Any IP address, not only loopback ones, so that a server +// started with --host 0.0.0.0 (the user's own serve flags follow the daemon's --host 127.0.0.1) can still be +// reached by the phone's LAN address. A request without Host is no browser's. +// - Origin: a browser adds it to every POST a page sends to another origin, and sends one without asking first (no +// preflight) only with a form's content types: text/plain, application/x-www-form-urlencoded or +// multipart/form-data. One with application/json it sends only after a preflight that allows it, and this server +// allows none unless started with --cors-origin. So a POST carrying an Origin must be application/json, unless +// --cors-origin allows that Origin. Origin itself is not refused: a daemon older than TurboLLM#287 forwarded its +// client's headers on its /v1/* pass-through, Origin included, so a web client of its API reached this port with +// one (since TurboLLM#287 the daemon sends no Origin here). Nor is application/json required of every POST: +// `curl -d` sends a form type, and that older daemon forwarded its client's Content-Type too (the daemon now +// sends the chat body it re-serializes as application/json). A web client of such a daemon that posted JSON +// labelled text/plain is refused here: from this port it looks like a page posting here. +std::string CrossSiteRefusal(const Request& req) { + if (req.headers.count("host")) { + std::string host = Lower(req.headers.at("host")); + host = host.substr(0, host[0] == '[' ? host.find(']') + 1 : host.find(':')); // drop the port + in6_addr ip; // room for either family + bool ip_literal = host.size() > 2 && host[0] == '[' && host.back() == ']' + ? ::inet_pton(AF_INET6, host.substr(1, host.size() - 2).c_str(), &ip) == 1 + : ::inet_pton(AF_INET, host.c_str(), &ip) == 1; + if (host != "localhost" && !ip_literal) return "Forbidden: Host must be localhost or an IP address"; + } + if (req.method == "POST" && req.headers.count("origin")) { + const std::string& origin = req.headers.at("origin"); + for (const auto& o : g_cors_origins) { + if (o == "*" || o == origin) return ""; + } + std::string type = req.headers.count("content-type") ? req.headers.at("content-type") : ""; + if (Lower(Trim(type.substr(0, type.find(';')))) != "application/json") { + return "Forbidden: a cross-origin POST must be application/json"; + } + } + return ""; +} + bool SendResponse(int fd, const Request& req, int status, const std::string& reason, const std::string& content_type, - const std::string& body) { + const std::string& body, const std::string& extra_headers = "") { std::string head = "HTTP/1.1 " + std::to_string(status) + " " + SafeReason(reason) + "\r\n"; head += std::string("Server: ") + kServerName + "/" + kServerVersion + "\r\n"; head += "Content-Type: " + content_type + "\r\n"; head += "Content-Length: " + std::to_string(body.size()) + "\r\n"; head += "Connection: close\r\n"; + head += extra_headers; head += CorsHeaders(req); head += "\r\n"; return SendAll(fd, head + body); } -bool SendJson(int fd, const Request& req, int status, const json& j) { +bool SendJson(int fd, const Request& req, int status, const json& j, const std::string& extra_headers = "") { return SendResponse(fd, req, status, ReasonFor(status), "application/json", - j.dump(2, ' ', false, json::error_handler_t::replace) + "\n"); + j.dump(2, ' ', false, json::error_handler_t::replace) + "\n", extra_headers); } // Python's send_error: the message is the status line's reason phrase. The body is an -// OpenAI-style error object rather than Python's HTML page. +// OpenAI-style error object rather than Python's HTML page. The message is often the runtime's own text, which need +// not be UTF-8 (a bad model file's magic bytes), and this runs inside catch handlers, so dump must not throw. bool SendError(int fd, const Request& req, int status, const std::string& message) { json body = {{"error", {{"message", message}, {"type", status >= 500 ? "server_error" : "invalid_request_error"}, {"code", status}}}}; - return SendResponse(fd, req, status, message, "application/json", body.dump() + "\n"); + return SendResponse(fd, req, status, message, "application/json", + body.dump(-1, ' ', false, json::error_handler_t::replace) + "\n"); } bool ReadRequest(int fd, Request& req) { @@ -639,6 +1333,8 @@ struct StreamQueue { // StreamCallback from its own thread with a pointer to the queue, so a request that leaves early (an exception while // tokens are still arriving: a malformed tool call, a JSON error) must not free the queue under it. On such an exit // this cancels the generation and waits for the final chunk; if none comes in 30 s the server exits (see below). +// A chunk marked final is not always the last callback (an unparsable tool call is one), so HandleChatCompletions also +// declares the queue before the conversation, whose deletion waits for the generation to end. class StreamGuard { public: StreamGuard(std::unique_ptr& q, LiteRtLmConversation* conv) : q_(q), conv_(conv) {} @@ -683,10 +1379,9 @@ void StreamCallback(void* data, const LiteRtLmStreamChunk* chunk) { item.text = t ? t : ""; item.final = L.litert_lm_stream_chunk_is_final(chunk); } - { - std::lock_guard lock(q->mu); - q->items.push_back(std::move(item)); - } + std::lock_guard lock(q->mu); + q->items.push_back(std::move(item)); + // Notify under the lock: once it is released the request thread may take the final chunk and free q. q->cv.notify_one(); } @@ -731,9 +1426,11 @@ Piece ParseChunk(const std::string& s) { return p; } -json OpenAiToolCalls(const json& calls, const std::string& now_str, bool with_index) { +// `first` is the number of the first call in `calls` within the turn: a stream sends a turn's calls over several +// chunks, and each call keeps its place in the turn's numbering (see the drain loop). +json OpenAiToolCalls(const json& calls, const std::string& now_str, bool with_index, int first = 0) { json out = json::array(); - int i = 0; + int i = first; for (const auto& tc : calls) { json fn = tc.is_object() && tc.contains("function") ? tc["function"] : tc; json args = fn.is_object() && fn.contains("arguments") ? fn["arguments"] : json::object(); @@ -794,20 +1491,218 @@ LiteRtLmThinkingConfig* MakeThinking(bool enable, int budget) { return t; } -// openai_common.parse_thinking_config: reasoning_effort wins; otherwise the config's thinking keys. -std::optional> ParseThinking(const json& body, const ModelConfig& mc) { - if (body.contains("reasoning_effort") && !body["reasoning_effort"].is_null()) { - if (!body["reasoning_effort"].is_string()) throw HttpError(400, "Invalid thinking parameters: ValueError: reasoning_effort must be a string"); - std::string e = Lower(body["reasoning_effort"].get()); - if (e == "none") return std::make_pair(false, 0); - if (e == "minimal" || e == "low" || e == "medium" || e == "high" || e == "xhigh") return std::make_pair(true, -1); - throw HttpError(400, "Invalid thinking parameters: ValueError: Invalid reasoning_effort value: '" + e + +// Thinking as a request asks for it. The server read only the top-level reasoning_effort and the config's thinking +// keys (openai_common.parse_thinking_config), and passed an empty extra context with every message, so what TurboLLM +// and other clients send was dropped: chat_template_kwargs (enable_thinking, reasoning_effort, preserve_thinking), +// which LiteRT-LM renders into the model's Jinja template as extra context (c/conversation.cc, CreateOptionalArgs), and +// thinking_budget_tokens, whose budget the runtime enforces (ThinkingBudgetConstraint, runtime/core/tasks.cc). On +// Qwen3-0.6B {"chat_template_kwargs": {"enable_thinking": false}} and {"thinking_budget_tokens": 0} both thought for +// 190 tokens; only reasoning_effort "none" turned it off. In order, the first that says: the top-level +// reasoning_effort (as the Python server, and what the daemon sends for a LiteRT-LM model), then +// chat_template_kwargs.enable_thinking, then chat_template_kwargs.reasoning_effort, then thinking_budget_tokens (0 is +// off, n > 0 a budget of n, below 0 no budget), then the config's thinking keys. A budget of n > 0 applies whichever +// turned thinking on. reasoning_effort's levels other than none turn it on with no budget, and none turns it off, as in +// the Python server. +struct Thinking { + bool enable; + int budget; // -1: none + std::string from; +}; + +// Whether a model thinks, as its file says: its metadata's supports_thinking, else its template's switch or a thinking +// channel (gemma-3-270m-it has neither). +bool Thinks(const ModelFileInfo& m) { return m.supports_thinking ? *m.supports_thinking : m.thinking_switch || m.channel; } + +// The 400 for thinking turned on, by `t`, for a model that does not think. +HttpError ThinkingRefused(const ModelFileInfo& m, const Thinking& t, const std::string& id) { + return HttpError(400, "Invalid thinking parameters: " + t.from + " turns thinking on, but " + id + + (m.supports_thinking ? " does not think (its metadata says so)" : " does not think")); +} + +std::optional ParseThinking(const json& body, const ModelConfig& mc) { + auto effort = [](const json& v, const std::string& field) -> Thinking { + if (!v.is_string()) throw HttpError(400, "Invalid thinking parameters: ValueError: " + field + " must be a string"); + std::string e = Lower(v.get()); + if (e == "none") return {false, 0, field}; + if (e == "minimal" || e == "low" || e == "medium" || e == "high" || e == "xhigh") return {true, -1, field}; + throw HttpError(400, "Invalid thinking parameters: ValueError: Invalid " + field + " value: '" + e + "'. Supported strings: none, minimal, low, medium, high, xhigh."); + }; + auto has = [](const json& o, const char* k) { return o.contains(k) && !o[k].is_null(); }; + std::optional t; + if (has(body, "reasoning_effort")) t = effort(body["reasoning_effort"], "reasoning_effort"); + if (has(body, "chat_template_kwargs")) { + const json& ctk = body["chat_template_kwargs"]; + if (!ctk.is_object()) throw HttpError(400, "Invalid thinking parameters: ValueError: chat_template_kwargs must be an object"); + if (has(ctk, "enable_thinking")) { + if (!ctk["enable_thinking"].is_boolean()) { + throw HttpError(400, "Invalid thinking parameters: ValueError: chat_template_kwargs.enable_thinking must be a boolean"); + } + if (!t) t = Thinking{ctk["enable_thinking"].get(), ctk["enable_thinking"].get() ? -1 : 0, + "chat_template_kwargs.enable_thinking"}; + } + if (has(ctk, "reasoning_effort")) { + Thinking e = effort(ctk["reasoning_effort"], "chat_template_kwargs.reasoning_effort"); + if (!t) t = e; + } } - if (!mc.thinking && !mc.thinking_budget) return std::nullopt; + if (has(body, "thinking_budget_tokens")) { + const json& b = body["thinking_budget_tokens"]; + if (!b.is_number_integer()) throw HttpError(400, "Invalid thinking parameters: ValueError: thinking_budget_tokens must be an integer"); + const long long n = b.get(); + if (n > INT32_MAX) throw HttpError(400, "Invalid thinking parameters: ValueError: thinking_budget_tokens is too large"); + if (!t) { + t = Thinking{n != 0, n > 0 ? static_cast(n) : n == 0 ? 0 : -1, "thinking_budget_tokens"}; + } else if (t->enable && n > 0) { + t->budget = static_cast(n); + } else if (t->enable != (n != 0)) { + Log("thinking_budget_tokens " + std::to_string(n) + " is not used: " + t->from + " turned thinking " + + (t->enable ? "on" : "off")); + } + } + if (t || (!mc.thinking && !mc.thinking_budget)) return t; bool thinking = mc.thinking ? *mc.thinking : (*mc.thinking_budget != 0); int budget = mc.thinking_budget ? *mc.thinking_budget : (thinking ? -1 : 0); - return std::make_pair(thinking, budget); + return Thinking{thinking, budget, "the config's thinking"}; +} + +// Repetition penalties, which LiteRT-LM 0.18.0 applies per message while it decodes (RepetitionPenaltyConfig, through +// litert_lm_conversation_optional_args_set_repetition_penalty_config; tasks.cc), and which the server dropped: +// presence_penalty and frequency_penalty (OpenAI's: subtracted from a token's logit once, or once per occurrence) and +// repetition_penalty, or repeat_penalty, llama.cpp's name, which the daemon sends (HuggingFace's multiplicative +// penalty, 1 = none). Its window: repeat_last_n when the request sets it, as llama.cpp reads it (0 turns the penalties +// off, -1 is the whole reply); else llama.cpp's own default of the last 64 tokens for a request that names the penalty +// repeat_penalty, llama.cpp's way; else the whole reply, as OpenAI's penalties and LiteRT-LM's default are. LiteRT-LM +// counts the tokens of the reply only, not the prompt's (repetition_penalty_config.h). A repetition penalty below 1, +// which in llama.cpp favours repeats, LiteRT-LM clamps to 1 without a word, so it is a 400. none: no penalty. +struct Penalties { + float repetition = 1, presence = 0, frequency = 0; + int window = 0; +}; + +std::optional ParsePenalties(const json& body) { + auto has = [&](const char* k) { return body.contains(k) && !body[k].is_null(); }; + auto number = [&](const char* k) { + if (!body[k].is_number() || !std::isfinite(body[k].get())) { + throw HttpError(400, std::string("Invalid sampler parameters: ValueError: ") + k + " must be a number"); + } + return body[k].get(); + }; + Penalties p; + const char* rep = has("repetition_penalty") ? "repetition_penalty" : has("repeat_penalty") ? "repeat_penalty" : nullptr; + if (rep) { + p.repetition = number(rep); + if (p.repetition < 1) { + throw HttpError(400, std::string("Invalid sampler parameters: ") + rep + " " + body[rep].dump() + + " is below 1, which LiteRT-LM does not take (1 is no penalty)"); + } + } + if (has("presence_penalty")) p.presence = number("presence_penalty"); + if (has("frequency_penalty")) p.frequency = number("frequency_penalty"); + p.window = rep && std::string(rep) == "repeat_penalty" ? 64 : 0; + if (has("repeat_last_n")) { + if (!body["repeat_last_n"].is_number_integer() || body["repeat_last_n"].get() < -1 || + body["repeat_last_n"].get() > INT32_MAX) { + throw HttpError(400, "Invalid sampler parameters: ValueError: repeat_last_n must be an integer of -1 or more"); + } + const int n = body["repeat_last_n"].get(); + if (n == 0) return std::nullopt; + p.window = n > 0 ? n : 0; + } + if (p.repetition == 1 && p.presence == 0 && p.frequency == 0) return std::nullopt; + return p; +} + +// The sampler a conversation is created with. A session's sampler parameters replace the model's own entirely: the +// runtime uses LlmMetadata's only when the session's type is UNSPECIFIED (SessionConfig::MaybeUpdateAndValidate in +// engine_settings.cc, LiteRT-LM 0.18.0). The server used to build one only when the request set any of temperature, +// top_p, top_k and seed, and then filled every field the request left out with a fixed value: top_k 1, which is +// greedy decoding (sampling_cpu_util.cc), top_p 0.95, temperature 1 and seed 0. So a temperature or top_p sent alone +// never had any effect (with temperature 1.5 and seeds 1, 2 and 3, LFM2.5-230M wrote the same sentence it writes +// with none), and the model's own parameters were thrown away whenever a request set one field. And the seed was 0 +// whenever the request gave none, the runtime's default as well (engine_settings.cc), so every conversation drew the +// same "random" sample and a Regenerate repeated the answer word for word. +struct Sampling { + int top_k; + float top_p, temperature; + int seed; +}; + +// The top_k that is no top-k limit. On CPU the largest k, which the CPU sampler clamps to the vocabulary +// (TopKTopPSampling in sampling_cpu_util.cc). On GPU the vocabulary itself, all the tokens there are: the GPU sampler +// is created for the k it is given and the model's number of logits (CreateGpuSampler in sampler_factory.cc; the +// sampler itself is not open source), so it gets no k larger than that. LiteRT-LM 0.18.0 sets no top_k limit of its +// own there: GpuConfig::max_top_k, 64, is read nowhere. Where the vocabulary is unknown, the largest k, as on CPU. +int NoTopKLimit(const std::string& backend, const ModelFileInfo& model) { + return backend == "gpu" && model.vocabulary > 0 ? model.vocabulary : INT32_MAX; +} + +// Each field the request sets, as it set it; the others from the model's own parameters, or, for a model that has +// none, LiteRT-LM's own defaults for CPU and GPU, the ones it would use (engine_settings.cc: TOP_P, k 1, p 0.95, +// temperature 1); and a random seed unless the request gives one. The CPU sampler implements TOP_P only +// (CreateCpuSampler in sampler_factory.cc), and TOP_P with p 1 is top-k sampling, so a model whose own type is TOP_K +// or GREEDY gets that. On NPU, and for an Artisan text decoder on GPU, which LiteRT-LM runs as GPU_ARTISAN, its +// default type is UNSPECIFIED (engine_settings.cc), which leaves sampling to the model, so there a sampler is set only +// for a request that sets one of the fields, as before. none: the runtime's own choice. +std::optional SamplingFor(const json& body, const ModelFileInfo& model, const std::string& backend, + const std::string& model_id, bool request_sets_any) { + const bool model_samples = (backend != "cpu" && backend != "gpu") || (backend == "gpu" && model.artisan); + if (model_samples && !request_sets_any) return std::nullopt; + Sampling s{1, 0.95f, 1.0f, 0}; + switch (model.sampler_type) { + case kLiteRtLmSamplerTypeTopP: + s = {model.top_k, model.top_p, model.temperature, 0}; + break; + case kLiteRtLmSamplerTypeTopK: + s = {model.top_k, 1.0f, model.temperature, 0}; + break; + case kLiteRtLmSamplerTypeGreedy: + s = {1, 1.0f, 1.0f, 0}; + break; + default: + break; + } + // A model that names TOP_P or TOP_K with no k (0) asks for no top-k limit; the runtime refuses k 0. + if (s.top_k <= 0) s.top_k = NoTopKLimit(backend, model); + auto has = [&](const char* k) { return body.contains(k) && !body[k].is_null(); }; + if (has("temperature")) s.temperature = body["temperature"].get(); + if (has("top_p")) s.top_p = body["top_p"].get(); + if (has("top_k")) { + const int k = static_cast(body["top_k"].get()); + if (k <= 0) { + s.top_k = NoTopKLimit(backend, model); + static std::set told; + if (told.insert(model_id + "," + backend).second) { + const char* what = s.top_k != INT32_MAX ? " (its vocabulary)" + : backend == "cpu" ? " (the CPU sampler takes the whole vocabulary)" + : " (its vocabulary is unknown)"; + Log("top_k " + std::to_string(k) + " is no top-k limit: top_k " + std::to_string(s.top_k) + " for " + model_id + + what); + } + } else { + s.top_k = k; // as set, on GPU too (see NoTopKLimit) + } + } else if (s.top_k == 1 && ((has("temperature") && s.temperature > 0) || (has("top_p") && s.top_p < 1))) { + // k 1 picks the likeliest token whatever the temperature or top_p, so the request's would have no effect: the + // model's default here is greedy decoding, and the request asks to sample. It gets no top-k limit. + s.top_k = NoTopKLimit(backend, model); + static std::set told; // once per model and backend, not per request + if (told.insert(model_id + "," + backend).second) { + Log("top_k is not set and " + model_id + "'s default is 1 (greedy), under which the request's " + + (has("temperature") ? "temperature" : "top_p") + " would have no effect; sampling on " + backend + + " with top_k " + std::to_string(s.top_k) + + (s.top_k == INT32_MAX ? " (no limit)" : " (its vocabulary: no limit)")); + } + } + if (has("seed")) { + s.seed = static_cast(body["seed"].get()); + } else { + // The CPU sampler seeds std::default_random_engine (minstd_rand0) with it, which takes 0 and 1 alike, so the + // range starts at 1; each conversation seeds its own generator (CreateNewContext). + static std::mt19937 gen{std::random_device{}()}; + s.seed = std::uniform_int_distribution(1, INT32_MAX - 1)(gen); + } + return s; } // openai_common.parse_response_format → (constraint type, schema/pattern), or none. @@ -876,7 +1771,10 @@ std::string NowStr(long long* created) { void HandleChatCompletions(int fd, const Request& req) { json body; try { - body = json::parse(req.body); + body = ParseBounded(req.body); + } catch (const JsonTooDeep& e) { + SendError(fd, req, 400, std::string("Invalid JSON: ") + e.what()); + return; } catch (...) { SendError(fd, req, 400, "Invalid JSON"); return; @@ -915,17 +1813,57 @@ void HandleChatCompletions(int fd, const Request& req) { } // conversation.normalize_message: a bare string is a user turn. if (prompt.is_string()) prompt = {{"role", "user"}, {"content", prompt}}; + // A conversation that ends with the assistant's own words (a client prefilling the start of the reply, which the + // daemon's gateway passes through) asks for that reply to be continued. LiteRT-LM 0.18.0 can do that only from + // C++ (OptionalArgs::has_pending_message, runtime/conversation/conversation.h); its C API has no way to, so the + // message was sent as a finished turn and the model began a new reply: [user "What is the capital of France?", + // assistant "The capital of France is"] got "The capital of France is Paris.", which a client appending it to its + // prefill doubles. A 400 that says so instead. An assistant turn of tool calls is the model's own, as before; one + // whose tool_calls is null or empty, as clients that write every optional field send a plain reply, has none. + const bool has_calls = prompt.is_object() && prompt.contains("tool_calls") && prompt["tool_calls"].is_array() && + !prompt["tool_calls"].empty(); + if (prompt.is_object() && prompt.value("role", "") == "assistant" && !has_calls) { + SendError(fd, req, 400, "Invalid messages: the last message is the assistant's, and this server cannot continue " + "a reply (LiteRT-LM's C API has no way to); end with a user or tool message"); + return; + } + // max_completion_tokens, or max_tokens, OpenAI's older name for it, which the server did not read: a client that + // sent max_tokens 5 got the whole reply. Below 1 is a 400, as OpenAI answers it: the runtime generated one token + // for 0 or -5 and the server called that "length". std::optional max_tokens; - if (body.contains("max_completion_tokens") && !body["max_completion_tokens"].is_null()) { - if (!body["max_completion_tokens"].is_number_integer()) { - SendError(fd, req, 400, "max_completion_tokens must be an integer"); + for (const char* k : {"max_completion_tokens", "max_tokens"}) { + if (max_tokens || !body.contains(k) || body[k].is_null()) continue; + if (!body[k].is_number_integer() || body[k].get() < 1 || body[k].get() > INT32_MAX) { + SendError(fd, req, 400, std::string(k) + " must be an integer of at least 1, got " + body[k].dump()); + return; + } + max_tokens = body[k].get(); + } + + // What LiteRT-LM's conversation API cannot give, said rather than dropped: it returns one reply per message, with no + // token log-probabilities (those exist only on its lower-level session API), so n above 1 got one choice and + // logprobs none, both with a 200. + { + auto has = [&](const char* k) { return body.contains(k) && !body[k].is_null(); }; + std::string why; + if (has("n") && !(body["n"].is_number_integer() && body["n"].get() == 1)) { + why = "n " + body["n"].dump() + ": this server gives one choice per request"; + } else if (has("logprobs") && !(body["logprobs"].is_boolean() && !body["logprobs"].get())) { + why = "logprobs " + body["logprobs"].dump() + ": LiteRT-LM gives no token log-probabilities"; + } else if (has("top_logprobs") && !(body["top_logprobs"].is_number_integer() && body["top_logprobs"].get() == 0)) { + why = "top_logprobs " + body["top_logprobs"].dump() + ": LiteRT-LM gives no token log-probabilities"; + } + if (!why.empty()) { + SendError(fd, req, 400, "Unsupported parameter: " + why); return; } - max_tokens = body["max_completion_tokens"].get(); } - // openai_chat_completions.parse_sampler_config + // openai_chat_completions.parse_sampler_config. A value the runtime cannot take is the request's mistake, a 400 that + // names the field: it was a 500 with the runtime's words ("INVALID_ARGUMENT: p must be in [0, 1].", "Temperature + // must be >= 0"), and a top_k or seed with a fraction was cut to an integer. top_k 0 or below is no top-k limit, + // as in llama.cpp (TurboLLM's Top K 0 means disabled; vLLM uses -1): see SamplingFor. bool has_sampler = false; for (const char* k : {"temperature", "top_p", "top_k", "seed"}) { if (body.contains(k) && !body[k].is_null()) { @@ -933,10 +1871,36 @@ void HandleChatCompletions(int fd, const Request& req) { SendError(fd, req, 400, std::string("Invalid sampler parameters: ValueError: ") + k + " must be a number"); return; } + const double v = body[k].get(); + const bool integral = std::isfinite(v) && v == std::floor(v) && v >= INT32_MIN && v <= INT32_MAX; + std::string why; + if (std::string(k) == "temperature" && !(v >= 0)) why = "temperature must be at least 0"; + if (std::string(k) == "top_p" && !(v >= 0 && v <= 1)) why = "top_p must be between 0 and 1"; + if ((std::string(k) == "top_k" || std::string(k) == "seed") && !integral) why = std::string(k) + " must be an integer"; + if (!why.empty()) { + SendError(fd, req, 400, "Invalid sampler parameters: ValueError: " + why + ", got " + body[k].dump()); + return; + } has_sampler = true; } } - auto thinking = ParseThinking(body, mc); + std::optional thinking = ParseThinking(body, mc); + std::optional penalties = ParsePenalties(body); + std::string note_headers; // headers that tell the client what the server did with its request + // min_p, which LiteRT-LM has no sampler for, was dropped without a word. A 400 for it would fail every chat of a + // thread or model whose saved settings have a Min P, chats that worked, so it is still not applied, but the reply + // says so (X-Unapplied-Parameters), and the log once per model (below, under the engine lock). + bool min_p_unapplied = false; + if (body.contains("min_p") && !body["min_p"].is_null()) { + if (!body["min_p"].is_number() || !std::isfinite(body["min_p"].get())) { + SendError(fd, req, 400, "Invalid sampler parameters: ValueError: min_p must be a number"); + return; + } + if (body["min_p"].get() > 0) { + note_headers += "X-Unapplied-Parameters: min_p\r\n"; + min_p_unapplied = true; + } + } auto constraint = ParseResponseFormat(body); json tools = json::array(); @@ -945,23 +1909,105 @@ void HandleChatCompletions(int fd, const Request& req) { if (t.is_object() && t.value("type", "") == "function") tools.push_back(t); } } + // tool_choice was ignored, so "none", which the daemon sends to make a model answer in text (its gateway after a + // repeated call, and its chat's retry after an empty answer), still offered the model the tools and got tool calls + // back. "none" now leaves the tools out of the conversation; the history keeps its earlier calls and results. + // "auto", or none at all, is what it was. LiteRT-LM 0.18.0's C API has no way to make a model call a tool (its + // constrained decoding keeps a call well formed, and Gemma's "function call only" mode is set by the model file, + // not per conversation), so "required" or a named function is a 400 that says so rather than a reply that may + // not call it. + if (body.contains("tool_choice") && !body["tool_choice"].is_null()) { + const json& choice = body["tool_choice"]; + const std::string mode = choice.is_string() ? choice.get() : ""; + if (mode == "none") { + tools = json::array(); + } else if (mode == "required" || choice.is_object()) { + SendError(fd, req, 400, "Invalid tool_choice: this server cannot make the model call a tool (" + choice.dump() + + "); LiteRT-LM can only offer it the tools, as tool_choice \"auto\" does"); + return; + } else if (mode != "auto") { + SendError(fd, req, 400, "Invalid tool_choice: " + choice.dump() + " (none, auto, required or a function)"); + return; + } + } - const bool stream = body.value("stream", false); - const bool include_usage = - body.contains("stream_options") && body["stream_options"].is_object() && body["stream_options"].value("include_usage", false); + // Absent or null is false for both, as the Python server's body.get() reads them, and as clients that write every + // optional field send them. json's value() threw type_error on a null, and the catch below made that a 500. Any + // other non-boolean is the client's mistake: a 400 that says so, not a 500. + bool stream = false; + if (body.contains("stream") && !body["stream"].is_null()) { + if (!body["stream"].is_boolean()) { + SendError(fd, req, 400, "stream must be a boolean"); + return; + } + stream = body["stream"].get(); + } + bool include_usage = false; + if (body.contains("stream_options") && body["stream_options"].is_object()) { + const json& so = body["stream_options"]; + if (so.contains("include_usage") && !so.at("include_usage").is_null()) { + if (!so.at("include_usage").is_boolean()) { + SendError(fd, req, 400, "stream_options.include_usage must be a boolean"); + return; + } + include_usage = so.at("include_usage").get(); + } + } + // OpenAI's stop: a string or an array of them. It was read nowhere, so the user's stop sequences (the daemon sends + // a thread's) and API clients' were silently ignored. The C API has no stop strings of its own, only the model's + // stop tokens, so the server ends the reply at the first of them itself, in the drain loop below. + std::vector stops; + if (body.contains("stop") && !body["stop"].is_null()) { + const json& st = body["stop"]; + for (const json& one : st.is_array() ? st : json::array({st})) { + if (!one.is_string()) { + SendError(fd, req, 400, "stop must be a string or an array of strings"); + return; + } + if (!one.get().empty()) stops.push_back(one.get()); + } + } + + // The parts TranslateMessage made of image_url and input_audio parts, or that the client sent as LiteRT-LM's own. + InputNeeds needs; + auto note_parts = [&](const json& m) { + if (!m.is_object() || !m.contains("content") || !m["content"].is_array()) return; + for (const auto& part : m["content"]) { + if (!part.is_object() || !part.contains("type") || !part["type"].is_string()) continue; + needs.image |= part["type"] == "image"; + needs.audio |= part["type"] == "audio"; + } + }; + for (const auto& m : translated) note_parts(m); + note_parts(prompt); std::lock_guard engine_lock(g_engine_mu); - LiteRtLmEngine* engine = GetOrCreateEngine(ref, mc); + // Thinking turned on for a model that does not think is refused before anything loads, as an image for a model + // without vision is (GetOrCreateEngine): refused once loaded, it first released the model that was loaded and + // loaded this one (gemma-3-270m-it with reasoning_effort "high": a 400 after 0.6 s, and the next request to the + // model before it paid its whole load again). A file whose header the server cannot read is left to the check + // below, once the runtime has loaded it, as before. + if (thinking && thinking->enable) { + const ModelFileInfo& info = FileInfo(ref); + if (info.capabilities_known && !Thinks(info)) throw ThinkingRefused(info, *thinking, ref.id); + } + LiteRtLmEngine* engine = GetOrCreateEngine(ref, mc, needs); + // Under the engine lock, as every other once-per-model note: each connection has its own thread, and two requests + // with a Min P at once (a chat and its title) inserted into this set together, which can corrupt it. + if (min_p_unapplied) { + static std::set told; + if (told.insert(ref.path).second) Log("min_p is not applied to " + ref.id + ": LiteRT-LM has no min-p sampling"); + } - // Engine.create_conversation, with exactly the options the serve path passes. + // Engine.create_conversation, with the options the serve path passes, but constrained decoding only with tools + // (below). LiteRtLmSessionConfig* sc = L.litert_lm_session_config_create(); - if (has_sampler) { + if (auto sampling = SamplingFor(body, g_engine.info, g_engine.backend, ref.id, has_sampler)) { LiteRtLmSamplerParams* sp = L.litert_lm_sampler_params_create(kLiteRtLmSamplerTypeTopP); - L.litert_lm_sampler_params_set_top_k(sp, body.contains("top_k") && !body["top_k"].is_null() ? body["top_k"].get() : 1); - L.litert_lm_sampler_params_set_top_p(sp, body.contains("top_p") && !body["top_p"].is_null() ? body["top_p"].get() : 0.95f); - L.litert_lm_sampler_params_set_temperature( - sp, body.contains("temperature") && !body["temperature"].is_null() ? body["temperature"].get() : 1.0f); - L.litert_lm_sampler_params_set_seed(sp, body.contains("seed") && !body["seed"].is_null() ? body["seed"].get() : 0); + L.litert_lm_sampler_params_set_top_k(sp, sampling->top_k); + L.litert_lm_sampler_params_set_top_p(sp, sampling->top_p); + L.litert_lm_sampler_params_set_temperature(sp, sampling->temperature); + L.litert_lm_sampler_params_set_seed(sp, sampling->seed); L.litert_lm_session_config_set_sampler_params(sc, sp); L.litert_lm_sampler_params_delete(sp); } @@ -974,17 +2020,101 @@ void HandleChatCompletions(int fd, const Request& req) { if (!context.empty()) L.litert_lm_conversation_config_set_messages(cc, context_str.c_str()); std::string tools_str = tools.dump(); if (!tools.empty()) L.litert_lm_conversation_config_set_tools(cc, tools_str.c_str()); - L.litert_lm_conversation_config_set_enable_constrained_decoding(cc, true); + // Constrained decoding is for tools only: with it, a data processor that can (Gemma 3's, Gemma 4's and + // FunctionGemma's, with the FST constraint library) builds a constraint from the tools so that the calls the model + // writes fit them (Conversation::CreateDecodeConfig, runtime/conversation/conversation.cc:401-413 in LiteRT-LM + // 0.18.0); the others build none (Qwen3's and the generic one never do: model_data_processor.h:101-103), and + // without tools none is built. A response_format does not use it: its constraint comes from the provider set + // below (conversation.cc:397-400). The Python server turns it on for every request, as this one did, and + // Conversation::Create passes it to the data processor, which may refuse it: LFM2's always does + // ("FAILED_PRECONDITION: Constrained decoding is not supported for Lfm2DataProcessor.", + // lfm2_data_processor.cc:53-56), and Gemma 3's, Gemma 4's and FunctionGemma's ("INTERNAL: Failed to create + // GemmaModelConstraintProvider.", gemma3_data_processor.cc:146-150, gemma4_data_processor.cc:96-101 and + // function_gemma_data_processor.cc:219-224) when the FST constraint library rejects the model's vocabulary, as it + // does gemma-3-270m-it's. Every request to such a model failed with a 500 before anything was generated. + const bool constrain_tools = !tools.empty() && !g_engine.tools_unconstrained; + L.litert_lm_conversation_config_set_enable_constrained_decoding(cc, constrain_tools); LiteRtLmConstraintProviderType provider = kLiteRtLmConstraintProviderTypeLlGuidance; if (constraint) L.litert_lm_conversation_config_set_constraint_provider(cc, &provider); + // The model's thinking. A model that does not think (its metadata says so, or it has neither the template's switch + // nor a thinking channel: gemma-3-270m-it) cannot be asked to: a 400, before it loads (above) when the server could + // read its header, else here. Turning it off there changes nothing, so nothing is set. A model whose file does not + // say, whose template has no switch, but that has a channel (LFM2.5-230M, which does not think, and + // LFM2.5-1.2B-Thinking, which always does) gets what was asked as asked: LiteRT-LM 0.18.0 can neither turn on nor + // stop thinking there by itself (a budget of 0 sets no constraint, tasks.cc), and nothing in such a file tells the + // two apart, so a 400 for one would refuse the other. The server says so in its log once and, since a client sees + // nothing of that log, in every such reply's X-Thinking header ("unverified"). + if (thinking) { + const ModelFileInfo& m = g_engine.info; + const bool thinks = Thinks(m); + if (!thinks && thinking->enable) { + L.litert_lm_conversation_config_delete(cc); + throw ThinkingRefused(m, *thinking, ref.id); + } + if (!thinks) { + thinking.reset(); + } else if (!m.supports_thinking && !m.thinking_switch) { + note_headers += "X-Thinking: unverified\r\n"; + static std::set told; + if (told.insert(ref.path).second) { + Log(ref.id + " does not say whether it thinks, and its template has no switch for it; thinking " + + (thinking->enable ? "on" : "off") + " (" + thinking->from + ") is passed to it as asked"); + } + } + } if (thinking) { - LiteRtLmThinkingConfig* t = MakeThinking(thinking->first, thinking->second); + LiteRtLmThinkingConfig* t = MakeThinking(thinking->enable, thinking->budget); L.litert_lm_conversation_config_set_thinking_config(cc, t); L.litert_lm_thinking_config_delete(t); } - L.litert_lm_clear_last_error(); - std::unique_ptr conv(L.litert_lm_conversation_create(engine, cc)); + // chat_template_kwargs is the template's extra context, for the preface and for the message, and its + // enable_thinking is what was decided above, which the template reads in place of the thinking config's + // (conversation.cc), so that the template and the decoder's budget agree. + json extra_context = body.contains("chat_template_kwargs") && body["chat_template_kwargs"].is_object() + ? body["chat_template_kwargs"] + : json::object(); + if (thinking && !extra_context.empty()) extra_context["enable_thinking"] = thinking->enable; + const std::string extra_context_str = extra_context.dump(); + if (!extra_context.empty()) L.litert_lm_conversation_config_set_extra_context(cc, extra_context_str.c_str()); + // Declared before the conversation so it is freed after it: deleting the conversation waits for its generation to + // end (~SessionAdvanced's WaitUntilDone), and until then the runtime may call StreamCallback with this queue, even + // after a chunk marked final (see the error branch of the drain loop below). + auto q_owner = std::make_unique(); + StreamQueue& q = *q_owner; + auto create = [&] { + L.litert_lm_clear_last_error(); + return std::unique_ptr(L.litert_lm_conversation_create(engine, cc)); + }; + std::unique_ptr conv = create(); + if (!conv && constrain_tools) { + // A model that refuses it still takes tools, unconstrained: the model writes its calls freely and its data + // processor parses them as always, as under `litert-lm run`, which never turns it on. Whether the flag was the + // cause is told by creating the conversation without it, not from the message, which differs by model. + std::string why = LastRuntimeError(); + L.litert_lm_conversation_config_set_enable_constrained_decoding(cc, false); + conv = create(); + if (conv) { + g_engine.tools_unconstrained = true; + Log("constrained decoding is not available for " + ref.id + (why.empty() ? "" : " (" + why + ")") + + "; its tool calls are not constrained"); + } + } + std::optional provider_refused; + if (!conv && constraint) { + // A response_format needs the LLGuidance constraint provider, which Conversation::Create builds from the model's + // tokenizer and stop tokens (conversation.cc:433-440) and which fails for a model with no single-token stop + // sequence (constraint_provider_factory.cc:43-56) or a tokenizer llguidance cannot take + // (llg_constraint_provider.cc:127-132). If the conversation can be created without it, the request asked for + // what this model cannot do: a 400 that says why, not a 500. + std::string why = LastRuntimeError(); + L.litert_lm_conversation_config_set_constraint_provider(cc, nullptr); + if (create()) provider_refused = why; + } L.litert_lm_conversation_config_delete(cc); + if (provider_refused) { + throw HttpError(400, "Invalid response_format parameters: this model cannot follow a response_format" + + (provider_refused->empty() ? "" : ": " + *provider_refused)); + } if (!conv) { std::string why = LastRuntimeError(); throw HttpError(500, "RuntimeError: Failed to create conversation" + (why.empty() ? "" : ": " + why)); @@ -993,33 +2123,55 @@ void HandleChatCompletions(int fd, const Request& req) { LiteRtLmConversationOptionalArgs* oa = L.litert_lm_conversation_optional_args_create(); if (max_tokens) L.litert_lm_conversation_optional_args_set_max_output_tokens(oa, *max_tokens); if (constraint) L.litert_lm_conversation_optional_args_set_constraint(oa, constraint->first, constraint->second.c_str()); + if (penalties) { + LiteRtLmRepetitionPenaltyConfig* rp = L.litert_lm_repetition_penalty_config_create(); + L.litert_lm_repetition_penalty_config_set_repetition_penalty(rp, penalties->repetition); + L.litert_lm_repetition_penalty_config_set_presence_penalty(rp, penalties->presence); + L.litert_lm_repetition_penalty_config_set_frequency_penalty(rp, penalties->frequency); + L.litert_lm_repetition_penalty_config_set_window_size(rp, penalties->window); + L.litert_lm_conversation_optional_args_set_repetition_penalty_config(oa, rp); // copied + L.litert_lm_repetition_penalty_config_delete(rp); + } long long created = 0; const std::string now_str = NowStr(&created); const std::string chunk_id = "chatcmpl_" + now_str; + // No "usage" key on these chunks, not even null as the Python server writes with include_usage: the daemon's + // gateway turns include_usage on for streaming clients that leave it unset, so it can count tokens, and takes the + // usage chunk back out of their stream, so a null on every other chunk reached clients that never asked for usage. + // The usage chunk at the end sets it. auto make_chunk = [&](json choices) { - json p = {{"id", chunk_id}, {"object", "chat.completion.chunk"}, {"created", created}, {"model", raw_model}, {"choices", choices}}; - if (include_usage) p["usage"] = nullptr; - return p; + return json{{"id", chunk_id}, {"object", "chat.completion.chunk"}, {"created", created}, {"model", raw_model}, {"choices", choices}}; }; - auto q_owner = std::make_unique(); - StreamQueue& q = *q_owner; std::string msg_str = prompt.dump(); L.litert_lm_clear_last_error(); - int rc = L.litert_lm_conversation_send_message_stream(conv.get(), msg_str.c_str(), "{}", oa, StreamCallback, &q); + g_engine.state_used = true; // even a send that fails may have prefilled part of the message + int rc = L.litert_lm_conversation_send_message_stream(conv.get(), msg_str.c_str(), extra_context_str.c_str(), oa, + StreamCallback, &q); L.litert_lm_conversation_optional_args_delete(oa); // Declared after the queue and the conversation, so it runs before either is destroyed. StreamGuard guard(q_owner, conv.get()); if (rc != 0) { guard.Finished(); // nothing was started std::string why = LastRuntimeError(); + // A response_format whose regex or JSON schema the provider cannot compile fails here, before anything is + // generated: the constraint is built as the message is sent (conversation.cc:397-400), and the provider reports + // "Failed to create LLGuidance constraint: " (llg_constraint_provider.cc:155-160), so the request is at + // fault: a 400, not a 500. + if (constraint && why.find("Failed to create LLGuidance constraint") != std::string::npos) { + throw HttpError(400, "Invalid response_format parameters: " + why); + } throw HttpError(500, "RuntimeError: litert_lm_conversation_send_message_stream failed" + (why.empty() ? "" : ": " + why)); } + // The context window the reply was made in, and the larger one asked for, if any, for the daemon to show. + std::string window_headers; + if (g_engine.window > 0) window_headers += "X-Context-Window: " + std::to_string(g_engine.window) + "\r\n"; + if (g_engine.window_asked > 0) window_headers += "X-Context-Window-Requested: " + std::to_string(g_engine.window_asked) + "\r\n"; if (stream) { std::string head = "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nCache-Control: no-cache\r\nConnection: close\r\n" + - CorsHeaders(req) + "\r\n"; + note_headers + window_headers + CorsHeaders(req) + "\r\n"; headers_sent = true; SendAll(fd, head + Sse(make_chunk(json::array({{{"index", 0}, {"delta", {{"role", "assistant"}}}, {"finish_reason", nullptr}}})))); } @@ -1031,6 +2183,41 @@ void HandleChatCompletions(int fd, const Request& req) { std::optional error; bool client_gone = false; bool cancelled = false; + bool prompt_too_long = false; + // The stop strings apply to the reply's text only, as OpenAI's do, not to its thinking or its tool calls. Text that + // could be the start of one is held back until the next chunk tells; at a stop string the reply ends before it, + // never showing it, and the generation is cancelled, which the loop below then takes as the quiet end it is. + std::string held; + bool stopped = false; + auto apply_stops = [&](std::string text) { + if (stops.empty()) return text; + if (stopped) return std::string(); + held += text; + size_t at = std::string::npos; + for (const auto& stop : stops) at = std::min(at, held.find(stop)); + if (at != std::string::npos) { + text = held.substr(0, at); + held.clear(); + stopped = true; + if (!cancelled) { + cancelled = true; + L.litert_lm_conversation_cancel_process(conv.get()); + } + return text; + } + size_t keep = 0; + for (const auto& stop : stops) { + for (size_t n = std::min(stop.size() - 1, held.size()); n > keep; --n) { + if (held.compare(held.size() - n, n, stop, 0, n) == 0) { + keep = n; + break; + } + } + } + text = held.substr(0, held.size() - keep); + held.erase(0, held.size() - keep); + return text; + }; // Drain the callback queue until the final chunk. Meanwhile watch the client, at most every 200 ms: a closed // connection cancels the generation (the Python server only noticed on its next write). Checked between chunks @@ -1039,9 +2226,15 @@ void HandleChatCompletions(int fd, const Request& req) { auto next_peer_check = std::chrono::steady_clock::now() + std::chrono::milliseconds(200); auto watch_peer = [&] { auto now = std::chrono::steady_clock::now(); - if (cancelled || now < next_peer_check) return; + if (now < next_peer_check) return; next_peer_check = now + std::chrono::milliseconds(200); - if (PeerGone(fd)) { + if (client_gone) { + // Cancel again on every tick until the final chunk. A cancel flags only the tasks the runtime has already + // queued, and the decode task is queued, with a fresh flag, only once the prefill task has finished: a cancel + // landing in between is lost, and the turn would run to its end for nobody, holding the engine. Cancelling a + // turn that has ended is a no-op. + L.litert_lm_conversation_cancel_process(conv.get()); + } else if (!cancelled && PeerGone(fd)) { client_gone = cancelled = true; L.litert_lm_conversation_cancel_process(conv.get()); } @@ -1064,9 +2257,27 @@ void HandleChatCompletions(int fd, const Request& req) { if (item.final) guard.Finished(); watch_peer(); if (item.is_error) { - if (item.text.find("CANCELLED") != std::string::npos) { + // Not necessarily the end of the generation: the runtime reports a tool call it cannot parse as soon as the + // call's closing tag streams in, marked final, and keeps decoding. Stop it whatever the error is (a no-op once + // the generation has ended), so deleting the conversation below does not wait for the rest of the turn while + // this request holds the engine. + if (!cancelled) { + cancelled = true; + L.litert_lm_conversation_cancel_process(conv.get()); + } + // The text is absl::Status::ToString(), code first. Match from the start: a parse error quotes the model's + // output, which may contain these words. + if (item.text.rfind("CANCELLED:", 0) == 0) { // a cancelled turn ends quietly, as in the Python client - } else if (item.text.find("Max number of tokens reached") != std::string::npos) { + } else if (item.text.rfind("INVALID_ARGUMENT: Input token ids are too long", 0) == 0) { + // The prompt alone does not fit the context window: the request's mistake, said with the window's size. + prompt_too_long = true; + error = "the prompt does not fit the model's context window" + + (g_engine.window > 0 ? " of " + std::to_string(g_engine.window) + " tokens" : std::string()) + " (" + + item.text + ")"; + } else if (item.text.rfind("RESOURCE_EXHAUSTED: Max number of tokens reached", 0) == 0) { + // A full context window, from a runtime that reports it as an error. LiteRT-LM 0.18.0's C API never does + // (see the token-count check below). finish = "length"; } else { error = item.text; @@ -1074,7 +2285,15 @@ void HandleChatCompletions(int fd, const Request& req) { break; } Piece p = ParseChunk(item.text); + p.text = apply_stops(p.text); + if (stopped) p.tool_calls = json::array(); if (!p.reasoning.empty()) ++reasoning_tokens; + // The runtime sends each tool-call block as its own chunk once the block closes, so parallel calls from a model + // that writes one block per call (Qwen's ) arrive in several chunks. An OpenAI stream numbers a turn's + // calls 0, 1, 2, ... across all its chunks, and clients and the daemon's gateways put the deltas that share an + // index together into one call; numbered per chunk, every such call was index 0 with the same id, and they came + // out as one call whose arguments were the calls' objects run together. Numbered as in the non-streaming reply. + const int first_call = static_cast(tool_calls_all.size()); if (!p.tool_calls.empty()) { for (auto& tc : p.tool_calls) tool_calls_all.push_back(tc); finish = "tool_calls"; @@ -1089,7 +2308,7 @@ void HandleChatCompletions(int fd, const Request& req) { } if (ok && !p.tool_calls.empty()) { ok = SendAll(fd, Sse(make_chunk(json::array( - {{{"index", 0}, {"delta", {{"tool_calls", OpenAiToolCalls(p.tool_calls, now_str, true)}}}, {"finish_reason", nullptr}}})))); + {{{"index", 0}, {"delta", {{"tool_calls", OpenAiToolCalls(p.tool_calls, now_str, true, first_call)}}}, {"finish_reason", nullptr}}})))); } if (!ok && !cancelled) { client_gone = cancelled = true; @@ -1100,33 +2319,58 @@ void HandleChatCompletions(int fd, const Request& req) { reasoning_all += p.reasoning; done = item.final; } + // The end of a reply that only looked as if it might become a stop string. + if (!held.empty() && !error) { + if (stream && !client_gone) { + SendAll(fd, Sse(make_chunk(json::array({{{"index", 0}, {"delta", {{"content", held}}}, {"finish_reason", nullptr}}})))); + } + text_all += held; + } if (client_gone) { Log("client disconnected — generation cancelled"); return; } json usage, timings; - UsageAndTimings(conv.get(), reasoning_tokens, usage, timings); - if (finish == "stop" && max_tokens && usage["completion_tokens"].get() >= *max_tokens) finish = "length"; + // Not after an error, whose reply carries neither: the generation may still be running then (the cancel above only + // asks it to stop), and the runtime copies the session's benchmark info under a lock its decode thread does not + // take while writing it. + if (!error) { + UsageAndTimings(conv.get(), reasoning_tokens, usage, timings); + if (finish == "stop" && !stopped && max_tokens && usage["completion_tokens"].get() >= *max_tokens) finish = "length"; + // A full context window ends the turn with an ordinary final chunk: the conversation API reports it as an error + // only with return_error_on_max_tokens_reached, which defaults to off and which LiteRT-LM 0.18.0's C API has no + // way to turn on. So it is told from the counts. The runtime stops once the KV cache holds max_num_tokens, and + // this request's conversation is new, so its cache holds just this prompt (earlier messages included) and the + // reply. The window is the engine's own, the one LiteRT-LM gives it (RuntimeWindow): the model's size when it is + // smaller than what was asked, and what the runtime picks when nothing was asked. + const int window = g_engine.window; + if (finish == "stop" && !stopped && window > 0 && usage["total_tokens"].get() >= window) finish = "length"; + } if (stream) { + // Stop at the first send that fails: to a client that has stopped reading, each further one would wait out the + // send timeout again while this request holds the engine. + bool ok; if (error) { // openai_common.OpenAIStreamFormatter.format_error - SendAll(fd, "event: response.error\ndata: " + json({{"error", "RuntimeError: " + *error}}).dump() + "\n\n"); + ok = SendAll(fd, "event: response.error\ndata: " + + json({{"error", "RuntimeError: " + *error}}).dump(-1, ' ', false, json::error_handler_t::replace) + "\n\n"); } else { json last = make_chunk(json::array({{{"index", 0}, {"delta", json::object()}, {"finish_reason", finish}}})); last["timings"] = timings; - SendAll(fd, Sse(last)); - if (include_usage) { + ok = SendAll(fd, Sse(last)); + if (ok && include_usage) { json u = make_chunk(json::array()); u["usage"] = usage; - SendAll(fd, Sse(u)); + ok = SendAll(fd, Sse(u)); } } - SendAll(fd, "data: [DONE]\n\n"); + if (ok) SendAll(fd, "data: [DONE]\n\n"); return; } + if (error && prompt_too_long) throw HttpError(400, "Invalid messages: " + *error); if (error) throw HttpError(500, "RuntimeError: " + *error); json message = {{"role", "assistant"}, {"content", text_all.empty() ? json(nullptr) : json(text_all)}}; if (!reasoning_all.empty()) message["reasoning_content"] = reasoning_all; @@ -1138,7 +2382,7 @@ void HandleChatCompletions(int fd, const Request& req) { {"choices", json::array({{{"index", 0}, {"message", message}, {"finish_reason", finish}}})}, {"usage", usage}, {"timings", timings}}; - SendJson(fd, req, 200, resp); + SendJson(fd, req, 200, resp, note_headers + window_headers); } catch (const HttpError& e) { Log("request failed: " + std::string(e.what())); if (!headers_sent) SendError(fd, req, e.status, e.what()); @@ -1156,7 +2400,11 @@ void HandleModels(int fd, const Request& req) { if (g_engine.engine) { struct stat st {}; long long mtime = ::stat(g_engine.key.path.c_str(), &st) == 0 ? static_cast(st.st_mtime) : 0; - data.push_back({{"id", g_engine.key.path}, {"object", "model"}, {"created", mtime}, {"owned_by", "litert-lm"}}); + json entry = {{"id", g_engine.key.path}, {"object", "model"}, {"created", mtime}, {"owned_by", "litert-lm"}}; + // The loaded model's context window (vLLM's name for it), and the larger one that was asked for, if any. + if (g_engine.window > 0) entry["max_model_len"] = g_engine.window; + if (g_engine.window_asked > 0) entry["max_model_len_requested"] = g_engine.window_asked; + data.push_back(entry); } SendJson(fd, req, 200, {{"object", "list"}, {"data", data}}); } @@ -1168,7 +2416,9 @@ void HandleConnection(int fd) { return; } if (g_verbose) Log(req.method + " " + req.path); - if (req.method == "OPTIONS") { + if (std::string why = CrossSiteRefusal(req); !why.empty()) { + SendError(fd, req, 403, why); + } else if (req.method == "OPTIONS") { SendResponse(fd, req, 200, "OK", "text/plain", ""); } else if (req.method == "GET" && req.path == "/health") { SendJson(fd, req, 200, {{"status", "ok"}}); @@ -1229,6 +2479,26 @@ void PrintHelp() { } // namespace int main(int argc, char** argv) { +#ifdef __linux__ + // Die with the process that started this one, by the kernel: SIGKILL the moment the thread that forked it ends. On + // Android that is the daemon's Node thread in the app process, so the server, and the model it holds, ends however + // the app process ends (Quit, a crash, Android killing the app), at once. The app's daemon starts it through + // libtether_exec.so, which arms the same signal before the exec (BUILDING-ENGINES.md, "The engine tether"); this + // covers every other way it is started: by a daemon without that helper, through the daemon's sh lifeline when the + // helper fails its self-test, or by hand. Armed before anything else runs, so even a server stuck loading its + // library goes. A parent process that ended before the prctl never sends it: this process was then already + // reparented to init, so the server does not start. This sees the process end, not the thread: had only the forking + // thread ended, this process would have moved to another thread of its parent and the signal would come when that + // one ends; the daemon's thread ends only with the daemon. A server that init starts on purpose is refused too; + // none is, as this binary ships only in the APK (host builds are for its tests). + if (::prctl(PR_SET_PDEATHSIG, SIGKILL) != 0) { + std::fprintf(stderr, "prctl(PR_SET_PDEATHSIG) failed: %s; the server will not end with its parent\n", + std::strerror(errno)); + } else if (::getppid() == 1) { + std::fprintf(stderr, "the process that started %s has already exited; not starting\n", kServerName); + return 1; + } +#endif std::string config_path, host = "127.0.0.1"; int port = 9379; std::vector args(argv + 1, argv + argc); @@ -1312,11 +2582,31 @@ int main(int argc, char** argv) { } int one = 1; ::setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one)); - // A client that connects and never sends a request must not hold a thread forever. Applies to reads only; - // a long generation's writes and the disconnect poll are unaffected. + // A client that connects and never sends a request must not hold a thread forever. timeval rcv_timeout{60, 0}; ::setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &rcv_timeout, sizeof(rcv_timeout)); - std::thread(HandleConnection, fd).detach(); + // Nor may one that stops reading. Chat completion and /v1/models responses are written under g_engine_mu, so a + // send blocked on a full socket held the engine, and with it every later chat and /v1/models request, for as long + // as the client kept the connection open. Now a send() gives up after 30 s: with a partial count if it got some + // bytes out, which SendAll takes as progress, else with an error, which ends a stream like a disconnect and cancels + // its generation. A client that reads nothing is dropped within one to two minutes: each send that still gets a few + // bytes out waits its own 30 s, and the kernel can free a little room once more after the buffers first fill. One + // that reads is served however slowly, as long as it never leaves the buffers full for 30 s. The disconnect poll is + // unaffected. + timeval snd_timeout{30, 0}; + ::setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &snd_timeout, sizeof(snd_timeout)); + try { + std::thread(HandleConnection, fd).detach(); + } catch (const std::exception& e) { + // No thread for this connection: std::system_error (EAGAIN) once the process is out of threads or address + // space, e.g. with hundreds of idle connections each holding a thread for up to 60 s, or std::bad_alloc. Uncaught, + // it ended the server (std::terminate) and with it the loaded model. Drop this connection and keep serving: + // threads come back as connections end. Pause as for accept errors above, so a backlog of pending connections is + // not dropped, and logged, in a tight loop. + Log(std::string("cannot start a connection thread: ") + e.what()); + ::close(fd); + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } } Log("Shutting down server..."); // Free the engine if no inference holds it; a generation still running is cut short by exit. diff --git a/engines/litert-lm-server/testdata/fake_litert_lm.cc b/engines/litert-lm-server/testdata/fake_litert_lm.cc new file mode 100644 index 0000000..af6ce5c --- /dev/null +++ b/engines/litert-lm-server/testdata/fake_litert_lm.cc @@ -0,0 +1,277 @@ +// A stand-in liblitert-lm.so for constrained-decoding.test.mjs. It implements the C API calls around creating a +// conversation and sending it a message the way LiteRT-LM 0.18.0 answers them for the kinds of model below, so the +// server's choices can be checked without a model. The test adds every other function the server resolves as one +// that does nothing. +// +// The kind of model is told by the start of the file name in the request's `model`: +// lfm2 refuses constrained decoding, as Lfm2DataProcessor::Create always does +// (runtime/conversation/model_data_processor/lfm2_data_processor.cc:53-56) +// fst-unsupported refuses it as Gemma3DataProcessor::Create does when the FST constraint library rejects the +// model's vocabulary (gemma3_data_processor.cc:124-152; gemma-3-270m-it is one) +// no-llg has no LLGuidance constraint provider (constraint_provider_factory.cc:43-56) +// broken cannot create any conversation +// anything else takes both (a Gemma model whose vocabulary the FST library supports) +// and, whatever the file's header says: an engine for gpu-broken cannot be created on gpu, as on a phone whose GPU +// path fails, and one for no-engine on no backend at all. +// Each conversation created is logged with what its config asked for: {"call": "conversation_create", "model": ..., +// "config": {"sampler": {...}, ...}}. +// A conversation answers with one chunk of text that says how it was created and sent, for example +// "constrained=1 tools=1 provider=0 constraint=0 creates=3", where creates counts every litert_lm_conversation_create +// call in the process; a message whose text starts with "say:" is answered with the rest of its text instead, three +// bytes to a chunk, as a model streams, and one whose text is "too long" fails as a prompt longer than the context +// window does. A regex that does not compile fails the send as llguidance does +// (llg_constraint_provider.cc:155-160). +// +// With FAKE_LITERT_LM_CALLS set to a file, the calls that say what the server asked the runtime for are appended to it, +// one JSON object per line ({"call": "", ...its arguments}), for +// model-settings.test.mjs to read. +#include +#include +#include +#include + +#include "c/conversation.h" +#include "c/engine.h" +#include "c/error_reporter.h" +#include "nlohmann/json.hpp" + +struct LiteRtLmEngineSettings { + std::string model, backend; +}; +struct LiteRtLmEngine { + std::string model; +}; +struct LiteRtLmSamplerParams { + nlohmann::json j; +}; +struct LiteRtLmSessionConfig { + nlohmann::json sampler; +}; +struct LiteRtLmThinkingConfig { + nlohmann::json j = nlohmann::json::object(); +}; +struct LiteRtLmConversationConfig { + bool constrained = false, tools = false, provider = false; + nlohmann::json config = nlohmann::json::object(); +}; +struct LiteRtLmConversation { + LiteRtLmConversationConfig config; +}; +struct LiteRtLmConversationOptionalArgs { + LiteRtLmConstraintType type = kLiteRtLmConstraintTypeNone; + std::string pattern; + nlohmann::json j = nlohmann::json::object(); // the other arguments, as logged with the message +}; +struct LiteRtLmRepetitionPenaltyConfig { + nlohmann::json j = {{"repetition_penalty", 1.0}, {"presence_penalty", 0.0}, {"frequency_penalty", 0.0}, {"window_size", 0}}; +}; +struct LiteRtLmStreamChunk { + std::string text; + const char* error; + bool final; +}; + +namespace { +std::string g_error; +int g_creates = 0; + +void Record(const nlohmann::json& call) { + static std::mutex mu; + const char* path = std::getenv("FAKE_LITERT_LM_CALLS"); + if (!path || !*path) return; + std::lock_guard lock(mu); + if (FILE* f = std::fopen(path, "a")) { + std::fputs((call.dump() + "\n").c_str(), f); + std::fclose(f); + } +} + +bool Is(const std::string& model, const char* kind) { return model.rfind(kind, 0) == 0; } + +std::string Basename(const char* path) { + std::string p = path ? path : ""; + size_t slash = p.find_last_of('/'); + return slash == std::string::npos ? p : p.substr(slash + 1); +} +} // namespace + +extern "C" { + +const char* litert_lm_get_last_error_message(void) { return g_error.c_str(); } +void litert_lm_clear_last_error(void) { g_error.clear(); } + +LiteRtLmEngineSettings* litert_lm_engine_settings_create(const char* model_path, const char* backend, + const char* vision_backend, const char* audio_backend) { + auto name = [](const char* b) { return b ? nlohmann::json(b) : nlohmann::json(nullptr); }; + Record({{"call", "engine_settings_create"}, {"model", Basename(model_path)}, {"backend", name(backend)}, + {"vision_backend", name(vision_backend)}, {"audio_backend", name(audio_backend)}}); + return new LiteRtLmEngineSettings{Basename(model_path), backend ? backend : ""}; +} +void litert_lm_engine_settings_set_max_num_tokens(LiteRtLmEngineSettings* s, int n) { + Record({{"call", "engine_settings_set_max_num_tokens"}, {"model", s->model}, {"max_num_tokens", n}}); +} +void litert_lm_engine_settings_delete(LiteRtLmEngineSettings* s) { delete s; } +LiteRtLmEngine* litert_lm_engine_create(const LiteRtLmEngineSettings* s) { + Record({{"call", "engine_create"}, {"model", s->model}, {"backend", s->backend}}); + if (Is(s->model, "gpu-broken") && s->backend == "gpu") { + g_error = "UNAVAILABLE: no usable OpenCL device"; + return nullptr; + } + if (Is(s->model, "no-engine")) { + g_error = "INTERNAL: cannot load the model on " + s->backend; + return nullptr; + } + return new LiteRtLmEngine{s->model}; +} +void litert_lm_engine_delete(LiteRtLmEngine* e) { delete e; } + +LiteRtLmSessionConfig* litert_lm_session_config_create() { return new LiteRtLmSessionConfig; } +void litert_lm_session_config_delete(LiteRtLmSessionConfig* c) { delete c; } +LiteRtLmSamplerParams* litert_lm_sampler_params_create(LiteRtLmSamplerType type) { + return new LiteRtLmSamplerParams{{{"type", type}}}; +} +void litert_lm_sampler_params_delete(LiteRtLmSamplerParams* p) { delete p; } +void litert_lm_sampler_params_set_top_k(LiteRtLmSamplerParams* p, int32_t k) { p->j["top_k"] = k; } +void litert_lm_sampler_params_set_top_p(LiteRtLmSamplerParams* p, float v) { p->j["top_p"] = v; } +void litert_lm_sampler_params_set_temperature(LiteRtLmSamplerParams* p, float v) { p->j["temperature"] = v; } +void litert_lm_sampler_params_set_seed(LiteRtLmSamplerParams* p, int32_t seed) { p->j["seed"] = seed; } +void litert_lm_session_config_set_sampler_params(LiteRtLmSessionConfig* c, const LiteRtLmSamplerParams* p) { + c->sampler = p->j; +} +void litert_lm_conversation_config_set_session_config(LiteRtLmConversationConfig* c, const LiteRtLmSessionConfig* s) { + c->config["sampler"] = s->sampler; +} + +LiteRtLmThinkingConfig* litert_lm_thinking_config_create() { return new LiteRtLmThinkingConfig; } +void litert_lm_thinking_config_delete(LiteRtLmThinkingConfig* c) { delete c; } +void litert_lm_thinking_config_set_enable_thinking(LiteRtLmThinkingConfig* c, bool enable) { c->j["enable"] = enable; } +void litert_lm_thinking_config_set_thinking_token_budget(LiteRtLmThinkingConfig* c, int32_t budget) { + c->j["budget"] = budget; +} +void litert_lm_conversation_config_set_thinking_config(LiteRtLmConversationConfig* c, const LiteRtLmThinkingConfig* t) { + c->config["thinking"] = t->j; +} +void litert_lm_conversation_config_set_messages(LiteRtLmConversationConfig* c, const char* messages_json) { + c->config["messages"] = nlohmann::json::parse(messages_json ? messages_json : "null", nullptr, false); +} +void litert_lm_conversation_config_set_extra_context(LiteRtLmConversationConfig* c, const char* extra_context_json) { + c->config["extra_context"] = nlohmann::json::parse(extra_context_json ? extra_context_json : "null", nullptr, false); +} + +LiteRtLmConversationConfig* litert_lm_conversation_config_create() { return new LiteRtLmConversationConfig; } +void litert_lm_conversation_config_delete(LiteRtLmConversationConfig* c) { delete c; } +void litert_lm_conversation_config_set_enable_constrained_decoding(LiteRtLmConversationConfig* c, bool enable) { + c->constrained = enable; +} +void litert_lm_conversation_config_set_tools(LiteRtLmConversationConfig* c, const char* tools_json) { + c->tools = tools_json && *tools_json; +} +void litert_lm_conversation_config_set_constraint_provider(LiteRtLmConversationConfig* c, + const LiteRtLmConstraintProviderType* type) { + c->provider = type != nullptr; +} + +// Conversation::Create's order: the data processor (which gets the constrained-decoding flag), then the provider. +LiteRtLmConversation* litert_lm_conversation_create(LiteRtLmEngine* engine, LiteRtLmConversationConfig* c) { + ++g_creates; + const std::string& m = engine->model; + Record({{"call", "conversation_create"}, {"model", m}, {"config", c->config}}); + if (Is(m, "broken")) { + g_error = "INTERNAL: cannot create a session"; + } else if (c->constrained && Is(m, "lfm2")) { + g_error = "FAILED_PRECONDITION: Constrained decoding is not supported for Lfm2DataProcessor."; + } else if (c->constrained && Is(m, "fst-unsupported")) { + g_error = "INTERNAL: Failed to create GemmaModelConstraintProvider."; + } else if (c->provider && Is(m, "no-llg")) { + g_error = + "INVALID_ARGUMENT: LlGuidanceConfig::eos_id wasn't set and no valid stop token was found in SessionConfig."; + } else { + return new LiteRtLmConversation{*c}; + } + return nullptr; +} +void litert_lm_conversation_delete(LiteRtLmConversation* conv) { delete conv; } +void litert_lm_conversation_cancel_process(LiteRtLmConversation*) { Record({{"call", "cancel_process"}}); } +LiteRtLmBenchmarkInfo* litert_lm_conversation_get_benchmark_info(LiteRtLmConversation*) { return nullptr; } + +LiteRtLmConversationOptionalArgs* litert_lm_conversation_optional_args_create() { + return new LiteRtLmConversationOptionalArgs; +} +void litert_lm_conversation_optional_args_delete(LiteRtLmConversationOptionalArgs* a) { delete a; } +void litert_lm_conversation_optional_args_set_max_output_tokens(LiteRtLmConversationOptionalArgs* a, int n) { + a->j["max_output_tokens"] = n; +} +LiteRtLmRepetitionPenaltyConfig* litert_lm_repetition_penalty_config_create() { return new LiteRtLmRepetitionPenaltyConfig; } +void litert_lm_repetition_penalty_config_delete(LiteRtLmRepetitionPenaltyConfig* c) { delete c; } +void litert_lm_repetition_penalty_config_set_repetition_penalty(LiteRtLmRepetitionPenaltyConfig* c, float v) { + c->j["repetition_penalty"] = v; +} +void litert_lm_repetition_penalty_config_set_presence_penalty(LiteRtLmRepetitionPenaltyConfig* c, float v) { + c->j["presence_penalty"] = v; +} +void litert_lm_repetition_penalty_config_set_frequency_penalty(LiteRtLmRepetitionPenaltyConfig* c, float v) { + c->j["frequency_penalty"] = v; +} +void litert_lm_repetition_penalty_config_set_window_size(LiteRtLmRepetitionPenaltyConfig* c, int n) { + c->j["window_size"] = n; +} +void litert_lm_conversation_optional_args_set_repetition_penalty_config(LiteRtLmConversationOptionalArgs* a, + const LiteRtLmRepetitionPenaltyConfig* c) { + a->j["repetition_penalty_config"] = c ? c->j : nlohmann::json(nullptr); +} +void litert_lm_conversation_optional_args_set_constraint(LiteRtLmConversationOptionalArgs* a, + LiteRtLmConstraintType type, const char* pattern) { + a->type = type; + a->pattern = pattern ? pattern : ""; +} + +int litert_lm_conversation_send_message_stream(LiteRtLmConversation* conv, const char* message, + const char* extra_context, const LiteRtLmConversationOptionalArgs* a, + LiteRtLmStreamCallback callback, void* data) { + Record({{"call", "send_message"}, + {"message", nlohmann::json::parse(message ? message : "null", nullptr, false)}, + {"extra_context", nlohmann::json::parse(extra_context ? extra_context : "null", nullptr, false)}, + {"args", a ? a->j : nlohmann::json(nullptr)}}); + const LiteRtLmConstraintType type = a ? a->type : kLiteRtLmConstraintTypeNone; + if (type == kLiteRtLmConstraintTypeRegex && a->pattern == "(") { + g_error = "INTERNAL: Failed to create LLGuidance constraint: invalid regex \"(\": unclosed group"; + return -1; + } + const LiteRtLmConversationConfig& c = conv->config; + std::string text = "constrained=" + std::to_string(c.constrained) + " tools=" + std::to_string(c.tools) + + " provider=" + std::to_string(c.provider) + " constraint=" + std::to_string(type) + + " creates=" + std::to_string(g_creates); + nlohmann::json m = nlohmann::json::parse(message ? message : "null", nullptr, false); + std::string said; + if (m.is_object() && m.contains("content")) { + const nlohmann::json& content = m["content"]; + if (content.is_string()) said = content.get(); + if (content.is_array() && !content.empty() && content[0].is_object()) said = content[0].value("text", ""); + } + if (said == "too long") { + LiteRtLmStreamChunk failed{"", "INVALID_ARGUMENT: Input token ids are too long. Exceeding the maximum number of " + "tokens allowed: 4209 >= 4096", + true}; + callback(data, &failed); + return 0; + } + std::vector chunks = {text}; + if (said.rfind("say:", 0) == 0) { + chunks.clear(); + for (size_t i = 4; i < said.size(); i += 3) chunks.push_back(said.substr(i, 3)); + } + for (const auto& piece : chunks) { + nlohmann::json out = {{"role", "assistant"}, {"content", {{{"type", "text"}, {"text", piece}}}}}; + LiteRtLmStreamChunk reply{out.dump(), nullptr, false}; + callback(data, &reply); + } + LiteRtLmStreamChunk end{"", nullptr, true}; + callback(data, &end); + return 0; +} + +const char* litert_lm_stream_chunk_get_error(const LiteRtLmStreamChunk* chunk) { return chunk->error; } +const char* litert_lm_stream_chunk_get_text(const LiteRtLmStreamChunk* chunk) { return chunk->text.c_str(); } +bool litert_lm_stream_chunk_is_final(const LiteRtLmStreamChunk* chunk) { return chunk->final; } + +} // extern "C" diff --git a/engines/litert-lm-server/testdata/harness.mjs b/engines/litert-lm-server/testdata/harness.mjs new file mode 100644 index 0000000..f79dab7 --- /dev/null +++ b/engines/litert-lm-server/testdata/harness.mjs @@ -0,0 +1,289 @@ +// What the server's tests share (constrained-decoding.test.mjs, model-settings.test.mjs): building the server for this +// machine with c++, as parent-death.test.mjs does, against testdata/fake_litert_lm.cc, a stand-in liblitert-lm.so; +// starting it; sending it requests, every wait bounded; and writing stand-in .litertlm files. +// Linux only, and skipped without a C and a C++ compiler. +import assert from 'node:assert/strict' +import { spawn, spawnSync } from 'node:child_process' +import { existsSync, readFileSync, writeFileSync } from 'node:fs' +import { createServer } from 'node:net' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' + +export const root = fileURLToPath(new URL('..', import.meta.url)) + +// A compiler that does not answer in 10 s counts as missing (the timeout sets .error), so a file skips cleanly. +const missing = ['cc', 'c++'].filter((c) => spawnSync(c, ['--version'], { stdio: 'ignore', timeout: 10_000 }).error) +export const skip = process.platform !== 'linux' ? 'the server is tested on Linux only' + : missing.length ? `no ${missing.join(' or ')} on this machine` : false +export const realLibrary = process.env.LITERT_LM_TEST_LIBRARY +export const realModels = (process.env.LITERT_LM_TEST_MODELS ?? '').split(',').filter(Boolean) +export const skipReal = skip || (!realLibrary || !realModels.length + ? 'set LITERT_LM_TEST_LIBRARY and LITERT_LM_TEST_MODELS to run the server on real models' : false) + +// Builds the stand-in library and the server into `work`; returns the server's path. The stand-in is +// testdata/fake_litert_lm.cc, and every other function the server resolves (src/litert_lm_api.h) as one that does +// nothing. Beside the server, where LoadLiteRtLm looks first after LITERT_LM_LIBRARY. Each build is bounded by +// itself: spawnSync holds the event loop, so a hook's own timeout cannot end one. +export function buildServer(work) { + const header = readFileSync(join(root, 'src', 'litert_lm_api.h'), 'utf8') + const names = [...header.matchAll(/^\s*X\((\w+)\)/gm)].map((m) => m[1]) + assert.ok(names.length > 40, 'the C API list in litert_lm_api.h was not found') + const source = readFileSync(join(root, 'testdata', 'fake_litert_lm.cc'), 'utf8') + const defined = new Set([...source.matchAll(/^\S.*?\b(litert_lm_\w+)\(/gm)].map((m) => m[1])) + writeFileSync(join(work, 'rest.c'), names.filter((n) => !defined.has(n)).map((n) => `void ${n}(void) {}\n`).join('')) + const cxx = (args) => { + const r = spawnSync('c++', args, { encoding: 'utf8', timeout: 120_000 }) + assert.equal(r.status, 0, r.error ? `c++: ${r.error.message}` : r.stderr) + } + cxx([ + '-std=c++17', '-shared', '-fPIC', '-x', 'c++', join(root, 'testdata', 'fake_litert_lm.cc'), '-x', 'c', join(work, 'rest.c'), + '-I', join(root, 'third_party', 'litert_lm'), '-I', join(root, 'third_party'), '-o', join(work, 'liblitert-lm.so'), + ]) + const binary = join(work, 'litertlm_server') + cxx([ + '-std=c++17', '-O0', '-w', '-I', join(root, 'third_party', 'litert_lm'), '-I', join(root, 'third_party'), + '-DLITERT_LM_VERSION="test"', join(root, 'src', 'main.cc'), join(root, 'src', 'litert_lm_api.cc'), + '-ldl', '-pthread', '-o', binary, + ]) + return binary +} + +// Every server started here, so a file's after() can kill what its tests left. +export const servers = [] +export function killServers() { + for (const s of servers) { + try { + process.kill(s.pid, 'SIGKILL') + } catch {} + } +} + +export function freePort() { + return new Promise((resolve, reject) => { + const s = createServer().once('error', reject) + s.listen(0, '127.0.0.1', () => { + const { port } = s.address() + s.close(() => resolve(port)) + }) + }) +} + +export async function startServer(binary, env, args = []) { + const port = await freePort() + const child = spawn(binary, ['serve', '--host', '127.0.0.1', '--port', String(port), ...args], { env, stdio: ['ignore', 'ignore', 'pipe'] }) + servers.push(child) + const server = { port, stderr: '', child } + child.stderr.setEncoding('utf8') + child.stderr.on('data', (d) => { server.stderr += d }) + await new Promise((resolve, reject) => { + // It logs this before it loads any model, so 30 s is plenty; the server is killed when the tests end. + const timer = setTimeout(() => reject(new Error(`the server did not start in 30 s: ${server.stderr}`)), 30_000) + const onData = () => { + if (server.stderr.includes('Starting OpenAI-compatible API server')) { + clearTimeout(timer) + resolve() + } + } + child.stderr.on('data', onData) + child.once('exit', (code) => { + clearTimeout(timer) + reject(new Error(`the server exited ${code}: ${server.stderr}`)) + }) + }) + return server +} + +export function stopServer(server) { + try { + process.kill(server.child.pid, 'SIGKILL') + } catch {} +} + +// fetch, bounded: without a signal undici waits 300 s for a reply's headers and then says only "fetch failed". +export async function request(url, init, ms) { + const signal = AbortSignal.timeout(ms) + try { + const res = await fetch(url, { ...init, signal }) + return { status: res.status, text: await res.text(), headers: res.headers } + } catch (err) { + if (!signal.aborted) throw err + throw new Error(`${init?.method ?? 'GET'} ${url} ${init?.body ?? ''}: no reply in ${ms / 1000} s`) + } +} + +// A stand-in answers at once; a real model gets longer, since it loads on its first request. +export async function chat(server, body, ms = 30_000) { + const res = await request(`http://127.0.0.1:${server.port}/v1/chat/completions`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }, ms) + const text = res.text + if (body.stream && res.status === 200) { + const events = text.split('\n\n').filter((e) => e.startsWith('data: ')).map((e) => e.slice(6)) + assert.equal(events.at(-1), '[DONE]', text) + const chunks = events.slice(0, -1).map((e) => JSON.parse(e)) + return { + status: res.status, + content: chunks.map((c) => c.choices[0]?.delta?.content ?? '').join(''), + reasoning: chunks.map((c) => c.choices[0]?.delta?.reasoning_content ?? '').join(''), + finish: chunks.at(-1).choices[0].finish_reason, + chunks, + headers: res.headers, + } + } + const json = JSON.parse(text) + if (res.status !== 200) return { status: res.status, error: json.error.message } + return { + status: res.status, + content: json.choices[0].message.content ?? '', + reasoning: json.choices[0].message.reasoning_content ?? '', + tool_calls: json.choices[0].message.tool_calls, + finish: json.choices[0].finish_reason, + usage: json.usage, + headers: res.headers, + } +} + +export const sleep = (ms) => new Promise((r) => setTimeout(r, ms)) + +// What the server has logged, all of it up to now: a reply can arrive before the lines logged while it was made. With +// --verbose the server logs each request as it reads it, so once a GET /health made now shows up, all are in. +export async function logged(server) { + const marks = () => server.stderr.split('\n').filter((l) => l === 'GET /health').length + const seen = marks() + assert.equal((await request(`http://127.0.0.1:${server.port}/health`, undefined, 5000)).status, 200) + for (const end = Date.now() + 5000; marks() === seen && Date.now() < end;) await sleep(10) + assert.ok(marks() > seen, 'the server did not log GET /health') + return server.stderr.split('\n') +} + +// The calls the stand-in library logged (FAKE_LITERT_LM_CALLS, one JSON object per line), in order. +export function calls(file) { + return existsSync(file) ? readFileSync(file, 'utf8').split('\n').filter(Boolean).map((l) => JSON.parse(l)) : [] +} + +// A FlatBuffer, laid out front to back from a tree of values: every uoffset points forward, as FlatBuffers requires, +// each vtable sits right before its table, and each table starts 8-aligned. fb.table([...]) lists a table's fields by +// index (undefined leaves one out); a field is fb.u8(v), fb.u32(v), fb.i32(v) or fb.u64(v), a string, a table, a +// vector of tables or strings (fb.vector([...])) or of int32s (fb.ints([...])). +export const fb = { + u8: (v) => ({ size: 1, put: (b, at) => b.writeUInt8(v, at) }), + u32: (v) => ({ size: 4, put: (b, at) => b.writeUInt32LE(v, at) }), + i32: (v) => ({ size: 4, put: (b, at) => b.writeInt32LE(v, at) }), + u64: (v) => ({ size: 8, put: (b, at) => b.writeBigUInt64LE(BigInt(v), at) }), + table: (fields) => ({ table: fields }), + vector: (items) => ({ vector: items }), + ints: (values) => ({ ints: values }), +} +export function flatbuffer(root) { + const parts = [] + let pos = 0 + const put = (b) => { parts.push(b); pos += b.length; return pos - b.length } + const pad = (to, extra = 0) => { const n = (to - ((pos + extra) % to)) % to; if (n) put(Buffer.alloc(n)) } + const patches = [] // [position of a uoffset, position of its target] + // Lays out `value` (a table, vector or string) and what it points to; returns its position. + const place = (value) => { + if (typeof value === 'string') { + pad(4) + const b = Buffer.alloc(4 + Buffer.byteLength(value) + 1) + b.writeUInt32LE(Buffer.byteLength(value)) + b.write(value, 4) + return put(b) + } + if (value.ints) { + pad(4) + const b = Buffer.alloc(4 + 4 * value.ints.length) + b.writeUInt32LE(value.ints.length) + value.ints.forEach((v, k) => b.writeInt32LE(v, 4 + 4 * k)) + return put(b) + } + if (value.vector) { + pad(4) + const at = put(Buffer.alloc(4 + 4 * value.vector.length)) + parts.at(-1).writeUInt32LE(value.vector.length) + value.vector.forEach((item, k) => patches.push([at + 4 + 4 * k, place(item)])) + return at + } + // A table: its soffset, then each field at its own alignment; the vtable gives each field's place in it. + const slots = [] + let size = 4 + for (const field of value.table) { + if (field === undefined) { slots.push(0); continue } + const width = field.size ?? 4 + size += (width - (size % width)) % width + slots.push(size) + size += width + } + size += (4 - (size % 4)) % 4 + const vtable = Buffer.alloc(4 + 2 * slots.length) + vtable.writeUInt16LE(vtable.length, 0) + vtable.writeUInt16LE(size, 2) + slots.forEach((at, k) => vtable.writeUInt16LE(at, 4 + 2 * k)) + pad(8, vtable.length) + put(vtable) + const table = Buffer.alloc(size) + table.writeInt32LE(vtable.length, 0) + const at = put(table) + const later = [] + value.table.forEach((field, k) => { + if (field === undefined) return + if (field.put) field.put(table, slots[k]) + else later.push([at + slots[k], field]) + }) + for (const [slot, child] of later) patches.push([slot, place(child)]) + return at + } + put(Buffer.alloc(8)) // the root uoffset, and room before the root's vtable + patches.push([0, place(root)]) + const out = Buffer.concat(parts) + for (const [at, target] of patches) out.writeUInt32LE(target - at, at) + return out +} + +// A stand-in .litertlm the server can read as the runtime does: "LITERTLM", three uint32 versions, padding, the +// uint64 header end, then the header FlatBuffer (schema/core/litertlm_header_schema.fbs) listing `sections`, each +// { type, bytes, items } (3 = TFLiteModel, 5 = LlmMetadataProto, 9 = ExecutorMetadataProto; items, the section's +// string key-value pairs, such as model_type), whose bytes follow at 16 KiB steps as in real files. The stand-in +// library does not read it. +export function litertlmFile(path, sections) { + let next = 16384 + const placed = sections.map((s) => { + const bytes = s.bytes ?? Buffer.alloc(0) + const begin = next + next = begin + Math.ceil(Math.max(bytes.length, 1) / 16384) * 16384 + return { ...s, bytes, begin } + }) + const item = ([key, value]) => fb.table([key, fb.u8(9), fb.table([value])]) // KeyValuePair, its value a StringValue + const header = flatbuffer(fb.table([undefined, fb.table([fb.vector(placed.map((s) => fb.table([ + s.items ? fb.vector(Object.entries(s.items).map(item)) : undefined, fb.u64(s.begin), fb.u64(s.begin + s.bytes.length), fb.u8(s.type), + ])))])])) + assert.ok(32 + header.length <= 16384, 'the header must fit in 16 KiB') + const u32 = (v) => { const b = Buffer.alloc(4); b.writeUInt32LE(v); return b } + const u64 = (v) => { const b = Buffer.alloc(8); b.writeBigUInt64LE(BigInt(v)); return b } + const preamble = Buffer.concat([Buffer.from('LITERTLM'), u32(1), u32(0), u32(0), u32(0), u64(32 + header.length)]) + const file = Buffer.alloc(next) + Buffer.concat([preamble, header]).copy(file) + for (const s of placed) s.bytes.copy(file, s.begin) + writeFileSync(path, file) + return path +} + +// Protobuf encoding, for the sections above: field(n, bytes or a nested array of fields), varint(n, v). +export const proto = { + varint(n, v) { + const out = [n << 3] + for (; v > 127; v >>>= 7) out.push((v & 127) | 128) + out.push(v) + return Buffer.from(out) + }, + field(n, value) { + const body = Buffer.isBuffer(value) ? value : Array.isArray(value) ? Buffer.concat(value) : Buffer.from(value) + const len = [] + for (let v = body.length; ; v >>>= 7) { + if (v > 127) len.push((v & 127) | 128) + else { len.push(v); break } + } + return Buffer.concat([Buffer.from([(n << 3) | 2, ...len]), body]) + }, +} diff --git a/engines/tether-exec/tether_exec.c b/engines/tether-exec/tether_exec.c new file mode 100644 index 0000000..bfc317e --- /dev/null +++ b/engines/tether-exec/tether_exec.c @@ -0,0 +1,65 @@ +// libtether_exec.so [args...] +// +// Ties an engine to the app process, by the kernel. The daemon starts each engine as +// `libtether_exec.so ` (TurboLLM's util/tether.ts). This arms +// PR_SET_PDEATHSIG(SIGKILL) and then execs the engine in place, so the engine keeps this process's +// pid, stdio and exit status, no extra process is left behind, and the kernel SIGKILLs the engine +// the moment the daemon's thread that started it ends: however the app process ends (Quit, a +// crash, Process.killProcess, the system killing the app). Without it an engine's only tie was +// system_server killing the app's cgroup after the app process had died. +// +// Why a separate executable: Node runs nothing in the child between fork and exec, and the death +// signal belongs to the process and is kept across execve(2) (not for a set-user-ID program, which +// no engine is), so a process that then becomes the engine has to arm it. It follows the thread +// that forked, not the process: libuv forks on the thread that calls spawn, which is the thread +// MainActivity starts node::Start on, and that thread lives exactly as long as the daemon. +// +// Named lib*.so because Android installs into nativeLibraryDir, the only place an app may execute +// files from, only the files under lib// that match lib*.so. Built for each ABI the APK ships +// by scripts/stage-tether-exec.mjs; BUILDING-ENGINES.md, "The engine tether". +// +// Exit status 127, without running the program, when: +// - the arguments are wrong, or the daemon pid is not a number above 1; +// - this process's parent is not that pid. The daemon process ended between the fork and the +// prctl, so this process was already reparented out of it and no signal would ever come (or +// the daemon passed another pid, which is a bug). This sees the process end, not the thread: +// had only the forking thread ended, the kernel would have moved this process to another +// thread of the daemon, getppid() would be unchanged, and the armed signal would come when +// that thread ends instead. The daemon's thread ends only with the daemon, so the two agree; +// - the exec fails (the message on stderr says why). +// If prctl itself fails, the program is started without the signal, with a message on stderr: the +// daemon's boot self-test then sees a program outlive its parent and uses its sh lifeline instead. +#include +#include +#include +#include +#include +#include +#include +#include + +int main(int argc, char **argv) { + if (argc < 3) { + fputs("usage: libtether_exec.so [args...]\n", stderr); + return 127; + } + char *end = NULL; + errno = 0; + long parent = strtol(argv[1], &end, 10); + if (errno != 0 || end == argv[1] || *end != '\0' || parent <= 1 || parent > INT_MAX) { + fprintf(stderr, "libtether_exec.so: bad daemon pid '%s'; not starting %s\n", argv[1], argv[2]); + return 127; + } + if (prctl(PR_SET_PDEATHSIG, SIGKILL) != 0) { + fprintf(stderr, "libtether_exec.so: prctl(PR_SET_PDEATHSIG) failed: %s; starting %s without it\n", + strerror(errno), argv[2]); + } else if (getppid() != (pid_t)parent) { + // Checked after the prctl: a daemon that ends from here on fires the signal. + fprintf(stderr, "libtether_exec.so: the parent is %d, not the daemon %ld, which has exited; not starting %s\n", + (int)getppid(), parent, argv[2]); + return 127; + } + execvp(argv[2], argv + 2); + fprintf(stderr, "libtether_exec.so: cannot run %s: %s\n", argv[2], strerror(errno)); + return 127; +} diff --git a/play-store-assets/listing-copy.md b/play-store-assets/listing-copy.md index 54b18d3..73787ac 100644 --- a/play-store-assets/listing-copy.md +++ b/play-store-assets/listing-copy.md @@ -52,8 +52,8 @@ WHAT YOU CAN DO model — fully offline, even in airplane mode. - Discover and download models straight from Hugging Face, right inside the app, with one tap. -- The inference engine installs itself automatically the first time you need - it — no manual setup, no compiling, no terminal. +- The inference engines come built into the app — no setup, no compiling, + no terminal. - Create custom AI agents with their own system prompt and skill/tool access, for different tasks — coding help, writing, research, or a completely blank slate with zero injected instructions. @@ -70,12 +70,21 @@ PRIVACY BY DESIGN - No account, no login, no cloud backend for your conversations. - No telemetry, no analytics, no crash-reporting SDK phoning home in this build. -- Chats and downloaded models live only in this app's private storage on - your device. -- Your prompts, files, and model weights are never sent to us or anyone - else. The only network requests this app makes are ones you triggered - yourself: downloading a model you picked, fetching the inference engine - binary the first time it's needed, or a tool call you explicitly approved. +- Your chats stay in this app's private storage on your device. Android's + own backup to your Google account and its transfer to a new phone also + copy them off it, if you use them (with a screen lock on Android 9 and + later, the backup is end-to-end encrypted). Downloaded models stay in + the app's storage unless you pick another folder for them. +- Your prompts, files, and model weights are never sent to us. They can + leave the phone through that backup (which leaves models out), a tool + call you approve, a device or service you connect the app to yourself, + or an image in a reply. Images in a reply or a model card load from + the website they name as soon as they show, without asking you, so a + reply can make the app contact that site and pass it text from the + chat in the image's address. Apart from those images, the app goes + online only for what you ask of it: browsing Hugging Face and + downloading the models you pick, a tool call you explicitly approve, + or a device or service you connect it to. WORTH KNOWING @@ -90,9 +99,9 @@ WHO IT'S FOR Privacy-conscious users, local-AI enthusiasts, developers who already run TurboLLM or llama.cpp on a computer, and anyone curious what a genuinely offline LLM chat feels like on a phone — under your control, on your -hardware, with nothing leaving the device. +hardware. ``` -(2,736 chars — well under the 4,000 limit; keyword density: "local" x6, "offline" x5, "AI"/"LLM" +(3,336 chars — well under the 4,000 limit; keyword density: "local" x6, "offline" x5, "AI"/"LLM" x9, "on-device" x2, "privacy/private" x3, "no cloud/account/subscription/telemetry" x5, plus feature nouns tuned for search — Hugging Face, GGUF, agents, tokens, llama.cpp.) diff --git a/play-store-assets/privacy-notes.md b/play-store-assets/privacy-notes.md index 55c4956..035b264 100644 --- a/play-store-assets/privacy-notes.md +++ b/play-store-assets/privacy-notes.md @@ -22,7 +22,8 @@ embedded daemon with: ```kotlin args = arrayOf("--no-telemetry", "--port", ANDROID_DAEMON_PORT.toString()), ``` -— `composeApp/src/androidMain/kotlin/com/turbollm/app/MainActivity.kt:69` +— `composeApp/src/androidMain/kotlin/com/turbollm/app/MainActivity.kt`, in the `startScript` call +that launches the daemon `--no-telemetry` is documented as a one-way kill switch in the telemetry doc — no runtime toggle can turn it back on. So the desktop-style telemetry system (usage levels, event log, opt-out UI @@ -45,43 +46,163 @@ Material — no Firebase, no Crashlytics, no ads SDK, no analytics SDK present * this file) and grepped all Kotlin source for network calls. Findings below describe the app AS CURRENTLY CODED; once Firebase is integrated, add a corresponding "app diagnostics" row. -1. **Native app code (Kotlin) only ever talks to `127.0.0.1`** — `DaemonClient.kt` and - `DaemonWebView.kt` both hit `http://127.0.0.1:7000` (the embedded daemon), nothing external. +1. **Native app code (Kotlin) only ever talks to `127.0.0.1`**: all of it goes to the embedded + daemon at `http://127.0.0.1:7000`. `DaemonWebView.kt` loads the UI from there, and + `DaemonAuth.kt` makes the app's other calls to it: the check that the daemon proves itself + before the app sends its secret, and the authenticated requests from `EngineService.kt` + (activity, downloads, status, and the request to write the chat database to disk before + Quit and before a close with nothing running ends the app), `ModelImport.kt` (adding the + imported-models folder) and `StorageAccess.kt` (rescan). (`DaemonClient.kt` in `commonMain` + points at `127.0.0.1:6996`; it is unused Spike A code that only the old `App()` screen, which + the Android app never shows, constructs.) The web UI in that WebView goes further, though: it + loads the images its pages point at from any https host, and from any host over plain http + on port 80. The daemon's CSP for the app allows that on purpose + (`APP_SECRET_CSP` in its `server.ts`: images and media from `'self'`, any `https:` host, + `http://*`, which with no port matches plain http on port 80 only, `data:` and `blob:`), and + `DaemonWebView.kt` refuses only other `127.0.0.1` ports. Those are the images in a model card + (the README in the Hugging Face dialog, which can embed images from any site) and markdown + images in chat replies and the other text the UI renders as markdown. An image in a reply loads + as soon as the reply shows, with no approval, so a reply (after a prompt injection in a pasted + file or a search result, say) can make the phone contact a host the model names and pass it + text from the chat in the image's URL. That behaviour is TurboLLM#287's "Known, not fixed + here" entry "Markdown images in chat", an owner decision; fill in the Data safety form with it + in mind. 2. **The embedded daemon (the same `turbollm` product code that runs on desktop) makes the same - categories of outbound request desktop does, all user-initiated:** + categories of outbound request desktop does, all user-initiated except where noted:** + - Searching and browsing Hugging Face from Discover: opening it lists popular repos, typing + searches them, and opening a repo fetches its details, its file list and its model card (the + README; for a repo of checkpoints, their config files too), all from `huggingface.co` over + HTTPS, with the user's Hugging Face token when one is set (`hf/hf.ts`). - Downloading a model you chose, from Hugging Face. - - Downloading the inference engine binary the first time it's needed (an NDK-cross-compiled - `llama-server`, hosted on TurboLLM's own GitHub/Hugging Face releases — ADR-391). + - Downloading an inference engine, which a normal install never does: the engines ship inside + the APK (the CPU `llama-server`, the LiteRT-LM server unless a build turns it off, and the + Vulkan `llama-server` in builds made with it) and are registered from there at every boot + (`engines/seed.ts`). The app has no way to install another: Android lets it run only code + installed with it, so its Engines page lists the built-in engines with no install control. + The daemon downloads one (TurboLLM's own Android build of `llama-server`, from Hugging Face) + in two cases only: once at boot, not started by you, if no engine in the APK could be + registered at all, and when a client calls its engine-install API directly, which nothing in + the app does. - Any in-chat tool call you explicitly approve (e.g. `web_search`), same approval-gated flow as desktop. - **Not present on this build:** the periodic telemetry upload — disabled per above. + - **Not present in the app either:** desktop's check for a newer TurboLLM, which fetches + `registry.npmjs.org/turbollm/latest` at every daemon start and when Settings reads a stale + answer. The app updates through Play, so the daemon skips that check inside it (TurboLLM#287, + "fix(update): the Android app does not check npm for an app update"); Termux installs, which + update through npm, keep it. 3. **`AndroidManifest.xml` permissions**, and why each exists: - `INTERNET` — the above network use. - - `REQUEST_IGNORE_BATTERY_OPTIMIZATIONS` — lets the user optionally exempt TurboLLM from the OS - killing an in-progress local inference run. - - No location, camera, microphone, contacts, notification, or broad storage permission - requested. (`FOREGROUND_SERVICE`/`FOREGROUND_SERVICE_DATA_SYNC`/`POST_NOTIFICATIONS` were - declared in earlier builds for a foreground-service notification that was never actually - implemented — the daemon just runs on a background thread inside the activity — and were - removed after Play's automated review flagged the unused `FOREGROUND_SERVICE_DATA_SYNC` - grant.) + - `FOREGROUND_SERVICE` and `FOREGROUND_SERVICE_SPECIAL_USE` — `EngineService`, a foreground + service of type `specialUse` that keeps the daemon, the engines and a loaded model running + while the app is in the background. It starts when the app is opened and runs until Quit in its + notification, or until the app is closed (Back from its main screen, or swiped away from + Recents) with nothing running: no model loaded or loading, nothing generating, importing or + downloading, no reply waiting for an approval, no routine running, and no LAN access, remote + access or Turbo Link for other devices on. That close ends the app's process about 3 seconds + later, as Quit does (Quit too waits those 3 seconds), whether notifications are on or off; + opening the app again within them, or after Quit bringing it back on screen, keeps it running. + Closed with something running, the service keeps running after that work ends or that access is + turned off, until Quit or a later close with nothing running (opening the app and closing it + again, or swiping away the card a Back left in Recents); with notifications off there is no + Quit, and on Android 13+ the system's list of active apps can stop it. Routines only scheduled + for later do not count as running: a close with nothing running stops the routine scheduler + with the app. `PLAY-FGS-DECLARATION.md` has the text for Play's declaration. + - `POST_NOTIFICATIONS` — that service's ongoing notification, which carries the Quit action and + the "phone is hot" warning. On Android 13+ the app asks for it each time it starts until it is + granted; Android stops showing the prompt once the user has declined it twice. The service + runs either way. + - `WAKE_LOCK` — a partial wake lock, held only while a reply is being generated, a model + being loaded, auto-tuned, imported or downloaded, or a skill written, so that work finishes + with the screen off; never while idle or while a reply waits for the user to approve a tool + call. + - `READ_EXTERNAL_STORAGE` and `WRITE_EXTERNAL_STORAGE`, Android 10 and older only + (`maxSdkVersion="29"`, with `requestLegacyExternalStorage`) — for a model folder the user adds + in shared storage, such as `/sdcard/AI`: read to scan and load its models, write because the + daemon downloads into the primary model folder. Asked for, after a dialog, only when such a + folder is set (`StorageAccess.kt`). On Android 11+ the app requests no storage permission, so + in shared storage it can read only files it created itself; models are imported with the + system file picker, which needs none. + - No location, camera, microphone or contacts permission, and no All files access + (`MANAGE_EXTERNAL_STORAGE`). `FOREGROUND_SERVICE_DATA_SYNC`, which an earlier build declared + for a foreground service it never implemented and Play's automated review flagged, is gone; + the foreground service now in the app is the `specialUse` one above. + `REQUEST_IGNORE_BATTERY_OPTIMIZATIONS`, which earlier builds declared but never used (no code + asked for the exemption), is gone too. ## Where data lives `HOME` is set to the app's private `filesDir` (`MainActivity.kt`), so the daemon's usual -`~/.turbollm`-equivalent state (chats, config, downloaded models/engines) lands entirely inside -Android's app-private storage — not shared storage, not readable by other apps. - -**One open item, not yet resolved in code — flag for the dev team before shipping:** -`AndroidManifest.xml` has `android:allowBackup="true"` with no `dataExtractionRules`/ -`fullBackupContent` exclusion file present. That means Android's own Auto Backup (to the user's -Google account, not to any TurboLLM server) could sweep up the app's private files — including -local chat history — into the user's device backup by default. This is a very different privacy -surface than "nothing ever leaves the device" (it's the user's own Google backup, not a third -party TurboLLM controls, but it's still data leaving the device to *somewhere*). Recommend either -(a) adding a backup-rules XML that excludes the daemon's data directory, or (b) disclosing it -plainly if the team decides the Google-account-only exposure is acceptable. Not fixed here — out -of scope for asset prep — but should not be silently left unaddressed. +`~/.turbollm`-equivalent state (chats, config, downloaded engines) lands inside Android's +app-private storage, which other apps cannot read. So do the models, unless the user adds a model +folder somewhere else: imported ones go to `files/models/`, and downloads go to the primary model +folder, which is one of the app's own unless the user picks another. A model folder the user adds +can be: +- in shared storage, such as `/sdcard/AI`: the app reads it on Android 10 and older, with the + storage permission, and downloads into it when it is the primary model folder. Other apps with + storage access, and a computer over USB, can read the models there. On Android 11+ the app + cannot read the models the user or another app put there, and offers to import them instead + (`StorageAccess.kt`). Android still lets it create files under `Download/` and `Documents/`, + though, so if the primary model folder is in one of those, the daemon's downloads can land in + shared storage there: the app reads them back, and a computer over USB or an app with All files + access can read them too. +- in the app's external files folder, `Android/data/com.turbollm/files/`: no permission needed, + and downloaded into when it is the primary model folder. A computer over USB can read it, and + on Android 10 and older so can other apps with the storage permission. + +Chats and settings stay in `filesDir` either way. + +**Backups — one open decision for the dev team before shipping:** +`AndroidManifest.xml` has `android:allowBackup="true"` with backup rules +(`res/xml/data_extraction_rules.xml` for Android 12+, `res/xml/backup_rules.xml` before). They only +exclude models, caches, logs and the extracted daemon under `files/`, and the app's external files +folder as a whole (below); everything else Auto Backup takes by default stays in. So Android's own +Auto Backup (to the user's Google account, not to any TurboLLM server; end-to-end encrypted with +the lock-screen credential on Android 9+ when the phone has one) and device-to-device transfer +carry the rest of the app's own files: + +- `files/`: the chat history and the rest of the daemon's state, including its settings file, + `files/.turbollm/config.json`. +- `app_webview/`: the WebView's own data. Among it are its cookie store (the only cookie the app + sets is the per-launch app secret, which no longer works after a restart) and the web UI's + localStorage: UI settings such as the theme and pinned models, the assistant and user names + and custom instructions set under personalization, each conversation's agent, reasoning effort + and thinking budget (by conversation id), the id of the chat last open, the draft last typed + into the text classification playground, and any API key pasted into the web UI's key prompt, + in plaintext (`tllm.authToken`). The rest is the WebView's other site storage and caches. +- `shared_prefs/`: the app's own preferences (whether the model-folder storage prompt was + answered), next to any preference files the WebView keeps there for itself (no user data: the + app grants pages no permissions). + +The app's external files folder, `Android/data/com.turbollm/files/` (app-specific external +storage, the "external" domain), is left out of both the backup and the transfer: the app writes +nothing there itself, but a model folder the user adds inside it holds multi-GB models (put there +by the user, or downloaded into it). In a cloud backup they would go over Android's 25 MB quota, +and Android would then skip the whole cloud backup, chats and `config.json` included. Like the +other excluded models, they can be copied or downloaded again. Android itself never backs up +`cache/`, `code_cache/` or `no_backup/`. + +When the cloud backup actually runs: Android's scheduled Auto Backup takes an app at most once a +day, and by default only while the phone is idle, charging and on Wi-Fi (any unmetered network, or +mobile data if the user allowed it). It also passes over an app that counts as in the foreground, +and an app running a foreground service does. `EngineService` is one, and it runs from the moment +the app is opened until Quit or until the app is closed with nothing running, so while it runs +Android puts TurboLLM's backup off and tries again 1 to 3 hours later. A phone that keeps the app +open day and night can go a long time without a new TurboLLM backup; one is taken on a night the +app is not running, after Quit or such a close, a reboot, or Android ending the process. +Excluding the models only keeps a backup that does run under the 25 MB quota. +`android:backupInForeground="true"` would remove the wait, but Android stops the app to back it +up, which would unload the model and cut off a reply, so it is not set. + +`config.json` also holds, in plaintext, every third-party credential the user has entered: the +Hugging Face and GitHub tokens, the Cloudflare tunnel token, the ngrok authtoken, the Tavily/Kagi +search keys, MCP server keys and env, and the raw Turbo Link peer tokens that grant access to the +user's other TurboLLM machines. (API keys the daemon issues are stored hashed there; one pasted +into the web UI's key prompt is in the localStorage above.) None of these are needed to restore +chats. Either disclose it plainly, or keep the credentials out of backups: moving them +to a separate file the rules exclude takes a daemon change and a config migration; excluding +`config.json` itself would also drop every setting on restore, and excluding `app_webview/` +would drop the web UI's own settings along with a pasted key. ## Suggested Play Console "Data safety" form answers @@ -94,9 +215,13 @@ of scope for asset prep — but should not be silently left unaddressed. usage data** (App activity → App interactions, Device or other IDs). Both are shared with Google as the processor (Play's form has a specific "shared with Firebase/Google for analytics" path — use it, don't mark this as "not shared"). Your own chat content, prompts, files, and - model weights are NOT collected or shared by either SDK or by TurboLLM. + model weights are NOT collected or shared by either SDK or by TurboLLM. TurboLLM itself + collects nothing, but an image in a model reply loads from the host it names (any https host, + or plain http on port 80) as soon as the reply shows, so text a reply puts in an image's URL + reaches that host (item 1 under "What this app actually does over the network"). - **Is data encrypted in transit?** Yes for Firebase's traffic (Google's SDKs use HTTPS/TLS). - N/A for the app's own loopback traffic; Hugging Face/GitHub downloads use HTTPS. + N/A for the app's own loopback traffic; Hugging Face/GitHub downloads use HTTPS. Images in a + reply or a model card can load over plain http, on port 80 (item 1). - **Can users request data deletion?** Yes for local app data — uninstalling removes it, and there is no TurboLLM-side account/record. For Firebase-held crash/analytics data, follow Firebase's own data deletion process (Google Analytics for Firebase supports user-level deletion requests) — diff --git a/scripts/stage-daemon-assets.mjs b/scripts/stage-daemon-assets.mjs index 6cc5d54..0b8f2d0 100644 --- a/scripts/stage-daemon-assets.mjs +++ b/scripts/stage-daemon-assets.mjs @@ -17,96 +17,312 @@ // bundled (turbollm/tsup.android.config.ts, ~6 MB) and staged here. // // Layout produced (mirrors what Node expects once extracted to filesDir/daemon): -// daemon/package.json {"type":"module"} — without it Node reads the -// ESM bundle as CommonJS and dies on `import` -// daemon/dist/… the bundle + android-preload.cjs + webdist/ +// daemon/package.json {"type":"module","version":…} — without the type +// Node reads the ESM bundle as CommonJS and dies on +// `import`; the version is what the daemon reports +// daemon/dist/… the bundle + android-preload.cjs + webdist/ (with +// build-inputs.json, the stamp of the web/ files it +// was built from) // daemon/node_modules/sql.js/… NOT bundled on purpose: sql.js resolves its own // .wasm relative to its package dir at runtime // // Usage: node scripts/stage-daemon-assets.mjs [path-to-turbollm-checkout] import { execSync } from 'node:child_process' -import { cpSync, existsSync, mkdirSync, rmSync, writeFileSync, readdirSync, statSync } from 'node:fs' +import { createHash } from 'node:crypto' +import { + cpSync, existsSync, lstatSync, mkdirSync, realpathSync, rmSync, writeFileSync, readFileSync, readdirSync, statSync, +} from 'node:fs' import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' -const appRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..') -const turbollm = resolve(process.argv[2] ?? join(appRoot, '..', 'turbollm')) -const assets = join(appRoot, 'composeApp', 'src', 'androidMain', 'assets', 'daemon') +// ── The web UI ──────────────────────────────────────────────────────────────────────────── +// +// The APK carries turbollm/src/webdist, the web UI's build output, which nothing in git holds. +// Copied as found, an APK built after a pull or a branch switch paired the daemon it had just +// bundled with whatever web UI was last built in that checkout, with no sign of it anywhere. +// So the web UI is built again here whenever the files it is built from are not the ones it was +// last built from here: a hash of them is kept in webdist/build-inputs.json (a name without a +// leading dot, which AGP would leave out of the APK's assets). A build with nothing changed under +// web/ costs that hash, under 50 ms; one with a change costs a web build, 40-55 s. + +/** The stamp in webdist: the web/ files the web UI there was built from. */ +export const WEB_STAMP = 'build-inputs.json' +const STAMP_VERSION = 1 -const bundle = join(turbollm, 'dist-android') -const webdist = join(turbollm, 'src', 'webdist') -const preload = join(turbollm, 'src', 'util', 'android-preload.cjs') -// Only the Node build of sql.js. The package also carries asm.js and browser variants plus -// debug builds — 24 MB in total, against ~1 MB for the two files actually loaded here. -const sqlJs = join(turbollm, 'node_modules', 'sql.js') +// What the web build reads is every file under web/ (Tailwind builds its CSS from the class names +// in all of them, tests included, and vite reads web/.env*), except its packages, which +// web/package-lock.json stands for; dot-directories (vite's and other caches); what the test runs +// write; and the files a desktop puts in folders it shows. +const NOT_INPUT_DIRS = new Set(['test-results', 'playwright-report', 'blob-report', 'coverage']) +const NOT_INPUT_FILES = new Set(['.DS_Store', 'Thumbs.db', 'desktop.ini']) -if (!existsSync(turbollm)) { - console.error(`FATAL: no turbollm checkout at ${turbollm}\nPass its path: node scripts/stage-daemon-assets.mjs `) - process.exit(1) +/** A SHA-256 over every web build input under [web]: its path relative to web/ with '/' between + * names, its length and its bytes, in the code-unit order of the paths, so Windows, macOS and + * Linux hash the same tree alike. A symlink to a file counts as that file; a dangling one, and a + * symlink to a directory, are left alone (the walk never follows one). On Windows, what a folder + * listing calls a link is any reparse point, which a OneDrive folder or file is too: lstat tells + * the links from those, and those count as the folder or file they are. A file or folder that goes + * away while the hash is taken (an editor's safe write moves the file it saves aside for a moment) + * is left out rather than failing the build: that only changes the hash, so the worst it can do is + * cost a web build. */ +export function webInputs(web) { + const files = [] + const walk = (dir, rel) => { + let entries + try { + entries = readdirSync(dir, { withFileTypes: true }) + } catch (e) { + if (rel && e.code === 'ENOENT') return + throw e + } + for (const e of entries) { + const path = join(dir, e.name) + const name = rel ? `${rel}/${e.name}` : e.name + // On Windows a listing types every reparse point as a link (libuv's scandir looks only at + // FILE_ATTRIBUTE_REPARSE_POINT), and so a OneDrive folder or file (a cloud files placeholder), + // a deduplicated file and the like say "link" too. Taken for links, every folder of sources in + // a OneDrive-synced checkout was skipped and a change in one went unseen. lstat calls only + // symlinks and junctions links and reports the rest as the folder or file they are. An entry + // lstat cannot read is handled as a link, as before. + let type = e + if (e.isSymbolicLink()) { + try { + type = lstatSync(path) + } catch {} + } + if (type.isSymbolicLink()) { + let target + try { + target = statSync(path) + } catch { + continue + } + if (target.isFile()) files.push([name, path]) + } else if (type.isDirectory()) { + if (e.name === 'node_modules' || e.name.startsWith('.') || (!rel && NOT_INPUT_DIRS.has(e.name))) continue + walk(path, name) + } else if (type.isFile() && !NOT_INPUT_FILES.has(e.name)) { + files.push([name, path]) + } + } + } + walk(web, '') + files.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + const hash = createHash('sha256') + let hashed = 0 + for (const [name, path] of files) { + let bytes + try { + bytes = readFileSync(path) + } catch (e) { + if (e.code === 'ENOENT') continue + throw e + } + hash.update(`${name}\0${bytes.length}\0`) + hash.update(bytes) + hashed++ + } + return { hash: hash.digest('hex'), files: hashed } } -// Always rebuild the Android bundle rather than reusing whatever is on disk. It takes ~200 ms, -// and the alternative is the failure mode this whole pipeline exists to prevent: an APK that -// looks fine and silently ships a daemon older than the source. The web UI is NOT rebuilt here — -// that one is slow, changes far less often, and has its own `npm run build` in web/. -console.log('building the Android daemon bundle…') -try { - execSync('npx tsup --config tsup.android.config.ts', { cwd: turbollm, stdio: 'inherit' }) -} catch { - console.error(`FATAL: the Android daemon bundle failed to build in ${turbollm}`) - process.exit(1) +/** Why the web UI in [webdist] is not the build of [inputs], or null when it is. */ +function webStale(webdist, inputs) { + if (!existsSync(join(webdist, 'index.html'))) return 'there is no web UI build' + let stamp + try { + stamp = JSON.parse(readFileSync(join(webdist, WEB_STAMP), 'utf8')) + } catch { + return 'the web UI build there has no record of what it was built from' + } + if (stamp?.version !== STAMP_VERSION || stamp.inputs !== inputs.hash) { + const then = stamp?.files ?? '?' + return `web/ has changed since the web UI was last built here (${then} files then, ${inputs.files} now)` + } + return null } -for (const [label, p] of [ - ['android bundle', bundle], - ["web UI (run: npm run build in turbollm/web)", webdist], - ['android preload', preload], - ['sql.js (run: npm install in turbollm)', sqlJs], -]) { - if (!existsSync(p)) { - console.error(`FATAL: missing ${label}: ${p}`) - process.exit(1) +/** Why web/node_modules is not what web/package-lock.json asks for, or null when it is. npm + * records what it installed in node_modules/.package-lock.json; every version there must be the + * lockfile's, and every package the lockfile has must be there, but for an optional one (another + * platform's esbuild or rollup binary). A pulled lockfile change built with the old packages would + * ship a UI built from other packages under a stamp that names the new ones. */ +export function webDepsProblem(web) { + const nodeModules = join(web, 'node_modules') + if (!existsSync(nodeModules)) return 'web/node_modules is missing' + let locked, installed + try { + locked = JSON.parse(readFileSync(join(web, 'package-lock.json'), 'utf8')).packages ?? {} + } catch { + return 'web/package-lock.json cannot be read' + } + try { + installed = JSON.parse(readFileSync(join(nodeModules, '.package-lock.json'), 'utf8')).packages ?? {} + } catch { + return 'web/node_modules has no record of what npm installed there (node_modules/.package-lock.json)' } + for (const [key, want] of Object.entries(locked)) { + if (!key.startsWith('node_modules/')) continue // web/ itself, or a linked folder + const name = key.slice(key.lastIndexOf('node_modules/') + 'node_modules/'.length) + const have = installed[key] + if (!have) { + if (want.optional || want.devOptional) continue + return `${name} ${want.version} is in web/package-lock.json but not in web/node_modules` + } + if (have.version !== want.version) { + return `web/node_modules has ${name} ${have.version}, web/package-lock.json ${want.version}` + } + } + return null } -rmSync(assets, { recursive: true, force: true }) -mkdirSync(join(assets, 'dist'), { recursive: true }) - -cpSync(bundle, join(assets, 'dist'), { recursive: true }) -cpSync(webdist, join(assets, 'dist', 'webdist'), { recursive: true }) -cpSync(preload, join(assets, 'dist', 'android-preload.cjs')) - -// sql.js's JS ends up bundled (the Android pass bundles everything), but its .wasm cannot be — -// it is a binary the module loads at runtime via `__dirname + "/sql-wasm.wasm"`. The banner in -// tsup.android.config.ts points `__dirname` at this dist/ directory, so the wasm goes here -// rather than into a node_modules tree. Only the wasm is needed; the package's other 23 MB is -// asm.js, browser and debug variants. -cpSync(join(sqlJs, 'dist', 'sql-wasm.wasm'), join(assets, 'dist', 'sql-wasm.wasm')) - -// `ws` is reached through a real runtime `require('ws')` (registerTerminalWs), not a static -// import, so esbuild cannot bundle it and it has to exist as a resolvable package — Node walks -// up from dist/ and finds daemon/node_modules/. It is the ONLY such case that matters here: -// the source has exactly three bare requires — 'net' (a builtin), 'node-pty' (native, and only -// reached by the Code feature, which is cut from Android) and this one. `ws` itself has no -// runtime dependencies, so copying the package alone is sufficient. -const wsSrc = join(turbollm, 'node_modules', 'ws') -if (!existsSync(wsSrc)) { - console.error(`FATAL: missing ws: ${wsSrc}`) - process.exit(1) +// `npm run build` in web/: what TurboLLM's build:web runs once web/'s packages are in (`cd web && +// npm ci && npm run build`), without the npm ci, which needs the network and whose job +// webDepsProblem has just checked. Through the shell, like `npx tsup` below: that is what finds +// npm.cmd on Windows, and the command holds no path that would need quoting. +function npmRunBuild(web) { + execSync('npm run build', { cwd: web, stdio: 'inherit' }) } -cpSync(wsSrc, join(assets, 'node_modules', 'ws'), { recursive: true }) -// The daemon bundle is ESM. Node decides that by walking up for the nearest package.json, -// and once extracted to filesDir there is no other one above it. -writeFileSync(join(assets, 'package.json'), '{"type":"module"}\n') +/** Builds the web UI in [turbollm] (into src/webdist, where web/'s vite config puts it) unless it + * is the build of web/ as it is now, then stamps it. Answers whether it built. Throws, with what + * to do, when web/node_modules does not match its lockfile or the build fails. */ +export function ensureWebUi(turbollm, { build = npmRunBuild, log = console.log } = {}) { + const web = join(turbollm, 'web') + const webdist = join(turbollm, 'src', 'webdist') + if (!existsSync(join(web, 'package.json'))) throw new Error(`no web UI sources at ${web}`) + const inputs = webInputs(web) + const stale = webStale(webdist, inputs) + if (!stale) { + log(`web UI up to date (built from these ${inputs.files} files under web/)`) + return false + } + const deps = webDepsProblem(web) + if (deps) { + throw new Error( + `the web UI has to be built, as ${stale}, but ${deps}.\n` + + `Install web/'s packages, then build the app again:\n npm --prefix "${web}" ci`, + ) + } + log(`building the web UI: ${stale}…`) + // Gone before the build starts, so an interrupted or failed build leaves no stamp vouching for it. + rmSync(join(webdist, WEB_STAMP), { force: true }) + const index = join(webdist, 'index.html') + const before = statSync(index, { throwIfNoEntry: false }) + try { + build(web) + } catch { + throw new Error(`the web UI failed to build in ${web} (its output is above)`) + } + // The build has to have written index.html here: one whose output went elsewhere (web/'s vite + // config moved its outDir, or its build script changed) would leave the old web UI in place, + // and the stamp would then vouch for it. A new index.html shows by its mtime: a real web build + // takes far longer than any file system's timestamp step (the inode can be the old file's). + const after = statSync(index, { throwIfNoEntry: false }) + if (!after || (before && after.mtimeMs === before.mtimeMs && after.ino === before.ino)) { + throw new Error(`the web UI build did not write ${index}`) + } + // The hash from before the build: a file changed while it ran makes the next build build again. + const stamp = { version: STAMP_VERSION, inputs: inputs.hash, files: inputs.files } + writeFileSync(join(webdist, WEB_STAMP), `${JSON.stringify(stamp)}\n`) + return true +} + +function main() { + const appRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..') + const turbollm = resolve(process.argv[2] ?? join(appRoot, '..', 'turbollm')) + const assets = join(appRoot, 'composeApp', 'src', 'androidMain', 'assets', 'daemon') -function du(dir) { - let total = 0 - for (const e of readdirSync(dir, { withFileTypes: true })) { - const p = join(dir, e.name) - total += e.isDirectory() ? du(p) : statSync(p).size + const bundle = join(turbollm, 'dist-android') + const webdist = join(turbollm, 'src', 'webdist') + const preload = join(turbollm, 'src', 'util', 'android-preload.cjs') + // Only the Node build of sql.js. The package also carries asm.js and browser variants plus + // debug builds — 24 MB in total, against ~1 MB for the two files actually loaded here. + const sqlJs = join(turbollm, 'node_modules', 'sql.js') + + if (!existsSync(turbollm)) { + console.error(`FATAL: no turbollm checkout at ${turbollm}\nPass its path: node scripts/stage-daemon-assets.mjs `) + process.exit(1) } - return total + + // Always rebuild the Android bundle rather than reusing whatever is on disk. It takes ~200 ms, + // and the alternative is the failure mode this whole pipeline exists to prevent: an APK that + // looks fine and silently ships a daemon older than the source. The web UI is rebuilt below + // only when web/ has changed since its last build (ensureWebUi): it takes 40-55 s. + console.log('building the Android daemon bundle…') + try { + execSync('npx tsup --config tsup.android.config.ts', { cwd: turbollm, stdio: 'inherit' }) + } catch { + console.error(`FATAL: the Android daemon bundle failed to build in ${turbollm}`) + process.exit(1) + } + + for (const [label, p] of [ + ['android bundle', bundle], + ['android preload', preload], + ['sql.js (run: npm install in turbollm)', sqlJs], + ]) { + if (!existsSync(p)) { + console.error(`FATAL: missing ${label}: ${p}`) + process.exit(1) + } + } + + try { + ensureWebUi(turbollm) + } catch (e) { + console.error(`FATAL: ${e.message}`) + process.exit(1) + } + + rmSync(assets, { recursive: true, force: true }) + mkdirSync(join(assets, 'dist'), { recursive: true }) + + cpSync(bundle, join(assets, 'dist'), { recursive: true }) + cpSync(webdist, join(assets, 'dist', 'webdist'), { recursive: true }) + cpSync(preload, join(assets, 'dist', 'android-preload.cjs')) + + // sql.js's JS ends up bundled (the Android pass bundles everything), but its .wasm cannot be — + // it is a binary the module loads at runtime via `__dirname + "/sql-wasm.wasm"`. The banner in + // tsup.android.config.ts points `__dirname` at this dist/ directory, so the wasm goes here + // rather than into a node_modules tree. Only the wasm is needed; the package's other 23 MB is + // asm.js, browser and debug variants. + cpSync(join(sqlJs, 'dist', 'sql-wasm.wasm'), join(assets, 'dist', 'sql-wasm.wasm')) + + // `ws` is reached through a real runtime `require('ws')` (registerTerminalWs), not a static + // import, so esbuild cannot bundle it and it has to exist as a resolvable package — Node walks + // up from dist/ and finds daemon/node_modules/. It is the ONLY such case that matters here: + // the source has exactly three bare requires — 'net' (a builtin), 'node-pty' (native, and only + // reached by the Code feature, which is cut from Android) and this one. `ws` itself has no + // runtime dependencies, so copying the package alone is sufficient. + const wsSrc = join(turbollm, 'node_modules', 'ws') + if (!existsSync(wsSrc)) { + console.error(`FATAL: missing ws: ${wsSrc}`) + process.exit(1) + } + cpSync(wsSrc, join(assets, 'node_modules', 'ws'), { recursive: true }) + + // The daemon bundle is ESM. Node decides that by walking up for the nearest package.json, + // and once extracted to filesDir there is no other one above it. It is also where the daemon + // reads its own version (cli.ts: `/../package.json`) — without one it fell back + // to '0.1.1' in the UI, telemetry, Turbo Link hello and the update check on every Android install. + const { version } = JSON.parse(readFileSync(join(turbollm, 'package.json'), 'utf8')) + if (typeof version !== 'string' || !version) { + console.error(`FATAL: no version in ${join(turbollm, 'package.json')}`) + process.exit(1) + } + writeFileSync(join(assets, 'package.json'), `${JSON.stringify({ type: 'module', version })}\n`) + + function du(dir) { + let total = 0 + for (const e of readdirSync(dir, { withFileTypes: true })) { + const p = join(dir, e.name) + total += e.isDirectory() ? du(p) : statSync(p).size + } + return total + } + console.log(`staged daemon assets → ${assets}`) + console.log(`payload: ${(du(assets) / 1e6).toFixed(1)} MB`) } -console.log(`staged daemon assets → ${assets}`) -console.log(`payload: ${(du(assets) / 1e6).toFixed(1)} MB`) + +// Run as a script; the test imports the web UI check without staging anything. +if (process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))) main() diff --git a/scripts/stage-daemon-assets.test.mjs b/scripts/stage-daemon-assets.test.mjs new file mode 100644 index 0000000..28d6b8c --- /dev/null +++ b/scripts/stage-daemon-assets.test.mjs @@ -0,0 +1,396 @@ +// Tests for the web UI step of scripts/stage-daemon-assets.mjs, which builds turbollm's web UI again whenever web/ +// is not what it was last built from: +// node --test scripts/stage-daemon-assets.test.mjs +// Everything runs on a fake turbollm checkout in a temp folder; nothing builds the real web UI. The tests of the whole +// script run it as Gradle's stageDaemonAssets does, from a copy in a temp app checkout (the script stages into the app +// checkout it sits in), against a checkout with a stand-in tsup and a web/ whose `npm run build` is a small node +// script. Those need npm and a POSIX shell for the stand-in tsup, so they are skipped on Windows. +import assert from 'node:assert/strict' +import { spawnSync } from 'node:child_process' +import fs, { + chmodSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, utimesSync, writeFileSync, +} from 'node:fs' +import { createHash } from 'node:crypto' +import { syncBuiltinESMExports } from 'node:module' +import { tmpdir } from 'node:os' +import { dirname, join } from 'node:path' +import { after, test } from 'node:test' +import { fileURLToPath } from 'node:url' + +import { WEB_STAMP, ensureWebUi, webDepsProblem, webInputs } from './stage-daemon-assets.mjs' + +const script = fileURLToPath(new URL('./stage-daemon-assets.mjs', import.meta.url)) +const work = mkdtempSync(join(tmpdir(), 'stage-daemon-assets-test-')) +after(() => rmSync(work, { recursive: true, force: true })) + +function write(path, text) { + mkdirSync(dirname(path), { recursive: true }) + writeFileSync(path, text) +} +const json = (path) => JSON.parse(readFileSync(path, 'utf8')) + +// web/'s build does what vite does with emptyOutDir: it replaces src/webdist with its output, here an index.html +// holding web/src/app.txt, and it notes each run in builds.log. Each build dates its index.html after the last +// build's and after anything the test wrote. ensureWebUi tells a new index.html by its mtime, as a real build takes +// far longer than a file system's timestamp step; these take under a millisecond, and the new index.html can get the +// old one's inode, so where timestamps step by a clock tick (Linux before 6.13) or a second, nothing else tells them +// apart. +const FAKE_BUILD = `import { appendFileSync, mkdirSync, readFileSync, rmSync, utimesSync, writeFileSync } from 'node:fs' +rmSync('../src/webdist', { recursive: true, force: true }) +mkdirSync('../src/webdist', { recursive: true }) +writeFileSync('../src/webdist/index.html', readFileSync('src/app.txt')) +appendFileSync('../builds.log', 'built\\n') +const t = Date.now() / 1000 + 10 * readFileSync('../builds.log').length +utimesSync('../src/webdist/index.html', t, t) +` +// The same build in this process, for ensureWebUi's injected build. +let clock = 0 +function build(web) { + const index = join(web, '..', 'src', 'webdist', 'index.html') + rmSync(join(web, '..', 'src', 'webdist'), { recursive: true, force: true }) + write(index, readFileSync(join(web, 'src', 'app.txt'))) + const t = Date.now() / 1000 + 10 * ++clock + utimesSync(index, t, t) + builds.push(web) +} +let builds = [] +const quiet = { build, log: () => {} } + +let made = 0 +// A turbollm checkout with what the staging script reads. Its web/package-lock.json has a package for another +// platform (optional) that web/node_modules does not, as on any machine. +function fakeTurbollm() { + const tl = join(work, `turbollm-${++made}`) + const web = join(tl, 'web') + write(join(tl, 'package.json'), '{"name":"turbollm","version":"1.2.3"}\n') + write(join(tl, 'src', 'util', 'android-preload.cjs'), '\n') + write(join(tl, 'node_modules', 'sql.js', 'dist', 'sql-wasm.wasm'), 'wasm') + write(join(tl, 'node_modules', 'ws', 'package.json'), '{"name":"ws","version":"8.0.0"}\n') + write(join(web, 'package.json'), `${JSON.stringify({ name: 'web', private: true, scripts: { build: 'node fake-build.mjs' } })}\n`) + write(join(web, 'package-lock.json'), `${JSON.stringify({ name: 'web', lockfileVersion: 3, requires: true, packages: { + '': { name: 'web' }, + 'node_modules/vite': { version: '7.3.7', dev: true }, + 'node_modules/@esbuild/win32-x64': { version: '0.25.0', dev: true, optional: true, os: ['win32'], cpu: ['x64'] }, + } })}\n`) + write(join(web, 'node_modules', '.package-lock.json'), `${JSON.stringify({ name: 'web', lockfileVersion: 3, requires: true, packages: { + 'node_modules/vite': { version: '7.3.7', dev: true }, + } })}\n`) + write(join(web, 'node_modules', 'vite', 'package.json'), '{"name":"vite","version":"7.3.7"}\n') + write(join(web, 'fake-build.mjs'), FAKE_BUILD) + write(join(web, 'index.html'), '
\n') + write(join(web, 'src', 'app.txt'), 'version 1\n') + return tl +} +const stampOf = (tl) => json(join(tl, 'src', 'webdist', WEB_STAMP)) + +test('a web UI built from web/ as it is now is not built again', () => { + builds = [] + const tl = fakeTurbollm() + assert.equal(ensureWebUi(tl, quiet), true, 'no web UI yet: built') + assert.equal(ensureWebUi(tl, quiet), false) + assert.equal(ensureWebUi(tl, quiet), false) + assert.equal(builds.length, 1) +}) + +test('a changed file under web/ builds the web UI again, then stamps it', () => { + builds = [] + const tl = fakeTurbollm() + const web = join(tl, 'web') + ensureWebUi(tl, quiet) + write(join(web, 'src', 'app.txt'), 'version 2\n') + assert.equal(ensureWebUi(tl, quiet), true) + assert.equal(readFileSync(join(tl, 'src', 'webdist', 'index.html'), 'utf8'), 'version 2\n') + assert.deepEqual(stampOf(tl), { version: 1, inputs: webInputs(web).hash, files: webInputs(web).files }) + // A file added, and a lockfile change (web/node_modules updated with it), each count as a change too. + write(join(web, 'src', 'new.tsx'), 'export {}\n') + assert.equal(ensureWebUi(tl, quiet), true) + const lock = json(join(web, 'package-lock.json')) + const installed = json(join(web, 'node_modules', '.package-lock.json')) + lock.packages['node_modules/vite'].version = installed.packages['node_modules/vite'].version = '7.3.8' + write(join(web, 'package-lock.json'), JSON.stringify(lock)) + write(join(web, 'node_modules', '.package-lock.json'), JSON.stringify(installed)) + assert.equal(ensureWebUi(tl, quiet), true) + assert.equal(ensureWebUi(tl, quiet), false) + assert.equal(builds.length, 4) +}) + +test('a web UI with no stamp, or a stamp of another kind, is built again', () => { + builds = [] + const tl = fakeTurbollm() + // Built by hand or by an older staging script: an index.html and no record of what it came from. + write(join(tl, 'src', 'webdist', 'index.html'), 'version 0\n') + assert.equal(ensureWebUi(tl, quiet), true) + assert.equal(readFileSync(join(tl, 'src', 'webdist', 'index.html'), 'utf8'), 'version 1\n') + write(join(tl, 'src', 'webdist', WEB_STAMP), JSON.stringify({ ...stampOf(tl), version: 0 })) + assert.equal(ensureWebUi(tl, quiet), true) + assert.equal(builds.length, 2) +}) + +test('a web build that fails, or that a change overtakes, is built again next time', () => { + const tl = fakeTurbollm() + const app = join(tl, 'web', 'src', 'app.txt') + const index = join(tl, 'src', 'webdist', 'index.html') + ensureWebUi(tl, quiet) + // A build that dies halfway, after writing some of its output: no stamp is left to vouch for what is there, so + // even with web/ back as the last stamp had it, the half-built UI is not taken for that build. + write(app, 'version 2\n') + const failing = { log() {}, build() { write(index, 'BROKEN\n'); throw new Error('vite died') } } + assert.throws(() => ensureWebUi(tl, failing), /^Error: the web UI failed to build in /) + assert.equal(existsSync(join(tl, 'src', 'webdist', WEB_STAMP)), false, 'no stamp after a failed build') + write(app, 'version 1\n') + assert.equal(ensureWebUi(tl, quiet), true) + assert.equal(readFileSync(index, 'utf8'), 'version 1\n') + // A file saved while the build runs: the UI was built from the file as it was before, which is what the stamp + // says, so the next staging builds again. + write(app, 'version 2\n') + const overtaken = { log() {}, build(web) { build(web); write(app, 'version 3\n') } } + assert.equal(ensureWebUi(tl, overtaken), true) + assert.equal(readFileSync(index, 'utf8'), 'version 2\n') + assert.equal(ensureWebUi(tl, quiet), true, 'built again: web/ changed while the last build ran') + assert.equal(readFileSync(index, 'utf8'), 'version 3\n') + assert.equal(ensureWebUi(tl, quiet), false) +}) + +test('a web build that writes no index.html into webdist fails, and stamps nothing', () => { + const tl = fakeTurbollm() + const stamp = join(tl, 'src', 'webdist', WEB_STAMP) + // What a web/ whose build puts its output somewhere else would do: exit 0 and leave webdist as it was. + const elsewhere = { log() {}, build() {} } + assert.throws(() => ensureWebUi(tl, elsewhere), /^Error: the web UI build did not write .*index\.html$/) + assert.equal(existsSync(stamp), false) + ensureWebUi(tl, quiet) + write(join(tl, 'web', 'src', 'app.txt'), 'version 2\n') + assert.throws(() => ensureWebUi(tl, elsewhere), /^Error: the web UI build did not write .*index\.html$/, + 'an older build there is not the new one') + assert.equal(existsSync(stamp), false) + assert.equal(readFileSync(join(tl, 'src', 'webdist', 'index.html'), 'utf8'), 'version 1\n') + assert.equal(ensureWebUi(tl, quiet), true) +}) + +test('missing or mismatched web/node_modules stops with the command to run, and builds nothing', () => { + builds = [] + const tl = fakeTurbollm() + const web = join(tl, 'web') + const command = `npm --prefix "${web}" ci` + const installedPath = join(web, 'node_modules', '.package-lock.json') + const installed = readFileSync(installedPath, 'utf8') + assert.equal(webDepsProblem(web), null, "another platform's optional package is not needed") + + write(installedPath, installed.replace('"7.3.7"', '"7.3.6"')) + assert.throws(() => ensureWebUi(tl, quiet), (e) => { + assert.match(e.message, /web\/node_modules has vite 7\.3\.6, web\/package-lock\.json 7\.3\.7/) + assert.ok(e.message.endsWith(command), e.message) + return true + }) + write(installedPath, JSON.stringify({ packages: {} })) + assert.throws(() => ensureWebUi(tl, quiet), /vite 7\.3\.7 is in web\/package-lock\.json but not in web\/node_modules/) + rmSync(join(web, 'node_modules'), { recursive: true }) + assert.throws(() => ensureWebUi(tl, quiet), (e) => { + assert.match(e.message, /^the web UI has to be built, as there is no web UI build, but web\/node_modules is missing\./) + assert.ok(e.message.endsWith(command), e.message) + return true + }) + assert.equal(builds.length, 0) + assert.equal(existsSync(join(tl, 'src', 'webdist')), false) + + // With nothing to build, web/'s packages are not needed. + write(installedPath, installed) + ensureWebUi(tl, quiet) + rmSync(join(web, 'node_modules'), { recursive: true }) + assert.equal(ensureWebUi(tl, quiet), false) +}) + +test('a dangling symlink, a symlink to a folder, packages, caches and test output are not web build inputs', (t) => { + const tl = fakeTurbollm() + const web = join(tl, 'web') + const before = webInputs(web) + write(join(web, '.vite', 'deps', 'chunk.js'), 'cache') + write(join(web, 'src', '.cache', 'x'), 'cache') + write(join(web, 'node_modules', 'vite', 'index.js'), 'package') + write(join(web, 'src', 'x', 'node_modules', 'y.js'), 'package') + write(join(web, 'test-results', 'run.json'), '{}') + write(join(web, 'playwright-report', 'index.html'), 'report') + write(join(web, 'src', '.DS_Store'), 'finder') + assert.deepEqual(webInputs(web), before) + // A dotfile is an input, though: vite reads web/.env*. + write(join(web, '.env.local'), 'VITE_X=1\n') + assert.notEqual(webInputs(web).hash, before.hash) + rmSync(join(web, '.env.local')) + + const outside = join(work, `outside-${made}`) + write(join(outside, 'a.tsx'), 'export {}\n') + try { + symlinkSync(join(web, 'src', 'no-such-file.tsx'), join(web, 'src', 'gone.tsx')) + symlinkSync(outside, join(web, 'src', 'linked'), 'dir') + } catch (e) { + if (process.platform === 'win32' && e.code === 'EPERM') return t.skip('symlinks need Developer Mode on Windows') + throw e + } + assert.deepEqual(webInputs(web), before) + assert.equal(ensureWebUi(tl, quiet), true, 'the walk gets through them') + // A symlink to a file counts as that file. + symlinkSync(join(outside, 'a.tsx'), join(web, 'src', 'a.tsx')) + assert.notEqual(webInputs(web).hash, before.hash) +}) + +test('a file or folder that goes away while web/ is hashed is left out, not an error', () => { + const tl = fakeTurbollm() + const web = join(tl, 'web') + const saved = join(web, 'src', 'App.tsx') + const tmp = join(web, 'src', 'tmp') + write(saved, 'export {}\n') + write(join(tmp, 'x.ts'), 'export {}\n') + // Each goes away after the walk has listed it and before it is read, as an editor's safe write moves a file aside + // for a moment (fs's own functions, which the script's imports of node:fs follow, are wrapped for this one call). + const { readFileSync: read, readdirSync: list } = fs + fs.readFileSync = (path, ...rest) => { + if (path === saved) rmSync(saved) + return read(path, ...rest) + } + fs.readdirSync = (path, ...rest) => { + if (path === tmp) rmSync(tmp, { recursive: true }) + return list(path, ...rest) + } + syncBuiltinESMExports() + let during + try { + during = webInputs(web) + } finally { + Object.assign(fs, { readFileSync: read, readdirSync: list }) + syncBuiltinESMExports() + } + assert.equal(existsSync(saved) || existsSync(tmp), false, 'both went away during the hash') + assert.deepEqual(during, webInputs(web), 'the hash of web/ without them') +}) + +test('folders and files that Windows lists as links but are not (OneDrive placeholders) are web build inputs', () => { + const tl = fakeTurbollm() + const web = join(tl, 'web') + const menu = join(web, 'src', 'components', 'Menu.tsx') + write(menu, 'export {}\n') + write(join(web, 'src', '.DS_Store'), 'finder') + write(join(web, '.vite', 'deps', 'chunk.js'), 'cache') + const outside = join(work, `outside-reparse-${made}`) + write(join(outside, 'a.tsx'), 'export {}\n') + try { + symlinkSync(outside, join(web, 'src', 'linked'), 'dir') // a real link, which is still not followed + } catch (e) { + if (process.platform !== 'win32' || e.code !== 'EPERM') throw e + } + const listed = webInputs(web) + // On Windows, libuv's scandir types every entry with FILE_ATTRIBUTE_REPARSE_POINT as a link, and Node's Dirent says + // what libuv says: in a folder OneDrive keeps, whose folders and files are placeholders (reparse points), every + // entry is listed as a link. That is done here to every folder and file under web/ (fs's readdirSync, which the + // script's import of node:fs follows, is wrapped for this). + const { readdirSync: list } = fs + fs.readdirSync = (path, options) => { + const entries = list(path, options) + if (!options?.withFileTypes) return entries + const { UV_DIRENT_LINK } = fs.constants + return entries.map((e) => (e.isDirectory() || e.isFile() ? new fs.Dirent(e.name, UV_DIRENT_LINK, path) : e)) + } + syncBuiltinESMExports() + let asOnWindows, edited + try { + const src = fs.readdirSync(web, { withFileTypes: true }).find((e) => e.name === 'src') + assert.ok(src.isSymbolicLink() && !src.isDirectory(), 'src/ is listed as a link') + asOnWindows = webInputs(web) + write(menu, 'export const menu = 1\n') + edited = webInputs(web) + } finally { + fs.readdirSync = list + syncBuiltinESMExports() + } + assert.deepEqual(asOnWindows, listed, 'every folder walked and the same files hashed as when listed as what they are') + assert.notEqual(edited.hash, asOnWindows.hash, 'a change in a subfolder is seen') +}) + +test('the inputs hash is the same on every run, whatever order the files were made in', () => { + // 'src/C.tsx' is before 'src/a.tsx' in code-unit order (C is U+0043, a U+0061) and after it in a locale's. + const files = { 'src/b.tsx': 'b', 'src/a/z.ts': 'z', 'index.html': '
', 'src/a.tsx': 'a', 'package.json': '{}', 'src/C.tsx': 'C' } + const one = join(work, `order-1-${made}`) + const two = join(work, `order-2-${made}`) + for (const [p, text] of Object.entries(files)) write(join(one, ...p.split('/')), text) + for (const [p, text] of Object.entries(files).reverse()) write(join(two, ...p.split('/')), text) + const first = webInputs(one) + assert.deepEqual(webInputs(one), first) + assert.deepEqual(webInputs(two), first) + // Its definition, which every platform computes alike: each file's '/'-joined path, length and bytes, sorted by path. + const want = createHash('sha256') + for (const p of Object.keys(files).sort()) want.update(`${p}\0${files[p].length}\0`).update(files[p]) + assert.deepEqual(first, { hash: want.digest('hex'), files: 6 }) + // A file moved to another folder is a change even with the same bytes. + write(join(two, 'src', 'a', 'b.tsx'), 'b') + rmSync(join(two, 'src', 'b.tsx')) + assert.notEqual(webInputs(two).hash, first.hash) +}) + +// ── The whole script ────────────────────────────────────────────────────────────────────── + +const npm = spawnSync('npm', ['--version'], { stdio: 'ignore', shell: process.platform === 'win32' }) +const scriptSkip = process.platform === 'win32' ? 'the stand-in tsup is a POSIX executable' + : npm.error || npm.status !== 0 ? 'no npm on this machine' : false + +// The checkout of fakeTurbollm, plus a tsup that writes dist-android/cli.js, as npx finds it. +function fakeCheckout() { + const tl = fakeTurbollm() + write(join(tl, 'node_modules', 'tsup', 'package.json'), '{"name":"tsup","version":"8.0.0","bin":{"tsup":"cli.mjs"}}\n') + write(join(tl, 'node_modules', 'tsup', 'cli.mjs'), `#!/usr/bin/env node +import { mkdirSync, writeFileSync } from 'node:fs' +mkdirSync('dist-android', { recursive: true }) +writeFileSync('dist-android/cli.js', 'export {}\\n') +`) + chmodSync(join(tl, 'node_modules', 'tsup', 'cli.mjs'), 0o755) + mkdirSync(join(tl, 'node_modules', '.bin'), { recursive: true }) + symlinkSync('../tsup/cli.mjs', join(tl, 'node_modules', '.bin', 'tsup')) + return tl +} +function stage(tl) { + const app = join(work, `app-${made}`) + mkdirSync(join(app, 'scripts'), { recursive: true }) + cpSync(script, join(app, 'scripts', 'stage-daemon-assets.mjs')) + // Bounded, so a stuck npx or npm fails this test by name instead of holding the run. The timeout signals only the + // staging script, so it leads a group of its own and a timed-out run takes its npm children with it. + const r = spawnSync(process.execPath, [join(app, 'scripts', 'stage-daemon-assets.mjs'), tl], { + encoding: 'utf8', env: { ...process.env, npm_config_update_notifier: 'false' }, timeout: 120_000, detached: true, + }) + if (r.error?.code === 'ETIMEDOUT') { + try { process.kill(-r.pid, 'SIGKILL') } catch { /* the group is gone */ } + } + assert.equal(r.error, undefined, `the staging script did not finish: ${r.error?.message}`) + const staged = join(app, 'composeApp', 'src', 'androidMain', 'assets', 'daemon', 'dist', 'webdist') + return { ...r, staged } +} +const builtTimes = (tl) => (existsSync(join(tl, 'builds.log')) ? readFileSync(join(tl, 'builds.log'), 'utf8').split('\n').length - 1 : 0) + +test('the staging script builds a web UI older than web/ before it stages it, and only then', { skip: scriptSkip }, () => { + const tl = fakeCheckout() + // Built from older sources, the way a pull or a branch switch leaves it. + write(join(tl, 'src', 'webdist', 'index.html'), 'version 0\n') + let r = stage(tl) + assert.equal(r.status, 0, r.stderr) + assert.equal(readFileSync(join(r.staged, 'index.html'), 'utf8'), 'version 1\n', 'the APK carries the web UI of web/ as it is') + assert.match(r.stdout, /building the web UI: the web UI build there has no record of what it was built from/) + assert.deepEqual(json(join(r.staged, WEB_STAMP)), stampOf(tl)) + r = stage(tl) + assert.equal(r.status, 0, r.stderr) + assert.match(r.stdout, /web UI up to date/) + assert.equal(builtTimes(tl), 1) + write(join(tl, 'web', 'src', 'app.txt'), 'version 2\n') + r = stage(tl) + assert.equal(r.status, 0, r.stderr) + assert.equal(readFileSync(join(r.staged, 'index.html'), 'utf8'), 'version 2\n') + assert.equal(builtTimes(tl), 2) +}) + +test('the staging script stops with the command to run when web/node_modules is missing', { skip: scriptSkip }, () => { + const tl = fakeCheckout() + write(join(tl, 'src', 'webdist', 'index.html'), 'version 0\n') + rmSync(join(tl, 'web', 'node_modules'), { recursive: true }) + const r = stage(tl) + assert.equal(r.status, 1) + assert.match(r.stderr, /FATAL: the web UI has to be built, .* but web\/node_modules is missing\./) + assert.ok(r.stderr.includes(`npm --prefix "${join(tl, 'web')}" ci`), r.stderr) + assert.equal(existsSync(r.staged), false, 'nothing staged') + assert.equal(builtTimes(tl), 0) +}) diff --git a/scripts/stage-litert-lm-engine.mjs b/scripts/stage-litert-lm-engine.mjs index 8efa079..f12a2e0 100644 --- a/scripts/stage-litert-lm-engine.mjs +++ b/scripts/stage-litert-lm-engine.mjs @@ -5,44 +5,65 @@ // Why it is built like this: on desktop and under Termux, TurboLLM runs LiteRT-LM as // `python -m litert_lm_cli.main serve`, and that Python server reaches the runtime through ctypes // into liblitert-lm.so — the one native library in Google's `litert-lm-api` wheel. The app has no -// Python and may only execute files installed with its APK, so it ships that same library unchanged -// and replaces the Python layer with a native one. Nothing here is recompiled from Google's sources: -// the library is the exact file Termux users run, taken from the pinned PyPI wheel and checked -// against the hash below. +// Python and may only execute files installed with its APK, so it ships that same library and +// replaces the Python layer with a native one. Nothing here is recompiled from Google's sources: +// the library is the file Termux users run, taken from the pinned PyPI wheel and checked against +// the hash below, with one change. Its dependency on libandroid.so, which the engine process +// cannot load, is renamed to a shim generated from the library itself (see "The engine process +// cannot load libandroid.so" below), and the patched file is pinned as well. // -// Output: //{liblitert-lm.so, liblitertlm_server.so}, which build.gradle.kts adds to the -// APK's jniLibs. Both names fit the lib*.so pattern Android requires before it will install a file -// into nativeLibraryDir — the only place an app may execute from (BUILDING-ENGINES.md). +// Output: //{liblitert-lm.so, liblitertlm_server.so, } (the shim is liblrtandr.so +// today), which build.gradle.kts adds to the APK's jniLibs. Every name fits the lib*.so pattern +// Android requires before it will install a file into nativeLibraryDir — the only place an app may +// execute from (BUILDING-ENGINES.md). In / the script writes those files and removes +// only shims an earlier run staged that this one does not (regular files named liblr*.so that +// shimFileName could have produced); it leaves everything else there alone, so may also hold +// other libraries, such as the llama.cpp engines in composeApp/src/androidMain/jniLibs. It does +// overwrite a file of its own names. The build directories (cmake-, shim-) and, unless +// --cache says otherwise, the wheel cache (cache) go in /.. . // // Usage (Gradle runs this; see stageLitertLmEngine in composeApp/build.gradle.kts): // node scripts/stage-litert-lm-engine.mjs --ndk --out [--abis arm64-v8a,x86_64] -// [--cmake ] [--ninja ] +// [--cmake ] [--ninja ] [--cache ] import { createHash } from 'node:crypto' import { execFileSync } from 'node:child_process' -import { copyFileSync, existsSync, mkdirSync, readFileSync, renameSync, statSync, writeFileSync } from 'node:fs' +import { + copyFileSync, existsSync, lstatSync, mkdirSync, readFileSync, readdirSync, realpathSync, renameSync, rmSync, statSync, + writeFileSync, +} from 'node:fs' import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { inflateRawSync } from 'node:zlib' // ── Pinned runtime ─────────────────────────────────────────────────────────────────────── -// Bumping LiteRT-LM = change the version and both hashes together (PyPI lists them under -// https://pypi.org/pypi/litert-lm-api//json, "digests.sha256"), then rebuild. The server -// talks to the library through its C API, so check c/engine.h and c/conversation.h for changes -// to the functions listed in engines/litert-lm-server/src/litert_lm_api.h. +// Bumping LiteRT-LM = change the version and all four hashes together, then rebuild: each wheel's +// (PyPI lists them under https://pypi.org/pypi/litert-lm-api//json, "digests.sha256") +// and each patched library's (the build stops and prints the new one, after its checks below +// have passed). The server talks to the library through its C API, so check c/engine.h and +// c/conversation.h for changes to the functions listed in engines/litert-lm-server/src/litert_lm_api.h. export const LITERT_LM_VERSION = '0.18.0' const WHEELS = { 'arm64-v8a': { file: `litert_lm_api-${LITERT_LM_VERSION}-py3-none-android_23_arm64_v8a.whl`, sha256: 'a045af401dc199c35b1f44c29e8899cecbf2712a8194c9e35489ab5e2a95d47d', + // liblitert-lm.so after the DT_NEEDED rename below; nothing else in it differs from the wheel's. + patchedSha256: 'ae7e50538d9298e113686bc2ac07ebf33aac1325e6e3cbff299a1024a1b16d6a', elfMachine: 0xb7, // EM_AARCH64 + triple: 'aarch64-linux-android', }, x86_64: { file: `litert_lm_api-${LITERT_LM_VERSION}-py3-none-android_23_x86_64.whl`, sha256: 'db5268c5eb8e60f5a7cdecea310f8084426ae174fdbe8efe2991c47f54d74889', + patchedSha256: '681f770323d7021ca47a0481ae49744699d80a62c79f759cd1ce443e21becb22', elfMachine: 0x3e, // EM_X86_64 + triple: 'x86_64-linux-android', }, } const LIB_IN_WHEEL = 'litert_lm/liblitert-lm.so' +// The app's minSdk. The server and the shims are built for it, and the NDK's stub libraries for +// it say what every supported device provides; the library itself is built for API 23. Also the +// engine tether's (stage-tether-exec.mjs), as are ndkToolchain and checkElf. +export const MIN_SDK = 24 const appRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..') const serverSrc = join(appRoot, 'engines', 'litert-lm-server') @@ -142,7 +163,7 @@ function extractFromZip(zipPath, member) { // ── ELF checks: right architecture, and 16 KB-aligned LOAD segments ──────────────────────── // A 4 KB-aligned file fails to load on a 16 KB-page device, and Play rejects updates that // carry one (targetSdk 35+). Checked here so a bad file stops the build, not a user's launch. -function checkElf(path, machine) { +export function checkElf(path, machine) { const b = readFileSync(path) if (b.readUInt32BE(0) !== 0x7f454c46) fail(`${path}: not an ELF file`) if (b[4] !== 2) fail(`${path}: not a 64-bit ELF`) @@ -159,6 +180,380 @@ function checkElf(path, machine) { } } +// ── The engine process cannot load libandroid.so ───────────────────────────────────────── +// liblitert-lm.so is built to be loaded by an app into its own process, and it lists +// libandroid.so among its dependencies. The server is not an app process: the daemon executes it +// from nativeLibraryDir, and in a freshly executed process the linker has to find libandroid.so's +// whole dependency tree itself (libandroid_runtime, libhwui, the media stack), which an app, +// forked from the zygote, already has loaded. On the owner's phone that failed, so no .litertlm +// ever loaded: +// dlopen failed: library "libstagefright.so" not found: needed by /system/lib64/libmediandk.so +// in namespace (default) +// The library uses libandroid.so only to read APK assets (AAssetManager_*, AAsset_*), which the +// server never asks for: it loads models from file paths. So the build replaces that dependency, +// working from the file it ships rather than from a list written for one LiteRT-LM version: +// - LOADABLE_LIBS names the libraries a standalone process can load. Every other DT_NEEDED entry +// is replaced. +// - Each GLOBAL import that, going by the NDK's stub libraries, only a replaced library provides +// gets a function in a generated shim that, if it is ever called, names itself on stderr and in +// logcat and aborts. It never returns a value it would have to make up. WEAK imports, which +// the library allows to be absent, get none; those a replaced library provides must also come +// from a loadable one (AHardwareBuffer_*, which libandroid.so also defines from API 26, now +// bind to libnativewindow.so's, which libEGL.so needs). The build fails, naming the symbols, if +// a shimmed import is data rather than a function, if a GLOBAL import has no provider at all, +// if a GLOBAL import comes from both a replaced and a loadable library (it would bind +// elsewhere), or if a WEAK one comes only from a replaced library (it would be left unresolved). +// Each of those is checked at every API level from minSdk to the newest the NDK has stubs for, +// since a device of any of them may run the app. +// - The DT_NEEDED string is renamed in place to the shim's name, which has the same length +// (libandroid.so -> liblrtandr.so). Nothing else in the file changes, and the result must +// match the pinned patchedSha256. +// - A probe executable is then linked against the staged library, the shims and the NDK stubs +// with --no-allow-shlib-undefined, so a GLOBAL import that nothing in the library's dependency +// list provides fails the build with the linker's own message. + +// The libraries a standalone executable can load: bionic's own (libc, libm, libdl, which every +// process has), liblog and libz, and the graphics libraries native system daemons link (EGL, +// GLES, Vulkan, nativewindow). On the owner's phone the linker, which loads breadth-first, got +// through every library liblitert-lm.so needs directly and failed levels down in libandroid.so's +// tree. libandroid.so is the framework's app-facing library, and so are the other NDK libraries +// left out here (media, camera, ...). Add one only with a reason. +export const LOADABLE_LIBS = [ + 'libc.so', 'libm.so', 'libdl.so', 'liblog.so', 'libz.so', 'libEGL.so', 'libGLESv1_CM.so', + 'libGLESv2.so', 'libGLESv3.so', 'libnativewindow.so', 'libvulkan.so', +] +// The other names a shim must not take. +const STAGED_FILES = ['liblitert-lm.so', 'liblitertlm_server.so'] + +const DT_NEEDED = 1 +const DT_STRTAB = 5 +const DT_SYMTAB = 6 +const DT_STRSZ = 10 +const DT_VERDEF = 0x6ffffffc +const DT_VERDEFNUM = 0x6ffffffd +const DT_VERNEED = 0x6ffffffe +const DT_VERNEEDNUM = 0x6fffffff +// Every dynamic tag whose value is a .dynstr offset: NEEDED, SONAME, RPATH, RUNPATH, AUXILIARY, FILTER. +const DT_STRING_TAGS = new Set([DT_NEEDED, 14, 15, 29, 0x7ffffffd, 0x7fffffff]) +const STB_GLOBAL = 1 +const STB_WEAK = 2 +const STT_NOTYPE = 0 +const STT_FUNC = 2 +const SYMBOL_TYPES = { 0: 'untyped symbol', 1: 'data object', 2: 'function', 6: 'TLS variable', 10: 'IFUNC' } + +// The dynamic linking view of a 64-bit little-endian ELF file (what both ABIs ship): its DT_NEEDED +// entries, its dynamic symbols, the libraries it needs symbol versions from, and every .dynstr +// offset anything refers to, so a rename can prove it changes one string only. Found through the +// program headers, as the device's linker finds it; the symbol count comes from the .dynsym +// section header, which the dynamic section does not record. +export function readDynamic(b, what) { + if (b.length < 64 || b.readUInt32BE(0) !== 0x7f454c46 || b[4] !== 2 || b[5] !== 1) { + throw new Error(`${what}: not a 64-bit little-endian ELF file`) + } + const u64 = (o) => Number(b.readBigUInt64LE(o)) + const loads = [] + let dynamic + for (let i = 0; i < b.readUInt16LE(56); i++) { + const o = u64(32) + i * b.readUInt16LE(54) + const seg = { type: b.readUInt32LE(o), off: u64(o + 8), vaddr: u64(o + 16), filesz: u64(o + 32) } + if (seg.type === 1) loads.push(seg) // PT_LOAD + else if (seg.type === 2) dynamic = seg // PT_DYNAMIC + } + if (!dynamic) throw new Error(`${what}: no dynamic section`) + const fileOffset = (vaddr) => { + const seg = loads.find((l) => vaddr >= l.vaddr && vaddr < l.vaddr + l.filesz) + if (!seg) throw new Error(`${what}: address 0x${vaddr.toString(16)} is in no loaded segment`) + return seg.off + vaddr - seg.vaddr + } + const entries = [] + for (let o = dynamic.off; o + 16 <= dynamic.off + dynamic.filesz && u64(o) !== 0; o += 16) { + entries.push({ tag: u64(o), val: u64(o + 8) }) + } + const value = (tag) => entries.find((e) => e.tag === tag)?.val + if ([DT_STRTAB, DT_STRSZ, DT_SYMTAB].some((t) => value(t) === undefined)) { + throw new Error(`${what}: dynamic section without DT_STRTAB, DT_STRSZ or DT_SYMTAB`) + } + const strtab = fileOffset(value(DT_STRTAB)) + const strsz = value(DT_STRSZ) + const refs = [] + const str = (off, by) => { + if (off >= strsz) throw new Error(`${what}: ${by} points outside .dynstr`) + const ref = { off, by } + refs.push(ref) + return { name: b.toString('utf8', strtab + off, b.indexOf(0, strtab + off)), ref } + } + + const needed = [] + for (const e of entries) { + if (!DT_STRING_TAGS.has(e.tag)) continue + const s = str(e.val, `dynamic tag 0x${e.tag.toString(16)}`) + if (e.tag === DT_NEEDED) needed.push({ name: s.name, off: e.val, ref: s.ref }) + } + + const symtab = fileOffset(value(DT_SYMTAB)) + let count + for (let i = 0; i < b.readUInt16LE(60); i++) { + const o = u64(40) + i * b.readUInt16LE(58) + if (b.readUInt32LE(o + 4) === 11 && u64(o + 24) === symtab) count = u64(o + 32) / 24 // SHT_DYNSYM + } + if (count === undefined) throw new Error(`${what}: no .dynsym section header for DT_SYMTAB`) + const symbols = [] + for (let i = 1; i < count; i++) { // 0 is the null symbol + const o = symtab + i * 24 + symbols.push({ + name: str(b.readUInt32LE(o), `symbol ${i}`).name, + bind: b[o + 4] >> 4, + type: b[o + 4] & 0xf, + defined: b.readUInt16LE(o + 6) !== 0, + }) + } + + // Elf64_Verneed {vn_version, vn_cnt, vn_file, vn_aux, vn_next} and its Elf64_Vernaux entries + // {vna_hash, vna_flags, vna_other, vna_name, vna_next}; Elf64_Verdef {vd_version, vd_flags, + // vd_ndx, vd_cnt, vd_hash, vd_aux, vd_next} and its Elf64_Verdaux {vda_name, vda_next}. + const versionedFrom = new Set() + const verneeds = value(DT_VERNEEDNUM) ?? 0 + for (let i = 0, o = verneeds && fileOffset(value(DT_VERNEED)); i < verneeds; i++, o += b.readUInt32LE(o + 12)) { + versionedFrom.add(str(b.readUInt32LE(o + 4), 'a version need').name) + for (let j = 0, a = o + b.readUInt32LE(o + 8); j < b.readUInt16LE(o + 2); j++, a += b.readUInt32LE(a + 12)) { + str(b.readUInt32LE(a + 8), 'a version need entry') + } + } + const verdefs = value(DT_VERDEFNUM) ?? 0 + for (let i = 0, o = verdefs && fileOffset(value(DT_VERDEF)); i < verdefs; i++, o += b.readUInt32LE(o + 16)) { + for (let j = 0, a = o + b.readUInt32LE(o + 12); j < b.readUInt16LE(o + 6); j++, a += b.readUInt32LE(a + 4)) { + str(b.readUInt32LE(a), 'a version definition') + } + } + return { what, strtab, needed, symbols, refs, versionedFrom } +} + +// The NDK's stub libraries for one ABI, in dir (sysroot/usr/lib/): apis, every API level +// it has stubs for from minApi up, in order, and exportsOf(lib, api), the symbols the stub of lib +// at that level exports (name -> st_type), or undefined when there is no such stub. +export function ndkStubs(dir, minApi) { + const apis = readdirSync(dir).filter((d) => /^\d+$/.test(d)).map(Number).filter((a) => a >= minApi).sort((a, b) => a - b) + if (apis[0] !== minApi) throw new Error(`${dir}: no stub libraries for API ${minApi}`) + const exported = new Map() + const exportsOf = (name, api) => { + const f = join(dir, String(api), name) + if (!exported.has(f)) { + const syms = existsSync(f) ? readDynamic(readFileSync(f), f).symbols : undefined + exported.set(f, syms && new Map(syms.filter((s) => s.defined && (s.bind === STB_GLOBAL || s.bind === STB_WEAK)).map((s) => [s.name, s.type]))) + } + return exported.get(f) + } + return { apis, exportsOf } +} + +// Which of the library's DT_NEEDED entries are replaced, and the functions each shim must define. +// exportsOf(lib, api) gives the NDK stub's exported symbols (name -> st_type), or undefined when +// the NDK has no such stub at that API level; apis lists the levels to check, minSdk first, up to +// the newest the NDK has. Every check runs at each of them, because a device of any of them may +// run the app, and a symbol can move between libraries at any level in between: a WEAK import +// libandroid.so gains at API N and a loadable library only at a later M goes unresolved on +// N..M-1, which comparing only the first and the last level would miss. +export function planShims(lib, exportsOf, apis) { + const [minApi] = apis + const replaced = lib.needed.map((n) => n.name).filter((n) => !LOADABLE_LIBS.includes(n)) + const problems = new Set() + const head = `${lib.what}: ${replaced.length ? `cannot stand in for ${replaced.join(', ')}` : 'imports nothing provides'}:` + for (const r of replaced) { + if (!exportsOf(r, minApi)) problems.add(`${r} has no NDK stub at API ${minApi}, so what only it provides is unknown`) + if (lib.versionedFrom.has(r)) problems.add(`${r}: imports from it carry symbol versions, which a shim does not define`) + } + if (problems.size) throw new Error(`${head}\n ${[...problems].join('\n ')}`) + const from = (libs, name, api) => libs.filter((l) => exportsOf(l, api)?.has(name)) + // A problem found at several levels is reported once, with them. In its text \0, which no name + // read from .dynstr can contain, stands for the levels. + const levels = new Map() + const note = (problem, api) => levels.set(problem, [...(levels.get(problem) ?? []), api]) + const shims = replaced.map((r) => ({ replaced: r, functions: [] })) + for (const s of lib.symbols) { + if (s.defined || (s.bind !== STB_GLOBAL && s.bind !== STB_WEAK)) continue + const weak = s.bind === STB_WEAK + for (const api of apis) { + const owners = from(replaced, s.name, api) + const loadable = from(LOADABLE_LIBS, s.name, api) + if (!weak && !owners.length && !loadable.length) note(`${s.name}: no library provides it at API \0`, api) + if (!weak && owners.length && loadable.length) { + note(`${s.name}: ${owners.join(', ')} and ${loadable.join(', ')} both provide it (API \0); without ${owners[0]} it would bind elsewhere`, api) + } + if (weak && owners.length && !loadable.length) { + note(`${s.name} (weak): only ${owners.join(', ')} provides it (API \0); without it, it would be left unresolved`, api) + } + } + const owner = from(replaced, s.name, minApi)[0] + if (weak || !owner) continue + const type = s.type !== STT_NOTYPE ? s.type : exportsOf(owner, minApi).get(s.name) + if (type !== STT_FUNC) { + problems.add(`${s.name}: a ${SYMBOL_TYPES[type] ?? `type ${type} symbol`} from ${owner}; a shim can only stand in for functions`) + } else if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(s.name)) { + problems.add(`${s.name}: from ${owner}, not a C identifier`) + } else { + shims.find((sh) => sh.replaced === owner).functions.push(s.name) + } + } + for (const [problem, at] of levels) problems.add(problem.replace('\0', apiRanges(at, apis))) + if (problems.size) throw new Error(`${head}\n ${[...problems].join('\n ')}`) + const taken = new Set([...lib.needed.map((n) => n.name), ...LOADABLE_LIBS, ...STAGED_FILES]) + for (const sh of shims) { + sh.shim = shimFileName(sh.replaced, taken) + taken.add(sh.shim) + sh.functions.sort() + } + return shims +} + +// The levels in at, in the order of apis, as runs of consecutive entries of it: "26-28, 31". +function apiRanges(at, apis) { + const runs = [] + for (const api of at) { + const run = runs[runs.length - 1] + if (run && apis.indexOf(api) === apis.indexOf(run[1]) + 1) run[1] = api + else runs.push([api, api]) + } + return runs.map(([first, last]) => (first === last ? `${first}` : `${first}-${last}`)).join(', ') +} + +// A shim's file name is exactly as long as the name it replaces, so the DT_NEEDED string can be +// rewritten in place: "lrt" and the start of the old name (libandroid.so -> liblrtandr.so), with +// a base-36 counter over its last characters while that is taken. The counter never covers the +// "lr", so every shim name starts with liblr: that is what lets isShimFileName tell a shim an +// earlier run staged from another library in the same directory (such as libomp.so, which a +// counter over all three letters of a libxyz.so could otherwise have been named). +export function shimFileName(replaced, taken) { + const m = /^lib(.+)\.so$/.exec(replaced) + if (!m) throw new Error(`${replaced}: not a lib*.so name`) + const n = m[1].length + if (n < 2) throw new Error(`${replaced}: too short for a shim name of its length (liblr*.so)`) + const base = `lrt${m[1]}`.slice(0, n) + for (let i = 0; i < 36 ** Math.min(n - 2, 3); i++) { + const tag = i ? i.toString(36) : '' + const name = `lib${base.slice(0, n - tag.length)}${tag}.so` + if (!taken.has(name)) return name + } + throw new Error(`no free shim name of ${replaced.length} characters for ${replaced}`) +} + +// Whether shimFileName can produce name: liblr, then either t and anything, or nothing, or a +// counter of up to three base-36 digits; then .so. +export function isShimFileName(name) { + return /^liblr(t.*|[1-9a-z][0-9a-z]{0,2})?\.so$/s.test(name) +} + +// Removes from dir the shims an earlier run staged that this one does not (a LiteRT-LM bump can +// rename or drop one, and it would otherwise stay in the APK), and nothing else: only regular +// files whose name isShimFileName accepts. dir can hold other libraries; with --out pointed at +// composeApp/src/androidMain/jniLibs it holds the llama.cpp engines, which are slow to rebuild. +export function removeStaleShims(dir, staged) { + const removed = [] + for (const f of readdirSync(dir)) { + if (staged.has(f) || !isShimFileName(f) || !lstatSync(join(dir, f)).isFile()) continue + rmSync(join(dir, f)) + removed.push(f) + } + return removed +} + +// Renames one DT_NEEDED entry in place, refusing if any other reference into .dynstr (a symbol +// name, a version entry, another dynamic tag) shares its bytes: those, and every other copy of the +// name in the file, such as the one in .rodata next to TFLite's NNAPI strings, are left alone. +export function renameNeeded(b, lib, from, to) { + const n = lib.needed.find((e) => e.name === from) + if (!n || to.length !== from.length) throw new Error(`${lib.what}: cannot rename DT_NEEDED ${from} to ${to}`) + const at = lib.strtab + n.off + if (b.toString('latin1', at, at + from.length + 1) !== `${from}\0` || b[at - 1] !== 0) { + throw new Error(`${lib.what}: DT_NEEDED ${from} is not a string of its own at .dynstr+${n.off}`) + } + const shared = lib.refs.filter((r) => r !== n.ref && r.off >= n.off && r.off < n.off + from.length) + if (shared.length) { + throw new Error(`${lib.what}: ${shared.map((r) => r.by).join(', ')} share(s) the string of DT_NEEDED ${from}`) + } + b.write(to, at, 'latin1') +} + +export function shimSource({ replaced, shim, functions }, origin) { + return `// Generated by scripts/stage-litert-lm-engine.mjs for ${origin}. Do not edit. +// +// liblitert-lm.so lists ${replaced} as a dependency, and the engine process cannot load +// ${replaced}, so the staging script renamed that dependency to ${shim}: this library. It +// defines the ${functions.length} function(s) liblitert-lm.so imports from ${replaced} and nothing else. None +// of them should run in the engine; if one does, it says which, on stderr and in logcat, and +// aborts: it does not return a value it would have to make up. +#include +#include +#include +#include + +#define WHY "which ${replaced} provides, but ${replaced} cannot be loaded in the engine process (${shim} stands in for it); stopping." + +__attribute__((noreturn)) static void stop(const char* function) { + dprintf(STDERR_FILENO, "liblitert-lm.so called %s(), %s\\n", function, WHY); + __android_log_print(ANDROID_LOG_FATAL, "litert-lm-server", "liblitert-lm.so called %s(), %s", function, WHY); + abort(); +} + +#define SHIM(name) \\ + __attribute__((visibility("default"), noreturn)) void name(void); \\ + void name(void) { stop(#name); } + +${functions.map((f) => `SHIM(${f})`).join('\n')} +` +} + +export function ndkToolchain(ndk) { + const prebuilt = join(ndk, 'toolchains', 'llvm', 'prebuilt') + const exe = process.platform === 'win32' ? 'clang.exe' : 'clang' + const host = existsSync(prebuilt) ? readdirSync(prebuilt).find((h) => existsSync(join(prebuilt, h, 'bin', exe))) : undefined + if (!host) fail(`no NDK clang under ${prebuilt}`) + return { clang: join(prebuilt, host, 'bin', exe), libs: join(prebuilt, host, 'sysroot', 'usr', 'lib') } +} + +function run(cmd, args, what) { + try { + execFileSync(cmd, args, { stdio: ['ignore', 'inherit', 'inherit'] }) + } catch { + fail(`${what} failed (the compiler's or linker's message is above)`) + } +} + +// Builds a generated shim and checks that it exports exactly its functions and needs only +// loadable libraries. +function buildShim(sh, abi, tc, workDir) { + const src = join(workDir, sh.shim.replace(/\.so$/, '.c')) + const out = join(workDir, sh.shim) + writeFileSync(src, shimSource(sh, `LiteRT-LM ${LITERT_LM_VERSION}, ${abi}`)) + run(tc.clang, [ + `--target=${WHEELS[abi].triple}${MIN_SDK}`, '-shared', '-fPIC', '-O2', '-fvisibility=hidden', + '-Wall', '-Wextra', '-Werror', `-Wl,-soname,${sh.shim}`, '-Wl,--no-undefined', + '-Wl,--exclude-libs,ALL', '-Wl,-z,max-page-size=16384', '-s', '-o', out, src, '-llog', + ], `building ${sh.shim} for ${abi}`) + const built = readDynamic(readFileSync(out), out) + const exported = built.symbols.filter((s) => s.defined && (s.bind === STB_GLOBAL || s.bind === STB_WEAK)).map((s) => s.name) + if (exported.sort().join() !== sh.functions.join()) fail(`${out} exports ${exported.join(', ') || 'nothing'}, expected ${sh.functions.join(', ') || 'nothing'}`) + const extra = built.needed.map((n) => n.name).filter((n) => !LOADABLE_LIBS.includes(n)) + if (extra.length) fail(`${out} needs ${extra.join(', ')}, which the engine process may not load`) + return out +} + +// lld checks every undefined symbol of a shared library whose DT_NEEDED entries are all on the +// link line (--no-allow-shlib-undefined), against what is on that line. So the probe links the +// staged library, its shims and the NDK stubs of exactly the other libraries it needs, and the +// link fails if any GLOBAL import has no provider among them, as the device's linker would. +function probeLink(libPath, abi, tc, shimPaths, workDir) { + const lib = readDynamic(readFileSync(libPath), libPath) + const deps = lib.needed.map((n) => shimPaths.get(n.name) ?? join(tc.libs, WHEELS[abi].triple, String(MIN_SDK), n.name)) + const missing = deps.filter((d) => !existsSync(d)) + if (missing.length) fail(`${libPath}: no ${missing.join(', ')} to check it against`) + const probe = join(workDir, 'probe.c') + writeFileSync(probe, 'int main(void) { return 0; }\n') + run(tc.clang, [ + `--target=${WHEELS[abi].triple}${MIN_SDK}`, '-o', join(workDir, 'probe'), probe, libPath, ...deps, + '-Wl,--no-allow-shlib-undefined', '-Wl,--no-undefined', + ], `the link check of ${libPath} against its dependencies`) +} + function findTool(explicit, name) { if (explicit && existsSync(explicit)) return explicit return process.platform === 'win32' ? `${name}.exe` : name // fall back to PATH @@ -175,8 +570,8 @@ function buildServer(abi, opts) { `-DCMAKE_MAKE_PROGRAM=${ninja}`, `-DCMAKE_TOOLCHAIN_FILE=${toolchain}`, `-DANDROID_ABI=${abi}`, - // Same floor as the app (minSdk 24); the library itself is built for API 23. - '-DANDROID_PLATFORM=android-24', + // Same floor as the app; the library itself is built for API 23. + `-DANDROID_PLATFORM=android-${MIN_SDK}`, // Static and unexported: see engines/litert-lm-server/src/litert_lm_api.h. '-DANDROID_STL=c++_static', '-DCMAKE_BUILD_TYPE=Release', @@ -192,25 +587,59 @@ function buildServer(abi, opts) { async function main() { const opts = parseArgs(process.argv.slice(2)) + const tc = ndkToolchain(opts.ndk) for (const abi of opts.abis) { const w = WHEELS[abi] if (!w) fail(`no LiteRT-LM build for ABI ${abi} (have: ${Object.keys(WHEELS).join(', ')})`) const outDir = join(opts.out, abi) + const workDir = join(opts.out, '..', `shim-${abi}`) mkdirSync(outDir, { recursive: true }) + mkdirSync(workDir, { recursive: true }) const wheel = await fetchWheel(abi, opts.cache) + const data = Buffer.from(extractFromZip(wheel, LIB_IN_WHEEL)) + const parsed = readDynamic(data, `${LIB_IN_WHEEL} (${abi})`) + const { apis, exportsOf } = ndkStubs(join(tc.libs, w.triple), MIN_SDK) + const shims = planShims(parsed, exportsOf, apis) + for (const sh of shims) renameNeeded(data, parsed, sh.replaced, sh.shim) + const lib = join(outDir, 'liblitert-lm.so') - writeFileSync(lib, extractFromZip(wheel, LIB_IN_WHEEL)) + writeFileSync(lib, data) checkElf(lib, w.elfMachine) + const shimPaths = new Map() + for (const sh of shims) { + const staged = join(outDir, sh.shim) + copyFileSync(buildShim(sh, abi, tc, workDir), staged) + checkElf(staged, w.elfMachine) + shimPaths.set(sh.shim, staged) + console.log(`liblitert-lm.so (${abi}): ${sh.replaced} replaced by ${sh.shim}, ` + + `which aborts if any of its ${sh.functions.length} function(s) is called: ${sh.functions.join(', ')}`) + } + probeLink(lib, abi, tc, shimPaths, workDir) + // Last, so that a LiteRT-LM bump learns the new hash only once every check above has passed. + const patched = sha256(data) + if (patched !== w.patchedSha256) { + fail(`${LIB_IN_WHEEL} (${abi}) with ${shims.map((sh) => `${sh.replaced} replaced by ${sh.shim}`).join(', ') || 'nothing replaced'}: ` + + `sha256 ${patched} does not match the pinned patchedSha256 ${w.patchedSha256}`) + } const server = join(outDir, 'liblitertlm_server.so') copyFileSync(buildServer(abi, opts), server) checkElf(server, w.elfMachine) + for (const f of removeStaleShims(outDir, new Set(shimPaths.keys()))) { + console.log(`removed ${join(outDir, f)}: a shim an earlier run staged and this one does not`) + } + console.log(`staged LiteRT-LM ${LITERT_LM_VERSION} for ${abi}: ` + `liblitert-lm.so ${(statSync(lib).size / 1e6).toFixed(1)} MB, ` + - `liblitertlm_server.so ${(statSync(server).size / 1e3).toFixed(0)} KB`) + `liblitertlm_server.so ${(statSync(server).size / 1e3).toFixed(0)} KB` + + [...shimPaths].map(([name, f]) => `, ${name} ${(statSync(f).size / 1e3).toFixed(0)} KB`).join('')) } } -main().catch((e) => fail(e instanceof Error ? e.message : String(e))) +// Run as a script; the test imports the functions above without staging anything. Real paths on +// both sides, so a symlinked checkout cannot make a Gradle run skip the staging and still pass. +if (process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))) { + main().catch((e) => fail(e instanceof Error ? e.message : String(e))) +} diff --git a/scripts/stage-litert-lm-engine.test.mjs b/scripts/stage-litert-lm-engine.test.mjs new file mode 100644 index 0000000..4b627b0 --- /dev/null +++ b/scripts/stage-litert-lm-engine.test.mjs @@ -0,0 +1,345 @@ +// Tests for the libandroid.so replacement in stage-litert-lm-engine.mjs: reading the library's +// dynamic section and the NDK's stubs, deciding what each shim defines, renaming the DT_NEEDED +// entry, naming the shim and removing a stale one. They run on small synthetic ELF files, so they +// need neither the NDK nor the wheel: +// node --test scripts/stage-litert-lm-engine.test.mjs +// (Every staging run also checks the real library: the shim build, the link probe and the pinned +// hash of the patched file.) +import assert from 'node:assert/strict' +import { mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { test } from 'node:test' + +import { + isShimFileName, ndkStubs, planShims, readDynamic, removeStaleShims, renameNeeded, shimFileName, shimSource, +} from './stage-litert-lm-engine.mjs' + +const GLOBAL = 1 +const WEAK = 2 +const OBJECT = 1 +const FUNC = 2 + +// A minimal 64-bit little-endian shared object, laid out as a linker would for the parts the +// script reads: one PT_LOAD over the whole file (vaddr = file offset), PT_DYNAMIC, .dynstr, +// .dynsym with its section header, optional version needs, and some bytes after them standing in +// for .rodata. Identical strings share one .dynstr entry, as lld emits them. +function fakeElf({ needed, symbols, verneed = [], rodata = '' }) { + const strings = [''] + const offsets = new Map([['', 0]]) + let strsz = 1 + const strOff = (s) => { + if (!offsets.has(s)) { + offsets.set(s, strsz) + strings.push(s) + strsz += Buffer.byteLength(s) + 1 + } + return offsets.get(s) + } + for (const n of needed) strOff(n) + for (const s of symbols) strOff(s.name) + for (const v of verneed) [v.file, ...v.versions].forEach(strOff) + const dynstr = Buffer.from(strings.map((s) => `${s}\0`).join(''), 'latin1') + + const align = (n) => (n + 7) & ~7 + const strtabOff = 64 + 2 * 56 + const symtabOff = align(strtabOff + dynstr.length) + const verneedOff = symtabOff + 24 * (symbols.length + 1) + const verneedSize = verneed.reduce((n, v) => n + 16 + 16 * v.versions.length, 0) + const dynamicOff = align(verneedOff + verneedSize) + const dyn = [ + ...needed.map((n) => [1, strOff(n)]), + [5, strtabOff], [10, dynstr.length], [6, symtabOff], + ...(verneed.length ? [[0x6ffffffe, verneedOff], [0x6fffffff, verneed.length]] : []), + [0, 0], + ] + const rodataOff = dynamicOff + 16 * dyn.length + const shoff = align(rodataOff + rodata.length) + const b = Buffer.alloc(shoff + 2 * 64) + + b.writeUInt32BE(0x7f454c46, 0) + b[4] = 2 // ELFCLASS64 + b[5] = 1 // ELFDATA2LSB + b[6] = 1 + b.writeUInt16LE(3, 16) // ET_DYN + b.writeUInt16LE(0xb7, 18) + b.writeUInt32LE(1, 20) + b.writeBigUInt64LE(64n, 32) // e_phoff + b.writeBigUInt64LE(BigInt(shoff), 40) + b.writeUInt16LE(64, 52) + b.writeUInt16LE(56, 54) + b.writeUInt16LE(2, 56) + b.writeUInt16LE(64, 58) + b.writeUInt16LE(2, 60) + const phdr = (o, type, off, size) => { + b.writeUInt32LE(type, o) + b.writeBigUInt64LE(BigInt(off), o + 8) + b.writeBigUInt64LE(BigInt(off), o + 16) + b.writeBigUInt64LE(BigInt(size), o + 32) + b.writeBigUInt64LE(BigInt(size), o + 40) + } + phdr(64, 1, 0, b.length) // PT_LOAD + phdr(64 + 56, 2, dynamicOff, 16 * dyn.length) // PT_DYNAMIC + + dynstr.copy(b, strtabOff) + symbols.forEach((s, i) => { + const o = symtabOff + 24 * (i + 1) + b.writeUInt32LE(strOff(s.name), o) + b[o + 4] = (s.bind << 4) | s.type + b.writeUInt16LE(s.defined ? 1 : 0, o + 6) + }) + let o = verneedOff + verneed.forEach((v, i) => { + b.writeUInt16LE(1, o) + b.writeUInt16LE(v.versions.length, o + 2) + b.writeUInt32LE(strOff(v.file), o + 4) + b.writeUInt32LE(16, o + 8) + b.writeUInt32LE(i < verneed.length - 1 ? 16 + 16 * v.versions.length : 0, o + 12) + v.versions.forEach((name, j) => { + const a = o + 16 + 16 * j + b.writeUInt32LE(strOff(name), a + 8) + b.writeUInt32LE(j < v.versions.length - 1 ? 16 : 0, a + 12) + }) + o += 16 + 16 * v.versions.length + }) + dyn.forEach(([tag, val], i) => { + b.writeBigUInt64LE(BigInt(tag), dynamicOff + 16 * i) + b.writeBigUInt64LE(BigInt(val), dynamicOff + 16 * i + 8) + }) + b.write(rodata, rodataOff, 'latin1') + const sh = shoff + 64 // [0] is the null section header + b.writeUInt32LE(11, sh + 4) // SHT_DYNSYM + b.writeBigUInt64LE(BigInt(symtabOff), sh + 24) + b.writeBigUInt64LE(BigInt(24 * (symbols.length + 1)), sh + 32) + b.writeBigUInt64LE(24n, sh + 56) + return b +} + +const fn = (name, bind = GLOBAL) => ({ name, bind, type: FUNC }) + +// Stand-ins for the NDK's stub libraries: each library's symbols, with their type and the API level +// the library gains them at. A library has no stub below the level of its first symbol, as the +// NDK has no libnativewindow.so below API 26 (where it and libandroid.so both gain +// AHardwareBuffer_acquire). +const STUBS = { + 'libandroid.so': { AAsset_close: [FUNC, 24], AAssetManager_open: [FUNC, 24], ALooper_forThread: [FUNC, 24], AHardwareBuffer_acquire: [FUNC, 26] }, + 'libc.so': { malloc: [FUNC, 24], stderr: [OBJECT, 24], dprintf: [FUNC, 24] }, + 'liblog.so': { __android_log_print: [FUNC, 24] }, + 'libnativewindow.so': { AHardwareBuffer_acquire: [FUNC, 26] }, +} +const APIS = [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] +const withExtra = (extra = {}) => Object.fromEntries([...new Set([...Object.keys(STUBS), ...Object.keys(extra)])] + .map((lib) => [lib, { ...STUBS[lib], ...extra[lib] }])) +const exportsAt = (spec, lib, api) => { + const syms = Object.entries(spec[lib] ?? {}).filter(([, [, since]]) => since <= api) + return syms.length ? new Map(syms.map(([name, [type]]) => [name, type])) : undefined +} +const stubs = (extra) => (lib, api) => exportsAt(withExtra(extra), lib, api) + +// Shaped like LiteRT-LM 0.18.0's liblitert-lm.so: libandroid.so first among its dependencies, +// APK-asset functions imported from it, the rest from bionic, AHardwareBuffer_* imported weak. +const litertLike = (extraSymbols = [], extra = {}) => ({ + needed: ['libandroid.so', 'liblog.so', 'libc.so'], + symbols: [ + { name: 'litert_lm_engine_create', bind: GLOBAL, type: FUNC, defined: true }, + fn('AAsset_close'), fn('AAssetManager_open'), fn('malloc'), fn('__android_log_print'), + { name: 'stderr', bind: GLOBAL, type: OBJECT }, + fn('AHardwareBuffer_acquire', WEAK), + ...extraSymbols, + ], + rodata: 'ANEURALNETWORKS_NO_ERROR\0libandroid.so\0', + ...extra, +}) +const plan = (spec, extraStubs) => planShims(readDynamic(fakeElf(spec), 'liblitert-lm.so'), stubs(extraStubs), APIS) + +test('readDynamic reads the DT_NEEDED entries, the dynamic symbols and the version needs', () => { + const lib = readDynamic(fakeElf(litertLike([], { verneed: [{ file: 'libc.so', versions: ['LIBC'] }] })), 'x') + assert.deepEqual(lib.needed.map((n) => n.name), ['libandroid.so', 'liblog.so', 'libc.so']) + assert.deepEqual(lib.symbols.find((s) => s.name === 'stderr'), { name: 'stderr', bind: GLOBAL, type: OBJECT, defined: false }) + assert.equal(lib.symbols.find((s) => s.name === 'litert_lm_engine_create').defined, true) + assert.deepEqual([...lib.versionedFrom], ['libc.so']) + assert.throws(() => readDynamic(Buffer.from('not an ELF file at all, just some bytes'.repeat(3)), 'x'), /not a 64-bit/) +}) + +test('libandroid.so gets a shim with exactly the functions only it provides', () => { + assert.deepEqual(plan(litertLike()), [{ replaced: 'libandroid.so', shim: 'liblrtandr.so', functions: ['AAssetManager_open', 'AAsset_close'] }]) +}) + +test('a new libandroid.so-only import adds a function to the shim', () => { + assert.deepEqual(plan(litertLike([fn('ALooper_forThread')]))[0].functions, ['AAssetManager_open', 'AAsset_close', 'ALooper_forThread']) +}) + +test('a library that needs only loadable libraries gets no shim', () => { + assert.deepEqual(plan({ needed: ['libc.so'], symbols: [fn('malloc')] }), []) +}) + +test('a data object only libandroid.so provides fails the build', () => { + const extra = { 'libandroid.so': { AConfiguration_table: [OBJECT, 24] } } + assert.throws(() => plan(litertLike([{ name: 'AConfiguration_table', bind: GLOBAL, type: OBJECT }]), extra), + /AConfiguration_table: a data object from libandroid.so; a shim can only stand in for functions/) +}) + +test('a GLOBAL import no library provides fails the build', () => { + assert.throws(() => plan(litertLike([fn('mystery_function')])), /mystery_function: no library provides it at API 24-35$/m) +}) + +test('a GLOBAL import both libandroid.so and a loadable library provide fails the build', () => { + const extra = { 'libandroid.so': { dprintf: [FUNC, 30] } } + assert.throws(() => plan(litertLike([fn('dprintf')]), extra), /dprintf: libandroid.so and libc.so both provide it \(API 30-35\)/) +}) + +test('a WEAK import only libandroid.so provides fails the build; one a loadable library also provides does not', () => { + const extra = { 'libandroid.so': { ANativeActivity_finish: [FUNC, 35] } } + assert.throws(() => plan(litertLike([fn('ANativeActivity_finish', WEAK)]), extra), + /ANativeActivity_finish \(weak\): only libandroid.so provides it \(API 35\)/) + // AHardwareBuffer_acquire, in litertLike, is the second case (libnativewindow.so from API 26). + assert.doesNotThrow(() => plan(litertLike())) +}) + +// Writes the stub libraries spec describes (shaped like STUBS) as the NDK lays them out, +// //, for each of apis, plus a file beside the level directories as the NDK has. +function stubTree(dir, spec, apis) { + for (const api of apis) { + mkdirSync(join(dir, String(api))) + for (const lib of Object.keys(spec)) { + const syms = exportsAt(spec, lib, api) + if (!syms) continue + const symbols = [...syms].map(([name, type]) => ({ name, bind: GLOBAL, type, defined: true })) + writeFileSync(join(dir, String(api), lib), fakeElf({ needed: [], symbols })) + } + } + writeFileSync(join(dir, 'libc++_shared.so'), 'not an API level') +} + +test('every API level from minSdk to the newest stubs is checked: a WEAK import libandroid.so gains before a loadable library does fails the build', () => { + const dir = mkdtempSync(join(tmpdir(), 'stage-litert-lm-stubs-')) + try { + // libandroid.so gains AHardwareBuffer_lockPlanes at 26, libnativewindow.so only at 29: on + // API 26-28 it would be left unresolved. API 24 and 30, the first and the newest level, both + // look fine. No libandroid.so stub exists below 24 (STUBS starts it there), so checking 21-23 + // would fail the build too. + stubTree(dir, withExtra({ + 'libandroid.so': { AHardwareBuffer_lockPlanes: [FUNC, 26] }, + 'libnativewindow.so': { AHardwareBuffer_lockPlanes: [FUNC, 29] }, + }), [21, 22, 23, 24, 25, 26, 27, 28, 29, 30]) + const { apis, exportsOf } = ndkStubs(dir, 24) + assert.deepEqual(apis, [24, 25, 26, 27, 28, 29, 30]) + const lib = (extraSymbols) => readDynamic(fakeElf(litertLike(extraSymbols)), 'liblitert-lm.so') + assert.deepEqual(planShims(lib([]), exportsOf, apis).map((sh) => sh.functions), [['AAssetManager_open', 'AAsset_close']]) + assert.throws(() => planShims(lib([fn('AHardwareBuffer_lockPlanes', WEAK)]), exportsOf, apis), + /AHardwareBuffer_lockPlanes \(weak\): only libandroid.so provides it \(API 26-28\)/) + assert.throws(() => ndkStubs(dir, 20), /no stub libraries for API 20/) + } finally { + rmSync(dir, { recursive: true, force: true }) + } +}) + +test('a replaced library imported with symbol versions fails the build', () => { + assert.throws(() => plan(litertLike([], { verneed: [{ file: 'libandroid.so', versions: ['LIBANDROID'] }] })), + /libandroid.so: imports from it carry symbol versions/) +}) + +test('renameNeeded changes the DT_NEEDED string and nothing else', () => { + const before = fakeElf(litertLike()) + const after = Buffer.from(before) + const lib = readDynamic(after, 'x') + renameNeeded(after, lib, 'libandroid.so', 'liblrtandr.so') + const changed = [...after.keys()].filter((i) => after[i] !== before[i]) + assert.deepEqual(changed, [3, 4, 5, 6, 7, 8, 9].map((i) => lib.strtab + lib.needed[0].off + i)) + const reread = readDynamic(after, 'x') + assert.deepEqual(reread.needed.map((n) => n.name), ['liblrtandr.so', 'liblog.so', 'libc.so']) + assert.deepEqual(reread.symbols, lib.symbols) + assert.ok(after.includes('ANEURALNETWORKS_NO_ERROR\0libandroid.so\0'), 'the .rodata copy is left alone') +}) + +test('renameNeeded refuses a string another reference shares', () => { + // A symbol with the same name shares the string (lld emits identical strings once). + const same = fakeElf(litertLike([fn('libandroid.so')])) + assert.throws(() => renameNeeded(same, readDynamic(same, 'x'), 'libandroid.so', 'liblrtandr.so'), /symbol \d+ share\(s\) the string/) + // So does one that points into its tail ("android.so"). + const tail = fakeElf(litertLike()) + const lib = readDynamic(tail, 'x') + const symtab = Number(tail.readBigUInt64LE(Number(tail.readBigUInt64LE(40)) + 64 + 24)) + tail.writeUInt32LE(lib.needed[0].off + 3, symtab + 24 * 2) // symbol 2, AAsset_close + assert.throws(() => renameNeeded(tail, readDynamic(tail, 'x'), 'libandroid.so', 'liblrtandr.so'), /symbol 2 share\(s\) the string/) + assert.throws(() => renameNeeded(Buffer.from(tail), lib, 'libandroid.so', 'liblrtandr2.so'), /cannot rename/) +}) + +test('renameNeeded refuses a DT_NEEDED entry that is the tail of a longer string', () => { + // Point DT_NEEDED at the "libandroid.so" that ends the symbol name my_libandroid.so: no other + // reference starts inside it, but renaming it would rename that symbol. + const b = fakeElf(litertLike([fn('my_libandroid.so')])) + const { strtab } = readDynamic(b, 'x') + const dynamic = Number(b.readBigUInt64LE(64 + 56 + 8)) // PT_DYNAMIC's p_offset + b.writeBigUInt64LE(BigInt(b.indexOf('my_libandroid.so\0', strtab) - strtab + 3), dynamic + 8) // entry 0 + const lib = readDynamic(b, 'x') + assert.equal(lib.needed[0].name, 'libandroid.so') + const before = Buffer.from(b) + assert.throws(() => renameNeeded(b, lib, 'libandroid.so', 'liblrtandr.so'), /DT_NEEDED libandroid.so is not a string of its own/) + assert.ok(b.equals(before), 'nothing is written') +}) + +test('shim names keep the length of the name they replace and stay unique', () => { + assert.equal(shimFileName('libandroid.so', new Set()), 'liblrtandr.so') + assert.equal(shimFileName('libfoo.so', new Set()), 'liblrt.so') + const taken = new Set() + for (const lib of ['libmediandk.so', 'libmediaxyz.so', 'libmediaabc.so', 'libabc.so', 'libxyz.so', 'libcamera2ndk.so']) { + const name = shimFileName(lib, taken) + assert.equal(name.length, lib.length, name) + assert.match(name, /^lib[0-9a-z]+\.so$/) + assert.ok(!taken.has(name), name) + taken.add(name) + } + assert.deepEqual([...taken].slice(0, 5), ['liblrtmedia.so', 'liblrtmedi1.so', 'liblrtmedi2.so', 'liblrt.so', 'liblr1.so']) + assert.throws(() => shimFileName('android.so', new Set()), /not a lib\*\.so name/) +}) + +test('isShimFileName accepts every name shimFileName can produce, and not the libraries beside the shims', () => { + // Stems of two, three and four characters, until no name is left: the counter takes over the + // most characters there. + for (const replaced of ['libab.so', 'libxyz.so', 'libandr.so']) { + const taken = new Set() + for (;;) { + let name + try { + name = shimFileName(replaced, taken) + } catch (e) { + assert.match(e.message, /no free shim name/) + break + } + assert.ok(isShimFileName(name), `${replaced}: ${name}`) + taken.add(name) + } + assert.equal(taken.size, { 'libab.so': 1, 'libxyz.so': 35, 'libandr.so': 1260 }[replaced], replaced) + } + assert.throws(() => shimFileName('libx.so', new Set()), /too short/) + // What src/androidMain/jniLibs holds beside them (the llama.cpp engines) and the staged files. + for (const other of ['libggml-base.so', 'libggml.so', 'libllama.so', 'libllama_server.so', 'libllama_server_vk.so', + 'libmtmd.so', 'libomp.so', 'liblog.so', 'liblitert-lm.so', 'liblitertlm_server.so', 'liblr0.so', 'liblrAB.so']) { + assert.ok(!isShimFileName(other), other) + } +}) + +test('removeStaleShims removes the shims an earlier run staged and nothing else in the directory', () => { + const dir = mkdtempSync(join(tmpdir(), 'stage-litert-lm-out-')) + try { + // As if --out were composeApp/src/androidMain/jniLibs: the llama.cpp engines stay. + const others = ['libggml-base.so', 'libggml-cpu.so', 'libggml.so', 'libllama-common.so', 'libllama-server-impl.so', + 'libllama.so', 'libllama_server.so', 'libllama_server_vk.so', 'libmtmd.so', 'libomp.so', 'README.txt', + 'liblitert-lm.so', 'liblitertlm_server.so', 'liblrtandr.so'] + for (const f of [...others, 'liblrtjnigraph.so', 'liblr1.so']) writeFileSync(join(dir, f), f) + mkdirSync(join(dir, 'liblrtdir.so')) // a directory, which no run stages + assert.deepEqual(removeStaleShims(dir, new Set(['liblrtandr.so'])).sort(), ['liblr1.so', 'liblrtjnigraph.so']) + assert.deepEqual(readdirSync(dir).sort(), [...others, 'liblrtdir.so'].sort()) + } finally { + rmSync(dir, { recursive: true, force: true }) + } +}) + +test('the shim source defines each function to report itself and abort', () => { + const src = shimSource({ replaced: 'libandroid.so', shim: 'liblrtandr.so', functions: ['AAssetManager_open', 'AAsset_close'] }, 'test') + assert.deepEqual(src.match(/^SHIM\(.*\)$/gm), ['SHIM(AAssetManager_open)', 'SHIM(AAsset_close)']) + assert.match(src, /libandroid.so cannot be loaded in the engine process \(liblrtandr.so stands in for it\)/) + assert.match(src, /abort\(\);/) + assert.doesNotMatch(src.replace(/^\/\/.*$/gm, ''), /\breturn\b/, 'no function returns') +}) diff --git a/scripts/stage-play-listing.mjs b/scripts/stage-play-listing.mjs index e29ea6f..ede310e 100644 --- a/scripts/stage-play-listing.mjs +++ b/scripts/stage-play-listing.mjs @@ -28,7 +28,10 @@ if (!existsSync(listingCopyPath)) { console.error(`FATAL: missing ${listingCopyPath}`) process.exit(1) } -const md = readFileSync(listingCopyPath, 'utf8') +// LF only from here on: Git for Windows checks the file out with CRLF by default (.gitattributes pins only the +// shell scripts to LF), which the fence regex below would not match, and the copy would reach Play with stray \r +// in it. +const md = readFileSync(listingCopyPath, 'utf8').replace(/\r\n/g, '\n') // Extracts the first fenced ``` block that follows a `## ` line matching `headingRe`. function extractFenced(headingRe) { diff --git a/scripts/stage-tether-exec.mjs b/scripts/stage-tether-exec.mjs new file mode 100644 index 0000000..7ce65c0 --- /dev/null +++ b/scripts/stage-tether-exec.mjs @@ -0,0 +1,91 @@ +#!/usr/bin/env node +// Build the engine tether that ships inside the APK: libtether_exec.so, from +// engines/tether-exec/tether_exec.c, for each ABI the APK carries. +// +// The daemon starts every engine through it (TurboLLM's util/tether.ts): it arms +// PR_SET_PDEATHSIG(SIGKILL) and execs the engine, so the kernel kills the engine the moment the +// daemon's thread ends, however the app process ends. The daemon checks it once at boot by +// watching it work, and uses a /system/bin/sh lifeline instead when the check fails or the file is +// missing (BUILDING-ENGINES.md, "The engine tether"). +// +// Output: //libtether_exec.so, which build.gradle.kts adds to the APK's jniLibs. The +// name fits the lib*.so pattern Android requires before it will install a file into +// nativeLibraryDir, the only place an app may execute from (BUILDING-ENGINES.md). Nothing else in +// is touched. Built with the NDK's clang for the app's minSdk, as a PIE with 16 KB-aligned +// LOAD segments, stripped, and checked with the LiteRT-LM staging's checkElf. +// +// Usage (Gradle runs this; see stageTetherExec in composeApp/build.gradle.kts): +// node scripts/stage-tether-exec.mjs --ndk --out [--abis arm64-v8a,x86_64] +import { execFileSync } from 'node:child_process' +import { mkdirSync, realpathSync, renameSync, rmSync, statSync } from 'node:fs' +import { dirname, join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' + +import { checkElf, MIN_SDK, ndkToolchain } from './stage-litert-lm-engine.mjs' + +export const HELPER = 'libtether_exec.so' +export const ABIS = { + 'arm64-v8a': { triple: 'aarch64-linux-android', elfMachine: 0xb7 }, // EM_AARCH64 + x86_64: { triple: 'x86_64-linux-android', elfMachine: 0x3e }, // EM_X86_64 +} +// Every build of the helper, the NDK's here and the test's host build, uses these. +export const CFLAGS = ['-O2', '-Wall', '-Wextra', '-Werror'] + +export const SOURCE = join(resolve(dirname(fileURLToPath(import.meta.url)), '..'), 'engines', 'tether-exec', 'tether_exec.c') + +function parseArgs(argv) { + const out = { abis: Object.keys(ABIS) } + for (let i = 0; i < argv.length; i++) { + const k = argv[i] + const v = argv[i + 1] + if (k === '--ndk') out.ndk = v + else if (k === '--out') out.out = v + else if (k === '--abis') out.abis = v.split(',').map((s) => s.trim()).filter(Boolean) + else throw new Error(`unknown argument ${k}`) + i++ + } + if (!out.ndk) throw new Error('--ndk is required') + if (!out.out) throw new Error('--out is required') + return out +} + +function fail(msg) { + console.error(`FATAL: ${msg}`) + process.exit(1) +} + +function main() { + const opts = parseArgs(process.argv.slice(2)) + const unknown = opts.abis.filter((abi) => !ABIS[abi]) + if (unknown.length) fail(`no tether build for ABI ${unknown.join(', ')} (have: ${Object.keys(ABIS).join(', ')})`) + const { clang } = ndkToolchain(opts.ndk) + for (const abi of opts.abis) { + const dir = join(opts.out, abi) + mkdirSync(dir, { recursive: true }) + const staged = join(dir, HELPER) + // Built beside its final name and renamed once checked, so a failed build leaves no helper the + // APK would carry: the packaging check in build.gradle.kts then stops the build. + const tmp = `${staged}.tmp` + rmSync(tmp, { force: true }) + try { + execFileSync(clang, [ + `--target=${ABIS[abi].triple}${MIN_SDK}`, ...CFLAGS, '-fPIE', '-pie', + '-Wl,-z,max-page-size=16384', '-s', '-o', tmp, SOURCE, + ], { stdio: ['ignore', 'inherit', 'inherit'] }) + } catch { + fail(`building ${HELPER} for ${abi} failed (the compiler's message is above)`) + } + checkElf(tmp, ABIS[abi].elfMachine) + renameSync(tmp, staged) + console.log(`staged ${HELPER} for ${abi}: ${(statSync(staged).size / 1e3).toFixed(1)} KB`) + } +} + +// Run as a script; the test imports the constants above without staging anything. +if (process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))) { + try { + main() + } catch (e) { + fail(e instanceof Error ? e.message : String(e)) + } +} diff --git a/scripts/stage-tether-exec.test.mjs b/scripts/stage-tether-exec.test.mjs new file mode 100644 index 0000000..df8547a --- /dev/null +++ b/scripts/stage-tether-exec.test.mjs @@ -0,0 +1,236 @@ +// Tests for the engine tether, libtether_exec.so (engines/tether-exec/tether_exec.c), and its +// staging script: +// node --test scripts/stage-tether-exec.test.mjs +// The helper is built for this machine with `cc` and run on it, which needs Linux (prctl and +// /proc); without either, those tests are skipped. The staging test builds both shipped ABIs with +// the NDK when one is found (ANDROID_NDK_HOME, ANDROID_NDK_ROOT, or the pinned version under +// ANDROID_HOME or ANDROID_SDK_ROOT), and is skipped otherwise. Every process a test starts is +// killed by pid when it ends. +import assert from 'node:assert/strict' +import { spawn, spawnSync } from 'node:child_process' +import { chmodSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { after, before, test } from 'node:test' +import { fileURLToPath } from 'node:url' + +import { checkElf } from './stage-litert-lm-engine.mjs' +import { ABIS, CFLAGS, HELPER, SOURCE } from './stage-tether-exec.mjs' + +const script = fileURLToPath(new URL('./stage-tether-exec.mjs', import.meta.url)) +const work = mkdtempSync(join(tmpdir(), 'tether-exec-test-')) +after(() => rmSync(work, { recursive: true, force: true })) + +const cc = spawnSync('cc', ['--version'], { stdio: 'ignore' }) +const hostSkip = process.platform !== 'linux' ? 'PR_SET_PDEATHSIG and /proc are Linux only' + : cc.error ? 'no cc on this machine' : false +const helper = join(work, 'tether_exec') +before(() => { + if (hostSkip) return + const built = spawnSync('cc', [...CFLAGS, '-o', helper, SOURCE], { encoding: 'utf8' }) + assert.equal(built.status, 0, `cc failed:\n${built.stderr}`) +}) + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)) + +// /proc//stat after the command name: the state and the parent pid. null when the process is gone. +function stat(pid) { + try { + const s = readFileSync(`/proc/${pid}/stat`, 'utf8') + const f = s.slice(s.lastIndexOf(')') + 2).split(' ') + return { state: f[0], ppid: Number(f[1]) } + } catch { + return null + } +} +function cmdline(pid) { + try { + return readFileSync(`/proc/${pid}/cmdline`, 'utf8').split('\0').filter(Boolean) + } catch { + return null + } +} +// A zombie counts as gone: it runs nothing, and who reaps it (PID 1 or a subreaper) is not the helper's business. +const gone = (pid) => { + const s = stat(pid) + return !s || s.state === 'Z' +} +async function waitFor(check, ms) { + for (const end = Date.now() + ms; Date.now() < end; await sleep(20)) if (check()) return true + return check() +} +function killAll(pids) { + for (const pid of pids) { + try { + process.kill(pid, 'SIGKILL') + } catch {} + } +} +// 'exit' can come before the child's last output has been read from its pipes; 'close' comes once its stdio has ended +// too, so a test that checks what the child wrote waits for that. The self-test waits for 'exit': there the program +// the sh started in the background holds the sh's stdout, so 'close' would wait for the program. +function exited(child) { + return new Promise((r) => child.once('exit', (code, signal) => r({ code, signal }))) +} +function closed(child) { + return new Promise((r) => child.once('close', (code, signal) => r({ code, signal }))) +} +function firstLine(stream) { + return new Promise((resolve, reject) => { + let buf = '' + stream.setEncoding('utf8') + stream.on('data', (d) => { + buf += d + if (buf.includes('\n')) resolve(buf.slice(0, buf.indexOf('\n'))) + }) + stream.on('end', () => reject(new Error(`no line before the end (got ${JSON.stringify(buf)})`))) + }) +} + +test('a daemon killed with SIGKILL takes the engine it started through the helper with it', { skip: hostSkip }, async () => { + const pids = [] + try { + // The stand-in daemon starts `sleep 300` through the helper, as the app's daemon starts an engine, and prints its pid. + const daemon = spawn(process.execPath, ['-e', ` + const c = require('child_process').spawn(process.argv[1], [String(process.pid), 'sleep', '300'], { stdio: 'ignore' }) + console.log(c.pid); setInterval(() => {}, 1000)`, helper], { stdio: ['ignore', 'pipe', 'inherit'] }) + pids.push(daemon.pid) + const engine = Number(await firstLine(daemon.stdout)) + pids.push(engine) + assert.ok(await waitFor(() => cmdline(engine)?.join(' ') === 'sleep 300', 1500), 'the helper did not exec the engine in place') + assert.equal(stat(engine).ppid, daemon.pid) + process.kill(daemon.pid, 'SIGKILL') + assert.ok(await waitFor(() => gone(engine), 1000), 'the engine outlived the daemon') + } finally { + killAll(pids) + } +}) + +test('a parent that is not the daemon pid it was given exits 127 without running the program', { skip: hostSkip }, async () => { + const pids = [] + try { + const marker = join(work, 'ran') + // A live process that is not the helper's parent (this process is), as a helper whose daemon died before the prctl + // sees: its parent is then whoever adopted it. + const other = spawn('sleep', ['30'], { stdio: 'ignore' }) + pids.push(other.pid) + const child = spawn(helper, [String(other.pid), 'sh', '-c', `echo ran > '${marker}'`], { stdio: ['ignore', 'ignore', 'pipe'] }) + let err = '' + child.stderr.on('data', (d) => (err += d)) + assert.deepEqual(await closed(child), { code: 127, signal: null }) + assert.equal(existsSync(marker), false, 'the program ran') + assert.match(err, new RegExp(`the parent is ${process.pid}, not the daemon ${other.pid}`)) + } finally { + killAll(pids) + } +}) + +test('a daemon pid that is not a number above 1 exits 127 without running the program', { skip: hostSkip }, async () => { + const marker = join(work, 'ran-bad-pid') + for (const pid of ['1', '0', '-5', 'abc', '', `${process.pid}x`, '99999999999']) { + const r = spawnSync(helper, [pid, 'sh', '-c', `echo ran > '${marker}'`], { encoding: 'utf8' }) + assert.equal(r.status, 127, `daemon pid ${JSON.stringify(pid)}`) + assert.match(r.stderr, /bad daemon pid/) + } + assert.equal(existsSync(marker), false, 'the program ran') + assert.equal(spawnSync(helper, [String(process.pid)]).status, 127, 'no program') +}) + +test('the program keeps the helper pid, stdin, stdout and exit status', { skip: hostSkip }, async () => { + const child = spawn(helper, [String(process.pid), 'sh', '-c', 'echo "$$"; printf "a\\000b\\n"; cat; exit 7'], { stdio: 'pipe' }) + const out = [] + child.stdout.on('data', (d) => out.push(d)) + child.stdin.end('from stdin') + assert.deepEqual(await closed(child), { code: 7, signal: null }) + assert.deepEqual(Buffer.concat(out), Buffer.from(`${child.pid}\na\0b\nfrom stdin`)) + + const missing = spawnSync(helper, [String(process.pid), join(work, 'no-such-program')], { encoding: 'utf8' }) + assert.equal(missing.status, 127) + assert.match(missing.stderr, /cannot run .*no-such-program: No such file or directory/) +}) + +test('a signal to the helper pid reaches the program, and its status passes through', { skip: hostSkip }, async () => { + const pids = [] + try { + const child = spawn(helper, [String(process.pid), 'sleep', '300'], { stdio: 'ignore' }) + pids.push(child.pid) + assert.ok(await waitFor(() => cmdline(child.pid)?.join(' ') === 'sleep 300', 1500)) + const status = exited(child) + child.kill('SIGTERM') + assert.deepEqual(await status, { code: null, signal: 'SIGTERM' }) + } finally { + killAll(pids) + } +}) + +// The daemon's boot self-test (TurboLLM's util/tether.ts; BUILDING-ENGINES.md, "The engine tether"), here with this +// machine's sh and sleep: the program must be seen running under the helper, exec'd in place, while the helper's +// parent sh lives, and then be gone within 1 s of that sh exiting. +async function selfTest(sh, helperPath) { + const pids = [] + try { + const parent = spawn(sh, ['-c', '"$0" "$$" sleep 30 & echo "$!"; read -r _', helperPath], { stdio: ['pipe', 'pipe', 'ignore'] }) + pids.push(parent.pid) + const program = Number(await firstLine(parent.stdout)) + pids.push(program) + const ran = await waitFor(() => { + const s = stat(program) + return s && s.state !== 'Z' && s.ppid === parent.pid && cmdline(program)?.join(' ') === 'sleep 30' + }, 1500) + if (!ran) return 'never ran' + const done = exited(parent) + parent.stdin.end() + await done + return (await waitFor(() => gone(program), 1000)) ? 'passed' : 'outlived its parent' + } finally { + killAll(pids) + } +} + +test('the helper passes the self-test the daemon runs at boot, and helpers that do not work fail it', { skip: hostSkip }, async () => { + const shells = ['/bin/sh', ...['bash', 'mksh'].filter((s) => !spawnSync(s, ['-c', 'exit 0']).error)] + // Stand-ins for a broken build: one that execs without arming the signal, and one that never runs the program. + const noSignal = join(work, 'no-signal.sh') + writeFileSync(noSignal, '#!/bin/sh\nshift\nexec "$@"\n') + chmodSync(noSignal, 0o755) + const exits127 = join(work, 'exits-127.sh') + writeFileSync(exits127, '#!/bin/sh\nexit 127\n') + chmodSync(exits127, 0o755) + for (const sh of shells) { + assert.equal(await selfTest(sh, helper), 'passed', sh) + assert.equal(await selfTest(sh, noSignal), 'outlived its parent', `${sh}, a helper without the signal`) + assert.equal(await selfTest(sh, exits127), 'never ran', `${sh}, a helper that exits 127`) + } +}) + +function findNdk() { + const pinned = (sdk) => sdk && join(sdk, 'ndk', '27.0.12077973') + const env = process.env + return [env.ANDROID_NDK_HOME, env.ANDROID_NDK_ROOT, pinned(env.ANDROID_HOME), pinned(env.ANDROID_SDK_ROOT)] + .find((d) => d && existsSync(join(d, 'toolchains', 'llvm', 'prebuilt'))) +} +const ndk = findNdk() + +test('the staging script builds a 16 KB-aligned PIE executable for each shipped ABI', { skip: !ndk && 'no NDK found' }, () => { + const out = join(work, 'jniLibs') + const r = spawnSync(process.execPath, [script, '--ndk', ndk, '--out', out], { encoding: 'utf8' }) + assert.equal(r.status, 0, r.stderr) + for (const [abi, { elfMachine }] of Object.entries(ABIS)) { + const f = join(out, abi, HELPER) + checkElf(f, elfMachine) // exits the process, failing this file, if the architecture or the alignment is wrong + const b = readFileSync(f) + assert.equal(b.readUInt16LE(16), 3, `${abi}: ET_DYN, a PIE`) + assert.equal(b.readUInt16LE(18), elfMachine, abi) + const phoff = Number(b.readBigUInt64LE(32)) + const interp = [...Array(b.readUInt16LE(56)).keys()].map((i) => phoff + i * b.readUInt16LE(54)) + .find((o) => b.readUInt32LE(o) === 3) // PT_INTERP + assert.ok(interp !== undefined, `${abi}: no PT_INTERP, so not an executable`) + const off = Number(b.readBigUInt64LE(interp + 8)) + assert.equal(b.toString('latin1', off, b.indexOf(0, off)), '/system/bin/linker64', abi) + assert.equal(existsSync(`${f}.tmp`), false) + } + + const bad = spawnSync(process.execPath, [script, '--ndk', ndk, '--out', out, '--abis', 'armeabi-v7a'], { encoding: 'utf8' }) + assert.equal(bad.status, 1) + assert.match(bad.stderr, /no tether build for ABI armeabi-v7a/) +}) diff --git a/scripts/webview-height-fix.test.mjs b/scripts/webview-height-fix.test.mjs new file mode 100644 index 0000000..d33d6e1 --- /dev/null +++ b/scripts/webview-height-fix.test.mjs @@ -0,0 +1,243 @@ +// Tests for HEIGHT_FIX_JS, the script the app's WebView runs after each page load +// (composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonWebView.kt): +// node --test scripts/webview-height-fix.test.mjs +// The script is read out of DaemonWebView.kt and turned into the string Kotlin builds from it (its ${'$'} is a $), so +// this runs what ships. It is run in Chromium on a page shaped like the web UI's Tailwind output, at the phone size +// the bug was reported at, under two hosts: a WebView that lays the page out at its own height (what MATCH_PARENT +// layout params give), and one that lays it out at height 0 (what WRAP_CONTENT gave: Chromium's +// --blink-settings=forceZeroLayoutHeight=true is the same Blink setting). Both are compared with the same page in +// plain Chromium with no script. Needs Playwright (require('playwright'), so NODE_PATH works) and the Chromium it +// launches, or the one CHROME_PATH names; those tests are skipped without them. +import assert from 'node:assert/strict' +import { mkdtempSync, readFileSync, rmSync } from 'node:fs' +import { createRequire } from 'node:module' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { test } from 'node:test' + +const KT = new URL('../composeApp/src/androidMain/kotlin/com/turbollm/app/DaemonWebView.kt', import.meta.url) + +// HEIGHT_FIX_JS as the app evaluates it. A `$` in a Kotlin raw string starts a template, so the only one allowed is +// the ${'$'} that stands for a literal $. +function heightFixJs() { + const m = /private const val HEIGHT_FIX_JS = """([\s\S]*?)"""/.exec(readFileSync(KT, 'utf8')) + assert.ok(m, 'HEIGHT_FIX_JS not found in DaemonWebView.kt') + const js = m[1].replaceAll("${'$'}", '$') + const template = /\$[\w{]/.exec(js) + assert.equal(template, null, `Kotlin would fill in a template in HEIGHT_FIX_JS: ${template && js.slice(template.index, template.index + 20)}`) + return js +} + +test('HEIGHT_FIX_JS is plain JavaScript once Kotlin has built it', () => { + assert.doesNotThrow(() => new Function(heightFixJs())) +}) + +// The fix itself, which no browser here can run: Chromium lays out the page of a WebView whose layout params are +// WRAP_CONTENT high at height 0, and AndroidView's holder adds a view that has none with WRAP_CONTENT, as soon as the +// factory returns. So the one WebView the factory builds (on first load and after a renderer crash) is given +// MATCH_PARENT both ways first. +test('the factory gives the WebView MATCH_PARENT layout params before AndroidView adds it', () => { + const kt = readFileSync(KT, 'utf8') + assert.equal(kt.match(/WebView\(context\)/g)?.length, 1, 'one place builds the WebView') + const first = /WebView\(context\)\.apply \{\s*(?:\/\/.*\s*)*layoutParams = ViewGroup\.LayoutParams\(\s*ViewGroup\.LayoutParams\.MATCH_PARENT,\s*ViewGroup\.LayoutParams\.MATCH_PARENT,?\s*\)/ + assert.ok(first.test(kt), 'the first statement in WebView(context).apply { } sets MATCH_PARENT layout params') +}) + +// Playwright makes its folders in the temp folder and leaves them there when Chromium does not start (the skip +// below), so this file's go in a temp folder of their own, removed when its process exits. Not in an after() hook: +// node:test runs those once the tests above are done, while the skip check below may still be launching Chromium. +const ownTmp = mkdtempSync(join(tmpdir(), 'webview-height-fix-')) +for (const k of ['TMPDIR', 'TMP', 'TEMP']) process.env[k] = ownTmp +process.on('exit', () => rmSync(ownTmp, { recursive: true, force: true })) + +let chromium = null +try { + ;({ chromium } = createRequire(import.meta.url)('playwright')) +} catch {} +const launchOptions = (zero) => ({ + executablePath: process.env.CHROME_PATH || undefined, + args: zero ? ['--blink-settings=forceZeroLayoutHeight=true'] : [], +}) +const skip = await (async () => { + if (!chromium) return 'Playwright is not installed' + try { + await (await chromium.launch(launchOptions(false))).close() + return false + } catch (e) { + return `Chromium does not start: ${String(e.message).split('\n')[0]}` + } +})() + +// The web UI's shapes, with the CSS Tailwind v4 (4.3.3) writes for them, in its utilities layer: html/body/#root at +// 100% (index.css), the model picker (ModelLoadMenu.tsx), arbitrary values with vh inside calc() and clamp(), calc() +// spelt with '_' for a space and without (Tailwind then spaces the operators out itself, so the class name is not the +// CSS), a vh term after a percentage, Tailwind's '!' (important), vmin, a custom property whose name has digits and +// "vh" in it, and the named utilities. One class's rule is in a stylesheet from another origin, which a page script +// cannot read. `pad` is the 4px padding and 1px border of the picker's menu, which is all that was left of it at +// height 0. +const OTHER_ORIGIN_CSS = 'http://other-origin.test/utilities.css' +const PAGE = `
+
+
+
+
+
+
+
+
+
+
+
+
+
+
` +const OTHER_ORIGIN_RULES = '@layer utilities { .h-\\[calc\\(100vh_-_3rem\\)\\] { height: calc(100vh - 3rem) } }' + +const IDS = ['root', 'picker', 'h45', 'calc', 'calcSpaced', 'percent', 'important', 'unreadable', 'clamp', 'vmin', + 'varname', 'shell', 'screen', 'portal', 'reclassed'] + +// Loads PAGE in a 412x925 phone, runs HEIGHT_FIX_JS twice (onPageFinished can come twice for one page) unless js is +// null, then does what the UI does later: a Radix portal with a vh class is added, and an existing element is given +// one (React reusing an element with a new className). Heights are measured then, after the class is taken away +// again, and after a keyboard-like resize to 412x600. With `unsized`, HEIGHT_FIX_JS runs as on a page that finished +// loading before the WebView was laid out: window.innerHeight reads 0 while it runs, and then the window gets its +// height (a resize event). `signal` is the test's: a test that times out leaves this waiting where it was (a +// page.evaluate whose promise never settles), so its finally never closes Chromium, and an open Chromium keeps +// `node --test` running. Closing it when the signal aborts ends that wait, so the run ends, the test failed by name. +async function run(zero, js, { unsized = false, signal } = {}) { + const browser = await chromium.launch(launchOptions(zero)) + const close = () => { browser.close().catch(() => {}) } + signal?.addEventListener('abort', close, { once: true }) + if (signal?.aborted) close() + try { + const page = await (await browser.newContext({ + viewport: { width: 412, height: 925 }, deviceScaleFactor: 2.625, isMobile: true, hasTouch: true, + })).newPage() + const warnings = [] + page.on('console', (m) => { if (m.type() === 'warning') warnings.push(m.text()) }) + await page.route(OTHER_ORIGIN_CSS, (route) => route.fulfill({ contentType: 'text/css', body: OTHER_ORIGIN_RULES })) + await page.setContent(PAGE, { waitUntil: 'load' }) + const vh100 = await page.evaluate(() => { + const d = document.createElement('div') + d.style.height = '100vh' + document.body.appendChild(d) + const h = d.getBoundingClientRect().height + d.remove() + return h + }) + if (unsized) { + await page.evaluate(() => { + window.realInnerHeight = Object.getOwnPropertyDescriptor(window, 'innerHeight') + Object.defineProperty(window, 'innerHeight', { get: () => 0, configurable: true, enumerable: true }) + }) + } + const first = js === null ? undefined : await page.evaluate(js) + const second = js === null ? undefined : await page.evaluate(js) + if (unsized) { + await page.evaluate(() => { + Object.defineProperty(window, 'innerHeight', window.realInnerHeight) + window.dispatchEvent(new Event('resize')) + }) + } + const decided = await page.evaluate(() => window.__turbollmHeightFix) + await page.evaluate(() => { + const portal = document.createElement('div') + portal.innerHTML = '
' + document.body.appendChild(portal) + document.getElementById('reclassed').className = 'max-h-[60vh] pad' + }) + const heights = () => page.evaluate((ids) => Object.fromEntries(ids.map((id) => + [id, document.getElementById(id).getBoundingClientRect().height])), IDS) + const at925 = await heights() + await page.evaluate(() => { document.getElementById('reclassed').className = 'pad' }) + const unclassed = (await heights()).reclassed + await page.setViewportSize({ width: 412, height: 600 }) + await page.waitForFunction(() => window.innerHeight === 600) + await page.evaluate(() => new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r)))) + const at600 = await heights() + const important = await page.evaluate(() => document.querySelectorAll('[style*="important"]').length) + const styles = await page.evaluate(() => document.querySelectorAll('style').length) + return { vh100, first, second, decided, warnings, at925, at600, unclassed, important, styles } + } finally { + signal?.removeEventListener('abort', close) + await browser.close() + } +} + +function sameHeights(actual, expected, what) { + for (const id of IDS) { + assert.ok(Math.abs(actual[id] - expected[id]) <= 0.5, `${what}: #${id} is ${actual[id]}px, ${expected[id]}px in plain Chromium`) + } +} + +let reference = null +const plain = async (signal) => (reference ??= await run(false, null, { signal })) + +const slow = { skip, timeout: 120_000 } + +test('in a WebView that lays the page out at its own height, HEIGHT_FIX_JS answers false and changes nothing', slow, async (t) => { + const ref = await plain(t.signal) + const r = await run(false, heightFixJs(), { signal: t.signal }) + assert.equal(r.vh100, 925) + assert.equal(r.first, false, 'the first run answers false: the probe found 100vh at its real height') + assert.equal(r.second, null, 'a second run on the same page does nothing') + assert.equal(r.important, 0, 'no inline style was set') + assert.equal(r.styles, ref.styles, 'no stylesheet was added') + sameHeights(r.at925, ref.at925, '412x925') + sameHeights(r.at600, ref.at600, '412x600') + assert.equal(r.unclassed, ref.unclassed) +}) + +test('in a WebView that lays the page out at height 0, HEIGHT_FIX_JS answers true and every height is as in Chromium', slow, async (t) => { + const ref = await plain(t.signal) + const r = await run(true, heightFixJs(), { signal: t.signal }) + assert.equal(r.vh100, 0, 'the host lays the page out at height 0 (100vh measures 0 before HEIGHT_FIX_JS runs)') + assert.equal(r.first, true, 'the first run answers true, which onPageFinished logs') + assert.equal(r.second, null, 'a second run on the same page does nothing') + assert.equal(r.styles, ref.styles + 1, 'one stylesheet for the named utilities, however many times it ran') + // The model picker's menu was 10px (its padding and border) with max-height min(0px, …). + assert.equal(Math.round(r.at925.picker), 555) + sameHeights(r.at925, ref.at925, '412x925') + sameHeights(r.at600, ref.at600, '412x600, after a resize') + assert.equal(r.unclassed, ref.unclassed, 'a class taken away takes its height with it') +}) + +test('on a page that loaded before the WebView had a height, HEIGHT_FIX_JS decides at the first resize that gives it one', slow, async (t) => { + const ref = await plain(t.signal) + const sized = await run(false, heightFixJs(), { unsized: true, signal: t.signal }) + assert.equal(sized.first, null, 'no answer while the window has no height') + assert.equal(sized.second, null) + assert.equal(sized.decided, false, 'the probe ran once the window had its height, and found 100vh real') + assert.equal(sized.important, 0) + assert.equal(sized.styles, ref.styles) + assert.deepEqual(sized.warnings, []) + sameHeights(sized.at925, ref.at925, 'own height, 412x925') + + const zero = await run(true, heightFixJs(), { unsized: true, signal: t.signal }) + assert.equal(zero.first, null, 'no answer while the window has no height') + assert.equal(zero.second, null) + assert.equal(zero.decided, true, 'the probe ran once the window had its height, and found 100vh at 0px') + assert.equal(zero.warnings.length, 1, 'onPageFinished has had its answer, so the page logs the warning itself') + assert.match(zero.warnings[0], /^100vh is 0px in this WebView/) + assert.equal(Math.round(zero.at925.picker), 555) + sameHeights(zero.at925, ref.at925, 'height 0, 412x925') + sameHeights(zero.at600, ref.at600, 'height 0, 412x600, after a resize') +})