diff --git a/BUILDING-ENGINES.md b/BUILDING-ENGINES.md index 79dad28..1ddcccc 100644 --- a/BUILDING-ENGINES.md +++ b/BUILDING-ENGINES.md @@ -394,7 +394,9 @@ the request sets a field, as before: LiteRT-LM leaves sampling to the model ther or below means no top-k limit, as in llama.cpp (TurboLLM's Top K 0 means disabled), and one above 0 is passed as set, on GPU too, where LiteRT-LM 0.18.0 sets no limit of its own; a `top_p` outside 0 to 1, a negative `temperature`, and a `top_k` or `seed` with a fraction are a 400 that -names the field instead of the runtime's 500. +names the field instead of the runtime's 500. LiteRT-LM takes these values and the repetition +penalties below as floats, so a `temperature` or penalty beyond a float's range (3.4028235e+38 +either way), which reached it as an infinity, is a 400 that names the field too. **Thinking is asked for the ways clients ask for it.** In order, the first that says: the top-level `reasoning_effort` (what the Python server reads, and what the daemon sends for a @@ -458,7 +460,8 @@ puts gpu first), as `litert-lm serve` picks it. The server picked CPU for every GPU-only bundle sent without a backend failed to load. When the model's own backend does not start, the engine falls back to the next one the model lists, CPU for most, and never to one it does not list, and the server logs why; when none starts, the error gives each one's reason. A -backend the user set is used as set, failing as before. +backend the user set is used as set, failing as before, also when an engine whose own backend fell +back is loaded: an engine is kept for the backend it runs on. **A conversation that ends with the assistant's own words is a 400.** A client that prefills the start of a reply asks for it to be continued, which LiteRT-LM 0.18.0 can do only from C++ @@ -477,10 +480,14 @@ AddressSanitizer caught an exception being destroyed by the other runtime. See ends, and it exits 1 without starting when its parent is already init (the starter died before the `prctl`). The daemon's [engine tether](#the-engine-tether) does the same before it execs the server; this covers a server started any other way: by a daemon without the tether, through the -daemon's `/system/bin/sh` fallback, or by hand. `node --test -engines/litert-lm-server/parent-death.test.mjs` builds the server for the machine (Linux, `cc` -and `c++`) against a stand-in `liblitert-lm.so` whose functions do nothing, and checks that a -stand-in daemon's SIGKILL takes the listening server with it and that an orphaned start exits. +daemon's `/system/bin/sh` fallback, or by hand. Off Android, a parent that is PID 1 of a +container's PID namespace, its entrypoint (tini, a shell), is taken for the server's starter, as +it usually is: the kernel keeps no record that would tell an orphan that PID 1 adopted from its +child. `node --test engines/litert-lm-server/parent-death.test.mjs` builds the server for the +machine (Linux, `cc` and `c++`) against a stand-in `liblitert-lm.so` whose functions do nothing, +and checks that a stand-in daemon's SIGKILL takes the listening server with it, that an orphaned +start exits (outside a container's PID namespace, where that test is skipped), and, where `unshare` +can make a PID namespace, that a server its PID 1 starts runs. ### GPU diff --git a/PLAY-FGS-DECLARATION.md b/PLAY-FGS-DECLARATION.md index c57bdb7..4d09ac6 100644 --- a/PLAY-FGS-DECLARATION.md +++ b/PLAY-FGS-DECLARATION.md @@ -40,9 +40,11 @@ in the foreground, also while idle, until the user quits or closes the app with rather than suggesting it runs only during generation. LAN access, remote access and Turbo Link are read from the daemon the app bundles: the `serving` field of its GET -/api/v1/status, from TurboLLM's feat(api): /api/v1/status says which server features are on. A daemon without that -field reports them all off (the app reads a missing or malformed `serving` as nothing served), so a close with only -one of them on would quit: the text below holds only for an APK built with a TurboLLM that has it. +/api/v1/status, from TurboLLM's feat(api): /api/v1/status says which server features are on, or the same field in the +`status` its GET /api/v1/activity carries too, from TurboLLM's fix(api): one activity request carries what the +Android poll reads. A daemon without that field reports them all off (the app reads a missing or malformed `serving` +as nothing served), so a close with only one of them on would quit: the text below holds only for an APK built with a +TurboLLM that has it. A close made while the app is still starting, before its daemon answers (a swipe from Recents during the splash screen, and before Android 12 Back there too), waits for the daemon, for up to 30 seconds, and is then decided as the diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt index dda5027..f3f91eb 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt @@ -26,8 +26,14 @@ internal data class DownloadNotice(val title: String, val text: String, val prog /** The running and queued rows of a GET /api/v1/downloads reply, or null when the reply is not the list the daemon * sends (the poll then counts as failed, and as idle, as before). A row with a field missing or of another type * reads that field as 0 instead of failing the whole list. */ -internal fun parseLiveDownloads(body: String): List? { - val list = runCatching { JSONObject(body).optJSONArray("downloads") }.getOrNull() ?: return null +internal fun parseLiveDownloads(body: String): List? = + runCatching { JSONObject(body) }.getOrNull()?.let(::liveDownloads) + +/** The running and queued rows of [json]'s `downloads`, as parseLiveDownloads reads them: the list GET + * /api/v1/downloads answers, and the one a GET /api/v1/activity reply carries too from a daemon that sends it there + * (TurboLLM's active-work.ts, its running and queued rows only). Null when [json] has no such list. */ +internal fun liveDownloads(json: JSONObject): List? { + val list = json.optJSONArray("downloads") ?: return null return (0 until list.length()).mapNotNull { i -> val row = list.optJSONObject(i) ?: return@mapNotNull null val status = row.optString("status") diff --git a/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt b/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt index d5b7e94..585e515 100644 --- a/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt +++ b/composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt @@ -341,13 +341,15 @@ class EngineService : Service() { /** Whether anything runs, asked of the daemon now (see appClosed): a model import, work in progress or a reply * waiting for a tool approval (GET /api/v1/activity's items, engineGenerating or waiting), a model download * running or queued (a paused or failed one does not count), or a model loaded or loading or a server feature on - * (queryStatus). A query that fails, or a reply it does not recognise, counts as something running. */ + * (queryStatus). The downloads and the status come from the same activity reply when it carries them + * (ActivityReply), else from their own routes. A query that fails, or a reply it does not recognise, counts as + * something running. */ private fun anythingRuns(): Boolean { if (ModelImport.isRunning()) return true - val activity = queryActivity() ?: return true - if (activity.work != null || activity.waiting) return true - if (queryDownloads()?.isEmpty() != true) return true - return queryStatus()?.runs != false + val reply = queryActivity() ?: return true + if (reply.activity.work != null || reply.activity.waiting) return true + if ((reply.downloads ?: queryDownloads())?.isEmpty() != true) return true + return (reply.status ?: queryStatus())?.runs != false } /** Ask the daemon to write its chat database to disk now (POST /api/v1/daemon/flush), waiting at most @@ -382,10 +384,15 @@ class EngineService : Service() { while (running) { // A model import counts as work too: the service must neither stop nor let the CPU sleep under the copy. val importing = ModelImport.isRunning() - // So does a model download (Discover): a stopped service or a sleeping CPU would stall a multi-GB transfer. - val downloads = queryDownloads() + // One request a tick, two loopback connections (DaemonAuth's proof and the request), to a daemon whose + // activity reply carries the downloads and the status too (ActivityReply); an older daemon's are asked of + // their own routes, as before, each with a proof of its own: four connections a tick, six while idle. + val reply = queryActivity() + val activity = reply?.activity + // A model download counts too (Discover): a stopped service or a sleeping CPU would stall a multi-GB + // transfer. + val downloads = reply?.downloads ?: queryDownloads() val downloading = !downloads.isNullOrEmpty() - val activity = queryActivity() // A poll that could not read the activity keeps its last work or approval line up for up to // ACTIVITY_MISSES polls in a row, as the download lines do below: the daemon can be slow to answer for a // moment mid-reply, and the line read "Idle" for that poll, while a close then counts the unreadable @@ -402,7 +409,11 @@ class EngineService : Service() { val nowBusy = activity?.work != null || importing || downloading // The loaded model's name, for the line shown while nothing runs: asked only then, and kept from the last // reading when the status cannot be read, so a slow answer does not drop the name for 3 s. - val status = if (work == null && !importing && !downloading && !waiting) queryStatus() else null + val status = if (work == null && !importing && !downloading && !waiting) { + reply?.status ?: queryStatus() + } else { + null + } val model = if (status != null) status.name else this.model // The query can take seconds. Under the lock, so the service being destroyed meanwhile is seen before the // wake lock is re-taken or a removed notification is posted again. @@ -449,10 +460,10 @@ class EngineService : Service() { } } - /** What the daemon is doing (GET /api/v1/activity, see DaemonActivity), or null when it could not be read: the poll + /** What the daemon is doing (GET /api/v1/activity, see ActivityReply), or null when it could not be read: the poll * reads that as idle for the wake lock and keeps its last line for a few polls, a close (anythingRuns) counts it * as something running. */ - private fun queryActivity(): DaemonActivity? = try { + private fun queryActivity(): ActivityReply? = try { val conn = DaemonAuth.open("/api/v1/activity") conn.connectTimeout = 1000 conn.readTimeout = 2000 @@ -465,7 +476,10 @@ class EngineService : Service() { // daemon's: unreadable, so a close does not take it for nothing running. JSONObject(conn.inputStream.bufferedReader().use { it.readText() }) .takeIf { it.optJSONArray("items") != null } - ?.let(::daemonActivity) + ?.let { json -> + val status = json.optJSONObject("status")?.let(::statusReading) + ActivityReply(daemonActivity(json), liveDownloads(json), status) + } } } finally { conn.disconnect() @@ -618,6 +632,17 @@ class EngineService : Service() { * (the daemon lists those chats under `waiting`, not `items`). */ private class DaemonActivity(val work: String?, val waiting: Boolean) +/** A GET /api/v1/activity reply: what the daemon is doing, and, from a daemon that sends them there too (TurboLLM's + * active-work.ts), its running and queued downloads ([downloads], as GET /api/v1/downloads gives them) and the fields + * of GET /api/v1/status the service reads ([status], in that route's names and shapes), so that one request a poll + * tells all three. Each is null when the reply does not carry it as that route would answer (an older daemon, or a + * field this does not recognise), and is then asked of its own route, as before. */ +private class ActivityReply( + val activity: DaemonActivity, + val downloads: List?, + val status: StatusReading?, +) + /** What GET /api/v1/status says. [runs]: something runs there, a model held or a server feature on (statusReading). * [name]: the model the notification names while nothing runs, null when none is loaded or the loaded one is only * an embedding model in a pool slot, which the status does not name. */ diff --git a/engines/litert-lm-server/model-settings.test.mjs b/engines/litert-lm-server/model-settings.test.mjs index 4f75b63..e3ea176 100644 --- a/engines/litert-lm-server/model-settings.test.mjs +++ b/engines/litert-lm-server/model-settings.test.mjs @@ -16,14 +16,14 @@ // Every server a test starts is killed by pid when that test ends, or at the latest when the tests end. Every wait is // bounded. import assert from 'node:assert/strict' -import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { mkdtempSync, readFileSync, rmSync, statSync, utimesSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' import { basename, join } from 'node:path' import { deflateSync } from 'node:zlib' import { after, before, test } from 'node:test' import { buildServer, calls, chat, fb, flatbuffer, killServers, litertlmFile, logged, proto, realLibrary, realModels, request, - skip, skipReal, startServer, stopServer, + skip, skipReal, sleep, startServer, stopServer, } from './testdata/harness.mjs' const visionModels = (process.env.LITERT_LM_TEST_VISION_MODELS ?? '').split(',').filter(Boolean) @@ -254,6 +254,54 @@ test('an image or audio part sent to a model without that input is a 400 that sa assert.equal((await chat(fake, { model: unreadable('unreadable'), messages: imageMessage('What color is this?') })).status, 200) }) +test('a model file is read once while it is unchanged, and again once it changes, in the same second at the same size or with its time put back', { skip, timeout: 60_000 }, async () => { + const path = join(work, 'rewritten.litertlm') + const reads = async (name) => (await logged(fake)).filter((l) => l.startsWith('Reading the header of ') && l.endsWith(`/${name}`)).length + // Takes no image, until it is rewritten in place (the same inode), at the same size (a section each), its + // modification time moved within the same second: then it has a vision encoder. + litertlmFile(path, [decoder, tflite('tf_lite_embedder')]) + const second = Math.floor(Date.now() / 1000) - 60 + utimesSync(path, second + 0.25, second + 0.25) + const before = statSync(path) + assert.deepEqual(await chat(fake, { model: path, messages: imageMessage('What is this?') }), { status: 400, error: 'Invalid messages: rewritten.litertlm takes no image input' }) + for (let i = 0; i < 3; i++) assert.equal((await chat(fake, { model: path, messages: hello })).status, 200) + assert.equal(await reads('rewritten.litertlm'), 1) + litertlmFile(path, [decoder, tflite('tf_lite_vision_encoder')]) + utimesSync(path, second + 0.75, second + 0.75) + const after = statSync(path) + assert.deepEqual([after.ino, after.size, Math.floor(after.mtimeMs / 1000)], [before.ino, before.size, second]) + assert.equal((await chat(fake, { model: path, messages: imageMessage('What is this?') })).status, 200) + assert.equal((await chat(fake, { model: path, messages: hello })).status, 200) + assert.equal(await reads('rewritten.litertlm'), 2) + // Rewritten again, as the file without the encoder, and given back the modification time it had (as cp -p, touch -r + // and tar do): the same inode, size and modification time to the nanosecond, its change time moved. + const was = statSync(path, { bigint: true }) + await sleep(20) + litertlmFile(path, [decoder, tflite('tf_lite_embedder')]) + utimesSync(path, second + 0.75, second + 0.75) + const now = statSync(path, { bigint: true }) + assert.deepEqual([now.ino, now.size, now.mtimeNs], [was.ino, was.size, was.mtimeNs]) + assert.notEqual(now.ctimeNs, was.ctimeNs) + assert.deepEqual(await chat(fake, { model: path, messages: imageMessage('What is this?') }), { status: 400, error: 'Invalid messages: rewritten.litertlm takes no image input' }) + assert.equal((await chat(fake, { model: path, messages: hello })).status, 200) + assert.equal(await reads('rewritten.litertlm'), 3) + // A header end the runtime cannot load is refused for every request, read once, and the engine loaded stays. + const bad = join(work, 'bad-offset.litertlm') + const preamble = Buffer.alloc(32) + preamble.write('LITERTLM') + preamble.writeBigUInt64LE(1n << 40n, 24) + writeFileSync(bad, Buffer.concat([preamble, Buffer.alloc(16384)])) + const log = await callsDuring(async () => { + for (let i = 0; i < 2; i++) { + assert.deepEqual(await chat(fake, { model: bad, messages: hello }), { status: 500, + error: `Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for ${bad}: invalid .litertlm header')` }) + } + assert.equal((await chat(fake, { model: path, messages: hello })).status, 200) + }) + assert.deepEqual(log.filter((c) => c.call === 'engine_create'), []) + assert.equal(await reads('bad-offset.litertlm'), 1) +}) + // The server's peak resident memory, from /proc (Linux, as the tests are). const peakMb = (server) => Number(/^VmHWM:\s+(\d+) kB/m.exec(readFileSync(`/proc/${server.child.pid}/status`, 'utf8'))[1]) / 1024 @@ -701,6 +749,35 @@ test('repetition penalties reach the model with llama.cpp\'s and OpenAI\'s meani assert.equal((await logged(fake)).filter((l) => l === 'min_p is not applied to plain.litertlm: LiteRT-LM has no min-p sampling').length, 1, fake.stderr) }) +test('a sampler or penalty value a float cannot hold is a 400 that names it, and one it can is passed as set', { skip, timeout: 60_000 }, async () => { + const model = readable('plain') + const range = (field, value) => `Invalid sampler parameters: ValueError: ${field} must be within a float's range (-3.4028235e+38 to 3.4028235e+38), got ${value}` + for (const [request, error] of [ + [{ temperature: 1e39 }, range('temperature', '1e+39')], + [{ repetition_penalty: 1e39 }, range('repetition_penalty', '1e+39')], + [{ repeat_penalty: 1e39 }, range('repeat_penalty', '1e+39')], + [{ presence_penalty: 1e39 }, range('presence_penalty', '1e+39')], + [{ presence_penalty: -1e39 }, range('presence_penalty', '-1e+39')], + [{ frequency_penalty: 1e39 }, range('frequency_penalty', '1e+39')], + [{ frequency_penalty: -1e39 }, range('frequency_penalty', '-1e+39')], + // Past FLT_MAX by more than half a float's step, so it rounds to infinity (3.4028235e38, below, is past it by less + // and rounds to FLT_MAX). + [{ temperature: 3.4028236e38 }, range('temperature', '3.4028236e+38')], + ]) { + for (const stream of [false, true]) { + assert.deepEqual(await chat(fake, { model, messages: hello, ...request, stream }), { status: 400, error }, JSON.stringify(request)) + } + } + // Within a float's range, as set: the sampler and the penalties get it. + const log = await callsDuring(async () => { + const reply = await chat(fake, { model, messages: hello, temperature: 3e38, repetition_penalty: 3e38, presence_penalty: -3e38, frequency_penalty: 3.4028235e38 }) + assert.equal(reply.status, 200, reply.error) + }) + assert.equal(log.filter((c) => c.call === 'conversation_create').at(-1).config.sampler.temperature, f(3e38)) + assert.deepEqual(log.filter((c) => c.call === 'send_message').at(-1).args.repetition_penalty_config, + { repetition_penalty: f(3e38), presence_penalty: f(-3e38), frequency_penalty: f(3.4028235e38), window_size: 0 }) +}) + test('min_p is said once per model, by a request that is answered, also when requests with it arrive at once', { skip, timeout: 60_000 }, async () => { // Each connection has its own thread. The note is taken under the engine lock, so requests refused before it (here // for their response_format) never touch it, and answered ones take turns. @@ -881,6 +958,36 @@ test('a prompt that does not fit the context window is a 400 that says the windo '(INVALID_ARGUMENT: Input token ids are too long. Exceeding the maximum number of tokens allowed: 4209 >= 4096)' }) }) +test('model output nested deeper than the server reads fails its request, streamed or not, and the server stays up', { skip, timeout: 60_000 }, async () => { + const model = readable('plain') + const nested = (n, stream) => chat(fake, { model, messages: [{ role: 'user', content: `nested:${n}` }], stream }, 30_000) + // A tool call's arguments nested as deep as a request's may be, 128 levels ({"a": ...} and 127 arrays), are read as + // before; so are 125, which LiteRT-LM 0.18.0 itself takes from a model. + for (const n of [100, 125, 127]) { + const reply = await nested(n, false) + assert.equal(reply.status, 200, `${n}: ${reply.error}`) + assert.equal(reply.finish, 'tool_calls') + assert.deepEqual(JSON.parse(reply.tool_calls[0].function.arguments), { a: JSON.parse('['.repeat(n) + ']'.repeat(n)) }) + } + // One level more is refused. Copied or written out a level at a time, 200,000 levels ran the connection thread out of + // stack. + const error = 'RuntimeError: the model\'s output, as the runtime parsed it, is nested deeper than 132 levels ' + + '(128 within a tool call\'s arguments)' + assert.deepEqual(await nested(128, false), { status: 500, error }) + assert.deepEqual(await nested(200_000, false), { status: 500, error }) + const res = await request(`http://127.0.0.1:${fake.port}/v1/chat/completions`, { + method: 'POST', headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ model, messages: [{ role: 'user', content: 'nested:200000' }], stream: true }), + }, 30_000) + assert.equal(res.status, 200) + assert.ok(res.text.includes(`event: response.error\ndata: ${JSON.stringify({ error })}\n\n`), res.text) + assert.ok(res.text.endsWith('data: [DONE]\n\n'), res.text) + // A chunk that is no JSON is the reply's text, as always, however many brackets it opens. + const text = await chat(fake, { model, messages: [{ role: 'user', content: 'nested-text:200' }] }) + assert.deepEqual([text.status, text.content], [200, `${'['.repeat(200)} not json`]) + assert.equal((await chat(fake, { model, messages: hello })).status, 200) +}) + for (const path of realModels.length ? realModels : ['']) { test(`a real model's context window is its own when more is asked, and a reply that fills it ends with "length": ${path}`, { skip: skipReal, timeout: 900_000 }, async () => { const config = join(work, 'no-cache.json') @@ -973,6 +1080,52 @@ test('with no backend set, the model\'s own is used, and one that does not start assert.deepEqual(await engines(recurrent), { status: 200, error: undefined, created: ['cpu'] }) }) +test('a backend the request sets after the model\'s own fell back gets an engine on it or its error, never the other', { skip, timeout: 60_000 }, async () => { + const engines = async (ref) => { + let reply + const log = await callsDuring(async () => { reply = await chat(fake, { model: ref, messages: hello }) }) + return { status: reply.status, error: reply.error, created: log.filter((c) => c.call === 'engine_create').map((c) => c.backend) } + } + const ok = (created) => ({ status: 200, error: undefined, created }) + const gpuError = (path) => ({ status: 500, created: ['gpu'], + error: `Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for ${path}: UNAVAILABLE: no usable OpenCL device')` }) + // The model's own gpu does not start, so its engine runs on cpu, and a request without a backend keeps that engine. + const broken = readable('gpu-broken') + assert.deepEqual(await engines(broken), ok(['gpu', 'cpu'])) + assert.deepEqual(await engines(broken), ok([])) + // gpu set: an engine on gpu, here the GPU's error, not the engine that runs on cpu. + assert.deepEqual(await engines(`${broken},gpu`), gpuError(broken)) + // cpu set: the engine that runs on cpu serves it, and requests without a backend keep it after. + assert.deepEqual(await engines(broken), ok(['gpu', 'cpu'])) + assert.deepEqual(await engines(`${broken},cpu`), ok([])) + assert.deepEqual(await engines(broken), ok([])) + // A model that gets a new engine for each conversation (an LFM2 one): gpu set is not the cpu its last one ran on. + const recurrent = litertlmFile(join(work, 'gpu-broken-lfm2-set.litertlm'), [tflite('tf_lite_prefill_decode', 'gpu,cpu'), llmMetadata(LFM2)]) + assert.deepEqual(await engines(recurrent), ok(['gpu', 'cpu'])) + assert.deepEqual(await engines(`${recurrent},gpu`), gpuError(recurrent)) + // An engine on the model's own backend serves a request that sets it; one on a backend a request set is not the + // model's choice, so a request without a backend gets the model's own. + const first = readable('gpu-first') + assert.deepEqual(await engines(first), ok(['gpu'])) + assert.deepEqual(await engines(`${first},gpu`), ok([])) + assert.deepEqual(await engines(first), ok([])) + assert.deepEqual(await engines(`${first},cpu`), ok(['cpu'])) + assert.deepEqual(await engines(first), ok(['gpu'])) + // A model file replaced by one whose own backend is another gets an engine on that one. + const replaced = readable('backend-replaced', [decoder]) + assert.deepEqual(await engines(replaced), ok(['cpu'])) + readable('backend-replaced', [tflite('tf_lite_prefill_decode', 'gpu'), llmMetadata(GENERIC)]) + assert.deepEqual(await engines(replaced), ok(['gpu'])) + // And one whose own backend is the one the old file's engine fell back to gets an engine of its own too, not the old + // file's: that one was keyed by the cpu it runs on, and served the new file with the old file's settings (here an + // LFM2 model's new engine for each conversation was lost to the old file's info). + const fellBack = readable('gpu-broken-replaced', [tflite('tf_lite_prefill_decode', 'gpu,cpu'), llmMetadata(GENERIC)]) + assert.deepEqual(await engines(fellBack), ok(['gpu', 'cpu'])) + readable('gpu-broken-replaced', [tflite('tf_lite_prefill_decode', 'cpu'), llmMetadata(LFM2)]) + assert.deepEqual(await engines(fellBack), ok(['cpu'])) + assert.deepEqual(await engines(fellBack), ok(['cpu']), 'and LFM2 gets a new engine for its next conversation') +}) + test('a developer message is a system message', { skip, timeout: 60_000 }, async () => { let log = await callsDuring(async () => { const reply = await chat(fake, { model: readable('plain'), messages: [{ role: 'developer', content: 'Answer in one word.' }, ...hello] }) diff --git a/engines/litert-lm-server/parent-death.test.mjs b/engines/litert-lm-server/parent-death.test.mjs index b5c3069..21cc7f6 100644 --- a/engines/litert-lm-server/parent-death.test.mjs +++ b/engines/litert-lm-server/parent-death.test.mjs @@ -6,7 +6,7 @@ // when it ends. import assert from 'node:assert/strict' import { spawn, spawnSync } from 'node:child_process' -import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { mkdtempSync, readdirSync, readFileSync, readlinkSync, rmSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' import { after, before, test } from 'node:test' @@ -118,6 +118,17 @@ test('a server whose parent exited before it started does not start', { skip }, t.skip(`an orphan is adopted by pid ${ppid} here, not init`) return } + // PID 1 of a container's PID namespace is its entrypoint, whose child the server is taken for (main.cc + // ParentExited), so an orphan it adopted starts there by design (the next test). The server is in this process's + // namespace; one whose /proc cannot be read counts as the root one, as the server takes it. + let ns = null + try { + ns = readlinkSync('/proc/self/ns/pid') + } catch {} + if (ns !== null && ns !== 'pid:[4026531836]') { + t.skip('in a container\'s PID namespace, where an orphan its PID 1 adopted starts by design') + return + } outer.stdio[3].end('go\n') assert.ok(await waitFor(() => gone(pid), 3000), 'the server started') assert.match(readFileSync(errFile, 'utf8'), /the process that started turbollm-litertlm-server has already exited; not starting/) @@ -126,6 +137,48 @@ test('a server whose parent exited before it started does not start', { skip }, } }) +// A PID namespace of its own, as a container has: `unshare --pid --fork` makes its command that namespace's PID 1. Root +// can make one, and elsewhere a user namespace (-r) can where the system allows those. --kill-child: unshare's child, +// the namespace's PID 1, and with it everything in the namespace, dies with unshare. +const unshare = skip ? undefined : [[], ['-r']].map((user) => [...user, '--pid', '--fork', '--kill-child', '--mount-proc']) + .find((flags) => spawnSync('unshare', [...flags, 'true'], { stdio: 'ignore', timeout: 10_000 }).status === 0) +const skipNamespace = skip || (!unshare && 'unshare cannot make a PID namespace here') +const tini = !skip && spawnSync('tini', ['--version'], { stdio: 'ignore', timeout: 10_000 }).status === 0 +// The pids whose parent is `pid`, from /proc. +function children(pid) { + return readdirSync('/proc').filter((d) => /^\d+$/.test(d)).map(Number).filter((p) => stat(p)?.ppid === pid) +} + +test('a server whose parent is PID 1 of its own PID namespace, as a container\'s entrypoint is, starts', { skip: skipNamespace }, async () => { + // A shell that starts the server as its child, as `docker run sh -c '... &'` does, and tini, as + // `docker run --init` does. + const inits = [['sh', '-c', 'echo "sh is pid $$" >&2; "$0" serve --host 127.0.0.1 --port 0 & wait', server]] + if (tini) inits.push(['tini', '--', server, 'serve', '--host', '127.0.0.1', '--port', '0']) + for (const init of inits) { + const pids = [] + try { + const outer = spawn('unshare', [...unshare, ...init], { stdio: ['ignore', 'ignore', 'pipe'] }) + pids.push(outer.pid) + let err = '' + outer.stderr.setEncoding('utf8') + outer.stderr.on('data', (d) => { err += d }) + const started = await waitFor(() => err.includes('Starting OpenAI-compatible API server') || err.includes('not starting'), 10_000) + assert.ok(started, `${init[0]}: no word from the server in 10 s: ${err}`) + assert.doesNotMatch(err, /not starting/, init[0]) + if (init[0] === 'sh') assert.match(err, /^sh is pid 1$/m) + // The server is the child of the namespace's PID 1 (unshare's child), and it ends with it. + const [first] = children(outer.pid) + const [serverPid] = first ? children(first) : [] + assert.ok(serverPid, `${init[0]}: the server is not the child of the namespace's PID 1`) + pids.push(first, serverPid) + process.kill(outer.pid, 'SIGKILL') + assert.ok(await waitFor(() => gone(serverPid), 3000), `${init[0]}: the server outlived its namespace's PID 1`) + } finally { + killAll(pids) + } + } +}) + test('--version still answers', { skip }, () => { const r = spawnSync(server, ['--version'], { encoding: 'utf8' }) assert.equal(r.status, 0, r.stderr) diff --git a/engines/litert-lm-server/src/main.cc b/engines/litert-lm-server/src/main.cc index d779ad7..d8e6727 100644 --- a/engines/litert-lm-server/src/main.cc +++ b/engines/litert-lm-server/src/main.cc @@ -655,40 +655,59 @@ ModelFileInfo ReadModelFileInfo(std::ifstream& in, const std::string& header) { return info; } -// ModelFileInfo for the model file, read once per version of it (its size and modification time). A .litertlm starts -// with "LITERTLM", three uint32 versions, 4 bytes of padding and a little-endian uint64 header-end offset (the same 32 -// bytes the daemon's models/litertlm.ts reads). The runtime allocates and zeroes header_end - 32 bytes before reading -// the header, and the std::bad_alloc a bogus offset raises crosses the C API into std::terminate, killing the whole -// server, so the offset is bounded here first. The bound is the runtime's own: it reads the header from the file's -// first 16 KiB only (kLitertLmHeaderMaxSize, LiteRT-LM 0.18.0's litert_lm_loader.h), so a larger offset can never load, -// yet would still cost an allocation up to the file's size (hundreds of MB on a phone). Checked before the loaded -// engine is released, which a bad file then leaves in place. A file that can't be read or lacks the magic is left to -// the runtime, which reports it cleanly and more precisely (e.g. "Permission denied"), and says nothing here. Caller -// holds g_engine_mu. +// ModelFileInfo for the model file, read once per version of it: its device, inode, size, and modification and change +// times to the nanosecond, from a stat, which is all a request costs while the file is unchanged. The change time moves +// with every write, and nothing can set it back, so a rewrite that puts the old modification time back (cp -p, +// touch -r, tar) is a new version too. Keyed by the size and the second, a model rewritten in place within the same +// second at the same size kept the old file's info (an image sent to a model that now takes images was refused), and +// every request opened the file and read it again. The stat comes before the read, so a file that changes in between is +// read again next time. A .litertlm starts with "LITERTLM", three uint32 versions, 4 bytes of padding and a +// little-endian uint64 header-end offset (the same 32 bytes the daemon's models/litertlm.ts reads). The runtime +// allocates and zeroes header_end - 32 bytes before reading the header, and the std::bad_alloc a bogus offset raises +// crosses the C API into std::terminate, killing the whole server, so the offset is bounded here first. The bound is +// the runtime's own: it reads the header from the file's first 16 KiB only (kLitertLmHeaderMaxSize, LiteRT-LM 0.18.0's +// litert_lm_loader.h), so a larger offset can never load, yet would still cost an allocation up to the file's size +// (hundreds of MB on a phone). Refused, for every request to that version of the file, before the loaded engine is +// released, which a bad file then leaves in place. A file that can't be read or lacks the magic is left to the runtime, +// which reports it cleanly and more precisely (e.g. "Permission denied"), and says nothing here. Caller holds +// g_engine_mu. const ModelFileInfo& FileInfo(const ModelRef& ref) { struct Cached { - long long size = -1, mtime = -1; + dev_t dev = 0; + ino_t ino = 0; + off_t size = -1; + timespec mtime{}, ctime{}; + std::string invalid; // the 500 for a header offset the runtime cannot load ModelFileInfo info; }; static std::map cache; + Cached& c = cache[ref.path]; struct stat st {}; - unsigned char h[32] = {}; - std::ifstream in(ref.path, std::ios::binary); - bool ok = ::stat(ref.path.c_str(), &st) == 0 && in.read(reinterpret_cast(h), sizeof h) && - std::memcmp(h, "LITERTLM", 8) == 0; - uint64_t header_end = 0; - for (int i = 31; i >= 24; --i) header_end = (header_end << 8) | h[i]; - constexpr uint64_t kRuntimeHeaderMaxSize = 16 * 1024; - if (ok && (header_end <= 32 || header_end > std::min(st.st_size, kRuntimeHeaderMaxSize))) { - throw HttpError(500, "Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for " + ref.path + - ": invalid .litertlm header')"); + if (::stat(ref.path.c_str(), &st) != 0) { + c = Cached(); + return c.info; + } + if (c.dev != st.st_dev || c.ino != st.st_ino || c.size != st.st_size || c.mtime.tv_sec != st.st_mtim.tv_sec || + c.mtime.tv_nsec != st.st_mtim.tv_nsec || c.ctime.tv_sec != st.st_ctim.tv_sec || + c.ctime.tv_nsec != st.st_ctim.tv_nsec) { + c = Cached{st.st_dev, st.st_ino, st.st_size, st.st_mtim, st.st_ctim, "", ModelFileInfo()}; + if (g_verbose) Log("Reading the header of " + ref.path); + unsigned char h[32] = {}; + std::ifstream in(ref.path, std::ios::binary); + if (in.read(reinterpret_cast(h), sizeof h) && std::memcmp(h, "LITERTLM", 8) == 0) { + uint64_t header_end = 0; + for (int i = 31; i >= 24; --i) header_end = (header_end << 8) | h[i]; + constexpr uint64_t kRuntimeHeaderMaxSize = 16 * 1024; + if (header_end <= 32 || header_end > std::min(st.st_size, kRuntimeHeaderMaxSize)) { + c.invalid = "Failed to load engine: RuntimeError('Failed to create LiteRT-LM engine for " + ref.path + + ": invalid .litertlm header')"; + } else { + std::string header(header_end - 32, '\0'); + if (in.read(&header[0], static_cast(header.size()))) c.info = ReadModelFileInfo(in, header); + } + } } - Cached& c = cache[ref.path]; - if (c.size == static_cast(st.st_size) && c.mtime == static_cast(st.st_mtime)) return c.info; - c = Cached{static_cast(st.st_size), static_cast(st.st_mtime), ModelFileInfo()}; - if (!ok) return c.info; - std::string header(header_end - 32, '\0'); - if (in.read(&header[0], static_cast(header.size()))) c.info = ReadModelFileInfo(in, header); + if (!c.invalid.empty()) throw HttpError(500, c.invalid); return c.info; } @@ -715,8 +734,9 @@ struct EngineHolder { bool tools_unconstrained = false; // What the model file says about itself (ModelFileInfo), as it was when this engine was created. ModelFileInfo info; - // The backend the engine runs on: key.backend, or CPU when the model's own backend did not start. - std::string backend; + // The model's own backend, when no backend was set for this engine (else empty): the one it runs on (key.backend), or + // the one it fell back from because it did not start. + std::string backend_chosen; // The context window this engine has, in tokens (-1: unknown), and the larger one that was asked for, if any. int window = -1, window_asked = -1; // The model's decoder keeps a recurrent state (info.recurrent_state), and a message has been sent on this engine @@ -845,6 +865,7 @@ LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc, co // start falls back to the next one it lists (below); one the user set does not. const bool backend_set = !key.backend.empty(); if (!backend_set) key.backend = info.text_backends.empty() ? "cpu" : info.text_backends.front(); + const std::string chosen = backend_set ? "" : key.backend; // A vision or audio executor is created only when the engine settings name a backend for it (c/engine.cc, // CreateEngineSettingsHelper), and none was named, so an image sent to LFM2.5-VL-450M got a 500 "Vision executor // should not be null, please TryLoadingVisionExecutor() first." although the model takes images. @@ -858,6 +879,22 @@ LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc, co if (info.capabilities_known && needs.audio && key.audio_backend.empty()) { throw HttpError(400, "Invalid messages: " + ref.id + " takes no audio input"); } + // An engine is keyed by the backend it runs on, so a backend the request or the config sets gets an engine on that + // backend or that backend's error. Keyed by the model's choice, an engine that had fallen back to CPU served a request + // that set gpu. A request that sets none takes the engine made for the same choice of the model's, on whichever + // backend it fell back to, so the model's own is not tried again until another engine is loaded. + if (!chosen.empty() && g_engine.engine && g_engine.backend_chosen == chosen) { + EngineKey fallen = key; + fallen.backend = g_engine.key.backend; + if (g_engine.key == fallen) key = fallen; + } + // Nor does a request that sets none take one made for another choice of the model's, which only a replaced file + // (the choice is its header's) brings, even when that engine fell back to the backend the new file chooses: it was + // loaded from the old file, and its ModelFileInfo (sampler, context, recurrent state) is the old file's. A file + // replaced by one of the same choice, or used with a backend set, still keeps the engine, as it always has: the key + // holds no version of the file. + const bool other_choice = !chosen.empty() && g_engine.engine && !g_engine.backend_chosen.empty() && + g_engine.backend_chosen != chosen; // A model whose decoder keeps a recurrent state gets a new engine for every conversation after the first. LiteRT-LM // 0.18.0 starts each new conversation on an engine at step 0 and keeps the executor's buffers as they are there @@ -868,8 +905,9 @@ LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc, co // France." after that same question's first turn and a title request, and with "The capital of Italy is **Rome**." // on a fresh one (LiteRT-LM #3165; the model card says to create a new Engine for each conversation). The C API has // no way to reset that state, so the engine is created again, with the same settings. - const bool fresh_state = g_engine.engine && g_engine.key == key && g_engine.info.recurrent_state && g_engine.state_used; - if (g_engine.engine && g_engine.key == key && !fresh_state) return g_engine.engine; + const bool fresh_state = + g_engine.engine && g_engine.key == key && !other_choice && g_engine.info.recurrent_state && g_engine.state_used; + if (g_engine.engine && g_engine.key == key && !other_choice && !fresh_state) return g_engine.engine; if (fresh_state) { static std::set told; // once per model file, not per conversation if (told.insert(ref.path).second) { @@ -931,12 +969,12 @@ LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc, co } return e; }; - // A new engine for a fresh state (above) runs where the one before it did, so a model whose own backend fell back is - // not tried on that backend again for every conversation. A backend other than CPU that the model picked and that - // does not start falls back to the next one the model lists, and only to one it lists: a GPU-only bundle was tried - // on CPU, which cannot run it, and the error then gave the CPU's reason instead of the GPU's. When none starts, the - // error gives each one's reason. - std::vector backends = {fresh_state ? g_engine.backend : key.backend}; + // A new engine for a fresh state (above) has the key of the one before it, so it runs where that one did, and a model + // whose own backend fell back is not tried on that backend again for every conversation. A backend other than CPU that + // the model picked and that does not start falls back to the next one the model lists, and only to one it lists: a + // GPU-only bundle was tried on CPU, which cannot run it, and the error then gave the CPU's reason instead of the + // GPU's. When none starts, the error gives each one's reason. + std::vector backends = {key.backend}; if (!backend_set && backends[0] != "cpu") { auto at = std::find(info.text_backends.begin(), info.text_backends.end(), backends[0]); if (at != info.text_backends.end()) backends.insert(backends.end(), at + 1, info.text_backends.end()); @@ -956,7 +994,8 @@ LiteRtLmEngine* GetOrCreateEngine(const ModelRef& ref, const ModelConfig& mc, co if (window <= 0) window = RuntimeWindow(info, -1, backend); g_engine.engine = e; g_engine.key = key; - g_engine.backend = backend; + g_engine.key.backend = backend; + g_engine.backend_chosen = chosen; // The same model and settings again: what was found out about the model still holds. if (!fresh_state) g_engine.tools_unconstrained = false; g_engine.info = info; @@ -993,20 +1032,22 @@ std::map BuildNameByToolCallId(const json& messages) { constexpr int kMaxJsonDepth = 128; struct JsonTooDeep : std::runtime_error { - JsonTooDeep() : std::runtime_error("nested deeper than " + std::to_string(kMaxJsonDepth) + " levels") {} + explicit JsonTooDeep(int max_depth = kMaxJsonDepth) + : std::runtime_error("nested deeper than " + std::to_string(max_depth) + " levels") {} }; // The DOM builder json::parse itself uses, counting the arrays and objects open: it throws JsonTooDeep at one that -// kMaxJsonDepth others enclose. Not json::parse with a parser callback, which would be simpler: that parser's +// max_depth others enclose. Not json::parse with a parser callback, which would be simpler: that parser's // end_object scans the whole enclosing array or object for a value the callback discarded, each time an object closes, // so a body holding N objects in one array or object took O(N^2). A normal chat request with an extra field of 50,000 // empty objects (575 KB) kept a core busy for 16 s, on the connection thread and before any lock, and nothing stops // that once the client has gone. This builder's start and end are what json::parse does, so it stays linear. struct BoundedDomParser : nlohmann::detail::json_sax_dom_parser { - using json_sax_dom_parser::json_sax_dom_parser; + BoundedDomParser(json& result, int max_depth) : json_sax_dom_parser(result, true), max_depth(max_depth) {} + const int max_depth; int depth = 0; bool start_object(std::size_t n) { - if (++depth > kMaxJsonDepth) throw JsonTooDeep(); + if (++depth > max_depth) throw JsonTooDeep(max_depth); return json_sax_dom_parser::start_object(n); } bool end_object() { @@ -1014,7 +1055,7 @@ struct BoundedDomParser : nlohmann::detail::json_sax_dom_parser { return json_sax_dom_parser::end_object(); } bool start_array(std::size_t n) { - if (++depth > kMaxJsonDepth) throw JsonTooDeep(); + if (++depth > max_depth) throw JsonTooDeep(max_depth); return json_sax_dom_parser::start_array(n); } bool end_array() { @@ -1023,11 +1064,11 @@ struct BoundedDomParser : nlohmann::detail::json_sax_dom_parser { } }; -// json::parse, throwing JsonTooDeep at an array or object that kMaxJsonDepth others enclose. A syntax error throws +// json::parse, throwing JsonTooDeep at an array or object that max_depth others enclose. A syntax error throws // json::parse_error, as with json::parse (the builder is made with allow_exceptions). -json ParseBounded(const std::string& s) { +json ParseBounded(const std::string& s, int max_depth = kMaxJsonDepth) { json result; - BoundedDomParser builder(result, true); + BoundedDomParser builder(result, max_depth); json::sax_parse(s, &builder); return result; } @@ -1402,12 +1443,26 @@ std::string ContentText(const json& content) { return out; } +// Throws JsonTooDeep for a JSON chunk nested deeper than a request may be, its envelope aside. The runtime's chunk +// carries the tool calls it parsed from the model's output, 4 levels down (the chunk, its tool_calls, the call, its +// function), so a call's arguments may nest as deep as a request's, kMaxJsonDepth. Copying the tool calls (below) and +// writing them out recurse once per level, and a stand-in's chunk with a tool call 200,000 levels deep killed the server +// with SIGSEGV. LiteRT-LM 0.18.0 bounds what it parses itself (a JSON tool call at 128 levels), but the server does not +// rely on what the runtime hands it. Piece ParseChunk(const std::string& s) { Piece p; if (s.empty()) return p; json j; try { - j = json::parse(s); + j = ParseBounded(s, kMaxJsonDepth + 4); + } catch (const JsonTooDeep&) { + // Text that is not JSON, however many brackets it opens, is plain text, as below. accept() builds nothing, so it + // does not recurse. + if (!json::accept(s)) { + p.text = s; + return p; + } + throw; } catch (...) { p.text = s; // the Python client treats unparseable chunks as plain text, too return p; @@ -1580,12 +1635,23 @@ struct Penalties { int window = 0; }; +// Why `v`, a number the request sets as `field`, cannot be passed to LiteRT-LM, which takes sampler values as floats, +// or empty. A number beyond a float's range (FLT_MAX, 3.4028235e+38) passed every check the server made as a double and +// became an infinity as a float: temperature 1e39 reached the sampler as inf, a penalty of 1e39 or -1e39 the decoder. +std::string OutOfFloatRange(const char* field, const json& v) { + if (std::isfinite(static_cast(v.get()))) return ""; + return std::string(field) + " must be within a float's range (-3.4028235e+38 to 3.4028235e+38), got " + v.dump(); +} + std::optional ParsePenalties(const json& body) { auto has = [&](const char* k) { return body.contains(k) && !body[k].is_null(); }; auto number = [&](const char* k) { if (!body[k].is_number() || !std::isfinite(body[k].get())) { throw HttpError(400, std::string("Invalid sampler parameters: ValueError: ") + k + " must be a number"); } + if (std::string why = OutOfFloatRange(k, body[k]); !why.empty()) { + throw HttpError(400, "Invalid sampler parameters: ValueError: " + why); + } return body[k].get(); }; Penalties p; @@ -1881,6 +1947,10 @@ void HandleChatCompletions(int fd, const Request& req) { SendError(fd, req, 400, "Invalid sampler parameters: ValueError: " + why + ", got " + body[k].dump()); return; } + if (std::string range = OutOfFloatRange(k, body[k]); !range.empty()) { + SendError(fd, req, 400, "Invalid sampler parameters: ValueError: " + range); + return; + } has_sampler = true; } } @@ -2002,7 +2072,7 @@ void HandleChatCompletions(int fd, const Request& req) { // Engine.create_conversation, with the options the serve path passes, but constrained decoding only with tools // (below). LiteRtLmSessionConfig* sc = L.litert_lm_session_config_create(); - if (auto sampling = SamplingFor(body, g_engine.info, g_engine.backend, ref.id, has_sampler)) { + if (auto sampling = SamplingFor(body, g_engine.info, g_engine.key.backend, ref.id, has_sampler)) { LiteRtLmSamplerParams* sp = L.litert_lm_sampler_params_create(kLiteRtLmSamplerTypeTopP); L.litert_lm_sampler_params_set_top_k(sp, sampling->top_k); L.litert_lm_sampler_params_set_top_p(sp, sampling->top_p); @@ -2284,7 +2354,20 @@ void HandleChatCompletions(int fd, const Request& req) { } break; } - Piece p = ParseChunk(item.text); + Piece p; + try { + p = ParseChunk(item.text); + } catch (const JsonTooDeep& e) { + // The model's output cannot be read: the request fails as on the runtime's own error, and the rest of the + // generation is not waited for. + if (!cancelled) { + cancelled = true; + L.litert_lm_conversation_cancel_process(conv.get()); + } + error = std::string("the model's output, as the runtime parsed it, is ") + e.what() + " (" + + std::to_string(kMaxJsonDepth) + " within a tool call's arguments)"; + break; + } p.text = apply_stops(p.text); if (stopped) p.tool_calls = json::array(); if (!p.reasoning.empty()) ++reasoning_tokens; @@ -2476,6 +2559,27 @@ void PrintHelp() { "The request's `model` field is the model path: [,[,]]\n"); } +#ifdef __linux__ +// Whether the process that started this one has already exited, which main() can tell only by this process now being +// init's: an orphan is adopted by PID 1. Only the machine's own init counts, PID 1 of the root PID namespace. PID 1 of +// a container's namespace is its entrypoint (tini, dumb-init, a shell), which starts the server as its child on purpose +// (`docker run --init`), and such a server was refused. The kernel keeps no record of who forked a process, so there +// an orphan, which that PID 1 adopts too, cannot be told from its child, and starts. On Android, where only the app +// starts this binary, a parent of PID 1 is always the app gone, also where Android runs in a container (ChromeOS's +// ARC++), so there it stays refused. +bool ParentExited() { + if (::getppid() != 1) return false; +#ifndef __ANDROID__ + char ns[64]; + const ssize_t n = ::readlink("/proc/self/ns/pid", ns, sizeof ns); + // The root PID namespace's inode, PROC_PID_INIT_INO in the kernel's proc_ns.h (since Linux 3.8). When /proc cannot + // be read, as before. + if (n > 0 && std::string(ns, static_cast(n)) != "pid:[4026531836]") return false; +#endif + return true; +} +#endif + } // namespace int main(int argc, char** argv) { @@ -2487,14 +2591,14 @@ int main(int argc, char** argv) { // covers every other way it is started: by a daemon without that helper, through the daemon's sh lifeline when the // helper fails its self-test, or by hand. Armed before anything else runs, so even a server stuck loading its // library goes. A parent process that ended before the prctl never sends it: this process was then already - // reparented to init, so the server does not start. This sees the process end, not the thread: had only the forking - // thread ended, this process would have moved to another thread of its parent and the signal would come when that - // one ends; the daemon's thread ends only with the daemon. A server that init starts on purpose is refused too; - // none is, as this binary ships only in the APK (host builds are for its tests). + // reparented to init, so the server does not start (ParentExited). This sees the process end, not the thread: had + // only the forking thread ended, this process would have moved to another thread of its parent and the signal would + // come when that one ends; the daemon's thread ends only with the daemon. A server that the machine's own init + // starts on purpose is refused too; none is on Android, where this binary ships only in the APK. if (::prctl(PR_SET_PDEATHSIG, SIGKILL) != 0) { std::fprintf(stderr, "prctl(PR_SET_PDEATHSIG) failed: %s; the server will not end with its parent\n", std::strerror(errno)); - } else if (::getppid() == 1) { + } else if (ParentExited()) { std::fprintf(stderr, "the process that started %s has already exited; not starting\n", kServerName); return 1; } diff --git a/engines/litert-lm-server/testdata/fake_litert_lm.cc b/engines/litert-lm-server/testdata/fake_litert_lm.cc index af6ce5c..e3d7ca9 100644 --- a/engines/litert-lm-server/testdata/fake_litert_lm.cc +++ b/engines/litert-lm-server/testdata/fake_litert_lm.cc @@ -19,8 +19,9 @@ // "constrained=1 tools=1 provider=0 constraint=0 creates=3", where creates counts every litert_lm_conversation_create // call in the process; a message whose text starts with "say:" is answered with the rest of its text instead, three // bytes to a chunk, as a model streams, and one whose text is "too long" fails as a prompt longer than the context -// window does. A regex that does not compile fails the send as llguidance does -// (llg_constraint_provider.cc:155-160). +// window does; one whose text is "nested:" is answered with a tool call whose arguments nest n arrays, and one whose +// text is "nested-text:" with a chunk that is no JSON, n brackets and some text. A regex that does not compile fails +// the send as llguidance does (llg_constraint_provider.cc:155-160). // // With FAKE_LITERT_LM_CALLS set to a file, the calls that say what the server asked the runtime for are appended to it, // one JSON object per line ({"call": "", ...its arguments}), for @@ -255,6 +256,25 @@ int litert_lm_conversation_send_message_stream(LiteRtLmConversation* conv, const callback(data, &failed); return 0; } + if (said.rfind("nested:", 0) == 0) { + // A tool call whose arguments nest n arrays, as the runtime parses one a model wrote with that many brackets. + // Written out by hand: nlohmann would build it recursively. + const size_t n = std::strtoul(said.c_str() + 7, nullptr, 10); + LiteRtLmStreamChunk call{"{\"role\":\"assistant\",\"tool_calls\":[{\"type\":\"function\",\"function\":{\"name\":\"f\"," + "\"arguments\":{\"a\":" + std::string(n, '[') + std::string(n, ']') + "}}}]}", + nullptr, false}; + callback(data, &call); + LiteRtLmStreamChunk end{"", nullptr, true}; + callback(data, &end); + return 0; + } + if (said.rfind("nested-text:", 0) == 0) { + LiteRtLmStreamChunk text{std::string(std::strtoul(said.c_str() + 12, nullptr, 10), '[') + " not json", nullptr, false}; + callback(data, &text); + LiteRtLmStreamChunk end{"", nullptr, true}; + callback(data, &end); + return 0; + } std::vector chunks = {text}; if (said.rfind("say:", 0) == 0) { chunks.clear(); diff --git a/play-store-assets/privacy-notes.md b/play-store-assets/privacy-notes.md index 035b264..20cb119 100644 --- a/play-store-assets/privacy-notes.md +++ b/play-store-assets/privacy-notes.md @@ -50,23 +50,23 @@ CURRENTLY CODED; once Firebase is integrated, add a corresponding "app diagnosti daemon at `http://127.0.0.1:7000`. `DaemonWebView.kt` loads the UI from there, and `DaemonAuth.kt` makes the app's other calls to it: the check that the daemon proves itself before the app sends its secret, and the authenticated requests from `EngineService.kt` - (activity, downloads, status, and the request to write the chat database to disk before - Quit and before a close with nothing running ends the app), `ModelImport.kt` (adding the - imported-models folder) and `StorageAccess.kt` (rescan). (`DaemonClient.kt` in `commonMain` - points at `127.0.0.1:6996`; it is unused Spike A code that only the old `App()` screen, which - the Android app never shows, constructs.) The web UI in that WebView goes further, though: it - loads the images its pages point at from any https host, and from any host over plain http - on port 80. The daemon's CSP for the app allows that on purpose - (`APP_SECRET_CSP` in its `server.ts`: images and media from `'self'`, any `https:` host, - `http://*`, which with no port matches plain http on port 80 only, `data:` and `blob:`), and - `DaemonWebView.kt` refuses only other `127.0.0.1` ports. Those are the images in a model card - (the README in the Hugging Face dialog, which can embed images from any site) and markdown - images in chat replies and the other text the UI renders as markdown. An image in a reply loads - as soon as the reply shows, with no approval, so a reply (after a prompt injection in a pasted - file or a search result, say) can make the phone contact a host the model names and pass it - text from the chat in the image's URL. That behaviour is TurboLLM#287's "Known, not fixed - here" entry "Markdown images in chat", an owner decision; fill in the Data safety form with it - in mind. + (activity, with downloads and status when the daemon's activity answer does not carry them, + and the request to write the chat database to disk before Quit and before a close with nothing + running ends the app), `ModelImport.kt` (adding the imported-models folder) and + `StorageAccess.kt` (rescan). (`DaemonClient.kt` in `commonMain` points at `127.0.0.1:6996`; it + is unused Spike A code that only the old `App()` screen, which the Android app never shows, + constructs.) The web UI in that WebView goes further, though: it loads the images its pages + point at from any https host, and from any host over plain http on port 80. The daemon's CSP + for the app allows that on purpose (`APP_SECRET_CSP` in its `server.ts`: images and media from + `'self'`, any `https:` host, `http://*`, which with no port matches plain http on port 80 + only, `data:` and `blob:`), and `DaemonWebView.kt` refuses only other `127.0.0.1` ports. Those + are the images in a model card (the README in the Hugging Face dialog, which can embed images + from any site) and markdown images in chat replies and the other text the UI renders as + markdown. An image in a reply loads as soon as the reply shows, with no approval, so a reply + (after a prompt injection in a pasted file or a search result, say) can make the phone contact + a host the model names and pass it text from the chat in the image's URL. That behaviour is + TurboLLM#287's "Known, not fixed here" entry "Markdown images in chat", an owner decision; + fill in the Data safety form with it in mind. 2. **The embedded daemon (the same `turbollm` product code that runs on desktop) makes the same categories of outbound request desktop does, all user-initiated except where noted:** - Searching and browsing Hugging Face from Discover: opening it lists popular repos, typing