Follow-ups to #8: backend fallback, container start, model file cache, float range, chunk depth, poll

BUILDING-ENGINES.md

@@ -394,7 +394,9 @@ the request sets a field, as before: LiteRT-LM leaves sampling to the model ther
 or below means no top-k limit, as in llama.cpp (TurboLLM's Top K 0 means disabled), and one above
 0 is passed as set, on GPU too, where LiteRT-LM 0.18.0 sets no limit of its own; a `top_p`
 outside 0 to 1, a negative `temperature`, and a `top_k` or `seed` with a fraction are a 400 that
-names the field instead of the runtime's 500.
+names the field instead of the runtime's 500. LiteRT-LM takes these values and the repetition
+penalties below as floats, so a `temperature` or penalty beyond a float's range (3.4028235e+38
+either way), which reached it as an infinity, is a 400 that names the field too.
 
 **Thinking is asked for the ways clients ask for it.** In order, the first that says: the
 top-level `reasoning_effort` (what the Python server reads, and what the daemon sends for a
@@ -458,7 +460,8 @@ puts gpu first), as `litert-lm serve` picks it. The server picked CPU for every
 GPU-only bundle sent without a backend failed to load. When the model's own backend does not
 start, the engine falls back to the next one the model lists, CPU for most, and never to one it
 does not list, and the server logs why; when none starts, the error gives each one's reason. A
-backend the user set is used as set, failing as before.
+backend the user set is used as set, failing as before, also when an engine whose own backend fell
+back is loaded: an engine is kept for the backend it runs on.
 
 **A conversation that ends with the assistant's own words is a 400.** A client that prefills the
 start of a reply asks for it to be continued, which LiteRT-LM 0.18.0 can do only from C++
@@ -477,10 +480,14 @@ AddressSanitizer caught an exception being destroyed by the other runtime. See
 ends, and it exits 1 without starting when its parent is already init (the starter died before
 the `prctl`). The daemon's [engine tether](#the-engine-tether) does the same before it execs the
 server; this covers a server started any other way: by a daemon without the tether, through the
-daemon's `/system/bin/sh` fallback, or by hand. `node --test
-engines/litert-lm-server/parent-death.test.mjs` builds the server for the machine (Linux, `cc`
-and `c++`) against a stand-in `liblitert-lm.so` whose functions do nothing, and checks that a
-stand-in daemon's SIGKILL takes the listening server with it and that an orphaned start exits.
+daemon's `/system/bin/sh` fallback, or by hand. Off Android, a parent that is PID 1 of a
+container's PID namespace, its entrypoint (tini, a shell), is taken for the server's starter, as
+it usually is: the kernel keeps no record that would tell an orphan that PID 1 adopted from its
+child. `node --test engines/litert-lm-server/parent-death.test.mjs` builds the server for the
+machine (Linux, `cc` and `c++`) against a stand-in `liblitert-lm.so` whose functions do nothing,
+and checks that a stand-in daemon's SIGKILL takes the listening server with it, that an orphaned
+start exits (outside a container's PID namespace, where that test is skipped), and, where `unshare`
+can make a PID namespace, that a server its PID 1 starts runs.
 
 ### GPU
 

PLAY-FGS-DECLARATION.md

@@ -40,9 +40,11 @@ in the foreground, also while idle, until the user quits or closes the app with
 rather than suggesting it runs only during generation.
 
 LAN access, remote access and Turbo Link are read from the daemon the app bundles: the `serving` field of its GET
-/api/v1/status, from TurboLLM's feat(api): /api/v1/status says which server features are on. A daemon without that
-field reports them all off (the app reads a missing or malformed `serving` as nothing served), so a close with only
-one of them on would quit: the text below holds only for an APK built with a TurboLLM that has it.
+/api/v1/status, from TurboLLM's feat(api): /api/v1/status says which server features are on, or the same field in the
+`status` its GET /api/v1/activity carries too, from TurboLLM's fix(api): one activity request carries what the
+Android poll reads. A daemon without that field reports them all off (the app reads a missing or malformed `serving`
+as nothing served), so a close with only one of them on would quit: the text below holds only for an APK built with a
+TurboLLM that has it.
 
 A close made while the app is still starting, before its daemon answers (a swipe from Recents during the splash
 screen, and before Android 12 Back there too), waits for the daemon, for up to 30 seconds, and is then decided as the

composeApp/src/androidMain/kotlin/com/turbollm/app/DownloadNotice.kt

@@ -26,8 +26,14 @@ internal data class DownloadNotice(val title: String, val text: String, val prog
 /** The running and queued rows of a GET /api/v1/downloads reply, or null when the reply is not the list the daemon
  *  sends (the poll then counts as failed, and as idle, as before). A row with a field missing or of another type
  *  reads that field as 0 instead of failing the whole list. */
-internal fun parseLiveDownloads(body: String): List<LiveDownload>? {
-    val list = runCatching { JSONObject(body).optJSONArray("downloads") }.getOrNull() ?: return null
+internal fun parseLiveDownloads(body: String): List<LiveDownload>? =
+    runCatching { JSONObject(body) }.getOrNull()?.let(::liveDownloads)
+
+/** The running and queued rows of [json]'s `downloads`, as parseLiveDownloads reads them: the list GET
+ *  /api/v1/downloads answers, and the one a GET /api/v1/activity reply carries too from a daemon that sends it there
+ *  (TurboLLM's active-work.ts, its running and queued rows only). Null when [json] has no such list. */
+internal fun liveDownloads(json: JSONObject): List<LiveDownload>? {
+    val list = json.optJSONArray("downloads") ?: return null
     return (0 until list.length()).mapNotNull { i ->
         val row = list.optJSONObject(i) ?: return@mapNotNull null
         val status = row.optString("status")

composeApp/src/androidMain/kotlin/com/turbollm/app/EngineService.kt

@@ -341,13 +341,15 @@ class EngineService : Service() {
     /** Whether anything runs, asked of the daemon now (see appClosed): a model import, work in progress or a reply
      *  waiting for a tool approval (GET /api/v1/activity's items, engineGenerating or waiting), a model download
      *  running or queued (a paused or failed one does not count), or a model loaded or loading or a server feature on
-     *  (queryStatus). A query that fails, or a reply it does not recognise, counts as something running. */
+     *  (queryStatus). The downloads and the status come from the same activity reply when it carries them
+     *  (ActivityReply), else from their own routes. A query that fails, or a reply it does not recognise, counts as
+     *  something running. */
     private fun anythingRuns(): Boolean {
         if (ModelImport.isRunning()) return true
-        val activity = queryActivity() ?: return true
-        if (activity.work != null || activity.waiting) return true
-        if (queryDownloads()?.isEmpty() != true) return true
-        return queryStatus()?.runs != false
+        val reply = queryActivity() ?: return true
+        if (reply.activity.work != null || reply.activity.waiting) return true
+        if ((reply.downloads ?: queryDownloads())?.isEmpty() != true) return true
+        return (reply.status ?: queryStatus())?.runs != false
     }
 
     /** Ask the daemon to write its chat database to disk now (POST /api/v1/daemon/flush), waiting at most
@@ -382,10 +384,15 @@ class EngineService : Service() {
         while (running) {
             // A model import counts as work too: the service must neither stop nor let the CPU sleep under the copy.
             val importing = ModelImport.isRunning()
-            // So does a model download (Discover): a stopped service or a sleeping CPU would stall a multi-GB transfer.
-            val downloads = queryDownloads()
+            // One request a tick, two loopback connections (DaemonAuth's proof and the request), to a daemon whose
+            // activity reply carries the downloads and the status too (ActivityReply); an older daemon's are asked of
+            // their own routes, as before, each with a proof of its own: four connections a tick, six while idle.
+            val reply = queryActivity()
+            val activity = reply?.activity
+            // A model download counts too (Discover): a stopped service or a sleeping CPU would stall a multi-GB
+            // transfer.
+            val downloads = reply?.downloads ?: queryDownloads()
             val downloading = !downloads.isNullOrEmpty()
-            val activity = queryActivity()
             // A poll that could not read the activity keeps its last work or approval line up for up to
             // ACTIVITY_MISSES polls in a row, as the download lines do below: the daemon can be slow to answer for a
             // moment mid-reply, and the line read "Idle" for that poll, while a close then counts the unreadable
@@ -402,7 +409,11 @@ class EngineService : Service() {
             val nowBusy = activity?.work != null || importing || downloading
             // The loaded model's name, for the line shown while nothing runs: asked only then, and kept from the last
             // reading when the status cannot be read, so a slow answer does not drop the name for 3 s.
-            val status = if (work == null && !importing && !downloading && !waiting) queryStatus() else null
+            val status = if (work == null && !importing && !downloading && !waiting) {
+                reply?.status ?: queryStatus()
+            } else {
+                null
+            }
             val model = if (status != null) status.name else this.model
             // The query can take seconds. Under the lock, so the service being destroyed meanwhile is seen before the
             // wake lock is re-taken or a removed notification is posted again.
@@ -449,10 +460,10 @@ class EngineService : Service() {
         }
     }
 
-    /** What the daemon is doing (GET /api/v1/activity, see DaemonActivity), or null when it could not be read: the poll
+    /** What the daemon is doing (GET /api/v1/activity, see ActivityReply), or null when it could not be read: the poll
      *  reads that as idle for the wake lock and keeps its last line for a few polls, a close (anythingRuns) counts it
      *  as something running. */
-    private fun queryActivity(): DaemonActivity? = try {
+    private fun queryActivity(): ActivityReply? = try {
         val conn = DaemonAuth.open("/api/v1/activity")
         conn.connectTimeout = 1000
         conn.readTimeout = 2000
@@ -465,7 +476,10 @@ class EngineService : Service() {
                 // daemon's: unreadable, so a close does not take it for nothing running.
                 JSONObject(conn.inputStream.bufferedReader().use { it.readText() })
                     .takeIf { it.optJSONArray("items") != null }
-                    ?.let(::daemonActivity)
+                    ?.let { json ->
+                        val status = json.optJSONObject("status")?.let(::statusReading)
+                        ActivityReply(daemonActivity(json), liveDownloads(json), status)
+                    }
             }
         } finally {
             conn.disconnect()
@@ -618,6 +632,17 @@ class EngineService : Service() {
  *  (the daemon lists those chats under `waiting`, not `items`). */
 private class DaemonActivity(val work: String?, val waiting: Boolean)
 
+/** A GET /api/v1/activity reply: what the daemon is doing, and, from a daemon that sends them there too (TurboLLM's
+ *  active-work.ts), its running and queued downloads ([downloads], as GET /api/v1/downloads gives them) and the fields
+ *  of GET /api/v1/status the service reads ([status], in that route's names and shapes), so that one request a poll
+ *  tells all three. Each is null when the reply does not carry it as that route would answer (an older daemon, or a
+ *  field this does not recognise), and is then asked of its own route, as before. */
+private class ActivityReply(
+    val activity: DaemonActivity,
+    val downloads: List<LiveDownload>?,
+    val status: StatusReading?,
+)
+
 /** What GET /api/v1/status says. [runs]: something runs there, a model held or a server feature on (statusReading).
  *  [name]: the model the notification names while nothing runs, null when none is loaded or the loaded one is only
  *  an embedding model in a pool slot, which the status does not name. */