diff --git a/CHANGELOG.md b/CHANGELOG.md index f61eaef..fd6d4e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ ## Unreleased ### Added +- **TensorFold status on the device heartbeat**: `uik-daemon` and `ide-agent-kit intent daemon` can publish the live status of each TensorFold server in the model registry under the device's `tensorfold` key, so CodeWatch can show a card per machine. Opt in with `INTENT_TENSORFOLD_REGISTRY=`; `INTENT_TENSORFOLD_SCOPE=local` (default, loopback entries only) or `all`. Read with GET `/health`, `/metrics` and `/v1/slide/graph` only, never a completion. Publishes model, warming, version/backend/device when the server reports them, memory, running and waiting requests, token counters, tok/s from counter deltas between reads, draft acceptance, TTFT p50, KV cache use, Living Weights fact counts (never fact text) and the metrics auth state. An unreachable server is `reachable: false`; an unreadable registry is an `error`, never an empty list. New `ide-agent-kit tensorfold-status [--json]` prints the same object. The registry gains `engine` (`"tensorfold"` marks an entry, any other name skips it, absent means auto-detect). A key is sent only to an entry marked TensorFold that names its own `keyFile`, and only on the two gated routes; the fleet-wide `LLM_API_KEY` fallbacks are never used for a status read. - **What is still waiting on you** (`bin/iak-pending.mjs`): asks every daemon the fleet knows about - this machine's plus every `watchdog-roster.json` entry with a `gate` - for its pending intents and prints them oldest first with the age in the leading column. Motivation: confirmation cards post to the room correctly and then scroll away; on 2026-09-20 four approval requests sat pending on one machine for up to 9 hours 12 minutes and the owner asked twice in one night what was waiting on him, because no surface anywhere could answer it. A host that could not be reached is reported under its own `COULD NOT ASK` heading and is never folded into "nothing pending"; a gate naming another machine by LAN IP is refused rather than probed and reported the same way. A daemon whose rows carry no `status` this version understands is a shape mismatch rather than an empty queue, and a fleet roster that exists but cannot be parsed or read exits `2` instead of silently shrinking the fleet to this machine (only a *missing* roster means "single-machine install"). An age that cannot be believed - missing, future-dated, or epoch-seconds - renders as `age unknown`, sorts to the top and survives `--older-than`, rather than being clamped to "fresh". Exit codes: `0` all clear, `10` pending, `11` some hosts unreachable, `12` no host reachable, `2` cannot run, precedence `12 > 11 > 10 > 0`; `--older-than` narrows what is printed and announced but can never produce a `0`. `--json` for machines. `--room` requires `--older-than` and announces each item at most once per escalation (threshold, then each doubling of its age), keeping a ledger at `~/.iak/pending-announced.json` that is written atomically before the post, carries forward entries for hosts it could not ask, rolls back when a post fails, and degrades to silence - never a storm - when it is missing, corrupt or unwritable. `examples/pending-launchd.plist` wires it to a timer. The gate token is resolved by the existing `resolveGateToken` helper, is sent only to hosts that helper already trusts, and never appears in output or argv. - **Session degradation watch** (`bin/iak-degradation-watch.mjs`): a per-machine daemon that tails watched agents' Claude Code session transcripts and posts a room alert (pushed to the owner's phone via CodeWatch) the moment a session's `message.model` leaves the configured accepted set or its `permissionMode` differs from the expected one. Motivation: on 2026-07-18 an agent's session silently slipped to a small fallback model with manual permission prompts while the owner was away; the degraded agent kept posting — looping on stale tasks and reposting completed instructions — and cost an evening to diagnose. Alerts fire immediately on first detection (a wrong model in a transcript is a fact, not a flap) with a 30-minute per-signal cooldown and a recovery note. Config: `degradation_watch` block (`interval_sec`, `alert_room`, `agents[]` with `transcript_dir`, `accepted_models` prefix list, `expected_permission_mode`); API key reuses `poller.api_key`. `--once` and `--dry-run` flags for testing. Run one instance per machine under process supervision. - **Send-to-session**: the :8788 daemon gains `POST /sessions/send {agent, text, from?}` — deliver text as user input into any named agent's live session — plus `GET /sessions/agents` for the target picker. Four delivery adapters, configured per agent under `mcp.confirmations.sessions.agents`: `wake` (spawn a wake/injection script), `gui` (same, chained behind a human-idle guard with `--wait` so it never types over the user), `tmux` (literal `send-keys` + Enter), and `forward` (relay to the owning machine's daemon, provenance passed through so the `[from …]` prefix is applied exactly once). Deliveries are accepted-async (202) and every attempt is receipted. This is the backend half of the CodeWatch send box; it drives any agent uniformly, including headless sessions the provider apps cannot reach. diff --git a/README.md b/README.md index 8934b98..a00eef2 100644 --- a/README.md +++ b/README.md @@ -486,6 +486,28 @@ npx uik-daemon # standalone equivalent, INTENT_* env vars The IDE Agent Kit and User Intent Kit are designed to be deployed together: IAK consumes the intent state for gating, UIK publishes your own heartbeats. Without the daemon your slot goes stale and other agents will treat you as offline. +#### TensorFold status on the device card + +IAK, TensorFold, CodeWatch and GroupMind are meant to work as one system. If +you run a [TensorFold](https://github.com/ThinkOffApp/TensorFold) server, the +heartbeat can carry its live status (model, memory, requests, tok/s, draft +acceptance, time to first token, KV cache use, Living Weights) so CodeWatch +can show a card for it on each machine. + +1. List the server in `config/models.json` with `"engine": "tensorfold"` (or + leave `engine` out and let it be auto-detected). See + `config/models.README.md`. +2. Check what the card will say: `ide-agent-kit tensorfold-status` (add + `--json` for the exact object). +3. Turn it on for the publisher: set + `INTENT_TENSORFOLD_REGISTRY=/path/to/config/models.json` in the + environment of `uik-daemon` or `ide-agent-kit intent daemon`, and restart it. + +It is read with GET requests only (`/health`, `/metrics`, `/v1/slide/graph`), +never a completion, and an unreachable server shows as unreachable instead of +stopping the heartbeat. The payload shape is in +`packages/user-intent-kit/README.md`. + UIK v0.2.2 or later recommended for the `uik-daemon` bin. v0.2.0 introduced the daemon, v0.2.1 fixed a silent-exit in the example file, and v0.2.2 added agent-status republish on the heartbeat cadence so the agent slot does not expire while the daemon is running. Pre-0.2.2 daemons publish device state correctly but the agent slot still goes stale after its TTL. ### Enrichment Configuration @@ -1185,6 +1207,7 @@ ide-agent-kit gateway [options] ide-agent-kit memory [options] ide-agent-kit rooms archive [--backfill] [--max-pages N] [--room R] ide-agent-kit rooms search "words" [--from handle] [--since 2026-09-20] [--regex] [--limit N] +ide-agent-kit tensorfold-status [--json] [--registry ] [--local] [--timeout ] ide-agent-kit init [--ide ] [--profile ] ide-agent-kit acp [options] ide-agent-kit keepalive [--pid-file ] [--heartbeat-sec ] diff --git a/bin/cli.mjs b/bin/cli.mjs index c53b99d..61ad5c7 100755 --- a/bin/cli.mjs +++ b/bin/cli.mjs @@ -100,6 +100,11 @@ Usage: state: overall_state + reachability + background gate summary patch: update this device's slot: intent patch key=value [key=value ...] daemon: long-running publisher (desktop state + agent heartbeat) [--interval ] + Add INTENT_TENSORFOLD_REGISTRY= to publish TensorFold status on each beat. + + ide-agent-kit tensorfold-status [--json] [--registry ] [--local] [--timeout ] [--config ] + Live status of every TensorFold server in the model registry, read with GET only + (/health, /metrics, /v1/slide/graph). Same object the device heartbeat publishes. ide-agent-kit poll --rooms --api-key --handle <@handle> [--interval ] [--config ] (Legacy) Poll GroupMind rooms with explicit CLI args. Prefer "rooms watch". @@ -562,6 +567,22 @@ async function main() { process.exit(1); } + // ── TensorFold status (GET only) ─────────────────────── + if (command === 'tensorfold-status') { + const opts = parseKV(args, 'tensorfold-status'); + const config = loadConfig(opts.config); + const { resolveTensorFoldRegistryPath, runTensorFoldStatus } = await import('../src/tensorfold-status.mjs'); + const registryPath = resolveTensorFoldRegistryPath({ cliPath: typeof opts.registry === 'string' ? opts.registry : undefined, config }); + const timeoutMs = opts.timeout ? Number(opts.timeout) : undefined; + if (timeoutMs !== undefined && (!Number.isFinite(timeoutMs) || timeoutMs <= 0)) { + console.error('Error: --timeout must be a positive number of milliseconds'); + process.exit(2); + } + const result = await runTensorFoldStatus({ registryPath, scope: opts.local ? 'local' : 'all', json: !!opts.json, timeoutMs }); + (result.exitCode === 0 ? console.log : console.error)(result.text); + process.exit(result.exitCode); + } + // ── Rooms ────────────────────────────────────────────── if (command === 'rooms') { const opts = parseKV(args, subcommand || 'rooms'); diff --git a/config/models.README.md b/config/models.README.md index daea29e..24b0281 100644 --- a/config/models.README.md +++ b/config/models.README.md @@ -5,7 +5,7 @@ Copy the example to `config/models.json` and edit it; `config/models.json` is the file `bin/model-capacity.mjs` reads by default. ```json -{ "models": [ { "id": ..., "host": ..., "port": ..., "kind": ..., "box": ..., "sharing": ..., "owner": ..., "auth": ..., "keyFile": ... } ] } +{ "models": [ { "id": ..., "host": ..., "port": ..., "kind": ..., "engine": ..., "box": ..., "sharing": ..., "owner": ..., "auth": ..., "keyFile": ... } ] } ``` A bare top-level array is also accepted. @@ -18,6 +18,7 @@ A bare top-level array is also accepted. | `host` | yes | **Tailnet MagicDNS name** (`asus1`, `mini`) or tailnet IP (`100.x.y.z`). See the rule below. | | `port` | yes | Integer 1-65535. The OpenAI-style HTTP port, not the ssh port. | | `kind` | yes | `openai` \| `vllm` \| `lmstudio`. Decides which list endpoint is called: `lmstudio` uses `/api/v1/models` and counts only loaded instances, the other two use `/v1/models`. | +| `engine` | no | Which server software answers on this port. `tensorfold` turns on the TensorFold status read-out for this entry (see below). Any other short name (`vllm`, `llama.cpp`, `ollama`) says "not TensorFold" and skips auto-detection. Omit it to let the status reader auto-detect. | | `box` | no | Free text for humans: which machine, where it lives. Shown in `--json`, never probed. | | `sharing` | no | `exclusive` \| `shared`, default `shared`. Advisory metadata for the switcher's ranking. The probe applies the same BUSY rules to both - it is documented in `docs/model-capacity.md` rather than silently inferred. | | `owner` | no | Handle of whoever to ask before taking the box. | @@ -64,6 +65,42 @@ two hosts behind the same router were in opposite path states to the same Berlin box at the same moment. A value written into this shared file is wrong for every caller except the one who wrote it. +## TensorFold entries + +`ide-agent-kit tensorfold-status` and the device heartbeat (when +`INTENT_TENSORFOLD_REGISTRY` points at this file) read the live status of +every TensorFold server listed here: model, warming, memory, requests, token +counters, tok/s, draft acceptance, time to first token, KV cache use and +Living Weights. They send **GET /health, GET /metrics and GET /v1/slide/graph +only**, never a completion. + +Which entries count as TensorFold: + +* `"engine": "tensorfold"`: always, reachable or not. +* no `engine`: auto-detected when `/metrics` carries `tensorfold:` series or + `/health` has TensorFold's shape. A server started with an API key hides + both, so mark a keyed server `"engine": "tensorfold"` explicitly. +* any other `engine`: never asked. + +**Keys.** A TensorFold server may gate `/metrics` and `/v1/slide/graph` +behind a key (`/health` stays open, and says only `{"status": "ok"}` while +keys are on). The status reader sends a key only to an entry marked +`"engine": "tensorfold"` that names its own `keyFile`, and only on those two +routes. It never uses `LLM_API_KEY` / `LLM_API_KEY_FILE`: those belong to some +other server as far as a status read knows. Without a usable key it reports +`metrics: "auth_required"` and the rest of the card reads unknown. + +```json +{ + "id": "gb10-tensorfold", + "host": "asus3", + "port": 8080, + "kind": "openai", + "engine": "tensorfold", + "keyFile": "~/.iak/tensorfold_gb10_api_key.txt" +} +``` + ## The rule: tailnet names, never LAN IPs `host` must not be a private LAN address. The loader rejects `10.x.x.x`, diff --git a/config/models.example.json b/config/models.example.json index 23d7d3a..d66e234 100644 --- a/config/models.example.json +++ b/config/models.example.json @@ -28,6 +28,28 @@ "box": "Mac mini (always-on)", "sharing": "shared", "owner": "@myagent" + }, + { + "id": "mb-tensorfold", + "host": "localhost", + "port": 8091, + "kind": "openai", + "engine": "tensorfold", + "box": "This Mac, TensorFold serving one model (loopback only)", + "sharing": "shared", + "owner": "@myagent" + }, + { + "id": "gb10-tensorfold", + "host": "asus3", + "port": 8080, + "kind": "openai", + "engine": "tensorfold", + "box": "ASUS Ascent GX10 (GB10), native TensorFold started with an API key file", + "sharing": "exclusive", + "owner": "@myagent", + "auth": "bearer", + "keyFile": "~/.iak/example_tensorfold_api_key.txt" } ] } diff --git a/packages/user-intent-kit/README.md b/packages/user-intent-kit/README.md index 8d60234..6366c43 100644 --- a/packages/user-intent-kit/README.md +++ b/packages/user-intent-kit/README.md @@ -123,6 +123,89 @@ export INTENT_MODEL_ENDPOINT=127.0.0.1:8080 as a last resort for a box with no endpoint to ask; a server that answers always outranks it. +### TensorFold servers on the device card + +Point the daemon at a model registry (ide-agent-kit's `config/models.json`) +and each TensorFold server in it gets its live status on this machine's +device card: + +```bash +export INTENT_TENSORFOLD_REGISTRY=~/ide-agent-kit/config/models.json +export INTENT_TENSORFOLD_SCOPE=local # default; `all` also reads remote entries +``` + +- **GET only.** `/health`, `/metrics` and `/v1/slide/graph`, never a + completion, so a status read cannot interfere with a Living Weights lesson. +- **Off unless set.** Nothing is read or published without the registry path. +- **Not on the heartbeat's path.** It reads on its own timer at the heartbeat + interval; the beat publishes the last reading and never waits on a socket. +- **One key, replaced whole.** Everything goes under `tensorfold`, so a server + removed from the registry disappears from the card on the next beat. +- **Unreachable is a reading.** A server that does not answer is published as + `reachable: false` with the reason; it never takes the heartbeat down. +- **Which entries:** marked `"engine": "tensorfold"`, or auto-detected from + their `/metrics`. See `config/models.README.md` in ide-agent-kit, including + how a keyed server names its key. +- **TensorFold's default port is 8080, and so is `INTENT_MODEL_ENDPOINT`'s.** + The served-model probe above may send one short generation request to a + loopback endpoint to prove what is loaded. On a machine where TensorFold + listens on 8080, set `INTENT_MODEL_GENERATE=0` so the heartbeat never asks it + for a token. + +The payload, per device: + +```json +"tensorfold": { + "schema": 1, + "checked_at": "2026-10-14T09:00:00.000Z", + "scope": "local", + "error": null, + "servers": [{ + "id": "mb-tensorfold", "host": "localhost", "port": 8091, "local": true, + "source": "registry", "reachable": true, "reason": null, + "model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit", "warming": false, + "version": null, "backend": null, "device": null, + "memory": { "used_gb": 18.7, "peak_gb": 40.7, "budget_gb": 96.2 }, + "requests": { "running": 0, "waiting": 0 }, + "tokens": { "prompt_total": 0, "generation_total": 0 }, + "tok_s": null, "tok_s_window_s": null, "decode_tok_s": null, + "draft": { "drafted": 0, "accepted": 0, "acceptance": null }, + "ttft_p50_s": null, "kv_cache_usage": 0, + "living_weights": { "available": false, "facts": null, "learning": null, "learned": null }, + "metrics": "ok", + "checked_at": "2026-10-14T09:00:00.000Z" + }] +} +``` + +A value that could not be read is `null`, never `0`. `error` is set (and +`servers` empty) when the registry itself could not be read, so "no servers" +and "could not tell" never look the same. + +What the numbers mean: + +- `local`: the server runs on this machine (loopback entry). `source` is + `registry` (marked `"engine": "tensorfold"`) or `detected`. +- `memory`: GB, base 10, one decimal, like `mem_total_gb`. `used` is the + engine's active memory, `budget` its configured budget. +- `tokens`: counters since the server started, for finished requests. +- `tok_s`: generated tokens of requests that finished between two reads, + divided by the seconds between them (`tok_s_window_s`). `null` on the first + read and after a server restart. `decode_tok_s` is the server's own live + decode rate, where the server reports one (the native build does). +- `draft.acceptance`: accepted over drafted MTP tokens since start, `null` + until something was drafted. +- `ttft_p50_s`: median time to first token since start, estimated from the + histogram buckets. +- `kv_cache_usage`: the fullest stream's cache use, 0 to 1. +- `living_weights.available`: `/v1/slide/graph` answered (`false` on a 404, + `null` when it could not be read). Counts only: the fact text stays on the + machine. The native build answers the route even without `--slide`, so + there `facts` is the better signal. +- `metrics`: `ok`, `auth_required`, `auth_rejected` or `unavailable`. +- `version`, `backend`, `device`: read from `/health` when the server reports + them (newer builds), `null` otherwise. + > Note: The older `examples/iak-integration.js` is a one-shot demo and will exit once its setInterval timer is unref'd. Use `uik-daemon` for long-running deployments. ## Quick start diff --git a/packages/user-intent-kit/bin/uik-daemon.js b/packages/user-intent-kit/bin/uik-daemon.js index dbefdb1..6b4193d 100755 --- a/packages/user-intent-kit/bin/uik-daemon.js +++ b/packages/user-intent-kit/bin/uik-daemon.js @@ -66,12 +66,27 @@ * Defaults under ~/.local/state/user-intent-kit per * user/device. Preserve it across restarts. Run exactly * ONE chooser-enabled device publisher per device. + * INTENT_TENSORFOLD_REGISTRY optional path to a model registry + * (config/models.json in ide-agent-kit). OFF when + * unset. Each TensorFold entry in it (marked + * "engine": "tensorfold", or auto-detected from its + * /metrics) is read with GET /health, GET /metrics + * and GET /v1/slide/graph on the heartbeat cadence + * and published under the device's `tensorfold` key. + * Never a generation request. An unreachable server + * publishes reachable:false; the heartbeat never + * waits on it. + * INTENT_TENSORFOLD_SCOPE local (default) | all. local = only entries on + * loopback, i.e. servers on THIS machine, so each + * machine's card carries its own servers. all = every + * entry, for a box that cannot run this daemon. */ import { hostname } from 'node:os'; import { execSync } from 'node:child_process'; import { IntentClient, IAKAdapter, DesktopAdapter, ServedModelProbe, ModelAvailabilityProbe, defaultSearchRoots, + TensorFoldMonitor, } from '../src/index.js'; const baseUrl = process.env.INTENT_API_BASE || 'https://groupmind.one/api/v1'; @@ -115,10 +130,18 @@ if (userId.toLowerCase() === agentHandle.replace(/^@/, '').toLowerCase()) { ); } +let tensorfold; +try { + tensorfold = new TensorFoldMonitor({ intervalMs: Math.min(pollIntervalMs, 60000) }); +} catch (err) { + console.error(`uik-daemon: ${err.message}`); + process.exit(1); +} + const client = new IntentClient({ baseUrl, apiKey, userId, deviceId }); const iak = new IAKAdapter(client, { agentHandle, machine: deviceId }); const desktop = new DesktopAdapter(client, { - pollIntervalMs, machine: deviceId, kind: deviceKind, model: deviceModel, modelProbe, availabilityProbe, + pollIntervalMs, machine: deviceId, kind: deviceKind, model: deviceModel, modelProbe, availabilityProbe, tensorfold, }); if (publishDevice) desktop.start(); @@ -158,6 +181,10 @@ console.log(`uik-daemon: model scan -> ${defaultSearchRoots().join(', ')}`); // loopback endpoint, off for anything else. Where it is off the honest // reading is `listed`, and the card says so rather than saying serving. console.log(`uik-daemon: generation -> ${modelProbe.generates ? 'on; a token proves what is loaded' : 'off; models report as listed, never served'}`); +// The TensorFold read-out, and only where it reads from. It is GET-only by +// construction; the line exists so a card without servers can be traced to +// "off" rather than guessed at. +console.log(`uik-daemon: tensorfold -> ${publishDevice ? tensorfold.describe() : 'off (INTENT_DEVICE_PUBLISH=0)'}`); const shutdown = async (sig) => { console.log(`uik-daemon: ${sig}, shutting down`); diff --git a/packages/user-intent-kit/src/adapters/desktop.js b/packages/user-intent-kit/src/adapters/desktop.js index c37f717..a418cf5 100644 --- a/packages/user-intent-kit/src/adapters/desktop.js +++ b/packages/user-intent-kit/src/adapters/desktop.js @@ -10,6 +10,7 @@ import { ServedModelProbe } from '../served-model.js'; import { ModelAvailabilityProbe } from '../model-availability.js'; import { StatePublisher } from '../state-publisher.js'; import { ModelSwitcher } from '../model-switcher.js'; +import { TensorFoldMonitor } from '../tensorfold-status.js'; /** * Desktop Adapter - detects active window and context on macOS. @@ -30,6 +31,7 @@ export class DesktopAdapter { #pollIntervalMs; #publisher; #switcher; + #tensorfold; /** * @param {import('../client.js').IntentClient} client @@ -43,10 +45,14 @@ export class DesktopAdapter { * @param {import('../model-availability.js').ModelAvailabilityProbe|null} [opts.availabilityProbe] * - what this box has on disk and could fit. Publishes the WEAKER claim, * in its own fields; pass null to turn the disk scan off entirely. + * @param {import('../tensorfold-status.js').TensorFoldMonitor|null} [opts.tensorfold] + * - GET-only TensorFold status for the `tensorfold` key. Omit it for the + * environment's configuration (OFF unless INTENT_TENSORFOLD_REGISTRY is + * set); pass null to turn it off whatever the environment says. */ constructor(client, { pollIntervalMs = 30000, machine, kind, model, modelProbe, availabilityProbe, - hostSources, modelSwitcher, chooserUrl, + hostSources, modelSwitcher, chooserUrl, tensorfold, } = {}) { this.#client = client; this.#switcher = modelSwitcher ?? new ModelSwitcher(client, { chooserUrl }); @@ -73,6 +79,11 @@ export class DesktopAdapter { if (!Number.isFinite(pollIntervalMs) || pollIntervalMs <= 0) throw new Error('Invalid pollIntervalMs'); // One timer owns both telemetry and liveness. Default server device TTL is 90s. this.#pollIntervalMs = Math.min(pollIntervalMs, 60000); + // Same cadence as the heartbeat, on its own timer: the beat reads the + // last snapshot and never waits on a TensorFold socket. + this.#tensorfold = tensorfold === undefined + ? new TensorFoldMonitor({ intervalMs: this.#pollIntervalMs }) + : tensorfold; this.#publisher = new StatePublisher(fields => this.#client.patchDevice(fields), { refreshMs: this.#pollIntervalMs, }); @@ -150,6 +161,12 @@ export class DesktopAdapter { // the heartbeat's path and the first beat goes out without waiting for it. if (!this.#switcher.enabled) this.#availabilityProbe?.start(); this.#switcher.start(); + // GET-only, so it is safe beside the chooser. The first read publishes + // as soon as it lands instead of waiting a whole interval for the card. + this.#tensorfold?.start()?.then(snapshot => { + if (snapshot && this.#pollTimer) return this.publishState(); + return undefined; + }).catch(() => {}); // this.#client.startHeartbeat() removed: the StatePublisher's own // refreshMs timer now re-sends the last state as the liveness beat, so a // second heartbeat mechanism only doubled the writes (c0ed66d, 18 Sep 2026). @@ -167,6 +184,7 @@ export class DesktopAdapter { this.#modelProbe?.stop(); this.#availabilityProbe?.stop(); this.#switcher?.stop(); + this.#tensorfold?.stop(); } /** @@ -212,6 +230,9 @@ export class DesktopAdapter { // running it". Empty when there is nothing measured to say. ...this.#listedModels(), ...(this.#availabilityProbe?.current() ?? {}), + // TensorFold servers' live status, one key replaced whole each beat. + // Empty until the first GET-only read lands; absent when off. + ...(this.#tensorfold?.fields() ?? {}), }; try { diff --git a/packages/user-intent-kit/src/index.js b/packages/user-intent-kit/src/index.js index 2c7f001..c08ebc6 100644 --- a/packages/user-intent-kit/src/index.js +++ b/packages/user-intent-kit/src/index.js @@ -7,6 +7,16 @@ export { DesktopAdapter } from './adapters/desktop.js'; export { BrowserAdapter } from './adapters/browser.js'; export { collectHostTelemetry } from './host-telemetry.js'; export { ServedModelProbe, servedModelEntry, readVerdict, parseEndpoint, VERDICTS } from './served-model.js'; +export { + TensorFoldMonitor, + probeTensorFold, + collectTensorFoldStatus, + parsePrometheus, + histogramQuantile, + TENSORFOLD_ENGINE, + STATUS_SCHEMA as TENSORFOLD_STATUS_SCHEMA, + METRICS_STATES as TENSORFOLD_METRICS_STATES, +} from './tensorfold-status.js'; export { ModelAvailabilityProbe, describeLocalModels, diff --git a/packages/user-intent-kit/src/model-capacity.js b/packages/user-intent-kit/src/model-capacity.js index f805bf2..6aa679b 100644 --- a/packages/user-intent-kit/src/model-capacity.js +++ b/packages/user-intent-kit/src/model-capacity.js @@ -170,6 +170,9 @@ const CREDENTIAL_VALUE_FIELDS = Object.freeze([ // and reported as ENOENT. const LOOKS_LIKE_A_TOKEN = /^(sk-|sk_|xfb_|antfarm_|ghp_|github_pat_|Bearer\s)|^[A-Za-z0-9+/_-]{40,}={0,2}$/; +// `engine` names server software, nothing else. +const ENGINE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,31}$/; + // Command names that mean "this box is taken". Matched against the process // NAME (basename of argv[0]), never against the whole command line, so that a // path component like /var/grid/logs cannot fake a running job. @@ -291,6 +294,17 @@ export function loadRegistry(source, { allowLan = false } = {}) { 'Set "auth": "bearer", or drop the keyFile.'); } } + // engine: which server software answers on this port. Only "tensorfold" + // changes behaviour today (the status read-out); any other value says + // "not TensorFold, do not auto-detect"; absent means "auto-detect". + let engine = null; + if (raw.engine !== undefined) { + if (typeof raw.engine !== 'string' || !ENGINE_NAME.test(raw.engine.trim())) { + throw new RegistryError( + `registry entry "${id}": "engine" must be a short name such as "tensorfold", "vllm" or "llama.cpp"`); + } + engine = raw.engine.trim().toLowerCase(); + } for (const field of MEASUREMENT_ONLY_FIELDS) { if (raw[field] !== undefined) { throw new RegistryError( @@ -310,7 +324,7 @@ export function loadRegistry(source, { allowLan = false } = {}) { 'Pass --allow-lan for a deliberate single-site run.'); } return { - id, host, port, kind, sharing, auth, keyFile, + id, host, port, kind, sharing, auth, keyFile, engine, box: typeof raw.box === 'string' ? raw.box : null, owner: typeof raw.owner === 'string' ? raw.owner : null, lanOverride: Boolean(lan), @@ -386,9 +400,13 @@ const NO_TOKEN = Object.freeze({ token: null, keySource: null, keyBlocked: false * A keyFile that is configured but unreadable does NOT fall through to the * environment. Quietly substituting a different credential for the one the * operator named would make "which key was refused?" unanswerable. + * + * `entryOnly` stops after the entry's own sources. The TensorFold status + * reader uses it: the fleet-wide LLM_API_KEY is some other server's key as + * far as a status read knows, and a key goes only where it was named. */ export async function resolveEntryToken(entry, { - env = process.env, readFileImpl = readFile, statImpl = stat, + env = process.env, readFileImpl = readFile, statImpl = stat, entryOnly = false, } = {}) { const auth = entry?.auth ?? (entry?.keyFile ? 'bearer' : 'none'); if (auth !== 'bearer') return { ...NO_TOKEN, auth }; @@ -396,6 +414,7 @@ export async function resolveEntryToken(entry, { if (entry?.keyFile) { return { ...(await readKeyFile(entry.keyFile, KEY_SOURCES.ENTRY_FILE, io)), auth }; } + if (entryOnly) return { ...NO_TOKEN, auth }; const envPath = typeof env?.LLM_API_KEY_FILE === 'string' ? env.LLM_API_KEY_FILE.trim() : ''; if (envPath) { return { ...(await readKeyFile(envPath, KEY_SOURCES.ENV_FILE, io)), auth }; diff --git a/packages/user-intent-kit/src/tensorfold-status.js b/packages/user-intent-kit/src/tensorfold-status.js new file mode 100644 index 0000000..9fbcc49 --- /dev/null +++ b/packages/user-intent-kit/src/tensorfold-status.js @@ -0,0 +1,634 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// +// tensorfold-status.js - what a TensorFold server is doing right now, read +// with GETs only and folded into an object small enough to ride on a device +// heartbeat. +// +// THREE GETS AND NOTHING ELSE. /health (always open), /metrics (Prometheus +// text, may need a key) and /v1/slide/graph (the Living Weights fact graph, +// 404 on a server started without --slide). Never a completion, never a chat +// request, never /health?reset_peak=1. A generated token during a Living +// Weights lesson is a lesson the heartbeat interfered with, and a status +// reader that can change what it reads is not a status reader. +// +// NEVER THROWS. Every failure becomes a field: an unreachable server is +// `reachable: false` with the reason, a 401 on /metrics is +// `metrics: "auth_required"`, a 404 graph is `living_weights.available: +// false`. A value we could not read is null, never 0: an idle server and an +// unreadable counter must not look the same on a dashboard. +// +// KEYS. A key is sent only to an entry the registry marks +// `"engine": "tensorfold"`, only when that same entry names its own key +// (`keyFile`, a path), and only on the two gated routes. The fleet-wide +// LLM_API_KEY / LLM_API_KEY_FILE fallbacks are NOT used here: those are some +// other server's credential as far as this reader knows. Auto-detected +// entries are probed without any key at all. Nothing here logs or returns a +// token; Prometheus `key="..."` labels (the native server labels its request +// counter by API key label) are never copied into the output either. +// +// FACT TEXT STAYS ON THE BOX. The slide graph carries what the model was +// taught, which can be private. Only counts leave this module. + +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { readRegistryFile, resolveEntryToken } from './model-capacity.js'; + +export const TENSORFOLD_ENGINE = 'tensorfold'; +export const STATUS_SCHEMA = 1; +export const DEFAULT_STATUS_TIMEOUT_MS = 3000; +/** A port that answered "not TensorFold" is asked again after this long. */ +export const DETECT_RECHECK_MS = 10 * 60 * 1000; +export const SCOPES = Object.freeze(['local', 'all']); +export const METRICS_STATES = Object.freeze({ + OK: 'ok', + AUTH_REQUIRED: 'auth_required', + AUTH_REJECTED: 'auth_rejected', + UNAVAILABLE: 'unavailable', +}); + +const MAX_BODY_BYTES = 1 << 20; +const MAX_GRAPH_BYTES = 8 << 20; +const BYTES_PER_GB = 1e9; // base 10, matching mem_total_gb in host-telemetry.js +const MIN_RATE_WINDOW_MS = 1000; + +// --- small parsers ---------------------------------------------------------- + +const isObject = v => v !== null && typeof v === 'object' && !Array.isArray(v); +const finite = v => (typeof v === 'number' && Number.isFinite(v) ? v : null); +const round = (v, places) => (v === null ? null : Math.round(v * 10 ** places) / 10 ** places); +const count = v => (Number.isFinite(v) && v >= 0 ? Math.round(v) : null); +const gb = bytes => (bytes === null || bytes < 0 ? null : round(bytes / BYTES_PER_GB, 1)); + +/** + * A short human label from /health (`version`, `backend`, `device`). The + * fields do not exist yet on 0.6.x servers, so absent is the normal case and + * reads as null. A `device` that turns out to be an object with a `name` is + * accepted too, because the shape is not settled upstream. + */ +export function healthLabel(value) { + let v = value; + if (isObject(v) && typeof v.name === 'string') v = v.name; + if (typeof v === 'number' && Number.isFinite(v)) v = String(v); + if (typeof v !== 'string') return null; + // eslint-disable-next-line no-control-regex + const clean = v.replace(/[\u0000-\u001f\u007f]/g, '').trim(); + return clean ? clean.slice(0, 120) : null; +} + +function unescapeLabel(s) { + return s.replace(/\\(.)/g, (_, c) => (c === 'n' ? '\n' : c)); +} + +/** + * Prometheus text exposition (0.0.4) into [{name, labels, value}]. Comment + * and malformed lines are skipped rather than fatal: one odd line from a + * newer server must not blank every other number on the card. + */ +export function parsePrometheus(text) { + const out = []; + for (const raw of String(text ?? '').split('\n')) { + const line = raw.trim(); + if (!line || line.startsWith('#')) continue; + const m = /^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{(.*)\})?\s+(\S+)(?:\s+-?\d+)?$/.exec(line); + if (!m) continue; + let value; + if (m[3] === '+Inf') value = Infinity; + else if (m[3] === '-Inf') value = -Infinity; + else value = Number(m[3]); + if (Number.isNaN(value)) continue; + const labels = {}; + for (const lm of (m[2] ?? '').matchAll(/([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*"((?:[^"\\]|\\.)*)"/g)) { + labels[lm[1]] = unescapeLabel(lm[2]); + } + out.push({ name: m[1], labels, value }); + } + return out; +} + +/** + * `histogram_quantile` over cumulative buckets [{le, count}], with the same + * linear interpolation Prometheus uses. It is an ESTIMATE bounded by the + * bucket edges, and null when nothing was observed. + */ +export function histogramQuantile(q, buckets) { + const sorted = (buckets || []) + .filter(b => Number.isFinite(b.count) && (Number.isFinite(b.le) || b.le === Infinity)) + .sort((a, b) => a.le - b.le); + if (!sorted.length) return null; + const total = sorted[sorted.length - 1].count; + if (!(total > 0)) return null; + const rank = q * total; + let prevLe = 0; + let prevCount = 0; + for (const b of sorted) { + if (b.count >= rank) { + if (b.le === Infinity) return sorted.length > 1 ? prevLe : null; + if (b.count === prevCount) return b.le; + return prevLe + (b.le - prevLe) * ((rank - prevCount) / (b.count - prevCount)); + } + prevLe = b.le; + prevCount = b.count; + } + return null; +} + +/** Does this /health body have TensorFold's shape? Used only to auto-detect. */ +export function looksLikeTensorFoldHealth(body) { + return isObject(body) + && typeof body.status === 'string' + && Array.isArray(body.model_ids) + && Number.isFinite(body.max_batch_size) + && typeof body.warming === 'boolean'; +} + +/** The /health fields this module uses. Missing fields are null. */ +export function summarizeHealth(body) { + const h = isObject(body) ? body : {}; + const mem = isObject(h.memory) ? h.memory : {}; + return { + status: typeof h.status === 'string' ? h.status.slice(0, 40) : null, + model: healthLabel(h.model), + warming: typeof h.warming === 'boolean' ? h.warming : null, + version: healthLabel(h.version), + backend: healthLabel(h.backend), + device: healthLabel(h.device), + memory: { active: finite(mem.active), peak: finite(mem.peak), budget: finite(mem.budget) }, + // The native server's own live decode rate (`live` block in /health). + decodeTokS: finite(isObject(h.live) ? h.live.decode_tokens_per_second : undefined), + }; +} + +/** The `tensorfold:` series this module uses. Missing series are null. */ +export function summarizeMetrics(series) { + const tf = (series || []).filter(s => s.name.startsWith('tensorfold:')); + const one = name => { + const s = tf.find(x => x.name === `tensorfold:${name}`); + return s ? s.value : null; + }; + const kv = tf.filter(s => s.name === 'tensorfold:kv_cache_usage_ratio').map(s => s.value).filter(Number.isFinite); + const buckets = tf + .filter(s => s.name === 'tensorfold:time_to_first_token_seconds_bucket' && s.labels.le !== undefined) + .map(s => ({ le: s.labels.le === '+Inf' ? Infinity : Number(s.labels.le), count: s.value })); + const drafted = count(one('mtp_drafted_total')); + const accepted = count(one('mtp_accepted_total')); + return { + detected: tf.length > 0, + requests: { running: count(one('requests_running')), waiting: count(one('requests_waiting')) }, + tokens: { prompt_total: count(one('prompt_tokens_total')), generation_total: count(one('generation_tokens_total')) }, + draft: { + drafted, + accepted, + acceptance: drafted > 0 && accepted !== null ? round(accepted / drafted, 3) : null, + }, + ttftP50: round(histogramQuantile(0.5, buckets), 3), + kvCacheUsage: kv.length ? round(Math.max(...kv), 3) : null, + // The native build reports engine memory here (CUDA), and /health hides + // its memory block when keys are on, so this is the fallback for `used`. + deviceMemory: finite(one('device_memory_bytes')), + deviceMemoryPeak: finite(one('device_memory_peak_bytes')), + }; +} + +/** Counts from /v1/slide/graph. Never the fact text. */ +export function summarizeSlideGraph(body) { + if (!isObject(body) || !Array.isArray(body.facts)) { + return { available: null, facts: null, learning: null, learned: null }; + } + const states = body.facts.map(f => (isObject(f) ? f.state : null)); + return { + available: true, + facts: body.facts.length, + learning: states.filter(s => s === 'learning').length, + learned: states.filter(s => s === 'learned').length, + }; +} + +// --- HTTP ------------------------------------------------------------------- + +function errorCodes(err) { + const all = []; + for (let e = err, depth = 0; e && depth < 6; e = e.cause, depth++) { + if (e.code) all.push(String(e.code)); + if (e.name) all.push(String(e.name)); + } + return all; +} + +function describeFetchError(err, timeoutMs) { + const all = errorCodes(err); + if (all.includes('TimeoutError') || all.includes('AbortError')) return `no response within ${timeoutMs} ms`; + const code = all.find(c => /^E[A-Z]+$/.test(c) || c.startsWith('UND_ERR')); + if (code) return `connect failed (${code})`; + return `request failed (${all.find(c => c !== 'TypeError' && c !== 'Error') || 'fetch error'})`; +} + +async function readCapped(response, maxBytes) { + if (!response.body?.getReader) { + const text = await response.text(); + if (text.length > maxBytes) throw new Error('response too large'); + return text; + } + const reader = response.body.getReader(); + const chunks = []; + let size = 0; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + size += value.byteLength; + if (size > maxBytes) { + await reader.cancel().catch(() => {}); + throw new Error('response too large'); + } + chunks.push(value); + } + return new TextDecoder().decode(Buffer.concat(chunks.map(c => Buffer.from(c)))); +} + +/** + * One GET. Resolves to {status, text, error}; never rejects. `status` is null + * when no HTTP answer came back at all. Never follows a redirect: a 302 is an + * unknown destination and following it would hand the key to it. + */ +async function get(url, { fetchImpl, timeoutMs, token = null, maxBytes = MAX_BODY_BYTES, accept = 'application/json' }) { + // `connection: close`: a status read every half minute has no use for a + // kept-alive socket, and the Python server parks a thread on every idle + // one. Measured 2026-10-14: without it, a CLI run that exited with the + // socket still open left a ConnectionResetError traceback in the 0.6.0 + // server's log. + const headers = { accept, connection: 'close' }; + if (token) headers.authorization = `Bearer ${token}`; + let response; + try { + response = await fetchImpl(url, { method: 'GET', headers, redirect: 'error', signal: AbortSignal.timeout(timeoutMs) }); + } catch (err) { + return { status: null, text: null, error: describeFetchError(err, timeoutMs) }; + } + if (!response.ok) { + try { await response.body?.cancel?.(); } catch { /* nothing to drain */ } + return { status: response.status, text: null, error: `HTTP ${response.status}` }; + } + try { + return { status: response.status, text: await readCapped(response, maxBytes), error: null }; + } catch (err) { + return { status: response.status, text: null, error: err.message === 'response too large' ? 'response too large' : describeFetchError(err, timeoutMs) }; + } +} + +function parseJson(text) { + try { return JSON.parse(text); } catch { return null; } +} + +export function baseUrlOf(target) { + const host = String(target.host); + const bracket = host.includes(':') && !host.startsWith('[') ? `[${host}]` : host; + return `http://${bracket}:${target.port}`; +} + +export function isLoopbackHost(host) { + const h = String(host || '').trim().toLowerCase().replace(/^\[|\]$/g, ''); + return h === 'localhost' || h === '::1' || /^127\.\d{1,3}\.\d{1,3}\.\d{1,3}$/.test(h); +} + +async function readCore(base, io, token) { + const [health, metrics] = await Promise.all([ + get(`${base}/health`, io), + get(`${base}/metrics`, { ...io, token, accept: 'text/plain' }), + ]); + return { health, metrics }; +} + +function readSlide(base, io, token) { + return get(`${base}/v1/slide/graph`, { ...io, token, maxBytes: MAX_GRAPH_BYTES }); +} + +// --- the status object -------------------------------------------------------- + +/** + * Fold the three answers into the published shape. `previous` is the last + * sample for this server ({generation_total, at}) or null; the returned + * `sample` is what to keep for the next call. + */ +export function buildStatus(target, { health, metrics, slide }, { + tokenSent = false, previous = null, clockMs, checkedAt, keyNote = null, +} = {}) { + const healthBody = health?.status === 200 ? parseJson(health.text) : null; + const h = summarizeHealth(healthBody); + const series = metrics?.status === 200 ? parsePrometheus(metrics.text) : []; + const m = summarizeMetrics(series); + + let metricsState; + if (metrics?.status === 401 || metrics?.status === 403) { + metricsState = tokenSent ? METRICS_STATES.AUTH_REJECTED : METRICS_STATES.AUTH_REQUIRED; + } else if (metrics?.status === 200 && m.detected) { + metricsState = METRICS_STATES.OK; + } else { + metricsState = METRICS_STATES.UNAVAILABLE; + } + const metricsOk = metricsState === METRICS_STATES.OK; + + const reachable = health?.status != null || metrics?.status != null || slide?.status != null; + const reasons = []; + if (!reachable) { + reasons.push(health?.error || metrics?.error || 'no answer'); + } else { + if (health?.status !== 200) reasons.push(`/health ${health?.error || 'no answer'}`); + else if (!healthBody) reasons.push('/health was not JSON'); + else if (h.status && h.status !== 'ok') reasons.push(`/health status "${h.status}"`); + if (metricsState === METRICS_STATES.AUTH_REQUIRED) reasons.push('metrics: auth required'); + else if (metricsState === METRICS_STATES.AUTH_REJECTED) reasons.push('metrics: key refused'); + else if (!metricsOk) { + const readable = metrics?.status === 200 && typeof metrics.text === 'string'; + reasons.push(readable ? 'metrics: no tensorfold series' : `metrics: ${metrics?.error || 'no answer'}`); + } + if (keyNote && !metricsOk) reasons.push(keyNote); + } + + let living = { available: null, facts: null, learning: null, learned: null }; + if (slide?.status === 200) living = summarizeSlideGraph(parseJson(slide.text)); + else if (slide?.status === 404) living = { available: false, facts: null, learning: null, learned: null }; + + const generation = metricsOk ? m.tokens.generation_total : null; + let tokS = null; + let windowS = null; + if (previous && generation !== null && Number.isFinite(previous.generation_total) + && generation >= previous.generation_total && clockMs - previous.at >= MIN_RATE_WINDOW_MS) { + windowS = round((clockMs - previous.at) / 1000, 1); + tokS = round((generation - previous.generation_total) / ((clockMs - previous.at) / 1000), 1); + } + // A counter that went DOWN means the server restarted: no rate this time, + // and the new value is the baseline for the next one. + const sample = generation !== null ? { generation_total: generation, at: clockMs } : previous; + + const status = { + id: target.id ?? null, + host: target.host, + port: target.port, + local: isLoopbackHost(target.host), + source: target.source ?? 'registry', + reachable, + reason: reasons.length ? reasons.join('; ').slice(0, 300) : null, + model: h.model, + warming: h.warming, + version: h.version, + backend: h.backend, + device: h.device, + memory: { + used_gb: gb(h.memory.active ?? (metricsOk ? m.deviceMemory : null)), + peak_gb: gb(h.memory.peak ?? (metricsOk ? m.deviceMemoryPeak : null)), + budget_gb: gb(h.memory.budget), + }, + requests: metricsOk ? m.requests : { running: null, waiting: null }, + tokens: metricsOk ? m.tokens : { prompt_total: null, generation_total: null }, + tok_s: tokS, + tok_s_window_s: windowS, + decode_tok_s: h.decodeTokS === null ? null : round(h.decodeTokS, 1), + draft: metricsOk ? m.draft : { drafted: null, accepted: null, acceptance: null }, + ttft_p50_s: metricsOk ? m.ttftP50 : null, + kv_cache_usage: metricsOk ? m.kvCacheUsage : null, + living_weights: living, + metrics: metricsState, + checked_at: checkedAt, + }; + return { status, sample, detected: m.detected || looksLikeTensorFoldHealth(healthBody) }; +} + +/** + * Probe one server that is known to be TensorFold: all three GETs in + * parallel. `token` is sent on /metrics and /v1/slide/graph only, never on + * /health (open by design; the native server answers it the same with or + * without a key). + */ +export async function probeTensorFold(target, { + fetchImpl = (...a) => fetch(...a), + timeoutMs = DEFAULT_STATUS_TIMEOUT_MS, + now = () => Date.now(), + clock = () => performance.now(), + token = null, + keyNote = null, + previous = null, +} = {}) { + const base = baseUrlOf(target); + const io = { fetchImpl, timeoutMs }; + const [core, slide] = await Promise.all([readCore(base, io, token), readSlide(base, io, token)]); + return buildStatus(target, { ...core, slide }, { + tokenSent: Boolean(token), previous, clockMs: clock(), checkedAt: new Date(now()).toISOString(), keyNote, + }); +} + +// --- the registry ----------------------------------------------------------- + +function expandHome(path) { + const s = String(path); + if (s === '~') return homedir(); + if (s.startsWith('~/')) return join(homedir(), s.slice(2)); + return s; +} + +/** + * The key for an entry the registry marks TensorFold, from that entry's own + * keyFile and nothing wider. Returns {token, note}; `note` says why a key + * could not be used, and never contains the key. + */ +async function entryToken(entry, env) { + if (entry.auth !== 'bearer') return { token: null, note: null }; + const credential = await resolveEntryToken(entry, { env, entryOnly: true }); + if (credential.token) return { token: credential.token, note: null }; + if (!entry.keyFile) return { token: null, note: 'the entry names no keyFile' }; + return { token: null, note: credential.keyWarning || null }; +} + +/** + * Status for every TensorFold server in a loaded registry. + * + * engine "tensorfold" always reported, reachable or not. + * engine absent auto-detected: reported when /metrics carries + * `tensorfold:` series or /health has TensorFold's + * shape. A port that answered as something else is + * asked again after DETECT_RECHECK_MS; one that was + * TensorFold earlier in this process and stops + * answering stays on the list as unreachable. + * any other engine skipped, no request sent. + * + * `samples` and `detected` are Maps the caller keeps between calls (the + * heartbeat does; a one-shot CLI passes fresh ones). Never throws. + */ +export async function collectTensorFoldStatus(entries, { + scope = 'all', + fetchImpl = (...a) => fetch(...a), + timeoutMs = DEFAULT_STATUS_TIMEOUT_MS, + now = () => Date.now(), + clock = () => performance.now(), + env = process.env, + samples = new Map(), + detected = new Map(), +} = {}) { + const io = { fetchImpl, timeoutMs }; + const inScope = (entries || []).filter(e => scope === 'all' || isLoopbackHost(e.host)); + const results = await Promise.all(inScope.map(async entry => { + const key = `${String(entry.host).toLowerCase()}:${entry.port}`; + const target = { id: entry.id, host: entry.host, port: entry.port }; + try { + if (entry.engine && entry.engine !== TENSORFOLD_ENGINE) { + return { skipped: { id: entry.id, reason: `registry says engine "${entry.engine}"` } }; + } + if (entry.engine === TENSORFOLD_ENGINE) { + const { token, note } = await entryToken(entry, env); + const { status, sample } = await probeTensorFold({ ...target, source: 'registry' }, { + ...io, now, clock, token, keyNote: note, previous: samples.get(key) ?? null, + }); + if (sample) samples.set(key, sample); + return { server: status }; + } + const seen = detected.get(key); + if (seen && seen.tensorfold === false && now() - seen.at < DETECT_RECHECK_MS) { + return { skipped: { id: entry.id, reason: 'not TensorFold (checked recently)' } }; + } + const base = baseUrlOf(target); + const core = await readCore(base, io, null); + const first = buildStatus(target, core, { clockMs: clock(), checkedAt: new Date(now()).toISOString() }); + if (!first.status.reachable) { + if (seen?.tensorfold) return { server: { ...first.status, source: 'detected' } }; + return { skipped: { id: entry.id, reason: `unreachable, engine unknown (${first.status.reason})` } }; + } + if (!first.detected) { + detected.set(key, { tensorfold: false, at: now() }); + return { skipped: { id: entry.id, reason: 'not TensorFold (no tensorfold: metrics, /health shape differs)' } }; + } + detected.set(key, { tensorfold: true, at: now() }); + const slide = await readSlide(base, io, null); + const { status, sample } = buildStatus({ ...target, source: 'detected' }, { ...core, slide }, { + previous: samples.get(key) ?? null, clockMs: clock(), checkedAt: new Date(now()).toISOString(), + }); + if (sample) samples.set(key, sample); + return { server: status }; + } catch (err) { + // Belt and braces: everything above is written not to throw, and a + // heartbeat that dies on a status read is worse than a missing card. + return { skipped: { id: entry.id, reason: `status read failed (${err?.name || 'error'})` } }; + } + })); + return { + servers: results.filter(r => r.server).map(r => r.server), + skipped: results.filter(r => r.skipped).map(r => r.skipped), + }; +} + +// --- the heartbeat side ------------------------------------------------------- + +/** + * Keeps a cached TensorFold snapshot for the device heartbeat. + * + * OFF unless a registry path is given (INTENT_TENSORFOLD_REGISTRY). It runs + * on its own timer and the heartbeat reads `fields()` synchronously, so a + * slow or dead server costs a stale card, never a late heartbeat. The + * registry file is re-read every refresh, so an edit applies without a + * restart. + * + * Published as ONE key, `tensorfold`, replaced whole on every beat. The + * intent server merges a PATCH key by key, so a field this monitor stopped + * sending would otherwise linger on the card forever. + */ +export class TensorFoldMonitor { + #registryPath; #entries; #scope; #intervalMs; #timeoutMs; #fetchImpl; #now; #clock; #env; #warn; + #readRegistry; #timer = null; #inFlight = null; #snapshot = null; + #samples = new Map(); #detected = new Map(); #lastWarning = -Infinity; + + constructor({ + registryPath = process.env.INTENT_TENSORFOLD_REGISTRY, + entries, + scope = process.env.INTENT_TENSORFOLD_SCOPE, + intervalMs = 30000, + timeoutMs = DEFAULT_STATUS_TIMEOUT_MS, + fetchImpl = (...a) => fetch(...a), + now = () => Date.now(), + clock = () => performance.now(), + env = process.env, + warn = message => console.warn(message), + readRegistry = readRegistryFile, + } = {}) { + const path = typeof registryPath === 'string' ? registryPath.trim() : ''; + this.#registryPath = path && !['0', 'off', 'false', 'no'].includes(path.toLowerCase()) ? expandHome(path) : null; + this.#entries = Array.isArray(entries) ? entries : null; + const wanted = typeof scope === 'string' && scope.trim() ? scope.trim().toLowerCase() : 'local'; + if (!SCOPES.includes(wanted)) throw new Error(`INTENT_TENSORFOLD_SCOPE must be one of ${SCOPES.join(', ')}`); + this.#scope = wanted; + if (!Number.isFinite(intervalMs) || intervalMs <= 0) throw new Error('Invalid tensorfold intervalMs'); + this.#intervalMs = intervalMs; + this.#timeoutMs = timeoutMs; + this.#fetchImpl = fetchImpl; + this.#now = now; + this.#clock = clock; + this.#env = env; + this.#warn = warn; + this.#readRegistry = readRegistry; + } + + get enabled() { return Boolean(this.#registryPath || this.#entries); } + + get scope() { return this.#scope; } + + /** For a log line. A path and a scope, never a key. */ + describe() { + if (!this.enabled) return 'off (set INTENT_TENSORFOLD_REGISTRY to a models.json to turn it on)'; + return `${this.#registryPath ?? 'in-memory registry'} (scope ${this.#scope}, every ${Math.round(this.#intervalMs / 1000)}s, GET only)`; + } + + /** What the heartbeat publishes: {} when off or before the first read. */ + fields() { + if (!this.enabled || !this.#snapshot) return {}; + return { tensorfold: structuredClone(this.#snapshot) }; + } + + #warning(reason) { + const t = this.#now(); + if (t >= this.#lastWarning && t - this.#lastWarning < 300000) return; + this.#lastWarning = t; + this.#warn(`uik tensorfold: ${reason}`); + } + + async #read() { + const checkedAt = new Date(this.#now()).toISOString(); + let entries; + try { + entries = this.#entries ?? await this.#readRegistry(this.#registryPath); + } catch (err) { + const message = String(err?.message || 'unknown error').slice(0, 200); + this.#warning(`registry unreadable, publishing the error instead of servers: ${message}`); + // An error is not an empty list: `error` says the list is unknown. + this.#snapshot = { schema: STATUS_SCHEMA, checked_at: checkedAt, scope: this.#scope, error: `registry unreadable: ${message}`, servers: [] }; + return this.#snapshot; + } + const { servers } = await collectTensorFoldStatus(entries, { + scope: this.#scope, fetchImpl: this.#fetchImpl, timeoutMs: this.#timeoutMs, + now: this.#now, clock: this.#clock, env: this.#env, samples: this.#samples, detected: this.#detected, + }); + this.#snapshot = { schema: STATUS_SCHEMA, checked_at: checkedAt, scope: this.#scope, error: null, servers }; + return this.#snapshot; + } + + /** One read. Single flight, never rejects. */ + refresh() { + if (!this.enabled) return Promise.resolve(null); + if (!this.#inFlight) { + this.#inFlight = this.#read() + .catch(err => { this.#warning(`status read failed: ${err?.name || 'error'}`); return this.#snapshot; }) + .finally(() => { this.#inFlight = null; }); + } + return this.#inFlight; + } + + /** Starts the timer; resolves when the first read lands (null when off). */ + start() { + this.stop(); + if (!this.enabled) return Promise.resolve(null); + const first = this.refresh(); + this.#timer = setInterval(() => { this.refresh(); }, this.#intervalMs); + this.#timer.unref?.(); + return first; + } + + stop() { + if (this.#timer) clearInterval(this.#timer); + this.#timer = null; + } +} diff --git a/src/tensorfold-status.mjs b/src/tensorfold-status.mjs new file mode 100644 index 0000000..2b80ace --- /dev/null +++ b/src/tensorfold-status.mjs @@ -0,0 +1,137 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// +// tensorfold-status - every TensorFold server in the model registry, read +// NOW with GET /health, GET /metrics and GET /v1/slide/graph, for +// `ide-agent-kit tensorfold-status`. +// +// The reading itself lives in packages/user-intent-kit/src/tensorfold-status.js, +// because the device heartbeat (uik-daemon, `ide-agent-kit intent daemon`) +// publishes the same object and must not depend on IAK. This file only finds +// the registry, runs one read and renders it for a person or for --json. +// +// One-shot: tok/s needs two samples, so the read-out says "needs two samples" +// rather than printing 0. The heartbeat keeps the previous sample and does +// publish a rate. + +import { homedir } from 'node:os'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { readRegistryFile, RegistryError, resolveCallerHost } from '../packages/user-intent-kit/src/model-capacity.js'; +import { collectTensorFoldStatus, STATUS_SCHEMA } from '../packages/user-intent-kit/src/tensorfold-status.js'; +import { resolveModelRegistryPath } from './model-selection.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); + +function expandHome(path) { + const s = String(path).trim(); + if (s === '~') return homedir(); + if (s.startsWith('~/')) return join(homedir(), s.slice(2)); + return s; +} + +/** + * Which registry to read: --registry, then INTENT_TENSORFOLD_REGISTRY (the + * file the heartbeat reads, so the CLI shows what the card will show), then + * the same path the model picker uses (mcp.confirmations.model_registry, or + * config/models.json in this checkout). + */ +export function resolveTensorFoldRegistryPath({ cliPath, env = process.env, config = {}, rootDir = ROOT } = {}) { + if (typeof cliPath === 'string' && cliPath.trim()) return expandHome(cliPath); + const fromEnv = typeof env.INTENT_TENSORFOLD_REGISTRY === 'string' ? env.INTENT_TENSORFOLD_REGISTRY.trim() : ''; + if (fromEnv && !['0', 'off', 'false', 'no'].includes(fromEnv.toLowerCase())) return expandHome(fromEnv); + return resolveModelRegistryPath(config, rootDir); +} + +const show = v => (v === null || v === undefined ? 'unknown' : String(v)); +const num = (v, unit = '') => (v === null || v === undefined ? 'unknown' : `${v}${unit}`); +const pct = v => (v === null || v === undefined ? 'unknown' : `${Math.round(v * 100)}%`); + +function livingLine(lw) { + if (lw.available === false) return 'not available (no /v1/slide/graph; serve with --slide)'; + if (lw.available === null) return 'unknown'; + return `${lw.facts} facts, ${lw.learned} learned, ${lw.learning} learning now`; +} + +function metricsLine(state) { + return { + ok: 'ok', + auth_required: 'auth required', + auth_rejected: 'key refused', + unavailable: 'unavailable', + }[state] ?? state; +} + +/** One server, several short lines. Plain words, nothing derived silently. */ +export function formatServer(s) { + const where = `${s.host}:${s.port}`; + const head = `${s.id ?? where} ${where} ${s.reachable ? 'reachable' : 'UNREACHABLE'}${s.source === 'detected' ? ' (auto-detected)' : ''}`; + const lines = [head]; + if (s.reason) lines.push(` note: ${s.reason}`); + if (!s.reachable) return lines.join('\n'); + const warming = s.warming === null ? 'unknown' : (s.warming ? 'yes' : 'no'); + lines.push(` model: ${show(s.model)} warming: ${warming}`); + lines.push(` version: ${show(s.version)} backend: ${show(s.backend)} device: ${show(s.device)}`); + lines.push(` memory: ${num(s.memory.used_gb, ' GB')} used, ${num(s.memory.peak_gb, ' GB')} peak, ${num(s.memory.budget_gb, ' GB')} budget`); + lines.push(` requests: ${num(s.requests.running)} running, ${num(s.requests.waiting)} waiting ` + + `tokens since start: ${num(s.tokens.prompt_total)} prompt, ${num(s.tokens.generation_total)} generated`); + const rate = s.tok_s === null ? 'needs two samples (the heartbeat publishes it)' : `${s.tok_s} over ${s.tok_s_window_s} s`; + const decode = s.decode_tok_s === null ? '' : ` server decode: ${s.decode_tok_s} tok/s`; + lines.push(` tok/s: ${rate}${decode}`); + const acceptance = s.draft.acceptance === null ? 'n/a' : pct(s.draft.acceptance); + lines.push(` draft: ${num(s.draft.drafted)} drafted, ${num(s.draft.accepted)} accepted, acceptance ${acceptance} ` + + `ttft p50: ${s.ttft_p50_s === null ? 'n/a' : `${s.ttft_p50_s} s`} kv cache: ${pct(s.kv_cache_usage)}`); + lines.push(` living weights: ${livingLine(s.living_weights)} metrics: ${metricsLine(s.metrics)}`); + return lines.join('\n'); +} + +export function formatReport(report) { + const out = [`TensorFold servers in ${report.registry} (read from ${report.from} at ${report.checked_at}, GET only):`]; + if (!report.servers.length) out.push('none found.'); + for (const s of report.servers) out.push('', formatServer(s)); + if (report.skipped.length) { + out.push('', 'Not TensorFold or not checked:'); + for (const k of report.skipped) out.push(` ${k.id}: ${k.reason}`); + } + return out.join('\n'); +} + +/** + * Run one read. Returns {exitCode, report, text}. Exit 0 when the registry + * was read (unreachable servers are reported, not fatal), 2 when it could + * not be. + */ +export async function runTensorFoldStatus({ + registryPath, + scope = 'all', + json = false, + fetchImpl, + timeoutMs, + env = process.env, + now = () => Date.now(), + callerHost, + readRegistry = readRegistryFile, +} = {}) { + let entries; + try { + entries = await readRegistry(registryPath); + } catch (err) { + const message = err instanceof RegistryError ? err.message : `cannot read registry ${registryPath}: ${err.message}`; + return { exitCode: 2, report: null, text: `Error: ${message}\nSee config/models.README.md for the registry format.` }; + } + const { servers, skipped } = await collectTensorFoldStatus(entries, { + scope, env, now, + ...(fetchImpl ? { fetchImpl } : {}), + ...(timeoutMs ? { timeoutMs } : {}), + }); + const report = { + schema: STATUS_SCHEMA, + registry: registryPath, + from: callerHost ?? await resolveCallerHost(), + checked_at: new Date(now()).toISOString(), + scope, + servers, + skipped, + }; + return { exitCode: 0, report, text: json ? JSON.stringify(report, null, 2) : formatReport(report) }; +} diff --git a/test/fixtures/tensorfold/health-mlx-0.6.0.json b/test/fixtures/tensorfold/health-mlx-0.6.0.json new file mode 100644 index 0000000..8fec722 --- /dev/null +++ b/test/fixtures/tensorfold/health-mlx-0.6.0.json @@ -0,0 +1 @@ +{"status": "ok", "model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit", "model_ids": ["NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit"], "max_batch_size": 8, "warming": false, "memory": {"active": 18747144612, "cache": 0, "peak": 40682525650, "budget": 96207267430, "mlx_budget": 92986041958, "footprint": 19203554976}} \ No newline at end of file diff --git a/test/fixtures/tensorfold/metrics-mlx-0.6.0.prom b/test/fixtures/tensorfold/metrics-mlx-0.6.0.prom new file mode 100644 index 0000000..4ab4563 --- /dev/null +++ b/test/fixtures/tensorfold/metrics-mlx-0.6.0.prom @@ -0,0 +1,57 @@ +# HELP tensorfold:requests_running Requests in prefill or decode. +# TYPE tensorfold:requests_running gauge +tensorfold:requests_running 0 +# HELP tensorfold:requests_waiting Requests queued or held until a lane is free. +# TYPE tensorfold:requests_waiting gauge +tensorfold:requests_waiting 0 +# HELP tensorfold:prompt_tokens_total Prompt tokens of finished requests. +# TYPE tensorfold:prompt_tokens_total counter +tensorfold:prompt_tokens_total 0 +# HELP tensorfold:generation_tokens_total Generated tokens of finished requests. +# TYPE tensorfold:generation_tokens_total counter +tensorfold:generation_tokens_total 0 +# HELP tensorfold:kv_cache_usage_ratio Tokens in a stream cache divided by that stream's context window. +# TYPE tensorfold:kv_cache_usage_ratio gauge +tensorfold:kv_cache_usage_ratio{pool="0"} 0 +# HELP tensorfold:mtp_drafted_total Draft tokens verified on finished requests. +# TYPE tensorfold:mtp_drafted_total counter +tensorfold:mtp_drafted_total 0 +# HELP tensorfold:mtp_accepted_total Draft tokens kept on finished requests. +# TYPE tensorfold:mtp_accepted_total counter +tensorfold:mtp_accepted_total 0 +# HELP tensorfold:request_latency_seconds Seconds from arrival to the reply leaving. +# TYPE tensorfold:request_latency_seconds histogram +tensorfold:request_latency_seconds_bucket{le="0.01"} 0 +tensorfold:request_latency_seconds_bucket{le="0.05"} 0 +tensorfold:request_latency_seconds_bucket{le="0.1"} 0 +tensorfold:request_latency_seconds_bucket{le="0.25"} 0 +tensorfold:request_latency_seconds_bucket{le="0.5"} 0 +tensorfold:request_latency_seconds_bucket{le="1"} 0 +tensorfold:request_latency_seconds_bucket{le="2.5"} 0 +tensorfold:request_latency_seconds_bucket{le="5"} 0 +tensorfold:request_latency_seconds_bucket{le="10"} 0 +tensorfold:request_latency_seconds_bucket{le="30"} 0 +tensorfold:request_latency_seconds_bucket{le="60"} 0 +tensorfold:request_latency_seconds_bucket{le="120"} 0 +tensorfold:request_latency_seconds_bucket{le="300"} 0 +tensorfold:request_latency_seconds_bucket{le="+Inf"} 0 +tensorfold:request_latency_seconds_sum 0 +tensorfold:request_latency_seconds_count 0 +# HELP tensorfold:time_to_first_token_seconds Seconds from arrival to the first generated token. +# TYPE tensorfold:time_to_first_token_seconds histogram +tensorfold:time_to_first_token_seconds_bucket{le="0.01"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="0.05"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="0.1"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="0.25"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="0.5"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="1"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="2.5"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="5"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="10"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="30"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="60"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="120"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="300"} 0 +tensorfold:time_to_first_token_seconds_bucket{le="+Inf"} 0 +tensorfold:time_to_first_token_seconds_sum 0 +tensorfold:time_to_first_token_seconds_count 0 diff --git a/test/fixtures/tensorfold/slide-graph-404.json b/test/fixtures/tensorfold/slide-graph-404.json new file mode 100644 index 0000000..ee1e01e --- /dev/null +++ b/test/fixtures/tensorfold/slide-graph-404.json @@ -0,0 +1 @@ +{"error": {"message": "unknown path /v1/slide/graph"}} \ No newline at end of file diff --git a/test/tensorfold-status.test.mjs b/test/tensorfold-status.test.mjs new file mode 100644 index 0000000..a5eae09 --- /dev/null +++ b/test/tensorfold-status.test.mjs @@ -0,0 +1,585 @@ +// SPDX-License-Identifier: AGPL-3.0-only +// +// TensorFold status on the device heartbeat. +// +// The fixtures under test/fixtures/tensorfold/ named *-mlx-0.6.0.* and +// slide-graph-404.json are byte-for-byte GET answers recorded from a live +// TensorFold server (tensorfold 0.6.0, MLX, 127.0.0.1:8091) on 2026-10-14. +// Its counters were all zero, so the traffic case below edits the counter +// values of that same recording rather than inventing a format. The 401 case +// mirrors the native server's key gate (zig/src/server/server.zig +// `authenticate` and status_routes.zig `health`), and the 200 slide graph +// mirrors zig/src/server/slide_graph.zig `document`. +// +// Every test server records every request, and the suite asserts that only +// GETs to /health, /metrics and /v1/slide/graph ever arrive: a status reader +// that sent a completion would teach the model something mid-lesson. The +// recorder has its own negative control so the check can fail. + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { createServer } from 'node:http'; +import { readFileSync } from 'node:fs'; +import { mkdtemp, writeFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { execFile } from 'node:child_process'; + +import { + parsePrometheus, histogramQuantile, summarizeMetrics, summarizeHealth, summarizeSlideGraph, + looksLikeTensorFoldHealth, probeTensorFold, collectTensorFoldStatus, TensorFoldMonitor, + DETECT_RECHECK_MS, METRICS_STATES, STATUS_SCHEMA, +} from '../packages/user-intent-kit/src/tensorfold-status.js'; +import { loadRegistry, resolveEntryToken, RegistryError, KEY_SOURCES } from '../packages/user-intent-kit/src/model-capacity.js'; +import { DesktopAdapter } from '../packages/user-intent-kit/src/adapters/desktop.js'; +import { resolveTensorFoldRegistryPath, runTensorFoldStatus, formatServer } from '../src/tensorfold-status.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const fixture = name => readFileSync(new URL(`./fixtures/tensorfold/${name}`, import.meta.url), 'utf8'); +const LIVE_HEALTH = fixture('health-mlx-0.6.0.json'); +const LIVE_METRICS = fixture('metrics-mlx-0.6.0.prom'); +const LIVE_SLIDE_404 = fixture('slide-graph-404.json'); +const LIVE_MODEL = 'NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit'; + +// Made up. The suite asserts it never appears in any output. +const DUMMY_TOKEN = 'tf-dummy-not-real'; +const ALLOWED = new Set(['GET /health', 'GET /metrics', 'GET /v1/slide/graph']); + +/** Replace the value of one exact series line in a Prometheus recording. */ +function setSeries(text, series, value) { + const escaped = series.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + const re = new RegExp(`^${escaped} \\S+$`, 'm'); + assert.match(text, re, `fixture has no line for ${series}`); + return text.replace(re, `${series} ${value}`); +} + +/** The live recording with traffic: edited values, same format. */ +function trafficMetrics({ generation = 1200 } = {}) { + let t = LIVE_METRICS; + t = setSeries(t, 'tensorfold:requests_running', 2); + t = setSeries(t, 'tensorfold:requests_waiting', 1); + t = setSeries(t, 'tensorfold:prompt_tokens_total', 5000); + t = setSeries(t, 'tensorfold:generation_tokens_total', generation); + t = setSeries(t, 'tensorfold:kv_cache_usage_ratio{pool="0"}', 0.25); + t = t.replace('tensorfold:kv_cache_usage_ratio{pool="0"} 0.25', 'tensorfold:kv_cache_usage_ratio{pool="0"} 0.25\ntensorfold:kv_cache_usage_ratio{pool="1"} 0.5'); + t = setSeries(t, 'tensorfold:mtp_drafted_total', 1000); + t = setSeries(t, 'tensorfold:mtp_accepted_total', 750); + const ttft = { '0.01': 0, '0.05': 0, '0.1': 2, '0.25': 7, '0.5': 9, '1': 10, '2.5': 10, '5': 10, '10': 10, '30': 10, '60': 10, '120': 10, '300': 10, '+Inf': 10 }; + for (const [le, n] of Object.entries(ttft)) t = setSeries(t, `tensorfold:time_to_first_token_seconds_bucket{le="${le}"}`, n); + t = setSeries(t, 'tensorfold:time_to_first_token_seconds_count', 10); + return t; +} + +/** A recording HTTP server. `routes` maps "GET /path" to a reply or a function returning one. */ +async function tfServer(t, routes) { + const seen = []; + const server = createServer((req, res) => { + seen.push({ line: `${req.method} ${req.url}`, auth: req.headers.authorization ?? null, connection: req.headers.connection ?? null }); + let route = routes[`${req.method} ${req.url}`]; + if (typeof route === 'function') route = route(req); + if (!route) { + res.writeHead(404, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ error: { message: `unknown path ${req.url}` } })); + return; + } + if (route.hang) return; + res.writeHead(route.status ?? 200, { 'content-type': route.type ?? 'application/json', ...(route.headers || {}) }); + res.end(route.body ?? ''); + }); + await new Promise(r => server.listen(0, '127.0.0.1', r)); + t.after(() => new Promise(r => { server.closeAllConnections(); server.close(r); })); + return { port: server.address().port, seen, server }; +} + +function assertGetOnly(seen) { + for (const r of seen) assert.ok(ALLOWED.has(r.line), `status reader sent ${r.line}`); +} + +const liveRoutes = (overrides = {}) => ({ + 'GET /health': { body: LIVE_HEALTH }, + 'GET /metrics': { body: LIVE_METRICS, type: 'text/plain; version=0.0.4; charset=utf-8' }, + 'GET /v1/slide/graph': { status: 404, body: LIVE_SLIDE_404 }, + ...overrides, +}); + +// The native server with keys on: /health is open and minimal, the rest 401. +const GATE_BODY = '{"error": {"message": "A valid API key is required", "type": "invalid_request_error", "param": null, "code": "invalid_api_key"}}'; +const gated = (body, type = 'application/json') => req => (req.headers.authorization === `Bearer ${DUMMY_TOKEN}` + ? { body, type } + : { status: 401, body: GATE_BODY, headers: { 'www-authenticate': 'Bearer' } }); +const authRoutes = () => ({ + 'GET /health': { body: '{"status": "ok"}' }, + 'GET /metrics': gated(LIVE_METRICS, 'text/plain; version=0.0.4; charset=utf-8'), + 'GET /v1/slide/graph': gated(JSON.stringify({ next: 1, facts: [] })), +}); + +const entry = (port, extra = {}) => loadRegistry([{ id: 'tf', host: '127.0.0.1', port, kind: 'openai', ...extra }])[0]; + +/** A 0600 key file holding `token`, removed after the test. */ +async function keyFileWith(t, token) { + const dir = await mkdtemp(join(tmpdir(), 'iak-tf-key-')); + t.after(() => rm(dir, { recursive: true, force: true })); + const path = join(dir, 'tensorfold_api_key.txt'); + await writeFile(path, `${token}\n`, { mode: 0o600 }); + return path; +} + +// --- the recorder can fail ------------------------------------------------------ + +test('negative control: the GET-only recorder catches a completion request', async t => { + const { port, seen } = await tfServer(t, liveRoutes()); + await fetch(`http://127.0.0.1:${port}/v1/chat/completions`, { method: 'POST', body: '{}' }); + assert.throws(() => assertGetOnly(seen), /POST \/v1\/chat\/completions/); +}); + +// --- parsing the live recording ------------------------------------------------- + +test('live 0.6.0 /metrics parses into tensorfold series with labels', () => { + const series = parsePrometheus(LIVE_METRICS); + assert.ok(series.length > 30); + assert.ok(series.every(s => s.name.startsWith('tensorfold:'))); + const kv = series.find(s => s.name === 'tensorfold:kv_cache_usage_ratio'); + assert.deepEqual(kv.labels, { pool: '0' }); + const inf = series.find(s => s.name === 'tensorfold:time_to_first_token_seconds_bucket' && s.labels.le === '+Inf'); + assert.equal(inf.value, 0); + const m = summarizeMetrics(series); + assert.equal(m.detected, true); + assert.deepEqual(m.requests, { running: 0, waiting: 0 }); + assert.deepEqual(m.draft, { drafted: 0, accepted: 0, acceptance: null }, 'no drafts is n/a, never 0%'); + assert.equal(m.ttftP50, null, 'nothing observed is null, never 0 s'); + assert.equal(m.kvCacheUsage, 0); +}); + +test('live 0.6.0 /health has the TensorFold shape; other servers do not', () => { + assert.equal(looksLikeTensorFoldHealth(JSON.parse(LIVE_HEALTH)), true); + assert.equal(looksLikeTensorFoldHealth({ status: 'ok' }), false, 'llama-server and a keyed TensorFold both say just this'); + assert.equal(looksLikeTensorFoldHealth(null), false); + const h = summarizeHealth(JSON.parse(LIVE_HEALTH)); + assert.equal(h.model, LIVE_MODEL); + assert.equal(h.warming, false); + assert.equal(h.version, null); + assert.equal(h.backend, null); + assert.equal(h.device, null); +}); + +test('version, backend and device are read when /health carries them', () => { + const future = { ...JSON.parse(LIVE_HEALTH), version: '1.1.0', backend: 'cuda', device: { name: 'NVIDIA GB10' } }; + const h = summarizeHealth(future); + assert.equal(h.version, '1.1.0'); + assert.equal(h.backend, 'cuda'); + assert.equal(h.device, 'NVIDIA GB10'); + assert.equal(summarizeHealth({ ...future, device: 'Apple M5 Max\u0007' }).device, 'Apple M5 Max'); +}); + +test('histogram p50 interpolates like Prometheus and is null with no observations', () => { + const b = (le, count) => ({ le, count }); + assert.equal(histogramQuantile(0.5, [b(0.1, 2), b(0.25, 7), b(Infinity, 10)]), 0.19); + assert.equal(histogramQuantile(0.5, [b(0.1, 0), b(Infinity, 0)]), null); + assert.equal(histogramQuantile(0.5, [b(1, 0), b(Infinity, 4)]), 1, 'mass in +Inf reports the highest finite edge'); + assert.equal(histogramQuantile(0.5, []), null); +}); + +test('slide graph: counts only, never the taught text', () => { + const graph = { + next: 4, + facts: [ + { id: 1, text: 'private fact one', source: 'chat', at: 1, state: 'learned', links: [] }, + { id: 2, text: 'private fact two', source: 'chat', at: 2, state: 'learning', links: [], phase: 'fitting' }, + { id: 3, text: 'private fact three', source: 'chat', at: 3, state: 'queued', links: [] }, + ], + }; + const lw = summarizeSlideGraph(graph); + assert.deepEqual(lw, { available: true, facts: 3, learning: 1, learned: 1 }); + assert.doesNotMatch(JSON.stringify(lw), /private/); + assert.deepEqual(summarizeSlideGraph({ error: 'x' }), { available: null, facts: null, learning: null, learned: null }); +}); + +// --- one server, end to end ------------------------------------------------------ + +test('live recording end to end: the published object, GET only, no key sent', async t => { + const { port, seen } = await tfServer(t, liveRoutes()); + const { status, sample } = await probeTensorFold({ id: 'mb-tensorfold-nemotron', host: '127.0.0.1', port }, { + now: () => Date.parse('2026-10-14T09:00:00Z'), clock: () => 5000, + }); + assertGetOnly(seen); + assert.deepEqual(seen.map(r => r.line).sort(), ['GET /health', 'GET /metrics', 'GET /v1/slide/graph']); + assert.ok(seen.every(r => r.auth === null)); + assert.ok(seen.every(r => r.connection === 'close'), 'no idle keep-alive socket parked on the server between reads'); + assert.deepEqual(status, { + id: 'mb-tensorfold-nemotron', + host: '127.0.0.1', + port, + local: true, + source: 'registry', + reachable: true, + reason: null, + model: LIVE_MODEL, + warming: false, + version: null, + backend: null, + device: null, + memory: { used_gb: 18.7, peak_gb: 40.7, budget_gb: 96.2 }, + requests: { running: 0, waiting: 0 }, + tokens: { prompt_total: 0, generation_total: 0 }, + tok_s: null, + tok_s_window_s: null, + decode_tok_s: null, + draft: { drafted: 0, accepted: 0, acceptance: null }, + ttft_p50_s: null, + kv_cache_usage: 0, + living_weights: { available: false, facts: null, learning: null, learned: null }, + metrics: 'ok', + checked_at: '2026-10-14T09:00:00.000Z', + }); + assert.deepEqual(sample, { generation_total: 0, at: 5000 }); + assert.ok(JSON.stringify(status).length < 1024, 'one server stays under 1 KB on the heartbeat'); +}); + +test('traffic: tok/s from generation_tokens_total deltas, acceptance, ttft p50, fullest kv pool', async t => { + const state = { generation: 1200 }; + const { port, seen } = await tfServer(t, liveRoutes({ + 'GET /metrics': () => ({ body: trafficMetrics({ generation: state.generation }), type: 'text/plain' }), + })); + const target = { id: 'tf', host: '127.0.0.1', port }; + const first = await probeTensorFold(target, { clock: () => 10_000 }); + assert.equal(first.status.tok_s, null, 'the first sample has no rate'); + assert.deepEqual(first.status.requests, { running: 2, waiting: 1 }); + assert.deepEqual(first.status.tokens, { prompt_total: 5000, generation_total: 1200 }); + assert.deepEqual(first.status.draft, { drafted: 1000, accepted: 750, acceptance: 0.75 }); + assert.equal(first.status.ttft_p50_s, 0.19); + assert.equal(first.status.kv_cache_usage, 0.5); + + state.generation = 1800; + const second = await probeTensorFold(target, { clock: () => 20_000, previous: first.sample }); + assert.equal(second.status.tok_s, 60); + assert.equal(second.status.tok_s_window_s, 10); + + state.generation = 40; // the server restarted + const third = await probeTensorFold(target, { clock: () => 30_000, previous: second.sample }); + assert.equal(third.status.tok_s, null, 'a counter that went down is a restart, not a negative rate'); + assert.deepEqual(third.sample, { generation_total: 40, at: 30_000 }); + assertGetOnly(seen); +}); + +test('401 on /metrics: "auth required", minimal /health reads as unknown, no key invented', async t => { + const { port, seen } = await tfServer(t, authRoutes()); + const { status } = await probeTensorFold({ id: 'gb10', host: '127.0.0.1', port }); + assertGetOnly(seen); + assert.ok(seen.every(r => r.auth === null)); + assert.equal(status.reachable, true); + assert.equal(status.metrics, METRICS_STATES.AUTH_REQUIRED); + assert.match(status.reason, /metrics: auth required/); + assert.equal(status.model, null); + assert.deepEqual(status.memory, { used_gb: null, peak_gb: null, budget_gb: null }); + assert.deepEqual(status.requests, { running: null, waiting: null }, 'unread is null, never 0'); + assert.equal(status.living_weights.available, null, 'a 401 graph is unknown, not absent'); + assert.match(formatServer(status), /metrics: auth required/); +}); + +test('keyFile: the entry key goes to /metrics and /v1/slide/graph only, never /health, never in output', async t => { + const { port, seen } = await tfServer(t, authRoutes()); + const warnings = []; + const e = entry(port, { engine: 'tensorfold', keyFile: await keyFileWith(t, DUMMY_TOKEN) }); + const { servers } = await collectTensorFoldStatus([e], { env: {} }); + assertGetOnly(seen); + const byPath = Object.fromEntries(seen.map(r => [r.line, r.auth])); + assert.equal(byPath['GET /health'], null); + assert.equal(byPath['GET /metrics'], `Bearer ${DUMMY_TOKEN}`); + assert.equal(byPath['GET /v1/slide/graph'], `Bearer ${DUMMY_TOKEN}`); + assert.equal(servers[0].metrics, METRICS_STATES.OK); + assert.deepEqual(servers[0].living_weights, { available: true, facts: 0, learning: 0, learned: 0 }); + assert.doesNotMatch(JSON.stringify(servers) + warnings.join(''), new RegExp(DUMMY_TOKEN)); + + seen.length = 0; + const other = entry(port, { engine: 'tensorfold', keyFile: await keyFileWith(t, 'some-other-made-up-key') }); + const wrong = await collectTensorFoldStatus([other], { env: {} }); + assert.equal(wrong.servers[0].metrics, METRICS_STATES.AUTH_REJECTED, 'a key that was sent and refused is a different repair'); +}); + +test('a redirect is never followed, so a key cannot be carried to another host', async t => { + const elsewhere = await tfServer(t, liveRoutes()); + const to = `http://127.0.0.1:${elsewhere.port}`; + const { port } = await tfServer(t, { + 'GET /health': { body: LIVE_HEALTH }, + 'GET /metrics': { status: 302, headers: { location: `${to}/metrics` } }, + 'GET /v1/slide/graph': { status: 302, headers: { location: `${to}/v1/slide/graph` } }, + }); + const e = entry(port, { engine: 'tensorfold', keyFile: await keyFileWith(t, DUMMY_TOKEN) }); + const { servers } = await collectTensorFoldStatus([e], { env: {} }); + assert.equal(elsewhere.seen.length, 0); + assert.equal(servers[0].metrics, METRICS_STATES.UNAVAILABLE); +}); + +test('a named key that is missing is reported, and the fleet-wide LLM_API_KEY is never substituted', async t => { + const { port, seen } = await tfServer(t, authRoutes()); + const env = { LLM_API_KEY: DUMMY_TOKEN, LLM_API_KEY_FILE: '' }; + const missing = join(tmpdir(), 'iak-tf-no-such-dir', 'key.txt'); + const named = await collectTensorFoldStatus([entry(port, { engine: 'tensorfold', keyFile: missing })], { env }); + assert.match(named.servers[0].reason, /metrics: auth required; cannot read key file .*key\.txt \(ENOENT\)/); + const bearerNoKey = await collectTensorFoldStatus([entry(port, { engine: 'tensorfold', auth: 'bearer' })], { env }); + assert.match(bearerNoKey.servers[0].reason, /names no keyFile/); + assert.ok(seen.every(r => r.auth === null), 'LLM_API_KEY went out to a TensorFold status read'); +}); + +test('unreachable: closed port and a socket that never answers both give reachable:false, never a throw', async t => { + const probe = createServer(); + await new Promise(r => probe.listen(0, '127.0.0.1', r)); + const closedPort = probe.address().port; + await new Promise(r => probe.close(r)); + const closed = await probeTensorFold({ id: 'gone', host: '127.0.0.1', port: closedPort }); + assert.equal(closed.status.reachable, false); + assert.match(closed.status.reason, /connect failed \(ECONNREFUSED\)/); + assert.equal(closed.status.metrics, METRICS_STATES.UNAVAILABLE); + assert.equal(closed.status.living_weights.available, null); + assert.equal(closed.sample, null); + + const { port } = await tfServer(t, { 'GET /health': { hang: true }, 'GET /metrics': { hang: true }, 'GET /v1/slide/graph': { hang: true } }); + const hung = await probeTensorFold({ id: 'hung', host: '127.0.0.1', port }, { timeoutMs: 150 }); + assert.equal(hung.status.reachable, false); + assert.match(hung.status.reason, /no response within 150 ms/); + + const thrown = await probeTensorFold({ id: 'x', host: '127.0.0.1', port }, { fetchImpl: () => { throw new Error('boom'); } }); + assert.equal(thrown.status.reachable, false); +}); + +// --- the registry --------------------------------------------------------------- + +test('registry: engine is a short name, lower-cased and kept; absent is null', () => { + const [e] = loadRegistry([{ id: 'a', host: 'localhost', port: 8091, kind: 'openai', engine: 'TensorFold' }]); + assert.equal(e.engine, 'tensorfold'); + assert.equal(e.auth, 'none'); + const [plain] = loadRegistry([{ id: 'b', host: 'localhost', port: 8091, kind: 'openai' }]); + assert.equal(plain.engine, null); + const bad = extra => () => loadRegistry([{ id: 'c', host: 'localhost', port: 1, kind: 'openai', ...extra }]); + assert.throws(bad({ engine: 'tensor fold!' }), /short name/); + assert.throws(bad({ engine: 42 }), RegistryError); + assert.throws(bad({ engine: 'tensorfold', apiKey: DUMMY_TOKEN }), /never a VALUE/, 'a TensorFold entry still refuses a pasted key'); +}); + +test('resolveEntryToken: entryOnly uses the entry keyFile and stops before the fleet-wide env', async t => { + const env = { LLM_API_KEY: 'fleet-wide-made-up' }; + const [withFile] = loadRegistry([{ id: 'a', host: 'localhost', port: 1, kind: 'openai', keyFile: await keyFileWith(t, DUMMY_TOKEN) }]); + const r = await resolveEntryToken(withFile, { env, entryOnly: true }); + assert.equal(r.token, DUMMY_TOKEN); + assert.equal(r.keySource, KEY_SOURCES.ENTRY_FILE); + const [bare] = loadRegistry([{ id: 'b', host: 'localhost', port: 1, kind: 'openai', auth: 'bearer' }]); + assert.equal((await resolveEntryToken(bare, { env })).token, 'fleet-wide-made-up', 'the picker keeps its fallbacks'); + assert.equal((await resolveEntryToken(bare, { env, entryOnly: true })).token, null); +}); + +// --- many servers --------------------------------------------------------------- + +test('collect: explicit, auto-detected, other engine, scope local, and the not-TensorFold recheck', async t => { + const tf = await tfServer(t, liveRoutes()); + const llama = await tfServer(t, { 'GET /health': { body: '{"status":"ok"}' } }); + const vllm = await tfServer(t, liveRoutes()); + const urls = []; + const fetchImpl = (url, init) => { urls.push(url); return fetch(url, init); }; + let now = Date.parse('2026-10-14T09:00:00Z'); + const entries = loadRegistry([ + { id: 'auto-tf', host: '127.0.0.1', port: tf.port, kind: 'openai' }, + { id: 'llama', host: 'localhost', port: llama.port, kind: 'openai' }, + { id: 'vllm', host: '127.0.0.1', port: vllm.port, kind: 'vllm', engine: 'vllm' }, + { id: 'remote', host: 'remote-box.invalid', port: 8091, kind: 'openai', engine: 'tensorfold' }, + ]); + const detected = new Map(); + const opts = { fetchImpl, now: () => now, detected, scope: 'local', env: {} }; + const one = await collectTensorFoldStatus(entries, opts); + assert.deepEqual(one.servers.map(s => [s.id, s.source]), [['auto-tf', 'detected']]); + assert.deepEqual(one.skipped.map(s => s.id), ['llama', 'vllm']); + assert.equal(vllm.seen.length, 0, 'an entry the registry calls something else gets no request'); + assert.ok(!urls.some(u => u.includes('remote-box.invalid')), 'scope local must not leave the machine'); + assert.ok(llama.seen.length > 0); + for (const s of [tf, llama]) assertGetOnly(s.seen); + + const before = llama.seen.length; + await collectTensorFoldStatus(entries, opts); + assert.equal(llama.seen.length, before, 'a port that answered "not TensorFold" is not asked again at once'); + now += DETECT_RECHECK_MS + 1; + await collectTensorFoldStatus(entries, opts); + assert.ok(llama.seen.length > before, 'and is asked again after the recheck interval'); + + const all = await collectTensorFoldStatus(entries, { ...opts, scope: 'all', timeoutMs: 1500 }); + const remote = all.servers.find(s => s.id === 'remote'); + assert.equal(remote.reachable, false); + assert.equal(remote.local, false); + + // A detected server that stops answering stays on the card as unreachable. + tf.server.closeAllConnections(); + await new Promise(r => tf.server.close(r)); + const down = await collectTensorFoldStatus(entries, opts); + const gone = down.servers.find(s => s.id === 'auto-tf'); + assert.equal(gone.reachable, false); + assert.equal(gone.source, 'detected'); +}); + +test('auto-detection never sends a key, even when the entry names one', async t => { + const { port, seen } = await tfServer(t, authRoutes()); + const e = entry(port, { keyFile: await keyFileWith(t, DUMMY_TOKEN) }); + const { servers, skipped } = await collectTensorFoldStatus([e], { env: {} }); + assert.equal(servers.length, 0); + assert.match(skipped[0].reason, /not TensorFold/); + assert.ok(seen.every(r => r.auth === null)); +}); + +// --- the heartbeat ---------------------------------------------------------------- + +function macSources() { + return { + platform: () => 'darwin', loadavg: () => [1, 1, 1], cpuCount: () => 10, + totalMemBytes: () => 16e9, freeMemBytes: () => 4e9, availableMemBytes: () => 12e9, + run: () => '', listProcessCommandLines: () => [], listThermalZones: () => [], + readThermalZone: () => '', readThermalCriticalMilli: () => '', + }; +} +const offSwitcher = { enabled: false, refresh: async () => ({}), start() {}, stop() {}, fields: () => ({}) }; + +test('monitor: off by default, network-free, publishes nothing', async () => { + for (const registryPath of ['', 'off', undefined]) { + const m = new TensorFoldMonitor({ registryPath, env: {}, fetchImpl: () => { throw new Error('network while off'); } }); + if (registryPath === undefined && process.env.INTENT_TENSORFOLD_REGISTRY) continue; + assert.equal(m.enabled, false); + assert.deepEqual(m.fields(), {}); + assert.equal(await m.refresh(), null); + assert.equal(await m.start(), null); + m.stop(); + assert.match(m.describe(), /^off/); + } + assert.throws(() => new TensorFoldMonitor({ registryPath: '/x', scope: 'everywhere' }), /local, all/); +}); + +test('monitor: an unreadable registry is an error on the card, not an empty list', async () => { + const warnings = []; + const m = new TensorFoldMonitor({ registryPath: join(tmpdir(), 'no-such-dir-iak', 'models.json'), warn: w => warnings.push(w) }); + await m.refresh(); + await m.refresh(); + const { tensorfold } = m.fields(); + assert.equal(tensorfold.schema, STATUS_SCHEMA); + assert.match(tensorfold.error, /^registry unreadable: /); + assert.deepEqual(tensorfold.servers, []); + assert.equal(warnings.length, 1, 'warned once, not once per beat'); +}); + +test('monitor: reads the registry file each refresh and keeps the rate between refreshes', async t => { + const state = { generation: 100 }; + const { port, seen } = await tfServer(t, liveRoutes({ + 'GET /metrics': () => ({ body: trafficMetrics({ generation: state.generation }), type: 'text/plain' }), + })); + const dir = await mkdtemp(join(tmpdir(), 'iak-tf-')); + t.after(() => rm(dir, { recursive: true, force: true })); + const path = join(dir, 'models.json'); + await writeFile(path, JSON.stringify({ models: [{ id: 'mb-tf', host: 'localhost', port, kind: 'openai', engine: 'tensorfold' }] })); + let clock = 0; + const m = new TensorFoldMonitor({ registryPath: path, scope: 'local', clock: () => clock, intervalMs: 30000 }); + assert.match(m.describe(), /scope local, every 30s, GET only/); + await m.refresh(); + clock = 30_000; + state.generation = 1600; + await m.refresh(); + const { tensorfold } = m.fields(); + assert.equal(tensorfold.error, null); + assert.equal(tensorfold.scope, 'local'); + assert.equal(tensorfold.servers.length, 1); + assert.equal(tensorfold.servers[0].tok_s, 50); + assert.equal(tensorfold.servers[0].tok_s_window_s, 30); + assertGetOnly(seen); + + await writeFile(path, JSON.stringify({ models: [{ id: 'mb-tf', host: 'localhost', port, kind: 'openai', engine: 'llama.cpp' }] })); + await m.refresh(); + assert.deepEqual(m.fields().tensorfold.servers, [], 'an edited registry applies without a restart'); +}); + +test('heartbeat: the device PATCH carries the tensorfold key; off and null publish no key', async t => { + const { port } = await tfServer(t, liveRoutes()); + const monitor = new TensorFoldMonitor({ entries: [entry(port, { engine: 'tensorfold' })], intervalMs: 60000 }); + const patched = []; + const client = { deviceId: 'testbox', patchDevice: async s => { patched.push(s); } }; + const base = { pollIntervalMs: 60000, hostSources: macSources(), modelProbe: null, availabilityProbe: null, modelSwitcher: offSwitcher }; + + const before = new DesktopAdapter(client, { ...base, tensorfold: monitor }); + await before.publishState(); + assert.equal(patched[0].tensorfold, undefined, 'nothing until the first read lands: no misleading empty list'); + + await monitor.refresh(); + const adapter = new DesktopAdapter(client, { ...base, tensorfold: monitor }); + await adapter.publishState(); + const beat = patched.at(-1); + assert.equal(beat.tensorfold.schema, STATUS_SCHEMA); + assert.equal(beat.tensorfold.error, null); + assert.equal(beat.tensorfold.servers[0].model, LIVE_MODEL); + assert.equal(beat.tensorfold.servers[0].reachable, true); + assert.equal(beat.ttl_sec, 90); + + const off = new DesktopAdapter(client, { ...base, tensorfold: null }); + await off.publishState(); + assert.equal('tensorfold' in patched.at(-1), false); +}); + +test('heartbeat: start() publishes again as soon as the first read lands', async t => { + const { port } = await tfServer(t, liveRoutes()); + const monitor = new TensorFoldMonitor({ entries: [entry(port, { engine: 'tensorfold' })], intervalMs: 60000 }); + const patched = []; + const adapter = new DesktopAdapter({ deviceId: 'testbox', patchDevice: async s => { patched.push(s); } }, { + pollIntervalMs: 60000, hostSources: macSources(), modelProbe: null, availabilityProbe: null, + modelSwitcher: offSwitcher, tensorfold: monitor, + }); + adapter.start(); + t.after(() => adapter.stop()); + for (let i = 0; i < 100 && !patched.some(p => p.tensorfold); i++) await new Promise(r => setTimeout(r, 20)); + assert.ok(patched.some(p => p.tensorfold?.servers?.[0]?.model === LIVE_MODEL)); +}); + +// --- the CLI -------------------------------------------------------------------- + +test('registry path: --registry, then INTENT_TENSORFOLD_REGISTRY, then the picker\'s path', () => { + assert.equal(resolveTensorFoldRegistryPath({ cliPath: '/a/models.json', env: { INTENT_TENSORFOLD_REGISTRY: '/b' } }), '/a/models.json'); + assert.equal(resolveTensorFoldRegistryPath({ env: { INTENT_TENSORFOLD_REGISTRY: '/b/models.json' } }), '/b/models.json'); + assert.equal(resolveTensorFoldRegistryPath({ env: {}, config: { mcp: { confirmations: { model_registry: '/c.json' } } } }), '/c.json'); + assert.equal(resolveTensorFoldRegistryPath({ env: { INTENT_TENSORFOLD_REGISTRY: 'off' }, rootDir: '/repo' }), join('/repo', 'config', 'models.json')); +}); + +test('read-out: human text says "unknown" for absent fields; --json is the heartbeat object', async t => { + const { port, seen } = await tfServer(t, liveRoutes()); + const dir = await mkdtemp(join(tmpdir(), 'iak-tf-cli-')); + t.after(() => rm(dir, { recursive: true, force: true })); + const path = join(dir, 'models.json'); + await writeFile(path, JSON.stringify({ models: [{ id: 'mb-tf', host: '127.0.0.1', port, kind: 'openai' }] })); + const human = await runTensorFoldStatus({ registryPath: path, callerHost: 'testhost', env: {} }); + assert.equal(human.exitCode, 0); + assert.match(human.text, /mb-tf {2}127\.0\.0\.1:\d+ {2}reachable {2}\(auto-detected\)/); + assert.match(human.text, /version: unknown {3}backend: unknown {3}device: unknown/); + assert.match(human.text, /memory: 18\.7 GB used, 40\.7 GB peak, 96\.2 GB budget/); + assert.match(human.text, /tok\/s: needs two samples/); + assert.match(human.text, /living weights: not available/); + const json = await runTensorFoldStatus({ registryPath: path, callerHost: 'testhost', json: true, env: {} }); + const report = JSON.parse(json.text); + assert.equal(report.schema, STATUS_SCHEMA); + assert.equal(report.servers[0].model, LIVE_MODEL); + assertGetOnly(seen); + + const missing = await runTensorFoldStatus({ registryPath: join(dir, 'nope.json'), callerHost: 'testhost' }); + assert.equal(missing.exitCode, 2); + assert.match(missing.text, /cannot read registry/); +}); + +test('cli: ide-agent-kit tensorfold-status --json runs end to end', async t => { + const { port } = await tfServer(t, liveRoutes()); + const dir = await mkdtemp(join(tmpdir(), 'iak-tf-bin-')); + t.after(() => rm(dir, { recursive: true, force: true })); + const path = join(dir, 'models.json'); + await writeFile(path, JSON.stringify({ models: [{ id: 'mb-tf', host: '127.0.0.1', port, kind: 'openai', engine: 'tensorfold' }] })); + const env = { ...process.env }; + delete env.INTENT_TENSORFOLD_REGISTRY; + const run = args => new Promise(resolve => { + execFile(process.execPath, [join(ROOT, 'bin', 'cli.mjs'), 'tensorfold-status', ...args], { cwd: dir, env, timeout: 20000 }, + (err, stdout, stderr) => resolve({ code: err ? err.code : 0, stdout, stderr })); + }); + const ok = await run(['--registry', path, '--json']); + assert.equal(ok.code, 0, ok.stderr); + const report = JSON.parse(ok.stdout); + assert.equal(report.servers[0].id, 'mb-tf'); + assert.equal(report.servers[0].source, 'registry'); + const bad = await run(['--registry', join(dir, 'missing.json')]); + assert.equal(bad.code, 2); +});