Require verified peer inventory for service decisions

SECURITY.md

@@ -87,10 +87,22 @@ credential store.
 
 ### LAN Metadata and Overlay Addresses
 
-In the **broker-managed deployment**, `/v1/node-info` serves unauthenticated
-plaintext hardware/utilization metadata even after pairing. Standalone node-info
-with `--cluster-dir` can instead use membership-gated mTLS; do not assume that
-mode is active in the desktop bundle.
+In the **broker-managed deployment**, `/v1/node-info` retains an unauthenticated
+plaintext hardware/utilization preview. It also serves inventory and broker-owned
+service metadata over pinned cluster mTLS on the same port. Services grant peer
+trust and use remote inventory for routing or scheduling only after that mTLS
+read proves the exact current certificate and matching response identity. Public
+previews have independent record keys and an explicit `unverified` marker; a
+claimed host UUID cannot replace or remove a verified record. Proofs are bound
+to the local admission and both certificates and are withdrawn on authorization
+changes or worker retirement. Older peers without authenticated service metadata
+remain previews, not service targets. Explicitly configured bare-engine manual
+routes remain a separately labelled bypass and supply no peer telemetry.
+Manual-entry kind is not yet persisted: after a manual-nodes worker restart,
+an address previously added for a PAIR peer can be routed as an unauthenticated
+plaintext bare engine if no PAIR endpoint answers there, including when another
+machine has taken that address. Persisting PAIR-vs-bare-engine intent is required
+to prevent that downgrade across worker restarts.
 
 PAIR can advertise/rank Tailscale CGNAT addresses (`100.64.0.0/10`) and use known
 peer address fallbacks. This is not arbitrary tailnet peer enumeration, and

desktop/docs/services-api.md

@@ -204,7 +204,7 @@
 | `ready` | notification (we consume) | ✅ yes |
 
 **Dynamic / unresolved notify sites (verify by hand — `npm run service-contracts` prints the line numbers):**
-- `method (var)  (daemon.go, 3 sites)`
+- `method (var)  (daemon.go, 2 sites)`
 
 ## nvpair-node-settings
 

docs/architecture.mdx

@@ -241,17 +241,18 @@ Not every surface is gated, and the exceptions are deliberate:
 | Workload replication, error synchronization | Mutual TLS |
 | Cluster membership | Mutual TLS, after pairing |
 | Pairing itself | Plaintext, authenticated by the PIN |
-| Host and GPU telemetry | Plaintext HTTP, **not** authenticated |
+| Service inventory and scheduling telemetry | Pinned cluster mutual TLS |
+| Public hardware preview | Plaintext HTTP, **not** authenticated; display-only |
 
 Pairing is plaintext because no trust exists yet at that point. The six-digit PIN
 is what authenticates the exchange that establishes the pinned certificates.
 
-Node telemetry is a considered trade-off, not an oversight. It is treated as the
-lowest-sensitivity inter-node surface and is read by consumers that are not
-cluster members, so it stays plaintext even on a clustered node. The practical
-consequence is that hostname, hardware inventory, and utilization for a node
-running PAIR are readable by anything on the same subnet that asks. Do not run
-PAIR on a network where that is unacceptable.
+Public hardware previews remain readable by nonmembers over plaintext HTTP,
+including after pairing. They are explicitly unverified and never authorize
+routing, scheduling, or peer trust. Services separately authenticate node-info
+and its broker-owned service metadata over pinned cluster mTLS on the same port.
+The public preview still exposes hostname, hardware inventory, and utilization
+to devices on the subnet; do not run PAIR where that disclosure is unacceptable.
 
 ### Two Personalities on One Port
 
@@ -756,8 +757,11 @@ of being crammed into TXT.
 PAIR identifies every node by a stable UUID, never by its hostname.
 
 An mDNS instance name defaults to the hostname, so two machines that share one
-would silently merge into a single node. Every service therefore advertises a
-per-host UUID, and the directory is keyed by it.
+must not merge. Verified peers use their authenticated host UUID as the directory
+key. Unverified discovery previews use a separate opaque `peerRecordId` and
+`identityVerification: "unverified"`; their claimed host UUID is never a storage
+key and cannot overwrite or remove a verified peer. Explicit manual engine
+configurations occupy their own namespace too.
 
 That UUID is the correlation key across the entire system: discovery entries,
 cluster membership, workload attribution, error reports, telemetry, and proxy
@@ -803,60 +807,62 @@ or prove that these failures are prevented.
 
 ### Node Enrichment
 
-Announcements establish that a node exists and which ports it serves. PAIR
-fetches the interesting facts afterwards, for each node, over HTTP:
-
-- GPU, CPU, and memory inventory from its node-info port
-- Its model list from its engine-manager port, as a flat union, a per-engine
-  breakdown, and the per-engine set of models currently loaded in memory
-
-Each enrichment keeps a last-good value, so one failed fetch dims nothing. A
-node's card does not blank out because a single poll timed out.
-
-With a cluster directory configured, a browsed peer whose `cluster-uuid=` matches
-a pin this node holds is annotated as trusted, which is how the interface can
-distinguish a paired node from a stranger it can merely see.
+Announcements supply candidate endpoints, not identity proofs or authoritative
+service ports. Before granting `trusted`, the scanner fetches node-info over
+pinned cluster mTLS, checks the exact current server certificate, and requires
+matching certificate, response, and broker-owned metadata identities. Only that
+metadata supplies service ports. The endpoint which completed authentication is
+promoted; other advertised addresses remain hints for subsequent authentication.
+
+The scanner then reads models, per-engine attribution, loaded models, engine
+self-reports and compact scheduling telemetry over pinned mTLS. Completed reads
+recheck their authorization after decoding, reject redirects, and enforce body
+limits. Each proof is bound to the local admission, local leaf and exact peer pin
+record, so pin rotation, readmission or membership loss cannot revive old cached
+inventory. Public hardware previews remain separately keyed and untrusted.
+
+Older peers without authenticated service metadata stay visible as previews but
+cannot contribute service inventory until upgraded. Desktop adoption of the
+additive record-key/provenance fields is a follow-up; routing and scheduling are
+already enforced in the Go services.
 
 ### Node Eviction
 
-The scanner scans every five seconds and tolerates three consecutive misses
-before it treats a node as gone. Even then it does not drop the node right away.
-It runs these checks and keeps the node if any of them answers:
-
-- Inference the node served in the last minute, which the proxies report
-- Recent telemetry from that node
-- A direct request to the node's node-info endpoint
-- A probe of the engine-manager port
-
-The activity check comes first because a node under load is the one most likely
-to miss an announcement, and evicting it would pull a working machine out of the
-routing pool at its busiest.
-
-The remaining checks deliberately avoid the inference proxy ports. Those ports
-exist to serve requests, and using them as a liveness test would put connection
-attempts on the serving path every time multicast dropped a packet.
-
-Multicast is lossy, and Wi-Fi, VPN adapters, and sleeping network interfaces all
-produce gaps that have nothing to do with a node's health. Without that probe a
-momentary gap would evict a perfectly healthy node and take its models out of the
-routing pool.
-
-Discovery holds record replacement to the same standard. A node can reappear with
-a new identity, for example after a reset. The old entry then looks superseded,
-and leaving both in place would list the machine twice. But a matching address
-and hostname arrive in an unauthenticated announcement that anything on the
-network can send, so those signals only mark the old record as a suspect.
-Discovery confirms the node's identity before dropping that record. A stale
-duplicate therefore cannot linger, and nothing can displace a peer by claiming
-its address.
+Public candidates retain the browser's missed-announcement and reachability
+policy. Their disappearance removes only their preview record, never a peer
+whose UUID they claim. Verified peers have an independent lifetime: a 15-second
+refresh authenticates their inventory again, and failed authentication expires
+the record after 45 seconds without a successful inventory read. Recent pinned
+inference activity can keep a busy peer present while its control plane is
+saturated. Pin or admission changes withdraw the proof immediately rather than
+waiting for that liveness deadline.
+
+The broker also rechecks proof bindings when publishing service targets,
+initializing the scheduler, and accepting or replaying telemetry. Retired worker
+callbacks cannot restore the old worker's records, and scanner retirement clears
+its sources without deleting a still-authenticated manual source. Withdrawal
+removes the verified record with its original proof token; it does not turn that
+record into a preview under the same host key.
 
 ### Manual Nodes
 
 Some networks block or filter multicast, so discovery is not the only path in.
 `nvpair-manual-nodes` takes an address you enter directly and probes it on a
-fixed interval, and a manual node that answers is folded into the same directory
-as a discovered one. It is initially keyed by the address you typed, and re-keyed
-to the peer's real UUID as soon as that node reports it.
+fixed interval. A PAIR peer enters the verified directory only after pinned
+cluster mTLS proves its current identity and service metadata; public HTTP or
+operator CA trust cannot substitute for that proof. Its configuration keeps an
+independent manual record key until authentication succeeds. There is no
+plaintext downgrade for peer hardware, models, engine state or scheduling data.
+Explicitly user-added bare engines keep the separate manual routing bypass and
+never establish a PAIR peer identity or provide scheduling telemetry. Recognizing
+a PAIR endpoint prevents a bare-engine downgrade for that configuration's lifetime
+in the manual-nodes worker, but this kind is not persisted. After that worker
+restarts, a previously added PAIR address can become an unauthenticated plaintext
+bare-engine route if no PAIR endpoint answers there, including if another machine
+answers instead. Persisting each entry's PAIR-vs-bare-engine kind in
+`manual-nodes.json` is a follow-up, alongside desktop handling of the additive
+`peerRecordId` and `identityVerification` fields without keying previews by a
+claimed `hostUuid`.
 
 ## Observability and Distributed Tracing
 

services/nvpair-manual-nodes/README.md

@@ -30,7 +30,7 @@ Uses bidirectional newline-delimited JSON-RPC 2.0. By default, communication is
 | `--client-cert <path>` | _(none)_ | PEM client certificate to present when probing TLS-enabled manual nodes (requires `--client-key`) |
 | `--client-key <path>` | _(none)_ | PEM client private key matching `--client-cert` |
 | `--ca-bundle <path>` | _(none)_ | PEM bundle of CAs to trust when verifying server certificates (additive to the system trust store) |
-| `--cluster-dir <path>` | _(none)_ | Cluster config dir; when set, a TLS manual node's node-info is probed over cluster mTLS, presenting this node's leaf and accepting any currently-pinned server cert |
+| `--cluster-dir <path>` | _(none)_ | Cluster config dir for pinned mTLS peer inventory. A manual peer is discovered through a pinned certificate, then re-read against that exact principal before publishing its proof |
 | `--log-level <level>` | `info` | `error` / `warn` / `info` / `debug`; falls back to `$NVPAIR_LOG_LEVEL`. Also changeable at runtime via the `log/set-level` method |
 | `--version` | | Print version and exit |
 
@@ -82,7 +82,9 @@ Emitted when a manually added node has been probed and its initial status determ
 }
 ```
 
-Each node is probed for inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`), LM Studio on its default `:1234` (`GET /v1/models`), and oMLX on its default `:1236` (OpenAI-compatible `/v1/models` or loopback-only 403 proxy identification). Nodes running `nvpair-engine-manager` on `:14322` are also probed for authoritative multi-engine inventory. `lmstudio_*` and `omlx_*` mirror the `ollama_*` fields and let a supervising broker bridge the node into `lmstudio-proxy` and `omlx-proxy` the same way it bridges Ollama into `ollama-proxy`. A node can run any engine combination.
+PAIR peers must first prove identity through pinned cluster mTLS node-info on `:14318` (or the configured `tls_port`). Matching certificate, response and broker-owned `peerMetadata` identities are required. Only authenticated metadata supplies service ports; models and engine state are then read over pinned mTLS with no HTTP or operator-CA downgrade. Completed reads recheck the exact current pin and admission-bound `peerRecordId`.
+
+Explicitly user-added bare engines retain independent manual routes: Ollama on `:11434`, LM Studio on `:1234`, and oMLX on `:1236`. They remain labelled `identityVerification: "manual"`, never acquire peer trust, and provide no hardware scheduling telemetry. Recognized PAIR configurations that cannot authenticate are labelled `unverified`, not silently downgraded into this bypass. The broker namespaces manual configurations separately from authenticated host UUIDs. Recognition lasts only for the configuration's lifetime in this worker: kind is not yet persisted in `manual-nodes.json`. After a worker restart, an address previously added for a PAIR peer can be routed as an unauthenticated plaintext bare engine if no PAIR endpoint answers there (including if another machine answers instead). Persisting PAIR-vs-bare-engine intent is follow-up work.
 
 `engineNative` is the remote engine's own report of itself — the counters, the rates and the active models the node's engine-manager publishes at `GET /v1/engine-stats`. It is read over the same transport as the model list beside it, because it is the same kind of data: a peer's engine counters are cluster data, so a clustered peer is read over cluster mTLS pinned to the principal the node advertises, and there is no weaker fallback for a peer. The field is absent when the node publishes no view — an engine with no admin API, an engine that could not be reached, or a node this one cannot read. A node whose view could not be re-read keeps the view it had (the last-good rule the node-scanner follows for a discovered node). It is read on a slower cadence than the probe (once a minute), because the engine's counters move slowly and every reading is a fresh observation.
 
@@ -142,7 +144,7 @@ A hostname is preferred over an IP literal: probe clients disable keep-alives sp
 |---|---|---|
 | `address` | Yes | IP address or hostname of the node, with no port |
 | `name` | No | Friendly name (used as node ID; defaults to `manual:<address>`) |
-| `tls_port` | No | Probe node-info over HTTPS on this port instead of plain HTTP on `14318`. Echoed back as `tls_enabled` |
+| `tls_port` | No | Probe pinned cluster mTLS node-info on this port instead of the primary `14318`. Echoed back as `tls_enabled`; no plaintext downgrade |
 | `mtls` | No | Stored and echoed back as `mtls_required`. The probe transport itself is chosen by `tls_port` and live cluster membership, so this field records intent rather than driving it |
 
 Response: the initial node status object.
@@ -200,7 +202,7 @@ Each manual node is probed every 10 seconds, with a 3-second timeout per leg, fo
 
 - **Ollama** on port 11434: health check (`GET /`) and model list (`GET /api/tags`)
 - **LM Studio** on port 1234: `GET /v1/models`, which doubles as the liveness check and the model list
-- **Node Info** on port 14318, or `tls_port` over HTTPS: hardware inventory and identity (`GET /v1/node-info`)
+- **Node Info** on port 14318, or `tls_port`: pinned cluster mTLS hardware inventory, matching identity and authenticated service metadata (`GET /v1/node-info`). Public responses cannot establish identity or scheduling measurements.
 
 A node can have any combination of these, or none if the target is unreachable. Status changes trigger `node/updated` events. Because change detection compares CPU, memory, and GPU values, a node running node-info emits a `node/updated` on most probe cycles as utilization moves.
 

services/nvpair-manual-nodes/enginenative_test.go

@@ -18,6 +18,7 @@ import (
 	"nvpair-shared/clustertrust"
 	"nvpair-shared/clustertrusttest"
 	"nvpair-shared/enginenative"
+	"nvpair-shared/noderec"
 )
 
 // The engine-native half of a manual node: the engine's own report of itself,
@@ -147,8 +148,9 @@ func startPeerEngineManager(t *testing.T, selfDir string, body map[string]any) (
 	})
 	ts := httptest.NewUnstartedServer(mux)
 	ts.TLS = peerMesh.ServerTLSConfig()
-	ts.StartTLS()
-	t.Cleanup(ts.Close)
+	ts.Config.ReadTimeout = 2 * time.Second
+	ts.Config.WriteTimeout = 2 * time.Second
+	t.Cleanup(func() { ts.CloseClientConnections(); ts.Close() })
 
 	_, portStr, err := net.SplitHostPort(ts.Listener.Addr().String())
 	if err != nil {
@@ -158,6 +160,20 @@ func startPeerEngineManager(t *testing.T, selfDir string, body map[string]any) (
 	if err != nil {
 		t.Fatalf("parse listener port: %v", err)
 	}
+	mux.HandleFunc("/v1/node-info", func(w http.ResponseWriter, r *http.Request) {
+		if _, ok := peerMesh.VerifyClientPin(r); !ok {
+			http.Error(w, "forbidden", http.StatusForbidden)
+			return
+		}
+		principal := "uuid-peer"
+		_ = json.NewEncoder(w).Encode(NodeInfoResponse{
+			HostUUID: principal, ClusterUUID: &principal,
+			PeerMetadata: &noderec.PeerMetadataParams{HostUUID: principal, Services: map[noderec.ServiceKey]int{
+				noderec.ServiceNodeInfo: port, noderec.ServiceEngineManager: port, noderec.ServiceOMLX: 1236,
+			}},
+		})
+	})
+	ts.StartTLS()
 	// "localhost" is not a loopback IP literal, so the read is a peer's read
 	// and not this node's own engine-manager.
 	return "localhost", port, stub
@@ -173,6 +189,7 @@ func newClusteredTestManager(t *testing.T, selfDir string, emPort int) *Manager
 	if err != nil {
 		t.Fatalf("NewManager: %v", err)
 	}
+	bindTestManager(t, m)
 	m.emPort = emPort
 	// The plain-HTTP and operator-TLS fallbacks must not answer: a peer is
 	// read over cluster mTLS, and a weaker transport is not a view.
@@ -237,7 +254,7 @@ func TestManualNodeCarriesThePeerEngineNativeView(t *testing.T) {
 	host, port, stub := startPeerEngineManager(t, selfDir, engineNativeBody())
 
 	m := newClusteredTestManager(t, selfDir, port)
-	entry := ManualEntry{Address: host, Name: "peer"}
+	entry := ManualEntry{Address: host, Name: "peer", TLSPort: port, MTLS: true}
 	m.addNode(entry)
 
 	status := waitForProbedNode(t, m, nodeID(entry))
@@ -281,7 +298,7 @@ func TestManualNodeWithNoEngineViewCarriesNone(t *testing.T) {
 	host, port, _ := startPeerEngineManager(t, selfDir, map[string]any{"engine": "omlx", "supported": false})
 
 	m := newClusteredTestManager(t, selfDir, port)
-	entry := ManualEntry{Address: host, Name: "peer"}
+	entry := ManualEntry{Address: host, Name: "peer", TLSPort: port, MTLS: true}
 	m.addNode(entry)
 
 	status := waitForProbedNode(t, m, nodeID(entry))
@@ -298,7 +315,7 @@ func TestManualNodeKeepsTheLastGoodEngineView(t *testing.T) {
 	host, port, stub := startPeerEngineManager(t, selfDir, engineNativeBody())
 
 	m := newClusteredTestManager(t, selfDir, port)
-	entry := ManualEntry{Address: host, Name: "peer"}
+	entry := ManualEntry{Address: host, Name: "peer", TLSPort: port, MTLS: true}
 	m.addNode(entry)
 
 	status := waitForProbedNode(t, m, nodeID(entry))