Skip to content

Keep nodes added by address persistent with explicit PAIR or engine intent - #45

Merged
JaredP94 merged 6 commits into
mainfrom
feat/manual-nodes-first-class
Nov 3, 2026
Merged

JaredP94 merged 6 commits into
mainfrom
feat/manual-nodes-first-class

Conversation

@JaredP94

@JaredP94 JaredP94 commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Description

Make addresses added across Tailscale, WireGuard or other routed links persistent first-class entries without weakening #43's pinned-mTLS trust model.

Users explicitly choose PAIR machine (save the address, then PIN-pair) or Engine only (a labelled manual bypass that trusts no PAIR claims). Failed pairing keeps an unverified entry with retry/removal controls. PAIR entries never become bare-engine routes after worker/broker restart or loss of node-info. Unverified PAIR entries are preview:<sha>; engine bypasses remain manual:<hash>.

Upstream context: NVIDIA#10 and NVIDIA#92. Broker persistence equivalent to NVIDIA/Personal-AI-Router@aed93b6 (NVIDIA#60) was already present on the fork and was retained, not duplicated. Adapted NVIDIA/Personal-AI-Router@692950c (NVIDIA#172) successful-unpair cleanup at the shared broker boundary, and NVIDIA/Personal-AI-Router@b3b25ae (NVIDIA#128) normative documentation to this fork. Upstream contributors are credited in the signed-off commit.

Scope

  • Persistence remains in the Go broker, solely in configs/manual-nodes.json; no desktop store or second replay path. Persist/replay kind: pair | engine and retain previously proven associations solely for offline removal, never routing authority.
  • Migration: entries without historical kind are explicitly rewritten on disk to pair. Classification cannot safely infer prior consent to an engine bypass. Former bare engines stop direct routing until removed/re-added as Engine only. Unknown kinds fail closed.
  • PAIR intent uses only pinned cluster mTLS inventory. Engine-only entries read no PAIR claims. Successful cluster removal forgets associated PAIR addresses; failed unpair keeps them. Current local roster addresses cover removal before the first inventory proof. Engine-only entries are independent.
  • Hostnames/IPs are preserved as entered; fresh dials resolve names. Validate bare addresses and explain host:port errors. Existing authenticated overlay routing already handled manual addresses after Require verified peer inventory for service decisions #43; no proxy runtime change was needed. Added hostname mTLS inventory and three proxy transport-selection tests.
  • Desktop Add node, typed manual-node bridge and Cluster settings listing/refresh/retry/removal. Matching TUI choices (a PAIR, e engine, p retry pairing). Shared desktop types are additive; node-list keying and card verification remain owned by Keep desktop peer previews separate from verified machines #44.
  • Component versions relative to v0.8.0: retain existing unreleased minor bumps for manual-nodes (1.3.0) and broker (2.5.0); bump TUI to 1.3.0 for its additive manual-kind surface and cluster-manager to 2.2.0 for the broker-internal authenticated pairing completion notification. Proxy changes are tests only; no proxy version changes. No release/changelog edit.

Validation

Local isolated worktree /tmp/pair-work/manual-nodes, own npm ci --no-audit --no-fund; macOS arm64, Go 1.27.3, Node 26.7.0.

  • Six touched service modules (manual-nodes, broker, TUI, all three proxies): gofmt -l, go vet ./..., go test -count=1 ./..., GOOS=linux go vet ./..., GOOS=windows go vet ./... passed. Worker/broker race suites passed as well.
  • Cross-process module: native/Linux/Windows vet passed. No cross-process tests were run locally. Added real-worker restart coverage for CI.
  • Rebased onto merged Keep desktop peer previews separate from verified machines #44 (origin/main at b7be184); retained nodeRecordKey, nodeVerificationLabel, canPairByAddress and the discovered-list modal regression. Post-rebase desktop typecheck, lint with --max-warnings=0, 858 tests across 87 files, dead-code check, regenerated service contracts and contract check passed.
  • Post-rebase six-module JSON test audit: 1,380 passing cases (including subtests), zero failures. The opt-in TestLiveOMLXCacheAffinityConversation skipped and remains untested; no live engine test was enabled. No other native unit skips.
  • SPDX: 1,296 checked, 0 missing. git diff --check passed.
  • Red/green evidence: the PAIR restart-intent and engine-ignores-PAIR-claims regressions both failed on current-main behavior before the fix. Disabling successful-unpair cleanup made the new removal test fail; disabling pre-proof roster matching made its regression fail. Restoring the fixes passed.

Independent-review follow-ups

  • 3489d28 serializes node-status/telemetry publication with replacement/removal, including exact tracked-entry checks for initial discovery. Kind replacement publishes an empty initial status immediately. TestEnginePublicationCannotFollowPairReplacement forces the reported old-engine/new-PAIR interleaving without sleeps; TestKindReplacementWithdrawsBeforeNewProbe requires immediate withdrawal. Both failed before the fix, pass under the race detector, and passed 50 repeated ordered runs.
  • 052299f emits broker-internal cluster:pairing-completed only after authenticated pairing commit, carrying the original dial hostname rather than the observed numeric roster address. The broker persists existing PAIR aliases' cleanup-only association before later cluster-manager responses, including completion while unpair is in flight. It never marks an entry verified or supplies service authority. TestPairingCompletionPublishesOriginalHostnameForCleanup and TestPairingCompletionForgetsHostnameBeforeInventoryProof failed before the fix; the latter includes broker restart before any node-info proof. TestPairingCompletionDuringUnpairForgetsHostname covers the ordered in-flight notification.
  • Post-fix go test -race -count=1 ./... and native/Linux/Windows vet passed in all seven affected Go modules, now including cluster-manager. Desktop gates still pass (858 tests); contracts regenerated with an explicit internal-notification exception. SPDX: 1,299 checked, 0 missing.
  • Initial run 41643032848 had five passing jobs; macOS Go modules failed the existing TestMacmonSupervisorBackoffDoublesCapsAndResets timing-cap assertion. Run 41645196846 at 3489d28 subsequently passed all six jobs. Neither is claimed as final-head certification.

Final lifecycle ordering and reassignment corrections

  • 7d4039c: completion publication now occurs under the producer's roster commit/removal lock, with cluster/admission binding. It is fire-and-forget and never waits for a broker acknowledgement or request reply; the independent reader consumer does no worker round trip or mutation-lock wait. The broker rejects obsolete/removed completions. At most 16 transient address hints support an already in-flight unpair when the roster file has already lost the peer; they expire on every exit and grant no identity/routing or persistence authority. TestPairingCompletionPublicationOrdersConcurrentRemoval and TestObsoletePairingCompletionCannotAssociateManualEntry failed before this fix. Additional tests cover removed-admission guards and hint expiry/bounds.
  • 98e0e67: a persisted peer association is authoritative over cached identities, reused addresses and hints. Cached inventory/re-add cannot overwrite it. Deletion rechecks association ownership atomically with the durable write, protecting a reassociation that arrives during removal. TestCurrentManualAssociationSurvivesStaleObservationAndReadd, TestUnpairAssociationOverridesCachedIdentityAndReusedAddress (hostname and numeric address), and TestUnpairRechecksAssociationChangedDuringRequest all failed before this fix and now pass.
  • At this final source head, all seven affected Go modules passed go test -race -count=1 ./..., native/Linux/Windows vet. Desktop gates passed again (858 tests), contracts regenerated/checked; SPDX: 1,303 checked, 0 missing. No cross-process suite was run locally.

Final-head CI (98e0e67, run 41664943412) and Linux artifact skip audit are pending.

Risk

Migration intentionally stops historical bare-engine routing until the user explicitly chooses Engine only. Pairing failures leave an unverified saved address, not an engine fallback. No opt-in unclustered PAIR route exists. Cleanup failure after successful unpair is reported explicitly; membership may already be removed, and the manual entry can be forgotten separately after resolving the error.

DNS names are not frozen to resolved addresses; live pinned connections can be reused, and fresh connections re-resolve. Local tests and CI are not a live overlay deployment or packaged-artifact certification. Live multi-machine operation, Windows runtime and desktop launch remain unverified.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.

JaredP94 and others added 6 commits November 3, 2026 17:41
Keep broker-owned manual addresses first-class without allowing a PAIR entry to downgrade after worker replay. Persist kind, conservatively migrate historical entries to PAIR, and reserve the labelled manual bypass for explicitly chosen engines. Retain proven associations only for lifecycle cleanup, and forget PAIR addresses after successful unpair for every client. Add matching TUI choices and regression coverage.

Broker persistence equivalent to NVIDIA/Personal-AI-Router@aed93b6 (NVIDIA#60) is already present in the fork; retain that single owner rather than re-porting it. Adapt the successful-unpair cleanup from NVIDIA/Personal-AI-Router@692950c (NVIDIA#172) at the broker boundary, and adapt the normative specification from NVIDIA/Personal-AI-Router@b3b25ae (NVIDIA#128) to the pinned-mTLS trust model.

Co-authored-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
Co-authored-by: Terve <ntervalon@nvidia.com>
Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Make Add node save explicit PAIR or engine-only intent through the broker before starting PIN pairing. Keep unsuccessful pairing as an unverified address, expose manual entry listing, retry and removal under Cluster settings, and label the engine-only bypass. Add typed bridge channels and interaction tests without changing node-card verification or discovery keying.

Remove the old eager, ignored manual removal from the desktop membership handler: successful-only unpair cleanup now belongs to the shared broker transaction.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Keep probe state commits and node notifications in the same critical section as replacement and removal. Publish an empty replacement status immediately so a changed kind withdraws its prior route before the next network probe. Late initial-discovery callbacks also check the exact tracked entry under the publication lock.

An ordered interleaving regression pauses an engine probe at its publication boundary, replaces it with PAIR intent, and detects the obsolete engine update restoring the bypass. A separate regression requires withdrawal before a blocked replacement probe completes. Both failed before this fix; the race suite and fifty repeated ordered runs pass.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Emit a broker-internal completion association only after the cluster manager commits authenticated pairing. Retain the original dial hostname, not the numeric source address stored in the roster, and persist matching PAIR aliases for cleanup before any node-info inventory proof. The association never enters renderer state or grants identity, inventory, telemetry, or routing authority.

Include completion associations received while unpair is in flight, so successful removal cannot leave a hostname for replay. Producer and broker regressions failed before the fix; they now cover hostname versus numeric roster address, broker restart before proof, independent engine entries, abandoned pairing, and an ordered in-flight completion. Add the internal contract exception, regenerate service documentation, and bump cluster-manager to 2.2.0 for its additive notification.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Publish authenticated pairing completion inside the roster commit/removal transaction, and include its cluster and admission epoch. The broker rejects obsolete or removed admissions before writing cleanup associations. An already in-flight unpair may retain at most sixteen transient original-address hints so it can forget an unassociated hostname even if the roster file was removed before notification consumption; hints expire on success or failure and never provide routing or identity authority.

The notification is fire-and-forget: publishing under rosterMu never waits for an acknowledgement or a broker request response. The independent broker worker read pump consumes it while the request caller may be relaying nodes:remove. Its consumer does no worker round trip and never waits on manualMutationMu, avoiding a request/publication lock cycle.

The ordered producer/removal regression and obsolete-completion regression both failed before this fix. Current race suites, three-platform vet, desktop gates and contract checks pass. Tests also cover a removed roster during an in-flight unpair, successful-removal admission guards, and bounded hint expiry.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Match an associated manual entry only against its current persisted peer UUID. Use cached identity, address and transient-hint fallbacks only for entries with no association. Cached inventory and ordinary re-adds can seed missing metadata but cannot overwrite a current pairing association.

Recheck ownership under the persistence lock when deleting each alias, not only when collecting pre/post-removal snapshots, so authenticated reassociation while the removal RPC is in flight preserves the new peer entry and its live worker record.

Regressions for A-to-B hostname reassociation, reused numeric addresses, stale cached proof and re-add, and in-flight reassociation all failed before this fix and now pass. Seven Go race suites, native/Linux/Windows vet, desktop gates and contract checks pass.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
@JaredP94
JaredP94 force-pushed the feat/manual-nodes-first-class branch from 98e0e67 to cd76f56 Compare November 3, 2026 15:51
@JaredP94
JaredP94 merged commit 043beba into main Nov 3, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant