Skip to content

Isolate cross-process tests, fix the failing broker tests, and add CI - #42

Merged
JaredP94 merged 10 commits into
mainfrom
test/isolation-and-ci
Nov 3, 2026
Merged

JaredP94 merged 10 commits into
mainfrom
test/isolation-and-ci

Conversation

@JaredP94

@JaredP94 JaredP94 commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Description

Port the app-data isolation in NVIDIA/Personal-AI-Router@937cf6c to this fork's separate proxies, diagnose the five reported cross-process failures, and add fork CI adapted from NVIDIA/Personal-AI-Router@b4a0cc5.

Every runtime launch in services/tests now goes through an isolated command factory. HOME, USERPROFILE, XDG_CONFIG_HOME, APPDATA, and LOCALAPPDATA point into disposable test fixtures and are inherited by supervised workers. Explicit fixture roots preserve restart state and remote-engine manifests; Go builds retain their normal caches. An AST guard rejects bypass constructors and environment replacement, a unit test checks all five roots, and a real LM Studio child verifies port persistence under its private root.

Root causes of the five reported failures

All five were reproduced after isolating the process environment. They are test/environment interference, not demonstrated product defects:

  • TestBrokerBridgesManualNodeIntoProxy: the fixture mocked Ollama but not node-info. A running local PAIR supplied a real cluster principal/UUID; the manual-node worker correctly avoided anonymous engine probes for that clustered identity, so the expected synthetic candidate never appeared. Supply a controlled, unclustered node-info fixture as well. The existing learned-UUID test still covers the UUID-keyed path.
  • TestBrokerBridgesManualNodeIntoLMStudioProxy: the same uncontrolled node-info dependency affected the LM Studio fixture. Supply its own unclustered node-info fixture too.
  • TestBrokerAcceptsClientErrorsReport: the IPv4-loopback-only port preflight missed an existing IPv6/wildcard listener. The supervised errors worker then exited on its wildcard bind, leaving no datastore to emit errors:update. Probe wildcard and both loopback families, ignoring unsupported IPv6, and skip when the required port is occupied.
  • TestBrokerAcceptsClientErrorsReportRequest: the same errors-worker bind failure. The broker's immediate acknowledgement explains ack=true, update=false; it is receipt, not confirmation of datastore mutation. The corrected shared preflight prevents the misleading timeout.
  • TestBrokerCrashSurfacingAndRestart: the same unavailable errors datastore prevented the proxy-crash report from surfacing, independently of proxy supervision. Use the corrected preflight. No product error-pipeline change is warranted by this reproduction.

The full isolated run also exposed missing occupied-port checks in workload and cluster-restart cases. Add the same explicit skips there rather than connecting test peers to an unrelated running service. A skip remains untested coverage, not a successful integration assertion.

Desktop intermittent failure investigation

Ran npm run -s test:unit 30 consecutive times, with separate captured logs and exit statuses: 30/30 passed, each with 78 files / 767 tests. The reported intermittent failure was not reproduced; no speculative desktop change was made.

Scope

  • Test-only isolation, fixture corrections, regression coverage, and service testing documentation.
  • CI on every pull request and push to main: Go formatting/vet/tests for all service modules on macOS arm64 and Ubuntu; isolated cross-process tests on macOS only; all requested desktop gates and SPDX checks. Exact action-version tags, npm/Go caches, hosted runners, read-only permissions, and no release automation.
  • Normalize three pre-existing gofmt differences so the formatting gate starts clean.
  • Small broker-internal probe-launch refactor: Go vet could not prove cancellation ownership through the reassigned loop variable. Return context/cancel/result together, retaining existing cancellation semantics. Add stop/restart/worker-exit cancellation coverage.
  • No protocol, engine lifecycle, packaging, or desktop production behavior changes. No changelog edits.
  • Version decision: retain the existing unreleased broker 2.5.0 bump versus 2.4.1 at v0.8.0; it already covers the internal refactor under the fork's unreleased-bump policy. Other binary implementations are unchanged.

Validation

Local environment: macOS arm64, Go 1.27.3, Node 26.7.0. CI pins Go 1.26.7 and Node 25.5.0.

  • go vet ./... and go test -count=1 -timeout=15m -v ./... in all 17 services modules: passed on the final source (untouched modules from the first run; all touched modules re-run).
  • Final committed services/tests suite (go test -count=1 -timeout=10m -v ./...): passed in 286.4 seconds, 18 occupied-port-dependent tests skipped. In particular, the two manual bridge tests ran and passed; the three reported error-pipeline cases skipped because the developer's running PAIR owns port 14319. Their end-to-end success still needs the clean hosted runner; no running developer service was stopped.
  • GOOS=linux go vet ./... and GOOS=windows go vet ./... in services/tests, nvpair-node-info, nvpair-ui-broker, and shared: passed. Native Linux/Windows tests were not run locally.
  • Repeated focused isolation/environment/persistence/IPv6/manual-bridge/remote-engine tests: -count=3, passed. Re-ran test-module vet for native/Linux/Windows after the final guard edits.
  • Red/green evidence: the source guard failed on the original launch helpers; omitting USERPROFILE failed the environment regression; the IPv6-listener regression failed against the original preflight; both manual bridge tests failed before their fixture fixes and passed after; removing exit cancellation failed all three cancellation regression cases, then passed when restored. The original broker vet failure is also recorded.
  • Desktop: typecheck, lint (zero warnings), test:unit, dead-code:check, service-contracts:check, and licenses:check: passed; plus the 30-run unit-suite investigation above.
  • gofmt -l services: empty. node scripts/spdx-headers.mjs: 0 missing/review/unclassified. git diff --check: clean.
  • actionlint v1.7.7: passed. Verified referenced action tags exist. Hosted workflow execution remains to be verified by this PR's checks.

Risk

Mainly test harness and CI behavior. Runtime children no longer share developer app-data; tests requiring an occupied fixed port now skip explicitly. Network discovery is still exercised, so this is app-data isolation, not a network sandbox. The broker refactor preserves the existing cancellation contract and has focused regression coverage. CI may expose runner-specific failures not reproduced locally. The desktop intermittent failure remains unidentified after the requested 30 attempts.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above, including skips and unverified hosted behavior.
  • I checked affected component versions: the existing unreleased broker bump already covers its internal refactor; no additional version bump or user-facing behavior change is needed.

JaredP94 and others added 6 commits November 3, 2026 00:09
Port NVIDIA/Personal-AI-Router@937cf6c to this fork’s standalone proxies. Route every runtime command through a per-test app-data environment, retain explicit fixture roots for restarts and remote engine manifests, and guard against bypassing the command factory or replacing its environment. Keep Go build caches outside disposable application state.

Own the manual bridge tests’ node-info fixture instead of inheriting a running PAIR instance’s cluster identity. Check wildcard and IPv6 listeners before tests require fixed worker ports, including the workload and cluster restart cases, so occupied developer ports are reported as skips rather than misleading product failures.

Co-authored-by: Som <somtri@iastate.edu>
Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Go vet cannot prove the lifetime of the cancellation function assigned inside the watch loop. Extract probe launch into a helper that returns its context, cancellation function, and buffered result together; watch retains the same completion, deadline, and exit cleanup.

Cover cancellation on stop, requested restart, and worker exit. Keep the existing unreleased broker 2.5.0 bump relative to v0.8.0, which already covers this internal refactor.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Adapt NVIDIA/Personal-AI-Router@b4a0cc5 for this fork without release automation. Run component formatting, vet, and tests on macOS arm64 and Ubuntu; run the isolated cross-process suite on macOS only. Check the desktop contracts, attribution, types, lint, unit tests, dead code, and repository SPDX headers.

Pin action tags and toolchains, cache npm and Go modules, use read-only checkout permissions, and normalize three existing test files so the formatting gate starts clean.

Co-authored-by: Noah Tervalon <ntervalon@nvidia.com>
Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Bind install claims to device, inode, and a generation timestamp: birth time on Darwin and ctime on Linux. A replaced Linux bundle must not inherit a claim when its inode is reused. Keep unchanged identities stable and preserve Darwin staging renames.

Add deterministic reused-inode regressions including timestamp nanoseconds, and document conservative invalidation of prior claims. The existing unreleased engine-manager 1.4.0 bump over v0.8.0 already covers this fix.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Install the pinned Go toolchain in the desktop job and download declared modules, including Windows-only dependencies, before the offline license check. Verify the preparation does not change service manifests.

The first run never acquired either macos-latest runner: both jobs had zero executed steps and runner_id 0. Pin the supported macos-15 arm64 image instead of raising test timeouts for an infrastructure queue failure. Emit verbose Go test progress for future slow-test diagnosis.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
The first acquired arm64 runner completed all tests in eight minutes, but real mDNS sends failed with no route to host on en0. Add an explicit on-link IPv4 multicast route on the disposable runner before running the complete isolated cross-process suite. Keep every discovery and workload assertion enabled.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Stream verbose Go JSON events to both the job log and an always-uploaded artifact. Keep the Go deadline below the job deadline so a stuck suite reports its active test and goroutines rather than losing all evidence to job cancellation.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
…ulticast gap

The macOS cross-process job failed 12 discovery tests on every run, and
not because of a hang: with the diagnostics in place the suite finished
in eight minutes. Hosted macOS runners cannot send IPv4 multicast at
all; every mDNS send to 224.0.0.251 fails with "no route to host", even
with an explicit 224.0.0.0/4 route, so no node is ever discovered.

Run the cross-process suite on ubuntu-latest as well, where multicast
works, so discovery is exercised in CI. Tests that need real mDNS now
probe whether this host can multicast and skip only when it cannot,
naming the environment; on any machine that can, including a developer
Mac, they run as before. Drop the route step, which did not help.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Review follow-up. The multicast probe tried only the default route, but
discovery sends through every eligible interface, so a multicast-capable
LAN without a default route would have skipped the discovery tests. The
probe now sends out of each up, multicast-capable IPv4 interface, runs
the tests if any succeeds, skips only when every one fails for lack of a
multicast route, and fails the test on any other socket error. The
decision has its own table test. Also corrects a stale workflow comment.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
Review follow-up. The probe stopped at the first interface that sent, so
an unexpected error on another went unseen. Every result is now
inspected: unexpected errors are logged even when another interface
sent (the test still runs, as discovery tolerates per-interface
failures), and fail the test when no interface could send.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
@JaredP94
JaredP94 force-pushed the test/isolation-and-ci branch from 81925f8 to cb6ce63 Compare November 3, 2026 05:41
@JaredP94
JaredP94 merged commit b715ad6 into main Nov 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant