[Feature]: Add durable Remote Link failure diagnostics and opt-in tunnel debug logging #7297

Activity

1. github-actions

   added

   enhancementNew feature or requestNew feature or request

   on Oct 18, 2026
2. arikon commented on Oct 18, 2026

   

   arikon

   on Oct 18, 2026

   ContributorAuthor

   More actions

   Additional results from the macOS Child after upgrading to 2.88.0. The link is Child-initiated (ssh -L); Home remains on 2.80.0. Personal identifiers, PIDs, addresses and raw headers are omitted.

   Observations

   • A failed request received HTTP 503 2.003492 seconds after request body compression completed in the Codex client log. The preceding request in the same conversation received 200, and the following request received 200 about six seconds after the failure. This measures client-observed elapsed time, not the server-side ownership lookup itself.
   • The proxy and supervised SSH process kept their original PIDs and start times. SSH remained established and owned the expected local listener when inspected; no Home restart was observed.
   • Among 92 first HTTP 503 responses following a logged body-compression event after the Child restart, 71 arrived within 1.95–2.25 seconds of that event. This is a timing sample, not 92 independently confirmed ownership timeouts.
   • Twelve later executions of the exact macOS listener-owner lsof command all found the expected SSH PID in approximately 0.19–0.20 seconds. These checks cannot establish its outcome during the earlier failure.
   • Current /v1/models requests returned 200; an intentionally invalid empty /v1/responses request reached validation and returned 400 without making a model call. Subsequent real Codex requests also received 200. No runtime, service or SSH restart was performed during diagnosis.

   Source correlation and uncertainty

   At main revision 6936d9885d8a9bda9301737affcbaf1a4ff00e17:

   • port-reclaim.ts:363 sets OWNER_LOOKUP_TIMEOUT_MS = 2_000.
   • port-reclaim.ts:448–466 runs lsof on macOS and converts lookup errors or deadline expiry into false.
   • link-tunnel.ts:623–652 requires fresh listener ownership proof for connected().
   • link-relay.ts:369–382 returns the generic link tunnel unavailable 503 if proof fails and the supervisor is not pending.

   An intermittent local ownership-lookup timeout is the leading hypothesis, not an established root cause. The repeated approximately two-second signature fits this path while the SSH process stays alive. OpenCodex does not log the failed lookup's duration, exit/output, error or timeout outcome, so the historical result and why it might have been delayed remain unknown.

   Other configured waits are weaker matches for this timing: reconnect hold 15 seconds, relay authentication 5 seconds, response headers 300 seconds; the 64-waiter cutoff is immediate.

   Please include a correlated ownership-check diagnostic distinguishing match, mismatch, lookup error and deadline expiry, with elapsed time and supervisor state. A stable failure reason such as listener_owner_lookup_timeout would let operators confirm or reject this hypothesis. This does not propose bypassing ownership validation or assume that increasing its deadline is the correct fix.
3. Ingwannu commented on Oct 18, 2026

   

   Ingwannu

   on Oct 18, 2026

   Owner

   More actions

   The timing is useful evidence, but it still does not identify the historical failed lookup. I checked current dev 8640c2f30edb: port-reclaim.ts retains the 2,000 ms owner-lookup limit and the tunnel admission path consumes a boolean ownership result. A successful later lsof sample cannot prove the failed call matched or timed out.

   A focused first PR can preserve that admission result while adding a bounded typed diagnostic: match/mismatch/lookup-error/deadline, elapsed time, check generation and supervisor state, correlated to the relay refusal. Cover injected timeout, spawn failure, mismatched PID, stale completion and success; diagnostics must not change retry/admission policy or let stale proof satisfy a new request. Default output should be metadata only, deduplicated/rate-limited and bounded on disk; opt-in SSH verbosity needs a separate bounded/redacted sink, since raw stderr/argv can contain identities and destinations. Please keep deadline tuning and debug-mode UX out of the first evidence slice. No live tunnel restart or bypass is required to test this.
4. Ingwannu

   added

   needs-design설계 필요. Implementation waits on a written design.설계 필요. Implementation waits on a written design.

   on Oct 18, 2026
5. arikon commented on Oct 19, 2026

   

   arikon

   on Oct 19, 2026

   ContributorAuthor

   More actions

   Follow-up: the generic 503 has appeared with both approximately two-second and five-second timing signatures. Timing alone cannot establish the historical internal failure category or why an operation was delayed. The earlier ownership-lookup hypothesis remains unconfirmed.

   The new result below is a controlled synthetic reproduction, independent of production infrastructure: the public five-second relay-auth deadline can produce the exact same generic 503 before any data request is dispatched. No raw production logs, incident statistics, deployment configuration, identifiers or credentials are included.

   Controlled reproduction of the five-second path

   Synthetic test against main revision 6936d9885d8a9bda9301737affcbaf1a4ff00e17 (2.88.0). With the checkout's dependencies installed, save this as repro-link-auth-timeout.ts in the repository root and run bun repro-link-auth-timeout.ts (approximately ten seconds total).

   The fake peer accepts the authentication request but deliberately never replies. It binds only a dynamically allocated loopback port and uses a synthetic key/identity. The injected positive gate isolates the real connection-bound transport from OS listener ownership checks; fetchImpl is not replaced. It does not modify a running tunnel or use production credentials.

   import { createServer } from 'node:http';
   import { fetchBoundLinkRelay } from './src/client/link-relay-transport.ts';
   import { relayLinkDataRequest } from './src/client/link-relay.ts';
   let authRequests = 0, dataRequests = 0;
   const server = createServer((req, res) => {
     if (req.url?.startsWith('/.well-known/opencodex/link-relay-auth')) {
       authRequests++;
       // Intentionally never respond to the synthetic authentication request.
     } else { dataRequests++; res.writeHead(400); res.end('{}'); }
   });
   await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve));
   const port = (server.address() as {port:number}).port;
   const target = {tunnelPort:port, admissionKey:'synthetic-not-a-real-key', linkId:'lnk_0123456789abcdef', apiKeyId:'synthetic'};
   try {
     let start = performance.now();
     try { await fetchBoundLinkRelay(target, `http://127.0.0.1:${port}/v1/responses`, {method:'POST',body:'{}'}); }
     catch(e:any) { console.log(JSON.stringify({probe:'transport',seconds:(performance.now()-start)/1000,name:e.name,message:e.message,code:e.code,causeName:e.cause?.name,authRequests,dataRequests})); }
     start=performance.now();
     const response=await relayLinkDataRequest(new Request('http://localhost/v1/responses',{method:'POST',body:'{}'}),target,{tunnel:{connected:()=>true,pending:()=>false,waitForConnected:async()=>false}});
     console.log(JSON.stringify({probe:'relay',seconds:(performance.now()-start)/1000,status:response.status,body:await response.json(),retryAfter:response.headers.get('retry-after'),authRequests,dataRequests}));
   } finally { server.closeAllConnections(); server.close(); }

   Observed output (the counters are cumulative across the two sequential probes):

   {"probe":"transport","seconds":5.006377042,"name":"AbortError","message":"The operation was aborted","code":"ABORT_ERR","causeName":"TimeoutError","authRequests":1,"dataRequests":0}
   {"probe":"relay","seconds":5.004834625,"status":503,"body":{"error":"link tunnel unavailable"},"retryAfter":"1","authRequests":2,"dataRequests":0}

   Thus the generic failure is reproducible after an accepted loopback connection, while the tunnel gate reports connected, before any data request reaches the peer. This proves the code path, not that the historical five-second failures necessarily took it.

   Source correlation / diagnostic scope

   At the same pinned revision:

   • relay-auth.ts:6 sets the authentication deadline to 5,000 ms.
   • link-relay-transport.ts:161–183 applies it to the auth probe and rethrows the resulting transport error. Deadline expiry here produces AbortError / ABORT_ERR with TimeoutError as its cause, rather than LinkRelayAuthenticationError.
   • link-relay.ts:392–405 therefore falls through to the generic link tunnel unavailable response, with no underlying-error diagnostic. The existing narrow refused-connection retry does not apply to this abort.

   For the requested bounded diagnostics: correlated refusal metadata should distinguish ownership lookup outcomes from relay-auth deadline expiry and other transport failures. A dedicated injected auth-timeout regression test should assert that no data request was sent and that diagnostic classification preserves the existing 503/admission/retry behavior. This is not a proposal to increase deadlines, bypass admission, or add automatic replay.
6. github-actions commented on Oct 19, 2026

   

   github-actionsbot

   on Oct 19, 2026 – with GitHub Actions

   Contributor

   More actions

   _{Automated translation bookkeeping — detected language: English.}