fix(researcher): pin the outbound fetch to the validated address (WALM-682)

[nikola0x0]

Closes WALM-682. Triage of the 22 Sep static security review (WALM-679); MW-03 confirmed against source — the old docstring named the gap itself.

What was wrong

assertPublicUrl resolved a user-supplied hostname, confirmed every answer was public, then returned the URL and threw the addresses away. fetch resolved the same name a second time, and nothing tied the two answers together. A host that replies with a public address to the check and a private one to the connection walks straight through — DNS rebinding. Every redirect repeated it.

Reachable from chat: the schema takes z.string().url(), and an application/pdf file part carries that URL to fetchPublicUrl unchanged.

The fix

fetch cannot express this — it takes a URL and does its own resolution. The request now goes out through node:http/node:https, which take the connect address and the TLS identity separately:

• the socket goes to an address the check approved;
• Host stays the original host, including a non-default port;
• servername stays the hostname, so SNI and certificate verification are unchanged — pinning costs nothing in TLS terms;
• an IP literal gets no SNI, because the certificate must then carry the address itself; rejectUnauthorized stays on either way.

The second resolution is simply gone, so there is no longer a window to win. Redirects keep being followed by hand and now re-pin as well as re-check on each hop; redirect bodies are cancelled rather than left streaming, and a 30s timeout replaces fetch's absence of one.

fetchPublicUrl takes an injectable transport. Two existing tests staged a public first hop by stubbing globalThis.fetch, which this no longer goes through; the seam lets them drive the real redirect loop instead of a monkeypatched global.

Scope note

The ticket's preferred shape — a server-issued upload id replacing the arbitrary download URL — removes the surface rather than fencing it, but it changes the chat file flow. Not done here.

Worth recording for severity: only the pdf-url branch fetches server-side. The plain url branch goes out through r.jina.ai, so Jina does that fetching, not us, and the response must parse as a PDF before any of it returns as text. Closer to blind SSRF than to clean internal-response exfiltration.

Verification

20 tests in this file — the 15 that already existed plus 5 for the pin, including one that dials a loopback server through a URL naming example.com, where the handler running at all is what proves the name was never re-resolved. Researcher suite 49 pass (44 before), build clean.

[ducnmm]

[bug] new Response(...) runs in the http/https callback with no try/catch. Status must be 200–599 and statusText must be a valid reason-phrase. Node still surfaces the rest: HTTP/1.1 600 and HTTP/1.1 101 throw RangeError, and a reason-phrase with \x01 throws TypeError: Invalid statusText. The throw is not on this promise. Reproduced on Node 26: uncaughtException (default is to exit the process) and the promise never settles — the status line is already parsed, so the 30s socket timeout does not reject. NULL_BODY_STATUS does not help 101; the body is dropped and status: 101 still throws. The previous fetch rejected into processSource. A chat pdf-url can crash the worker or hang ingest.

Suggestion: Construct the Response inside try/catch, res.destroy(), and reject with ChatbotError. Map any status outside 200–599 to that error before calling Response. If statusMessage is not tab/SP/VCHAR/obs-text, pass "".

[nikola0x0]

Fixed in 03507ed. Confirmed first: new Response() throws RangeError for 101/199/600 and TypeError for a \x01 reason-phrase, and inside the callback that was an uncaughtException with the promise left hanging. Now a status outside 200–599 destroys the response and rejects with ChatbotError, an invalid reason-phrase becomes "", and construction is wrapped. 101/103 are out of NULL_BODY_STATUS since they can no longer reach it. Raw-socket tests send HTTP/1.1 600, a bare 101, and a \x01 phrase. All three fail on the previous commit with your errors and pass now.

[ducnmm]

[suggestion] Only addresses[0] is dialed. lookup(..., { all: true }) does not set ADDRCONFIG and defaults to verbatim order. Here example.com resolved AAAA-first, then IPv4. Setting connect host to that one IP is what drops the second lookup, and it also disables autoSelectFamily (on by default), which undici fetch used to fall over to the other family. The app image is node:22-alpine, which often has no IPv6 egress, so a dual-stack PDF URL can fail even though a later A record was already public. Not a rebinding bypass — every entry was checked.

Suggestion: On ENETUNREACH / EHOSTUNREACH / connect timeout, try the next address from this same list. Do not lookup again and do not put the hostname back in host.

[nikola0x0]

Fixed in 03507ed, your way: each validated address is tried in order, no second lookup, and the hostname never goes back into host. It only falls through on failures before a connection exists (refused, unreachable, or a new 10s connect deadline, since req.setTimeout is idle-only). Once connected, a TLS/HTTP failure is the server's answer and isn't retried against a sibling address.

[ducnmm]

Summary

The uncaught Response crash is fixed, and an immediate connect refusal (ECONNREFUSED / ENETUNREACH) now falls through the validated address list. The connect-timeout path does not. On Node 22 the global agent’s 5s socket timer fires while the handshake is still pending, req.setTimeout reports that as a plain ChatbotError, and fetchFirstReachable stops on addresses[0]. The 10s connect deadline never runs. A dual-stack host whose AAAA is dropped rather than refused still fails the whole PDF fetch on the first address.

The pin still holds: no second lookup, and servername / rejectUnauthorized stay on the original hostname.

Issue counts by severity

• bugs: 1
• suggestions: 0
• nits: 0

[harrymove-ctrl]

Approved. Follow-up commits address uncaught exception on out-of-range HTTP status lines and reason phrases when pinning outbound fetch to validated addresses. CI checks pass.