Fall back to non-EDNS queries on FORMERR

[lattwood]

What

ZDNS always adds an EDNS OPT record to queries. Some authoritative servers return FORMERR when EDNS is present but answer the same query normally without it.

If we get FORMERR without an OPT record in the response, retry the same nameserver without EDNS.

There are a few guardrails:

• Don’t fall back when DNSSEC is enabled.
• Don’t fall back when explicit EDNS options were requested.
• Don’t fall back when the FORMERR response contains OPT—the server understood EDNS in that case.
• Give the fallback its own rate-limit admission and network timeout.
• Don’t cache EDNS support, since that can leave long-running resolvers stuck with stale capability state.

This applies to UDP, TCP, DoT, and DoH, and leaves the existing UDP-to-TCP truncation behavior alone.

Testing

Added coverage for:

• UDP and TCP fallback.
• FORMERR responses with and without OPT.
• DNSSEC and explicit EDNS options.
• Rate-limit admission for both attempts.
• A delayed FORMERR followed by a delayed successful response through ExternalLookup, verifying that both wire operations get a full NetworkTimeout.

Also ran:

• go test ./...
• Focused tests under the race detector.
• golangci-lint run ./...
• The original query against the Microsoft authoritative server.

Fixes #617

[yarrrly]

Not a maintainer. I was looking at how DNS measurement tools handle EDNS-intolerant servers and ended up reading this, so two notes that may save review time.

The condition still reproduces, and it is wider than the one name in #617. Every *.mail.protection.outlook.com name I tried answers FORMERR to an EDNS query and answers normally without one, on both authoritative servers (2026-08-20):

$ dig +norecurse microsoft-com.mail.protection.outlook.com @ns1-proddns.glbdns.protection.outlook.com
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 56406
;; flags: qr; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

$ dig +norecurse +noedns microsoft-com.mail.protection.outlook.com @ns1-proddns.glbdns.protection.outlook.com
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21361
;; ANSWER SECTION:
microsoft-com.mail.protection.outlook.com. 10 IN A 52.101.10.2
microsoft-com.mail.protection.outlook.com. 10 IN A 52.101.60.68
microsoft-com.mail.protection.outlook.com. 10 IN A 52.101.50.10

Same for harvard-edu. and the reporter's 6thfamily-com., against ns1 and ns2. microsoft.com's own MX target is one of these names, so this is the Microsoft 365 mail-routing tier rather than a single misconfigured server, and any iterative scan that resolves MX targets meets it.

Two things in the diff I had to convince myself of:

• shouldRetryWithoutEDNS is narrower than "retry on FORMERR". It requires no OPT in the response and no DNSSEC and no EDNS options in the request, which is the RFC 6891 section 7 case and nothing beyond it. The FORMERR-with-OPT case has its own table entry asserting that no retry happens.
• useEDNS is threaded through the truncation path, so a plain UDP answer that comes back truncated retries over TCP still without an OPT record instead of quietly re-adding EDNS.

The fallback spends a second rate-limiter token against that nameserver, which the tests assert. Only servers that answer FORMERR-without-OPT pay it, which for a scanner seems the right place for the cost to land.