fix(consul): run the catalog and health watchers as independent timers

[nic-6443]

Description

Consul discovery can fail a whole fetch round with a bogus resty.http error when the catalog and health indexes change together. This PR stops killing the watchers, which is what causes it.

connect() spawns the catalog and health watchers as light threads, waits for the first one and kills both. The watchers go through resty.consul, which reads the response body via resty.http, and resty.http reads a body in a child coroutine. If the watcher that gets killed is still reading its body (typically the health watch streaming a large /v1/health/state/any response while the catalog watch returns first), ngx.thread.kill doesn't cancel the read pending in that child coroutine. When it completes, it resumes connect() at whatever it's waiting on, usually the status-line receive of the next catalog fetch, and that fetch fails with:

fetch_services_from_server(): connect consul: ... by sub url: /catalog/services, got catalog result: ".../resty/http.lua:479: bad argument #1 to 'str_sub' (string expected, got boolean)"

Now each index has its own watcher timer, and one run is one blocking query, so nothing gets killed and every run frees its memory when it ends. The watcher that sees a change runs the round itself: validate, fetch, back off on errors, same code as before. Rounds don't overlap: a watcher that sees a change while another round is running parks, and gets re-armed together with the round's watcher when it ends, so it queries again against the updated indexes and only triggers another fetch if something is still new. watch_catalog / watch_health now run a single query and return nothing when the index didn't change; the retry moved to the timer, so stop_flag and worker exit are checked between queries.

What stays the same: indexes are only committed after a successful fetch, the retry/backoff progression is still one per server, and without keepalive the rounds are still paced by fetch_interval. What changes: a watcher that didn't report a change keeps its blocking query open across rounds instead of being killed and restarted; each server runs two timers instead of one timer with two light threads; and if a timer can't be created, only that watcher stops instead of the whole loop.

t/discovery/consul-watch-race.t reproduces the race with a mock Consul that bumps both indexes together, answers the catalog watch shortly after the health watch starts streaming a slow chunked body, and delays the non-blocking catalog read so the health body finishes while the fetch is waiting. It fails on master with the got boolean error and a 503, and passes with this change.

Which issue(s) this PR fixes:

N/A

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change
• I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open findings

• watch_result_is_valid returns false both for error indexes (index <= 0) and for 'no longer new'… · New
• watch_catalog/watch_health now conceptually run 'one blocking query per timer run', but they… · New
• The PR description says: 'if a timer can't be created, only that watcher stops instead of the whole… · New
• This test relies on several wall-clock sleeps (mock server timing + ngx.sleep(6)) but only allows… · New
• This test relies on several wall-clock sleeps (mock server timing + ngx.sleep(6)) but only allows… · New

What changed in this PR

Fixes a Consul discovery race where killing one watcher mid-body-read can corrupt subsequent fetches, by running catalog/health watches as independent timer-driven blocking queries and avoiding ngx.thread.kill.

Changes:

• Reworked Consul discovery loop to use two independent timer-based watchers that serialize fetch “rounds” (in_round + parking).
• Updated Consul client watchers (watch_catalog/watch_health) to perform a single blocking query per run and return “no change” as nil.
• Added an integration test reproducing the prior race (consul-watch-race.t).

File	Description
t/​discovery/​consul-watch-race.t	Adds a regression test that reproduces the watcher-kill/body-read race and asserts it no longer breaks fetches.
apisix/​discovery/​consul/​init.lua	Refactors the polling loop to two timer-driven watchers and introduces “round” serialization/parking logic.
apisix/​discovery/​consul/​client.lua	Changes watchers to run a single blocking query and return nil when indexes don’t change.

🧠 Review effort: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.