Skip to content

fix(tandem): stop handing the same loopback port out twice - #658

Merged
xiaods merged 1 commit into
mainfrom
fix/tandem-test-address-collision
Oct 19, 2026
Merged

xiaods merged 1 commit into
mainfrom
fix/tandem-test-address-collision

Conversation

@xiaods

@xiaods xiaods commented Oct 19, 2026

Copy link
Copy Markdown
Owner

失败现象

CI 里 TestTandemPersistentTxnCompareTargets 挂了,报的是:

[store] 2026/10/19 07:17:11 node loaded in 2.976471ms (1274 bytes)
--- FAIL: TestTandemPersistentTxnCompareTargets (15.03s)
    persistence_integration_test.go:81: rqlite failed readiness
    persistence_integration_test.go:261: fatal: HTTP and Raft addresses must differ

第二行才是真正的错误,第一行的 readiness 超时只是它的表象。

根因

testAddress 用「监听 127.0.0.1:0 → 立刻 Close」的方式取空闲端口,端口在返回前就已经被释放了。于是连续两次调用可能拿到同一个端口——分配器会把刚释放的再发出来。

rqlite 发现 HTTP 地址和 Raft 地址是同一个,直接拒绝启动(fatal: HTTP and Raft addresses must differ);而测试侧只看到 /readyz 15 秒超时,报错指向时钟,完全看不出是端口撞了。

出问题的调用点:

位置 场景
persistence_integration_test.go:289 address, raft := testAddress(t), testAddress(t) 背靠背两次
persistence_integration_test.go:237,240 同一个进程里多次分配
cluster_integration_test.go:168-169 每个成员的 httpAddr/raftAddr 一对
cluster_integration_test.go:261-262 单成员 fixture 同上

修法

登记已分配的端口,撞了就重试:

testPortMu.Lock()
duplicate := testPortsUsed[port]
if !duplicate { testPortsUsed[port] = true }
testPortMu.Unlock()
if !duplicate { return addr }

条目永不释放,所以一对 HTTP/Raft、以及多节点 fixture 里每个成员的地址,在整个测试进程内都保持互不相同。

新增测试

TestTestAddressNeverRepeatsAPort 覆盖两种关键形状:rqlite 的一对地址,以及多节点 fixture 每成员的地址序列。

关于这个测试我要说清楚:我在本机上没能让旧实现失败——10 次连续运行、每次 200 个调用,无重复。内核是否复用刚释放的端口是分配细节,测试从外部逼不出来。所以它断言的是不变量而不是复现故障,产线 CI 上的 flake 才是"冲突真实存在"的证据。这一点写进了测试注释,免得有人看到绿了以为已经复现。

验证

gofmt -l          → 干净
go vet(tandem_integration + tandem_fault 两种 tag)→ 干净
go test -run TestTestAddressNeverRepeatsAPort -count=3 → ok

go test -tags tandem_integration ./pkg/tandem/ 有 20 个 --- FAIL,但全部是 TANDEM_TEST_* 环境变量未设置的既有失败(需要 Linux 版 Tandem 二进制,CI 里有)。我做了 A/B:把改动 stash 掉回到 main,失败集合逐一相同,所以本次改动没有引入任何新失败。

顺带说明

同一次 git stash pop 曾让 sandboxd/ 出现冲突。我已重置到 HEAD 状态,并逐个文件校验 sandboxd/src/{exec,main,pty,background}.zig 与 main 的 md5 完全一致——PR #653 的退出码修复没有被破坏。

TestTandemPersistentTxnCompareTargets failed with

    rqlite failed readiness
    fatal: HTTP and Raft addresses must differ

testAddress picked a free port by listening on 127.0.0.1:0 and closing before
returning it. The second of two back-to-back calls could therefore receive the
same port, because the allocator recycles what was just released. rqlite, given
one address for both its HTTP API and its Raft listener, refuses to start, and
the fixture only sees a readiness timeout — a message that names a clock problem
rather than the port collision it actually is.

Record each handed-out port and retry on a collision. Entries are never
released, so the HTTP/Raft pair and every member of a multi-node fixture stay
distinct for the whole process run.

Test added: TestTestAddressNeverRepeatsAPort covers both shapes that matter —
the rqlite pair, and the per-member address sequence.

Honest note on the test: I could not make the old implementation fail here, on
this host, across 10 consecutive runs of 200 calls each. Whether the kernel
recycles a just-freed port is allocation detail, not something a test can force
from outside. So this asserts the invariant rather than reproducing the failure,
and the production CI flake is the evidence the collision is real. It is stated
in the test comment so nobody reads a green run as proof of a repro.

Call sites now deduped rather than colliding:

  persistence_integration_test.go:289   address, raft := testAddress(t), testAddress(t)
  persistence_integration_test.go:81    the readiness timeout that surfaced it
  cluster_integration_test.go:168-169    per-member httpAddr/raftAddr pair
  cluster_integration_test.go:261-262    same, in the single-member fixture
@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

978 tests  ±0   951 ✅ ±0   6m 58s ⏱️ +4s
120 suites ±0    27 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 0312e92. ± Comparison against base commit 40062af.

@xiaods
xiaods merged commit 4e9d875 into main Oct 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant