Repository navigation
fix(tandem): stop handing the same loopback port out twice - #658
Merged
Merged
Conversation
TestTandemPersistentTxnCompareTargets failed with
rqlite failed readiness
fatal: HTTP and Raft addresses must differ
testAddress picked a free port by listening on 127.0.0.1:0 and closing before
returning it. The second of two back-to-back calls could therefore receive the
same port, because the allocator recycles what was just released. rqlite, given
one address for both its HTTP API and its Raft listener, refuses to start, and
the fixture only sees a readiness timeout — a message that names a clock problem
rather than the port collision it actually is.
Record each handed-out port and retry on a collision. Entries are never
released, so the HTTP/Raft pair and every member of a multi-node fixture stay
distinct for the whole process run.
Test added: TestTestAddressNeverRepeatsAPort covers both shapes that matter —
the rqlite pair, and the per-member address sequence.
Honest note on the test: I could not make the old implementation fail here, on
this host, across 10 consecutive runs of 200 calls each. Whether the kernel
recycles a just-freed port is allocation detail, not something a test can force
from outside. So this asserts the invariant rather than reproducing the failure,
and the production CI flake is the evidence the collision is real. It is stated
in the test comment so nobody reads a green run as proof of a repro.
Call sites now deduped rather than colliding:
persistence_integration_test.go:289 address, raft := testAddress(t), testAddress(t)
persistence_integration_test.go:81 the readiness timeout that surfaced it
cluster_integration_test.go:168-169 per-member httpAddr/raftAddr pair
cluster_integration_test.go:261-262 same, in the single-member fixture
|
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



失败现象
CI 里
TestTandemPersistentTxnCompareTargets挂了,报的是:第二行才是真正的错误,第一行的 readiness 超时只是它的表象。
根因
testAddress用「监听127.0.0.1:0→ 立刻 Close」的方式取空闲端口,端口在返回前就已经被释放了。于是连续两次调用可能拿到同一个端口——分配器会把刚释放的再发出来。rqlite 发现 HTTP 地址和 Raft 地址是同一个,直接拒绝启动(
fatal: HTTP and Raft addresses must differ);而测试侧只看到/readyz15 秒超时,报错指向时钟,完全看不出是端口撞了。出问题的调用点:
persistence_integration_test.go:289address, raft := testAddress(t), testAddress(t)背靠背两次persistence_integration_test.go:237,240cluster_integration_test.go:168-169cluster_integration_test.go:261-262修法
登记已分配的端口,撞了就重试:
条目永不释放,所以一对 HTTP/Raft、以及多节点 fixture 里每个成员的地址,在整个测试进程内都保持互不相同。
新增测试
TestTestAddressNeverRepeatsAPort覆盖两种关键形状:rqlite 的一对地址,以及多节点 fixture 每成员的地址序列。关于这个测试我要说清楚:我在本机上没能让旧实现失败——10 次连续运行、每次 200 个调用,无重复。内核是否复用刚释放的端口是分配细节,测试从外部逼不出来。所以它断言的是不变量而不是复现故障,产线 CI 上的 flake 才是"冲突真实存在"的证据。这一点写进了测试注释,免得有人看到绿了以为已经复现。
验证
go test -tags tandem_integration ./pkg/tandem/有 20 个--- FAIL,但全部是TANDEM_TEST_*环境变量未设置的既有失败(需要 Linux 版 Tandem 二进制,CI 里有)。我做了 A/B:把改动 stash 掉回到main,失败集合逐一相同,所以本次改动没有引入任何新失败。顺带说明
同一次
git stash pop曾让sandboxd/出现冲突。我已重置到 HEAD 状态,并逐个文件校验sandboxd/src/{exec,main,pty,background}.zig与main的 md5 完全一致——PR #653 的退出码修复没有被破坏。