Conversation
…andoff (dae#1013) Two-layer fix for silent traffic blocking after staged same-port handoff: Layer 0 — DNS fingerprint fix: - Exclude runtime-tunable params (OptimisticCache, OptimisticCacheTtl, MaxCacheSize) from dnsConfigFingerprint; they are atomic-tunable via DnsController.UpdateRuntime and do not affect BPF map state. - Add dnsRoutingUnchanged flag to ControlPlane; when true, skip clearReloadDomainRoutingMap + replayDnsReloadCache in both CommitPreparedDatapath and the non-deferred commit path. Layer 1 — Tiered reload via bpfDatapathChanged(): - New function detects all B-class map changes (routing rules, groups, DNS routing, interface bindings, conn-state map size, SoMarkFromDae). - When BPF datapath config changes, force a full reload (fresh BPF objects) instead of staged handoff, eliminating the shared-map race window where the old plane loses domain routing. - Safe (pure userspace) fields — log_level, check URLs, dial_mode, sniffing_timeout, TLS, bandwidth, optimistic_cache, etc. — continue to use staged handoff for zero-downtime reload. Tests: 11 new TestBpfDatapathChanged_* cases + updated fingerprint coverage test. go vet + gofmt clean, full cmd/control suites pass.
|
我还以为可以把 reload 删掉，逃 |
实机测试，还有问题
|
…, and recovery steps Records: - Latest deployed version: v2.0-custom @ 151de49 - GitHub PR status (daeuniverse#1009, daeuniverse#1011, daeuniverse#1016) - 2.4 server fault timeline (2026-06-18 10:11 ~ 11:00) - Root cause analysis (config parse error + tail timeout 5s) - Known PR daeuniverse#1016 limitation (same-config reload still uses staged handoff) - Configuration pitfalls (Chinese comments cause token error) - Deployment and recovery commands
b0cc523 to
45f4f7f
Compare
45f4f7f 实机验证结果
测试过程
测试结果
验证截图日志确认流量正常经 dae 代理: 结论
|
When config is unchanged and a reload is triggered, the staged handoff path shares BPF objects between old and new generations. The old generation's closeTail() calls core.Close() synchronously within a 5s timeout window. core.Close() runs TC filter detachment (netlink.FilterDel) which can exceed the timeout, causing "control plane close tail timed out" warnings (dae#1013). In staged handoff, by the time closeTail() runs, EjectBpf() has already transferred BPF ownership to the new generation. core.Close() only needs to detach TC filters and release the UDP tracker — neither is time-critical because the new generation already has its own TC filters attached. Run core.Close() in a background goroutine when isBpfEjected() is true. This check covers all staged handoff scenarios including the first reload (P0→P1) where sharedBpfReload is false but EjectBpf has been called. Using bpfEjected rather than sharedBpfReload ensures the async path is taken based on actual BPF ownership state, not creation history. This preserves zero-downtime staged handoff while eliminating the closeTail timeout. Reported-by: @itoywh See: dae#1013
45f4f7f to
9f65298
Compare
|
感谢 @itoywh 的实机验证！ 基于反馈重新审查后，我们对 问题
staged handoff 的设计目的就是让策略变更无缝迁移。#1013 的根因是 DNS 运行时参数变更触发 修正
验证在 dae.lan（Kernel 6.17, AVX2）上测试全部通过：
关键点：路由规则变更现在正确走 staged handoff（零停机），而非 full reload。 |
真机测试，还是遇到卡死情况
|
…routing_map race When DNS routing config changes (upstream, routing rules, etc.), the non-staged constructor path calls clearReloadDomainRoutingMap + replayDnsReloadCache on the shared BPF object. Under staged handoff the old generation keeps routing traffic on the same map during its drain period, so the synchronous clear creates a race window where the old plane loses domain routing entries and returns Operation not permitted (dae#1013). The previous fix (9f65298) removed bpfDatapathChanged() believing Layer 0 (dnsRoutingUnchanged flag) was sufficient. Layer 0 only covers runtime-tunable parameters (optimistic_cache etc.) that are excluded from the fingerprint; when DNS routing itself changes, dnsRoutingUnchanged is false and clear+replay still runs on the shared map during staged handoff. Gate stagedHotHandoff on dnsConfigEqual(conf, newConf) so that DNS routing changes fall back to non-staged reload, where the old generation is ejected immediately and clear+replay runs synchronously in the new plane's constructor with no drain-period race. Reported by @itoywh: daeuniverse#1016
|
感谢 @itoywh 的详细复现和实机验证！这个分析完全正确。 根因确认
但当 DNS 路由本身变更（upstream、routing 规则等）时， 这与 #1013 原始报告是同一个竞态，只是触发条件不同：
修复已推送 stagedHotHandoff := !portChanged && listener != nil && dnsConfigEqual(conf, newConf)DNS 路由变更时不再走 staged handoff，而是走非 staged 路径： 修复后行为矩阵
DNS 路由变更失去零停机是已知的 trade-off：staged handoff 的 clear+replay 设计在共享 map 上有结构性竞态，full reload 虽有短暂中断但比静默阻断安全。后续可以考虑增量更新 domain_routing_map（只删不用的、不全清）来恢复这个场景的零停机，但那是独立优化。 能否帮忙在 |
1 similar comment
|
感谢 @itoywh 的详细复现和实机验证！这个分析完全正确。 根因确认
但当 DNS 路由本身变更（upstream、routing 规则等）时， 这与 #1013 原始报告是同一个竞态，只是触发条件不同：
修复已推送 stagedHotHandoff := !portChanged && listener != nil && dnsConfigEqual(conf, newConf)DNS 路由变更时不再走 staged handoff，而是走非 staged 路径： 修复后行为矩阵
DNS 路由变更失去零停机是已知的 trade-off：staged handoff 的 clear+replay 设计在共享 map 上有结构性竞态，full reload 虽有短暂中断但比静默阻断安全。后续可以考虑增量更新 domain_routing_map（只删不用的、不全清）来恢复这个场景的零停机，但那是独立优化。 能否帮忙在 |
|
|
真机测试，还是有问题 ，头大 |
…aeuniverse#1039) Add post-commit datapath validation that checks dae0 (handle 0x2022) and all LAN/WAN interfaces (handle 0x2023) have TC filters attached after commitInterfaceBindings. Silent bind failures would otherwise cause traffic to bypass the proxy with zero error feedback (daeuniverse#1013 class of bugs). Validates both reload paths (non-deferred + staged handoff). On failure, returns descriptive error listing every missing binding. Testability: linkByName / filterLister are package-level vars so unit tests can inject mocks without touching the real netlink stack. Adds control_plane_core_test.go covering present/missing/link-not-found cases. Relates: daeuniverse#1013 daeuniverse#1016 daeuniverse#1037 daeuniverse#1038
3 participants
Summary
Fixes silent traffic blocking (
Operation not permitted) after staged same-port hot handoff reload, reported in #1013.Root Cause
When a user modifies DNS runtime parameters (e.g.,
optimistic_cache,optimistic_cache_ttl,max_cache_size) and triggers a reload, the staged handoff path shares the BPF object between old and new control planes. The DNS fingerprint included these runtime-tunable fields, causingdomain_routing_mapto be cleared synchronously and replayed asynchronously (via a capacity-1024 channel with silent drops) while the old control plane was still actively routing packets on the shared map. This created a race window where the old plane lost domain routing entries, causing transient routing failures.Two-Layer Fix
Layer 0 — DNS fingerprint fix:
OptimisticCache,OptimisticCacheTtl,MaxCacheSize) fromdnsConfigFingerprint; they are atomic-tunable viaDnsController.UpdateRuntimeand do not affect BPF map state.dnsRoutingUnchangedflag toControlPlane(passed via constructor); when true,CommitPreparedDatapathand the non-deferred commit path skipclearReloadDomainRoutingMap+replayDnsReloadCache.Layer 1 — Tiered reload via
bpfDatapathChanged():SoMarkFromDae.log_level, check URLs,dial_mode,sniffing_timeout, TLS settings, bandwidth limits,optimistic_cache, etc. — continue to use staged handoff for zero-downtime reload.Reload Decision Matrix
optimistic_cache, etc.)log_level, check URLs,dial_mode, etc.)Test Coverage
TestBpfDatapathChanged_*test cases covering all detected fields + safe-field exclusionsTestDNSConfigFingerprintCoversAllDnsFieldswith excluded-set contractgofmt,go vetclean; fullcmd+controltest suites pass with zero regressionsTest Plan
go build -tags dae_stub_ebpf ./cmd/passesgo vet -tags dae_stub_ebpf ./cmd/ ./config/ ./control/passesgofmt -lclean on all modified filesgo test -tags dae_stub_ebpf ./cmd/ ./control/— all passFixes #1013