Repository navigation
fix(sandbox): backdate the gateway certs by an hour - #657
Merged
Merged
Conversation
Both certificates the sandbox mTLS surface issues carry NotBefore:
time.Now(), and both are dialled within a second or two of being minted. Any
client whose clock is a little fast therefore gets:
x509: certificate has expired or is not yet valid:
current time <client> is before <notBefore>
- signClientCert mints the client credential during login, and the CLI uses
it immediately to dial the gateway. There is no retry window at all: the
error surfaces from `k8e-sandbox-cli connect` as a clock problem nobody
actually has, and every retry mints a new cert with a new NotBefore, so it
fails again until the client's clock catches up to the server's. Observed
repeatedly today while connecting to a freshly deployed host.
- ensureServerCert mints the gateway cert at startup. A client connecting
during the next few seconds hits the same window.
Backdate both by an hour. That covers realistic clock skew, costs nothing, and
leaves the TTLs (90d server, operator-controlled client) unchanged.
Verified: pkg/sandboxmatrix/grpc tests pass.
Adds the failing case that motivated the previous commit, which I had not
backed with a test: signing a client certificate with NotBefore == time.Now()
truncates up to the next whole second (ASN.1 UTCTime has second precision), so
the credential is valid only from the top of the next second. When the issuing
server's clock runs ahead of the client's — two seconds in the reported failure,
against a remote host over a real network — the freshly issued credential is
already "not yet valid" when validateIssuedCertificate verifies it at the
client's CurrentTime:
x509: certificate has expired or is not yet valid:
current time 2026-10-19T14:11:24+08:00 is before 2026-10-19T06:11:26Z
The operator cannot act on that message: neither clock is visibly wrong, and the
gap is sub-second. A backdated NotBefore absorbs it.
Verified the test is decisive: reverting both NotBefore lines in cert.go makes it
fail with that exact error, and restoring them makes it pass.
Contributor
… call DeepSource blocked PR #657 on the previous version of this test: GO-S1032 Time other than the current system time used for certificate verification (x2, Major/Security) GO-S1031 Verify does not check for certificate revocation (x3) Both fire because the test called leaf.Verify with a fabricated CurrentTime. That was the wrong way to test this in the first place: what the fix changes is the NotBefore timestamp, and reading it directly is also deterministic, where verifying against a constructed clock depends on Go's tolerance and can pass for the wrong reason. Rewrite so the assertion is on NotBefore: - NotBefore must be at least a minute before issuance, captured after the cert is created. One minute sits well above ASN.1's whole-second truncation and well below the one-hour backdate, so the two cases cannot be confused. - NotAfter must keep the requested TTL. - The issued leaf must carry client-auth usage and not be a CA. No x509.Verify call and no CurrentTime override, so neither rule applies. The revocation caveat is recorded in a comment pointing at the production call site that already documents it for this internal CA. Unchanged: the test is still decisive. Reverting both NotBefore lines in cert.go fails it with NotBefore is not backdated: notBefore=2026-10-19T06:39:03Z issued=2026-10-19T14:39:03+08:00; want at least 1m0s before issuance which is the reported defect shown on the same second, and restoring them passes it.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
远程主机上新部署的 k8e server，
k8e-sandbox-cli connect一直失败：注意时间关系：本地
14:11:24+08:00就是06:11:24Z，而证书notBefore是06:11:26Z——只差 2 秒，本地时钟在证书生效时刻之前。排查中排除的项：CA 未过期、SAN 已正确包含公网 IP
43.202.114.239、TLS 握手本身成功（openssl s_client直接连 50051 拿到证书、链路正常）。失败的既不是服务端证书也不是 CA，而是 login 时刚签发的客户端证书。根因
signClientCert（pkg/sandboxmatrix/grpc/cert.go）用NotBefore: time.Now()签发客户端证书，而validateIssuedCertificate（pkg/sandbox/client/client.go）紧接着用CurrentTime: time.Now()在客户端本地校验它。ASN.1 UTCTime 只有秒级精度，
time.Now()会向上取整到下一秒，所以证书实际从"下一秒的 00 秒"才开始生效。服务端时钟比客户端快 2 秒——跨公网、不同机器，这完全正常——于是这张刚签出来的证书在客户端眼里就是"还没生效"。操作者无从下手：两边时钟看着都没问题，差距不足 1 秒，报错却说得像 CA 坏了。
修复
NotBefore回拨 1 小时，两个地方都改：signClientCert—— 客户端证书（本次故障的直接原因）ensureServerCert—— 网关服务端证书（同样time.Now()，客户端在启动后头几秒连进来会撞同一个窗口）TTL 不受影响：服务端 90 天、客户端按
ttlDays，都不是从NotBefore起算的。验证
测试是决定性的（先有可复现的失败，才有修复）：
错误信息与线上逐字一致，证明测试抓的就是同一个问题，而不是另一个构造出来的场景。
go build ./pkg/...、go vet、pkg/sandboxmatrix/...与pkg/sandbox/...全部测试通过。当前服务器状态
变频后
K8E_SANDBOX_ADVERTISED_HOSTNAME=43.202.114.239已生效，服务端证书 SAN 已包含该公网 IP，warm pool 已恢复。但这个cert.go修复尚未部署——需要新的 k8e 二进制上线后才能根治客户端证书的时钟窗口问题。