tokio-rt-worker panic causing crash w/o restarting Kubernetes Pod

[gl-jkeys]

PgDog version
e.g., v0.1.34 or a41e41d1

Helm chart v0.58

Description
pgdog issues a tokio panic, but then failed to restart the affected pod. This left our environment in a bad state, and required a manual restart of the underlying Kubernetes Pod to remediate.

Logs
Attach trace logs, if the issue can be reproduced locally. To get trace logs, set RUST_LOG=trace, e.g.:

thread 'tokio-rt-worker' (7) panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/time/entry.rs:181:13:
mark_pending called when the timer entry is in an invalid state

Configuration

pgdog.toml:
----
[general]
port                      =   5432
workers                   =   2
default_pool_size         =   10
log_format                =   "text"
log_level                 =   "info"
pooler_mode               =   "transaction"
healthcheck_interval      =   30_000
idle_healthcheck_interval =   30_000
idle_healthcheck_delay    =   5_000
healthcheck_timeout       =   5_000
rollback_timeout          =   5_000
load_balancing_strategy   =   "round_robin"
read_write_strategy       =   "conservative"
tls_client_required       =   false
tls_verify                =   "prefer"
shutdown_timeout          =   60_000
prepared_statements       =   "extended"
passthrough_auth          =   "disabled"
connect_timeout           =   5_000
connect_attempts          =   1
connect_attempt_delay     =   0
checkout_timeout          =   5_000
dry_run                   =   false
client_idle_timeout       =   60000
mirror_queue              =   128
mirror_exposure           =   1.0
auth_type                 =   "scram"
cross_shard_disabled      =   false
openmetrics_port = 9090
openmetrics_namespace = "pgdog_"
server_lifetime = 86400000
log_connections = true
log_disconnections = true
expanded_explain = false

[levkk]

Hello! Please upgrade to latest PgDog version - I think this might be a bug in Tokio. We've seen something like this before in #1052. The latest version of PgDog also uses the latest version of Tokio.

On the issue of restart: it's not usually necessary; Tokio will re-spawn the thread and continue to serve traffic. I suspect the bad state in your case came from the applications misbehaving when their requests get interrupted, but I can't be sure. It would be good to have more details on what actually happened, e.g.,:

1. Couldn't create new connections to pgdog
2. Existing clients crashed

Thanks!

[gl-jkeys]

Thanks for the quick response as always! We will upgrade ASAP.

Regarding the bad state: we saw this panic and, after it happened, all requests from every client began to fail with: connect ETIMEDOUT (NodeJS applications). That seems to indicate that the requests are being issued correctly, but the TCP packets are being swallowed upstream of the client?

[levkk]

That's a bad one. I suspect Tokio landed in an unrecoverable state. Typically, a panic inside a thread is fine, Tokio just spawns another one, but a bug in the runtime can certainly stop all requests from flowing through.

Let me know if this happens again after you upgrade. Let's keep this issue open in the meantime.

[gl-jkeys]

Quick update. So far so good on the update to Helm chart 0.68. We have not observed any additional tokio panics since then.

I will keep this issue open a while longer, but so far, it does appear that the more recent version is stable for us.

(Side question: I noticed that the chart uses the latest tag for the pgdog repo. Does that mean the Helm chart installs are not idempotent, in the sense of which pgdog version gets installed?)

[levkk]

(Side question: I noticed that the chart uses the latest tag for the pgdog repo. Does that mean the Helm chart installs are not idempotent, in the sense of which pgdog version gets installed?)

Our chart? No, it uses appVersion: https://github.com/pgdogdev/helm/blob/main/Chart.yaml#L4. Fwiw, you should just tag the image you want like this:

image:
   tag: v0.1.48

[gl-jkeys]

Ah okay, thanks. I must've missed that. We will definitely pin to the current version we're on once we've validated its stability.

Thanks again for all your help!!

[levkk]

Anytime!

[hydeta]

We hit what looks like the same failure on v0.1.44 (tokio 1.52.3), 2026-07-20

Setup: GKE Autopilot, Helm chart, 3 replicas, workers = 2, pooler_mode = "session", ~150 pool size, asyncpg clients.

At 13:53 UTC one replica started panicking and then panicked on every new client connection for ~85 minutes:

thread 'tokio-rt-worker' (7) panicked at .../tokio-1.52.3/src/util/linked_list.rs:195:9:
assertion failed: self.tail.is_none()

Different panic site than the OP (linked_list.rs vs time/entry.rs) but it looks like the same timer-wheel corruption family as #1052. Matching the OP's report: the pod kept accepting TCP and completing SCRAM auth, then dropped each connection (clients saw connection was closed in the middle of operation), and Kubernetes never restarted it - our readiness probe is TCP-only, so the pod stayed Ready and kept taking ~1/3 of traffic until we manually killed the affected pod.

Two questions:

1. Is this expected to be fixed by fix: switch to alternative Tokio timer for >2 workers #1234 (alternative Tokio timer)? If so, is there an ETA for a tagged release containing it?
2. In this poisoned state, would the healthcheck_port HTTP endpoint have started failing? We're adding it to our probes and want to know if it can detect this, given the panics only fired on new connection handling.

[levkk]

I don't think so. Try upgrading to latest version. We've been running it pretty hot lately and haven't seen this crash, but this is almost certainly a race so a negative doesn't prove anything.

Worth looking through Tokio's backlog too to see if this was fixed.

[hydeta]

Ok thanks, will for now upgrade and report back if we encounter this again.

[meskill]

Hi @gl-jkeys @hydeta Have you experienced this or related issues with tokio after the upgrade?

[meskill]

We reproduced the issue and this has the same root cause like #1017 and the updates will be in this issue