Skip to content

Rust EClient fails to reconnect and causes 100% CPU infinite spin-loop when connection is lost #399

Description

@x-ira

Environment

  • Crate Version: v0.7.1 (or latest commit)
  • Language Binding: Rust
  • OS: macOS / Linux (Laptop sleep/resume or network interruption)

Problem Description

When running a Rust client, if the connection is interrupted (e.g., closing the laptop lid, putting the system to sleep, or temporary network drops), the ibx engine fails to reconnect and immediately traps the process in an infinite spin-loop, driving CPU usage to 100%.

Root Cause & Log Analysis

Upon waking up the system or experiencing a network flash-drop, the hot_loop continuously spams the following warning at a zero-delay frequency:

2026-07-31T09:48:08.364531Z ERROR ibx::engine::hot_loop::ccp: CCP connection lost: connection closed
2026-07-31T09:48:11.537858Z  WARN ibx::engine::hot_loop: CCP auto-reconnect skipped: no credentials
2026-07-31T09:48:13.579228Z ERROR ibx::engine::hot_loop::hmds: HMDS connection lost: Connection reset by peer
2026-07-31T09:48:18.489932Z  WARN ibx::engine::hot_loop: Farm auto-reconnect skipped: no credentials (host empty or auth missing)
2026-07-31T09:50:21.157930Z  WARN ibx::engine::hot_loop: CCP auto-reconnect skipped: no credentials
... (repeats indefinitely, hogging the entire CPU core)

This behavior stems directly from the known gap where EClient::connect_inner (src/api/client/mod.rs) never propagates credentials via update_reconnect_auth to the hot_loop for Rust callers, while Python handles it perfectly.

Because auth.host remains empty, the reconnect schedulers (CCP, HMDS, and Farms) all bail out early via safety guards like:

let auth = match self.reconnect_auth.as_ref() {
    Some(a) if !a.host.is_empty() => a,
    _ => return, 
};

However, since it returns instantly without executing any asynchronous network I/O or performing any explicit time-deferred backoff yields (because the network attempt was technically skipped), the event loop or retry mechanism drops into a tight busy-wait / infinite spin. It constantly retries, prints the warning, and loops back instantly, starving the CPU thread.

Impact

  • Any temporary connection loss or system sleep completely freezes/overloads the Rust process, making it impossible to recover without killing the entire application.
  • The beautiful jittered backoff mechanics introduced in #218 and #219 become entirely unexecuted dead code for Rust callers under this state.

Suggested Solution

  1. Fix the Core Gap: Ensure update_reconnect_auth is properly invoked within connect_inner for Rust callers, passing the four authentication items just like the Python path does.
  2. Defensive Looping: In the hot_loop or the scheduler, if a reconnect attempt is explicitly skipped due to missing credentials, ensure the task explicitly yields or sleeps (e.g., tokio::time::sleep) instead of instantly slamming back into the loop cycle, avoiding 100% CPU spinning.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions