Skip to content

agentmother heartbeat loop dies while A2A server stays up (passive agent goes silently offline) #174

Description

@NeilJo-GY

Scope note

The fix lives in the agentmother agent codebase (agent-side), NOT in ACN.
Tracked here only because this is the team's active issue tracker. Move to the
agentmother repo if/when that tracker exists.

Symptom

A paid Store order's store.order_paid message did not trigger agentmother's
/a2a -> poll_orders() in real time. Investigation showed:

  • The agent's HTTP A2A server was up and reachable (http://43.160.236.79:80/a2a).
  • But ACN considered the agent offline: the Redis alive key had expired
    (TTL 60min, renewed on each 15min heartbeat), so the message was parked in the
    inbox instead of being pushed.

Conclusion: the heartbeat loop stopped for >60min while the HTTP server kept
running
— classic "async background task swallowed an exception and exited
silently".

Why it matters

Even with the ACN-side fix (ADR-0012 "reachable == online", commit 5683143),
which now probes reachable endpoints regardless of the alive key, a robust
agent should keep its own liveness signal healthy. A dead heartbeat also
corrupts online/offline stats and any logic that reads agent status.

Action

  • Find why agentmother's heartbeat loop terminated (>60min gap)
  • Make the heartbeat task supervised/restarting: never let an unhandled
    exception kill it silently; log + auto-restart with backoff
  • Consider an implicit-heartbeat fallback: renew liveness on any successful
    inbound A2A handling, not only via the dedicated heartbeat call

Unblock the stuck order

Order 53cce555-6dae-4d1c-b62e-eaf19402de83 is paid and sitting in the inbox;
bringing the agent online (or its cron inbox-pull) will process it. Not lost.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready-for-humanNeeds human implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions