Skip to content

Circuit breaker for direct-delivery to long-down Mode A agents #173

Description

@NeilJo-GY

Context

Follow-up to ADR-0012 "reachable == online" (commit 5683143).

MessageRouter.route() now ALWAYS attempts the direct HTTP push for Mode A
agents regardless of the Redis alive key, with a bounded 5s connect timeout
to cap the cost of probing a down host. On failure for a believed-offline
agent it parks the message gracefully in the inbox (no DLQ, no raise).

Problem

For a Mode A agent that is genuinely down for a long time (e.g. registered
then abandoned) while senders keep messaging it, every message now pays up to a
~5s connect timeout before falling back to the inbox. The cost is bounded per
message and attempts can run concurrently, but it is still wasted work at scale.

Proposal

Add an opt-in circuit breaker so we stop probing an endpoint that has been
failing consistently:

  • Track recent consecutive direct-delivery failures per agent (Redis counter
    keyed acn:agents:{id}:push_fail, short TTL window).
  • When failures exceed a threshold N within the window, trust "offline" and
    route straight to the inbox without the HTTP probe.
  • Any successful delivery (or touch_alive) resets the counter.
  • Keep it strictly additive to the current behaviour; default thresholds tuned
    so a momentary blip never trips the breaker.

Deferred deliberately

This was deferred in the ADR-0012 work pending evidence that the short connect
timeout is insufficient. Only worth building if/when we observe meaningful
probe cost against dead Mode A agents in production.

Acceptance

  • Failure-count tracking + threshold gate in route() direct path
  • Reset on success / touch_alive
  • Unit tests: breaker trips after N failures, resets on success, blip does
    not trip
  • Configurable threshold/window (with safe defaults)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions