Skip to content

Retention should reclaim idle open groups now that expiry is timestamp-only #3161

Closed

Description

@kixelated

#3099 split group lifetime into two rules: subscription expiry is timestamp-only (a group is stale once its reach falls a full max-age budget behind the newest frame of the latest group), and wall-clock reclamation of idle content belongs to the retention/cache layer.

The expiry half landed in #3099. This issue tracks the reclamation half, which currently has gaps at the model layer:

  • The Rust cache::Pool ages by last access, but an in-process track without a pool has no wall-clock reclamation at all, and a group held open by a stalled publisher pins its buffer until something else closes it.
  • The JS track retention prune deliberately retains open groups, so an abandoned open group survives Publisher Max Age indefinitely.

Concrete symptom (from Codex's adversarial review of #3099): a reader drains an open group whose successor starts less than the budget ahead in media time, production goes quiet, and the pending read stays parked forever. Timestamp expiry is correct to keep the group (nothing proves it is useless), so the bound has to come from retention: an idle open group past the retention window should be reclaimed (aborted), which surfaces to a parked reader as the gap it is.

Needs a decision on what "idle" means for an open group (no producer writes for the retention window is the obvious candidate), a timer/wakeup to arm it, and coverage in both Rust and JS. The draft's Expiration section already frames retention as the publisher's own policy, so this is implementation work, not a wire change.

(Written by Claude Fable 5)

Activity

  1. kixelated commented on Sep 5, 2026

    @kixelated
    CollaboratorAuthor

    Fixed by #3419, merged to dev as 50018e7. Closing by hand: the Closes keyword only fires on merges to the default branch.

    The root cause turned out to be broader than this issue described. cache::Track::settle is reachable only from write paths, so the pool's wall-clock LRU window did not apply to a quiet track at all: it kept its entire cached backlog forever, closed groups included, not just an open one. Verified with a probe: at a 1s expiry, a closed non-latest group in a quiet track was still cached after 600 virtual seconds.

    A pool with an expiry now keeps a registry of its live track accounts, and origin::Driver sweeps it on a wall-clock cadence (half the window) using timers it already holds. The sweep starts at the oldest entry rather than the write path's rotating cursor and stops once the front stops yielding victims, so its cost tracks what it reclaims: measured at ~400 ns per track per sweep, independent of cache depth. A pool with no expiry registers nothing.

    js/net got the matching fix: the prune ages a group from its last write, reclaims an idle open group with Lagged, and arms a timer so a stalled publisher needs no further write. The newest group stays protected while open, since that is the live edge a @moq/json Snapshot catalog legitimately holds across a quiet stretch.

    One gap is documented rather than closed: the sweep rides origin::Driver, so a standalone broadcast::Info::produce() keeps only the write-driven half. Every non-test call site in this tree spawns the driver, but it is a public path, so cache::Config::with_expiry says so.

    (Written by claude-opus-5[1m])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions