Back the discovery registry with a file instead of shared memory — Closes #353 by conradbzura · Pull Request #441 · wool-labs/wool

conradbzura · 2026-09-18T20:49:36Z

Summary

Replace the POSIX shared-memory segment backing LocalDiscovery's registry with an ordinary file, and extend the resulting ownership model to every artifact a namespace holds.

The motivating defect is not the segment but its custodian. multiprocessing.shared_memory.SharedMemory enrols every segment with CPython's resource_tracker, a service wool does not use and, below Python 3.13, cannot decline, so the module carried 271 lines whose sole function was escaping that enrolment. Two of the faults the machinery existed to work around — #336 and #340 — are tracker accounting errors rather than wool errors, and are unreachable once no tracker participates.

Address the file at fixed offsets through a held descriptor rather than mapping it. On Apple platforms CPython issues a device-level flush (F_FULLFSYNC) whenever it maps a file descriptor — a cost of milliseconds per mapping, imposed on the whole host — while pread and pwrite obtain the same cross-process visibility through the page cache without it. This departs from the mechanism the issue specifies, and the access pattern is the warrant: the registry is a header and fixed-width slots addressed by offset, never traversed as memory, so the mapping purchased nothing the descriptor does not provide.

A file-backed registry makes the ownership model expressible, which is the larger half of the change. Each entry of a LocalDiscovery context mints a generation — a directory holding that owner's registry, notification file and worker blocks — published by renaming a pointer atomically into place. The owner owns every artifact within it and stamps the sizes every borrower binds at; its exit frees all of it. A borrower that outlives its owner raises DiscoveryNamespaceNotFound at its next read or write, rather than reaching a successor's registry or serving the snapshot it last read, and that error now reaches the caller rather than settling on a task nobody awaits.

Two residuals are accepted rather than solved. An owner killed outright leaves its generation on disk until something re-enters the namespace — which, for the per-lifecycle pool-<uuid> namespaces a WorkerPool mints, is never; containment under a single wool/ directory bounds the recovery at rm -rf <root>/wool. A process forked from an entered owner disowns the claim but cannot release it, so the namespace remains in use until that fork exits.

Breaking changes

DiscoveryNamespaceInUse loses its segment field and LocalDiscovery.Publisher loses its block_size parameter, which the owner now stamps. Every discovery exception's field becomes required rather than optional, and DiscoveryNamespaceNotFound renders a message that no longer asserts the registry is absent, since it fires while a successor's may be present. Of these four, two alter the args shape cls(*exc.args) reconstructs — the segment removal and the required-field change — and are therefore wire-compatibility breaks; Publisher is not an exception and carries no args, and the message change leaves args at (namespace,) on both sides. WorkerProxy now raises TypeError for an async generator passed as discovery, which it previously accepted: pass the subscriber itself rather than the generator its call returns. LocalDiscovery's block_size floor rises from 5 to the length prefix plus the smallest serialized worker metadata — 68 under the current schema, and derived from it, so a schema change moves the floor rather than stranding the validator — and constructions between the two floors, which previously succeeded and then failed every publish, now raise ValueError at construction. Finally, an entry whose namespace directory is replaced faster than it can be claimed raises DiscoveryNamespaceContended, a new exported exception, where it previously retried without bound.

Closes #353
Closes #376
Closes #345
Closes #340

Proposed changes

Back the registry with a file

Remove multiprocessing.shared_memory and multiprocessing.resource_tracker from local.py, along with _suppressing, _create, _attach, _tracker_lock, _shims, _reinit_tracker_state, the sys.version_info gate and _short_hash. tests/integration/test_local_discovery_tracker.py observed a fault emitted by a process wool does not control, and with the tracker gone it has no subject.

Take every lock through fcntl directly and drop the portalocker dependency. local.py was the package's only consumer, and routing one of four locks through a wrapper produced two spellings of one syscall, two exception vocabularies and two sets of flag constants. The wrapper's stated justification was a portability the module no longer possesses: it imports fcntl and os.pread unconditionally, and the package classifies only macOS and Linux.

Narrow DiscoveryNamespaceInUse to name the namespace rather than a shared-memory segment, and make every discovery exception's field required. No raise site in the tree produces the unknown case — each supplies a concrete value — so the optionality obliged every handler to narrow a value the library never leaves unset.

Scope every artifact to an owner's generation

<root>/wool/<namespace>/     the claim; outlives any one owner
├── current                  symlink to the live generation
└── <generation>/            one per owner entry
    ├── registry
    ├── registry.tmp         transient, between write and rename
    ├── notify
    └── blocks/<worker-uuid>

Generation names are minted per entry and never reused, so a borrower pinned to one generation cannot find a different owner's file at a path it remembers. Blocks sit one level down, so the directory a subscriber watches carries registry traffic alone.

The owner stamps both capacity and block_size into the registry header, and a borrower binds at those values rather than supplying its own. Sizing an artifact the owner owns is not a borrower's decision: two publishers with different block sizes would otherwise leave differently-sized blocks in one generation, determined by whichever registered each worker first.

A publisher no longer owns the blocks it creates. A drop removes the block and nulls its slot within one critical section under the registry lock, which collapses the four-state slot-block space and eliminates the unsynchronised deletion that allowed one publisher's teardown to unlink a block another was writing. A publisher's exit releases handles and removes nothing, so the leak in #345 has no remaining path: no artifact's reclamation depends on a borrower surviving to perform it.

Both borrower kinds bind through one _Binding primitive — resolve the pointer, read the header, pin a descriptor on the generation, probe the owner — so a publisher and a subscriber cannot disagree about what a missing generation, a replaced directory or a dead owner means, and a descriptor exhaustion cannot present as a vanished namespace.

Claiming a namespace sweeps whatever a killed predecessor left before staging. The sweep is sound only because a binding now ends with its owner: holding the claim proves the absence of a live writer. It resolves nothing — it lists and unlinks through descriptors, decides directory-ness by fstat rather than by the errno of a failed unlink, and descends only through O_NOFOLLOW, because /dev/shm is mode 1777 and shared by every user on the host. What it cannot remove it reports as a ResourceWarning rather than suppressing: a stranded generation is the larger of the two leaks this module can produce.

Detect an owner that did not exit

Removing a generation is what makes a stale borrower fail, and it covers an owner that exits and an owner swept by a successor. It does not cover an owner killed outright with nothing replacing it, whose files survive and remain readable.

The owner therefore holds an exclusive flock on its generation directory for that generation's life, and a borrower tests the lock — shared, non-blocking, through a descriptor pinned at bind — within work it already performs. A lock that cannot be taken proves the owner's process alive; one that can be taken proves the kernel dropped it, which it does however the owner died. This is the oracle LMDB and SQLite employ. Only contention answers "alive": a descriptor this process no longer holds answers "dead" because it proves nothing, and anything the probe cannot answer is raised rather than guessed.

Probing the generation rather than the claim ensures a borrower can never cause a rival claimant's lock attempt to fail; taking the probe shared ensures concurrent borrowers do not exclude one another. Pinning the descriptor costs 583 ns against 10.4 µs for reopening per check — roughly 1% of a publish, measured at 55–65 µs, and 0.35% of a scan over eight workers, measured at ~167 µs. The probe adds latency and never contention: while the owner lives, every probe fails immediately without acquiring anything.

A parked subscription wakes on _LIVENESS_INTERVAL to re-verify that its owner still holds the generation. The wake is a liveness check, not a rescan — two non-blocking locks and a readlink, against a rescan's read of every registered worker's block and an event per worker — so a subscription with no poll interval stays silent over an idle namespace for as long as it lives, and one with a long poll interval still detects a lost owner without waiting that interval out.

Read a block read-only, through an unbuffered handle. The subscriber opens one block per registered worker per scan and never writes, while _File's reads and writes go directly to pread/pwrite on the descriptor, so the buffered read-write handle paid for a write mode it never used and an 8 KiB buffer nothing touched. Measured on APFS: 18,647 ns to 11,039 ns per block, a 39% reduction.

Disown an inherited claim in a fork

A process forked from an entered owner inherits its descriptors and its atexit teardown, so the fork's ordinary exit reclaimed the live parent's namespace. An owner now records the pid that entered it, and teardown running in a process whose pid no longer matches removes nothing. That check is the entire mechanism, and it is the stronger of the two available: it also covers fork(2) issued from C, which an os.register_at_fork handler never observes.

Claiming is split from staging so that a failure has one meaning at each step. Until the claim's lock is held this process owns nothing, and a failure there closes only what it opened — releasing instead would purge the incumbent's generation, which an ENOLCK on a network filesystem or a KeyboardInterrupt landing mid-call would otherwise trigger. Staging owns the descriptor it opens and closes it on any failure, so the retry loop leaks nothing.

Deliver a failure to every consumer of a shared subscription

Fanout multicasts one source to several consumers and delivered an exception only to whichever consumer was pulling. The shared iterator remained bound to the generator its own exception had closed, so the next consumer received StopAsyncIteration from it — every consumer but one ended as though the source had run out.

The pull now belongs to the container rather than to the consumer that starts it. A consumer waits on that task without owning it, so a cancellation reaches only that consumer: the pull runs on, the item still reaches every queue, and the shared generator is never thrown into. This is what renders one consumer's teardown its own business, and the property matters in production: two WorkerProxy instances on one namespace share a Fanout, so stopping one would otherwise freeze the other's membership feed silently.

Cleanup closes the iterator rather than the source, which is what actually runs the subscriber's finally and releases its descriptor and watchdog thread; a source with no aclose is no longer mistaken for a teardown failure, and a genuine close failure propagates to the pool, which knows which resource it belongs to.

Surface a dead subscription to the caller

WorkerProxy consumes its subscription in a detached task, so the exception settled on a task nobody awaited until teardown. A pool whose namespace lost its owner waited out its whole quorum timeout and reported asyncio.TimeoutError; a later dispatch reported NoWorkersAvailable. The class already documented the intended behaviour; the implementation now provides it.

The failure is read from the sentinel task rather than cached, so no window exists in which the task holds the answer and the proxy does not. The sentinel wakes the quorum wait from its own finally, and the placement is load-bearing: every other setter of that event lies on a path the sentinel owns, so a caller parked in the wait would otherwise wait out the timeout for an event nothing could set.

Teardown reports the failure only when nothing else is reporting one. A failure a dispatch already raised is not raised again, and one that would displace an exception already unwinding the caller's block is logged instead — a teardown report is worth less than the failure the caller is already handling. Each report restores the chaining state the failure held when first reported, so a caller's traceback is the origin plus their own call rather than the origin plus every call that preceded it.

A proxy whose subscription failed stays started — it retains the balancer, the channel-pool hold and whatever membership it last held — so every later dispatch the membership cannot serve reports the same failure, and retrying those does not help. Recovery is stopping the proxy and starting another, or leaving and re-entering the pool context. With no quorum configured there is no wait for the cause to arrive in, so a dispatch issued first still reports NoWorkersAvailable, and the cause reaches the dispatch after it; closing that first-dispatch window requires a readiness seam through the subscription pipeline, whose design #444 carries.

Release a pooled subscriber's factory with its subscriptions

SubscriberMeta registered a construction's factory in a module-global dict that was never pruned. A caller minting a fresh key per lifecycle — as WorkerPool does, one pool-<uuid> namespace per instance — left one closure behind for the life of the process, under a key tuple holding the subscriber class with it.

The registry was redundant. A shared subscription already carries the construction its factory closes over, so one fact had two homes with different lifetimes: the subscription's copy dies with it, the registry's never did. The factory now travels inside the cache key, a frozen pair whose factory member is excluded from equality and hashing, so sharing is decided by the identity alone, exactly as before, while the closure is reachable only from the subscriptions that could use it.

Registering in the old dict hashed the key there, so a key callable returning an unhashable value failed at the construction that supplied it. Carrying the factory in the key would defer every hash to the pool and report the caller's mistake from inside the first iteration, so the identity is hashed at construction instead.

A per-acquisition factory parameter on ResourcePool.get was considered and rejected: the pool already lets the caller choose the key, so the payload belongs in the key rather than beside it, and a parameter silently ignored on a cache hit promises more than it delivers.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	`TestLocalDiscovery`	A namespace no owner holds	An owner enters it	It stages a generation and publishes the pointer to it	Claim and stage
2	`TestLocalDiscovery`	A live owner holding a namespace	A second owner claims the same namespace	It raises `DiscoveryNamespaceInUse`	Single ownership
3	`TestLocalDiscovery`	An owner whose registry is staged	A borrower probes during the rename	It binds a stamped registry or none	Atomic publication
4	`TestLocalDiscovery`	A predecessor killed with its generation intact	A fresh owner claims the namespace	It sweeps the predecessor and leaves only its own generation	Sweep on claim
5	`TestLocalDiscovery`	A rejected claim registered for teardown, and an unrelated directory on the descriptor number it released	That rejected instance is exited	It leaves the unrelated directory untouched	Claim sentinel
6	`TestLocalDiscovery`	A namespace whose generation publication fails partway through	An owner enters and exits it	It holds no more descriptors than before	Staging owns what it opens
7	`TestLocalDiscovery`	An incumbent owner and a filesystem whose lock call fails for a reason other than contention	A second owner is entered	It raises that error and leaves the incumbent's worker discoverable	Claim failure reclaims nothing
8	`TestLocalDiscovery`	An owner and a child forked from it	The child runs the inherited teardown and exits	It removes nothing and the parent's namespace still binds	Fork disown
9	`TestLocalDiscovery`	An owner whose block size exceeds the default	A separately constructed publisher publishes oversized metadata	It publishes successfully	Owner stamps the sizes
10	`TestLocalDiscovery`	An owner with a capacity of one and a worker registered	A borrower registers a second worker	It raises `DiscoveryCapacityExhausted` naming the owner's capacity	Borrower binds the capacity
11	`TestLocalDiscovery`	An owned namespace whose registry is overwritten unstamped or truncated	A publisher binds it	It raises `DiscoveryNamespaceNotFound`	Header is validated once, at bind
12	`TestLocalDiscovery`	An owner whose teardown removals fail	The owner exits	It reports each entry it could not remove as a `ResourceWarning`	Sweep failures are not silent
13	`TestLocalDiscoveryPublisher`	A publisher with a registered worker	The worker is dropped	It frees the slot and removes the block together	Slot and block lifetime
14	`TestLocalDiscoveryPublisher`	A publisher bound while its owner was live	The owner exits and a successor claims the namespace	It raises `DiscoveryNamespaceNotFound` and reaches no successor	Binding ends with its owner
15	`TestLocalDiscoveryPublisher`	A namespace whose owner was killed outright	A publisher binds it	It raises `DiscoveryNamespaceNotFound`	Liveness at bind
16	`TestLocalDiscoveryPublisher`	A publisher whose context has been exited	It publishes again	It raises `RuntimeError` naming the misuse	Lifecycle is explicit
17	`TestLocalDiscoverySubscriber`	A subscription streaming a worker	Its owner exits mid-iteration	It raises rather than serving the snapshot	Loud failure over stale reads
18	`TestLocalDiscoverySubscriber`	A registry slot naming a block that was removed	A subscriber scans	It skips the slot and still reports the readable worker	Carry-forward
19	`TestSubscriberLiveness`	A subscription whose owner is killed with the registry left readable	The subscription scans again	It raises `DiscoveryNamespaceNotFound`	Liveness lock
20	`TestSubscriberLiveness`	A live owner, one worker, and a drained subscription with no poll interval	No publisher writes across several wake boundaries	It emits nothing	An idle namespace costs nothing
21	`TestSubscriberLiveness`	A subscription whose poll interval far exceeds the liveness interval	Its owner is killed	It raises on the liveness interval, not the poll one	The bound holds for every interval
22	`TestSubscriberLiveness`	A failed subscription with no iteration left open	A successor claims the namespace and a fresh subscriber iterates	It discovers the successor's worker	Recovery after failure
23	`TestFanout`	A fanout over a source with `__aiter__` and no `aclose`	Cleanup runs after the iterator is initialised	It runs the underlying stream's `finally`	Cleanup closes the iterator
24	`TestFanout`	A fanout whose source raises while closing	Cleanup runs	It raises that failure and still signals every consumer	Close failures reach the pool
25	`TestFanoutConsumer`	Two consumers sharing a source that raises part way through	Each pulls again	Both raise that failure	Failure reaches every consumer
26	`TestFanoutConsumer`	Two consumers sharing a suspended source, one cancelled mid-pull	The source yields and the survivor pulls	It receives that item and no failure is recorded	Cancellation is one consumer's own
27	`TestFanoutConsumer`	Three consumers of one fanout over a source counting its advances	All three pull concurrently	It advances the source once and hands that item to each	Single-flight pull
28	`TestWorkerProxy`	A namespace with no owner and a quorum that can never be met	A non-lazy proxy is entered	It raises `DiscoveryNamespaceNotFound` well inside the quorum timeout	Quorum wait faults
29	`TestWorkerProxy`	A started proxy whose subscription has since raised	A task is dispatched	It raises that failure rather than `NoWorkersAvailable`	Dispatch faults
30	`TestWorkerProxy`	A proxy whose subscription failed and whose dispatch reported it	The proxy is stopped	It returns cleanly	Reported once
31	`TestWorkerProxy`	A proxy whose subscription failed with nothing having reported it	Its context exits carrying an exception from the body	It lets the body's exception reach the caller	Teardown displaces nothing
32	`TestWorkerProxy`	A proxy whose subscription failed	Many dispatches each report that failure	It hands every caller a traceback of the same depth	Reports do not accumulate
33	`TestSharedSubscription`	Two iterations of one shared subscription over a suspended source	One is cancelled mid-pull and the source yields	It delivers the event to the survivor	A peer's teardown is not the end
34	`TestCrossProcessOwnership`	An owner in an independent interpreter, killed with `SIGKILL`	A successor claims the namespace	It claims, repels a rival, and its exit removes the directory	Cross-process reclaim
35	`TestCrossProcessOwnership`	Four interpreters racing one wall-clock deadline	All claim the same namespace	Exactly one wins	Mutual exclusion
36	`TestCrossProcessBorrowing`	A publisher in an independent interpreter whose owner has exited	It publishes a worker	It reports itself stranded and the successor's registry stays empty	No inherited borrowers
37	`TestNamespaceResidue`	A borrowing publisher killed with `SIGKILL`	Its owner exits	It reclaims the directory and leaves nothing on disk	#345
38	`TestNamespaceResidue`	Three pool lifecycles on default namespaces	Each enters, dispatches and exits	Each creates one namespace directory and removes it	Containment
39	`TestPoolNamespaceOwnership`	A namespace no owner holds	A durable pool borrowing it is entered eagerly	It raises `DiscoveryNamespaceNotFound` without waiting out the quorum timeout	#376
40	`TestDiscoveryFailureIsolation`	A live two-worker pool whose registry is unlinked externally	The pool tears down	Both workers are reaped and each announcement failure is logged	External removal
41	`TestSubscriberMeta`	A subscriber class constructed but never iterated	Every reference is dropped and a collection runs	A weak reference to the class clears	Factory released with its subscriptions
42	`TestSubscriberMeta`	A key callable omitting a constructor argument, and two constructions differing in it	The surviving one is iterated	It builds with that construction's argument	Build serves the construction it is made for
43	`TestSubscriberMeta`	A key callable returning an unhashable value	A subscriber is constructed	It raises `TypeError` at the construction	Hashability reported at its source
44	`TestSharedSubscription`	A subscription whose subscriber the pool has retired	The same subscription is iterated again	It builds a second subscriber and yields from it	Rebuild after retirement

A namespace was reported in use both when a live owner held it and when a killed one had left its registry behind, so the error carried the backing segment's name and a hint to remove it by hand. Ownership is now a kernel-held claim the owner's death releases, which leaves no state to clean up and no second meaning for the error: it is raised only against an owner that is still running. BREAKING CHANGE: DiscoveryNamespaceInUse no longer takes or exposes a segment name, and its message drops the removal hint. Callers reading the segment attribute have nothing to read, and none is needed — a namespace reported in use has a live owner.

The registry and each worker's metadata block lived in POSIX shared memory, which enrolls every segment with CPython's resource tracker. Wool cleans up its own segments, so the tracker only got in the way, and most of this module's machinery existed to evade it: a fork-reinitialized attach lock, paired register and unregister calls against the tracker's ledger, and a version gate around a CPython bug that unlinks a live segment. A failed create also left the tracker holding an unregister for a name it never knew, killing its process with a KeyError. Plain files need none of that. A namespace is now one directory holding the registry, the notify file subscribers watch, and one block file per published worker, so the namespace has a single place to create, inspect and reclaim. Ownership is an exclusive flock on that directory, held for the owner's life and released by the kernel when it dies. The claimant re-checks that the path still names the directory it locked, so a namespace being reclaimed cannot hand two owners a lock on different inodes. Publishers lock the registry file itself rather than a separate lock file, which keeps the lock and the data it guards on one inode: an orphaned publisher can only write to the registry it bound, never to a successor's. The registry is written to a staging file and renamed into place, so no borrower binds a registry before its header is stamped. Owner exit removes the directory, and the last block of an orphaned publisher reclaims one its owner left, so a torn-down namespace leaves nothing behind. The files are read and written with pread and pwrite rather than mapped. CPython asks for F_FULLFSYNC whenever it maps a file descriptor on Apple platforms, which flushes the device, costs milliseconds per mapping and imposes that flush on the whole host; descriptor reads and writes carry the same cross-process visibility without it. Publishing costs roughly 250 to 460 microseconds against 147 for shared memory, all of it in creating and opening files, and wool publishes once per worker lifecycle. BREAKING CHANGE: entering a namespace whose owner was killed now claims it and replaces the stranded registry, where it previously raised DiscoveryNamespaceInUse until the segment was removed by hand.

The READMEs described LocalDiscovery as shared-memory IPC and named a namespace's owner as the instance that created its registry. Neither holds now: a namespace is a directory of ordinary files, and ownership is a claim that ends when the owner exits or its process dies, rather than a fact about who created what. Say file-backed rather than memory-mapped, since the files are deliberately read and written rather than mapped.

Drops the tests and fixtures that existed to police the resource tracker's ledger, since there is no tracker to keep balanced, and with them the last places the unit suite reached into the module for a segment name. Covers what ownership now means instead: a claim rejected while its owner lives, a claim taken over a dead owner's stranded registry, a directory removed on exit and left behind for an orphaned publisher, the two races where a directory is replaced or removed under a claimant, and a staging write that fails before the namespace is claimed. Adds a shared helper for a namespace's directory that mirrors the module's own choice of root rather than importing it, so the tests assert against the path a separate process would find.

Deletes the tracker integration suite, whose subject was CPython's resource tracker, and keeps its two real contracts — a namespace that survives a borrower's exit — by moving them to the teardown suite and asserting them behaviourally, through a publisher that still binds, rather than by reading a tracker's ledger. Rewrites the killed-owner test around reclaim: a successor now claims the namespace the killed owner stranded, where it used to assert the claim was refused. Guards the test against passing vacuously by first asserting the stranded registry is there to be reclaimed. Adds the acceptance test for the residue this work removes: pool lifecycles that publish and exit leave no namespace directories and no lock files behind. The lock now lives on the registry file, which a borrower cannot lock before an owner creates it, so the lock-timeout holder waits for the registry to appear and hands back a second line once it holds the lock. Its tests enter the owner first and wait for that line, rather than assuming a holder can wedge the lock ahead of them.

The worker pool suite patched multiprocessing.shared_memory through a fixture that around sixty tests requested. The module under test never read that name, so the patch changed nothing and the parameter only suggested the tests exercised shared memory. Removing it also empties one test whose sole arrangement was the mock, and leaves another covering default entry, which it now asserts rather than merely reaching.

The example's stub body sat on its own line, which ruff's formatter folds onto the signature now that docstring-code-format extends to Python blocks in markdown. Unrelated to the surrounding work; it was the one file in the tree that reported as unformatted.

REF_WIDTH and NULL_REF describe the registry's on-disk slot layout and have no reference outside this module, no public marker, and no entry in wool's __all__, yet they sat unprefixed directly above five structurally identical constants that are underscored. Rename both and group the module's two genuinely shared names ahead of the private block, so the prefix is what distinguishes them rather than position.

An owner removed its namespace's files by recomputed path rather than relative to the claim it holds. When a third party removed the directory out from under a live owner, a successor could claim the freed namespace and the first owner's ordinary exit then unlinked the successor's registry and notification file, silently and with exit status 0. A later borrower saw DiscoveryNamespaceNotFound. This could not happen while the claim was a shared-memory segment, because removing the directory never produced two live owners. Resolve the three removals against the claim's own descriptor so they cannot land in a directory this claim does not hold, and re-check that the path still names the claimed directory before removing it. Give _unlink_quietly a directory and a name rather than a joined path. With the two joined, a caller could pass one namespace's path and another's descriptor, removing the second while the warning named the first; split apart, that mismatch is unexpressible and the warning still names the file an operator can act on.

Three call sites computed the same "does this path still name this open file" comparison and each treated failure differently: the file handle caught FileNotFoundError only, the claim loop relied on an enclosing handler, and teardown caught OSError. A reader could not tell which policy was intended where. Extract the comparison alone and leave the policy at each site, because the three genuinely differ. A reclaimed handle reports itself stale, so it answers False to anything the filesystem cannot answer -- previously NotADirectoryError escaped as far as publish, undocumented. Teardown must not raise, so it answers False too. The claim loop must discriminate: a directory that vanished or was replaced by a non-directory is another process making progress and is retried, while anything else reaches the caller rather than spinning a loop that has no timeout and no sleep.

The namespace was interpolated into a filesystem path unvalidated, so a caller passing text it did not control could claim a directory outside the namespace scheme, write a registry and notification file into it, and unlink both on exit. In the non-escaping case a namespace containing a separator surfaced as a bare FileNotFoundError out of context entry. Validate it in all three constructors, beside the checks the classes already perform on their other parameters; the domain is one non-empty path component. An empty namespace was silently replaced by a generated one, because the default was applied with "or" rather than by testing for None. A caller who passes a namespace now gets the namespace they passed, or an error. Separately, block sizes of 1 to 4 constructed cleanly and then made every publish raise DiscoveryBlockExhausted forever, since the length prefix alone fills them. Reject sizes that do not exceed the prefix. The bound is necessary rather than sufficient and the docstrings say so: a block a few bytes above the prefix still cannot hold the smallest serialized metadata. BREAKING CHANGE: LocalDiscovery(""), and a block_size of 4 or below, now raise ValueError instead of being accepted.

…values The filter predicate was offered at both altitudes -- a constructor default plus a per-call override on subscribe -- but the rescan interval existed only on subscribe and on the Subscriber constructor. Every pool that spawns workers builds its own LocalDiscovery internally and passes neither, so the fallback cadence was fixed at None with no way to reach it, and the only route to another value forked to a primitive that cannot publish. Accept poll_interval alongside filter and have subscriber and subscribe fall back to it the same way. Note the scope honestly: this reaches a pool that is handed a discovery service, not one that constructs its own. An interval of zero was admitted and drove the rescan wait into a continuous loop, which is the behaviour the lock's own notes argue against, while the error message promised a positive value. Narrow the check to match what it says. BREAKING CHANGE: a poll_interval of 0 now raises ValueError instead of producing a continuous rescan.

A publisher cached one registry handle for its life and every publish locked and unlocked that single open file description. A file lock is held per description with no nesting count, so a second publish on the same publisher took a lock the first already held, and the first release dropped the lock outright while the other coroutine was still inside its critical section. An unrelated process could then enter the section mid-write. Distinct descriptions do serialize, which is why the suite missed it: every lock test established contention either from another interpreter or from two Publisher objects. Open a fresh handle per publish and release it on return, so the descriptor a publish locks is the descriptor it writes. That restores the semantics the separate lock file had before the lock moved onto the registry inode, and it makes the invariant structural rather than asserted. It also removes the only way one publish could close a handle another was holding locked: with no shared handle, neither a rebind nor context exit can pull a descriptor out from under a peer, so the ValueError that masked the original error on that path is unreachable. The cached handle, its rebinding, and the exit that closed it are all deleted. The regression test lands with the re-add recovery path it needs to reach a suspension point inside the held lock.

Publishing worker-added for a worker whose block file had vanished nulled the stale slot and fell through to a fresh registration, but the publisher still held a pool reference to that block. The pool returns a cached object on a key hit, so the registration got back the same handle on the unlinked inode. The write succeeded, because the size comes from the open descriptor rather than the path, and the slot was published naming a file no reader could open. Every subscriber scanning that registry then failed on it. Release the ledger's handle before falling through, so the pool's last reference drops and the factory creates the block afresh. The existing release at the end of the method stays for the displacement case, which is a different path. The docstring already promised a re-add "registers the worker fresh", and the existing coverage missed this because it re-added from a second publisher, which holds no such handle and cannot reach the path. Also lands the concurrency regression test for the per-publish registry handle: this recovery is the only suspension point a fresh publish can reach inside the held lock, so two concurrent publishes on one publisher silently lost a worker before that change.

…s slot A block acquired from the pool was never checked against the path it was created under. The pool returns a cached object on a key hit, so a publisher whose block file had been reclaimed externally -- by an operator, a temporary-file sweeper, or container storage pressure -- got back its own handle on the unlinked inode. The write succeeded and the slot was published naming a path that does not exist. The existing recovery only covers the case where the vanished block still has a matching slot in the same registry. It does not fire when the scan finds no match, which is exactly what happens once the publisher has bound a successor's freshly zeroed registry, and nothing repairs the slot afterwards. Check the handle against its name and, on a mismatch, drop the ledger's reference before re-acquiring. Dropping first is the whole point: at that moment two references exist on the cache entry, so releasing only the one this method took leaves the count above zero and the pool hands back the same stale handle. Written as a retry loop it would hang inside the cross-process lock; written as a single bounded retry without the drop it publishes the slot it was meant to prevent. Separately, worker-updated reached the same scenario through a path that raised a bare FileNotFoundError, undocumented on publish. A slot whose block has been reclaimed names a registration that exists in name only, so report it as an unregistered worker; worker-added is the caller's route back and recreates the block.

The guard around acquiring a block, writing it, and claiming its registry slot caught Exception, so a CancelledError escaped without closing the handle. The exit stack was dropped unentered, the pool entry's reference count stayed at one with nothing able to release it, and the block file and its descriptor survived until the publisher exited. A cancelled pool entry propagates into exactly this call through the gather that starts each worker, and the file handle helper in this same module already uses BaseException for the identical shape. Separately, the release of a displaced handle sat between popping the previous ledger entry and installing the new one, so a cancellation there left the ref with no handle at all while its registry slot was already written. Install first, then release, so no await separates the pop from the install.

Teardown ended with a bare close and never cleared the attribute, and the re-entry guard covers only entry, so a second exit closed a descriptor that was already closed and raised OSError out of a method documented never to raise. Exiting an instance that was never entered raised AttributeError for the same reason. That became sharper once removals resolve against the claim: a repeat release passes a closed -- and possibly reissued -- descriptor to three unlinks and a stat, so once the number is recycled the removals land in an unrelated directory. Worse, the recycled number can be another live owner's claim, and the stray close strips that owner's lock and admits a third. Exchange the claim for a sentinel before anything acts on it and return when it is already spent, so a second release removes nothing and closes nothing. The removals and the directory stay ahead of the close, because the directory has to go while the claim is still held, so the close moves into a finally around the whole body rather than being hoisted. Declare the claim and the shutdown hook alongside the instance's other attributes; they were the only two neither declared nor initialised, which is what let an unentered instance fail on attribute lookup.

…subscription The subscriber's rescan reads the slot array in one pass and then opens each worker's block by name, with no lock and no guard. A publisher's drop completing between the two leaves a slot naming a block that is gone, and the resulting FileNotFoundError propagated out of the async generator and ended the subscription permanently -- for every worker in the namespace, not just the one that failed. No race is even required: a publisher that exits without dropping its worker leaves exactly that state, which the publish contract acknowledges. Skip any slot this scan cannot read rather than allowlisting the two exception types, because the same loop can raise ValueError on a short or garbage slot, struct.error on a truncated prefix, and MemoryError from an unvalidated length prefix. Report the worker from the last scan that read it rather than omitting it. Omitting a worker the cache still holds makes the diff emit worker-dropped for a live worker and worker-added again on the next wake, so a transient bad read becomes an eviction and readmission. Give the carry a floor, or a persistent fault -- a corrupted registry, a foreign file -- becomes a subscription serving stale metadata forever with no signal at all, which is worse for an operator than the dead subscription this fixes. After three consecutive scans the worker is still reported, but each further scan says so. Read the prefix and the payload in one read sized by the block, so a refresh landing between them cannot pair a new size with an old payload, and a torn or foreign prefix cannot drive a 4 GiB allocation. The atomicity is stated honestly: one read against one write narrows tearing to a page-level property of the platform rather than guaranteeing it. The test that pinned the old behaviour asserted a dead subscription was correct. Inverted: it now asserts the readable worker is still reported.

Each notification from the observer thread scheduled a task that took the scan lock before setting an already-set event. The scan holds that lock across every yield, so a burst of publishes piled one task per notification onto the waiter queue, each waiting to do nothing. Those tasks were also created with no retained reference, which makes them eligible for collection mid-flight, and they could still be pending after the generator's cleanup had stopped the observer. The lock bought nothing: the scan clears the notification before it reads the registry, so a set arriving at any point during a scan is preserved and wakes the next one. Set the event directly from the loop thread and drop the lock, the tasks, and the two helpers that existed only to bridge them.

Both branches that build their own discovery constructed it with every limit defaulted, and a pool given neither a spawn count nor a discovery service resolves its worker count from the CPU count. On a host with more than 128 cores the registration of the 129th worker was refused for want of a slot, the failure was collected into the group that aborts context entry, and the first example in the class docstring -- a pool constructed with no arguments at all -- did not start. The pool exposes no capacity of its own, so the only escape was to hand it a hand-built discovery service. Size the registry from the count the pool has already resolved. The pool owns this namespace outright, so nothing else registers in it. Name the backend's default rather than repeating the literal at a second site. Covered by a constructor spy rather than a behavioural test: exercising the boundary honestly would mean spawning more real worker processes than the default capacity. The test abandons entry as soon as the capacity has been observed, so no worker is spawned.

Each example of these six enters a real pool, whose discovery now publishes its workers under a cross-process lock that genuinely serializes them rather than letting every publish reuse one lock. That made an example's wall-clock cost scale with the generated worker count, which pushed the pool suite past Hypothesis's default per-example deadline and produced failures that named a different test on each run. A timing bound on a test doing real cross-process work measures the machine, not the property. Drop the deadline on the six that enter a pool and record why.

The block-exhaustion error told the caller to re-register the worker under a larger block size. Re-registering opens the existing block by name and never consults the configured size, so the advice the public exception hands an operator re-raises the same error forever. Drop the false clause, keep the one that works, and point at the statement that a block's size is fixed at its first registration. The capacity error recorded a rationale the storage change retired: capacity was fixed for the registry's lifetime because the registry was a mapped segment whose size was fixed when mapped. Files are not, so keep the part a caller can act on -- the ceiling does not grow, and dropping a worker frees a slot -- and drop the mechanism claim. The in-use error stated that a namespace is only ever in use while its owner lives. A claim is held per open file description, so a process forked after entry inherits it and the namespace stays in use until every holder exits. State the condition the mechanism actually enforces.

The ownership paragraph said a killed owner's claim ends with its process, while the fork paragraph seven lines below said the claim lives until every process holding it has exited. A lock is held per open file description, so the second is right and the first is false whenever anything inherited the claim. State the condition the mechanism enforces, hoist it out of the implementation rubric where it was already correct, and say plainly what a process forked after entry inherits. Record the reclamation limitation rather than implying it away: the shutdown fallback is interpreter state, so a signal death leaves the directory and one block per registered worker behind, and a successor replaces the registry without sweeping them. That is bounded by the filesystem, not by this module, and a container stopped the ordinary way takes that path. Record why a fixed-width registry is a choice now rather than a constraint, since files can grow where a mapped segment could not, and why a publish opens its own handle. Smaller corrections: the reference parser documents a NULL that it never raises for; the unstamped-header error named a service that exists nowhere in the package; the lock's notes claimed the holder is by definition another process, which is false and pointed a reader away from the exclusion defect; two notes still described a registry nothing maps; the root helper promised a directory check it does not perform; one of its two coverage pragmas ran on every call and suppressed real coverage; the usage examples were not runnable; and two public properties had no return annotation. On the pool: say that a discovery instance is entered for the pool's lifetime and spent by that entry, so the documented retry needs a fresh instance and the factory form is what supports it, and document the discovery errors entry can raise along with the group that carries them.

Name the registry and notification files once rather than spelling the literals at each call site, so a test that needs the file an owner publishes or the one a publisher locks says which it means without spreading the name through the suite. Add a fixture that mints discovery namespaces and removes their directories afterwards. A namespace whose owner is killed keeps its directory by contract, since nothing survives a SIGKILL to reclaim it, so every test that kills an owner leaves one behind. The fixture removes directories only: the residue assertions stay in the tests that make them, because a fixture asserting them would mask the very leaks they pin.

Several tests asserted things that were true regardless of the code under test. The interleaving test wrapped removals only for paths whose parent was the namespace directory, but it runs against per-example namespaces derived from that one, which are siblings. The schedule was never consumed and the generated mask injected nothing, so the test passed whatever teardown did. Scope the wrapper to every directory carrying the namespace, assert the schedule was actually consumed so it can never go inert unnoticed again, and widen the mask from "vanished" to "vanished or refused" now that it has something to inject into. The no-owner subscriber test derived the namespace directory inline rather than through the helper, so on a platform whose root is not the temporary directory it inspected a path that never existed. Rename what the storage change made inaccurate: address-space exhaustion is capacity exhaustion, and two names had lost the underscore between "the" and "namespace". Move two publisher tests into the publisher's own class. Delete the worker-reference hashing test and its class. It reached into the module for a private type -- the last such reach in the unit suite -- to pin equality and hashing that nothing in the module calls, since both ledgers key on the rendered string.

Every existing lock test established contention by patching the locking call, so the lock could have moved onto a different file, onto the directory, or onto nothing at all and the suite would have stayed green. Take a real lock on the registry inode from a second descriptor and assert a borrower times out naming the namespace, so the file the lock lives on is pinned rather than assumed. Add its companion: a publish that raises under the lock still releases it, which a second publisher proves by acquiring on its first attempt. Pin that the registry is never observable unstamped, by probing at the moment the staged file is renamed into place: a borrower binding there finds no registry rather than an unstamped one. An implementation that truncated and stamped in place would pass every other test in the file. Cover the fork contract the class documents and nothing exercised: a fork that closes its inherited stack and leaves reclaims the namespace, and the parent's own exit then neither raises nor warns. Cover the subscriber configuration the discovery service actually produces. Its default rescans only when a publisher writes, and no iteration in the suite used it, so both the initial snapshot and a late publish are asserted under it. Cover the watch-start guard, which is new code, by making the observer fail to start and asserting the namespace is reported missing with the original failure as the cause. Cover the successor case for removals resolved against the claim: an owner whose directory was removed out from under it leaves a successor's namespace alone.

The re-add test ended its subprocess by killing it, so neither the context exit nor the shutdown fallback ever ran. With nothing left that could have removed anything, the assertion that the worker survived could not fail for the reason it stated. End the child cleanly over its input, require a clean exit status and a line printed after the loop has returned, and keep the kill only as a leak guard. The helper that waits for a lock holder to report accepted a timeout and then ignored it, reading on a blocking call instead. A holder that never reported wedged the whole run rather than failing, and it gated two of the three tests in that file. Honour the timeout with the reader-thread pattern the shared fixtures already use, so a silent holder fails where it happens. Assert the fields the exhaustion errors carry. They are the only way a caller learns why a registration was refused, and nothing checked that publishing populates them.

Repair two tests that carried no information. The successor-admission test asserted only that a claim was printed, in both of its arms, so the parametrisation distinguished nothing and the killed arm never checked that the kill stranded anything; give each arm the precondition that makes it specific. The racing-claimants test let its winner leave the namespace the instant it printed, so a late claimant could claim an unheld namespace and report itself a winner too -- a live flake that reported two winners; hold the claim past the deadline every loser races. The borrowing test reused the publisher script, which binds a publisher and publishes, so the read-side borrow it claimed to isolate never ran alone. Give it a subscriber-only script and assert through the owner's own subscription rather than a weaker probe. Assert the residue property that dominates the historical leak: a borrow that fails creates no inode, which the claim that follows it could not distinguish before, since entering makes the directory either way. Then cover what no test reached. An orphaned publisher in another interpreter reaches a successor's registry and is bound by the successor's capacity, not the capacity it bound under. Notification crosses a process boundary, asserted with no poll interval so that polling cannot satisfy the oracle. A pool cycling through namespaces leaves none of them behind, with a guard so a prefix that stops matching cannot pass silently. Pin the honest boundary of the reclamation work: a killed borrower leaves its directory standing, because the reclaim runs in the borrower's own shutdown fallback and nothing survives a SIGKILL to run it. Named as a deliberate pin, so both a fix and a regression have to be a deliberate edit.

The class read __init__, __aiter__, _stream, __reduce__, so a private method sat between two dunders and stranded __reduce__ below it. The style guide's class ordering puts private methods last and names this exact shape in its own counter-example. __aiter__ resolves the method by name at call time, so the move is behaviour-neutral.

Nine claims that were false, duplicated, or incomplete. Each is a caller-facing contract, so a reader acts on it. DiscoveryBlockExhausted directed callers to a Publisher block_size this branch removed, and publish stated the inverse of the new behaviour by attributing a block's size to the publisher that created it. Both now attribute it to the size the namespace's owner stamped, and name the one knob that exists. The exceptions module stated one of three rebuild outcomes. A keyword field riding args raises for too many positional arguments; kept out of args it either raises for a missing argument or, where it has a default, succeeds while silently substituting it. Only the middle case loses data without reporting anything, and that was the case the text omitted, so it was misleading rather than merely short. WorkerPool documented three exception types a bare except can never match, because every one of them arrives wrapped in an ExceptionGroup from the gather that starts the workers. Each entry now says so and names except*, and the group entry says it happens always rather than only when several workers fail. _Binding's header parameter was undocumented, and it is the one that carries the ownership invariant - the sizes are the owner's to fix and a borrower's only to read, which is what makes them unforgeable. The claim loop's note asserted that a claim resolves rather than livelocking as though unconditional. It holds among cooperating processes, where completed progress per retry means a legitimate claim takes one iteration. It does not defend against a co-tenant on a world-writable root, and the mitigation there is an unshared root. param block_size restated the validator's own necessary-and-not- sufficient paragraph almost sentence for sentence. The claim is caller- facing and the validator is private and never rendered, so the real gate now lives on publish's raises entry, with the parameter keeping only its domain and a pointer. FanoutConsumer's queue growth and the consequence it carries for a discovery feed are recorded where the queue is, and on the subscription that binds a generic fanout to discovery events: a dropped worker-dropped cannot be recovered by a later scan, because the diff forgets a worker as soon as it reports the drop. The fork-generation grouping gains a comment recording that it is deliberate, so a later reader does not relocate it. One correction in _cancel_sentinel is incomplete and is carried as an open finding rather than fixed here: it bounds the retention to one proxy run, which is right for a proxy that is stopped and wrong for one that stays started, which its own dispatch docstring documents as the normal outcome of a lost namespace.

The explicit-stack sweep opened every child directory as the listing named it and pushed the already-open descriptor onto the stack, so no child was closed until its whole subtree had drained. Peak descriptor use was therefore proportional to the breadth of the frontier rather than the depth of the tree: 400 sibling directories held 400 at once, where the recursion this replaced held one per level. That is the same threat the stack was introduced for. A co-tenant on a world-writable root who can plant a tree thousands deep can plant one thousands wide for no more effort, and exhausting the process's descriptors takes every unrelated open down with it rather than just this sweep. CPython's shutil._rmtree_safe_fd defers the open to the pop for this reason; the stack shape was taken from it and the laziness was not. A child is now opened when its own tag pops. Worse, the open's failure was absorbed as a race. The comment named two causes a listing cannot avoid, but the clause caught every OSError, including the permission the sticky bit denies on another user's directory and the descriptor budget the eager opening had just exhausted. Each of those abandoned a whole subtree with no report at all, against a contract promising that anything beyond an entry already gone surfaces as a ResourceWarning. Measured under a descriptor limit, 185 of 300 planted directories were left behind and two were named. The clause is now narrowed to the races it claimed, and a subtree that cannot be entered is reported, since losing one silently costs more than failing to remove a single entry. The loop also had no finally over the descriptors it held, where the recursion closed each child in one. A ResourceWarning a filter has promoted to an error, or an interrupt between pops, stranded the whole frontier; three descriptors leaked permanently in the reproduction. One stranded entry is now named once. Every directory above it is left non-empty in turn, so each reported the same leak again; the removal of a directory something was already reported beneath now tolerates a non-empty directory. The quiet removal reports whether the entry is gone so the sweep can tell the two cases apart, which is what the tolerance is keyed on, and it treats an entry already absent as removed rather than merely tolerated. Its stacklevel parameter is deleted. No call site passed it and no test exercised it, and the one value it ever took is now written where the warning is raised.

Two claims in the LocalDiscovery contract were left behind by the removal primitives being consolidated. The first named _unlink_quietly as the door every teardown removal transits. That was false by omission when the sweep had its own inline removals, and false by misdirection once they were merged: the shared primitive is _remove_quietly, and _unlink_quietly is one of two thin wrappers over it. A reader auditing the never-raises guarantee followed the named symbol and found it did not cover most of what a teardown removes. The second claimed every removal resolves against a descriptor rather than a path. Four do not - the failed-creation unlink, the namespace rmdir, and the two block removals - and the module contradicted itself twelve lines later with a comment explaining why one of them resolves a name. The claim now says which removals are descriptor-relative and why the rest are safe without being, which is the argument the sweep's own threat model rests on rather than a blanket assertion a reader can disprove by grepping.

The fork-generation counter and the owner-pid check sat side by side in one disjunction, and the counter could never decide it. Both fields were stamped together, and the counter's only mutation ran in an at-fork child handler, so the counter differed from the global exactly when the process had forked since the stamp -- which is also exactly when the pid differed. The disjunction was equal to either operand alone. The pid check is the more general of the two: a fork issued from C runs no Python at-fork handler, so it changes the pid without bumping the counter. Deleting the counter therefore loses no case, while keeping it cost a module global, a mutator, an at-fork registration, a field, two stampings, and a comment defending a member ordering the style guide has no position for. A forked child's teardown still removes nothing and leaves the live parent's namespace bindable, verified by neutralizing each oracle in turn: either alone keeps the fork-disown test green, and deleting both turns it red.

A subscription guarded observer.start() with a bare except OSError and rewrote every failure into DiscoveryNamespaceNotFound: the ENOSPC an exhausted inotify watch limit raises, EMFILE, ENOMEM. The justifying comment described only the case the guard was written for -- an owner that exited between the bind and the watch -- and that case presents as a missing directory. Reporting the rest as a missing namespace costs more than the errno. The exception's documented remedy is to re-bind, and re-binding a namespace whose owner is alive fails the same way every time, so the caller is directed to retry the one thing that cannot succeed. The accompanying message compounds it by asserting the namespace has no live owner while the test holds one open. The narrowed pair is the form two other sites in this module already use, so the observer guard was the outlier rather than the precedent.

DiscoveryNamespaceNotFound asserted in four places -- its summary, its body, its namespace parameter and its rendered message -- that the namespace has no live owner to borrow from. The condition it reports is per-binding, and the class body two lines down already said so: a binding ends with the owner it was made against, so a successor that claims the namespace afterwards does not serve it. The handoff case is exercised by a passing test in this suite, which raises the error inside a live successor's context manager. The operator-facing string therefore denied the existence of an owner that the same test had just entered and was holding open. Of the six other sites repeating the claim, five are reached only from a bind, where resolving the live pointer and finding no owner does make the namespace-level statement true. The sixth is publish, which runs against an already-pinned binding, and is corrected with it. The test pinning the message already reasoned correctly about this in a comment -- that a successor may hold the namespace -- while asserting a message that said otherwise.

A dispatch the balancer declines with NoWorkersAvailable, while the proxy holds no workers of its own, substitutes the subscription's failure for that decline. The substitution was made from inside the except handler, and CPython stamps __context__ at raise time, so the interpreter overwrote the chaining FailureReport had just restored with the very NoWorkersAvailable the proxy had judged the wrong answer. The caller therefore received a traceback headed "During handling of the above exception" naming the exception that was deliberately not reported, which inverts the causality the substitution exists to express. Accumulation stays bounded either way, so the existing test measuring traceback depth passed throughout. Reporting once the handler has completed leaves the interpreter nothing to stamp. The retry helper now returns the failure rather than its depth, so the new assertion needs no second copy of its loop.

FailureReport's docstring said it holds "that chaining state" and restores it before each raise. Two things were missing from that. __cause__ is not among what it holds. The slots capture the traceback, the context and the context-suppression flag only, so an explicit cause set by a later raise-from is permanent across reports. The restore also cannot survive a raise the interpreter restamps: CPython sets __context__ at raise time, so reporting from inside an active except replaces the held context with whatever is being handled there. That is a precondition on the caller rather than something this class can defend, and it is now stated where a caller will read it. What the class bounds is growth, not retention. The depth a caller receives is the same at the hundredth report as at the first, which is the accumulation it exists to prevent, while the origin traceback is held for the report's life along with whatever its frames reach. Both remediations recorded against that retention were measured, and neither bounds it to the origin: resetting the traceback after the raise drops the reporting helper's frame, and holding the traceback below the catch drops the sentinel task's frame. One frame each, out of a chain whose length is set by where the failure came from. Documented here rather than implemented.

The class had no tests of its own, while two subsystems depend on it to hand one exception to many callers without accumulating. Five cover its public surface: the held instance reaches a caller unwrapped and by identity, the context captured at construction is restored on every report rather than lost to the first propagation, the origin's frames are reported alongside the caller's, and repeated reports hand out one depth rather than a growing one.

Three claims in WorkerProxy outlived the narrowing that made the sentinel report conditional, and they have to move together because each cites the next. The class docstring promised the held failure reaches "the next caller to need the membership", naming dispatch without qualification, while dispatch's own raises entry already said the report waits for the balancer to decline. The class-level copy was the stale one. Dispatch then said every later dispatch reports the same failure. A dispatch the already-reported workers can still serve is served, which is the whole point of consulting the sentinel after the balancer rather than before it, so the promise holds only for dispatches the membership cannot serve. That same sentence is what the sentinel's retirement cited to bound what a dead task retains to one start rather than the proxy's life. It reaches that bound through _reset_state, which only a stop runs, and the path dispatch documents is one where no stop happens -- so on that path the bound is the proxy's life. The unwind docstring already said as much, and now agrees with the two above it.

The raises entry said the TypeError fires when discovery does not implement DiscoverySubscriberLike. The constructor rejects three things: an async generator, a callable, and an object with no __aiter__ -- and only the third is a protocol failure. The async generator is the one a caller cannot predict from the entry as written. __aiter__ is the protocol's only member and a generator has one, so such an object passes isinstance and passes a type checker, and is rejected regardless because it is its own iterator and is unpicklable. The code tests for it ahead of the protocol check for exactly that reason, and said so in a comment, while the entry a caller reads disclosed none of it. While the guard was unreachable the entry was merely incomplete. Hoisting the guard made it wrong about the error a caller will actually get.

The release ordering was justified by publish consulting the liveness lock "as its only oracle". Publish consults that lock and current() together, and its own comment two hundred lines away says so in those words, so one file gave two opposite accounts of the same check. The ordering argument survives restating. Releasing the lock last would leave an instant where it reports the owner live while the files are already gone, which leaves current() alone between a cross-process publisher and a write into an unlinked inode that reports success. That is the single-oracle dependence the pair was introduced to remove. Stated the other way round, the comment was an argument for deleting one of the two guards: a reader who believes publish consults only the lock has no reason to keep current(), and a reader who sees current() catching the case the comment describes has no reason to keep the lock. Each covers a window the other cannot, which publish documents, so this points there rather than restating it.

WorkerPool carried two raises entries for TimeoutError under two spellings, and asyncio.TimeoutError has been an alias of the builtin since 3.11. One said the quorum timeout propagates bare; the other said the lock timeout arrives wrapped in an ExceptionGroup and needs except*. Both are true of the same class, so a caller reading the first and writing except TimeoutError silently misses the lock timeout. Merged into one entry naming both origins and both arrival shapes, which is what a caller needs to write the two clauses that catch it. That also falsifies the claim below them that DiscoveryNamespaceNotFound is the one type needing both an except and an except*: TimeoutError occupies exactly that position. The entry now names both, and the group's own count no longer includes a type the next paragraph carves out of it. The asyncio.TimeoutError spelling appears at fourteen further sites across other modules, most of them except clauses that predate this work. Standardising those is a sweep of its own and is left alone; the two entries merged here are the ones whose disagreement was the defect.

DiscoveryBlockExhausted told a caller to raise the block ceiling by entering the namespace as an owner with a larger block_size. Entering a namespace a live owner holds raises DiscoveryNamespaceInUse, so the one remedy offered for a permanent per-worker failure was unavailable exactly when a caller would reach for it. Its closing clause also inverted the borrowing contract. It described the new generation as one that every borrower re-binds to, where the contract is that a borrower fails and its caller decides whether to re-bind -- nothing is re-pointed underneath anyone. What is true is narrower and more useful: the ceiling is settled where the namespace is entered, by its owner's block_size, and cannot be raised for a namespace already in use. Shrinking the metadata remains the actionable remedy. This now points at the ownership contract rather than paraphrasing it, which is how the inverted clause got written.

The check that a borrower's hold is still good -- the registry's path still naming the file it holds, and the owner that was live at bind still holding the generation -- appeared byte for byte at three call sites, with only the comments above them differing. Each site could therefore lose either half on the authority of a comment at another, which is the disagreement binding through one primitive exists to prevent. _Binding now owns that check, and with it the namespace it was bound against, which bind previously resolved and discarded. A borrower's failure therefore names one namespace rather than each holder keeping its own spelling of it. The remaining single-oracle check in bind is not a fourth copy. Its registry handle is deliberately closed before the probe runs, so there is nothing there to ask whether a path still resolves.

Deleting the liveness check from publish left the whole suite green: 205 unit tests and 29 integration tests passed while a publish wrote into a registry no live owner served. The check was the only thing standing between a stranded publisher and that write, and nothing measured it. The window is reachable because a killed owner runs no teardown. Its registry survives and its path still names it, so the handle opens and only the lock the kernel dropped reports the loss -- which is the one state in which neither half of the check can be inferred from the other.

Staging opened, sized, wrote and closed a raw descriptor, which left _File.create -- the module's own create-and-open primitive, with the same open and the same zero fill -- holding a single caller. Two deltas come with routing through it, both narrower than they look. The open is now exclusive rather than truncating, which the fresh generation name two lines above turns into an assertion: a staging file that already exists means that name collided, not that an earlier attempt left one behind. And a failure inside the create now removes the file it made, where before it was left for the next attempt's sweep. Staging still promises only that the generation directory may survive a failure, so the contract holds and the residue shrinks.

The logger is module-private and carried no leading underscore. Six modules in the package already spell theirs _log, and nothing outside this file reads it by either name.

The test helpers respelled the discovery layout as literals -- the root entry, the pointer, the registry, the notify file and the blocks directory -- so the two sides could come to name different paths with nothing to notice. They can now only disagree if the module disagrees with itself. Two ways they diverged, both measured. Renaming the root entry left the helpers pointing at a path the product never creates, and the residue assertions are negative, so four of them went on passing while a namespace leaked its whole directory: a genuine leak reported as clean. Separately, the module resolves its root once and caches it while the copy recomputed, so a temporary directory set after that first resolution moved only one of them. The call sites are untouched; only the seven helper bodies change, plus the two places outside them that spelled a layout name directly. This reads module-private names from a test module, which the test guide otherwise forbids. The coupling is not new -- it was there and silent, and sixteen sites in this suite already patch private names on this module. Naming it is what makes the two sides move together.

The unit and integration suites each carried a probe answering whether a fresh borrower can still bind a namespace. The bodies were the same six lines; only the shape differed, one folding a loop in and one leaving it to the caller -- and four of the five callers of the async one re-added the loop themselves. One async helper now serves both, in the module both suites already import from, and the two callers that want it synchronously say so. It needs only public names, so it adds no reach into the module under test.

It was defined between the third and fourth test classes, around four thousand seven hundred lines below both of its callers, where every other helper this module adds sits in one block ahead of the first test class. It now sits beside the strategy whose forests it walks. Nothing required the old placement. The comment on that strategy insists only that the strategy itself precede the test classes, since a decorator argument is read at class-definition time -- which is why a function called from inside a test body was free to sit anywhere.

The floor rejected only sizes that could not hold the 4-byte length prefix, so anything from 5 bytes upward constructed, claimed the namespace and stamped the header, and then raised DiscoveryBlockExhausted on every publish, permanently and per worker. The validator's own docstring conceded the bound was necessary and not sufficient. Derive the floor from the schema instead: the length prefix plus the smallest WorkerMetadata the message serializes to, measured once at import so a field added to the message moves the floor with it rather than leaving a validator that admits a block nothing can be published into. What stays reachable is a block above the floor but short of what a particular worker needs, which depends on that worker's metadata down to the length of its version string. The floor is the smallest block that can work rather than the smallest that will, and that is now documented in one place rather than restated in four. BREAKING CHANGE: LocalDiscovery rejects a block_size below the length prefix plus the smallest serialized worker metadata, 68 bytes on this schema. A block_size between 5 and 67 previously constructed and then failed every publish.

The loop had no cap and no sleep. All three of its retry paths spin immediately, and the one a hostile co-tenant drives is the one with no syntactic marker at all: the falsy _same_file check that detects a directory replaced under the claim, which has neither a continue nor an except clause. Entry is synchronous and is reached from async code, so that spin holds the caller's thread and issues a sweep's worth of syscalls per turn. Bound it with a for/else over _CLAIM_ATTEMPT_LIMIT, which needs no counter and reaches the raise holding nothing: every arm either breaks on success or exchanges its claim for the sentinel and closes it before looping, so exhaustion needs no teardown of its own. Exhaustion raises DiscoveryNamespaceContended rather than reusing DiscoveryNamespaceInUse. Nothing holds the claim in this case, and an exception asserting that something did would be exactly the kind of false claim its sibling was corrected for. The prose describing the absent cap goes with it, including the claim that the mitigation was a root no co-tenant shares rather than anything this loop could do about it. A cap is precisely what the loop can do. What survives is the distinction: a cap converts the spin into a failure, while an unshared root is what makes a legitimate claim resolve. BREAKING CHANGE: LocalDiscovery.__enter__ raises DiscoveryNamespaceContended after eight failed claim attempts instead of retrying without end, and WorkerPool propagates it from context entry. A legitimate claim takes one attempt, so this is reachable only where the namespace directory is replaced repeatedly during entry.

The coalescing the notification event performs was documented without saying what it costs when the backend underneath does not help. A backend that batches events before delivering them collapses a burst before the subscriber rescans; one that delivers each event as it happens leaves the scan count following the publish count, and since every scan reads every registered worker's block, the cost then grows with the burst times the workers already registered. Assert that bound rather than only describing it. Sixteen gathered publishes must report every worker within a small multiple of their number, not a rescan's worth of worker-updated per publish. The observable is the events the subscriber yields, which is public and needs no reach into the scan loop. The assertion is gated to the backends where it can fail. On a batching backend the burst collapses into a single scan, which emits sixteen worker-added events and nothing else, so the test passes there without being able to discriminate. Measured at exactly sixteen across five runs, against the hundred and thirty-six an un-coalesced burst would emit.

The pool entry declares its fanout as Fanout of DiscoveryEvent, but the key's factory was typed as returning Any, so the one site that constructs the fanout inferred Fanout of Any and assigned it into that field unchecked. A subscriber yielding anything at all satisfied the construction, and the mismatch surfaced only as an AttributeError from inside the first iteration, on the line that reads a uid off the event's metadata. Type the factory as building a DiscoverySubscriberLike, which is where the erasure starts rather than one frame below it. The identity stays Any because it is the caller's own opaque value; what the factory builds is iterated by this module, so its element type is this module's business to state.

Narrowing the close to async generators stopped closing a class-based iterator with a correct asynchronous aclose. The subscriber contract offers the generator as an example rather than a requirement, so such an iterator is admitted, iterated, and then never closed, while cleanup still documents that it raises whatever closing the iterator raised. Neither existing cleanup test covers the shape, because both use generators, so the suite was green against it. The comment justifying that narrowing had it backwards. It claimed an attribute probe was what would await whatever a third party happened to name aclose, but isinstance against AsyncGenerator is itself a structural probe, over five names rather than one, so the narrowing admits exactly that object as long as it also names asend and athrow. Close on either predicate, since neither covers the other: a native generator's aclose is a builtin that iscoroutinefunction does not report, and a class-based iterator implementing aclose without asend is not an AsyncGenerator. The pair still excludes the case that motivated the narrowing, a synchronous aclose, which is now pinned by a test so a later simplification to a bare attribute probe goes red rather than reintroducing the failure it caused.

The accepted residual said what is not done on the two paths that evict without running finalizers, and not what that costs. Every comparable residual in this subsystem names the consequence: the sibling paragraph spells out that a discarded worker-dropped keeps a consumer routing to a worker that is gone, and the discovery module spells out the residue a killed owner leaves. Name it here too. The source iterator is a subscriber iteration whose exit stack holds its binding, its registry handle and its observer's retirement, so a skipped close strands a watcher thread and the descriptors it pinned, and the uncancelled pull surfaces as a pending task destroyed without naming the fanout or the subscriber it belonged to. Record the asymmetry with it: the other resource owner in this subsystem has an atexit fallback and a fanout has no second route at all.

conradbzura added 19 commits September 17, 2026 15:03

conradbzura self-assigned this Sep 18, 2026

conradbzura added this to the v0.16 milestone Sep 18, 2026

conradbzura added 9 commits September 19, 2026 20:33

conradbzura added 15 commits October 15, 2026 14:06

conradbzura changed the title ~~Back the discovery registry with a file instead of multiprocessing.shared_memory — Closes #353~~ Back the discovery registry with a file instead of shared memory — Closes #353 Oct 22, 2026

conradbzura added 13 commits October 23, 2026 11:18

refactor: Privatize this module's logger name

faa41fc

The logger is module-private and carried no leading underscore. Six modules in the package already spell theirs _log, and nothing outside this file reads it by either name.

conradbzura mentioned this pull request Oct 26, 2026

Report a dead discovery subscription's cause from the first dispatch #444

Open

Navigation Menu

Repository navigation

Back the discovery registry with a file instead of shared memory — Closes #353 - #441

Back the discovery registry with a file instead of shared memory — Closes #353#441
conradbzura wants to merge 87 commits into
wool-labs:mainfrom
conradbzura:353-mmap-local-discovery

conradbzura commented Sep 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

conradbzura commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking changes

Proposed changes

Back the registry with a file

Scope every artifact to an owner's generation

Detect an owner that did not exit

Disown an inherited claim in a fork

Deliver a failure to every consumer of a shared subscription

Surface a dead subscription to the caller

Release a pooled subscriber's factory with its subscriptions

Test cases

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

conradbzura commented Sep 18, 2026 •

edited

Loading