Skip to content

fix: Gate model stages on current process memory - #819

Draft
buildingjoshbetter wants to merge 1 commit into
fix/model-residency-publish-297from
fix/model-process-admission-publish-297-v2
Draft

buildingjoshbetter wants to merge 1 commit into
fix/model-residency-publish-297from
fix/model-process-admission-publish-297-v2

Conversation

@buildingjoshbetter

@buildingjoshbetter buildingjoshbetter commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Change

The shared daemon can begin another model stage while current process memory is already above its configured budget. This adds opt-in current-RSS admission before model construction, after construction, before every embedding/reranking slice, and before CPU transfer or retry. Stacked on #816; partial progress toward #297, which remains open.

TRUEMEMORY_MODEL_SERVER_MAX_RSS_MB defaults to zero, disabled pending calibration. For enabled budget B = configured MiB * 1,048,576, current RSS U < B admits work; U >= B or an unavailable/invalid measurement refuses it with server_busy. A synthetic 1 MiB budget admits 1,048,575 bytes and refuses 1,048,576 and 1,048,577 bytes. This uses current psutil RSS, separately from historical peak reporting.

Refusal preserves inference ownership, complete-output semantics and sticky CPU recovery. Failed accelerator caches cannot evade CPU selection through a cache hit. Fast-lane refusal cannot fall through to main inference. RSS measurement consumes the same request deadline, so a slow sensor cannot start construction or recovery after expiry. Obsolete model references remain released before distinct replacements.

Models, rerankers, dtype, vector dimensions, result ordering and existing request, batch and retrieval limits are unchanged. Existing synthetic fixture loaders/doubles accept the new private helper/signatures without changing any of their 793 original assertions. The new regression fixture retains all 100 v1 assertions and adds 35. The publication branch also carries the already reviewed config fixture correction: actual delegated config directory/file/lock paths are isolated, all 20 prior assertions remain and eight checks are added. No production config or throttler change.

Validation

  • Independent review found no remaining blockers. Seven saved deadline/generation probes, 25 expiry/refusal-priority cases and 10 CPU-transfer expiry/reload cases pass.
  • Current-RSS regression suite: 24 tests / 93 subtests pass. Original-source and removed-guard controls each fail 36 assertions with zero errors. The rejected v1 deadline control fails 32 assertions with zero errors.
  • Exact combined author head abdeff27839ee0c2b58d70b57dba291a48f65b78: 3,063 tests and 2,612 subtests passed, 17 skipped, 3 deselected. Unprivileged isolated run with actual sqlite-vec; exit 0, no timeout/OOM. Log SHA256: a473773393247e37a59bd3d6e0e801c23187b35857035245baa3642de09cce4e.
  • Exact publication source: 2,253 tests and 656 subtests passed, 17 skipped, 3 deselected, under the same isolation and native-vector conditions. Exit 0, no timeout/OOM. Log SHA256: 594a02a08ac71a07a6ebc665e7df5a5dfe9ac910ffa15d65ca9074ac60cc5191.
  • Installed-wheel smoke uses actual current RSS: 41,443,328 bytes > 1,048,576-byte synthetic budget. Three requests refuse before model construction; model caches stay empty and owners release. Installed production source hash matches the reviewed candidate. Log SHA256: f923e15dcb4e6c5cf26ed91e26905c771d111c1c412050f0b58d6cfd9531c272.
  • Nine code/test files match the reviewed author. The additional platform fixture matches integration. Documentation preserves the publication base plus exact author suffix. Ruff, Python 3.10 grammar, whitespace, scoped privacy and public commit-identity checks pass.

Rejected v1 source and failed receipts are retained. Independent review caught a constructor-after-deadline defect despite initial focused tests. Full v1 also exposed an unrelated-throttler dependency in the new synthetic fixture, corrected by separating independent sensor cases. The first publication gate retained two inherited config-fixture path failures; the known fixture correction passes without weakening assertions. All 34 hosted checks pass at exact draft head 4bc096095d834095db962a20ee1547b1426a7cd0.

Remaining acceptance

This is sampled, opt-in admission, not a hard memory bound or reservation. Concurrent lanes or one native call can overshoot; current RSS does not establish unified-memory footprint or native allocator reclamation. A calibrated finite default, predicted workspace reservations, system headroom, bounded recovery and real Mac/MPS/thermal validation remain required.

All data is synthetic. This draft does not prove the original 26.24 GB Mac symptom fixed, close #297, merge main or release anything.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant