fix: restore default semantic fallback and reduce macOS watcher work

[justrach]

Current integration status (September 26, 2026)

The watcher repair has been isolated into draft PR #758; its local checks and exact-head CI pass, but the PR-ready publication preflight still rejects the large touched watcher source file (justrach/codegraff#1304). The installer hardening is isolated in draft PR #759 with normal-install registration and installed-hook coverage; its branch checks are tracked there. Both changes remain in this combined PR's diff until the release base advances, so this PR is not a shortcut around either review gate.

A fresh semantic-index probe against a disposable HOME and synthetic 53-byte source file, with telemetry disabled, received HTTP 530 on both compact and legacy signed embedding attempts. The old client reported that non-standard 5xx as EmbeddingProviderRejected; draft PR #761 corrects the classification to EmbeddingProviderUnavailable, but the hosted request still fails. Live hosted fallback, the flagged ranking/latency trade-off, signing/notarization, and release publication remain unverified. Keep this combined PR draft.

What changed

• A fresh default-hybrid project now maps missing OpenPuffer metadata/graphs to the existing hosted exact fallback instead of surfacing FileNotFound. The PR also keeps bounded transport retry and generation-publication safety checks.
• Installer downloads require a unique verified release checksum before replacing a binary. The branch incorporates the newer MCP client registrations from release/0.2.5856 rather than reverting them.
• On local macOS volumes, a healthy recursive FSEvents stream no longer audits the full tree every two seconds. It verifies a rotating set of up to 64 known files per tick and audits directories every 60 seconds. Excluded file events are filtered before they fill the pending cap, while directory/symlink/unknown-type and ignore-policy events remain visible. Dropped history still forces recovery; unsupported filesystems and aliases retain the conservative fallback.
• Added watcher dispatch-path regressions, security/fault fixtures, and observed retrieval/CPU evidence. Earlier commits in this PR also change ranking: whole-query BM25 joins the lexical union, implementation requests discount documentation-heavy ANN chunks, and unique compound definitions or explicit test filenames can receive priority. Hosted Jina, calibration, hosted exact fallback, and the 4× OpenPuffer ANN candidate pool are unchanged.

Why

Missing-sidecar errors prevented the intended no-setup semantic fallback. The initial watcher follow-up then erased its own CPU benefit by walking every directory on a two-second timer, and excluded-file bursts could trigger a false overflow rescan. A larger speed shortcut—removing the audit or shrinking retrieval candidates—would risk missed filesystem changes or retrieval breadth. The bounded probe plus slower audit retains recovery while keeping quiet-tree work low. Exact-metadata collisions can take longer to detect in large trees; the fallback watcher keeps its two-second audit.

Verification — exact head ade2b8c6c4622c00bb8b413c6bd30a78f1045e82

• Local: zig build test --summary all — 33/33 steps succeeded; 1,476/1,489 tests passed, 13 skipped. zig build test -Dtest-filter=issue-756 --summary all — 3/3 passed. zig build -Doptimize=ReleaseFast passed; the copied candidate binary has SHA-256 f912b2698ecdcdfd2e64fbd80341d4de5e15ce939968d0b1192662350916d55f.
• Local: python3 scripts/e2e_mcp_test.py --binary .graff/verify/codedb-pr750-issue756 --project "$PWD" — 76/76 passed. The lazy MCP suite passed all six scenarios; deterministic semantic fault/recovery/security checks passed 27/27. Python 3.12 installer safety tests, update-worker tests, watcher stress in recursive and forced-fallback modes, formatting, and whitespace checks passed.
• Local: python3 scripts/repro_watcher_748.py --binary <candidate> --seconds 12 on a disposable 1,200-directory/4,800-file corpus — 1.00% of one core quiet and 0.92% under excluded-file churn; the published v0.2.5856 binary measured 8.16% and 30.54%. Each arm's index stayed stable, but their final file counts differed by one; this short sample is diagnostic, not parity-qualified benchmark evidence. A 125-second candidate sample covering two scheduled audits measured 1.42% on a larger synthetic tree.
• Earlier frozen hosted evaluation of the ranking code (before this watcher repair) improved original-six top-one accuracy from 113/128 to 119/128 and top-five coverage from 127/128 to 128/128; the fresh 20-question ItsDangerous holdout stayed 18/20 first. Requests question 17 fell from second to third, and several observed live-latency metrics exceeded the 10% review threshold. All seven repositories are now observed; these receipts are not a fresh current-head evaluation or a passed latency gate.
• Local blocker: python3 scripts/test_semantic_lifecycle.py --binary .graff/verify/codedb-pr750-issue756 ... stopped on its first hosted request with EmbeddingProviderRejected (later traced to an HTTP 530 response); an explicit synthetic semantic-index build failed too. The published binary stops earlier with FileNotFound, so this run neither establishes a candidate regression nor verifies live hosted behavior. The previous branch's 11 live lifecycle passes are historical, not validation of this head.
• Remote: bench-regression run 36223857312 completed successfully for exact head ade2b8c6c4622c00bb8b413c6bd30a78f1045e82: macos-resource-regression and bench passed. Separate guard checks were skipped.

Keep this PR draft. Exact-head CI passed, but do not merge or publish a new version until live hosted fallback and the flagged ranking/latency trade-off are resolved. The published v0.2.5856 tag remains unchanged; its release branch has newer installer commits that are included in this PR's base.

[github-actions]

👋 Thanks for the contribution! Quick heads-up: this repo lands changes on the current release/* branch, not main.

Please retarget this PR via Edit → base branch to the active release/* branch shown in the repository branch list.

(Automated hint — reply here if you need a hand.)

[yousafsabir]

Two observations on the watcher commit (0c93dc3) from the reporter of #756, against the same 16.6k-file / 5,100-dir APFS root. Read of the diff plus measurements on main; I could not build the branch locally (zig 0.17-dev required).

1. The periodic audit still walks the whole tree every 2 s. retainUnchangedChild and the top-of-walk shortcut both return early unless parents.affected is set, and it is only set for events_only passes. The poll_due path calls incrementalDiffDirty → events_only=false, so every directory is still opened and every known file stat'd each cycle. The comment says this is intentional. With file watches fully armed on main (max_watched = 20000, no overflow reads) that walk alone measured 44% of one core here over 45 s, so I would predict roughly that after this PR on this tree, versus the 1.75% per client on the 1,200-dir fixture. While recursive != null a healthy FSEvents stream already reports loss via MustScanSubDirs / KernelDropped / UserDropped, so the audit is redundant at 2 s; backing it off to tens of seconds when the stream is healthy would remove most of the remaining steady-state cost.

2. Excluded paths count toward the 2,048 pending cap, and cap overflow forces a whole-tree re-read. receive inserts every path under the root; shouldSkip/shouldSkipFile filtering happens only after drain. With FileEvents set, a build writing into target/ or node_modules/ can exceed 2,048 pending paths between drains, which sets rescan, which marks all known files and dirs dirty and forces dirtyForcesFileRead on all of them. On a Rust workspace with cargo watch running, that is a full-tree hash per rebuild. Suggest applying the skip filter inside receive before counting, and on cap overflow marking directories dirty (metadata audit) rather than forcing content reads.