macOS: watcher burns ~60% of a core per MCP client on a 16k-file root with no worktrees (overflow re-hash + unconditional 2 s full-tree audit) #756

Follow-up to #751 / #754, on a root that is not a worktree forest (0 .worktrees directories). Opened as requested in the #751 close comment.

Summary

On 0.2.5856 (55483a0, macOS 15.8, Apple M1 Pro, 8 cores, 16 GB), every codedb mcp process attached to my main project sits at ~60% of one core indefinitely with scan: ready, no retrieval calls, and no edits. With three MCP clients open on the same root (Zed context server, Codex app-server launched by Zed, Claude Code) that is ~1.8 cores permanently, ~600 MB RSS each. This is what Activity Monitor shows right after a reboot: three codedb mcp at ~62% each.

Startup is not the problem: the snapshot loads in ~6 s. The cost is the steady-state watcher loop, and it has two independent components — overflow re-hashing (already named in #751) and the unconditional two-second full-tree audit, which costs about the same even when every file is natively watched.

Relation to draft PR #750 (added 2026-09-20)

The watcher commit in #750 (0c93dc3) addresses cost 1 below. With the recursive FSEvents stream active, addUnwatched no longer runs on the two-second tick, so the overflow set is never forced through dirtyForcesFileRead and the ~314 MB read+hash per cycle goes away.

Cost 2 (the unconditional two-second full-tree audit) and cost 3 (one watcher per MCP process) are unchanged in that branch: the polling pass calls incrementalDiffDirty with events_only=false, so parents.affected is never set and retainUnchangedChild cannot retain anything. Details and one regression risk in the pending cap are in #750 (comment). On this tree I would expect roughly the 44% figure from run B after #750, so this issue should stay open for the audit cadence rather than close on that release.

Tree

lsof on a live MCP process: 1032 DIR fds, 7 REG fds — the entire 1024-vnode budget is consumed by directory watches (armKqueue admits directories first, src/watcher.zig:1692), so every one of the 16.6k files is in the overflow set.

Measurements (process CPU-time delta from ps / wall time; 100% = one core)

Live processes 11 minutes after boot, no tool calls except codedb_status:

  PID  %CPU      TIME ELAPSED
 2953 100.0   6:39.01   11:09   (Zed)
 3142 100.0   6:37.89   11:06   (Codex via Zed)
 3280  99.6   6:35.08   11:01   (Claude Code)

→ 6:35 CPU / 11:01 wall ≈ 60% per process, sustained. seq in codedb_status stayed at 16671 the whole time (nothing re-indexed), so this is verification work, not indexing.

Controlled single-process runs (harness below; measured from t=20 s to t=65 s after initialize, quiet tree):

Same two runs with CODEDB_NO_WARMUP=1, measured later (t=60 s to t=105 s) to rule out the warmup thread: 55% (default) and 44% (max_watched=20000). So neither startup nor warmup is involved; this is the incrementalLoop steady state.

Per-cycle CPU deltas in run A alternate between ~1.2 s and ~2.7 s per ~3 s wall, i.e. the loop is busy for most of every two-second period.

Where the time goes (0.2.5856, 55483a0)

1. Overflow re-hash of the whole tree every 2 s. incrementalLoop (src/watcher.zig:2646-2652) calls watch.addUnwatched(&polling_dirty) on the two-second cadence; addUnwatched (:1665) puts every overflow path into the dirty set. dirtyForcesFileRead (:2972) then makes applyKnownFile (:2834) skip the exact-metadata shortcut (:2851) and read + Wyhash the file. With zero file watches, that is all 16.6k files / ~314 MB read and hashed every two seconds, per process. This is the "residual periodic overflow hashing" macOS: residual periodic watcher CPU is multiplied across MCP clients after #748 #751 left open; on this tree it is not residual, it is the primary cost.

2. Unconditional full-tree audit every 2 s, even with nothing dirty. The same poll_due block calls incrementalDiffDirty regardless of whether the dirty set is empty. incrementalDiffSelected with events_only=false rebuilds the parent index, re-initialises FilteredWalker (re-reads every nested .gitignore), and walks all ~5,100 directories stat-ing every entry — the comment at :2755 says this is deliberate ("polling entry points deliberately retain full discovery semantics, including callers that pass an empty dirty set"). Run B shows this alone is ~44% of a core on this tree: raising the budget removed the preadvs but not the walk.

3. Per-client multiplication. Each MCP process spawns its own incrementalLoop (src/commands.zig:538); the singleton lock is cli-daemon-only. Three clients → three full copies of (1)+(2). README line 131 acknowledges this.

Reproduction

Any macOS tree with more directories than max_watched (so file watches overflow to zero) and a few thousand files. Mine is proprietary, but the shape is ordinary: 7 repos side by side, one of them with a large postman/collection-dir (2,820 dirs / 9,544 small JSON files).

# harness2.py — spawn `codedb mcp`, send initialize + roots, then sample CPU-time deltas
import json, os, subprocess, sys, time, threading
ROOT = sys.argv[1]; args = sys.argv[2:]
p = subprocess.Popen(["codedb", *args, "mcp"], cwd=ROOT, stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True, bufsize=1)
def send(o): p.stdin.write(json.dumps(o) + "\n"); p.stdin.flush()
def reader():
    for line in p.stdout:
        try: m = json.loads(line)
        except Exception: continue
        if m.get("method") == "roots/list":
            send({"jsonrpc": "2.0", "id": m["id"], "result": {"roots": [{"uri": "file://" + ROOT, "name": "root"}]}})
threading.Thread(target=reader, daemon=True).start()
send({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2025-06-18", "capabilities": {"roots": {"listChanged": True}}, "clientInfo": {"name": "h", "version": "0"}}})
send({"jsonrpc": "2.0", "method": "notifications/initialized"})
def cpu():
    o = subprocess.run(["ps", "-o", "cputime=", "-p", str(p.pid)], capture_output=True, text=True).stdout.strip()
    m, s = o.split(":")[-2:]; return int(m) * 60 + float(s)
time.sleep(20); c0 = cpu(); w0 = time.time()
time.sleep(45); c1 = cpu(); w1 = time.time()
print(f"steady-state {(c1 - c0) / (w1 - w0) * 100:.0f}% of one core")
p.terminate()

python3 harness2.py /path/to/tree                                   # run A
printf 'max_watched = 20000\n' > big.codedbrc
python3 harness2.py /path/to/tree --config-file=$PWD/big.codedbrc   # run B
lsof -p <pid> | awk 'NR>1{print $5}' | sort | uniq -c                # DIR vs REG split

Expected

A quiet tree with scan: ready should cost close to zero CPU per client, and N clients on the same root should not cost N× the watcher. Concretely:

• Overflow verification should not re-read files whose exact metadata (mtime_ns, ctime_ns, inode, size) is unchanged on a local APFS volume, or should at least be rate-limited to a small slice per cycle rather than the whole set every 2 s.
• The two-second directory audit should not walk clean subtrees when no policy file changed and no event arrived (the same bound the event path already has).
• Admit file watches before directory watches, or share one watcher across MCP processes on the same root (the FSEvents rewrite in draft PR fix: restore default semantic fallback and reduce macOS watcher work #750 looks like the right direction).

Workarounds I found

• max_watched = 20000 in .codedbrc: stops the preadv storm (58% → 44%) but the audit walk remains.
• Excluding the big fixture tree via .codedbignore, or closing all but one MCP client, only scales the same loop down.

Happy to run a candidate binary against this tree and report the same numbers.