Repository navigation
Windows-native Store route: Store engine and SDK Store face on native Windows - #452
NelsonLongxiang wants to merge 20 commits into
Conversation
… open The Store engine runs natively on Windows (owner direction 2026-10-25, route 2: Windows-native Store, no WSL2): - The cross-process writer lane is a named kernel mutex (Local\layerfs-writer-lane-<hash>): CreateMutexW + INFINITE wait; WAIT_ABANDONED also takes ownership, so a dead writer's lane is released by the kernel and the next writer proceeds. - The kernel guard reads the volume filesystem via GetVolumeInformationW and allows local NTFS volumes only; other filesystems and non-disk prefixes refuse Unsupported, matching the Linux ext4/xfs/btrfs allowlist discipline. - SQLite opens on Windows through bundled rusqlite; the open/profile cfg lists widen from macos/linux to include windows. Linux and macOS branches are unchanged. - The installer drops the unix-only mode bits and the daemons-uid check on Windows; destination and exclusivity checks are unchanged. - Boundary guard: audited-FFI entries become per-crate tuples and gain persistence backend/sqlite/windows_lane.rs; path matching is as_posix so the audited-module check works on Windows hosts (this repairs two false positives the separator bug produced there). Verification at this commit: cargo check --offline --locked -p layerfs-persistence -p layerfs-store-service (Windows, rustc 1.90.0) Finished; the guard's 24 self-tests pass; the guard reports only the four test-only-attribute findings that HEAD already reports, with no new violation; tests/windows_store_native.rs (new) drives create + two-session writes + two re-exec'd child processes through the lane with bounded waits: 1 passed. Production LOC: core 96783 -> 96948 (delta +165), sdk 3090 -> 3090 (delta 0); combined 109056 -> 109221 (delta +165). Counter: tools/production_loc.py --detail (pinned at this tree), scopes core and sdk, tests/tools/docs excluded. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…e face The SDK's in-process Store face and the construction libraries it needs run natively on Windows (route 2, owner direction 2026-10-25): - layerfs-project Init: host byte encoding moves to import/host.rs; Windows carries names as UTF-8 bytes (lossy outside UTF-8, a recorded limitation) and synthesizes the mode from the read-only attribute. Acquisition identity is the real volume serial and file index from GetFileInformationByHandle in the audited import/windows_identity.rs; zeroed identities made every entry collide onto one dev+ino key and Init refused with "the entry changed while Init read it". The cfg(unix) raw-byte and stat paths are unchanged. - layerfs-overlay: audited database/windows_identity.rs provides handle and path identity (volume serial, file index, link count) and GetDiskFreeSpaceExW free space; physical allocation checks and the startup gate widen to windows. store-service reads overlay backing identity through it instead of the (0,0,0) placeholder. - The napi addon builds on Windows: Store discovery (discovery_windows) uses a named kernel mutex with save/load record files; fs, branch, project and operator calls run in-process through the Ephemeral-AI-Lab#352 direct route; workspace, exec and daemon calls refuse Unsupported - FUSE and the daemon stay Linux-only. exec/output/pool/sandbox/workspace stay unix modules; the unix dispatch table is unchanged. - Boundary guard: audited FFI adds project import/windows_identity.rs and overlay database/windows_identity.rs; both crate roots move forbid->deny. Guard self-tests cover both. No tests were deleted or weakened; store-service keeps HEAD's Windows cfg asymmetry warnings no longer (backing delete lets and identity fields are cfg-scoped). Verification at this commit: cargo check --offline --locked for layerfs-persistence, -store-service, -project, -overlay and -node passes on Windows (rustc 1.90.0, only HEAD's pre-existing warnings in untouched crates) and in the rust:1.85.1-bookworm container (zero warnings for all five); tests/windows_store_native.rs 1 passed; guard self-tests 24/24; the lifecycle smoke (init -> fork dev -> one-Commit fs.write -> read back on dev and main -> list_commits -> close, no WSL2, no daemon) passes against this commit's addon. Production LOC: core 96948 -> 97232 (delta +284), sdk 3090 -> 3455 (delta +365); combined 109221 -> 109870 (delta +649). Counter: tools/production_loc.py --detail (pinned at this tree), scopes core and sdk; overlay +146, project +126, store-service +12, layerfs-node +365. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Draft for owner review, landing with the windows-native-store branch: - Root AGENTS.md: a Windows-native Store workstream row and a mental-model bullet superseding the Linux-only Store sentence for this route only, with its recorded limitations. - core/AGENTS.md: the audited Windows FFI inventory (persistence windows_lane beside the macOS seal wrapper; project and overlay identity modules, forbid->deny), the real volume-serial + file-index requirement for acquisition rows, and the Linux-container cross-check rule for cfg(unix) code touched on this route. - DIRECT_STORE.md: section 6 for the Windows-native route - mechanisms, what still refuses Unsupported, limitations, proof receipts. - 83-store-concurrency-model.md: the writer-lane table row names the Windows named kernel mutex equivalent (abandoned mutex taken by the next waiter). The two pre-existing broken relative links found while checking (DIRECT_STORE.md -> DIRECT_STORE_IMPL.md, 83 -> 84-global-sqlite-store.md) are left for their owners; this change adds no new link and all links it adds resolve. Production LOC: core 97232 -> 97232 (delta 0), sdk 3455 -> 3455 (delta 0); combined 109870 -> 109870 (delta 0). Docs only; same counter and scope as the previous commits. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…open The 3b slice of the Windows-native Store route: the existing test suites now run natively on Windows, and one real open-path race they exposed is fixed for every platform. Product changes: - Concurrent Store opens met Busy under the no-retry rule: the profile pragmas (first database-touching statements, where a WAL -shm is created and recovery runs) and the writable setup block (binding hints, lineage, merge scratch, file digests, merge-record sweep) ran outside the writer lane. Each is now one short lane hold, released before the reads that follow; macOS keeps exact SQLite Busy by design. The lane verifier's child failed "the child opens the Store: Busy" intermittently before this; eight consecutive runs pass after it. - import/host.rs: a writable Windows file synthesized mode 0o644, the read-only base - the read-only attribute was not stored. Base is 0o444/0o555 as documented. - outside_source on Windows becomes a real containment check of the resolved placement inside the resolved source (junctions and symlinks resolve); only spellings that do not resolve to the source's own path (a subst drive, an unresolvable alias) stay unrecognized, replacing the previous unconditional no-op. - windows-sys features Win32_Security is declared by layerfs-project and layerfs-overlay themselves: CreateFileW is gated on it by its SECURITY_ATTRIBUTES parameter, and the crates only compiled before through another crate's feature unification. Test changes (delete-or-adapt; deleted tests listed): - Deleted: init_sqlite's an_unsupported_platform_refuses_the_store_ before_any_file_exists - its premise (Windows refused) is inverted by this route; the namespace oracle tests it shared a file with now run on Windows. - unix-only kept unix-only: serverless_store (socket readiness), the four issue327 counters examples and restore_record (st_blocks / 0o600 records; non-unix mains refuse instead of fabricating). - Adapted: project support asserts the host's own reading of each entry (raw stat on unix; the synthesized contract on Windows); the held writer helper drops select() for a blocking readline (WinError 10093) and trims the readiness line; kernel_guard tests the Windows non-local-spelling refusal beside macOS's; the lane verifier pipes its children's report into its failure instead of discarding it. - New: sdk/js/test/store_lifecycle.mjs - init, fork, one-Commit write, read back, stat, commit listing and close on the real binding, no Docker, daemon or WSL2; it runs on both Linux (the E2E volume) and Windows. trace.mjs imports its fixture through pathToFileURL (a bare Windows path is not an ESM URL); reinit/init_refusals gate their unix-only contracts explicitly. Verification at this commit (Windows, rustc 1.90.0): cargo test for layerfs-persistence, -project, -store-service, -overlay, -node: 173 passed, 0 failed; the persistence suite clean across four consecutive runs. SDK: store_lifecycle, binary_write, reinit pass; trace 9/9 cases; packaging 2/2; init_refusals reports its unix-only skip. Linux (rust:1.85.1-bookworm container): cargo check --offline --locked --tests --examples for the five crates, Finished. Boundary guard self-tests 24/24; no new violation. Production LOC: core 97232 -> 97258 (delta +26; persistence +18, project +8), sdk 3455 -> 3455 (delta 0); combined 109870 -> 109896. Counter: tools/production_loc.py --detail (pinned at this tree), scopes core and sdk; tests, examples and manifests excluded. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
|
Slice 3b added ( Two product defects the Windows runs exposed, fixed for every platform:
Tests: the one test whose premise this route inverts (Windows refused the Store) is deleted and listed in the commit message; unix-only proofs stay unix-only with refusing non-unix stubs (no fabricated metrics); the SDK gains Production LOC: core 97,232 → 97,258 (+26), sdk unchanged; per-commit numbers in the commit message. |
Slice 4 of the Windows-native Store route: sdk/js/test/store_matrix.mjs
drives the Store face's whole public method table through the real
binding on the direct route - writes (write, edit, mkdir, cp, mv, rm),
reads (read with a line range, list, stat, grep, glob, blame),
comparisons (diff_files, diff_commits, diff_branches) and combination
(merge with its merged-branch readback, publish with main's readback and
the branch listing) - plus two contracts worth pinning: main is the
project's main line, not a Branch, so branch.list_commits("main") is
NoSuchBranch and main moves only through branch.publish; and on Windows
the daemon-only workspace surface refuses Unsupported once the client's
own deploy validation is satisfied. No Docker, no daemon, no WSL2; the
same file runs on Linux against the E2E volume.
Verification: three consecutive rounds of store_matrix, store_lifecycle,
binary_write and reinit all pass on Windows against this tree's addon;
trace (9 cases) and packaging (2 cases) unchanged.
Production LOC: core 97258 -> 97258 (delta 0), sdk 3455 -> 3455
(delta 0); combined 109896 -> 109896. Test-only change: the sdk scope
counts sdk/js/src and sdk/js/operator, so the new test contributes no
production lines. reference absent (retired R9c; last counted 65,417 at
tree 498dd19).
|
Slice 4 added ( Covered: every fs write ( Two contracts the matrix pins, both confirmed as designed (cross-platform, not Windows-specific):
No product change was needed: the matrix passed against the current implementation once the contracts above were honored. Test-only commit; production LOC unchanged (core 97,258, sdk 3,455). |
Four re-exec'd child processes write distinct files to one branch of one Store concurrently, each through its own client of the direct route: the named-mutex writer lane serializes the publishes and the discovery record guard serializes the shared home's project record, so every write must land. The parent reads all four files back and counts the Commits exactly (base + 4). Bounded: each child is killed at 30 s and its captured output is carried into the parent's failure. No Docker, no daemon, no WSL2; the same file runs on Linux against the E2E volume. Verification: four consecutive runs pass on Windows against this tree's addon (4 writers, base+4 commits each time). Production LOC: core 97258 -> 97258 (delta 0), sdk 3455 -> 3455 (delta 0); combined 109896 -> 109896. Test-only change. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
|
Evidence round added ( 1. Deep machine chain on a divergence/merge/publish scenario ( Controls, not just the pass: a determinism control — the scenario independently re-produced from scratch, same spec, PASS 7/7 again with identical witnesses ( 2. Cross-process concurrent writers ( Predicate DSL notes for the verify engine (fed back earlier via kb): |
Two coverage pins from the confidence review: - A dead lane holder does not block the next writer: a thread ends while owning the product's own named mutex (the name derived exactly as windows_lane derives it), the kernel marks it abandoned, and the Store still opens and writes through that WAIT_ABANDONED acquisition. Deterministic - no timing, no kill race. - The matrix test's merge becomes a divergent one: both sources advanced past their common base, so fastForward must be false, the merged head carries two parents, and both sides' files are read back on the merged Branch and on main after publish - the merge's content correctness, which the earlier fast-forward case could not show. Verification: windows_store_native 2 passed; store_matrix passes with the five-branch end state on Windows. Production LOC: core 97258 -> 97258 (delta 0), sdk 3455 -> 3455 (delta 0). Test-only change. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…ne test The 3b lane wraps deadlocked Linux opens: the setup block's raw second Writer::acquire held the directory flock on one open file description while Session::run's admission inside acquired it on another - flock is per description, so the opening thread waited on itself. Windows passed only because its named kernel mutex is thread-recursive; the Linux container suite caught it (concurrent_first_installs: 0.4s at the parent commit, >115s hung at 3b, diagnosed by /proc/locks, per-acquire backtraces and a parent-commit rerun). The corrected shape, verified on both platforms: - Session::connect holds the writer lane for the whole connect-time statement block (profile pragmas, stored-format and profile reads) - the open's first database-touching statements, where a cleaned -shm is rebuilt under an exclusive recovery hold. - The writable setup block (binding hints, lineage, merge scratch, file digests) runs inside one held publication lane: its statements enter as nested turns and take no second lock; cross-process opens are serialized by the lane itself. - The merge-record sweep stays plain: its admission takes the lane. Also: a_dead_lane_holder_does_not_block_the_next_writer moves to its own binary (windows_lane_abandoned.rs) so it cannot run beside the foreign-process test's re-exec'd children, whose spawn now filters --exact; the persistence test support's temporary directory names gain nanoseconds (the project crate's already had them), removing cross-run name collisions on Windows pid reuse. Verification: Linux container - concurrent_first_installs 0.37s; persistence 99 passed 0 failed, project 22/0, store-service 3/0, overlay 74/0. Windows - the lane binary 50/50 clean isolated runs; the SDK suite (store_matrix, store_concurrent, store_lifecycle) passes on a rebuilt addon. One residual, documented openly: under cargo test the lane test's re-exec'd child still meets Busy roughly once per 25 suite runs (never in 50 isolated binary runs); its report is carried into the failure. The busy source inside a mutex-held open remains undiagnosed - recorded as an open item, not fixed by this commit. Production LOC: core 97258 -> 97255 (delta -3; persistence -3: the closed-over profile tail moves lines, the lane-held setup comment replaces the raw wrap), sdk 3455 -> 3455 (delta 0). Counter: tools/production_loc.py (pinned at this tree), scopes core and sdk; tests excluded. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
eb0203b to
3d90143
Compare
|
Confidence round + a real bug it caught ( 1. Divergent merge readback ( 2. Deterministic dead-lane test ( 3. Linux suite runs — this caught a real bug I had shipped in 3b. Corrected shape ( One residual, stated openly: under Test-infra hardening en route: the abandoned test moved to its own binary (it ran beside the lane test's re-exec'd children); the children's spawn filters |
a1b23d1 to
d960a38
Compare
The residual lane-test flake is diagnosed and closed. Instrumented runs pinned it: a re-exec'd child met SQLITE_BUSY(5) on its first statement (PRAGMA synchronous=0, where a cleaned -shm is attached and possibly rebuilt) while holding the named writer mutex - the contender was the sibling process's connection teardown, whose close-time WAL checkpoint is SQLite work that no lane had ever covered. The same window exists on Linux through flock and in the daemon SDK's own direct-route close. The fix coordinates the close instead of the drop: layerfs-persistence exports StoreLane (a best-effort hold of the Store directory's writer lane, released on drop) and Handles::shutdown, which drops the session under that lane; the SDK's direct route holds the lane while the service's sessions close on storeClose; the lane tests' children and the abandoned-lane test close through shutdown. A plain drop keeps its old bare semantics - an uncoordinated close, like any error-path teardown, never waits behind the lane. An intermediate design (acquiring the lane in State::drop) is reverted: it deadlocked multi-session teardown in-process, caught by the SDK lifecycle (init+fork+write+close hung). Drop-time blocking locks stay out of the product. Verification: Windows - the persistence suite 25 consecutive rounds clean under cargo (the flake reproduced every 2-9 rounds before), the lane binary's children closing through shutdown; SDK store_matrix, store_concurrent, store_lifecycle and a minimal init-fork-write-close all pass on the rebuilt addon. Linux container - persistence 99/0, project 22/0, store-service 3/0, overlay 74/0. Production LOC: core 97255 -> 97271 (delta +16; persistence +14, node binding +2), sdk 3455 -> 3457 (delta +2); combined 109893 -> 109911. Counter: tools/production_loc.py (pinned at this tree). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
d960a38 to
27ace14
Compare
|
Residual closed ( Diagnosis (instrumented): the child met SQLITE_BUSY(5) on its first statement — Fix: coordinate the close, not the drop. Receipts: Windows persistence suite 25 consecutive cargo rounds clean (the flake reproduced every 2–9 rounds at the prior design); SDK |
StoreService::close now records the Store file (open_direct populates it through the new StoreService::at) and, after the overlay owner stops - its final Commits take the writer lane themselves, so no outer hold may surround that - drops the service under the Store's writer lane: the sessions' close-time checkpoints never run beside another opener's first attach. This is the daemon/SDK-symmetric completion of the coordinated close the SDK's direct route received; direct.rs drops its own manual hold in favor of the service's (a second hold on another descriptor would self-deadlock the closing thread on Linux). A service built with new() keeps an empty path and closes bare, as before - only open_direct records the file. Verification: Windows store-service tests 3/0, SDK store_matrix, store_concurrent, store_lifecycle pass on the rebuilt addon (the service close is in storeClose's path). Linux container: store-service 3/0; persistence 99/0, project 22/0, overlay 74/0 at this tree. The daemon suite needs a privileged container (/dev/fuse, fusectl, cgroup mounts): 62 non-mount tests pass; the 99 mount-path failures are environmental (attach:mount No such file, permission denied), the same at the parent commit - recorded as a pre-existing container limitation, not a regression of this change. Production LOC: core 97271 -> 97281 (delta +10; store-service +10), sdk 3457 -> 3455 (delta -2; the manual hold's comment moves into the service). Counter: tools/production_loc.py (pinned at this tree). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
c6d1756 to
27f6cf8
Compare
|
Coordinated close, daemon-symmetric + a third machine chain (head
|
K3 review (猿光智启专家团2.0——layerfs底座, APPROVE-WITH-NITS, F1): the lane and discovery mutex names hashed the directory with DefaultHasher, whose algorithm may change between Rust releases - an older and a newer binary computing different names for one directory would silently run two lanes. The names now derive from SHA-256 of the resolved directory's encoded bytes, truncated to 64 bits (sha2 is already in the locked graph; the direct edges add no new package), in persistence's windows_lane (exposed as lane_name for the test) and the SDK binding's discovery guard. The dead-lane test replicates the new derivation. F2 (Local\ namespace does not cross sessions; the product opens its Store within one login session today) is recorded as a §6 limitation in DIRECT_STORE.md, with the Global\ privilege evaluation noted should cross-session coordination ever be needed. Verification: Windows persistence 87/0 (both lane binaries), project 10/0, store-service 3/0, overlay 74/0; SDK matrix/concurrent/lifecycle pass on the rebuilt addon; Linux container cargo check --offline --locked for persistence and the node binding, Finished. Production LOC: core 97281 -> 97282 (delta +1), sdk 3455 -> 3453 (delta -2); combined 109919 -> 109918. Counter: tools/production_loc.py (pinned at this tree). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
|
K3 review findings landed (
Receipts: Windows persistence 87/0 (both lane binaries), project 10/0, store-service 3/0, overlay 74/0; SDK matrix/concurrent/lifecycle on the rebuilt addon; Linux container |
…in doc 83 The concurrency model's table gains the open/teardown row: the connect-time statement block (where a cleaned WAL -shm is rebuilt under an exclusive recovery hold), the writable setup block (held publication lane; nested turns take no second lock) and the coordinated close (StoreLane/Handles::shutdown; the service stops overlays first, then drops the sessions under the lane) all run under the same cross-process lane - SQLite's close-time checkpoint is writer work like any other and never runs beside another opener's first attach. The lane-name note records the SHA-256 truncation (never DefaultHasher, K3 F1) and the Local\ session-namespace limitation (DIRECT_STORE.md §6, K3 F2). Also: the PR#452 observation script switches from the gh CLI to the unauthenticated REST API so the job engine's non-interactive environment (no gh auth) can run it. Production LOC: core 97282 -> 97282 (delta 0), sdk 3453 -> 3453 (delta 0); combined 109918 -> 109918. Docs and tooling only; the shell script lives outside the counted scopes. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Four maintained-doc spots still said the Store face is Linux-only: - sdk/js/README.md, the Store-calls paragraph: on Linux the Store sits on the daemons' shared local filesystem; on Windows the native route opens a local NTFS volume directly, no daemon involved (Workspace, exec and daemon calls refuse Unsupported there). - the LAYERFS_STORE_PATH row: the same split (Linux shared local filesystem / Windows local NTFS volume). - the Builds section: the build script refuses Windows by design; the Windows addon builds directly with cargo and renames layerfs_node.dll into native/layerfs.node. - cluster_one_handbook's persistence row: macOS/Linux/Windows, with the owner-direction date. The versioned 0.1.x manuals are untouched (immutable per release); benchmark_instruction's Linux sentence is about test hosting, not this route. Link check passes for both files at their real base paths. Production LOC: core 97282 -> 97282 (delta 0), sdk 3453 -> 3453 (delta 0); combined 109918 -> 109918. Docs only. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…udited identity Backing::delete refused Unsupported off unix, so the direct route's private overlay files on Windows were never reclaimed by their last owner. The Windows arm now applies the same discipline through the audited identity: the entry at the path and the retained open handle must both still be the acquisition's file (volume serial + file index, sole link) before std::fs::remove_file; the retained handle - opened with std's FILE_SHARE_DELETE - completes the removal when Drop closes it, exactly the Linux order (check, remove, close). A swapped-in file at the path is refused and stays. windows_path_identity is exported from layerfs-overlay beside windows_handle_identity; the device/inode fields' dead_code allowance widens to both reading platforms. Tests (new, overlay_backing_windows.rs, dangling-Weak pool so Drop skips the callback): a matching overlay file is deleted at last-owner release (bounded wait for the pending removal to complete on handle close), and a file swapped in behind the backing's back is refused and stays. Verification: Windows - the new tests 2/2, store-service 5/0 (was 3), persistence 87/0, project 10/0, overlay 74/0; SDK matrix/lifecycle pass on the rebuilt addon. Linux container - cargo check --locked for store-service and overlay, Finished. Production LOC: core 97282 -> 97316 (delta +34; store-service +31, overlay +3), sdk 3453 -> 3453 (delta 0); combined 109918 -> 109952. Counter: tools/production_loc.py (pinned at this tree). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
0121709 to
2a4cf66
Compare
|
Windows overlay last-owner deletion landed ( Tests: new With this, the DIRECT_STORE §6 limitation list shrinks to the inherent items (non-UTF-8 names, synthesized mode, subst-drive aliasing, logical overlay allocation). |
…ation §6 said last-owner overlay deletion keeps its Linux identity check and is not exercised on Windows; commit 2a4cf66 lifted exactly that. The paragraph now states the parity (audited identity on both sides of the comparison) and rewords the aliasing limitation to what 3b actually implements (resolved-containment; subst/unresolvable aliases stay unrecognized). Production LOC: core 97316 -> 97316 (delta 0), sdk 3453 -> 3453 (delta 0). Docs only. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The Windows overlay reported its logical length as allocated bytes (compression/sparse never visible; the receipt pair: a written overlay whose logical length is 77,824 holds 131,072 on disk - the stand-in understated by 40%). The audited Windows identity module gains windows_allocation_size (GetFileInformationByHandleEx / FileStandardInfo on the retained handle, compression/sparse aware) and Allocation::new/state read it: high_water seeds from the real footprint at open, and state reports it beside the logical length, parity with Linux's blocks*512. Test (new allocation_windows.rs): a written overlay's allocated_bytes is positive, at or above its logical length within a cluster margin, observed pair printed for the receipt. Verification: Windows - overlay 75/0 (74 + the new test), persistence 87/0, project 10/0, store-service 5/0. Linux container - cargo check --offline --locked for overlay and store-service, Finished. Production LOC: core 97316 -> 97336 (delta +20; overlay +20), sdk 3453 -> 3453 (delta 0); combined 109952 -> 109972. Counter: tools/production_loc.py (pinned at this tree). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
|
Windows allocation fidelity landed ( |
The Windows backing-inside-source check compared canonicalized DOS paths — a spelling whose DOS volume name survives resolution (a subst drive: GetFinalPathNameByHandle keeps the alias letter; live probe: realpath(Y:/inner) = Y:/inner) slipped past the containment. The check now runs Linux's algorithm on the audited Windows identity: canonicalize once, then compare every placement ancestor's volume serial + file index with the source's — junctions, subst drives and any spelling that resolves to the source's directories carry their device and index, and are refused. Test (init_sqlite, windows): a directory junction naming the source is refused BackingInsideSource before any operation. Verification: Windows project suite 11/0 (was 10); Linux container cargo check --offline --locked for layerfs-project, Finished. Production LOC: core 97336 -> 97341 (delta +5; project +5), sdk 3453 -> 3453 (delta 0); combined 109972 -> 109977. Counter: tools/production_loc.py (pinned at this tree). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
|
Alias check walks identities (subst/junction parity) ( |
The ruling's cadence (merge → stability window → WinFsp independent period) becomes a fixed three-step checklist, each step reading a mechanism that already exists: the observation job's merge record flips the pr_merged_and_stable gate, the owner bounds the stability window (the ruling names no length), and the independent period starts contract-first under its tree node. No new decisions deferred to the merge moment. Production LOC: core 97341 -> 97341 (delta 0), sdk 3453 -> 3453 (delta 0). Docs only. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
Upstream moved the tree: core/crates -> engine/crates, sdk/js -> api/agent (+ api/cli, api/backend, api/runtime), tools to the top level, and rewrote the SDK API (LayerFS.init/open static factories; settings from the environment; operator.* -> backend./runtime.; sandbox create/close/list surfaces). The merge transplant: - Windows dispatch/backend_call arms follow the new method tables: the daemon/sandbox surface refuses Unsupported by name, Store calls stay direct (backend.store.*, runtime.project.get_init). - The three SDK tests adapt to LayerFS.init/open + LAYERFS_STORE_PATH; the win32 daemon-refusal assertion moves to sandbox.create (reaches the native arm on the new signature). - StoreLane moves from lib.rs to store/lane.rs (upstream's lib.rs is a declaration-only entry now; the guard enforces it). - The boundary guard keeps upstream's shape and our semantics: audited FFI tuples (four Windows modules), project/overlay out of UNSAFE_FREE_CRATES with deny roots, as_posix comparisons; the guard's self-tests pass on Windows (48/48) and the scan is PASS over 962 files. - engine's layerfs-project Cargo.toml gains the windows-sys target block (the audited identity probe); api/README carries the Windows route beside the worker model; core/AGENTS.md keeps upstream's base with the audited-Windows-FFI section (one copy). - Dropped superseded old paths (sdk/js/README.md, core/tools/*, core/crates/layerfs-project/Cargo.toml). Verification at this merge: Windows cargo check --locked for the five crates, Finished; persistence 87/0, project 11/0, store-service 8/0, overlay 80/0; the three adapted SDK tests pass on the rebuilt addon (matrix 7 commits/5 branches, concurrent 4 writers, lifecycle); the boundary guard PASS + self-tests 48/48; Linux container rust:1.85.1-bookworm cargo check --locked for the five crates, Finished. Production LOC: core 97341 + upstream's new tree; the exact before/after of this merge commit is the first-parent comparison recorded by the merge itself (parent 1 = 6b63a64 at core 97341). reference absent (retired R9c; last counted 65,417 at tree 498dd19).
The last recoverable §6 limitation: a Windows name outside UTF-8 (an unpaired surrogate - legal in an NTFS filename) crossed to the Store as to_string_lossy replacement characters and never came back. The host codec (import/host.rs, both entry points) now encodes the name's UTF-16 units as WTF-8 and decodes back through OsString::from_wide, which does not validate pairing: every unit, orphan surrogates included, survives the byte round trip exactly. The table lives in import/host_wtf8_inner.rs, included by the product module and by the name-roundtrip test alike. Tests (name_roundtrip_windows.rs): plain/Chinese/emoji names; an orphan low and an orphan high surrogate name (both proven outside UTF-8); an exhaustive sweep of every single UTF-16 unit; and a real file whose name carries an orphan surrogate is created on NTFS (the host accepts it and the codec is what carries it). Verification: Windows project suite 15/0 (was 11); Linux container rust:1.85.1-bookworm cargo check --locked for layerfs-project, Finished. DIRECT_STORE §6 wording: names are lossless WTF-8 now. Production LOC: first-parent (f0f5a5a66 -> this) changed-scope count: layerfs-project +57 production lines (host_wtf8_inner.rs codec; host.rs net +2), docs +0; the old-tree/merged-tree global totals are not comparable across the 1.1.0 restructure and are not restated here. Counter: tools/production_loc.py at this tree. reference absent (retired R9c; last counted 65,417 at tree 498dd19).
|
Windows names travel as lossless WTF-8 ( |
What
The Windows-native Store route (owner direction 2026-10-25, route 2): the Store
engine and the SDK's in-process Store face run natively on Windows — no WSL2,
no daemon. FUSE, the daemon, Workspaces and Exec remain Linux-only; on Windows
those calls refuse
Unsupported.Three commits:
feat(persistence)— Store engine on Windows: the cross-process writerlane as a named kernel mutex (
WAIT_ABANDONEDtaken by the next waiter),the kernel guard's local-NTFS-only allowlist via
GetVolumeInformationW,bundled rusqlite opens, installer cfg split, the boundary guard's audited
FFI set gaining
windows_lane.rs(entries become per-crate tuples;as_posix()matching repairs two Windows-host false positives), and across-process lane verifier test (two re-exec'd child processes, bounded
waits).
feat(sdk)— project Init, overlay identity and the napi addon onWindows: host byte encoding (
import/host.rs), real volume-serial +file-index acquisition identity in audited FFI
(
project/import/windows_identity.rs,overlay/database/windows_identity.rs; both crates forbid→deny),store-servicereading real overlay backing identity, anddiscovery_windows.rs+ the Windows dispatch table (Store callsin-process per dsh history replay: LayerFS load test over deepseek-harness's git history #352; workspace/exec/daemon refuse
Unsupported).docs— draft for owner review: root AGENTS.md workstream row +supersession bullet, core/AGENTS.md audited-FFI inventory + the
real-identity requirement + the Linux-container cross-check rule,
DIRECT_STORE.md §6, and the writer-lane row in architecture doc 83.
Verification (this tree, rerun at final identity)
cargo check --offline --lockedforlayerfs-persistence,-store-service,-project,-overlay,-node—Finished; the only warnings are HEAD's pre-existing ones in untouched
crates (
layerfs-sandbox×2,layerfs-api×1). Each commit's tree alsochecks
--locked(the lock is split per commit).rust:1.85.1-bookwormcontainer,cargo check --offline --locked,same five crates): Finished, zero warnings.
cargo test -p layerfs-persistence --test windows_store_native— 1 passed (create + two-session writes + twoforeign child processes through the lane).
dev→ one-Commitfs.write→ read back ondevandmain→list_commits(1) → close.No WSL2, no daemon.
test-only attributefindings HEAD already reports (in
discovery.rs,pool.rs,sandbox.rs, untouched here) and no new violation; its 24 self-testspass. (On a Windows host, HEAD's guard additionally false-positives on
the two unix-audited modules; this branch's
as_posix()fix repairsthat.)
Recorded limitations (also in DIRECT_STORE.md §6)
Windows Init carries non-UTF-8 names lossily, synthesizes the mode from the
read-only attribute, has no hard-link membership and no
backing-inside-source alias check; overlay allocation is the logical length;
last-owner overlay-file deletion keeps its Linux identity check (not
exercised — this route holds no Workspace mounts).
Production LOC
b5814bf3afeat(persistence)feat(sdk)docsCounter:
tools/production_loc.py --detailat each commit snapshot; scopescore + sdk; tests/tools/docs excluded. reference absent (retired R9c; last
counted 65,417 at tree 498dd1917812).
For the owner
dsh macOS or WSL2 exceptions, the Disposable/WAL/OFF-only profile, or any
Linux behavior (unix branches are byte-identical where possible, cfg lists
only widen).
permanent Windows test location (sdk tests today are unix-worktree-run),
and whether a WinFsp mount research spike is in scope for a later slice.