Skip to content

vfs: integrate with CJS and ESM module loaders - #63653

Open
mcollina wants to merge 43 commits into
nodejs:mainfrom
mcollina:vfs-module-loader-integration
Open

vfs: integrate with CJS and ESM module loaders#63653
mcollina wants to merge 43 commits into
nodejs:mainfrom
mcollina:vfs-module-loader-integration

Conversation

@mcollina

@mcollina mcollina commented May 30, 2026

Copy link
Copy Markdown
Member

Makes require() and import resolve files served by node:vfs. Before this PR, mounted VFS files were only visible through fs.*; the loaders went straight to the real filesystem.

Design

vfs.mount() takes no arguments and returns the reserved absolute mount point of the instance: ${os.devNull}/vfs/<layerId> (for example /dev/null/vfs/0). os.devNull is a character device on POSIX and a device-namespace path on Windows; neither can have child filesystem entries, so no real path can ever exist under this root.

Everything about ownership is decidable from the path alone:

  • Dispatch: one path normalization, one prefix comparison against the namespace root, one map lookup on the layer id. O(1) in the number of mounted layers — the benchmark (benchmark/vfs/bench-fs-dispatch.js) reports flat per-call latency across 1..10 mounted layers.
  • Real-fs paths short-circuit with a single prefix check.
  • Unmount purges loader caches by prefix-scanning the mount point — no per-VFS ownership tracking is needed, and no other-VFS or real-fs entries are ever touched.
  • Symlink correctness: realpathSync of a VFS entry always resolves to another path under the same mount point, so cache entries hidden behind symlinks are captured by the prefix scan.
  • Dynamic-import identity: resolved URLs are plain file: URLs under the mount point; import(import.meta.resolve(x)) re-hits the same module job.

Two instances mounting simultaneously never collide (each gets its own layer-<id> segment), so there is no overlap validation and no ordering hazard between mounts.

Loader integration

Toggleable wrappers in the loaders. Null fast-path when no VFS is mounted; otherwise the VFS answers stat / readFile / realpath / legacyMainResolve / getFormatOfExtensionlessFile and the four package.json C++-binding calls.

Module identity follows the path: __filename, module.filename, and import.meta.url are the plain absolute path (or file: URL) of the module under the mount point — no synthetic decorations.

Review guide

Suggested reading order:

# File Role
1 lib/internal/vfs/router.js Reserved-namespace helpers: getVfsRoot, getLayerRoot, getLayerIdFromPath.
2 lib/internal/vfs/file_system.js mount() returns the layer's reserved mount point.
3 lib/internal/vfs/setup.js Heart of the PR. findVFS (O(1) lookup), fs handler, loader overrides with parity to src/node_modules.cc / src/node_file.cc, prefix-scan cache purge.
4 lib/internal/modules/helpers.js Hook surface: loader* wrappers, setLoaderFsOverrides / setLoaderPackageOverrides, purgeRealpathCacheForPrefix.
5 lib/internal/modules/cjs/loader.js stat() + TS read routed through wrappers; purgeModuleCachesForPrefix for unmount.
6 lib/internal/modules/esm/resolve.js legacyMainResolve + internalModuleStat + toRealPath routed. No URL decoration.
7 lib/internal/modules/esm/load.js getSourceSync reads via the wrapper.
8 lib/internal/modules/esm/get_format.js Extensionless format detection routed.
9 lib/internal/modules/package_json_reader.js 4 C++ binding calls routed; purgePackageJSONCacheForPrefix.
10 lib/fs.js statSync / lstatSync honour throwIfNoEntry:false on ENOENT from the VFS handler.
11 doc/api/vfs.md Mount semantics + Module loader integration section.

Tests (all gated by --experimental-vfs): test-vfs-mount, test-vfs-mount-errors, test-vfs-multi-mount, test-vfs-require, test-vfs-import, test-vfs-module-hooks, test-vfs-module-hooks-cleanup, test-vfs-package-json, test-vfs-package-json-cache, test-vfs-invalid-package-json, test-vfs-scoped-cache-purge, test-vfs-layer-id, test-vfs-layer-tag-prefix.

Refs

The reserved-namespace design follows the "no interference with valid paths in the file system" requirement from the SEA VFS requirements doc.

Out of scope

SEA + VFS, overlay/stacking of multiple VFS layers under one prefix, migrating the C++ package_configs_ cache, broader permission-model integration.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/loaders

@nodejs-github-bot nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels May 30, 2026
@codecov

codecov Bot commented May 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.36994% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.16%. Comparing base (0e2126d) to head (013de9f).
⚠️ Report is 284 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/vfs/setup.js 88.22% 49 Missing ⚠️
src/node_modules.cc 77.96% 4 Missing and 9 partials ⚠️
lib/internal/vfs/file_system.js 97.18% 2 Missing ⚠️
lib/internal/vfs/router.js 96.42% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #63653      +/-   ##
==========================================
- Coverage   92.04%   90.16%   -1.88%     
==========================================
  Files         381      746     +365     
  Lines      169208   243310   +74102     
  Branches    25926    45856   +19930     
==========================================
+ Hits       155746   219380   +63634     
- Misses      13174    15422    +2248     
- Partials      288     8508    +8220     
Files with missing lines Coverage Δ
lib/fs.js 98.36% <100.00%> (+4.13%) ⬆️
lib/internal/modules/cjs/loader.js 98.16% <100.00%> (+18.59%) ⬆️
lib/internal/modules/esm/get_format.js 95.20% <100.00%> (+20.89%) ⬆️
lib/internal/modules/esm/load.js 91.40% <100.00%> (+8.14%) ⬆️
lib/internal/modules/esm/resolve.js 99.03% <100.00%> (+12.10%) ⬆️
lib/internal/modules/helpers.js 98.78% <100.00%> (+8.00%) ⬆️
lib/internal/modules/package_json_reader.js 99.47% <100.00%> (+11.87%) ⬆️
src/node_modules.h 100.00% <ø> (ø)
lib/internal/vfs/file_system.js 99.69% <97.18%> (+0.47%) ⬆️
lib/internal/vfs/router.js 97.46% <96.42%> (+12.68%) ⬆️
... and 2 more

... and 539 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollina

Copy link
Copy Markdown
Member Author

@joyeecheung take a look, should be easier to review.

Comment thread lib/internal/modules/esm/load.js
Comment thread lib/internal/modules/esm/resolve.js
@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from 6321e08 to 51b033a Compare June 1, 2026 15:54
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jun 3, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jun 3, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@joyeecheung joyeecheung left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A design question recently occurred to me: have we explored the versioning of the mounting?

Comment thread lib/internal/vfs/setup.js Outdated
@mcollina

mcollina commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

A design question recently occurred to me: have we explored the versioning of the mounting?

what do you mean? You mean multiple vfs layers on top of each other?

@joyeecheung

joyeecheung commented Jun 3, 2026

Copy link
Copy Markdown
Member

what do you mean? You mean multiple vfs layers on top of each other?

For the stacks to have some kind of version number/ID to identify the current status?

BTW I just noticed that there's no mention of unmount() and mount() in the VFS docs..

@mcollina

mcollina commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

BTW I just noticed that there's no mention of unmount() and mount() in the VFS docs..

I did purge them when doing the splitting; I forgot to bring them back. I'll add them to this PR.

@mcollina

mcollina commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

For the stacks to have some kind of version number/ID to identify the current status?

No but we totally should.

@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from 51b033a to 294a19c Compare June 4, 2026 07:23
@mcollina

mcollina commented Jun 5, 2026

Copy link
Copy Markdown
Member Author

@joyeecheung PTAL

@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from 75bb5c7 to 99a5a5c Compare June 15, 2026 09:00
mcollina added 6 commits July 13, 2026 12:23
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from d1d72c3 to 0d5fb61 Compare July 13, 2026 13:59
@mcollina

Copy link
Copy Markdown
Member Author

@joyeecheung done

@trivikr
trivikr requested a review from joyeecheung July 15, 2026 17:27

@joyeecheung joyeecheung left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove the (AI generated?) comments that just restate what happens in code with prose or stating how other code calls it currently (which can get out of sync very quickly and is anti-encapsulation?)

Also I think the documentation still doesn't clarify a core question: how is the module lookup algorithm altered once an VFS is mounted. It needs to specify that precisely in the module loading spec in modules.md instead of just leaving some hand wavy descriptions like "the module loader will consult it, it will work" (but how and where in the algorithm? and when will they NOT be consulted?)

Comment thread lib/internal/modules/esm/load.js Outdated
Comment thread lib/internal/modules/helpers.js Outdated
Comment on lines +71 to +77
// Toggleable loader hooks for VFS support. Each `loader*` wrapper below
// funnels through a single factory: if an override is registered for the
// original function it is called first; anything but `undefined` short-
// circuits the wrapper, otherwise the wrapper falls through to the
// pinned native. This keeps the wrapper surface a one-liner per binding
// - adding a new binding is one `wrapLoaderMethod` call, no extra state
// variables or per-hook setter branches to keep in sync.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Toggleable loader hooks for VFS support. Each `loader*` wrapper below
// funnels through a single factory: if an override is registered for the
// original function it is called first; anything but `undefined` short-
// circuits the wrapper, otherwise the wrapper falls through to the
// pinned native. This keeps the wrapper surface a one-liner per binding
// - adding a new binding is one `wrapLoaderMethod` call, no extra state
// variables or per-hook setter branches to keep in sync.
// Toggleable loader hooks for VFS support.

This is just describing what obviously happens below with 7 lines of prose?

Comment thread lib/internal/modules/helpers.js Outdated
Comment on lines +673 to +674
// internal/vfs/setup.js) map directly into this array; indexes 0..6
// use packageConfig.main as the prefix, 7..9 use pkgPath directly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// internal/vfs/setup.js) map directly into this array; indexes 0..6
// use packageConfig.main as the prefix, 7..9 use pkgPath directly.
// internal/vfs/setup.js) map directly into this array.

This is repeating what comments below says.

Comment thread benchmark/vfs/bench-fs-dispatch.js Outdated
Comment thread doc/api/vfs.md Outdated
Comment thread doc/api/vfs.md Outdated
Comment thread doc/api/vfs.md Outdated
Comment on lines +326 to +332
participate in module resolution and loading. Both
`require()` / `require.resolve()` (CommonJS) and `import` /
`import.meta.resolve()` (ECMAScript modules) consult the VFS through
the same toggleable hooks that `node:fs` uses, so files served from
the VFS are first-class modules: `package.json` is honoured,
extensionless files are sniffed for Wasm vs. JavaScript, conditional
`exports` / `imports` work, and so on.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
participate in module resolution and loading. Both
`require()` / `require.resolve()` (CommonJS) and `import` /
`import.meta.resolve()` (ECMAScript modules) consult the VFS through
the same toggleable hooks that `node:fs` uses, so files served from
the VFS are first-class modules: `package.json` is honoured,
extensionless files are sniffed for Wasm vs. JavaScript, conditional
`exports` / `imports` work, and so on.
participate in module resolution and loading. Both
`require()` / `require.resolve()` (CommonJS) and `import` /
`import.meta.resolve()` (ECMAScript modules) consult the VFS through
the same toggleable hooks that `node:fs` uses, so files served from
the VFS are first-class modules: `package.json` is honoured,
extensionless files are sniffed for Wasm vs. JavaScript, conditional
`exports` / `imports` work, and so on.

This paragraph is listing example without actually clarifying anything - exactly how do they appear in the module lookup algorithm, and with what precedence?

mcollina added 2 commits July 18, 2026 15:10
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
mcollina and others added 5 commits July 24, 2026 02:34
Co-authored-by: Joyee Cheung <joyeec9h3@gmail.com>
The microbenchmark drove 1000+ iterations to reach the optimizing tier,
which is not representative of real workloads. Module-loading against a
mounted VFS is the meaningful signal and can be measured with the
existing benchmark harness.
Measures loading a large CJS or ESM module graph from a mounted VFS,
relying on the unmount cache purge so every iteration is a cold load.
@mcollina

Copy link
Copy Markdown
Member Author

@joyeecheung PTAL, I think I did all the fix you suggested.

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@joyeecheung joyeecheung left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if it's the AI talking (looks like it is) but this is now changing/deleting comments of code that aren't changed in this PR...

Comment thread doc/api/vfs.md
Comment on lines +300 to +302
instead, every file system operation those algorithms perform
(existence probes, file reads, `package.json` lookups, real-path
resolution) is dispatched on the path being probed: paths under a

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
instead, every file system operation those algorithms perform
(existence probes, file reads, `package.json` lookups, real-path
resolution) is dispatched on the path being probed: paths under a
instead, every file system operation those algorithms perform is dispatched
on the path being probed: paths under a

These implementation details should not be enumerated in the docs.

Comment thread doc/api/vfs.md
Comment on lines +305 to +306
behave as first-class modules: `package.json` is honoured,
conditional [`"exports"`][] / [`"imports"`][] work, and so on.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
behave as first-class modules: `package.json` is honoured,
conditional [`"exports"`][] / [`"imports"`][] work, and so on.
behave as first-class modules.

These look like random hand-wavy examples that seem to be meaningful but actually clarify nothing again - if we have to explain with an example it's better to provide a snippet etc. to make it more concrete.

Comment thread doc/api/vfs.md
Comment on lines +316 to +318
root: `package.json` scope lookups (for [`"type"`][],
[`"exports"`][], [`"imports"`][], and the CommonJS directory
[`"main"`][]) and [loading from `node_modules` folders][] stop at

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
root: `package.json` scope lookups (for [`"type"`][],
[`"exports"`][], [`"imports"`][], and the CommonJS directory
[`"main"`][]) and [loading from `node_modules` folders][] stop at
root: `package.json` scope lookups and [loading from `node_modules` folders][] stop at

Again we should not enumerate them here. If we must, simply linking to the documentation in packages is better.

Comment thread doc/api/vfs.md
[`"exports"`][], [`"imports"`][], and the CommonJS directory
[`"main"`][]) and [loading from `node_modules` folders][] stop at
the mount point instead of continuing into the real file system
([the global folders][], such as `NODE_PATH`, are legacy CommonJS

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "instead of continuing into the real file system" part is leading users to spare a thought about a case that is not practical in the first place, because the mount point cannot actually have real files in the file system, no continuation can be performed even if Node.js wants to. I think it's clearer to simply list an example lookup, for example in /foo/bar/main.cjs, require('baz') should look up

- `$MOUNT_POINT/foo/bar/node_modules/baz`
- `$MOUNT_POINT/foo/node_modules/baz`
- `$MOUNT_POINT/node_modules/baz`
- If $NODE_PATH is specified, search folders listed in $NODE_PATH
- `$HOME/.node_modules/baz`
- `$HOME/.node_libraries/baz`
- `$PREFIX/lib/node/baz`

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll need to verify, but I think it's going to recursively to the root of the fs, e.g. down to /.

I would not change this specifically for the vfs.

return providerPath;
}

// ==================== FS Operations (Sync) ====================

@joyeecheung joyeecheung Jul 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it deleting JSDocs and other unrelated comments in this file?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll restore all the JSDoc

Comment thread doc/api/vfs.md Outdated
myVfs.unmount();
```

For ECMAScript modules, convert mounted paths to `file:` URLs before

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: I wonder if we should simply supply vfs.mountPointURL, then users don't have to convert it...another solution is to provide path methods like vfs.resolve/vfs.resolveAsURL, then users don't need to go with real path methods to build VFS paths. Also what actually happens under the URL conversion should be much simpler, because there won't be any poking (is it Windows? etc.), it's just simple concatenation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'll add vfs.mountPointURL, does not cost us anything and it's more ergonomic.

I'm not sure we should add resolve, it's already super-heavy with code and we have that functionality already.

Comment thread doc/api/vfs.md Outdated
Comment thread doc/api/vfs.md
Comment on lines +375 to +378
Mounting and unmounting do not invalidate ESM modules that are
already executing. As with any other module-system teardown,
unmounting a VFS while the import graph below it is still loading is
the caller's responsibility to avoid.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Mounting and unmounting do not invalidate ESM modules that are
already executing. As with any other module-system teardown,
unmounting a VFS while the import graph below it is still loading is
the caller's responsibility to avoid.
Mounting and unmounting do not stop any module execution that is already started,
or invalidate any objects materialized from VFS modules that are already executed.
As with modules in the real file system, the callers are responsible of avoiding
removal or invalidation of modules in the virtual file system while they are being loaded.

You could technically unmount CJS while it is executing too (if you make the vfs object reference available e.g. with globals, then call unmount at the top level of a CJS module being loaded from that VFS).

Comment thread lib/internal/modules/helpers.js Outdated
Comment on lines +74 to +76
stat: (filename) => internalFsBinding.internalModuleStat(filename),
readFile: (filename, options) => fs.readFileSync(filename, options),
realpath: (requestPath) => fs.realpathSync(requestPath, {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
stat: (filename) => internalFsBinding.internalModuleStat(filename),
readFile: (filename, options) => fs.readFileSync(filename, options),
realpath: (requestPath) => fs.realpathSync(requestPath, {
internalModuleStat: (filename) => internalFsBinding.internalModuleStat(filename),
readFileSync: (filename, options) => fs.readFileSync(filename, options),
realpathSync: (requestPath) => fs.realpathSync(requestPath, {

we should use exact names to avoid ambiguity with other methods that actually use that name

Comment thread lib/internal/modules/helpers.js Outdated
};
}

const loaderStat = wrapLoaderMethod(nativeLoaderMethods.stat);

@joyeecheung joyeecheung Jul 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not just do something like

const wrappers = { __proto__: null };
const loaderMethodKeys = ObjectKeys(nativeLoaderMethods);
for (let i = 0; i < loaderMethodKeys.length; ++i) {
  const key = loaderMethodKeys[i];
  wrappers[key] = nativeLoaderMethods[key];
}

Then we can simply use the key to identify overrides, and setLoaderFsOverrides and setLoaderPackageOverrides can be merged into

setLoaderOverrides(overrides) {
  for (let i = 0; i < loaderMethodKeys.length; ++i) {
    const key = loaderMethodKeys[i];
    if (overrides[key]) {
      loaderOverrides.set(key, override);
    } else {
      loaderOverrides.delete(key);
    }
  }
  hasLoaderOverrides = loaderOverrides.size > 0;
}

And then we just exports the wrappers to other modules instead of enumerating them all in the export list below. This means in the future if anything is moved to C++ that leads to an overridable point, we no longer have to add like 4 lines of boilerplate below to hook it up, simply adding a key-value pair in nativeLoaderMethods is enough and everything else is automatic.

@mcollina

Copy link
Copy Markdown
Member Author

I am not sure if it's the AI talking (looks like it is) but this is now changing/deleting comments of code that aren't changed in this PR...

I told it to clean it up some long comments that were added previously and clean up the whole file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. vfs Issues and PRs related to the virtual filesystem subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants