vfs: integrate with CJS and ESM module loaders - #63653
Conversation
|
Review requested:
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #63653 +/- ##
==========================================
+ Coverage 90.10% 90.13% +0.02%
==========================================
Files 752 752
Lines 252197 252868 +671
Branches 47440 47580 +140
==========================================
+ Hits 227245 227925 +680
+ Misses 16251 16246 -5
+ Partials 8701 8697 -4
🚀 New features to boost your workflow:
|
|
@joyeecheung take a look, should be easier to review. |
6321e08 to
51b033a
Compare
joyeecheung
left a comment
There was a problem hiding this comment.
A design question recently occurred to me: have we explored the versioning of the mounting?
what do you mean? You mean multiple vfs layers on top of each other? |
For the stacks to have some kind of version number/ID to identify the current status? BTW I just noticed that there's no mention of |
I did purge them when doing the splitting; I forgot to bring them back. I'll add them to this PR. |
No but we totally should. |
51b033a to
294a19c
Compare
|
@joyeecheung PTAL |
75bb5c7 to
99a5a5c
Compare
Co-authored-by: Joyee Cheung <joyeec9h3@gmail.com>
The microbenchmark drove 1000+ iterations to reach the optimizing tier, which is not representative of real workloads. Module-loading against a mounted VFS is the meaningful signal and can be measured with the existing benchmark harness.
Measures loading a large CJS or ESM module graph from a mounted VFS, relying on the unmount cache purge so every iteration is a cold load.
Co-authored-by: Joyee Cheung <joyeec9h3@gmail.com>
db320eb to
7f564e4
Compare
|
Can we get full CI on this? I‘d really like to avoid having this pick up more conflicts with main. Thanks! |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
@joyeecheung PTAL, I plan to land this on Monday/when I can get a clean CI. If there are more things to fix, could you please open an issue? |
Makes
require()andimportresolve files served bynode:vfs. Before this PR, mounted VFS files were only visible throughfs.*; the loaders went straight to the real filesystem.Design
vfs.mount()takes no arguments and returns the reserved absolute mount point of the instance:${os.devNull}/vfs/<layerId>(for example/dev/null/vfs/0).os.devNullis a character device on POSIX and a device-namespace path on Windows; neither can have child filesystem entries, so no real path can ever exist under this root.Everything about ownership is decidable from the path alone:
benchmark/vfs/bench-fs-dispatch.js) reports flat per-call latency across 1..10 mounted layers.realpathSyncof a VFS entry always resolves to another path under the same mount point, so cache entries hidden behind symlinks are captured by the prefix scan.file:URLs under the mount point;import(import.meta.resolve(x))re-hits the same module job.Two instances mounting simultaneously never collide (each gets its own
layer-<id>segment), so there is no overlap validation and no ordering hazard between mounts.Loader integration
Toggleable wrappers in the loaders. Null fast-path when no VFS is mounted; otherwise the VFS answers
stat/readFile/realpath/legacyMainResolve/getFormatOfExtensionlessFileand the fourpackage.jsonC++-binding calls.Module identity follows the path:
__filename,module.filename, andimport.meta.urlare the plain absolute path (orfile:URL) of the module under the mount point — no synthetic decorations.Review guide
Suggested reading order:
lib/internal/vfs/router.jsgetVfsRoot,getLayerRoot,getLayerIdFromPath.lib/internal/vfs/file_system.jsmount()returns the layer's reserved mount point.lib/internal/vfs/setup.jsfindVFS(O(1) lookup), fs handler, loader overrides with parity tosrc/node_modules.cc/src/node_file.cc, prefix-scan cache purge.lib/internal/modules/helpers.jsloader*wrappers,setLoaderFsOverrides/setLoaderPackageOverrides,purgeRealpathCacheForPrefix.lib/internal/modules/cjs/loader.jsstat()+ TS read routed through wrappers;purgeModuleCachesForPrefixfor unmount.lib/internal/modules/esm/resolve.jslegacyMainResolve+internalModuleStat+toRealPathrouted. No URL decoration.lib/internal/modules/esm/load.jsgetSourceSyncreads via the wrapper.lib/internal/modules/esm/get_format.jslib/internal/modules/package_json_reader.jspurgePackageJSONCacheForPrefix.lib/fs.jsstatSync/lstatSynchonourthrowIfNoEntry:falseon ENOENT from the VFS handler.doc/api/vfs.mdTests (all gated by
--experimental-vfs):test-vfs-mount,test-vfs-mount-errors,test-vfs-multi-mount,test-vfs-require,test-vfs-import,test-vfs-module-hooks,test-vfs-module-hooks-cleanup,test-vfs-package-json,test-vfs-package-json-cache,test-vfs-invalid-package-json,test-vfs-scoped-cache-purge,test-vfs-layer-id,test-vfs-layer-tag-prefix.Refs
The reserved-namespace design follows the "no interference with valid paths in the file system" requirement from the SEA VFS requirements doc.
Out of scope
SEA + VFS, overlay/stacking of multiple VFS layers under one prefix, migrating the C++
package_configs_cache, broader permission-model integration.