Creating an in-browser tool to deal with a node_modules file tree such as "running Webpack in browser" often requires essentially running "npm install" in browser. This benchmark is to compare that performance across alternatives, which is WebContainer (by StackBlitz), Nodebox (by CodeSandBox), and npm-in-browser.
I (Naru) built npm-in-browser as an open-source alternative way to run "npm install" in browser without relying on black-box runtime / CDN and wanted to see whether it's practical to be used for creating tools such as JS Bundle Lab.
In summary, the benchmark has shown that npm-in-browser is performant enough to be used to do "npm install" typical frontend packages such as react-dom. Interestingly, it was faster than WebContainer in this case. For heavy packages such as installing typescript, WebContainer was faster but npm-in-browser was not extremely slow. Note that Nodebox was always extremely faster than the others in all the settings. I believe this is because Nodebox is using highly optimized format to deliver npm packages via their own CDN and they also use their own algorithm of "npm install", which does not seem to be using the actual "npm" CLI.
Note that as of 2023-10-18, both WebContainer and Nodebox have some restrictions on its usage such as commercial usage and their frontend/backend source code is not public. npm-in-browser is just a small open-source library without commercial usage restrictions.
I ran the benchmark on using my MacBook Air (M2, 2022) under a high-speed internet connection, which should represent a typical environment at home in Japan.
The rules are the following:
I used Puppeteer to measure "cold startup time" and "warm startup time". For each way, the same browser instance loaded the same page 6 times in succession and the first one is considered as "cold startup time" and the remaining 5 are considered as "warm startup time". I repeated this 100 times to plot 100 points in "cold startup time" and 500 points in "warm startup time" for each way.
Note that I did not use network throttling / CPU throttling since I was not confident that it is working correctly for everything including Web Workers, WASM, and Service Workers.
npm-in-browser requires us to supply Node.js fs compatible instance and it turned out that the performance of this fs matters. In this benchmark, I used memfs and "custom-fs", which is a custom in-memory filesystem I built just to make npm-in-browser work. Note that this "custom-fs" is not battle-tested and it has many edge case bugs since I only implemented absolutely necessary parts. Also note that memfs is order of magnitude slower without a setImmediate polyfill. All the benchmarks using memfs here are using the polyfill.
In this case, it installs lightweight frontend packages essentially by running npm install react-dom@18.2.0 react@18.2.0 zod@3.22.2 framer-motion@10.16.4. This should represent a situation where we want to create tools like "frontend bundle size checker" such as JS Bundle Lab.
We see that for this case npm-in-browser performs better than WebContainer although its variance is higher.
Time to finish "npm install" since visiting the page, without browser cache. This should approximates the experience for first-time visitors.
Time to finish "npm install" since reloading the page. For the most of files, browser cache should be used. This should approximates the experience for revisitors.
In this case, it installs heavyweight build packages essentially by running npm install typescript@5.2.2 webpack@5.88.2 @babel/core@7.23.0 next@13.5.4. We might want to install this type of packages when we want to replicate Node.js build environment in a browser.
We see that Nodebox is extremely fast and npm-in-browser is slower than WebContainer. We also see that for npm-in-browser, replacing memfs with a custom simpler in-memory fs ("custom-fs") improved the performance.
Time to finish "npm install" since visiting the page, without browser cache. This should approximates the experience for first-time visitors.
Time to finish "npm install" since reloading the page. For the most of files, browser cache should be used. This should approximates the experience for revisitors.