Increase webpack performance by up to 7x 🚀🚀🚀🌑

[purfectliterature]

Profiling environment

Profiled on	Specification
CPU	Intel(R) Core(TM) i7-8550U
RAM	16.0 GB total, 10.0 GB on WSL2
OS	Windows 10 21H2 on Ubuntu 20.04 on WSL2
Architecture	64-bit
Node	16.17.0
webpack	5.74.0
@babel-core	7.19.0
Environment	development

• Fresh builds mean foreman start.
• Rebuilds mean recompilations during hot module reloads.
• To compute rebuild times, two words were removed/replaced from a form.
• Timings were obtained from webpack's CLI output.
• Profiles were run by invoking the new foreman start -f Procfile.profile command that uses webpack's built-in ProgressPlugin. See 6d8967b.

Optimization results

Changes	Fresh builds	Rebuilds	Delta
Nothing	139.4 s (2.3 mins)	17.0 s	Reference 🐌🐌🐌
babel-loader + fork-ts-checker-webpack-plugin	70.1 s (1.2 mins)	5.7 s	2-3x faster 🚀
Ditto + cacheDirectory*	32.6 s	4.1 s	4x faster 🚀🚀
Ditto + MUI/lodash tree-shaking	27.9 s	2.4 s	5-7x faster 🚀🚀🚀

* after building once.

So, what's the main bottleneck?

TL;DR: ts-loader's built-in typechecking is extremely 🐌 slow 🐌. [+100% 🚀]

Setting ts-loader to transpileOnly: true and using fork-ts-checker-webpack-plugin for typechecking on a separate process yielded almost similar build times as the one with only babel-loader.

Interestingly, we have @babel/preset-typescript, but babel-loader is set to only run on .js(x) files, so essentially this preset did nothing. As of Babel 7, Babel can already natively handle TypeScript files, so technically using babel-loader only is sufficient for our project. As such, ts-loader is removed in favour for a leaner webpack build process. However, we do lose typechecking because Babel is only a transpiler; typechecking is done by tsc.

Microsoft outlined ways to circumvent this caveat, but the author ultimately decided to integrate babel-loader with fork-ts-checker-webpack-plugin to bring back true TypeScript typechecking, although it increased fresh builds by ~8-10s. The author believed typechecking is very crucial to our codebase, and the tradeoff is worth it. Anyway, the typechecking gets slow when there are many files to be checked, but that would not be the case in HMRs, thus rebuilds are still (mostly) unaffected by fork-ts-checker-webpack-plugin.

But, that's not enough!

Enabling Babel caching [+60% 🚀]

cacheDirectory is an option in babel-loader that allows for caching non-changing Babel-transpiled codes for future webpack builds. According to their report and the author's tests, webpack builds after 1-2 builds sped up by up to 60%. cacheCompression is disabled to further speed up Babel's transpilation.

Babel 🌳 tree-shaking for MUI and lodash exports [+10% 🚀]

The author noticed during webpack profiling that there was many babel transpilations done on all @mui/icons-material, some @mui/material-ui components, and lodash modules. This is mainly caused by named imports, e.g., import { Something } from 'something';. Path imports, e.g., import Something from 'something/Something'; is actually faster because not the whole library's modules are transpiled.

Note that this is different from webpack's tree-shaking, because webpack's tree-shaking is more focused on production bundle size, i.e., by removing unused exports (usedExports: true). Babel tree-shaking happens when we start webpack, so the lack of it contributed to long startup times.

The author applied babel-plugin-import to convert all MUI named imports to path imports as of 093f8d2 and babel-plugin-lodash to do the same for lodash as of
42d307d.

lodash-webpack-plugin was not installed, despite the advertised performance benefits, because it might lead to some unexpected behaviours of lodash functions, although so far we only use isNumber in scribing.js and debounce in form/fields/TextField.jsx.

We might consider lodash-es for smaller build sizes, people say.

Limiting css-loader, postcss-loader, and sass-loader [+2% 🚀]

• css-loader now only works on rc-slider/assets/index.css because apparently it is used by VideoPlayerSlider.jsx.
• postcss-loader now only works on theme/index.css that contains Tailwind's default directives, and custom utilities in the future, if any.
• sass-loader now excludes node_modules.

Additional webpack development optimisations [+1% 🚀]

The author removed webpack output path info to relieve garbage collection and disabled additional webpack chunk clean-ups in development.

Takeaways and additional notes

⏱️ A new command to profile webpack

There is now a separate webpack configuration for profiling, webpack.profile.js, that uses webpack's built-in ProgressPlugin. There is a new command to run Coursemology on profile mode. This command will run yarn build:profile on client. See 6d8967b.

foreman start -f Procfile.profile

Developers can now monitor the webpack build progress and see the time taken for plugins, loaders, etc. to process what files in our codebase. Note that running webpack on profile mode will be slower than on normal development, notably because of console outputs. The results you obtain may be an overestimation of what the development build times will be(, which is good anyway).

ℹ️ CSS and Sass loading should be on whitelist

We are lucky that only one CSS file needs to be transpiled, except Tailwind's index.css. Since we are transitioning to Tailwind, we should only use Tailwind's utilities, and if there are custom utilities needed, define them only in index.css.

If for some reasons we need to add CSS files, we should add it to include: in css-loader's options in webpack.common.js. The author did not do the same for sass-loader because we are still amidst React pages migration, and we might still have some Sass files around.

❗ Please do NOT add more loose CSS or Sass files moving forward.

ℹ️ Use evaluatable translations identifier for intl.formatMessage

Do not do intl.formatMessage({ ...translations.something }), which is weird 👿. You may face an error like the following:

[React Intl] Messages must be statically evaluate-able for extraction
[webpack-dev-server] ERROR in ./app/bundles/course/leaderboard/pages/LeaderboardIndex/index.tsx
Module build failed (from ./node_modules/babel-loader/lib/index.js):
SyntaxError: coursemology2/client/app/bundles/course/leaderboard/pages/LeaderboardIndex/index.tsx: [React Intl] Messages must be statically evaluate-able for extraction.
    at File.buildCodeFrameError (coursemology2/client/node_modules/@babel/core/lib/transformation/file/file.js:249:12)
    at NodePath.buildCodeFrameError (coursemology2/client/node_modules/@babel/traverse/lib/path/index.js:145:21)
    at evaluatePath (coursemology2/client/node_modules/babel-plugin-formatjs/utils.js:16:16)
    at getMessageDescriptorKey (coursemology2/client/node_modules/babel-plugin-formatjs/utils.js:22:12)
    at propPaths.reduce.id (coursemology2/client/node_modules/babel-plugin-formatjs/utils.js:42:21)
    at Array.reduce (<anonymous>)
    at createMessageDescriptor (coursemology2/client/node_modules/babel-plugin-formatjs/utils.js:41:22)
    at processMessageObject (coursemology2/client/node_modules/babel-plugin-formatjs/visitors/call-expression.js:47:68)
    at PluginPass.visitor (coursemology2/client/node_modules/babel-plugin-formatjs/visitors/call-expression.js:120:13)
    at newFn (coursemology2/client/node_modules/@babel/traverse/lib/visitors.js:181:21)

The author fixed this issue as of f924c1a. This issue was not caught because react-intl's translations extractor was handled by babel-plugin-formatjs, and our TypeScript files were transpiled by ts-loader.

ℹ️ Think twice before installing 📦 big packages and importing them 🤔

The big devils in our codebase is moment, lodash, Immutable.js, and MUI. MUI and lodash have been handled properly by tree-shaking above. The problem with these packages for development build startups is named imports. Essentially, if there is a way to optimize them, e.g., with Babel plugins, tree-shaking, etc., then sure, use named imports for best DX. Otherwise, for small, one-time-good-one things, consider path imports. Violate the abstraction principle, the author could not care less, so long HMR times are kept fast.

ℹ️ Consider using DllPlugin for webpack

We should move our vendors from chunk-splitting to DllPlugins. Dll-ing is basically moving chunks/parts of code that do not change very often so that they need only be built once and not anymore. This is good for vendor libraries or our own stable libraries. If we move our vendors to Dlls, then our startups will be astronomically faster because we need not repeatedly build recorderjs, fabric, etc. The author did not do this because no time leh.

🤔 OkAy, CoOl, bUt WhAT AbOuT ✨ esbuild ✨?

The author hereby declares his love ❤️ for Vue.js and Evan You, and so esbuild is cool, too. The author did not have time to test esbuild, but here are some pointers that he wishes to extend:

• esbuild is currently not yet v1.0.0.
• To integrate esbuild to our builds, the best is probably to replace babel-loader to esbuild-loader.
• esbuild-loader can help us with TypeScript and polyfills, and typechecking is handled separately anyway.
• But, we must give away our Babel plugins because Babel plugins are not compatible with esbuild-loader. We will lose MUI and lodash tree-shaking (should be a way, somehow), but most importantly, react-intl processing with babel-plugin-formatjs. See Support the esbuild plug-in system？ evanw/esbuild#111.
• At the same time, we cannot even use esbuild plugins, so it is pretty much a lose-lose situation.

If we are going to try esbuild-loader, which the author very much like to, we need to:

• stop complicating our Babel builds, because we will lose them.
• figure out how to do tree-shaking with esbuild-loader for MUI and lodash.
• move from react-intl to other (better) libraries, e.g., i18next, that has esbuild support or no reliance on Babel.
• stop complicating our webpack configurations and figure out deployment stuff, if we are going to move totally to esbuild.

---

Thanks for coming this far; hope you enjoy this PR 👉👈. Maybe I should start a dev.to blog.

[ekowidianto]

Awesome work and thanks for this!! @purfectliterature