HMR taking much longer with webpack@5 and next@canary

[PepijnSenders]

What version of Next.js are you using?

10.0.5-canary.9

What version of Node.js are you using?

v15.3.0

What browser are you using?

Chrome

What operating system are you using?

macOS

How are you deploying your application?

next start

Describe the Bug

In dev mode with webpack@5 HMR takes considerably longer than with webpack@4 for the same change and exactly the same codebase.

Expected Behavior

HMR being faster or at the same speed.

To Reproduce

Still need to isolate the issue, which is really hard because this page is around 700kB total!

The webpack@4 baseline uses the following versions:

├─ next@npm:10.0.3
│  └─ webpack@npm:4.44.1 (via npm:4.44.1)

The baseline refresh time of HMR is around a second:

Compared with doing the same action with webpack@5:

├─ next@npm:10.0.5-canary.9
│  └─ webpack@npm:5.11.1 (via npm:5.11.1)

This takes around 12 seconds (vs. 1 second with webpack@4) and there's an extra event as well:

{
	"event": "serverOnlyChanges",
	"pages": ["/legacy/plans"]
}

and the built event has some errors regarding anonymous arrow functions that didn't have anything to do with the change.

We initially thought this issue was related to the context issue #18917, but it's not fixed on the latest canary.

[Timer]

We need a full reproduction before we can take a look at this!

[PepijnSenders]

@Timer I just tested config.cache.type = 'memory'; which seems much faster:

If you have any recommendations in the meantime on how to isolate this issue, please let me know!

[PepijnSenders]

Was hoping the speed-measure-webpack-plugin would give some more insights, but it doesn't show any anomalies in the plugins/loaders that we use either:

Webpack 4:

 SMP  ⏱  
General output time took 0.143 secs

 SMP  ⏱  Plugins
WellKnownErrorsPlugin took 0.065 secs
NextJsRequireCacheHotReloader took 0.005 secs
DefinePlugin took 0.001 secs
ChunkNamesPlugin took 0 secs
NextJsSsrImportPlugin took 0 secs
PagesManifestPlugin took 0 secs

 SMP  ⏱  
General output time took 0.169 secs

 SMP  ⏱  Plugins
NextJsRequireCacheHotReloader took 0.006 secs
DefinePlugin took 0.001 secs
ReactFreshWebpackPlugin took 0 secs
ChunkNamesPlugin took 0 secs
DropClientPage took 0 secs
HotModuleReplacementPlugin took 0 secs
WellKnownErrorsPlugin took 0 secs
ReactLoadablePlugin took 0 secs
BuildManifestPlugin took 0 secs

Webpack 5 (cache.type = 'memory'):

 SMP  ⏱  
General output time took 8.28 secs

 SMP  ⏱  Plugins
WellKnownErrorsPlugin took 0.001 secs
PagesManifestPlugin took 0.001 secs
DefinePlugin took 0 secs
NextJsSsrImportPlugin took 0 secs
NextJsRequireCacheHotReloader took 0 secs

 SMP  ⏱  Loaders
next took 0.177 secs
  module count = 1





 SMP  ⏱  
General output time took 8.75 secs

 SMP  ⏱  Plugins
ReactLoadablePlugin took 0.008 secs
BuildManifestPlugin took 0.002 secs
NextJsRequireCacheHotReloader took 0.002 secs
ReactFreshWebpackPlugin took 0.001 secs
HotModuleReplacementPlugin took 0.001 secs
WellKnownErrorsPlugin took 0.001 secs
ProvidePlugin took 0 secs
DefinePlugin took 0 secs
DropClientPage took 0 secs

 SMP  ⏱  Loaders
@next/react-refresh-utils, and 
next took 0.382 secs
  module count = 1

Webpack 5 (cache.type = 'fs'):

 SMP  ⏱  
General output time took 8.46 secs

 SMP  ⏱  Plugins
NextJsRequireCacheHotReloader took 0.001 secs
DefinePlugin took 0 secs
NextJsSsrImportPlugin took 0 secs
WellKnownErrorsPlugin took 0 secs
PagesManifestPlugin took 0 secs

 SMP  ⏱  Loaders
next took 0.123 secs
  module count = 1

 SMP  ⏱  
General output time took 9.19 secs

 SMP  ⏱  Plugins
ReactLoadablePlugin took 0.006 secs
ProvidePlugin took 0.001 secs
WellKnownErrorsPlugin took 0.001 secs
BuildManifestPlugin took 0.001 secs
ReactFreshWebpackPlugin took 0 secs
DefinePlugin took 0 secs
DropClientPage took 0 secs
HotModuleReplacementPlugin took 0 secs
NextJsRequireCacheHotReloader took 0 secs

 SMP  ⏱  Loaders
@next/react-refresh-utils, and 
next took 0.432 secs
  module count = 1

[Timer]

I just tried switching from filesystem to memory and it did not yield a meaningful speedup. Can you please share your project so we can trace the slowdown?

[PepijnSenders]

It's not a public project, and yea the switch also didn't yield a meaningful speed difference when you look at the stats in #20760 (comment).

What kind of project are you testing this on? I'm afraid it might only show up when you have a significant amount of code that needs to be compiled.

[Timer]

We're testing it on vercel.com, so a large project!

[PepijnSenders]

I believe it, I will try some more and see if removing certain parts isolate the issue. When you try it on vercel.com there's no change at all between webpack 5 and 4 refresh times?

[PepijnSenders]

Some more after hour debugging:

If the initial page that you compile is a smaller page, the compile and refresh time is very fast, like with webpack@4. If you compile a bigger and slower page after and then go back to the smaller page, it is suddenly very slow to compile as well. Both cache types act the same way.

Going to see if I can find anything that gets stuck in the memory heap.

[Timer]

On vercel.com webpack 5 seems a bit faster than webpack 4 for us.

If the initial page that you compile is a smaller page, the compile and refresh time is very fast, like with webpack@4. If you compile a bigger and slower page after and then go back to the smaller page, it is suddenly very slow to compile as well. Both cache types act the same way.

This is a good observation. Could you define the size of a "big" page? Are you using a lot of app-code, or very heavy npm dependencies?

[PepijnSenders]

On vercel.com webpack 5 seems a bit faster than webpack 4 for us.

Seeing the same on fresh projects.

This is a good observation. Could you define the size of a "big" page? Are you using a lot of app-code, or very heavy npm dependencies?

This is from the build output:

├ λ /legacy/plans                              68.9 kB         509 kB

It's a lot of app code, lots of dynamic imports, and experiments.

[PepijnSenders]

I will check if I can see anything in heap snapshots tomorrow. It might be interesting to compare a heap snapshot of when a smaller page is still "fast", and then after going to a bigger page and it being very slow after. I'm afraid though that it's going to be full of in-memory eval strings. Do you have any other ideas where and how I could debug this?

[Timer]

509 kB is pretty large... it might be due to a ton of dynamic imports. Can you see if this reproduces on a "large page" that doesn't use import() or next/dynamic?

My theory is that dynamic imports are slowing down compilation.

[PepijnSenders]

@Timer Another find, when I don't clear the .next/cache the small pages will also stay slow.

[PepijnSenders]

Also from my investigation on the heap snapshots, it seems like a lot of timers are set but never removed: https://github.com/webpack/webpack/blob/master/lib/cache/IdleFileCachePlugin.js#L140. Will need to check some more if that's actually the case or if there a just a lot of them.

[Timer]

Could you elaborate on "don't clear" @PepijnSenders? You should never be clearing .next/cache.

[PepijnSenders]

@Timer sorry missed your response, I know you should never clear it but my hunch is that the new webpack is adding a lot unnecessary things to the cache making everything slower. So as a test I removed the .next/cache folder between runs and when I start with a clean project it's much faster (if I don't go to bigger pages). Thought that might be a hint in the right direction.

I also see that the ticket was added to iteration 16, do you need more from me?

[Timer]

Yes (still need more info), it was only added to the iteration for investigation, but we're blocked on a full reproduction.

Can you make a mock app with large pages that reproduces this?

[PepijnSenders]

@Timer Do you jump on calls with people maybe we can dive into it together? I think creating a mock repo would be a lot of work, which I can do if you don't do that.

[Timer]

Unfortunately a lot of our profiling is done with private tool/tooling that wouldn't make it easy to do over a call. A mock repo would be the best.

[Timer]

Can you share your next.config.js?

[PepijnSenders]

Interesting but I don't think it's specifically webpack 5 in this case, because when I force it to webpack 5 it works. When I set .future.webpack5 to true though in next.config.js the aliases are suddenly not defined anymore.

[cullylarson]

Doesn't .future.webpack5 force weback 5?

[PepijnSenders]

It does and it also does a couple of other things apparently which breaks resolve.alias in the webpack config.

[cullylarson]

Oh, I see what you mean. When you say "force it to webpack 5", you mean force webpack 5 without using the .future.webpack5 flag.

I wonder if .future.webpack5 also turns on native ES modules.

[bfunc]

I have an issue with auto update page. It stopped working (no updating triangle) on versions of next,js greater than 10. Even default installation from homepage does not not work. Constantly getting error net::ERR_INCOMPLETE_CHUNKED_ENCODING on _next/webpack-hmr?page=/ http request. Tried version 10.0.7 on two diffrent machines. My luck that version 10.0.0 works ok.

[PepijnSenders]

Will open a new issue for this one: https://github.com/PepijnSenders/nextjs-aliases-issue. View it here #23130

[vvo]

For what it's worth: I too experienced slower rebuild times in dev using webpack 5 vs webpack 4 (like a 2x slowdown in general).

For example on an index.js page of 1000 lines with ~20 imports (internal/external modules).

After breaking up the page into different components (multiple files, example: components/Home/Pricing components/Home/Testimonials), with each component being ~100 lines then the rebuild speed is now almost instant when I change Testimonials (vs ~5 seconds before: the time it takes to rebuild the whole index.js page).

tl;dr; rebuilding a complete landing page of 1000 lines will be slow: break it down!

[timneutkens]

@vvo are you using next-transpile-modules?

[vvo]

@timneutkens yes, for one module indeed, could it be the reason?

[timneutkens]

Yes for sure: martpie/next-transpile-modules#179

[vvo]

Thanks, @timneutkens, does the "break pages into external file components" advice still make sense you think?

Just removed next transpile module (not used on the page I am modifying) and on the big page it's better but still slower than updating a component part of the page.

Also, not on an M1 so I guess it could be even faster :D

[timneutkens]

does the "break pages into external file components" advice still make sense you think?

Depends, in general the difference between webpack 4/5 shouldn't be much. Breaking up the components makes things faster as there's less code to be parsed/compiled for a single line change update.

If the project is open source we can have a look at it as well (if not feel free to dm me on twitter) 👍

[timneutkens]

Hey @PepijnSenders, can this one be closed now? Afaik all issues around this were solved.

[PepijnSenders]

All solved and working great. Thanks for the good work!

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.