Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't work with "multiple files" import #4433

Closed
fatfisz opened this issue May 19, 2018 · 12 comments · Fixed by #4639
Closed

Doesn't work with "multiple files" import #4433

fatfisz opened this issue May 19, 2018 · 12 comments · Fixed by #4639

Comments

@fatfisz
Copy link
Contributor

fatfisz commented May 19, 2018

Describe the bug
This code causes the "handle-import" babel plugin to break:

import(
  /* webpackInclude: /\.docs\.js$/, webpackChunkName: "whatever-[request]" */
  `__cwd/${path}`,
)

To Reproduce

  1. Include the snippet above in a code.
  2. Receive Module build failed: TypeError: Cannot read property '0' of undefined error from node_modules\next\dist\server\build\babel\plugins\handle-import.js:35:30

Expected behavior
I'd expect Next.js to be able to handle this kind of import usage, as it's supported by webpack. The old way to do the same, namely require.context, is not breaking like this (though it's breaking in other ways, as I'll bescribe in "additional context").

System information

  • OS: Windows 10
  • Version of Next.js: 6.0.3

Additional context
Related issue: #2690

This problem is directly caused by the "handle-import" plugin expecting the first argument to be a string literal (which has the value property). Since in my case it's a template literal, it doesn't have a value, only expressions and quasis.


For a while now I've been trying to create a wrapper around Next.js that's able to work with dynamically imported files (a tool for documentation which includes all *.docs.js files in the project).

First of all, I can't use pages/ - I want documentation files to be close to the scripts they document. I want the original structure of the source to be reflected by the structure of documentation, so replicating the directory structure in two places (original and pages) seems like an unmaintainable idea.

That's how I arrived at a solution that uses import-all.macro. The problem with that is, it's powered by babel. So when, as a user, I'm adding a new documentation file, I'll need to stop the tool, purge the cache (since all imports are inlined in a file), and restart the tool. I'm losing the file-listening capabilities of webpack-dev-server, and I need to go around the limitations of babel caching.

That's how I arrived at a solution that uses require.context. It almost works! I can't get the SSR story correctly though - while the server renders the appropriate script, the chunk doesn't seem to be ready when dynamic is rendering the component. Actually, the chunk doesn't seem to be generated at all! That's why I switched to import, to try and see if I could get the chunk-creating machine going at least with the other way of importing. As it is now, I've spent several days on this, I'm spinning in place, and I'm determined to fix the problem.

@fatfisz fatfisz changed the title Doesn't work with multiple import Doesn't work with "multiple files" import May 19, 2018
@fatfisz
Copy link
Contributor Author

fatfisz commented May 19, 2018

There seems to be an existing solution: https://github.com/faceyspacey/react-universal-component together with https://github.com/faceyspacey/webpack-flush-chunks. I'll see if I can get them working with Next.js.

If they work alright would it be a viable option to replace next/dynamic with react-universal-component in Next.js? Or is it in the domain of Next.js plugins?

@fatfisz
Copy link
Contributor Author

fatfisz commented May 22, 2018

I managed to make everything work. I'm very excited, because for so many days I was hoping that "the next solution will finally work" and there was always something lacking. But to get to the final solution, I had to change Next.js in a few ways - I'd like to get those changes into the package, as I think they'll benefit not only me, but also Next.js in the long run.

TL;DR:

  • Fixing the condition in handle-import to ignore template literals is very important (the current behavior of just crashing with an unhelpful error message is very undesirable)
  • Fixing getAvailableChunks to remove only the last hyphen and what follows it is also very important (it's because of how Webpack constructs chunk names - I can't do much here)
  • Upgrading to Webpack 4 would help a tiny bit

Below is a comprehensive description of whats and whys.

The handle-import plugin

As far as I know, the only way to set chunk names for "context" imports is to use this construct:

import(
	/* "webpackChunkName": "chunks/[request]" */
	`__cwd/${path}.docs.js`,
)

__cwd is resolved to process.cwd(), which is not the same directory where Next.js is running from

This is quite inconvenient because of three things:

  • It's impossible to get a map of all paths from this, as I described here: Add require.context.keys-like feature to import webpack/webpack#7283
  • Because of that I have to retrieve the map of paths using require.context('__cwd', true, /\.docs\.js/, 'weak');, which forces me to use a hack (I'll describe it below)
  • Webpack docs describe syntax that's only available in v. 4, so I wasted some time on discovering that webpackChunkName has to be put in quotes in v. 3 (otherwise this just breaks and the error message is not helpful)

What I'd benefit here from (a bit) is to have Webpack upgraded to v. 4, but that's a small issue.

As an extra thing, I have to actually use something like this:

const marker = '|';
import(
	/* "webpackChunkName": "chunks/[request]" */
	`__cwd/${marker}${path}${marker}.docs.js`,
)

Here I'm using the fact that Webpack doesn't know what marker is at the build time, so it just looks up the same paths as before. Afterwards though, I can easily get the actual path by getting the one between the markers (the ${path} argument comes from require.context, which returns full paths, so the result might look something like this: ././path/to/file.docs.js.docs.js, and after marking it's ./|./path/to/file.docs.js|.docs.js).

Ok, but at this point everything's still failing, because of the bug in server/build/babel/plugins/handle-import.js. What I did to fix the situation was to change the first condition inside CallExpression visitor to:

if (path.node.callee.type === TYPE_IMPORT && path.node.arguments[0].type !== 'TemplateLiteral') {
  ...
}

This just makes the plugin ignore template literals. I'd like it very much to see this change added to Next.js soon - if Next.js is not doing anything meaningful with import(`foo${bar}`), then it could at least not explode.

The getAvailableChunks utility function

So at this point I've added a few things here and there to be able to import files dynamicly like a civilised human. I got to know a bit about Webpack internals, and I've added an extension that not only allows me to get the chunk name for any file in import(`foo${bar}`), but also sets the proper value of module.__webpackChunkName so that Next.js knows such and such chunk was used when rendering a page. I don't really need this to be added to Next.js, but I'm leaving this here; maybe it will be useful someday when context imports come to Next.js:

The big hack that solves all my chunk-mapping-related problems
const marker = JSON.stringify('|');
const getExtraLazy = chunkNameMap => `
var chunkNameMap = ${JSON.stringify(chunkNameMap, null, '\t')};

function webpackContextResolveChunk(req) {
  if (!map.hasOwnProperty(req)) {
    throw new Error("Cannot find module '" + req + "'.");
  }
  return chunkNameMap[map[req][1]];
};

function extraWebpackAsyncContext(req) {
  if (req.indexOf(${marker}) === -1) {
    throw new Error("Cannot handle unmarked module request '" + req + "'.");
  }

  var unmarkedReq = req.slice(req.indexOf(${marker}) + 1, req.lastIndexOf(${marker}));

  return webpackAsyncContext(unmarkedReq).then(function(module) {
    var chunkId = webpackContextResolveChunk(unmarkedReq);
    module.__webpackChunkName = chunkId;
    return module;
  })
};

module.exports = extraWebpackAsyncContext;
`;

function getChunkNameMap(module) {
  return fromPairs(flatten(
    module.blocks
      .filter(block => block.dependencies[0].module)
      .map(block => (block.chunks || []).map(chunk => [chunk.id, chunk.name.slice('chunks/'.length)])),
  ));
}

class ExtraLazyTemplatePlugin {
  apply(moduleTemplate) {
    moduleTemplate.plugin('module', (source, module) => {
      if (module.async !== 'lazy') {
        return source;
      }
      const chunkNameMap = getChunkNameMap(module);
      return new ConcatSource(source, getExtraLazy(chunkNameMap));
    });
  }
}

class ExtraLazyPlugin {
  apply(compiler) {
    compiler.plugin('compilation', compilation => {
      compilation.moduleTemplate.apply(new ExtraLazyTemplatePlugin());
    });
  }
}

I'm almost completely set, but I still see a flash of "Loading..." after opening the page (proper markup is sent because of SSR, then "Loading..." appears, then the page is rendered again). And the reason for that is that my chunks are still being ignored by Next.js. I tracked it down to the following line in getAvailableChunks:

const chunkName = filename.replace(/-.*/, '')

If I understand correctly, the goal of this line is to strip the hash preceeded by a hyphen. Unfortunately with my chunks named chunks/[request], I get a lot of hyphens in the name - Webpack changes all slashes to hyphens. But because we're interested only in the last hyphen and what comes after it, my suggested change is this:

const chunkName = filename.replace(/-[^-]*$/, '');

It correctly strips the file name of the last hyphen and the remaining part.


Sorry if the write up is too long. It was a long ride for me after all, and I needed to get this off my chest. My biggest hope right now is that maintainers will agree with two small fixes I suggested. I'm here to help and I'd gladly prepare the needed PRs.

@fatfisz
Copy link
Contributor Author

fatfisz commented May 23, 2018

@timneutkens I hope I'm not overstepping my bounds if I ask you to give me any feedback on this? Right now I could prepare PRs for changes that I suggested - I just want to make sure that the problems I highlighted are something that you'd also like to see fixed.

@timneutkens
Copy link
Member

timneutkens commented May 23, 2018

@fatfisz let me read up on this soon! it's quite a lot of research 😄, I've been thinking about making some changes to next/dynamic anyway 👍

@fatfisz
Copy link
Contributor Author

fatfisz commented May 31, 2018

Hi, could I just create said PRs? They'll be very small and easy to merge - I promise 🙂

I'd like to move forward with my project and while I can patch webpack to do what I need, Next.js is much harder to adjust (I'd need to release my own version, monkey patching is impossible).

I'm sure that those changes won't affect Next.js negatively regardless of what direction next/dynamic will take.

@timneutkens
Copy link
Member

Sure go ahead with creating a PR, I can review them and we have extensive tests for the feature anyway, so if you break something you'll know 😄

@fatfisz
Copy link
Contributor Author

fatfisz commented Jun 6, 2018

Just a heads up that both branches seem to be passing. I just updated them with the latest canary.

timneutkens pushed a commit that referenced this issue Jun 7, 2018
Fixes one of the problems described in #4433.

The old regexp was removing everything after a hyphen, so with a chunk name like so:

```
chunks/path-to-a-file-[hash].js
```

the saved chunk name was

```
chunks/path
```

This caused problems, because webpack by default changes `/` to `-` in chunk names generated e.g. by ``import(`foo/${bar}`)``.

After this change the chunk name will be

```
chunks/path-to-a-file
```
timneutkens pushed a commit that referenced this issue Jun 9, 2018
This fixes a missed bug introduced in #4510.

Because the regexp was `/-[^-]*/` and not `/-[^-]*$/`, a wrong part of the filename was being removed:

```
bad:
'foo-bar-0123456789abcdef-0123456789abcdef.js' -> 'foo-0123456789abcdef-0123456789abcdef.js'

good:
'foo-bar-0123456789abcdef-0123456789abcdef.js' -> 'foo-bar-0123456789abcdef'
```

By a stroke of luck this didn't affect the existing dynamically generated chunks. To prevent regression I've added unit tests for the function that generates the name.

Btw. in the original issue (#4433) I used the right regexp, I just used the wrong regexp in #4510.

cc @timneutkens
lependu pushed a commit to lependu/next.js that referenced this issue Jun 19, 2018
…el#4510)

Fixes one of the problems described in vercel#4433.

The old regexp was removing everything after a hyphen, so with a chunk name like so:

```
chunks/path-to-a-file-[hash].js
```

the saved chunk name was

```
chunks/path
```

This caused problems, because webpack by default changes `/` to `-` in chunk names generated e.g. by ``import(`foo/${bar}`)``.

After this change the chunk name will be

```
chunks/path-to-a-file
```
lependu pushed a commit to lependu/next.js that referenced this issue Jun 19, 2018
This fixes a missed bug introduced in vercel#4510.

Because the regexp was `/-[^-]*/` and not `/-[^-]*$/`, a wrong part of the filename was being removed:

```
bad:
'foo-bar-0123456789abcdef-0123456789abcdef.js' -> 'foo-0123456789abcdef-0123456789abcdef.js'

good:
'foo-bar-0123456789abcdef-0123456789abcdef.js' -> 'foo-bar-0123456789abcdef'
```

By a stroke of luck this didn't affect the existing dynamically generated chunks. To prevent regression I've added unit tests for the function that generates the name.

Btw. in the original issue (vercel#4433) I used the right regexp, I just used the wrong regexp in vercel#4510.

cc @timneutkens
@dharmilsan
Copy link

Looks like some of the changes that @fatfisz suggested have been merged. But dynamic imports (using dynamic paths) still don't work with NextJS v6.1.1

@timneutkens
Copy link
Member

Will be fixed with #4639

@fatfisz
Copy link
Contributor Author

fatfisz commented Jul 2, 2018

This sounds awesome - so you'll be dropping handle-import and using react-loadable instead?

@timneutkens
Copy link
Member

@fatfisz yep that's what I'm working on, works great so far 👍

@briankoey
Copy link

Replace the line with require(variable/template string), what is the consequence of doing this?

@lock lock bot locked as resolved and limited conversation to collaborators Jul 24, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants