Importing assets from templates

[3dos]

Hi @SudoCat . It's me again (yeah still using this loader :D ) Is there a way to use import statements or anything parsed by other webpack loaders ? (like, for example the url (asset.png) in css)

I tried the following with no success:

{% import '../images/app/favicon.png' as favicon %}

with the following error message:

Error: /project/path/node_modules/nunjucks/browser/nunjucks-slim.js?:955
                      if(err) { throw err; }
                                ^
  Template render error: (orbital.njk)
    Template render error: (orbital.njk)
    TypeError: path.dirname is not a function
  
  - nunjucks-slim.js?:183 Object.exports.prettifyError
    [.]/[nunjucks]/browser/nunjucks-slim.js?:183:16
  
  - nunjucks-slim.js?:943 eval
    [.]/[nunjucks]/browser/nunjucks-slim.js?:943:32
  
  - loader.js:120 new_cls.root [as rootRenderFunc]
    [orbital.njk?.]/[html-webpack-plugin]/lib/loader.js:120:3
  
  - nunjucks-slim.js?:936 new_cls.render
    [.]/[nunjucks]/browser/nunjucks-slim.js?:936:16
  
  - nunjucks-slim.js?:767 eval
    [.]/[nunjucks]/browser/nunjucks-slim.js?:767:36
  
  - nunjucks-slim.js?:689 createTemplate
    [.]/[nunjucks]/browser/nunjucks-slim.js?:689:26
  
  - nunjucks-slim.js?:704 handle
    [.]/[nunjucks]/browser/nunjucks-slim.js?:704:26
  
  - nunjucks-slim.js?:718 eval
    [.]/[nunjucks]/browser/nunjucks-slim.js?:718:22
  
  - nunjucks-slim.js?:356 next
    [.]/[nunjucks]/browser/nunjucks-slim.js?:356:14
  
  - nunjucks-slim.js?:363 Object.exports.asyncIter
    [.]/[nunjucks]/browser/nunjucks-slim.js?:363:6

Sorry if I look dumb with this but my knowledge of loaders is quite limited. Will try to take the time to read the docs about writing loaders so I could help you support this one!

[SudoCat]

Hey there.

I'm really sorry, but I actually have no idea 😖 I've never actually used image loaders or anything similar to really know how they work. I'll probably need to have a look through some other templating language loaders and see what they're doing.

I imagine that part of the problem is how sketchy this nunjucks loader actually is. Unfortunately, nunjucks really is not tailored towards being turned into a loader; I've had to do a whole lot of work arounds to get this working. It's possible that because of this webpack is failing to find the potential dependencies within the templates.

I've got quite a busy work day today, so I doubt I'll get a chance to look into this until later this evening. If you're interested in trying to solve it, that'd be amazing. I'd be happy to support you if you have any questions about the code or if there are any other ways I can help!

[3dos]

Thank you very much! I'll throw myself into the Webpack docs and hopefully will be able to help you out with this.

Have a nice day!

[SudoCat]

Awesome, good luck! :D

[3dos]

Hiya, I think I know how to do this (thanks to the docs) but first, can you explain a bit how's the code working?

In fact, I only use it with the HtmlWebpackPlugin so, for now, I only test this context but if you tell me more about the loader, I'll be more prepared to start coding this feature. (Sorry, I'm quite new to Webpack but I'm motivated :D )

Thanks in advance!

[SudoCat]

Sure thing! Right, so the code I have written only handles the HtmlWebpackPlugin situation. If you're targetting web, then it instead loads the other package, nunjucks-loader - this should mean that we don't need to worry about maintaining the functionality used for web builds.

Now then, as for what my code actually does. It's been some time since I wrote this, so apologies if I'm a little shaky it on it. It's a little convoluted due to some of the requirements of loaders and Nunjucks, Nunjucks really did not want to be used like this, so there's been quite a lot of workarounds to glue this all together.

To understand the loader, first you need to understand the requirements, and the issues.

In order for HtmlWebpackPlugin to be able to pass data to the templates, the loader needs to return a module which exports a render function. This render function would then take the data passed to it, and compile the the template into a text string, with the data inserted into the right place. This is the requirement for our loader.

Unfortunately, that's not so easy in Nunjucks. Other templating languages, such as EJS and Pug, have very simple functionality, and require no knowledge of the current context in order to compile. This means they can both use very simplified portable compile functions to return the string. Nunjucks, on the other hand, has support for things like extends (which is the main reason I love it!). This means that Nunjucks needs a whoooole lot more information to be able to compile its templates.

Luckily, Nunjucks has a partial solution to this; precompiled templates. Nunjucks allows you to "precompile" the templates, essentially working out all of the complicated bits first, then outputting a simplified version of the templates for real-time compilation at a later point. This is how Nunjucks is usually used for web.

In order to precompile the templates, you need to use the nunjucks node environment loader (https://github.com/mozilla/nunjucks/blob/master/src/node-loaders.js). However, when I tried use this inside of a webpack loader, I encountered a host of recursive dependency errors. This turned out to be because of the require statement on line 35 of the nunjucks node loader.

Because of this, I decided to clone the node-loader, and remove the undesirable code which was creating the problems. The result is the fs-loader.js (https://github.com/SudoCat/Nunjucks-Isomorphic-Loader/blob/master/src/fs-loader.js). This creates a loader suitable for precompiling templates from within a webpack loader. You can see this being used when creating the nunjucks environment at https://github.com/SudoCat/Nunjucks-Isomorphic-Loader/blob/master/src/node-loader.js#L46

The next problem I encountered was how precompiled templates work. They are intended to be used live on the front end of a website, compiling them with javascript when needed. Because of this, all of the precompiled templates get automatically stored in the global window object... which doesn't exist in node. This meant I needed to create a fake window object for nunjucks to store the templates in. This is what you see at https://github.com/SudoCat/Nunjucks-Isomorphic-Loader/blob/master/src/node-loader.js#L64

With the precompiled templates safely stored in the loader's export module, the module then creates a new nunjucks slim environment. This provides us with the function we need to finalise the compilation, which gets done at https://github.com/SudoCat/Nunjucks-Isomorphic-Loader/blob/master/src/node-loader.js#L74

Hopefully this makes sense. Please let me know if there's anything you want me to clarify!

In short the loader:

1. Generates precompiled nunjucks templates
2. Returns a function for HtmlWebpackPlugin to use to compile the precompiled templates.

[3dos]

Wow, thank you very much for taking the time to explain this to me! Will read this and start coding this weekend. Oh time, if only we could get more of that stuff :')

[SudoCat]

That's no problem, I should probably have it written down anyway, just so I don't forget 😆

Haha, you can say that again!

[3dos]

So, I gave this a try and for now I'm stuck with paths from HtmlWebpackPlugin. I chose the following syntax as it is valid nunjucks: {{ require('./asset.jpg') }} considering require as a macro.

In the loader, at line 50 I just changed this:

        this.addContextDependency(paths[0]);

        // get the template content
	var templateContent = fs.readFileSync(name, { encoding: 'utf8' });

	// Search for require macros and require assets
	templateAsString = templateContent.replace(
		/\{\{\s*require\([\'\"]{1}(.+)[\'\"]{1}\)\s*\}\}/,
		function (match, $1) {
			console.log('require(' + utils.stringifyRequest(this, utils.urlToRequest(path.relative(paths[0], $1))) + ');');
			return eval('require(' + utils.stringifyRequest(this, utils.urlToRequest(path.relative(paths[0], $1))) + ');');
		}
	);

	var precompiledTemplates = nunjucks.precompileString(templateAsString, {
		name: name,
		env: env,
		include: [/.*\.(njk|nunjucks|html|tpl|tmpl)$/]
	});

So basically, I just changed the nunjucks.precompile to nunjucks.precompileString and tried to update the require macros by the actual asset paths.

Here's the error I get

ERROR in ./node_modules/html-webpack-plugin/lib/loader.js!./test.njk
    Module build failed: Error: Cannot find module './asset.jpg'
        at Function.Module._resolveFilename (module.js:513:15)
        at Function.Module._load (module.js:463:25)
        at Module.require (module.js:556:17)
        at require (internal/module.js:11:18)
        at eval (eval at <anonymous> (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:60:11), <anonymous>:1:1)
        at /Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:60:11
        at String.replace (<anonymous>)
        at Object.module.exports (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:55:37)
        at Object.module.exports (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/index.js:16:44)

I'm not sure about how to generate the right require path and I'm stuck with it. I think, this could work outside of the HtmlWebpackPlugin but didn't tried it out.

I'll try to check this out again next week but for now, if you have any advice, let me know :)

[3dos]

Well, my bad. Here's the right way to require the needed asset:

// Search for require macros
templateAsString = templateContent.replace(
	/\{\{\s*require\([\'\"]{1}(.+)[\'\"]{1}\)\s*\}\}/,
	function (match, $1) {
		console.log(path.resolve(paths[0], $1));
		// return 'dum'
		return require(path.resolve(paths[0], $1));
	}
);

This gets the right path to the wanted asset but I still got a problem as it seems the file-loader isn't called in this context. I have the following in my tests webpack config:

// Images
{
  test: /\.(jpe*g|png)/,
  use: [
    {
      loader: 'file-loader',
      options: {
         name: '[path][hash:8].[ext]'
      }
    }
  ]
}

And get the following error (looks like it tries to load the binary as a JS module):

ERROR in ./node_modules/html-webpack-plugin/lib/loader.js!./test.njk
    Module build failed: /Users/gomoon/Documents/workspace/Sandbox/nunjucks-tests/asset.jpg:1
    (function (exports, require, module, __filename, __dirname) { ����
                                                                  ^
    
    SyntaxError: Invalid or unexpected token
        at createScript (vm.js:80:10)
        at Object.runInThisContext (vm.js:139:10)
        at Module._compile (module.js:576:28)
        at Object.Module._extensions..js (module.js:623:10)
        at Module.load (module.js:531:32)
        at tryModuleLoad (module.js:494:12)
        at Function.Module._load (module.js:486:3)
        at Module.require (module.js:556:17)
        at require (internal/module.js:11:18)
        at /Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:61:11
        at String.replace (<anonymous>)
        at Object.module.exports (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:56:37)
        at Object.module.exports (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/index.js:16:44)

So now I get the asset but it's not loader by the file-loader. I also tried this.resolve instead of require but got this error:

ERROR in ./node_modules/html-webpack-plugin/lib/loader.js!./test.njk
    Module build failed: TypeError: Cannot read property 'missing' of undefined
        at Resolver.resolve (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-tests/node_modules/enhanced-resolve/lib/Resolver.js:83:31)
        at Object.resolve (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-tests/node_modules/webpack/lib/NormalModule.js:133:14)
        at /Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:62:16
        at String.replace (<anonymous>)
        at Object.module.exports (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/src/node-loader.js:57:37)
        at Object.module.exports (/Users/gomoon/Documents/workspace/Sandbox/nunjucks-isomorphic-loader/index.js:16:44)

[andreyvolokitin]

The issue should really be about using nunjucks import functionality. I have the same error while trying to import macro from one .njk file into another: {% import './path/to/macro.njk' as macro %}

[andreyvolokitin]

Okay, I found the source of the error form this issue. You can't use a relative path inside nunjucks imports (probably a related bug in nunjucks repo). Instead you need to use an absolute path relative to query.root path from the loader options.

The other problem though is that webpack doesn't watch changes in these imported assets (probably because they are not in a "webpack land"/not handled by webpack require statements). But this is another issue