Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken output (error in mkdirp() of createOutputWriter.js) #92

Closed
Trainmaster opened this issue Jun 14, 2018 · 24 comments · Fixed by #114
Closed

Broken output (error in mkdirp() of createOutputWriter.js) #92

Trainmaster opened this issue Jun 14, 2018 · 24 comments · Fixed by #114

Comments

@Trainmaster
Copy link
Contributor

As of version 3.6.1 there is an error thrown when using this plugin in a webpack configuration running a gulp task with the help of webpack-stream. Here some information about my setup:

// package.json
"assets-webpack-plugin": "3.6.1",
"gulp": "4.0.0",
"webpack": "4.12.0",
"webpack-cli": "3.0.6",
"webpack-merge": "4.1.3",
"webpack-stream": "4.0.3",
// and some more packages ...
// gulpfile.js
const gulp = require('gulp');
const webpackStream = require('webpack-stream');
const webpack = require('webpack');
const webpackDevConfig = require('./webpack.dev.config');
const webpackProdConfig = require('./webpack.prod.config');

// some different code ...

gulp.task('bundle', () => {
  const webpackConfig = isProduction ? webpackProdConfig : webpackDevConfig;
  return gulp.src('./frontend/app.js')
    .pipe(webpackStream(webpackConfig, webpack))
    .pipe(gulp.dest(options.targetFolder));
});

And here the error:

Error in plugin "webpack-stream"
Message:
    invalid argument
Details:
    domain: [object Object]
    domainThrown: true

The error isn't helpful at all, so I started debugging. First I enabled the showStack flag in /node_modules/plugin-error/index.js. After that, the following stack trace is displayed:

Error
    at /home/xxx/node_modules/webpack-stream/index.js:146:32
    at finalCallback (/home/xxx/node_modules/webpack/lib/Compiler.js:157:39)
    at hooks.done.callAsync.err (/home/xxx/node_modules/webpack/lib/Compiler.js:206:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/home/xxx/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook [as _callAsync] (/home/xxx/node_modules/webpack/node_modules/tapable/lib/Hook.js:35:21)
    at emitRecords.err (/home/xxx/node_modules/webpack/lib/Compiler.js:204:22)
    at Compiler.emitRecords (/home/xxx/node_modules/webpack/lib/Compiler.js:319:39)
    at emitAssets.err (/home/xxx/node_modules/webpack/lib/Compiler.js:198:10)
    at hooks.afterEmit.callAsync.err (/home/xxx/node_modules/webpack/lib/Compiler.js:305:14)
    at _err1 (eval at create (/home/xxx/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:16:1)
    at /home/xxx/node_modules/webpack-stream/index.js:192:9
    at _err0 (eval at create (/home/xxx/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:12:1)
    at /home/xxx/node_modules/assets-webpack-plugin/index.js:125:9
    at /home/xxx/node_modules/assets-webpack-plugin/lib/output/createQueuedWriter.js:14:7
    at /home/xxx/node_modules/assets-webpack-plugin/lib/output/createOutputWriter.js:19:16
    at Immediate._onImmediate (/home/xxx/node_modules/memory-fs/lib/MemoryFileSystem.js:282:5)

Then I stumbled upon #90. Those changes were introduced with 3.6.1. So I tried the previous version 3.6.0 which worked without throwing an error.

In a next step I thought it would be useful to get some information about the data around the mkdirp() call. I added the following lines lib/output/createOutputWriter.js.

mkdirp(options.path, function (err) {
  console.log('options', options);
  console.log('fs', fs);
  console.log('err', err);
  // ...

Using version 3.6.0 the following is displayed afterwards:

options { path: '.',
  filename: 'webpack-assets.json',
  prettyPrint: false,
  update: false,
  fullPath: true,
  processOutput: [Function] }
fs { constants: 
   { O_RDONLY: 0,
     O_WRONLY: 1,
     O_RDWR: 2,
     S_IFMT: 61440,
     S_IFREG: 32768,
     S_IFDIR: 16384,
     S_IFCHR: 8192,
     S_IFBLK: 24576,
     S_IFIFO: 4096,
     S_IFLNK: 40960,
     S_IFSOCK: 49152,
     O_CREAT: 64,
     O_EXCL: 128,
     O_NOCTTY: 256,
     O_TRUNC: 512,
     O_APPEND: 1024,
     O_DIRECTORY: 65536,
     O_NOATIME: 262144,
     O_NOFOLLOW: 131072,
     O_SYNC: 1052672,
     O_DSYNC: 4096,
     O_DIRECT: 16384,
     O_NONBLOCK: 2048,
     S_IRWXU: 448,
     S_IRUSR: 256,
     S_IWUSR: 128,
     S_IXUSR: 64,
     S_IRWXG: 56,
     S_IRGRP: 32,
     S_IWGRP: 16,
     S_IXGRP: 8,
     S_IRWXO: 7,
     S_IROTH: 4,
     S_IWOTH: 2,
     S_IXOTH: 1,
     F_OK: 0,
     R_OK: 4,
     W_OK: 2,
     X_OK: 1,
     UV_FS_COPYFILE_EXCL: 1,
     COPYFILE_EXCL: 1 },
  Stats: [Function: Stats],
  F_OK: 0,
  R_OK: 4,
  W_OK: 2,
  X_OK: 1,
  access: [Function],
  accessSync: [Function],
  exists: [Function],
  existsSync: [Function],
  readFile: [Function],
  readFileSync: [Function],
  close: [Function],
  closeSync: [Function],
  open: [Function],
  openSync: [Function],
  read: [Function],
  readSync: [Function],
  write: [Function],
  writeSync: [Function],
  rename: [Function],
  renameSync: [Function],
  truncate: [Function],
  truncateSync: [Function],
  ftruncate: [Function],
  ftruncateSync: [Function],
  rmdir: [Function],
  rmdirSync: [Function],
  fdatasync: [Function],
  fdatasyncSync: [Function],
  fsync: [Function],
  fsyncSync: [Function],
  mkdir: [Function],
  mkdirSync: [Function],
  readdir: [Function],
  readdirSync: [Function],
  fstat: [Function],
  lstat: [Function],
  stat: [Function],
  fstatSync: [Function],
  lstatSync: [Function],
  statSync: [Function],
  readlink: [Function],
  readlinkSync: [Function],
  symlink: [Function],
  symlinkSync: [Function],
  link: [Function],
  linkSync: [Function],
  unlink: [Function],
  unlinkSync: [Function],
  fchmod: [Function],
  fchmodSync: [Function],
  chmod: [Function],
  chmodSync: [Function],
  fchown: [Function],
  fchownSync: [Function],
  chown: [Function],
  chownSync: [Function],
  _toUnixTimestamp: [Function: toUnixTimestamp],
  utimes: [Function],
  utimesSync: [Function],
  futimes: [Function],
  futimesSync: [Function],
  writeFile: [Function],
  writeFileSync: [Function],
  appendFile: [Function],
  appendFileSync: [Function],
  watch: [Function],
  watchFile: [Function],
  unwatchFile: [Function],
  realpathSync: [Function: realpathSync],
  realpath: [Function: realpath],
  mkdtemp: [Function],
  mkdtempSync: [Function],
  copyFile: [Function],
  copyFileSync: [Function],
  createReadStream: [Function],
  ReadStream: 
   { [Function: ReadStream]
     super_: 
      { [Function: Readable]
        ReadableState: [Function: ReadableState],
        super_: [Object],
        _fromList: [Function: fromList] } },
  FileReadStream: 
   { [Function: ReadStream]
     super_: 
      { [Function: Readable]
        ReadableState: [Function: ReadableState],
        super_: [Object],
        _fromList: [Function: fromList] } },
  createWriteStream: [Function],
  WriteStream: 
   { [Function: WriteStream]
     super_: { [Function: Writable] WritableState: [Function: WritableState], super_: [Object] } },
  FileWriteStream: 
   { [Function: WriteStream]
     super_: { [Function: Writable] WritableState: [Function: WritableState], super_: [Object] } } }
err null

But using version 3.6.1 the output is different:

options { path: '.',
  filename: 'webpack-assets.json',
  prettyPrint: false,
  update: false,
  fullPath: true,
  processOutput: [Function] }
fs MemoryFileSystem { data: { home: { '': true, 'xxx': [Object] } } }
err { Error: invalid argument
    at pathToArray (/home/xxx/node_modules/memory-fs/lib/MemoryFileSystem.js:44:10)
    at MemoryFileSystem.mkdirpSync (/home/xxx/node_modules/memory-fs/lib/MemoryFileSystem.js:139:13)
    at MemoryFileSystem.(anonymous function) [as mkdirp] (/home/xxx/node_modules/memory-fs/lib/MemoryFileSystem.js:279:34)
    at writeOutput (/home/xxx/node_modules/assets-webpack-plugin/lib/output/createOutputWriter.js:17:8)
    at AssetsWebpackPlugin.queuedWriter [as writer] (/home/xxx/node_modules/assets-webpack-plugin/lib/output/createQueuedWriter.js:29:7)
    at afterEmit (/home/xxx/node_modules/assets-webpack-plugin/index.js:121:12)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/home/xxx/node_modules/webpack/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:7:1)
    at AsyncSeriesHook.lazyCompileHook [as _callAsync] (/home/xxx/node_modules/webpack/node_modules/tapable/lib/Hook.js:35:21)
    at asyncLib.forEach.err (/home/xxx/node_modules/webpack/lib/Compiler.js:302:27)
    at done (/home/xxx/node_modules/neo-async/async.js:2809:11)
    at /home/xxx/node_modules/neo-async/async.js:2760:7
    at MemoryFileSystem.writeFile (/home/xxx/node_modules/memory-fs/lib/MemoryFileSystem.js:328:9)
    at writeOut (/home/xxx/node_modules/webpack/lib/Compiler.js:286:29)
    at asyncLib.forEach (/home/xxx/node_modules/webpack/lib/Compiler.js:296:7)
    at baseEach (/home/xxx/node_modules/neo-async/async.js:2371:9)
    at Object.each (/home/xxx/node_modules/neo-async/async.js:2798:9)
  code: 'EINVAL',
  errno: 18,
  message: 'invalid argument',
  path: '.' }

So do you have any idea how to fix this?

@ztoben
Copy link
Owner

ztoben commented Jun 14, 2018

@Kronuz, any ideas here?

@ztoben
Copy link
Owner

ztoben commented Jun 15, 2018

Try 3.8.0 and see if you still have the same problem.

@Trainmaster
Copy link
Contributor Author

@ztoben Fixed with 3.8.0, thanks!

@Kronuz
Copy link
Contributor

Kronuz commented Jun 18, 2018

@Trainmaster there are a few things to consider here:

First, you seem to be using some configuration that ends up in webpack compiling things in it's memory filesystem (which is why you received MemoryFileSystem in the fs). Perhaps you're using some development server or webpack-hot-middleware.

Secondly, when that happens the now reverted patch from #90 is no longer putting the assets available in the memory file system, so webpack will no longer be able to properly open the assets file (for example to get the assets file from the running server, as it won't exist inside the virtual file system in memory)

Finally, the error says "invalid argument" when passing the argument to pathToArray(), which receives path and has a value of "." . mkdirp(".") from "mkdirp" package always succeeds, but in MemoryFilesystem it throws Error: invalid argument. We need to know where that "." is coming from and/or try normalizing paths starting with "./" and "." to remove that part before passing it to mkdirp() as well as not passing empty paths (as such also raise the error).

I strongly suggest we dig deeper here, as of right now, the assets file is being written in the real file system, while the actual assets are not.

@Kronuz
Copy link
Contributor

Kronuz commented Jun 18, 2018

@ztoben, What I'd do is reapply the pull request and normalize the path, making it absolute, by using resolve():

var path = require("path");

...

    var absolutePath = path.resolve(options.path);
    fs.mkdirp(absolutePath, function (err) {
      if (err) {
        return next(error('Could not create output folder ' + absolutePath, err))
      }

      var outputPath = fs.join(absolutePath, options.filename)

      ...
    }

in here: https://github.com/kossnocorp/assets-webpack-plugin/blob/804aa0ed2c6d797ff61a29d84d6ceda7bbb5586d/lib/output/createOutputWriter.js#L17

Maybe @Trainmaster could help verifying that fixes it.

Edit: Apparently, paths should be absolute here, but they aren't, in this edit, I propose using resolve() to make the path absolute before passing to mkdirp().

@ztoben
Copy link
Owner

ztoben commented Jun 21, 2018

@Kronuz, @Trainmaster: I've published a new alpha build with the changes suggested. Would you mind trying it out and seeing if it works for your setups? The version is 3.8.4-alpha.0. Thanks!

@Kronuz
Copy link
Contributor

Kronuz commented Jun 22, 2018

We also need to check the internal memory filesystem content, to see if the assets are being created where they should.

@ztoben
Copy link
Owner

ztoben commented Jun 22, 2018

@Kronuz, I know you're probably busy but any chance you could get a PR in for that? This isn't exactly my forte.

@Trainmaster
Copy link
Contributor Author

@ztoben It's not working with 3.8.4-alpha.0.

@ztoben
Copy link
Owner

ztoben commented Jun 25, 2018

@Trainmaster, same error? If not could you post a stacktrace?

@Kronuz
Copy link
Contributor

Kronuz commented Jul 10, 2018

I'll be working on resolving this in 3.8.4-alpha.0... there must be something we're missing.

@ztoben, if you could post any additional information (stacktrace or whatever), that'd be very helpful.

@Kronuz
Copy link
Contributor

Kronuz commented Jul 10, 2018

@Trainmaster, do you have options.path set to "."? Supposedly, that must be an absolute path, I get this error in Webpack 4:

Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
 - configuration[0].output.path: The provided value "." is not an absolute path!
   -> The output directory as **absolute path** (required).

@Kronuz
Copy link
Contributor

Kronuz commented Jul 10, 2018

@ztoben, in this line here:

var absolutePath = path.resolve(options.path)

options.path should be absolute (from Webpack's requirement since v2.3.0)... so I'm not sure how it ends up being ".". So that absolutePath shouldn't be necessary; perhaps an error, like in: https://github.com/webpack/webpack-dev-middleware/blob/ac17265d942b48217887d091ed1ffd461740427f/lib/fs.js#L57

@Trainmaster, what is your actual webpack configuration? also, what Webpack version are you using?

@ztoben
Copy link
Owner

ztoben commented Jul 11, 2018

@Kronuz unfortunately I don't have a stacktrace as I'm unable to reproduce this issue. I was hoping @Trainmaster would be able to supply one. If you make any progress on this let me know.

@Kronuz
Copy link
Contributor

Kronuz commented Jul 11, 2018

@ztoben, the thing is the problem is only there when options.path is not absolute; but options.path should never be "." nor a relative path.

Webpack, since v2.3.0, only accepts absolute paths in the configuration and default is also ends up being absolute, so we don't need to do that path.resolve(). The only thing I can think of is @Trainmaster is using an old Webpack version, in which case I think we should also warn about that output.path being non-absolute and bail (the same way webpack-dev-middleware does it).

@ztoben
Copy link
Owner

ztoben commented Jul 11, 2018

From the OP, he's using webpack 4.12.0 so I'm not sure what the issue would be.

My only guess would be that it's being caused by webpack-stream? Maybe we could wrap this all in a flag so we can provide both functionalities. Thoughts, @Kronuz?

Edit: On top of that it looks like webpack-stream doesn't fully support webpack 4 yet so that may be part of the issue. shama/webpack-stream#180

@Kronuz
Copy link
Contributor

Kronuz commented Jul 11, 2018

It'd be useful to have a minimal example which fails (one with just a couple files and a mini configuration to reproduce the issue). @Trainmaster, willing to help us with that?

@Kronuz
Copy link
Contributor

Kronuz commented Jul 11, 2018

@ztoben, I created a gist with what we have, but this one is working fine... @Trainmaster, could you please check if anything else is missing so it fails?

Edit: This is the Gist: https://gist.github.com/Kronuz/ac406b1b43bcca7d44622a2db50f853b

@Trainmaster
Copy link
Contributor Author

@ztoben Sorry for the late reply. The stack trace with version 3.8.4-alpha.0 using the latest webpack version 4.16.0 is:

Error
    at /home/xxx/node_modules/webpack-stream/index.js:146:32
    at finalCallback (/home/xxx/node_modules/webpack/lib/Compiler.js:210:39)
    at hooks.done.callAsync.err (/home/xxx/node_modules/webpack/lib/Compiler.js:259:14)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/home/xxx/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:6:1)
    at AsyncSeriesHook.lazyCompileHook [as _callAsync] (/home/xxx/node_modules/tapable/lib/Hook.js:35:21)
    at emitRecords.err (/home/xxx/node_modules/webpack/lib/Compiler.js:257:22)
    at Compiler.emitRecords (/home/xxx/node_modules/webpack/lib/Compiler.js:372:39)
    at emitAssets.err (/home/xxx/node_modules/webpack/lib/Compiler.js:251:10)
    at hooks.afterEmit.callAsync.err (/home/xxx/node_modules/webpack/lib/Compiler.js:358:14)
    at _err1 (eval at create (/home/xxx/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:16:1)
    at /home/xxx/node_modules/webpack-stream/index.js:192:9
    at _err0 (eval at create (/home/xxx/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:12:1)
    at /home/xxx/node_modules/assets-webpack-plugin/dist/index.js:129:9
    at /home/xxx/node_modules/assets-webpack-plugin/dist/lib/output/createQueuedWriter.js:16:7
    at /home/xxx/node_modules/assets-webpack-plugin/dist/lib/output/createOutputWriter.js:51:18
    at Immediate._onImmediate (/home/xxx/node_modules/memory-fs/lib/MemoryFileSystem.js:303:5)

@Kronuz
The options.path property is not set. The webpack config looks like:

const webpack = require('webpack');
const path = require('path');
const AssetsPlugin = require('assets-webpack-plugin');
const { VueLoaderPlugin } = require('vue-loader');
const includePaths = require('./webpack.includePaths');

module.exports = {
  entry: {
    app: './app.js',
  },
  output: {
    filename: '[name]_[chunkhash].js',
    publicPath: './js/',
  },
  module: {
    rules: [
      {
        parser: {
          amd: false,
        },
      },
      {
        test: /\.js$/,
        loader: 'babel-loader',
        include: includePaths,
      },
      {
        test: /\.vue$/,
        loader: 'vue-loader',
        include: includePaths,
      },
      {
        test: /\.css$/,
        use: [
          'vue-style-loader',
          'css-loader',
        ],
      },
      {
        test: /\.scss$/,
        use: [
          {
            loader: 'vue-style-loader',
          },
          {
            loader: 'css-loader',
          },
          {
            loader: 'sass-loader',
          },
        ],
      },
      {
        test: require.resolve('js-cookie'),
        use: [{
          loader: 'expose-loader',
          options: 'Cookies',
        }],
      },
      {
        test: require.resolve('moment'),
        use: [{
          loader: 'expose-loader',
          options: 'moment',
        }],
      },
      {
        test: require.resolve('jquery'),
        use: [{
          loader: 'expose-loader',
          options: 'jQuery',
        }, {
          loader: 'expose-loader',
          options: '$',
        }],
      },
      {
        test: require.resolve('lodash'),
        use: [{
          loader: 'expose-loader',
          options: '_',
        }],
      },
    ],
  },
  optimization: {
    runtimeChunk: {
      name: 'manifest',
    },
    splitChunks: {
      chunks: 'initial',
      cacheGroups: {
        default: false,
        vendor: {
          test: /[\\/]node_modules[\\/]/,
          name: 'vendor',
        },
      },
    },
  },
  plugins: [
    new webpack.ContextReplacementPlugin(/moment[\\/]locale$/, /^\.\/(de|fr|en|es|it|ja|tr|zh-cn)$/),
    new webpack.NamedModulesPlugin(),
    new webpack.NamedChunksPlugin((chunk) => {
      if (chunk.name) {
        return chunk.name;
      }
      return Array.from(chunk.modulesIterable)
        .map(module => path.basename(module.request, '.js'))
        .join('_');
    }),
    new AssetsPlugin(),
    new VueLoaderPlugin(),
  ],
};

@Kronuz
Copy link
Contributor

Kronuz commented Jul 11, 2018

@Trainmaster, could you please check what’s missing from the Gist I published in my previous post, so it fails? I just can’t make it fail here. Are you running just gulp command? Because the memory file system seems like some plugin is enabling that (I’ve only seen webpack-dev-middleware and webpack-dev-server doing that)

@Trainmaster
Copy link
Contributor Author

@Kronuz I commented your Gist

@Kronuz
Copy link
Contributor

Kronuz commented Jul 31, 2018

@ztoben, I know where the problem is coming from:

path: '.',

Path is hardcoded there, as ".", but it should come from compiler.options.output.path instead (at the time where the plugin is applied)

So that line above (path: '.') should not exist, and line

could be something like:

self.options.path = self.options.path || compiler.options.output.path;

The thing is path must be absolute, and should default to the path from the compiler options output path.

@ztoben
Copy link
Owner

ztoben commented Jul 31, 2018

@Trainmaster, could you try 3.8.4-alpha.1 and let me know if it's fixed for you? Thanks!

@Trainmaster
Copy link
Contributor Author

@ztoben It's working with 3.8.4-alpha.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants