webpack & google-cloud #1821

brandonmp · 2016-11-26T00:09:46Z

Environment details

OS: ubuntu 16.10
Node.js version: 4.3.2
npm version: 2.14.12
google-cloud-node version: 0.44

Steps to reproduce

I'm using a webpack setup to build a nodejs project, the dependencies of which include google-cloud

Webpack sort of loses its mind w/ google-cloud, though, so hoping someone has some insight on how to fix.

The first error, ironically, appeared to be a deeply nested aws-sdk dependency, b/c I guess aws-sdk doesn't play nicely w/ webpack. I was able to fix that by including a pre-packaged AWS in my webpack-config.js, which is here:

// webpack.config.js
var fs = require('fs');
var path = require("path");
var webpack = require('webpack')
var prePackagedAwsPath = path.join('aws-sdk', 'dist', 'aws-sdk');

var nodeModules = {};
fs.readdirSync('node_modules')
    .filter(function (x) {
        return ['.bin'].indexOf(x) === -1;
    })
    .forEach(function (mod) {
        nodeModules[mod] = 'commonjs ' + mod;
    });

module.exports = {
    entry: path.join(__dirname, '/src/index.js'),
    output: {
        path: path.join(__dirname, "/dist/"),
        filename: "[name].js"
    },
    resolve: {
        alias: {
            'aws-sdk': prePackagedAwsPath
        }
    },
    target: "node",
    externals: nodeModules,
    module: {
        loaders: [
            {
                test: /\.node/,
                loader: 'node'
            },
            {
                test: require.resolve(prePackagedAwsPath),
                loader: 'imports?window=>null'
            },
            {
                test: /\.js$/,
                exclude: /node_modules/,
                loader: 'babel'
            },
            {
                test: /\.json$/,
                loader: 'json'
            }
        ]
    }
};

Now, however, the problem lies with the JSONStream dependency. The file apparently includes a # character at the top of its index.js, and that's breaking the build process.

Here's the error:

ERROR in ./~/@wayhome/wayhome-utils/~/google-cloud/~/JSONStream/index.js
Module parse failed: /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/JSONStream/index.js Unexpected character '#' (1:0
)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected character '#' (1:0)
    at Parser.pp$4.raise (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2221:15)
    at Parser.pp$7.getTokenFromCode (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2756:10)
    at Parser.pp$7.readToken (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2477:17)
    at Parser.pp$7.nextToken (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2468:15)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:515:10)
    at Object.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:3098:39)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/lib/Parser.js:902:15)
    at DependenciesBlock.<anonymous> (/home/bmp/code/scraping/node_modules/webpack/lib/NormalModule.js:104:16)
    at DependenciesBlock.onModuleBuild (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
    at nextLoader (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
    at /home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
    at Storage.finished (/home/bmp/code/scraping/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
    at /home/bmp/code/scraping/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/@google-cloud/prediction/src/model.js 25:17-38

I'd normally call this a JSONStream issue and file it there, but google-cloud is throwing warnings everywhere (below) as well, so I assume there's some overarching issue w/ webpack.

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/hash-stream-validation/index.js
Module not found: Error: Cannot resolve module 'fast-crc32c' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/hash-s
tream-validation
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/hash-stream-validation/index.js 5:8-30

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/src/node/src/grpc_extension.js
Critical dependencies:
38:14-35 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/src/node/src/grpc_extension.js 38:14-35

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/bytebuffer/dist/bytebuffer-node.js
Module not found: Error: Cannot resolve module 'memcpy' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/bytebuffer/
dist
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/bytebuffer/dist/bytebuffer-node.js 29:38-55

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/pre-binding.js
Critical dependencies:
19:22-48 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/pre-binding.js 19:22-48
WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/versioning.js
Critical dependencies:
15:20-67 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/versioning.js 15:20-67

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/nw-pre-gyp/index.html
Module parse failed: /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node_modules/node-pre-gyp/lib/util/nw-pre-gy
p/index.html Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected token (1:0)
    at Parser.pp$4.raise (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2221:15)
    at Parser.pp.unexpected (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:603:10)
    at Parser.pp$3.parseExprAtom (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1822:12)
    at Parser.pp$3.parseExprSubscripts (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1715:21)
    at Parser.pp$3.parseMaybeUnary (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1692:19)
    at Parser.pp$3.parseExprOps (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1637:21)
    at Parser.pp$3.parseMaybeConditional (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1620:21)
    at Parser.pp$3.parseMaybeAssign (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1597:21)
    at Parser.pp$3.parseExpression (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1573:21)
    at Parser.pp$1.parseStatement (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:727:47)
    at Parser.pp$1.parseTopLevel (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:638:25)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:516:17)
    at Object.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:3098:39)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/lib/Parser.js:902:15)
    at DependenciesBlock.<anonymous> (/home/bmp/code/scraping/node_modules/webpack/lib/NormalModule.js:104:16)
    at DependenciesBlock.onModuleBuild (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
    at nextLoader (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
    at /home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
    at Storage.finished (/home/bmp/code/scraping/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
    at /home/bmp/code/scraping/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib ^\.\/.*$

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js
Module not found: Error: Cannot resolve module 'node-gyp' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node
_modules/node-pre-gyp/lib/util
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js 29:28-55

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js
Module not found: Error: Cannot resolve module 'npm' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node_modu
les/node-pre-gyp/lib/util
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js 39:27-49

The text was updated successfully, but these errors were encountered:

stephenplusplus · 2016-11-28T15:53:48Z

We haven't been testing against webpack, so I'm quite confident it would be pure luck if all of our dependencies would be compatible. I'm going to play around with it as time allows, but feel free to dig into it and google around for issues if you want to help out. After a quick search, I found this issue which might get you past the JSONStream error: webpack/webpack#2168

If you do dig into this, please keep me updated!

brandonmp · 2016-12-01T19:07:50Z

thanks @stephenplusplus ! that helped get around the !# error, though i've hit a few more I'll detail here for posterity. if anyone has ideas on how to address the unsolved problem, would be grateful :) note that this sort of stuff is a little outta my league, so some of these fixes may be suboptimal

!# error

error: wepback won't parse .js files that start with a !# arg
solution: install a special loader to handle

$ npm install shebang-loader

// in webpack.config.js, add:
module: {
  loaders: [
...
     {
      // several gcloud dependencies have !#, so we'll apply loader broadly
      test: /node_modules\/.*\.js$/,
      loaders: ['shebang', 'babel']
     }
...
  ]
}

source: webpack/webpack#2168

node-forge

error: node-forge throws Invalid left-hand side in assignment
solution: this one feels a bit janky. we'll minimize and sideload it into our webpack.config.js:

npm i node-forge
cd node_modules/node-forge
npm i
node node_modules\requirejs\bin\r.js -o minify.js optimize=none out=js/forge.bundle.js
// copy forge.bundle.js elsewhere if you like. mine is in src/webpack/

// webpack.config.js:
var forge = require('node-forge')

//then modify your babel-loader module loader to add forge.bundle.js, and add the file to `noParse`:
module: {
    loaders: [
...
          {
                test: /\.js$/,
                exclude: /(node_modules|forge\.bundle\.js)/,
                loader: 'babel-loader'
            }
...
    ],
     noParse: [
            path.resolve(__dirname, './whatever/path/you/chose/for/forge.bundle.js')
        ]

source: digitalbazaar/forge#198

node-pre-gyp

error: haven't researched this one yet. here's error text:

ERROR in ./~/@google-cloud/storage/~/@google-cloud/common/~/grpc/~/node-pre-gyp/~/tar/tar.js
Module build failed: SyntaxError: Invalid number (108:10)

  106 | // values for the mode field
  107 | var modes =
> 108 |   { suid: 04000 // set uid on extraction
      |           ^
  109 |   , sgid: 02000 // set gid on extraction
  110 |   , svtx: 01000 // set restricted deletion flag on dirs on extraction
  111 |   , uread:  0400

 @ ./~/@google-cloud/storage/~/@google-cloud/common/~/grpc/~/node-pre-gyp/lib/install.js 56:24-38

stephenplusplus · 2016-12-06T15:06:49Z

I'm going to close this issue, since we don't have any plans to support being run through webpack at this time. Please subscribe to #531 and if we end up carving out some time to spend towards this goal, you can follow along. Sorry I can't be of more help at this time.

brandonmp · 2016-12-07T04:28:10Z

thanks @stephenplusplus , though my interests were more on the-server-side--specifically for aws-lambda/google-cloud functions. webpack is pretty clutch in that workflow, so I'm hopeful the plan to support it changes in the near future.

mikelambert · 2016-12-13T17:49:15Z

@brandonmp , I believe the node-pre-gyp code would require a change:
{ suid: parseInt("4000", 8)
and so on.
(Or if node-pre-gyp is relying on ES6, which I'm not sure is safe...one could write 0o4000)

Seems webpack is enforcing the fact that 0400 is a non-standard way to write an octal number, which I'd classify as a bug, except that maybe node-pre-gyp is assuming it only ever runs in a node environment.

jonwoodring · 2017-02-25T19:52:31Z

Here's a webpack loader to get around the octal number issue:
https://www.npmjs.com/package/octal-number-loader
https://github.com/jonwoodring/octal-number-loader

olegccc · 2017-04-30T15:09:08Z

based on the comments above I assume the only safe way is to exclude this module from bundling using externals webpack option or webpack-node-externals module.

slootzky · 2017-06-08T13:51:19Z

any news with this issue? trying to use Google's pubsub from Azure functions & Firebase functions so we're using webpack which fails (currently at node-pre-gyp / pre-binding.js in this part :
if (!existsSync(package_json_path)) { throw new Error("package.json does not exist at " + package_json_path); }

gvillenave · 2018-01-19T05:00:17Z

I'm seeing the same error when building a React website with Firebase using Gatsby. Webpack is THE bundling standard nowadays, it seems quite odd that the Google Cloud Platform SDK would not be able to work with it...
Why was that issue closed in the first place?

yankeeinlondon · 2018-02-14T04:49:56Z

I've tried bundling with webpack, rollup, and browserify (all targeting node) and they all have problems with the @google-cloud/* libraries. Everything else works flawlessly.

dundas · 2018-04-10T02:17:11Z

I dont imagine anyone has come up with an answer for this. this bug is making me crazy right now. any work around?

ohabash · 2018-12-02T04:10:53Z

@dundas did you find a solution?

How do use firebase-admin in angular unversal server?

odedniv · 2019-03-10T21:02:16Z

Took me a while but I finally got it. In Google Cloud Functions you don't need webpack because you don't upload the function artifact with the dependencies, you upload it with package.json and GCF installs the dependencies itself.

It won't help AWS/Azure users, but at least I can finally use GCP resources from GCF...

tettoffensive · 2019-03-20T23:31:35Z

@brandonmp @dundas any solutions or workarounds?

I've been building an internal content management system with vue + webpack. I am using the firebase libraries for the database.

I just want to be able to upload an image to the storage. firebase.storage() isn't available in node. so then I read that I'm supposed to use google-cloud/storage, but that doesn't work either.

For simply adding const { Storage } = require('@google-cloud/storage');

index.js?1c59:45 Uncaught TypeError: Expected `input` to be a `Function` or `Object`, got `undefined`
    at module.exports (index.js?1c59:45)
    at eval (index.js?838e:22)
    at Object../node_modules/gtoken/build/src/index.js (app.js:3626)
    at __webpack_require__ (app.js:679)
    at fn (app.js:89)
    at eval (jwtclient.js?e0ab:26)
    at Object../node_modules/google-auth-library/build/src/auth/jwtclient.js (app.js:3153)
    at __webpack_require__ (app.js:679)
    at fn (app.js:89)
    at eval (googleauth.js?983a:37)

yankeeinlondon · 2019-10-07T02:47:29Z

I have added the following to externals:

module.exports = {
  // ...
  externals: [ 'aws-sdk', /@google-cloud/ ]
}

seems to do the trick.

stephenplusplus added help wanted core labels Nov 28, 2016

stephenplusplus mentioned this issue Dec 6, 2016

React Native? #1863

Closed

stephenplusplus closed this as completed Dec 6, 2016

derdrdirk mentioned this issue Jan 10, 2018

Webpack compatibility googleapis/nodejs-dialogflow#16

Closed

stephenplusplus mentioned this issue Feb 2, 2018

angular universal monitoring googleapis/nodejs-logging#58

Closed

stephenplusplus mentioned this issue May 8, 2018

Make it work with Angular googleapis/nodejs-speech#72

Closed

n-riesco mentioned this issue May 25, 2018

Support for Google BigQuery plotly/falcon#404

Open

stephenplusplus mentioned this issue Jul 30, 2018

Run Time Error- Can not find Module "Child_Process" googleapis/nodejs-speech#122

Closed

grempe mentioned this issue Jun 26, 2019

Errors with wrangler build and @google-cloud/pubsub cloudflare/wrangler-legacy#275

Closed

googleapis deleted a comment from JustinPlute Sep 2, 2019

JustinBeckwith added the web label Jun 25, 2020

JustinBeckwith assigned stephenplusplus Feb 1, 2021

eric-gt mentioned this issue Feb 18, 2021

Scripts for pulling one-time data from partner data sources needed in the Admin Dashboard UI curiouslearning/followthelearners#513

Closed

gssakash-SxT mentioned this issue May 10, 2023

Error: Google Analytics Library Issue with WebPack #4269

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

webpack & google-cloud #1821

webpack & google-cloud #1821

brandonmp commented Nov 26, 2016

stephenplusplus commented Nov 28, 2016

brandonmp commented Dec 1, 2016

stephenplusplus commented Dec 6, 2016

brandonmp commented Dec 7, 2016

mikelambert commented Dec 13, 2016 •

edited

Loading

jonwoodring commented Feb 25, 2017

olegccc commented Apr 30, 2017

slootzky commented Jun 8, 2017 •

edited

Loading

gvillenave commented Jan 19, 2018 •

edited

Loading

yankeeinlondon commented Feb 14, 2018

dundas commented Apr 10, 2018

ohabash commented Dec 2, 2018

odedniv commented Mar 10, 2019

tettoffensive commented Mar 20, 2019

yankeeinlondon commented Oct 7, 2019 •

edited

Loading

webpack & google-cloud #1821

webpack & google-cloud #1821

Comments

brandonmp commented Nov 26, 2016

Environment details

Steps to reproduce

stephenplusplus commented Nov 28, 2016

brandonmp commented Dec 1, 2016

!# error

node-forge

node-pre-gyp

stephenplusplus commented Dec 6, 2016

brandonmp commented Dec 7, 2016

mikelambert commented Dec 13, 2016 • edited Loading

jonwoodring commented Feb 25, 2017

olegccc commented Apr 30, 2017

slootzky commented Jun 8, 2017 • edited Loading

gvillenave commented Jan 19, 2018 • edited Loading

yankeeinlondon commented Feb 14, 2018

dundas commented Apr 10, 2018

ohabash commented Dec 2, 2018

odedniv commented Mar 10, 2019

tettoffensive commented Mar 20, 2019

yankeeinlondon commented Oct 7, 2019 • edited Loading

mikelambert commented Dec 13, 2016 •

edited

Loading

slootzky commented Jun 8, 2017 •

edited

Loading

gvillenave commented Jan 19, 2018 •

edited

Loading

yankeeinlondon commented Oct 7, 2019 •

edited

Loading