Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google and Firebase - Unexpected end of JSON input #309

Closed
gastonmorixe opened this issue Jan 17, 2018 · 21 comments
Closed

Google and Firebase - Unexpected end of JSON input #309

gastonmorixe opened this issue Jan 17, 2018 · 21 comments

Comments

@gastonmorixe
Copy link

gastonmorixe commented Jan 17, 2018

This is a (Bug Report / Feature Proposal)

Bug

Description

For bug reports:

  • What went wrong?
⟩ serverless deploy

{}
Serverless: Bundling with Webpack...
Time: 637ms
   Asset     Size  Chunks             Chunk Names
index.js  4.32 kB       0  [emitted]  main
   [0] ./handler.js 1.57 kB {0} [built]
   [1] external "firebase-admin" 42 bytes {0} [not cacheable]
   [2] external "firebase-functions" 42 bytes {0} [not cacheable]

  Syntax Error -------------------------------------------

  Unexpected end of JSON input

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           9.3.0
     Serverless Version:     1.25.0


  • What did you expect should have happened?

Deployed correctly

  • What was the config you used?
service: wrud

provider:
  name: google
  runtime: nodejs
  project: XXXX
  credentials: ~/.gcloud/XXXX.json

plugins:
  - serverless-google-cloudfunctions
  - serverless-webpack

custom:
  webpackIncludeModules:
    packagePath: './package.json'    

# needs more granular excluding in production as only the serverless provider npm
# package should be excluded (and not the whole node_modules directory)
package:
  exclude:
    - node_modules/**
    - .gitignore
    - .git/**

functions:
  hello:
    handler: helloWorld
    events:
      - http: path
  • What stacktrace or error message from your provider did you see?
  Stack Trace --------------------------------------------

SyntaxError: Unexpected end of JSON input
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at BbPromise.try (/Users/work1/Projects/wrud/wrud/node_modules/serverless-webpack/lib/packExternalModules.js:236:47)
    at tryCatcher (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/method.js:39:29)
    at BbPromise.fromCallback.then.depJson (/Users/work1/Projects/wrud/wrud/node_modules/serverless-webpack/lib/packExternalModules.js:236:35)
    at tryCatcher (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/promise.js:638:18)
    at /Users/work1/Projects/wrud/wrud/node_modules/bluebird/js/release/nodeback.js:42:21
    at childProcess.exec (/Users/work1/Projects/wrud/wrud/node_modules/serverless-webpack/lib/packExternalModules.js:233:16)
    at ChildProcess.exithandler (child_process.js:279:5)
    at ChildProcess.emit (events.js:159:13)
    at maybeClose (internal/child_process.js:943:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:220:5)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:368:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:386:17)
    at ServerlessWebpack.BbPromise.bind.then.then.then (/Users/work1/Projects/wrud/wrud/node_modules/serverless-webpack/index.js:90:51)
    at runCallback (timers.js:773:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate [as _immediateCallback] (timers.js:711:5)

Additional Data

package.json

{
  "name": "google-nodejs",
  "version": "0.1.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "serverless.com",
  "license": "MIT",
  "dependencies": {
    "babel-runtime": "^6.26.0",
    "firebase-admin": "^5.8.0",
    "firebase-functions": "^0.8.0",
    "jsonschema": "^1.2.2",
    "serverless-google-cloudfunctions": "^1.1.1"
  },
  "devDependencies": {
    "babel-core": "^6.26.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-transform-runtime": "^6.23.0",
    "babel-preset-env": "^1.6.1",
    "babel-preset-stage-3": "^6.24.1",
    "eslint": "^4.15.0",
    "eslint-config-standard": "^11.0.0-beta.0",
    "eslint-plugin-import": "^2.8.0",
    "eslint-plugin-node": "^5.2.1",
    "eslint-plugin-promise": "^3.6.0",
    "eslint-plugin-standard": "^3.0.1",
    "serverless-webpack": "^4.2.0",
    "webpack": "^3.10.0",
    "webpack-node-externals": "^1.6.0"
  }
}
  • Serverless-Webpack Version you're using: 4.2.0
  • Webpack version you're using: 3.10.0
  • Serverless Framework Version you're using: 1.25.0
  • Operating System: OSX 10.13.2 (17C88) High Sierra
  • Stack Trace (if available):
@HyperBrain
Copy link
Member

HyperBrain commented Jan 17, 2018

Hi @gastonmorixe, thanks for reporting.

According to the stacktrace, the plugin crashed, because it got an invalid output from the npm ls -prod -json -depth=1 command execution.

Can you try to execute it manually and post the output, so that we can check what exactly is going wrong there?

@gastonmorixe
Copy link
Author

@gastonmorixe
Copy link
Author

@HyperBrain also


npm ERR! peer dep missing: firebase-admin@~5.6.0, required by [email protected]
npm ERR! extraneous: [email protected] /Users/work1/Projects/wrud/wrud/node_modules/node-pre-gyp

@gastonmorixe
Copy link
Author

@HyperBrain sorry to add more comments, it seems that with npm install instead of yarn I get this

⟩ npm install
npm WARN rm not removing /Users/work1/Projects/wrud/wrud/node_modules/.bin/pbjs as it wasn't installed by /Users/work1/Projects/wrud/wrud/node_modules/protobufjs
npm WARN rm not removing /Users/work1/Projects/wrud/wrud/node_modules/google-gax/node_modules/grpc/node_modules/.bin/pbjs as it wasn't installed by /Users/work1/Projects/wrud/wrud/node_modules/google-gax/node_modules/grpc/node_modules/protobufjs
npm WARN rm not removing /Users/work1/Projects/wrud/wrud/node_modules/.bin/semver as it wasn't installed by /Users/work1/Projects/wrud/wrud/node_modules/semver
npm WARN rm not removing /Users/work1/Projects/wrud/wrud/node_modules/@google-cloud/common-grpc/node_modules/grpc/node_modules/.bin/node-pre-gyp as it wasn't installed by /Users/work1/Projects/wrud/wrud/node_modules/@google-cloud/common-grpc/node_modules/grpc/node_modules/node-pre-gyp

> [email protected] install /Users/work1/Projects/wrud/wrud/node_modules/fsevents
> node install

[fsevents] Success: "/Users/work1/Projects/wrud/wrud/node_modules/fsevents/lib/binding/Release/node-v59-darwin-x64/fse.node" is installed via remote

> [email protected] install /Users/work1/Projects/wrud/wrud/node_modules/google-gax/node_modules/grpc
> node-pre-gyp install --fallback-to-build --library=static_library

sh: node-pre-gyp: command not found
npm WARN [email protected] requires a peer of firebase-admin@~5.6.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

npm ERR! file sh
npm ERR! code ELIFECYCLE
npm ERR! errno ENOENT
npm ERR! syscall spawn
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build --library=static_library`
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/work1/.npm/_logs/2018-01-17T21_00_21_979Z-debug.log
2018-01-17 18:00:22.277 osascript[75656:3819841] ApplePersistence=NO

@HyperBrain
Copy link
Member

Thanks a lot. It seems that exactly the same happens during execution of the plugin (there might be a bug in the plugin that prevents noticing the npm error and so the parser fails in the end).

I will try if I get some time tomorrow, so that I can prepare exactly the same package.json, so that I can try to find out why npm fails here. (currently the plugin is based entirely on npm internally). I'll let you know if I found out anything.

@gastonmorixe
Copy link
Author

I moved to firebase-cli tool. It seems even with all the dependencies solved, it still not working.

@alopes
Copy link

alopes commented Jan 21, 2018

Hey, I believe I might be having the same issue. Works fine running locally with $sls offline start but once I try to deploy it I get that same error. Any ideas on how to tackle this issue?
Thanks

➜  SLS_DEBUG=* serverless deploy
Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command webpack
Serverless: Load command dynamodb
Serverless: Load command dynamodb:migrate
Serverless: Load command dynamodb:seed
Serverless: Load command dynamodb:start
Serverless: Load command dynamodb:noStart
Serverless: Load command dynamodb:remove
Serverless: Load command dynamodb:install
Serverless: Load command offline
Serverless: Load command offline:start
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
Time: 874ms
     Asset     Size  Chunks             Chunk Names
handler.js  6.49 kB       0  [emitted]  handler
   [0] ./handler.js 3.65 kB {0} [built]
   [1] external "jwks-rsa" 42 bytes {0} [not cacheable]
   [2] external "jsonwebtoken" 42 bytes {0} [not cacheable]
   [3] external "babel-polyfill" 42 bytes {0} [not cacheable]
Time: 862ms
     Asset     Size  Chunks             Chunk Names
handler.js  6.49 kB       0  [emitted]  handler
   [0] ./handler.js 3.65 kB {0} [built]
   [1] external "jwks-rsa" 42 bytes {0} [not cacheable]
   [2] external "jsonwebtoken" 42 bytes {0} [not cacheable]
   [3] external "babel-polyfill" 42 bytes {0} [not cacheable]
Time: 860ms
     Asset     Size  Chunks             Chunk Names
handler.js  6.49 kB       0  [emitted]  handler
   [0] ./handler.js 3.65 kB {0} [built]
   [1] external "jwks-rsa" 42 bytes {0} [not cacheable]
   [2] external "jsonwebtoken" 42 bytes {0} [not cacheable]
   [3] external "babel-polyfill" 42 bytes {0} [not cacheable]
Serverless: Invoke webpack:package

  Syntax Error -------------------------------------------

  Unexpected end of JSON input

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

SyntaxError: Unexpected end of JSON input
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at BbPromise.try (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/lib/packExternalModules.js:236:47)
    at tryCatcher (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/method.js:39:29)
    at BbPromise.fromCallback.then.depJson (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/lib/packExternalModules.js:236:35)
    at tryCatcher (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/promise.js:638:18)
    at /Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/node_modules/bluebird/js/release/nodeback.js:42:21
    at childProcess.exec (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/lib/packExternalModules.js:233:16)
    at ChildProcess.exithandler (child_process.js:278:5)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:927:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
From previous event:
    at PluginManager.invoke (/Users/alopes/.nvm/versions/node/v8.6.0/lib/node_modules/serverless/lib/classes/PluginManager.js:368:22)
    at PluginManager.spawn (/Users/alopes/.nvm/versions/node/v8.6.0/lib/node_modules/serverless/lib/classes/PluginManager.js:386:17)
    at ServerlessWebpack.BbPromise.bind.then.then.then (/Users/alopes/Code/Clients/xx-redacted-xx/services/job-user-profiles/node_modules/serverless-webpack/index.js:90:51)
    at runCallback (timers.js:781:20)
    at tryOnImmediate (timers.js:743:5)
    at processImmediate [as _immediateCallback] (timers.js:714:5)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.6.0
     Serverless Version:     1.25.0

Offline version

✗ sls offline start
Serverless: Bundling with Webpack...
Time: 762ms
     Asset     Size  Chunks             Chunk Names
handler.js  6.49 kB       0  [emitted]  handler
   [0] ./handler.js 3.65 kB {0} [built]
   [1] external "jwks-rsa" 42 bytes {0} [not cacheable]
   [2] external "jsonwebtoken" 42 bytes {0} [not cacheable]
   [3] external "babel-polyfill" 42 bytes {0} [not cacheable]
Serverless: Watching for changes...
Dynamodb Local Started, Visit: http://localhost:8000/shell
Serverless: DynamoDB - created table jobs-user-profile-dev
Serverless: Starting Offline: dev/eu-west-2.

Serverless: Routes for customAuthorizer:
Serverless: (none)

Serverless: Routes for public:
Serverless: GET /public

Serverless: Routes for private:
Serverless: GET /private
Serverless: Configuring Authorization: private customAuthorizer

Serverless: Offline listening on http://localhost:3051

package.json

  "dependencies": {
    "app-root-path": "^2.0.1",
    "auth0": "^2.5.0",
    "babel-polyfill": "^6.26.0",
    "bluebird": "^3.4.6",
    "globule": "^1.2.0",
    "jsonwebtoken": "^7.4.1",
    "jwks-rsa": "^1.2.1",
    "require-dir": "^0.3.2",
    "serverless-dynamodb-local": "^0.2.27",
    "serverless-offline": "^3.16.0",
    "systemjs": "^0.20.19",
    "webpack": "^3.8.1",
    "webpack-node-externals": "^1.6.0"
  },
  "devDependencies": {
    "babel-cli": "^6.26.0",
    "babel-core": "^6.26.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-istanbul": "^4.1.5",
    "babel-plugin-transform-async-to-generator": "^6.24.1",
    "babel-plugin-transform-class-properties": "^6.24.1",
    "babel-preset-es2015": "^6.24.1",
    "babel-preset-stage-0": "^6.24.1",
    "babel-register": "^6.26.0",
    "chai": "^4.1.2",
    "commitizen": "^2.9.6",
    "cross-env": "^5.1.1",
    "cz-conventional-changelog": "^2.1.0",
    "dotenv": "^4.0.0",
    "eslint": "^4.12.0",
    "format-json": "^1.0.3",
    "mocha": "^4.0.1",
    "nyc": "^11.3.0",
    "rimraf": "^2.6.2",
    "semantic-release": "^8.2.0",
    "serverless": "^1.24.1",
    "serverless-dotenv-plugin": "^1.0.1",
    "serverless-webpack": "^4.1.0"
  }

@alopes
Copy link

alopes commented Jan 21, 2018

Looks like this is caused by using yarn.
By deleting node_modules and then using npm, it works fine.

@HyperBrain
Copy link
Member

Hi @alopes . Thanks for the information and verification.

I think, having #286 done would make these cases very rare, as the plugin can then be configured to the packager that the user normally uses in his environment.

@shwanton
Copy link

shwanton commented Feb 6, 2018

Just got caught by this while using yarn, there is section in the README about yarn that I missed.

@HyperBrain
Copy link
Member

@shwanton The section has been added just a few days ago ;-)

@samwiseduzer
Copy link

I"m having this problem without using yarn, with identical results to @alopes

@HyperBrain
Copy link
Member

@samwiseduzer Can you check your npm version and try to delete the npm cache (see npm cache --help)

@samwiseduzer
Copy link

Deleting and reinstalling my packages seemed to take care of this

forabi added a commit to forabi/track-performance that referenced this issue Feb 28, 2018
`nodegit` caused all sorts of issues, and it takes too long to compile whether locally or on CI.  Even worse, due to [a bug in `serverless-webpack`](serverless-heaven/serverless-webpack#309) we have to compile it twice, which takes over 15 minutes on Travis.

This PR drops `nodegit` and replaces it with a regular `git` binary [compiled specifically for AWS Lambda](https://github.com/pimterry/lambda-git).

This may or may not work!
forabi added a commit to forabi/track-performance that referenced this issue Feb 28, 2018
`nodegit` caused all sorts of issues, and it takes too long to compile whether locally or on CI.  Even worse, due to [a bug in `serverless-webpack`](serverless-heaven/serverless-webpack#309) we have to compile it twice, which takes over 15 minutes on Travis.

This PR drops `nodegit` and replaces it with a regular `git` binary [compiled specifically for AWS Lambda](https://github.com/pimterry/lambda-git).

This may or may not work!
forabi added a commit to hollowverse-archive/track-performance that referenced this issue Feb 28, 2018
`nodegit` caused all sorts of issues, and it takes too long to compile whether locally or on CI.  Even worse, due to [a bug in `serverless-webpack`](serverless-heaven/serverless-webpack#309) we have to compile it twice, which takes over 15 minutes on Travis.

This PR drops `nodegit` and replaces it with a regular `git` binary [compiled specifically for AWS Lambda](https://github.com/pimterry/lambda-git).

This may or may not work!
@samwiseduzer
Copy link

I ran into this problem again because of unmet peer dependencies (which I discovered after running "npm ls -prod -json -depth=1"). Fixing the peer dependency took care of the issue for me.

@HyperBrain
Copy link
Member

@alopes Yarn support is available in master now and will be part of the next release. Please give it a try and check if it now correctly handles yarn.lock and works with yarn as expected. The README in master contains the exact steps on how to configure it - and adjust the webpack plugin configuration in serverless.yml properly to the new format.

@HyperBrain
Copy link
Member

Update: Yarn support is just released with 5.1.0 💥

@HyperBrain
Copy link
Member

HyperBrain commented Mar 14, 2018

@gastonmorixe Is there any update for this issue on your side. In case you use yarn, you could try plugin version 5.1.0 and configure it to use yarn as packager (if you don't use it, you can try that too ;-) ).

@gastonmorixe
Copy link
Author

gastonmorixe commented Mar 14, 2018 via email

@HyperBrain
Copy link
Member

Ok. I'm closing the issue now. Feel free to continue here if needed 😄

@HyperBrain
Copy link
Member

@gastonmorixe @alopes I finally found the root cause for the original Unexpected end of JSON error. It was an improper error messaging and internally a buffer overflow of the stdio buffers when using child_process.exec. With #373 I switched the packager invocations to use child_process.spawn now. This function uses streams to transfer the stdio stream data to the parent process as opposed to exec which uses fixed buffers (that are broken in some environments).

The fix will be released soon with version 5.1.2. If you have some time, you can check out if the current master branch works for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants