Support custom packager scripts #344

HyperBrain · 2018-03-09T16:56:23Z

What did you implement:

Closes #342
Closes #343

How did you implement it:

Scripts given in the packagerOptions.scripts array will be executed by npm run or yarn run after the complete function modules have been installed. I added a new interface method runScripts to the packager interface and the created package.json now contain the scripts.

How can we verify it:

# serverless.yml
custom:
  webpack:
    packagerOptions:
      scripts:
        - echo "test 1"
        - echo "test 2"

should print the output on each function or service packaging. Of course the echo does not make sense but replace it with a script that you need (e.g. the one from the original issue).

However, you should always carefully think about the problem you want to solve with a custom script. In most cases a webpack plugin or loader can do the same in a more stable way ;-)

Todos:

Write tests
Write documentation
Fix linting errors
Make sure code coverage hasn't dropped
Provide verification config / commands / resources
Enable "Allow edits from maintainers" for this PR
Update the messages below

Is this ready for review?: YES
Is it a breaking change?: NO

pflugs30 · 2018-03-09T17:24:10Z

Testing this now. My first command test sls package --package my-lambda failed due to a STD_OUT: maxBuffer exceeded error. Re-running the test with the --verbose keyword in an attempt to get more information.

HyperBrain · 2018-03-09T17:33:18Z

maxbuffer exceeded means you need to set a bigger buffer for stdout:

custom:
  webpack:
    packExternalModulesMaxBuffer: 614400

You can try even bigger values until the error is gone. The reason is, that the script might return more than the 200k default output

pflugs30 · 2018-03-09T17:33:57Z

The command failed again. Here's a dump of my console with the full path of my project hidden as {BASE_PATH}. Looking at the package.json file still left in the

{BASE_PATH}>sls package --verbose --package new-sls-webpack-test
Serverless: Using configuration:
{
  "includeModules": true,
  "packagerOptions": {
    "scripts": [
      "npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux"
    ]
  },
  "webpackConfig": "webpack.config.js",
  "packager": "npm",
  "packExternalModulesMaxBuffer": 204800
}
Serverless: Removing {BASE_PATH}\.webpack
Serverless: Bundling with Webpack...
Time: 14684ms
     Asset     Size  Chunks             Chunk Names
handler.js  16.3 kB       0  [emitted]  main
   [0] external "babel-runtime/regenerator" 42 bytes {0} [not cacheable]
   [1] external "babel-runtime/helpers/asyncToGenerator" 42 bytes {0} [not cacheable]
   [2] ./handler.js 9.99 kB {0} [built]
   [3] external "babel-runtime/core-js/json/stringify" 42 bytes {0} [not cacheable]
   [4] ./toscaConfig.js 279 bytes {0} [built]
   [5] ./googleConfig.js 273 bytes {0} [built]
   [6] external "request-promise" 42 bytes {0} [not cacheable]
   [7] external "@google-cloud/pubsub" 42 bytes {0} [not cacheable]
   [8] ./functions.js 2.49 kB {0} [built]
   [9] external "babel-runtime/core-js/promise" 42 bytes {0} [not cacheable]
Serverless: Fetch dependency graph from {BASE_PATH}\package.json
Serverless: Adding explicit peers for dependency request-promise
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, request-promise@^4.2.2, request@^2.83.0, @google-cloud/pubsub@^0.16.4
Serverless: Package took [182083 ms]
Serverless: Adding explicit peers for dependency request-promise
Serverless: Copy modules: {BASE_PATH}\.webpack\service [83610 ms]
Serverless: Prune: {BASE_PATH}\.webpack\service [17148 ms]

  Error --------------------------------------------------

  stdout maxBuffer exceeded

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     win32
     Node Version:           6.10.0
     Serverless Version:     1.26.1

pflugs30 · 2018-03-09T17:34:25Z

oops, sorry. Messages crossed. Trying your suggestion now.

pflugs30 · 2018-03-09T18:00:13Z

I think I've gotten past the STDOUT error. Now I see evidence the script in my serverless.yml file ran, but apparently I copied the script wrong since the script encountered an error. See below if interested.

It also seems that the package script abandoned everything as-is. That's probably fine in this case to help debug any errors or see what happened. Eventually, it may be worth having some error handling options around script errors.

Output:

{BASE_PATH}>sls package --verbose --package new-sls-webpack-test2
Serverless: Using configuration:
{
  "packExternalModulesMaxBuffer": 1000000,
  "includeModules": true,
  "packagerOptions": {
    "scripts": [
      "npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux"
    ]
  },
  "webpackConfig": "webpack.config.js",
  "packager": "npm"
}
Serverless: Removing {BASE_PATH}\.webpack
Serverless: Bundling with Webpack...
Time: 10768ms
     Asset     Size  Chunks             Chunk Names
handler.js  16.3 kB       0  [emitted]  main
   [0] external "babel-runtime/regenerator" 42 bytes {0} [not cacheable]
   [1] external "babel-runtime/helpers/asyncToGenerator" 42 bytes {0} [not cacheable]
   [2] ./handler.js 9.99 kB {0} [built]
   [3] external "babel-runtime/core-js/json/stringify" 42 bytes {0} [not cacheable]
   [4] ./toscaConfig.js 279 bytes {0} [built]
   [5] ./googleConfig.js 273 bytes {0} [built]
   [6] external "request-promise" 42 bytes {0} [not cacheable]
   [7] external "@google-cloud/pubsub" 42 bytes {0} [not cacheable]
   [8] ./functions.js 2.49 kB {0} [built]
   [9] external "babel-runtime/core-js/promise" 42 bytes {0} [not cacheable]
Serverless: Fetch dependency graph from {BASE_PATH}\package.json
Serverless: Adding explicit peers for dependency request-promise
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, request-promise@^4.2.2, request@^2.83.0, @google-cloud/pubsub@^0.16.4
Serverless: Package took [181991 ms]
Serverless: Adding explicit peers for dependency request-promise
Serverless: Copy modules: {BASE_PATH}\.webpack\service [118329 ms]
Serverless: Prune: {BASE_PATH}\.webpack\service [57344 ms]

  Error --------------------------------------------------

  Command failed: npm run script0
node-pre-gyp ERR! Tried to download(403): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v48-linux-x64-unknown.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v48 ABI, unknown) (falling back to source compile with node-gyp)
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit ({USER_PATH}\AppData\Roaming\nvm\v6.10.0\node_modules\npm\node_modules\node-gyp\lib\build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Windows_NT 10.0.16299
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "{USER_PATH}\\AppData\\Roaming\\nvm\\v6.10.0\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module={USER_PATH}\\Documents\\Code\\Excelion\\tosca\\.webpack\\service\\node_modules\\grpc\\src\\node\\extension_binary\\node-v48-linux-x64-unknown\\grpc_node.node" "--module_name=grpc_node" "--module_path={USER_PATH}\\Documents\\Code\\Excelion\\tosca\\.webpack\\service\\node_modules\\grpc\\src\\node\\extension_binary\\node-v48-linux-x64-unknown"
gyp ERR! cwd {BASE_PATH}\.webpack\service\node_modules\grpc
gyp ERR! node -v v6.10.0
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd build --fallback-to-build --library=static_library --module={BASE_PATH}\.webpack\service\node_modules\grpc\src\node\extension_binary\node-v48-linux-x64-unknown\grpc_node.node --module_name=grpc_node --module_path={BASE_PATH}\.webpack\service\node_modules\grpc\src\node\extension_binary\node-v48-linux-x64-unknown' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> ({BASE_PATH}\.webpack\service\node_modules\grpc\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack     at emitTwo (events.js:106:13)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:877:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
node-pre-gyp ERR! System Windows_NT 10.0.16299
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "{USER_PATH}\\Documents\\Code\\Excelion\\tosca\\.webpack\\service\\node_modules\\grpc\\node_modules\\node-pre-gyp\\bin\\node-pre-gyp" "install" "--fallback-to-build" "--library=static_library"
node-pre-gyp ERR! cwd {BASE_PATH}\.webpack\service\node_modules\grpc
node-pre-gyp ERR! node -v v6.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.39
node-pre-gyp ERR! not ok

npm ERR! Windows_NT 10.0.16299
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "rebuild" "grpc" "--target=6.1.0" "--target_arch=x64" "--target_platform=linux"
npm ERR! node v6.10.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build --library=static_library`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-pre-gyp install --fallback-to-build --library=static_library'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the grpc package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-pre-gyp install --fallback-to-build --library=static_library
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs grpc
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls grpc
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     {BASE_PATH}\.webpack\service\npm-debug.log

npm ERR! Windows_NT 10.0.16299
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "script0"
npm ERR! node v6.10.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] script0: `npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] script0 script 'npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the tosca-container-tracking package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs tosca-container-tracking
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls tosca-container-tracking
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     {BASE_PATH}\.webpack\service\npm-debug.log


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     win32
     Node Version:           6.10.0
     Serverless Version:     1.26.1

HyperBrain · 2018-03-09T18:07:22Z

This shows that the script execution works. I think now there's only a parameter error left for the script itself. The guy in the issue you referenced had the same problem, that some download did not work.

I'm quite confident that the plugin itself will now do its work. I fully agree with some improved error handling and will add something to make it better (maybe emitting the error and cleaning up the directories should be fine)

HyperBrain · 2018-03-09T18:09:01Z

@pflugs30 [email protected] and [email protected] (node-v48 ABI, unknown) can you check on their site which native images are available as precompiled ones?

I'm sure that if you set the correct node version in the params (maybe linux is missing), it will work.

pflugs30 · 2018-03-09T18:10:28Z

yes, I'd forgotten to copy one of the parameters for the build. The full command SHOULD have been:
npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc

Running the package command again now. It's amazing how long this takes for this (seemingly simple) Lambda function. These four dependencies I have branch out a LOT.

HyperBrain · 2018-03-09T18:19:37Z

BTW: Now with the scripts possible, I'll probably also increase the max buffer default. It is very likely that scripts output more than 200k of data to the console.

pflugs30 · 2018-03-09T18:34:38Z

Agreed. Here's my current serverless.yml file:

custom:
  webpack:
    packExternalModulesMaxBuffer: 1000000
    includeModules: true # enable auto-packing of external modules
    packagerOptions:
      scripts:
        - npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc

My most recent test with the corrected command completed successfully, including running the script. Here's the new log:

{BASE_PATH}>sls package --verbose --package new-sls-webpack-test3
Serverless: Using configuration:
{
  "packExternalModulesMaxBuffer": 1000000,
  "includeModules": true,
  "packagerOptions": {
    "scripts": [
      "npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc"
    ]
  },
  "webpackConfig": "webpack.config.js",
  "packager": "npm"
}
Serverless: Removing {BASE_PATH}\.webpack
Serverless: Bundling with Webpack...
Time: 31529ms
     Asset     Size  Chunks             Chunk Names
handler.js  16.3 kB       0  [emitted]  main
   [0] external "babel-runtime/regenerator" 42 bytes {0} [not cacheable]
   [1] external "babel-runtime/helpers/asyncToGenerator" 42 bytes {0} [not cacheable]
   [2] ./handler.js 9.99 kB {0} [built]
   [3] external "babel-runtime/core-js/json/stringify" 42 bytes {0} [not cacheable]
   [4] ./toscaConfig.js 279 bytes {0} [built]
   [5] ./googleConfig.js 273 bytes {0} [built]
   [6] external "request-promise" 42 bytes {0} [not cacheable]
   [7] external "@google-cloud/pubsub" 42 bytes {0} [not cacheable]
   [8] ./functions.js 2.49 kB {0} [built]
   [9] external "babel-runtime/core-js/promise" 42 bytes {0} [not cacheable]
Serverless: Fetch dependency graph from {BASE_PATH}\package.json
Serverless: Adding explicit peers for dependency request-promise
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, request-promise@^4.2.2, request@^2.83.0, @google-cloud/pubsub@^0.16.4
Serverless: Package took [334282 ms]
Serverless: Adding explicit peers for dependency request-promise
Serverless: Copy modules: {BASE_PATH}\.webpack\service [109501 ms]
Serverless: Prune: {BASE_PATH}\.webpack\service [23829 ms]
Serverless: Run scripts: {BASE_PATH}\.webpack\service [26009 ms]
Serverless: Zip service: {BASE_PATH}\.webpack\service [19935 ms]
Serverless: Packaging service...
Serverless: Remove {BASE_PATH}\.webpack

I also opened the ZIP file in the package folder, and I see the Linux version of the binary file listed, meaning the script did what I wanted it to do. Note: the Windows version is also there, but that's because I never executed a script to remove it.

I'm now going to test a deployment of the Lambda function, and I do expect that would work.

pflugs30 · 2018-03-09T18:36:03Z

I was able to successfully deploy the package to AWS using sls deploy, and my curl test of the method shows that it was invoked correctly. I have a new error, but it's unrelated to this topic. So, I'm bummed it didn't "just work", but I'm thrilled with your help, @HyperBrain, on getting past this specific issue. Much kudos and thanks!

HyperBrain · 2018-03-09T18:37:59Z

@pflugs30 I'm happy to hear it works. I'll finalize the PR tomorrow and add some documentation to the README. The merge and release will be done soon then. And thanks for testing 👍

HyperBrain · 2018-03-12T10:40:15Z

The error handling should be handled by a separate issue. On other errors (anywhere else) the .webpack folder is not deleted too. So that is a general one.
Having a different issue for this also let's us discuss if a cleanup would probably make the investigation of issues more difficult.

rgandco · 2018-03-15T18:07:18Z

@HyperBrain Thanks for this piece of work, however for my particular set-up I have had to force myself to use webpack 3.11.0 because of an issue where babel-runtime was not correctly being picked up as a dependency.

Full post here:
sst/guide#112 (comment)

Appreciate it's slightly off-topic, but any ideas on how to solve that one, so that I can use your fix above to get my grpc deployment working?

HyperBrain · 2018-03-15T18:17:59Z

@rgandco Right now I have no concrete idea. It could be that Webpack 4's new optimization algorithms and tree shaking now starts to add partial dependencies. You see that in the "external" log. It does not report babel-runtime as dependency (like Webpack 3 did) but babel-runtime/core-js/json/stringify for example.
If you can confirm that this is the difference between Webpack 3 and Webpack 4 using serverless-webpack, it is a bug in the Webpack 4 support of the plugin. In this case, can you create a bug here and I'll try to fix it properly.

However the forceInclude should be a proper workaround until it is fixed.

HyperBrain · 2018-03-15T19:40:04Z

Can you additionally post your package.json here? I'd like to check if prod/dev dependencies are correct. I just checked the code and the dependency detection seems to be ok in general.

rgandco · 2018-03-15T19:40:55Z

@HyperBrain Apologies - I totally missed in my haze of the day that I hadn't updated serverless-webpack to 5.1.0 - with webpack 3.11.0 - the fix above now works for me as well, much appreciated.

I'll have a think about Webpack 4 and get back to you.

rgandco · 2018-03-15T22:19:49Z

@HyperBrain Package.json as it stands now - which works, but as mentioned doesn't work if I upgrade webpack.

…rt-scripts Support custom packager scripts

HyperBrain added this to the 5.1.0 milestone Mar 9, 2018

This was referenced Mar 9, 2018

Need to bundle specific gRPC binary with Lambda function using serverless webpack #342

Closed

Make sure the Node library works on AWS Lambda grpc/grpc#6443

Closed

HyperBrain force-pushed the support-scripts branch from 0408678 to 459072e Compare March 10, 2018 10:59

Frank Schmid added 4 commits March 12, 2018 11:41

Allow to configure package scripts

d8a5f8d

Unit tests

3fcf387

Remaining unit tests

91a914f

Added documentation

83d5c6c

HyperBrain force-pushed the support-scripts branch from 3a74fd4 to 83d5c6c Compare March 12, 2018 10:41

HyperBrain changed the base branch from yarn-support to master March 12, 2018 10:45

HyperBrain merged commit 321dd88 into master Mar 13, 2018

HyperBrain deleted the support-scripts branch March 13, 2018 11:05

jamesmbourne pushed a commit to jamesmbourne/serverless-webpack that referenced this pull request Oct 15, 2019

Merge pull request serverless-heaven#344 from serverless-heaven/suppo…

e53225d

…rt-scripts Support custom packager scripts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support custom packager scripts #344

Support custom packager scripts #344

HyperBrain commented Mar 9, 2018 •

edited

Loading

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

HyperBrain commented Mar 9, 2018 •

edited

Loading

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

HyperBrain commented Mar 12, 2018

rgandco commented Mar 15, 2018

HyperBrain commented Mar 15, 2018 •

edited

Loading

HyperBrain commented Mar 15, 2018

rgandco commented Mar 15, 2018

rgandco commented Mar 15, 2018

Support custom packager scripts #344

Support custom packager scripts #344

Conversation

HyperBrain commented Mar 9, 2018 • edited Loading

What did you implement:

How did you implement it:

How can we verify it:

Todos:

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

HyperBrain commented Mar 9, 2018 • edited Loading

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

pflugs30 commented Mar 9, 2018

HyperBrain commented Mar 9, 2018

HyperBrain commented Mar 12, 2018

rgandco commented Mar 15, 2018

HyperBrain commented Mar 15, 2018 • edited Loading

HyperBrain commented Mar 15, 2018

rgandco commented Mar 15, 2018

rgandco commented Mar 15, 2018

HyperBrain commented Mar 9, 2018 •

edited

Loading

HyperBrain commented Mar 9, 2018 •

edited

Loading

HyperBrain commented Mar 15, 2018 •

edited

Loading