sharp is not working ( install ) in aws lambda ( serverless-webpack ) #2230

ghost · 2020-05-28T07:15:35Z

[I used]
1. serverless framework ( aws lambda, aws-nodejs-typescript template )
2. node version 12.14.1
3. sharp version 0.25.2 ( I did test 0.25.0 ~ 0.25.3 )

I want to resize my lambda zip size.

So I did install serverless-webpack plugins and bundle my lambda.

Before bundled by webpack, only transpile typescript to javascript and deploy using serverless and works well.

But, lambda bundled by serverless-webpack is not working.

[Error message]
darwin-x64' binaries cannot be used on the 'linux-x64' platform. Please remove the 'node_modules/sharp/vendor'

I don't know how to do.

I did check node_modules/sharp/vendor and saw linux-64 target.

but lambda is darwin-x64 logged.

I don't want to use lambda layer.

How to do?

I did

Delete node_modules and package-lock.json and install dependencies ( also installed sharp )
Delete node_modules/sharp and install sharp ( lambda environment - linux, x64, node version )
Set serverless-webpack configuration in serverless : packagerOptions ( scrips ) - rebuild sharp lambda enviroment
But, lambda is not working

upload and delete lambda work well and resized zip size!!

but get ( resize image ) lambda is not.

Please help me.

Thank you

The text was updated successfully, but these errors were encountered:

lovell · 2020-05-28T12:56:39Z

Did you see the following?

https://sharp.pixelplumbing.com/install#aws-lambda

https://github.com/serverless-heaven/serverless-webpack#node-modules--externals

ghost · 2020-05-29T01:31:30Z

Yes, I did it.

rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux --target=12.14.1 sharp

I did run above command in npm scripts ( before sls deploy )

also, I did

custom:
  webpack:
     packagerOptions:
         scripts:
            - npm rebuild --arch=x64 --platform=linux --target=12.14.1 sharp

And I checked my node_modules/sharp and version information. ( = linux )

But. lambda logged "darwin-64" not "linux-x64".

Additionally,

rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux --target=12.14.1 sharp

This command works well in lambda without serverless-webpack.

But, lambda with serverless-webpack is not working.

lovell · 2020-05-29T09:18:30Z

Hi, the commands documented at https://sharp.pixelplumbing.com/install#aws-lambda for you to use e.g. in the scripts directive are:

rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux sharp

Please note this is not the same as npm rebuild and there is no --target flag.

ghost · 2020-06-01T00:36:11Z

Yes, I also did it.

ex1.
rm -rf node_modules/sharp && npm i --arch=x64 --platform=linux sharp

ex2.
rm -rf node_modules/sharp && npm i --arch=x64 --platform=linux -- target=12.14.1 sharp

ex3.
npm rebuild --arch=x64 --platform=linux --target=12.14.1 sharp

But, above examples is not working.

I think that serverless-webpack plugin-in with sharp is not working properly.

Another issue repo said that using lambda layer.

But, I don't want to use lambda layer.. sorry..

lovell · 2020-06-01T09:04:58Z

Please can you provide a complete, standalone example with all dependencies, code and config, perhaps as a separate repo, that would allow someone else to reproduce this.

coreysnyder04 · 2020-06-02T03:17:03Z

I'm getting this same issue. I pretty much followed this example and then added sharp via this AWS Example.

I wonder if the build layer portion of this is what is screwing things up because it runs:

#!/bin/bash
set -eo pipefail
mkdir -p lib/nodejs
rm -rf node_modules lib/nodejs/node_modules
npm install --production
mv node_modules lib/nodejs/

Which may be installing the wrong version of sharp. When you run npm install --arch=x64 --platform=linux sharp does it save something to the package.json file or the lock file to say use this exact config when you run npm install again? Or does it go and grab whatever is best for that particular build environment?

It certainly doesn't seem like it saves enough info for the latter npm install to know which special version to use.

"dependencies": {
    "aws-xray-sdk-core": "1.1.2",
    "sharp": "^0.25.3"
  },

I tried to update the build layer to

#!/bin/bash
set -eo pipefail
mkdir -p lib/nodejs
rm -rf node_modules lib/nodejs/node_modules
npm install --production
rm -rf node_modules/sharp
npm install --arch=x64 --platform=linux sharp --save
mv node_modules lib/nodejs/

And then run the deploy script.

Deploy Script

#!/bin/bash
set -eo pipefail
ARTIFACT_BUCKET=$(cat bucket-name.txt)
aws cloudformation package --template-file template.yml --s3-bucket $ARTIFACT_BUCKET --output-template-file out.yml
aws cloudformation deploy --template-file out.yml --stack-name rr-image-service --capabilities CAPABILITY_NAMED_IAM

Unfortunately, this results in a new error I've never seen before:

$ ./3-deploy.sh

Unable to upload artifact lib/. referenced by ContentUri parameter of libs resource.
ZIP does not support timestamps before 1980

Here's the full output:

$ ./2-build-layer.sh

> [email protected] install /my-service/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached ~/.npm/_libvips/libvips-8.9.1-darwin-x64.tar.gz
added 94 packages from 179 contributors and audited 611 packages in 10.79s

5 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities


  > [email protected] install /my-service/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install --runtime=napi) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached ~/.npm/_libvips/libvips-8.9.1-linux-x64.tar.gz
+ [email protected]
added 519 packages from 462 contributors and audited 611 packages in 17.998s

13 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

$ ./3-deploy.sh

Unable to upload artifact lib/. referenced by ContentUri parameter of libs resource.
ZIP does not support timestamps before 1980

ghost · 2020-06-02T05:54:38Z

Thank you.

[package.json]

 "scripts": {
    "dev": "sls offline start",
     "build:prod": "rm -rf node_modules/sharp && npm install --arch=x64 --platform=linux sharp",
    "deploy:dev": "npx serverless deploy --aws-profile "profile" --stage dev --env development",
    "deploy:prod": "npx serverless deploy --aws-profile "profile" --stage prod --env production"
  },
 "dependencies": {
    "busboy": "^0.3.1",
    "http-errors": "^1.7.3",
    "sharp": "^0.25.2",
    "tslib": "^1.11.1",
    "uuid": "^7.0.3"
  },
  "devDependencies": {
    "@types/http-errors": "^1.6.3",
    "@types/sharp": "^0.25.0",
    "@types/aws-lambda": "^8.10.17",
    "@types/busboy": "^0.2.3",
    "@types/node": "^10.12.18",
    "@types/uuid": "^7.0.3",
    "aws-lambda": "^1.0.6",
    "aws-sdk": "^2.685.0",
    "serverless-dotenv-plugin": "^2.4.2",
    "serverless-offline": "^6.1.4",
    "serverless-webpack": "^5.2.0",
    "ts-loader": "^5.3.3",
    "typescript": "^3.2.4",
    "webpack": "^4.29.0",
    "webpack-node-externals": "^1.7.2"
  },

[serverless.yml]

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true
    packagerOptions:
         scripts:
            - npm rebuild --arch=x64 --platform=linux --target=12.14.1 sharp

[webpack.config.js]

const path = require("path");
const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");

module.exports = {
  context: __dirname,
  mode: slsw.lib.webpack.isLocal ? "development" : "production",
  entry: slsw.lib.entries,
  devtool: slsw.lib.webpack.isLocal
    ? "cheap-module-eval-source-map"
    : "source-map",
  resolve: {
    extensions: [".mjs", ".json", ".ts"],
    symlinks: false,
    cacheWithContext: false,
  },
  output: {
    libraryTarget: "commonjs",
    path: path.join(__dirname, ".webpack"),
    filename: "[name].js",
  },
  target: "node",
  externals: [nodeExternals()],
  module: {
    rules: [
      // all files with a `.ts` or `.tsx` extension will be handled by `ts-loader`
      {
        test: /\.(tsx?)$/,
        loader: "ts-loader",
        exclude: [
          [
            path.resolve(__dirname, "node_modules"),
            path.resolve(__dirname, ".serverless"),
            path.resolve(__dirname, ".webpack"),
          ],
        ],
        options: {
          transpileOnly: true,
          experimentalWatchApi: true,
        },
      },
    ],
  },
  plugins: [
    // new ForkTsCheckerWebpackPlugin({
    //   eslint: true,
    //   eslintOptions: {
    //     cache: true
    //   }
    // })
  ],
};

I did change script command that commented previous examples.

But, didn't work.

lovell · 2020-06-02T12:30:01Z

Cross-posted at https://stackoverflow.com/questions/62039078/error-darwin-x64-binaries-cannot-be-used-on-the-linux-x64-platform-aws-lamb

ghost · 2020-06-03T00:51:35Z

Thank you..

I wrote that post..

coreysnyder04 · 2020-06-03T02:10:54Z

So no luck still Seo? Are you also using layers when you upload/deploy your app? I wonder if that's the issue? Or at least related to it. I'm surprised others haven't faced this same problem.

ghost · 2020-06-03T02:21:08Z

I did change module ( sharp to jimp )

Jimp is working. ( zero native dependency )

But, Jimp slower than sharp..

Jimp : https://www.npmjs.com/package/jimp

sharp and jimp performance : https://sharp.pixelplumbing.com/performance

coreysnyder04 · 2020-06-04T01:31:47Z

It seems I just got past this issue but updating my build layer to use:

npm install --arch=x64 --platform=linux [email protected]
find lib/node_modules/. -mtime +10950 -print -exec touch {} \; #2239  Reference: https://github.com/aws/aws-cli/issues/2639#issuecomment-391255985

After doing this it works.

ghost · 2020-06-04T03:41:40Z

Please, can you provide your serverless.yml ?

I don't want to use lambda layer.

rickcha · 2020-07-06T12:34:47Z

Same issue. It looks like serverless is rebuilding all packages ignoring my local node_modules while deploying. I also tried using docker to npm install sharp as linux, but no luck.

ghost · 2020-08-03T05:39:42Z

I solved this problem using aws codebuild.

Thank you.

https://stackoverflow.com/questions/62039078/error-darwin-x64-binaries-cannot-be-used-on-the-linux-x64-platform-aws-lamb/63224217#63224217

ahmedmehanna · 2020-09-25T00:22:54Z

I fixed this issue by forcing the webpack to run the scripts
I am using serverless-bundle So the solution to fix it, it was the following ...

[serverless.yml]

plugins:
  - serverless-offline
  - serverless-pseudo-parameters
  - serverless-iam-roles-per-function
  - serverless-bundle

custom:
  bundle:
    packagerOptions:
      scripts:
        - rm -rf node_modules/sharp
        - npm install --arch=x64 --platform=linux sharp

https://sharp.pixelplumbing.com/install#aws-lambda
https://www.npmjs.com/package/serverless-bundle

bros4president · 2020-11-13T06:57:35Z

I fixed this issue by forcing the webpack to run the scripts
I am using serverless-bundle So the solution to fix it, it was the following ...
[serverless.yml]

plugins:
  - serverless-offline
  - serverless-pseudo-parameters
  - serverless-iam-roles-per-function
  - serverless-bundle

custom:
  bundle:
    packagerOptions:
      scripts:
        - rm -rf node_modules/sharp
        - npm install --arch=x64 --platform=linux sharp
https://sharp.pixelplumbing.com/install#aws-lambda
https://www.npmjs.com/package/serverless-bundle

I run into the same problem this thread was describing, developing on Windows and deploying to lambda.

Your answer helped me a lot as I was using a similar stack of plugins. So thanks a lot.

One caveat is that rm -rf node_modules/sharp threw an error so I added only the next line to scripts and it worked.

I assume it overrode the Windows installation, but I’m not sure if it’s safe.

Thanks!

hoonblizz · 2021-10-07T18:22:31Z

@ahmedmehanna Thanks for the solution. Its works like charm.

uniqueinx · 2023-04-15T15:24:08Z

For those using esbuild plugin

custom: {
    esbuild: {
      bundle: true,
      minify: false,
      sourcemap: true,
      target: 'node14',
      define: { 'require.resolve': undefined },
      platform: 'node',
      concurrency: 10,
      external: ['sharp'],
      packagerOptions: {
        scripts: [
          'rm -rf node_modules/sharp',
          'npm install --arch=x64 --platform=linux sharp',
        ],
      },
    },
  },

taojunnan · 2024-09-06T13:58:49Z

I did change module ( sharp to jimp )

Jimp is working. ( zero native dependency )

But, Jimp slower than sharp..

Jimp : https://www.npmjs.com/package/jimp

sharp and jimp performance : https://sharp.pixelplumbing.com/performance

thanks!

ghost added the question label May 28, 2020

ghost closed this as completed Aug 3, 2020

This was referenced May 24, 2022

Cannot find module '../build/Release/sharp-{platformAndArch}.node on typescript project #3126

Closed

Update install.md #3235

Merged

lmarini mentioned this issue Jun 27, 2022

linux/arm64 frontend github action fails because of sharp dependency clowder-framework/clowder2#34

Closed

robertu7 mentioned this issue Jul 26, 2022

fix(ci): move sharp install to webpack's packagerOptions thematters/serverless-file-post-processing#627

Merged

gronxb mentioned this issue Jun 26, 2024

fix(node): support npm versions 10 and above sst/ion#549

Merged

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sharp is not working ( install ) in aws lambda ( serverless-webpack ) #2230

sharp is not working ( install ) in aws lambda ( serverless-webpack ) #2230

ghost commented May 28, 2020 •

edited by ghost

Loading

lovell commented May 28, 2020

ghost commented May 29, 2020 •

edited by ghost

Loading

lovell commented May 29, 2020

ghost commented Jun 1, 2020

lovell commented Jun 1, 2020

coreysnyder04 commented Jun 2, 2020 •

edited

Loading

ghost commented Jun 2, 2020 •

edited by ghost

Loading

lovell commented Jun 2, 2020

ghost commented Jun 3, 2020

coreysnyder04 commented Jun 3, 2020

ghost commented Jun 3, 2020 •

edited by ghost

Loading

coreysnyder04 commented Jun 4, 2020

ghost commented Jun 4, 2020

rickcha commented Jul 6, 2020

ghost commented Aug 3, 2020

ahmedmehanna commented Sep 25, 2020

bros4president commented Nov 13, 2020 •

edited

Loading

hoonblizz commented Oct 7, 2021

uniqueinx commented Apr 15, 2023 •

edited

Loading

taojunnan commented Sep 6, 2024

sharp is not working ( install ) in aws lambda ( serverless-webpack ) #2230

sharp is not working ( install ) in aws lambda ( serverless-webpack ) #2230

Comments

ghost commented May 28, 2020 • edited by ghost Loading

I did

lovell commented May 28, 2020

ghost commented May 29, 2020 • edited by ghost Loading

lovell commented May 29, 2020

ghost commented Jun 1, 2020

lovell commented Jun 1, 2020

coreysnyder04 commented Jun 2, 2020 • edited Loading

ghost commented Jun 2, 2020 • edited by ghost Loading

lovell commented Jun 2, 2020

ghost commented Jun 3, 2020

coreysnyder04 commented Jun 3, 2020

ghost commented Jun 3, 2020 • edited by ghost Loading

coreysnyder04 commented Jun 4, 2020

ghost commented Jun 4, 2020

rickcha commented Jul 6, 2020

ghost commented Aug 3, 2020

ahmedmehanna commented Sep 25, 2020

bros4president commented Nov 13, 2020 • edited Loading

hoonblizz commented Oct 7, 2021

uniqueinx commented Apr 15, 2023 • edited Loading

taojunnan commented Sep 6, 2024

ghost commented May 28, 2020 •

edited by ghost

Loading

ghost commented May 29, 2020 •

edited by ghost

Loading

coreysnyder04 commented Jun 2, 2020 •

edited

Loading

ghost commented Jun 2, 2020 •

edited by ghost

Loading

ghost commented Jun 3, 2020 •

edited by ghost

Loading

bros4president commented Nov 13, 2020 •

edited

Loading

uniqueinx commented Apr 15, 2023 •

edited

Loading