Cannot find module '../build/Release/sharp-{platformAndArch}.node on typescript project

[chris-kck]

Possible install-time or require-time problem

• I have read the documentation relating to installation.
• I have ensured that the architecture and platform of Node.js used for npm install is the same as the architecture and platform of Node.js used at runtime.

Are you using the latest version of sharp?

• I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

If you cannot confirm this, please upgrade to the latest version and try again before opening an issue.

If you are using another package which depends on a version of sharp that is not the latest, please open an issue against that package instead.

Is this a problem with filesystem permissions? NO

If you are using npm v6 or earlier and installing as a root or sudo user, have you tried with the npm install --unsafe-perm flag?

If you are using npm v7 or later, does the user running npm install own the directory it is run in?

If you are using the ignore-scripts feature of npm, have you tried with the npm install --ignore-scripts=false flag?

What is the complete output of running npm install --verbose sharp?

Too Long, but no visible error during this step.

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

System:
OS: macOS 12.2
CPU: (8) x64 Apple M1
Memory: 35.98 MB / 8.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 14.19.0 - ~/.nvm/versions/node/v14.19.0/bin/node
Yarn: 1.22.17 - ~/.nvm/versions/node/v14.19.0/bin/yarn
npm: 6.14.16 - ~/.nvm/versions/node/v14.19.0/bin/npm
npmPackages:
sharp: ^0.30.2 => 0.30.2

Suspicions and additional context

I do understand the platform and arch are different but sharp is clever enough to detect the platform and install the appropriate pre-built binaries. In addition, the installation process is restarted on the lambda.
Lastly, this error is only occuring at runtime. Project is setup to use typescript. I have tried using different kinds of imports like require('sharp'), import {sharp} from 'sharp', import * as sharp from 'sharp' and import sharp from 'sharp' as in other issues but with no luck.

Error on lambda function

@lovell does this library currently support typescript? I dove deep into the documentation but could not find how to use it with typescript. This image shows the error I was encountering both on my local on Mac and on AWS lambda. Please assist.

Error on M1 showing the file actually exists

[lovell]

Hi, did you see https://sharp.pixelplumbing.com/install#aws-lambda ?

[chris-kck]

Hello, yes I checked it out. However I am not deploying with a zip file with node_modules. It is being deployed via a code pipeline. I think if I am able to solve for my local dev, it should be able to solve for the lambda.

[lovell]

Are you populating node_modules using Code Pipeline? If so, what is the output of npm install sharp --verbose when run on Code Pipeline?

[chris-kck]

Yes, all that is taken care of when the serverless lambdas are deployed. Looking more critically at the problem at my local with fewer variables, I think its relates to typescript and esbuild that is bundling the project files. I saw some mention of externals in other related issues but this is now beyond my understanding, but I suspect the challenge lies around there.

[lovell]

I've just added a section to the docs (not published yet) about use with esbuild - see https://github.com/lovell/sharp/blob/main/docs/install.md#esbuild

[chris-kck]

Thank you, this pins down the challenge I was having. Thank you for your amazing library!

[lovell]

I hope this information helped. Please feel free to re-open with more details if further assistance is required.

[SayefReyadh]

For AWS lambda deployment with Sharp module from win32-x64 OS, the following worked for me when using serverless, esbuild and serverless-esbuild. Changed the serverless.yml file with the below configuration. It is basically telling esbuild to download sharp again with the following --arch=x64 --platform=linux considering your lambda uses x64 arch. Check serverless-esbuild packager and packagerOptions options for more understanding.

esbuild:
    # keep existing configurations
    external:
      - sharp
    packagerOptions:
      scripts:
        - npm install --arch=x64 --platform=linux sharp

Note: If you are using AWS code build for automatic deployment no other additional script was needed for my end. Also, you need to comment out the script npm install --arch=x64 --platform=linux sharp while testing with serverless-offline from win32-x64 OS

[chris-kck]

@SayefReyadh thank you very much this makes sense. Thank you for the added explanation of what is going on. Out of interest, how did you get to that solution?

[SayefReyadh]

@chris-kck similar implementations were done for serverless-webpack and discussed in #2230

[flosky]

I am having issues making sharp work on aws lambda (with cdk) and esbuild. I am creating one .js file via esbuild and then deploy my lambda function via cdk with that one .js file (instead of using a zip folder).

When I run my lambda function I get this error:

{
    "errorType": "Error",
    "errorMessage": "\nSomething went wrong installing the \"sharp\" module\n\nCannot find module '../build/Release/sharp-linux-x64.node'\nRequire stack:\n- /var/task/processMarketingFile.js\n- /var/runtime/UserFunction.js\n- /var/runtime/Runtime.js\n- /var/runtime/index.js\n\nPossible solutions:\n- Install with verbose logging and look for errors: \"npm install --ignore-scripts=false --foreground-scripts --verbose sharp\"\n- Install for the current linux-x64 runtime: \"npm install --platform=linux --arch=x64 sharp\"\n- Consult the installation documentation: https://sharp.pixelplumbing.com/install",
    "stack": [
        "Error: ",
        "Something went wrong installing the \"sharp\" module",
        "",
        "Cannot find module '../build/Release/sharp-linux-x64.node'",
        "Require stack:",
        "- /var/task/processMarketingFile.js",
        "- /var/runtime/UserFunction.js",
        "- /var/runtime/Runtime.js",
        "- /var/runtime/index.js",
        "",
        "Possible solutions:",
        "- Install with verbose logging and look for errors: \"npm install --ignore-scripts=false --foreground-scripts --verbose sharp\"",
        "- Install for the current linux-x64 runtime: \"npm install --platform=linux --arch=x64 sharp\"",
        "- Consult the installation documentation: https://sharp.pixelplumbing.com/install",
        "    at /var/task/processMarketingFile.js:82:67968",
        "    at /var/task/processMarketingFile.js:1:222",
        "    at /var/task/processMarketingFile.js:83:121",
        "    at /var/task/processMarketingFile.js:1:222",
        "    at /var/task/processMarketingFile.js:83:63844",
        "    at /var/task/processMarketingFile.js:1:222",
        "    at Object.<anonymous> (/var/task/processMarketingFile.js:90:16737)",
        "    at Module._compile (internal/modules/cjs/loader.js:1085:14)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)",
        "    at Module.load (internal/modules/cjs/loader.js:950:32)"
    ]
}

I have read the installation guidelines on lambda and esbuild but I don't think that makes sense here. I need to have the sharp package present in the lambda context if I should exclude it in esbuild (which I am doing with the aws-sdk package). But how do I then get it to be present in my lambda?
If I follow the lambda instructions on the other hand it also doesn't work and I get the above error.

Any hints for me on this one?

[SayefReyadh]

@flosky I have faced similar issues while deploying with esbuild and serverless framework, I am not sure if aws cdk supports YAML, do consider having a look at our existing discussion in this PR #3235

[flosky]

@SayefReyadh Thanks for the hint. As far as I can see the mentioned PR is not relevant here. The cdk setup with esbuild seems to be different from serverless.

I managed to get it to work which also made me realized what is missing. Since we have to mark sharp as external dependency with esbuild, we have to make sure the lib the present in the lambda function. For this I had to change the Lambda setup in cdk from using a single js file (which was the whole point for using esbuild) to create an S3 asset with all the files (including the node_modules).
In the end I did a hybrid version. I created the single js file bundle with esbuild but excluded sharp and then added another package.json that only has sharp as dependency to create a node_modules only with sharp and its dependencies (I then also had to run the code that installs sharp for the right target architecture) and upload all of that to S3 as my Lambda code.

Not a great solution, I think. Ideally I would like to just use esbuild without marking sharp as external but I guess that is not possible due to the compiled source files. Another option would be to offer a lambda layer with sharp and all it's dependencies installed. I saw that people created this but it wasn't very well maintained. Would be better to offer this from the main maintainer.

[flosky]

@SayefReyadh I found a good solution that should be added to the sharp docs, I think.

You can set this up quite easily by using the NodejsFunction from aws-cdk-lib/aws-lambda-nodejs. It has esbuild as bundler built in and allows for some useful bundling configurations, which help in this case:

1. You can tell the bundler to keep certain packages in their node_modules file and not bundle them into the final .js file
2. You can force the bundler to always run inside a docker container which aligns with the target architecture. So no need to additionally call SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm install --arch= arm64 --platform=linux --libc=glibc sharp. I tested it with setting the Lambda architecture to Architecture.ARM_64 and it worked

Just as a reminder, this is for cdk Lambda projects using sharp

Here is a code snippet:

new NodejsFunction(this, 'Lambda', {
            role: someRole,
            handler: 'handler',
            entry: 'path/to/handlerFile.ts',
            functionName: 'sharpImageProcessingLambda',
            runtime: Runtime.NODEJS_14_X,
            architecture: Architecture.ARM_64,
            bundling: {
                externalModules: ['aws-sdk', 'sharp'],
                sourceMap: true,
                nodeModules: ['sharp'],
                forceDockerBundling: true
            }
        });

[SayefReyadh]

NodejsFunction

@flosky Thanks, appreciated the update.