AWS Lambda Problem Node.js & Latest

[SouthBelair]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

We had an existing aws lambda integration in place and noticed that some of our lambdas were missing from the configure list. In trying to correct this we created a new installation and selected an existing project and aws stack, this updated our lambdas to the "latest". The lambdas immediately began producing errors (see below).

Here's what changed:

Lambda ARN changed from arn:aws:lambda:us-east-1:943013980633:layer:SentryNodeServerlessSDK:184 to arn:aws:lambda:us-east-1:943013980633:layer:SentryNodeServerlessSDK:245

NODE_OPTIONS env value changed from -r @sentry/serverless/dist/awslambda-auto to -r @sentry/aws-serverless/cjs/awslambda-auto

We had to disable sentry on all of our node.js lambdas so that they would continue to operate, in order for us to restore sentry reporting we'll have to manually update every lambda with the old layer and NODE_options values, we have hundreds of lambdas.

node:internal/modules/cjs/loader:597
throw e;
^
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json
at new NodeError (node:internal/errors:405:5)
at exportsNotFound (node:internal/modules/esm/resolve:371:10)
at packageExportsResolve (node:internal/modules/esm/resolve:718:9)
at resolveExports (node:internal/modules/cjs/loader:590:36)
at Module._findPath (node:internal/modules/cjs/loader:664:31)
at Module._resolveFilename (node:internal/modules/cjs/loader:1126:27)
at Module._load (node:internal/modules/cjs/loader:981:27)
at internalRequire (node:internal/modules/cjs/loader:174:19)
at Module._preloadModules (node:internal/modules/cjs/loader:1569:5)
at loadPreloadModules (node:internal/process/pre_execution:628:5) {
code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}
Node.js v18.20.2
INIT_REPORT Init Duration: 156.91 ms Phase: init Status: error Error Type: Runtime.ExitError
node:internal/modules/cjs/loader:597
throw e;
^
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json
at new NodeError (node:internal/errors:405:5)
at exportsNotFound (node:internal/modules/esm/resolve:371:10)
at packageExportsResolve (node:internal/modules/esm/resolve:718:9)
at resolveExports (node:internal/modules/cjs/loader:590:36)
at Module._findPath (node:internal/modules/cjs/loader:664:31)
at Module._resolveFilename (node:internal/modules/cjs/loader:1126:27)
at Module._load (node:internal/modules/cjs/loader:981:27)
at internalRequire (node:internal/modules/cjs/loader:174:19)
at Module._preloadModules (node:internal/modules/cjs/loader:1569:5)
at loadPreloadModules (node:internal/process/pre_execution:628:5) {
code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}
Node.js v18.20.2
INIT_REPORT Init Duration: 634.31 ms Phase: invoke Status: error Error Type: Runtime.ExitError
START RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Version: $LATEST
RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8
REPORT RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Duration: 655.34 ms Billed Duration: 656 ms Memory Size: 256 MB Max Memory Used: 15 MB

Expected Result

N/A

Actual Result

N/A

Product Area

Settings - Integrations

Link

No response

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[fnfilho]

Having the same problem in production today. Had to rollback and remove our Sentry setup.

[andreiborza]

hello, sorry that you've experienced that.

Version 184 of the Lambda layer was still running the v7 version of our SDK.

While it's not a true fix for your problem, you can continue to use a v7 version of the layer. We specifically published one for 7.116.0, please see https://docs.sentry.io/platforms/javascript/guides/aws-lambda/layer/#lambda-layer-for-v7 for the ARN (unfortunately, you need to modify it yourself for your specific region).

Let me know if that helps at all.

[fnfilho]

Hey @andreiborza,

Thank you for following up. I am still not able to use AWS Lambda Layer.
Ticket #12012 got me confused as well to what is the right configs to set and I can't seem to find a definitive guide and docs of which Lambda Layer Version refers to what SDK Version.

So this is what I've tried so far, all combinations of the following values. Nothing seems to work.

ARNS:
arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:184
arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:240
arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:250
arn:aws:lambda:us-west-1:943013980633:layer:SentryNodeServerlessSDKv7:1

NODE_OPTIONS
--require @sentry/aws-serverless/cjs/awslambda-auto
--require @sentry/aws-serverless/dist/awslambda-auto
--require @sentry/aws-serverless/awslambda-auto

SENTRY_DSN
https://[email protected]/XXX

SENTRY_TRACES_SAMPLE_RATE
1.0

[paultheurer]

Having this issue as well. Had to turn off sentry lambda layer for all of our lambdas.

Updating to the latest version of the lambda layer via the sentry UI caused this.

[andreiborza]

For arn:...:SentryNodeServerlessSDK:235 and lower, and arn:...:SentryNodeServerlessSDKv7:1 (these use the v7 version of our SDK) the NODE_OPTIONS should include --require @sentry/serverless/dist/awslambda-auto

[fnfilho]

Thank you @andreiborza
I can confirm version 235 and NODE_OPTIONS @sentry/serverless/dist/awslambda-auto builds and runs correctly.

I also have set SENTRY_DSN: https://[email protected]/xxx and SENTRY_TRACES_SAMPLE_RATE: 1.0

With that said, neither errors nor performance data is being sent to Sentry. Is there a way to debug it better?

[andreiborza]

@fnfilho try setting debug: true and checking your aws logs. If you don't spot anything, please paste it here.

[fnfilho]

Hey @andreiborza
How exactly should I set debug: true? Is there an environment variable that has Sentry logs?

I did try by changing the package, but still did not work. This is what I did:

1. Downloaded the 235 version package;
2. Changed the awslambda-auto.js file, adding the debug: true parameter;
3. Uploaded a new zip file as a Custom AWS Layer (file attached);
4. Updated a function with the new custom AWS Layers that runs my own package with the parameter;
5. Ran the function

Logging than said: [Sentry] Cannot initialize SDK with debug option using a non-debug bundle.

sentry7.zip

[andreiborza]

@fnfilho you should be able to set SENTRY_DEBUG=1 in the env variables you provide to your lambda function. Let me know if you can't and I'll have another look.

[fnfilho]

Thank you for following up, @andreiborza.
I rolled back to package 235 and had the following env variables set:

NODE_OPTIONS: --require @sentry/serverless/dist/awslambda-auto --enable-source-maps --stack-trace-limit=1000
SENTRY_DEBUG: 1
SENTRY_DSN: https://[email protected]/xxx
SENTRY_TRACES_SAMPLE_RATE: 1.0

Than I forced an error to happen by using throw new Error("Sentry test"). No erros captured by Sentry and no logs in AWS about Sentry neither.

[fnfilho]

I tried updating to the new version 252 (arn: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:252). Can confirm the problem persists.

Error:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json

Env variables:
NODE_OPTIONS: -r @sentry/aws-serverless/cjs/awslambda-auto
SENTRY_DSN: https://[email protected]/xxx
SENTRY_TRACES_SAMPLE_RATE: 1.0

[andreiborza]

@fnfilho that's because all the versions above 235 use the new export via --require @sentry/aws-serverless/awslambda-auto.

[fnfilho]

Hey @andreiborza
I've tried to do some extra debugging here to figure out why errors are not shown on sentry.

This is my configuration:
ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:255
NODE_OPTIONS: --require @sentry/aws-serverless/awslambda-auto --enable-source-maps --stack-trace-limit=1000
SENTRY_DSN: https://[email protected]/xxx
SENTRY_TRACES_SAMPLE_RATE: 1.0

This builds correctly.
I than added a code to change the debug param:

const Sentry = require("@sentry/aws-serverless");
Sentry.getClient().getOptions().debug = true;
console.log("Sentry initiated.", { sentryConfig: Sentry.getClient().getOptions() });

Which prints something that looks ok:

INFO	Sentry initiated. {
  sentryConfig: {
    transport: [Function: d9],
    dsn: 'https://[email protected]/xxx',
    Ie: { sdk: [Object] },
    defaultIntegrations: [[Object]],
    integrations: [[Object]],
    autoSessionTracking: false,
    tracesSampleRate: 1,
    stackParser: [Function (anonymous)],
    platform: 'node',
    runtime: { name: 'node', version: 'v18.20.3' },
    serverName: '169.254.8.137',
    debug: true
  }
}

I than throw an error:

try {
    throw new Error("sentry testing 2");
} catch (error: any) {
    const response = Sentry.captureException(error);
    console.log("Sentry exception captured.", { response });
}

Which prints me a log with an ID that I think is the error Code from Sentry:

INFO	Sentry exception captured. { response: 'b6ca4a76e0434e048c45f357910c3b53' }

But still. No errors are tracked in Sentry at all.

[webjay]

I can add that I thought the reason why I didn't see my test errors etc was caused by us being over our limit, but I don't see any dropped errors.

The 3 errors I got in was not via the layer, but with @sentry/aws-serverless v8.11.0

So I'm in the same ⛵ as @fnfilho 👋

[lforst]

@fnfilho @webjay can you guys try calling await Sentry.flush() at the end of your handlers?

[webjay]

@lforst I tried this and still see nothing in Sentry. In the response I do see a Sentry id.

export const getAppVersionsController = async () => {
  const sentryId = Sentry.captureMessage("test message");
  await Sentry.flush();
  return createResponse({
    android: ANDROID_VERSION,
    ios: IOS_VERSION,
    sentryId,
  });
};

[fnfilho]

Hey @lforst
I'm getting the same thing as @webjay

Tried that a few hours ago.

const response = Sentry.captureException(error);
await Sentry.flush();
console.log("Sentry exception captured.", { response });

I also get the "Sentry exception captured" log with a response Id.
Still no bugs being capture in Sentry

[andreiborza]

captureException generates ids on the client, it's not an indicator that things went through.

My recommendation is to work backwards from the code snippet I posted here and see if it works and then gradually replace code with your own implementation.

Also note that we are still working on better support for ESM, see this issue: #12409.

[webjay]

To make it work, I have to:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
});

and Sentry.flush()

Is that how it is supposed to work @andreiborza

[andreiborza]

@webjay if you don't configure the DSN via env variables yes. Try it with this:

const Sentry = require("@sentry/aws-serverless");

Sentry.init({
  dsn: "__DSN__",
  tracesSampleRate: 1.0,
});

// Wrap your handler with `Sentry.wrapHandler`
exports.handler = Sentry.wrapHandler(async function(event, context) {    
    throw new Error('This should show up in sentry.')
});

[webjay]

@andreiborza I would say I do configure the DSN via env variables:

import { LayerVersion } from "aws-cdk-lib/aws-lambda";
import { type StackContext } from "sst/constructs";

export function SentryStack({ app, stack }: StackContext) {
  const sentryLayer = LayerVersion.fromLayerVersionArn(
    stack,
    "SentryLayer",
    `arn:aws:lambda:${app.region}:943013980633:layer:SentryNodeServerlessSDK:255`,
  );

  app.addDefaultFunctionLayers([sentryLayer]);

  app.addDefaultFunctionEnv({
    // SENTRY_ENABLED: app.local ? "false" : "true",
    SENTRY_DSN: process.env.SENTRY_DSN ?? "",
    SENTRY_ENVIRONMENT: app.stage,
    SENTRY_TRACES_SAMPLE_RATE: "1.0",
  });
}

However, it only works if I also do this:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  tracesSampleRate: 1.0,
});

If I use Sentry.wrapHandler I don't need to Sentry.flush()

I am not sure what the purpose of the layer is, when I still need to init and wrap?

[andreiborza]

@webjay please add SENTRY_DEBUG: 1 and paste some logs.

Here is another lambda function I successfully tried and the environment variables for it, without calling Sentry.init and without wrapping the handler.

Key	Value
NODE_OPTIONS	--require @sentry/aws-serverless/awslambda-auto --enable-source-maps --stack-trace-limit=1000
SENTRY_DEBUG	1
SENTRY_DSN	redacted DSN
SENTRY_TRACES_SAMPLE_RATE	1.0

const Sentry = require("@sentry/aws-serverless");

exports.handler = async function(event, context) {
    Sentry.captureException(new Error('This error should show in sentry.'))
    await Sentry.flush()
};

[webjay]

log-events-viewer-result.csv

Should I use NODE_OPTIONS with SentryNodeServerlessSDK:255?

Is it a problem to use the layer and have @sentry/[email protected] installed?

[andreiborza]

@webjay are there any more logs? Looks like it's missing some earlier parts.

Should I use NODE_OPTIONS with SentryNodeServerlessSDK:255?

You don't have to, you can. Depending on how you want to init sentry.

Is it a problem to use the layer and have @sentry/[email protected] installed?

Installed where? The layer comes with its own version.

[fnfilho]

Hey @andreiborza and @lforst

I did as requested and tried to simplify my code to something as similar as what you had there. Still no errors show up in Sentry. This is it.
I also created a new project and have a testing DSN in place.

FYI. I'm using serverless to build and deploy to lambda using version ^3.35.2. I have configured esbuild to for @sentry/profiling-node as per documentation as well.

import * as Sentry from "@sentry/aws-serverless";
import { nodeProfilingIntegration } from "@sentry/profiling-node";

Sentry.init({
  dsn: "https://[email protected]/4507538786549760",
  integrations: [nodeProfilingIntegration()],
  tracesSampleRate: 1.0,
  profilesSampleRate: 1.0,
  debug: true,
});

// Handler
export const main = Sentry.wrapHandler(async function (event, context) {
  try {
    console.log("options", JSON.stringify(Sentry.getClient().getOptions()));

    console.log("## ENVIRONMENT VARIABLES: " + serialize(process.env));
    console.log("## CONTEXT: " + serialize(context));
    console.log("## EVENT: " + serialize(event));

    Sentry.captureException(new Error("This error should show in sentry 10."));
    await Sentry.flush();
  } catch (err) {
    return {
      statusCode: 500,
      body: JSON.stringify(err),
    };
  }
});

var serialize = function (object: any) {
  return JSON.stringify(object, null, 2);
};

There are a few errors in log, all like this below. Full logs are here.
log-events-viewer-result.csv

ERROR	Sentry Logger [error]: Failed to read file: /src/domain/sentry-test/sentryTest.ts. Error: Error: ENOENT: no such file or directory, open '/src/domain/sentry-test/sentryTest.ts'
ERROR	Sentry Logger [error]: Failed to read file: /src/domain/sentry-test/sentryTest.ts. Error: Error: ENOENT: no such file or directory, open '/src/domain/sentry-test/sentryTest.ts'
ERROR	Sentry Logger [error]: Failed to read file: /node_modules/@sentry/aws-serverless/src/sdk.ts. Error: Error: ENOENT: no such file or directory, open '/node_modules/@sentry/aws-serverless/src/sdk.ts'
ERROR	Sentry Logger [error]: Failed to read file: /node_modules/@sentry/aws-serverless/src/sdk.ts. Error: Error: ENOENT: no such file or directory, open '/node_modules/@sentry/aws-serverless/src/sdk.ts'

[andreiborza]

@fnfilho is your function compiled to CJS? We are still working on proper ESM support (see #12409).

[andreiborza]

My colleague @Lms24 has overworked the documentation around our AWS lambda offering.

Please see https://docs.sentry.io/platforms/javascript/guides/aws-lambda for various options we support. Especially note the different installation methods we offer, and how to decide which to choose based on your setup.

We also removed documentation around the AWS Lambda container, as we don't support that anymore.

If you still have issues, please feel free to file a new issue.

Thank you all for the discussion and help that allowed us to fix underlying issues and bugs.