Sentry upgrade from 7.118.0 to 8.26.0 leak memory

[hiroshinishio]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Self-hosted/on-premise

Which SDK are you using?

@sentry/node

SDK Version

8.26.0

Framework Version

No response

Link to Sentry event

No response

Reproduction Example/SDK Setup

my sentry init in 7.118.0

    init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.NODE_ENV || "none",
        release: require("../../package.json").version,
        serverName: `${cluster.bot}-c=${cluster.id}-first=${cluster.first}-last=${cluster.last}` || "none",
        integrations: [
            new Integrations.Postgres({ module: require("pg") }),
            new Integrations.Modules(),
            new Integrations.FunctionToString(),
            new Integrations.LinkedErrors(),
            new Integrations.Console(),
            new Integrations.Http({ breadcrumbs: true, tracing: true }),
            rewriteFramesIntegration({ root: path.join(__dirname, "..") }),
        ],
        // Performance Monitoring
        tracesSampleRate: 1.0, //  Capture 100% of the transactions
    });

my sentry init in 8.6.0

    Sentry.init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.NODE_ENV || "none",
        release: require("../../package.json").version,
        serverName: `${cluster.bot}-c=${cluster.id}-first=${cluster.first}-last=${cluster.last}` || "none",
        integrations: [
            Sentry.modulesIntegration(),
            Sentry.functionToStringIntegration(),
            Sentry.linkedErrorsIntegration(),
            Sentry.consoleIntegration(),
            Sentry.httpIntegration({ breadcrumbs: true }),
            Sentry.rewriteFramesIntegration({ root: path.join(__dirname, "..") }),
            Sentry.onUnhandledRejectionIntegration(),
            Sentry.onUncaughtExceptionIntegration(),
            Sentry.redisIntegration(),
            Sentry.postgresIntegration(),
        ],
        // To avoid sending too much data to Sentry, we can reduce the sample rate of traces and profiles
        tracesSampleRate: 1.0,
        profilesSampleRate: 1.0,
    });

Steps to Reproduce

I'll try removing some of the integrations to see what's causing the problem.

Expected Result

A normal memory usage

Actual Result

anormal memory usage

[hiroshinishio]

Hey thanks for writing in!

We'll look into your issue next week as this week is Hackweek at Sentry (see getsentry#13421).

[hiroshinishio]

Hi, would you be able to provide a memory snapshot with the Node/v8 profiler so that we can look at what is holding the references causing the leak? Feel free to also shoot us an email or twitter dm if you don't want to publicly share it. Thanks!

[gitauto-ai]

Click the checkbox below to generate a PR!

• Generate PR

@hiroshinishio, You have 2 requests left in this cycle which refreshes on 2024-11-04 02:56:25+00:00.
If you have any questions or concerns, please contact us at [email protected].

[gitauto-for-dev]

Click the checkbox below to generate a PR!

• Generate PR - gitauto-wes

@hiroshinishio, You have 3 requests left in this cycle which refreshes on 2024-11-05 22:45:17+00:00.
If you have any questions or concerns, please contact us at [email protected].

[hiroshinishio]

I can't give you an snapshot because I have a lot of private information.

[hiroshinishio]

I made a snapshot, I'll examine it and show you

[hiroshinishio]

[hiroshinishio]

Set, Span and NonRecordingSpan are from Sentry

[hiroshinishio]

Yeah that looks like Sentry. Would you mind digging around a bit ant examine what holds the references to the spans?

[hiroshinishio]

[hiroshinishio]

[hiroshinishio]

Set, Span and NonRecordingSpan are from Sentry

This indicates that Sentry objects are retained in memory. It doesn't mean they are causing this retention though!
E.g. the expanded Span object has the distance (from root) of 15. The sentryRootSpan - distance of 12. It means something else is holding the reference to it, something that is closer to the root.
You might want to explore the bottom part of the memory profiler - the Retainers section

[hiroshinishio]

[hiroshinishio]

timerListMap[180000] - my understanding is that the app has started a timer for 180s (3min).
Any chance it's your app code rather than Sentry?

[hiroshinishio]

I don't think so. Before switching to sentry v8, someone warned me about the memory leak.

[hiroshinishio]

I have a few questions to narrow this down further. I am not ruling out that our SDK is causing the leak:

• What backend framework are you using - if at all?
• Are you manually creating spans somewhere?
• Can you share any more code that is relevant to Sentry?
• What Node.js version are you using? We have found that there are memory leaks in Node.js with regards to timers?
• Can you double-check whether you have any setTimeouts or setIntervals?
• One way we have found things may "leak" is if you create a span and don't end it. That way children get attached and they will be held indefinitely.

[hiroshinishio]

Hello,
I use sentry on a discord bot not a website
Node version: v20.17.0

[hiroshinishio]

Would you mind answering the questions I asked. It's important we have answers to them so we can rule out certain things.

[hiroshinishio]

I've already double-checked my code
When you talk to me about span, I think of HTML elements, but I don't do web.

[hiroshinishio]

I am talking about Sentry spans. Which you would start with Sentry.startSpan(). (https://docs.sentry.io/concepts/key-terms/tracing/distributed-tracing/#traces-transactions-and-spans)

Do you happen to have very long running requests in your process? Like server-sent-events, or streaming?

[hiroshinishio]

On my side I never call Sentry.startSpan()

[hiroshinishio]

Do you have any very long running requests / request handlers in your code?

[hiroshinishio]

I wouldn't say long requests, but I have a lot of requests per second, over 110req/s at times, and I receive a lot of messages via websocket.

[hiroshinishio]

here you can see that the memory leak arrived at the same time as my sentry update, and since then I've had the performance tab working.

[hiroshinishio]

I believe you saying this is correlated with the update. Can you share a bit more about your program architecture? Is discord opening a websocket request on your server or is your server making a websocket request to discord?

Also, are you initializing Sentry more than once per process?

[hiroshinishio]

From what I can tell from your profiler screenshots, something is creating a Span (a root span) that seemingly never finishes, and spans keep getting attached to that span and that ends up leaking memory. It would be nice to be able to figure out, what it is that is creating this root span.

[hiroshinishio]

• I initialize sentry only once
• On each cluster I open 16 websocket connections to discord (clients)

I'll try to remove the profillerRate line, and we'll see if it comes from that.

[hiroshinishio]

If you could somehow also try logging console.log(Sentry.getRootSpan()) somewhere in your websocket code, and share what is logged, that would be cool!

[hiroshinishio]

Sentry.getRootSpan() requires and argument

[hiroshinishio]

Sorry right. Can you try

const s = Sentry.getActiveSpan();
if (s) {
  console.log('root span', Sentry.getRootSpan(s));
}

[hiroshinishio]

On the active process i used eval()

NonRecordingSpan {
  _spanContext: {
    traceId: 'ba6488d048422cfba347c2a2b9b1eca5',
    spanId: '5df94fad8fd85299',
    traceFlags: 0,
    traceState: [TraceState]
  }
}

[hiroshinishio]

Thanks! We are struggling trying to understand what is creating this non-recording Span. Usually that shouldn't happen unless you set a traces sample rate of <1 or you manually continue a trace.

Can you try setting debug: true in your ˚Sentry.init()`? It will probably generate a lot of logs, but it may tell you what exactly is continued or not sampled and so on.

[hiroshinishio]

can I enable logs without restarting my process ?

[hiroshinishio]

I don't think so :/

[hiroshinishio]

Another thing that we just noticed: Have you properly followed the migration guide for how and when to call Sentry.init()? You may be calling Sentry.init() too late, which may create weird spans. https://docs.sentry.io/platforms/javascript/guides/node/migration/v7-to-v8/#updated-sdk-initialization

[hiroshinishio]

in my case I don't require sentry first

[hiroshinishio]

can you try doing so?

[hiroshinishio]

Is it normal for it to load integrations that I haven't put in?

[hiroshinishio]

yes, there is a set of integrations that are enabled by default, but don't worry, as long as you do not use (ie import/require) these packages these integrations are no-ops.

[hiroshinishio]

okay,

good now ?

NonRecordingSpan {
  _spanContext: {
    traceId: 'd6ce7aec75e1fbdac62a14ca1a5d3353',
    spanId: 'eea933c24e347f1d',
    traceFlags: 0,
    traceState: [TraceState]
  }
}

[hiroshinishio]

I can't make out from the screenshot what you did tbh. If you are still importing other things inside ./extras/sentry, that's wrong.

[hiroshinishio]

This corresponds to sentry

[hiroshinishio]

That setup looks good 👌 Can you share more of the debug logs? It would be good to see logs up until and including when you do things like send requests and database queries and similar. Thanks!

[hiroshinishio]

there was a query to my db but it doesn't seem to appear

[hiroshinishio]

Would you mind sharing the start of your application up to a certain point in text format? Thanks!

[hiroshinishio]

Do you have discord ?
It would be easier to talk and send logs

[hiroshinishio]

Yes! Feel free to join https://discord.com/invite/sentry and ping ``

[hiroshinishio]

After some back and fourth we have discovered that the memory leak in this issue happens due to the httpIntegration creating a NonRecordingSpan for a very long-lived websocket request. The span is never ended because the request basically never ends for the entire lifetime of the process and stuff (ie. child spans) keep getting attached to that span.

The workaround for now is to do the following:

Sentry.init({
  integrations: [
    Sentry.httpIntegration({
      ignoreOutgoingRequests(url, request) {
        return true;
      },
    }),
  ],
})

Thanks for the collaboration !!

---

Action items (varying degrees of possible):

• Stop attaching stuff to spans
• Create an upper bound for stuff being attached to spans

[hiroshinishio]

No problem 👍

[gitauto-for-dev]

Sorry, we have an error. Please try again.

Have feedback or need help?
Feel free to email [email protected].

[gitauto-for-dev]

Sorry, we have an error. Please try again.

Have feedback or need help?
Feel free to email [email protected].

[gitauto-ai]

Sorry, we have an error. Please try again.

Have feedback or need help?
Feel free to email [email protected].