Memory leak

[romilpunetha]

There's probably a memory leak in the instrumentation jar. Two of the services required a revert, one of them as soon as 10 mins. Im attaching screenshots to show the performance after the code was deployed with instrumentation. Both these services were deployed at around 10.

Some of the common libraries are

org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.3.61
com.fasterxml.jackson.module:jackson-module-kotlin:jar:2.9.10
com.amazonaws:aws-java-sdk-s3:jar:1.11.676 and other aws ki lib
org.springframework.boot:spring-boot-starter-actuator:jar:2.1.12
de.codecentric:spring-boot-admin-starter-client
de.codecentric:spring-boot-admin-starter-server:jar:2.1.1
org.springframework.boot:spring-boot-starter-undertow:jar:2.1.12.RELEASE
org.springframework.boot:spring-boot-starter-quartz:jar:2.1.12.RELEASE
org.springframework.boot:spring-boot-starter-data-redis:jar:2.1.12.RELEASE
org.springframework.boot:spring-boot-starter-data-jpa:jar:2.1.12.RELEASE
org.springframework.cloud:spring-cloud-starter-netflix-hystrix:jar:2.1.0.M3
org.springframework.boot:spring-boot-starter-web:jar:2.1.12.RELEASE

[trask]

Hi @romil-punetha!

The image you shared in gitter caught my interest:

because the image shows that the context store for the HttpServletResponse -> HttpServletRequest mapping is using WeakConcurrentMap instead of using a direct field mapping, which means that the agent wasn't able to add a field to HttpServletResponse.

You mentioned in gitter that you are using the DataDog agent at the same time, so I tried that also, and I got the same issue (not the memory leak, but using WeakConcurrentMap instead of direct field mapping).

Interestingly, if I swap the order of the OpenTelemetry and DataDog agents, then the OpenTelemetry agent uses direct field mapping, and the DataDog agent ends up using the WeakConcurrentMap.

Are you able to run the OpenTelemetry agent by itself without the DataDog agent to confirm whether that resolves the memory leak you are seeing?

[romilpunetha]

I ran it without the datadog agent, and i don't think there's a leak anymore. The memory footprint looks exactly like the one when only dd agent was injected. However, does this mean that I need to choose between using dd agent or the otel agent? Cuz I'm using them for different purposes.

[iNikem]

@romil-punetha Can you describe your use-case? What are those different purposes and why Otel agent cannot satisfy them both for you?

[romilpunetha]

@iNikem
I'm using datadog for APM only, and otel for in-house tracing, ie, my own hosted jaeger deployments and elasticsearch cluster.

[iNikem]

It is in general quite a bad idea to use several instrumenting agents at the same time. Especially those of very similar functionality, like DD and Otel. Have you considered using just Otel for both use-cases? DataDog certainly plans to migrate to Otel instrumentation. And it should be already possible to use Otel instrumentation and Collector and to ingest data into DataDog. Their support can help you with that as well, as Tyler mentioned in Gitter.

[romilpunetha]

Ok, will get in touch with them.

[trask]

I ran it without the datadog agent, and i don't think there's a leak anymore. The memory footprint looks exactly like the one when only dd agent was injected. However, does this mean that I need to choose between using dd agent or the otel agent? Cuz I'm using them for different purposes.

@romil-punetha, this is great to hear (and a relief)!

Sounds like we can close this, but keep us posted if you uncover anything new.