gcr.io/paketo-buildpacks/opentelemetry: otel javaagent extension

[germatpreis]

Hello,

I am using the gcr.io/paketo-buildpacks/opentelemetry buildpack in my spring-boot application via the spring-boot-maven-plugin (see below) and would like to provide an extension to the java agent (which at the moment only attaches some baggage properties to all spans) or if this is not possible, package my extension together with the agent and have the buildpack pull then this artifact from somewhere. I'd like to think this could be a rather standard use case.

I have not yet done a deep dive into how buildpacks work in detail - so far I just have been using them for a couple of months without having to know (which is great and I think one of the premises of buildpacks).

Is there any way the gcr.io/paketo-buildpacks/opentelemetry can help with that, or could someone point me into the right direction how achieve this?

Any help here would be very much appreciated. Also if it just is: you have to read into how buildpacks work.

Current configuration in the pom.xml:

<plugin>
	<artifactId>spring-boot-maven-plugin</artifactId>
	<configuration>
		<image>
			<name>some-service:latest</name>
			<pullPolicy>IF_NOT_PRESENT</pullPolicy>
			<buildpacks>
				<buildpack>urn:cnb:builder:paketo-buildpacks/java</buildpack>
				<buildpack>gcr.io/paketo-buildpacks/opentelemetry</buildpack>
			</buildpacks>
			<env>
				<BP_OPENTELEMETRY_ENABLED>true</BP_OPENTELEMETRY_ENABLED>
				<BP_PLATFORM>linux/arm64/v8</BP_PLATFORM>
			</env>
		</image>
	</configuration>
	<executions>
		<execution>
			<goals>
				<goal>build-image-no-fork</goal>
			</goals>
		</execution>
	</executions>
	<groupId>org.springframework.boot</groupId>
</plugin>

[dmikusa]

So there's no support in the buildpack directly for extensions, but it's possible you may still be able to enable them. How would you enable this extension if you were not using buildpacks? What steps would you take?

[germatpreis]

I apologize for the delay in an answer - I needed to find time to build a working example of what I want to do (without buildpacks - just via an ordinary Dockerfile).

The steps that I have taken are:

• for simplicity, packaged my own extension agents/opentelemetry/opentelemetry-javaagent-foobar-ext together with the official opentelemetry-javaagent (done in agents/opentelemetry/opentelemetry-javaagent-foobar) which results in a single agent jar opentelemetry-javaagent-foobar-2.6.0.jar.
• built a simple spring-boot based service (foobar-service) that has a single controller and which should be targeted for instrumentation.
• created a docker file (foobar-service.Dockerfile) that adds both artifacts (the agent and the service) to the image, sets the JAVA_TOOL_OPTIONS to pick up the agent and configures a few OTEL_* environment variables.

This setup works but I'd rather continue to use buildpacks - if the above is achievable. I assume that I need at the minimum to do a single thing:

• replace the downloaded agent with my custom built one (assuming I keep the same artifact name as the official agent) by ie. overriding the url where the agent is downloaded from

or maybe a bit cleaner in two steps (which would get rid of the upload / download step during our CI build):

• don't download the customized agent artifact from anywhere but pick it up from a local directory (monorepo) and
• override the reference in the JAVA_TOOL_OPTIONS - I'd rather have the customized agent be named opentelemetry-javaagent-foobar-2.6.0.jar then just opentelemetry-javaagent (which is a bit misleading).

Any hints how I can achieve this would be greatly appreciated!

[dmikusa]

replace the downloaded agent with my custom built one (assuming I keep the same artifact name as the official agent) by ie. overriding the url where the agent is downloaded from

This get's tricky. We offer ways to change where you pull the resources from but the expectation is that you will get the exact same download. It's verified by a hash, so you can't change the binaries that are downloaded by the buildpack if you run a mirror or change the dependency mappings.

Would it be possible to bundle your changes in a separate JAR file? The buildpack could then fetch your changes separately and add that to the classpath or take some action to make your changes available to the agent. That's the idea anyway, I'm not totally sure if/how the agent would see your customizations.

If it's not possible to separate the two, we can have the buildpack fetch a customized agent from somewhere. The trick is doing that in a way that doesn't open up the possibility for someone to inject bad code into your application.

Without making any changes to the buildpack, what you'd need to do is to host your custom JAR file somewhere and then update the metadata in buildpack.toml with the new URL and hash for that download. This doesn't require a fork of the buildpack, but you would need to have some sort of automation to checkout the buildpack, modify buildpack.toml, then package it as your own separate image file. You could then point your builds to your image and it'll use your custom JAR.

The other possibility that wouldn't require code change is that you could ship your custom JAR with your application and then just set JAVA_TOOL_OPTIONS to point to that JAR file. i.e. manually add the required flags to enable the OTEL agent.

The buildpack is doing three basic things for you: 1. download & install the agent, 2. enable the agent, 3. configure the agent. If you're bundling the agent with your app, then you should be able to know the path to the file so you can enable the agent. You'd just need to add any OTEL configuration variables manually, instead of having the buildpack enable them. It's a little extra work, but it should let you enable the agent with your customizations.

If we're going to make some code changes here, I'd like to try and follow this pattern that we have in other buildpacks for enabling extensions, it's a set of env variables like $BP_OPENTELEMETRY_EXT_*. These would point to the custom code, allowing it to be downloaded and verified. The trick for this pattern to work is that it needs to be separate from the core agent.

See https://github.com/paketo-buildpacks/new-relic?tab=readme-ov-file#configuration

[germatpreis]

This get's tricky. We offer ways to change where you pull the resources from but the expectation is that you will get the exact same download. It's verified by a hash, so you can't change the binaries that are downloaded by the buildpack if you run a mirror or change the dependency mappings.

Understood.

Would it be possible to bundle your changes in a separate JAR file? The buildpack could then fetch your changes separately and add that to the classpath or take some action to make your changes available to the agent. That's the idea anyway, I'm not totally sure if/how the agent would see your customizations.

Yeah sure, that would be the 'official / correct' way to provide OTEL extensions anyway. In my head, packing them together would have simplified things. But if the extension should be separate, one needs to set a VM parameter -Dotel.javaagent.extensions and point it to the extension jar. I can update my example hopefully by the end of this week to do this flavour.

Without making any changes to the buildpack, what you'd need to do is to host your custom JAR file somewhere and then update the metadata in buildpack.toml with the new URL and hash for that download. This doesn't require a fork of the buildpack, but you would need to have some sort of automation to checkout the buildpack, modify buildpack.toml, then package it as your own separate image file. You could then point your builds to your image and it'll use your custom JAR.

Ok, this would be doable. We have our automation as Github Actions and I would expect that our extension won't change all that often.

The other possibility that wouldn't require code change is that you could ship your custom JAR with your application and then just set JAVA_TOOL_OPTIONS to point to that JAR file. i.e. manually add the required flags to enable the OTEL agent.

The buildpack is doing three basic things for you: 1. download & install the agent, 2. enable the agent, 3. configure the agent. If you're bundling the agent with your app, then you should be able to know the path to the file so you can enable the agent. You'd just need to add any OTEL configuration variables manually, instead of having the buildpack enable them. It's a little extra work, but it should let you enable the agent with your customizations.

So if I understand this option correctly, bake the opentelemetry-javaagent-foobar-2.6.0.jar that contains both (the agent and the extension) also into the service image and set all OTEL_* env variables that I already set in the Dockerfile and in addition set the env variable JAVA_TOOL_OPTIONS and point it to the location where I have added the uber-jar. If this is easily doable, this would actually be my preferred (short term) option. But it sounds to me, that by adding the jar file manually and enabling the agent & extension via JAVA_TOOL_OPTIONS and -Dotel.javaagent.extensions I would be already doing steps 1, 2 and 3 that the paketo-buildpacks/opentelemetry already does? Is the build-pack still needed then? What am I missing? At any rate, could you point to an example where the basic tasks here (I assume: adding a file as layer to an image, setting / overriding env variables) are shown?

If we're going to make some code changes here, I'd like to try and follow this pattern that we have in other buildpacks for enabling extensions, it's a set of env variables like $BP_OPENTELEMETRY_EXT_*. These would point to the custom code, allowing it to be downloaded and verified. The trick for this pattern to work is that it needs to be separate from the core agent.

See https://github.com/paketo-buildpacks/new-relic?tab=readme-ov-file#configuration

Oh yes, a configuration like for this new-relic build-pack would be a very neat and easily understandable way of providing extensions to the OTEL agent - even for developers new to build packs!

[dmikusa]

I would be already doing steps 1, 2 and 3 that the paketo-buildpacks/opentelemetry already does? Is the build-pack still needed then? What am I missing?

No, I think you've got it. You wouldn't need the buildpack at that point. You're doing all the set up yourself, so you can just drop that buildpack for the time being.

At any rate, could you point to an example where the basic tasks here (I assume: adding a file as layer to an image, setting / overriding env variables) are shown?

You wouldn't be adding a layer, just include the customized JAR file with your app. So you could have Maven/Gradle just bundle it up with all of your other JAR files. The trick is that /workspace is the top-level of your JAR so when you reference your JAR in JAVA_TOOL_OPTIONS, you can use the path /workspace/where/you/put/your/file.jar.

For env variables, you can just set them when your app runs. You don't need to set them at build time, unless you want to embed them in the image. Then you can use these instructions to embed those settings into the image -> https://paketo.io/docs/howto/configuration/#image-embedded-environment-variables

[dmikusa]

I created an issue for this also -> paketo-buildpacks/opentelemetry#156

[germatpreis]

Thanks for your support!