Failed to enable JFR when integrate AppDynamic in an app

[zhoufenqin]

paketo-buildpacks/microsoft-openjdk: 2.6.0
paketo-buildpacks/libjvm v1.40.0

when build and deploy a java app with AppDynamics, then try to enable JFR at runtime, --env BPL_JFR_ENABLED, then the app fail to start

> k logs -f app-dynamic-default-11-89d457655-jp9xg -c app-dynamic
Setting Active Processor Count to 8
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS
Enabling Java Flight Recorder with args: dumponexit=true,filename=/tmp/recording.jfr
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx437150K -XX:MaxMetaspaceSize=99425K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1G, Thread Count: 250, Loaded Class Count: 15140, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Configuring AppDynamics properties
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_microsoft-openjdk/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -javaagent:/layers/paketo-buildpacks_appdynamics/appdynamics-java/javaagent.jar -XX:ActiveProcessorCount=8 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.local.only=true -Dmanagement.endpoints.jmx.exposure.include=health,metrics -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dspring.jmx.enabled=true -Dserver.tomcat.mbeanregistry.enabled=true -Dfile.encoding=UTF8 -Dspring.config.import=optional:configserver:/ -XX:StartFlightRecording=dumponexit=true,filename=/tmp/recording.jfr -XX:MaxDirectMemorySize=10M -Xmx437150K -XX:MaxMetaspaceSize=99425K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
Java 9+ detected, booting with Java9Util enabled.
Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_APPLICATION_NAME] for application name [fenzho]
Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_TIER_NAME] for tier name [standard]
Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_NODE_NAME] for node name [node-0]
Full Agent Registration Info Resolver using selfService [false]
Full Agent Registration Info Resolver using selfService [false]
Full Agent Registration Info Resolver using ephemeral node setting [false]
Full Agent Registration Info Resolver using application name [fenzho]
Full Agent Registration Info Resolver using tier name [standard]
Full Agent Registration Info Resolver using node name [node-0]
Install Directory resolved to[/layers/paketo-buildpacks_appdynamics/appdynamics-java]
getBootstrapResource not available on ClassLoader
Class with name [com.ibm.lang.management.internal.ExtendedOperatingSystemMXBeanImpl] is not available in classpath, so will ignore export access.
[AD Agent init] Mon Dec 19 12:36:30 UTC 2022[DEBUG]: JavaAgent - Setting AgentClassLoader as Context ClassLoader
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[INFO]: JavaAgent - Low Entropy Mode: Attempting to swap to non-blocking PRNG algorithm
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[INFO]: JavaAgent - UUIDPool size is 10
Agent conf directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf]
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[INFO]: JavaAgent - Agent conf directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf]
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver is running
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_APPLICATION_NAME] for application name [fenzho]
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_TIER_NAME] for tier name [standard]
[AD Agent init] Mon Dec 19 12:36:31 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_NODE_NAME] for node name [node-0]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using ephemeral node setting [false]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using application name [fenzho]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using tier name [standard]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using node name [node-0]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver finished running
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Agent runtime directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Agent node directory set to [node-0]
Agent runtime conf directory set to /layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: AgentInstallManager - Agent runtime conf directory set to /layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: JavaAgent - JDK Compatibility: 1.8+
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: JavaAgent - Using Java Agent Version [Server Agent #22.8.0.34104 v22.8.0 GA compatible with 4.4.1.0 r76b2bb8f45a498afcc5ecaf5a1a9480ed0d4998b release/22.8.0]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: JavaAgent - Running IBM Java Agent [No]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: JavaAgent - Java Agent Directory [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: JavaAgent - Java Agent AppAgent directory [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104]
Agent logging directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/logs]
[AD Agent init] Mon Dec 19 12:36:32 UTC 2022[INFO]: JavaAgent - Agent logging directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/logs]
getBootstrapResource not available on ClassLoader
Started AppDynamics Java Agent Successfully.
bootstrap method initialization exception
Error occurred during initialization of VM
Failure when starting JFR on_create_vm_3

Expected Behavior

JFR is enabled when the java app integrate with APM like AppDynamics

Current Behavior

Failure when starting JFR

Possible Solution

Steps to Reproduce

Motivations

[dmikusa]

Started AppDynamics Java Agent Successfully.
bootstrap method initialization exception
Error occurred during initialization of VM
Failure when starting JFR on_create_vm_3

Are you able to run this combination elsewhere? Like on your laptop?

From what I can tell, the Java buildpack exec.d helper is adding the flags as it is designed to do. The problem occurs when the JVM is trying to start, so I'm not sure this is really a buildpacks issue. Perhaps an incompatibility with AppDynamics & Flight Recorder? They both try to do roughly the same thing, so perhaps a conflict?

[zhoufenqin]

I try it in my laptop and use docker to run the image, the same issue happened.

Besides, it works when I use AppliationInsights instead of AppDynamic, it maybe caused by AppDynamic buildpack?

> docker run -it --rm -v /tmp:/tmp -v /mnt/d/code/github/java-sample/bindings:/bindings/dynamic --env SERVICE_BINDING_ROOT=/bindings --env BPL_JFR_ENABLED=true dynamic
Setting Active Processor Count to 8
Enabling Java Flight Recorder with args: dumponexit=true,filename=/tmp/recording.jfr
Calculating JVM memory based on 15122228K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx14510802K -XX:MaxMetaspaceSize=99425K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 15122228K, Thread Count: 250, Loaded Class Count: 15140, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Configuring AppDynamics properties
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_microsoft-openjdk/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -javaagent:/layers/paketo-buildpacks_appdynamics/appdynamics-java/javaagent.jar -XX:ActiveProcessorCount=8 -XX:StartFlightRecording=dumponexit=true,filename=/tmp/recording.jfr -XX:MaxDirectMemorySize=10M -Xmx14510802K -XX:MaxMetaspaceSize=99425K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
Java 9+ detected, booting with Java9Util enabled.
Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_APPLICATION_NAME] for application name [gateway]
Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_TIER_NAME] for tier name [standard]
Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_NODE_NAME] for node name [node-0]
Full Agent Registration Info Resolver using selfService [false]
Full Agent Registration Info Resolver using selfService [false]
Full Agent Registration Info Resolver using ephemeral node setting [false]
Full Agent Registration Info Resolver using application name [gateway]
Full Agent Registration Info Resolver using tier name [standard]
Full Agent Registration Info Resolver using node name [node-0]
Install Directory resolved to[/layers/paketo-buildpacks_appdynamics/appdynamics-java]
getBootstrapResource not available on ClassLoader
Class with name [com.ibm.lang.management.internal.ExtendedOperatingSystemMXBeanImpl] is not available in classpath, so will ignore export access.
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[DEBUG]: JavaAgent - Setting AgentClassLoader as Context ClassLoader
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: JavaAgent - Low Entropy Mode: Attempting to swap to non-blocking PRNG algorithm
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: JavaAgent - UUIDPool size is 10
Agent conf directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: JavaAgent - Agent conf directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver is running
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_APPLICATION_NAME] for application name [gateway]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_TIER_NAME] for tier name [standard]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found env variable [APPDYNAMICS_AGENT_NODE_NAME] for node name [node-0]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using ephemeral node setting [false]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using application name [gateway]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using tier name [standard]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using node name [node-0]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver finished running
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Agent runtime directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104]
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Agent node directory set to [node-0]
Agent runtime conf directory set to /layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: AgentInstallManager - Agent runtime conf directory set to /layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/conf
[AD Agent init] Tue Dec 20 00:51:45 UTC 2022[INFO]: JavaAgent - JDK Compatibility: 1.8+
[AD Agent init] Tue Dec 20 00:51:46 UTC 2022[INFO]: JavaAgent - Using Java Agent Version [Server Agent #22.8.0.34104 v22.8.0 GA compatible with 4.4.1.0 r76b2bb8f45a498afcc5ecaf5a1a9480ed0d4998b release/22.8.0]
[AD Agent init] Tue Dec 20 00:51:46 UTC 2022[INFO]: JavaAgent - Running IBM Java Agent [No]
[AD Agent init] Tue Dec 20 00:51:46 UTC 2022[INFO]: JavaAgent - Java Agent Directory [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104]
[AD Agent init] Tue Dec 20 00:51:46 UTC 2022[INFO]: JavaAgent - Java Agent AppAgent directory [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104]
Agent logging directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/logs]
[AD Agent init] Tue Dec 20 00:51:46 UTC 2022[INFO]: JavaAgent - Agent logging directory set to [/layers/paketo-buildpacks_appdynamics/appdynamics-java/ver22.8.0.34104/logs]
getBootstrapResource not available on ClassLoader
Registered app server agent with Node ID[8184043] Component ID[3791123] Application ID [1222894]
Started AppDynamics Java Agent Successfully.
bootstrap method initialization exception
Error occurred during initialization of VM
Failure when starting JFR on_create_vm_3

[showpune]

Hi @dmikusa ,
We use the BP from tanzu-java-azure-buildpack/java-azure:8.8.0 , so the components inside is compatible?

[pivotal-david-osullivan]

As @dmikusa mentioned, this does look like an incompatibility between the operations of the AppDynamics agent and JFR with Java 11. Can you confirm you were using 11?

I cannot reproduce this when the JVM version is set to 17 or 8. I also cannot reproduce when the AppDynamics agent attempts to start but backs away, e.g. due to missing config - JFR enables successfully with 11.

If this is the case, a bug might need to be raised with AppDynamics

[dmikusa]

I try it in my laptop and use docker to run the image, the same issue happened.

@zhoufenqin - Sorry, I meant can you run your app not built through buildpacks with these two options enabled? i.e. just a regular java -jar or java -cp .. <class> on your laptop and enable both AppDynamics & JFR?

[dmikusa]

@showpune - I don't think there would be any guarantee of compatibility there. Buildpacks are just bundling and pulling together these different pieces of software. The buildpack you've referenced has both of those, but whether you can use the two together is really up to the individual tool (i.e. AppDynamics and JFR) not the buildpacks. The buildpacks just enable the flags and install the software.

[dmikusa]

If this is the case, a bug might need to be raised with AppDynamics

+1 This doesn't look like something related to the buildpacks. It is probably just an incompatibility with the two tools. The fact that AppInsight works with JFR is just another data point that supports it being an issue with AppDynamics & JFR not the buildpacks.