Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wildfly agent issue #85

Open
montyz opened this issue Feb 22, 2021 · 5 comments
Open

Wildfly agent issue #85

montyz opened this issue Feb 22, 2021 · 5 comments
Assignees
@willarmiros
Labels
enhancement New feature or request

Comments

@montyz
Copy link

montyz commented Feb 22, 2021

I'm trying to run the disco agent with wildfly 19.0.0. It's acting as if if doesn't know where the disco-java-agent.jar is, but it obviously found it to initialize X-Ray.

Picked up JAVA_TOOL_OPTIONS: -javaagent:/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar=pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins
Feb 19, 2021 11:52:45 PM com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration init
INFO: Initializing the X-Ray Agent Recorder
Feb 19, 2021 11:52:49 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Feb 19, 2021 11:52:49 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
java.lang.NoClassDefFoundError: software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl$ContainerExecutor.execute(ServiceContainerImpl.java:945)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:795)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.commitInstallation(ServiceControllerImpl.java:313)
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:752)
        at [email protected]//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at [email protected]//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at [email protected]//org.jboss.as.controller.ControlledProcessStateService.addService(ControlledProcessStateService.java:59)
        at [email protected]//org.jboss.as.server.BootstrapImpl.internalBootstrap(BootstrapImpl.java:118)
        at [email protected]//org.jboss.as.server.BootstrapImpl.bootstrap(BootstrapImpl.java:80)
        at [email protected]//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Feb 19, 2021 11:52:49 PM org.jboss.as.server.SystemExiter$1 logExit
FATAL: WFLYSRV0239: Aborting with exit code 1

Any idea what might be going wrong?

Thanks!
@willarmiros
Copy link
Contributor

Hi @montyz thanks for raising this. This is rather strange that clearly the X-Ray component of premain is being executed but there's a problem with finding one of the disco libraries. Can you try enabling Disco debug logging described here and posting those results? You can also try to use jar -tf disco-java-agent.jar to inspect its contents and see if the class software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice is actually unavailable in the JAR or just somehow not being found.

@montyz
Copy link
Author

montyz commented Feb 22, 2021 

➜  .deploy git:(master) jar -tf /Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar | grep ExecutorInterceptor
software/amazon/disco/agent/concurrent/ExecutorInterceptor.class
software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice.class
software/amazon/disco/agent/concurrent/ScheduledThreadPoolExecutorInterceptor.class
software/amazon/disco/agent/concurrent/ScheduledThreadPoolExecutorInterceptor$DecorateTaskAdvice.class

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /Users/monty/lark/wildfly

  JAVA: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java

  JAVA_OPTS:  -server  -Xms512M -Xmx4096M -Dpolyglot.js.nashorn-compat=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/goldfinch.hprof -Djava.net.preferIPv4Stack=true -Dlogging.level.com.amazonaws.xray=DEBUG  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

Picked up JAVA_TOOL_OPTIONS: -javaagent:/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar=pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) finished parsing argument list: pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: disco-java-agent-aws-plugin.jar
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: disco-java-agent-web-plugin.jar
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: disco-java-agent-sql-plugin.jar
[software.amazon.disco.agent.plugin.PluginDiscovery] DiSCo(Core) attempting to load JAR file into system classloader: aws-xray-agent-plugin.jar
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to HttpServletServiceInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to AWSClientInvokeInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to JdbcExecuteInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to AWSClientBuilderInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ExecutorInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ApacheHttpClientInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to DataAccessor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinPoolInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ForkJoinTaskInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ScheduledThreadPoolExecutorInterceptor to process
[software.amazon.disco.agent.DiscoAgentTemplate] DiSCo(Core) passing arguments to ThreadSubclassInterceptor to process
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.web.servlet.HttpServletServiceInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.awsv1.AWSClientInvokeInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.sql.JdbcExecuteInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.awsv2.AWSClientBuilderInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ExecutorInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.web.apache.httpclient.ApacheHttpClientInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.interception.templates.DataAccessor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinPoolInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ForkJoinTaskInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ScheduledThreadPoolExecutorInterceptor
[software.amazon.disco.agent.interception.InterceptionInstaller] DiSCo(Core) attempting to install software.amazon.disco.agent.concurrent.ThreadSubclassInterceptor
Feb 22, 2021 10:31:19 PM com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration init
INFO: Initializing the X-Ray Agent Recorder
[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-web-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.web.servlet.HttpServletServiceInterceptor, software.amazon.disco.agent.web.apache.httpclient.ApacheHttpClientInterceptor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor, software.amazon.disco.agent.interception.templates.DataAccessor

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-sql-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.sql.JdbcExecuteInterceptor

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: disco-java-agent-aws-plugin.jar
        Bootstrap: no
        Installables: software.amazon.disco.agent.awsv1.AWSClientInvokeInterceptor, software.amazon.disco.agent.awsv2.AWSClientBuilderInterceptor

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) Plugin name: aws-xray-agent-plugin.jar
        Bootstrap: no
        Init: com.amazonaws.xray.agent.runtime.AgentRuntimeLoader

[software.amazon.disco.agent.DiscoAgent] DiSCo(Agent) agent startup complete
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Feb 22, 2021 10:31:24 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Feb 22, 2021 10:31:24 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
java.lang.NoClassDefFoundError: software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl$ContainerExecutor.execute(ServiceContainerImpl.java:945)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:795)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.commitInstallation(ServiceControllerImpl.java:313)
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:752)
        at [email protected]//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at [email protected]//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at [email protected]//org.jboss.as.controller.ControlledProcessStateService.addService(ControlledProcessStateService.java:59)
        at [email protected]//org.jboss.as.server.BootstrapImpl.internalBootstrap(BootstrapImpl.java:118)
        at [email protected]//org.jboss.as.server.BootstrapImpl.bootstrap(BootstrapImpl.java:80)
        at [email protected]//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Feb 22, 2021 10:31:24 PM org.jboss.as.server.SystemExiter$1 logExit
FATAL: WFLYSRV0239: Aborting with exit code 1

Nothing obvious to me.

@stale
Copy link

stale bot commented Jun 11, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in next 7 days. Thank you for your contributions.

@stale stale bot added the stale label Jun 11, 2021
@willarmiros willarmiros added the enhancement New feature or request label Jun 11, 2021
@stale stale bot removed the stale label Jun 11, 2021
@willarmiros
Copy link
Contributor

Sorry for the delayed reply here. It seems like this is a problem with the X-Ray agent being incapable of instrumenting Wildfly server applications. Unfortunately our team is not able to prioritize adding this support at the moment and so we can't provide an ETA. However, X-Ray does have full compatibility with the latest stable release of the OpenTelemetry Java auto-instrumentation agent, which does have wildfly support. You can check out the OpenTelemetry Java for use with X-Ray documentation here.

In the meantime I'll leave this open as a feature request for wildfly support.

@adrian-baker
Copy link

adrian-baker commented Aug 28, 2021 •
edited
Loading

I hit the same exception in a standard Spring Boot application. The cause ended up being that I had used the maven distribution (not the github zip release), which gives you a jar with a subtlety different MANIFEST.MF than the one in the release, and this relies on the name of the jar not being changed:

Boot-Class-Path: disco-java-agent-0.11.0.jar

In my case, my gradle script had renamed this to disco-java-agent.jar, which then seemed to trigger the NoClassDefFoundError.

Oddly even with the renamed jar it worked fine using JDK 16, but not on JDK 11.

So maybe just confirm whether the Boot-Class-Path entry inside disco-java-agent.jar/META-INF/MANIFEST.MF matches your actual file name.

Note that although using the git release is recommended, using the maven dist makes it easier to reuse gradle's internal dependency resolution, and the associated caching and parallelisation benefits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@willarmiros willarmiros

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@montyz @adrian-baker @willarmiros