Jolokia trait failure

[squakez]

I've started to see this recently:

❌ TestHealthTrait (0s)
❌ TestHealthTrait/Readiness_condition_with_stopped_route_scaled (0s)
...
❌ TestJolokiaTrait (0s)
❌ TestJolokiaTrait/Run_Java_with_Jolokia (0s)

This seems to be because jolokia agent is not properly started. For some reason, the dependency has started failing suddenly and was never detected previously by tests.

[squakez]

The error we're getting is the following one:

jolokia integration I> No access restrictor found, access to any MBean is allowed
jolokia integration Could not start Jolokia agent: java.lang.IllegalStateException: Cannot load service org.jolokia.service.serializer.JolokiaSerializer,10 defined in jar:file:/deployments/dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar!/META-INF/jolokia/services-default : java.lang.IllegalArgumentException: Can not instantiate org.jolokia.service.serializer.JolokiaSerializer: java.lang.reflect.InvocationTargetException. Aborting

The strange thing is that this started out of the blue, without apparently any change on our side. @tadayosi do you have any clue what could be the root cause by any chance?

[squakez]

The problem is some kind of classpath issue, as it seems we must include the agent in the classpath by specifying it explicitly. Running the application with these parameters is working:

    - -javaagent:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar=discoveryEnabled=false,host=*,port=8778
    - -cp
    - ./resources:/etc/camel/application.properties:/etc/camel/resources:/etc/camel/resources.d/_configmaps:/etc/camel/resources.d/_secrets:/etc/camel/sources/camel-k-embedded-flow.yaml:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar:dependencies/*:dependencies/app/*:dependencies/lib/boot/*:dependencies/lib/main/*:dependencies/quarkus/*
    - io.quarkus.bootstrap.runner.QuarkusEntryPoint

We can certainly workaround that. However, it is still unanswered why this problems happened from one moment to another. I thought it may be some change in the jdk base docker image, but, trying the application with some older image is having the same effect.

[tadayosi]

Hmm, not sure. Could it be related to this issue?

• JAVA_APP_DIR should be set for container images quarkusio/quarkus#39022

[squakez]

Hmm, not sure. Could it be related to this issue?

* [`JAVA_APP_DIR` should be set for container images quarkusio/quarkus#39022](https://github.com/quarkusio/quarkus/issues/39022)

Thanks for looking. No, it seems a different kind of problem. More something directly related to Java and classpaths probably.

[tadayosi]

I wonder if this issue can be reproduced locally with the same set of exec parameters.

As you can see at the above quarkus issue, the container image entrypoint has changed between Quarkus 3.5 and 3.7. This can cause changes in the classpaths. Maybe we can podman inspect the two container images and see if there's any changes between the working and non-working ones.

[squakez]

It can be reproduced. The non working classpath is:

     - ./resources:/etc/camel/application.properties:/etc/camel/resources:/etc/camel/resources.d/_configmaps:/etc/camel/resources.d/_secrets:/etc/camel/sources/camel-k-embedded-flow.yaml:dependencies/*:dependencies/app/*:dependencies/lib/boot/*:dependencies/lib/main/*:dependencies/quarkus/*

and the working one requires the jolokia agent to be specified:

   - ./resources:/etc/camel/application.properties:/etc/camel/resources:/etc/camel/resources.d/_configmaps:/etc/camel/resources.d/_secrets:/etc/camel/sources/camel-k-embedded-flow.yaml:dependencies/lib/main/org.jolokia.jolokia-agent-jvm-2.0.2-javaagent.jar:dependencies/*:dependencies/app/*:dependencies/lib/boot/*:dependencies/lib/main/*:dependencies/quarkus/*

what is strange is that the first classpath was working until a couple of days ago. The only thing that may have changed without our control is the base JDK image.