Elasticsearch instrumentation

[AlexanderWert]

This issue groups all the tasks that are required to allow proper usage of APM agent within Elasticsearch itself.

• fix log4j initialization within the agent: Fix log4j init with SecurityManager #2568
• fix class not found java.lang.Throwable when agent is loaded within ES, adding system module access to security policy fixed that.
• Related but optional (would make debugging easier) Add tests for Security Manager #2539
• internal spans within ES are created as transactions, whereas they are spans with OTel SDK.
• HTTP attributes in transaction are not properly mapped (might be in apm-server mapping):
  • difference between OTLP intake and OTel bridge apm-server#8067
  • Fix missing otel transaction attributes #2657
  • remove transaction.status from otel transactions #2660
• Security Manager might prevent proper creation of log file: Security Manager log file fallback #2581
• Security Manager + agent remote configuration

  WARN  org.stagemonitor.configuration.ConfigurationOption - Error in Java System Properties: no protocol:  Default value 'http://localhost:8200' for 'server_url' will be applied
  

Relates to

• Instrument Elasticsearch with APM elasticsearch#84369

[pugnascotia]

So far, the only change I've made to the agent is as follows:

diff --git a/apm-agent-plugin-sdk/src/main/java/co/elastic/apm/agent/sdk/logging/LoggerFactory.java b/apm-agent-plugin-sdk/src/main/java/co/elastic/apm/agent/sdk/logging/LoggerFactory.java
index c61257d21..4bcf2ee57 100644
--- a/apm-agent-plugin-sdk/src/main/java/co/elastic/apm/agent/sdk/logging/LoggerFactory.java
+++ b/apm-agent-plugin-sdk/src/main/java/co/elastic/apm/agent/sdk/logging/LoggerFactory.java
@@ -18,6 +18,9 @@
  */
 package co.elastic.apm.agent.sdk.logging;
 
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+
 public class LoggerFactory {
 
     private static volatile ILoggerFactory iLoggerFactory;
@@ -32,11 +35,16 @@ public class LoggerFactory {
      * @param name The name of the logger.
      * @return logger
      */
-    public static Logger getLogger(String name) {
+    public static Logger getLogger(final String name) {
         if (iLoggerFactory == null) {
             return NoopLogger.INSTANCE;
         }
-        return iLoggerFactory.getLogger(name);
+        return AccessController.doPrivileged(new PrivilegedAction<Logger> () {
+            @Override
+            public Logger run() {
+                return iLoggerFactory.getLogger(name);
+            }
+        });
     }
 
     /**

This may not ultimately be what we want to do, but I got me further.

The problem with java.lang.Throwable not being found was (I believe) due to the ES code lacking the accessSystemModules permission.

CC @ChrisHegarty

[felixbarny]

I've noticed that the apm-agent module doesn't have all the sources of the agent, which makes debugging a challenge.

May not be entirely in scope of this issue but having the ability to pull in all sources to that the agent can be debugged within a project that it's integrated in seems like an important capability. That's not only in the context of Elasticsearch but arguably specifically important there.

[pugnascotia]

I tried running a 3 node cluster in docker-compose, and noticed that the service map doesn't break down the cluster, which I think would be useful if one was in a degraded state. I set a different service_node_name on each, which is reflected in the "JVMs" tab.

[SylvainJuge]

Regarding the spans that are created as transactions: all the parent-child relationships are created through the context propagator, as a consequence even the spans that are on the same node are considered to be remote.

We should add a distinction between:

• local parent span, stored as SpanContext or Context, probably within ES ThreadContext as a transient object.
• remote parent span, which are created from the context propagator.

Given the parent/child relationship for inherited ThreadContext isn't really straightforward, it would be more efficient to discuss this with someone familiar with the ES codebase.

Also, with OTel (and also our internal agent API), the "currently active span" is the one that is picked as parent when another child span is created. Ideally, in the Tracer implementation, onTraceStarted would be able to retrieve the parent span from TraceContext and use it. Because the ThreadContext inherited fields are just plain strings, we have to find a way to store and inherit a reference (or alternatively an ID if it's not directly possible) in order to preserve the link to the parent span in place of using a remote parent.