Introduce Cloud Logging Exporter

cf-java-logging-support-opentelemetry-agent-extension/README.md

@@ -1,14 +1,15 @@
 # OpenTelemetry Java Agent Extension for SAP Cloud Logging
 
 This module provides an extension for the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/).
-The extension scans the service bindings of an application for SAP Cloud Logging.
+The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to that service.
 Thus, this extension provides a convenient auto-instrumentation for Java applications running on SAP BTP.
 
-The extension provides two main features:
+The extension provides the following main features:
 
-* auto-configuration of the OpenTelemetry connection to SAP Cloud Logging
-* adding resource attributes to describe the CF application
+* additional exporters for logs, metrics and traces for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* auto-configuration of the generic OpenTelemetry connection to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* adding resource attributes describing the CF application
 
 See the section on [configuration](#configuration) for further details.
 
@@ -18,7 +19,7 @@ Any Java application can be instrumented with the OpenTelemetry Java Agent and t
 
 ```sh
 java -javaagent:/path/to/opentelemetry-javaagent-<version>.jar \
-     -Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<versions>.jar \
+     -Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<version>.jar \
      # your Java application command
 ```
 
@@ -37,27 +38,76 @@ java -javaagent:BOOT-INF/lib/opentelemetry-javaagent-<version>.jar \
 
 See the [example manifest](../sample-spring-boot/manifest-otel-javaagent.yml), how this translates into a deployment description.
 
-For the instrumentation to send observability data to SAP Cloud Logging, the application needs to be bound to a corresponding service instance.
-The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
+Once the agent is attached to the JVM with the ectension in place, there are two ways, which can be used to send data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging):
+
+1. Use the `cloud-logging` exporters explicitly as provided by the extension.
+This can be achieved via system properties or environment variables:
+```sh
+-Dotel.logs.exporter=cloud-logging \
+-Dotel.metrics.exporter=cloud-logging \
+-Dotel.traces.exporter=cloud-logging
+
+#or
+
+export OTEL_LOGS_EXPORTER=cloud-logging
+export OTEL_METRICS_EXPORTER=cloud-logging
+export OTEL_TRACES_EXPORTER=cloud-logging
+java #...
+```
+
+2. Use the default `otlp` exporter with the provided default configuration from the extension:
+
+```sh
+-Dotel.logs.exporter=otlp \
+-Dotel.metrics.exporter=otlp # default value \
+-Dotel.traces.exporter=otlp # default value
+
+#or
 
-Note, that the OpenTelemetry Java Agent currently only sends traces and metrics by default.
-To enable logs, the additional property `-Dotel.logs.exporter=otlp` is required.
+export OTEL_LOGS_EXPORTER=otlp
+export OTEL_METRICS_EXPORTER=otlp # default value
+export OTEL_TRACES_EXPORTER=otlp # default value
+java #...
+```
+
+Note, that the OpenTelemetry Java Agent currently sends traces and metrics by default using the `otlp` exporter.
+That means, without any configuration the agent with the extension will forward metrics and traces to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
+See TODO for the difference between `cloud-logging` and `otlp` exporters.
+The benefit of the `cloud-logging` exporter is, that it can be combined with a different configuration of the `otlp` exporter.
+
+For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging), the application needs to be bound to a corresponding service instance.
+The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
 
 ## Configuration
 
 The OpenTelemetry Java Agent supports a wide variety of [configuration options](https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/).
 As the extension provides configuration via SPI, all its configuration takes lower precedence than other configuration options for OpenTelemetry.
 Users can easily overwrite any setting using environment variables or system properties.
 
+### Using the Extension
+
+The extension needs to be started with the OpenTelemetry Java Agent as outlined in the [Quick Start Guide](#quickstart-guide).
+You need to enable shipping data either by using the `cloud-logging` exporters or relying on the `otlp` exporters for each signal type.
+Multiple different exporters can be configured with comma separation.
+Using the custom `cloud-logging` exporter will enable you, to use the default `otlp` exporter for different services.
+The extension will configure a default endpoint and credentials for the `otlp` endpoints, so no further configuration is required.
+
+Note, that the `cloud-logging` exporter is just a facade for the `otlp` exporter to allow configuration of multiple data sinks.
+There is no custom network client provided by this extension.
+
 ### Configuring the Extension
 
 The extension itself can be configured by specifying the following system properties:
 
 | Property | Default Value | Comment |
 |----------|---------------|---------|
-| `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
-| `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
-| `otel.javaagent.extension.sap.cf.resource.enabled` or `env(OTEL_JAVAAGENT_EXTENSION_SAP_CF_RESOURCE_ENABLED)` | `true` | Whether to add CF resource attributes to all events. |
+| `otel.javaagent.extension.sap.cf.binding.cloud-logging.label` or `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` or `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.user-provided.label` | `user-provided` | The label of a user-provided service binding to bind to. Note, this label is defined by the Cloud Foundry instance. |
+| `otel.javaagent.extension.sap.cf.resource.enabled` | `true` | Whether to add CF resource attributes to all events. |
+
+> The `otel.javaagent.extension.sap.*` properties are preferred over the `com.sap.otel.extension.*` properties, which are kept for compatibility. 
+Each `otel.javaagent.extension.sap.*` property can also be provided as environment variable `OTEL_JAVAAGENT_EXTENSION_SAP_*`.
 
 The extension will scan the environment variable `VCAP_SERVICES` for CF service bindings.
 User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" by default).
@@ -90,7 +140,7 @@ The [OpenTelemetry Java Instrumentation project](https://github.com/open-telemet
 
 ## Using User-Provided Service Instances
 
-The extension provides support not only for managed service instance of SAP Cloud Logging but also for user-provided service instances.
+The extension provides support not only for managed service instance of [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) but also for user-provided service instances.
 This helps to fine-tune the configuration, e.g. leave out or reconfigure the syslog drain.
 Furthermore, this helps on sharing service instances across CF orgs or landscapes.
 
@@ -103,7 +153,7 @@ The extension requires four fields in the user-provided service credentials and
 | `ingest-otlp-cert`| The mTLS client certificate in PEM format matching the client key. Line breaks as `\n`. |
 | `server-ca` | The trusted mTLS server certificate in PEM format. Line breaks as `\n`. |
 
-If you have a SAP Cloud Logging service key, you can generate the required JSON file with jq:
+If you have a [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) service key, you can generate the required JSON file with jq:
 
 ```bash
 cf service-key cls test \
@@ -119,5 +169,17 @@ Using this file, you can create the required user-provided service:
 ```
 
 Note, that you can easily feed arbitrary credentials to the extension.
-It does not need to be SAP Cloud Logging.
+It does not need to be [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 You can even change the tag using the configuration parameters of the extension.
+
+## Implementation Differences between Cloud-Logging and OTLP Exporter
+
+The `cloud-logging` exporter provided by this extension is a facade for the `OtlpGrpcExporter` provided by the OpenTelemetry Java Agent, just like the `otlp` exporter.
+The difference is just during the bootstrapping phase.
+The main differences are:
+
+* The `cloud-logging` exporter will send data to all found bindings to SAP Cloud Logging.
+The auto-instrumentation of the `otlp` exporter will just configure the first binding it finds priotizing user-provided services.
+* The `otlp` configuration will write the required certificates and keys to temporary files, which are deleted when the JVM is shut down. The `cloud-logging` exporter will keep the secrets in memory.
+* Since the `otlp` exporter is the default for traces and metrics, just attaching the extension and binding to Cloud Logging will result in metrics and traces being forwarded. 
+The `cloud-logging` exporter needs to be configured explictly as does the `otlp` exporter for logs.

cf-java-logging-support-opentelemetry-agent-extension/dependency-reduced-pom.xml

@@ -32,6 +32,10 @@
                 <excludes>
                   <exclude>io.opentelemetry</exclude>
                   <exclude>com.fasterxml.jackson.core</exclude>
+                  <exclude>com.squareup.okhttp3</exclude>
+                  <exclude>com.squareup.okio</exclude>
+                  <exclude>org.jetbrains.kotlin</exclude>
+                  <exclude>org.jetbrains</exclude>
                 </excludes>
               </artifactSet>
             </configuration>
@@ -58,7 +62,13 @@
       <groupId>io.opentelemetry</groupId>
       <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
       <version>1.31.0</version>
-      <scope>provided</scope>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-exporter-otlp</artifactId>
+      <version>1.31.0</version>
+      <scope>compile</scope>
     </dependency>
     <dependency>
       <groupId>org.slf4j</groupId>

cf-java-logging-support-opentelemetry-agent-extension/pom.xml

@@ -49,7 +49,10 @@
         <dependency>
             <groupId>io.opentelemetry</groupId>
             <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
-            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>io.opentelemetry</groupId>
+            <artifactId>opentelemetry-exporter-otlp</artifactId>
         </dependency>
         <dependency>
             <groupId>io.pivotal.cfenv</groupId>
@@ -83,6 +86,10 @@
                                 <excludes>
                                     <exclude>io.opentelemetry</exclude>
                                     <exclude>com.fasterxml.jackson.core</exclude>
+                                    <exclude>com.squareup.okhttp3</exclude>
+                                    <exclude>com.squareup.okio</exclude>
+                                    <exclude>org.jetbrains.kotlin</exclude>
+                                    <exclude>org.jetbrains</exclude>
                                 </excludes>
                             </artifactSet>
                         </configuration>

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/CloudLoggingConfigurationCustomizerProvider.java

@@ -13,6 +13,8 @@ public class CloudLoggingConfigurationCustomizerProvider implements AutoConfigur
     public void customize(AutoConfigurationCustomizer autoConfiguration) {
         autoConfiguration
                 .addPropertiesSupplier(new CloudLoggingBindingPropertiesSupplier(cfEnv));
+
+        // ConfigurableLogRecordExporterProvider
     }
 
 }

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/attributes/CloudFoundryResourceCustomizer.java

@@ -34,14 +34,16 @@ public Resource apply(Resource resource, ConfigProperties configProperties) {
         CfApplication cfApp = cfEnv.getApp();
         ResourceBuilder rb = Resource.builder();
         rb.put("service.name", cfApp.getApplicationName());
-        rb.put("sap.cf.source_id", getString(cfApp, "source_id"));
+        rb.put("sap.cf.source_id", cfApp.getApplicationId());
         rb.put("sap.cf.instance_id", cfApp.getInstanceIndex());
         rb.put("sap.cf.app_id", cfApp.getApplicationId());
         rb.put("sap.cf.app_name", cfApp.getApplicationName());
         rb.put("sap.cf.space_id", cfApp.getSpaceId());
         rb.put("sap.cf.space_name", cfApp.getSpaceName());
         rb.put("sap.cf.org_id", getString(cfApp, "organization_id"));
         rb.put("sap.cf.org_name", getString(cfApp, "organization_name"));
+        rb.put("sap.cf.process.id", getString(cfApp, "process_id"));
+        rb.put("sap.cf.process.type", getString(cfApp, "process_type"));
         return rb.build();
     }
 

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudLoggingBindingPropertiesSupplier.java

@@ -1,5 +1,7 @@
 package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;
 
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+import io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties;
 import io.pivotal.cfenv.core.CfEnv;
 import io.pivotal.cfenv.core.CfService;
 
@@ -17,18 +19,34 @@
 public class CloudLoggingBindingPropertiesSupplier implements Supplier<Map<String, String>> {
 
     private static final Logger LOG = Logger.getLogger(CloudLoggingBindingPropertiesSupplier.class.getName());
-    private static final String CLOUD_LOGGING_LABEL = System.getProperty("com.sap.otel.extension.cloud-logging.label", "cloud-logging");
-    private static final String CLOUD_LOGGING_TAG = System.getProperty("com.sap.otel.extension.cloud-logging.tag", "Cloud Logging");
-    private static final String USER_PROVIDED_LABEL = "user-provided";
-    public static final String OTLP_ENDPOINT = "ingest-otlp-endpoint";
-    public static final String OTLP_CLIENT_KEY = "ingest-otlp-key";
-    public static final String OTLP_CLIENT_CERT = "ingest-otlp-cert";
-    public static final String OTLP_SERVER_CERT = "server-ca";
+    private static final String OTLP_ENDPOINT = "ingest-otlp-endpoint";
+    private static final String OTLP_CLIENT_KEY = "ingest-otlp-key";
+    private static final String OTLP_CLIENT_CERT = "ingest-otlp-cert";
+    private static final String OTLP_SERVER_CERT = "server-ca";
 
-    private final CfEnv cfEnv;
+    private final CloudLoggingServicesProvider cloudLoggingServicesProvider;
 
     public CloudLoggingBindingPropertiesSupplier(CfEnv cfEnv) {
-        this.cfEnv = cfEnv;
+        Map<String, String> defaults = new HashMap<>();
+        defaults.put("com.sap.otel.extension.cloud-logging.label", "cloud-logging");
+        defaults.put("com.sap.otel.extension.cloud-logging.tag", "Cloud Logging");
+        defaults.put("otel.javaagent.extension.sap.cf.binding.user-provided.label", "user-provided");
+        ConfigProperties configProperties = DefaultConfigProperties.create(defaults);
+        this.cloudLoggingServicesProvider = new CloudLoggingServicesProvider(configProperties, cfEnv);
+    }
+
+    private static boolean isBlank(String text) {
+        return text == null || text.trim().isEmpty();
+    }
+
+    private static File writeFile(String prefix, String suffix, String content) throws IOException {
+        File file = File.createTempFile(prefix, suffix);
+        file.deleteOnExit();
+        try (FileWriter writer = new FileWriter(file)) {
+            writer.append(content);
+            LOG.fine("Created temporary file " + file.getAbsolutePath());
+        }
+        return file;
     }
 
     /**
@@ -41,10 +59,7 @@ public CloudLoggingBindingPropertiesSupplier(CfEnv cfEnv) {
      */
     @Override
     public Map<String, String> get() {
-        Stream<CfService> userProvided = cfEnv.findServicesByLabel(USER_PROVIDED_LABEL).stream();
-        Stream<CfService> managed = cfEnv.findServicesByLabel(CLOUD_LOGGING_LABEL).stream();
-        return Stream.concat(userProvided, managed)
-                .filter(svc -> svc.existsByTagIgnoreCase(CLOUD_LOGGING_TAG))
+        return cloudLoggingServicesProvider.get()
                 .findFirst()
                 .map(this::createEndpointConfiguration).orElseGet(Collections::emptyMap);
     }
@@ -94,18 +109,4 @@ private Map<String, String> createEndpointConfiguration(CfService svc) {
         }
     }
 
-    private static boolean isBlank(String text) {
-        return text == null || text.trim().isEmpty();
-    }
-
-    private static File writeFile(String prefix, String suffix, String content) throws IOException {
-        File file = File.createTempFile(prefix, suffix);
-        file.deleteOnExit();
-        try (FileWriter writer = new FileWriter(file)) {
-            writer.append(content);
-            LOG.fine("Created temporary file " + file.getAbsolutePath());
-        }
-        return file;
-    }
-
 }

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudLoggingServicesProvider.java

@@ -0,0 +1,49 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;
+
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+import io.pivotal.cfenv.core.CfEnv;
+import io.pivotal.cfenv.core.CfService;
+
+import java.util.List;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CloudLoggingServicesProvider implements Supplier<Stream<CfService>> {
+
+    private static final String DEFAULT_USER_PROVIDED_LABEL = "user-provided";
+    private static final String DEFAULT_CLOUD_LOGGING_LABEL = "cloud-logging";
+    private static final String DEFAULT_CLOUD_LOGGING_TAG = "Cloud Logging";
+
+    private final List<CfService> services;
+
+    public CloudLoggingServicesProvider(ConfigProperties config, CfEnv cfEnv) {
+        String userProvidedLabel = getUserProvidedLabel(config);
+        String cloudLoggingLabel = getCloudLoggingLabel(config);
+        String cloudLoggingTag = getCloudLoggingTag(config);
+        List<CfService> userProvided = cfEnv.findServicesByLabel(userProvidedLabel);
+        List<CfService> managed = cfEnv.findServicesByLabel(cloudLoggingLabel);
+        this.services = Stream.concat(userProvided.stream(), managed.stream())
+                .filter(svc -> svc.existsByTagIgnoreCase(cloudLoggingTag))
+                .collect(Collectors.toList());
+    }
+
+    private String getUserProvidedLabel(ConfigProperties config) {
+        return config.getString("otel.javaagent.extension.sap.cf.binding.user-provided.label", DEFAULT_USER_PROVIDED_LABEL);
+    }
+
+    private String getCloudLoggingLabel(ConfigProperties config) {
+        String fromOwnProperties = System.getProperty("com.sap.otel.extension.cloud-logging.label", DEFAULT_CLOUD_LOGGING_LABEL);
+        return config.getString("otel.javaagent.extension.sap.cf.binding.cloud-logging.label", fromOwnProperties);
+    }
+
+    private String getCloudLoggingTag(ConfigProperties config) {
+        String fromOwnProperties = System.getProperty("com.sap.otel.extension.cloud-logging.tag", DEFAULT_CLOUD_LOGGING_TAG);
+        return config.getString("otel.javaagent.extension.sap.cf.binding.cloud-logging.tag", fromOwnProperties);
+    }
+
+    @Override
+    public Stream<CfService> get() {
+        return services.stream();
+    }
+}