Adjust Documentation for Cloud-Logging

Add references to the `cloud-logging`exporter to the documentation.
Describe the new config properties to control the extension.

Signed-off-by: Karsten Schnitter <[email protected]>

cf-java-logging-support-opentelemetry-agent-extension/README.md

@@ -1,14 +1,15 @@
 # OpenTelemetry Java Agent Extension for SAP Cloud Logging
 
 This module provides an extension for the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/).
-The extension scans the service bindings of an application for SAP Cloud Logging.
+The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to that service.
 Thus, this extension provides a convenient auto-instrumentation for Java applications running on SAP BTP.
 
-The extension provides two main features:
+The extension provides the following main features:
 
-* auto-configuration of the OpenTelemetry connection to SAP Cloud Logging
-* adding resource attributes to describe the CF application
+* additional exporters for logs, metrics and traces for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* auto-configuration of the generic OpenTelemetry connection to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* adding resource attributes describing the CF application
 
 See the section on [configuration](#configuration) for further details.
 
@@ -18,7 +19,7 @@ Any Java application can be instrumented with the OpenTelemetry Java Agent and t
 
 ```sh
 java -javaagent:/path/to/opentelemetry-javaagent-<version>.jar \
-     -Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<versions>.jar \
+     -Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<version>.jar \
      # your Java application command
 ```
 
@@ -37,27 +38,76 @@ java -javaagent:BOOT-INF/lib/opentelemetry-javaagent-<version>.jar \
 
 See the [example manifest](../sample-spring-boot/manifest-otel-javaagent.yml), how this translates into a deployment description.
 
-For the instrumentation to send observability data to SAP Cloud Logging, the application needs to be bound to a corresponding service instance.
-The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
+Once the agent is attached to the JVM with the ectension in place, there are two ways, which can be used to send data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging):
+
+1. Use the `cloud-logging` exporters explicitly as provided by the extension.
+This can be achieved via system properties or environment variables:
+```sh
+-Dotel.logs.exporter=cloud-logging \
+-Dotel.metrics.exporter=cloud-logging \
+-Dotel.traces.exporter=cloud-logging
+
+#or
+
+export OTEL_LOGS_EXPORTER=cloud-logging
+export OTEL_METRICS_EXPORTER=cloud-logging
+export OTEL_TRACES_EXPORTER=cloud-logging
+java #...
+```
+
+2. Use the default `otlp` exporter with the provided default configuration from the extension:
+
+```sh
+-Dotel.logs.exporter=otlp \
+-Dotel.metrics.exporter=otlp # default value \
+-Dotel.traces.exporter=otlp # default value
+
+#or
 
-Note, that the OpenTelemetry Java Agent currently only sends traces and metrics by default.
-To enable logs, the additional property `-Dotel.logs.exporter=otlp` is required.
+export OTEL_LOGS_EXPORTER=otlp
+export OTEL_METRICS_EXPORTER=otlp # default value
+export OTEL_TRACES_EXPORTER=otlp # default value
+java #...
+```
+
+Note, that the OpenTelemetry Java Agent currently sends traces and metrics by default using the `otlp` exporter.
+That means, without any configuration the agent with the extension will forward metrics and traces to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
+See TODO for the difference between `cloud-logging` and `otlp` exporters.
+The benefit of the `cloud-logging` exporter is, that it can be combined with a different configuration of the `otlp` exporter.
+
+For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging), the application needs to be bound to a corresponding service instance.
+The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
 
 ## Configuration
 
 The OpenTelemetry Java Agent supports a wide variety of [configuration options](https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/).
 As the extension provides configuration via SPI, all its configuration takes lower precedence than other configuration options for OpenTelemetry.
 Users can easily overwrite any setting using environment variables or system properties.
 
+### Using the Extension
+
+The extension needs to be started with the OpenTelemetry Java Agent as outlined in the [Quick Start Guide](#quickstart-guide).
+You need to enable shipping data either by using the `cloud-logging` exporters or relying on the `otlp` exporters for each signal type.
+Multiple different exporters can be configured with comma separation.
+Using the custom `cloud-logging` exporter will enable you, to use the default `otlp` exporter for different services.
+The extension will configure a default endpoint and credentials for the `otlp` endpoints, so no further configuration is required.
+
+Note, that the `cloud-logging` exporter is just a facade for the `otlp` exporter to allow configuration of multiple data sinks.
+There is no custom network client provided by this extension.
+
 ### Configuring the Extension
 
 The extension itself can be configured by specifying the following system properties:
 
 | Property | Default Value | Comment |
 |----------|---------------|---------|
-| `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
-| `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
-| `otel.javaagent.extension.sap.cf.resource.enabled` or `env(OTEL_JAVAAGENT_EXTENSION_SAP_CF_RESOURCE_ENABLED)` | `true` | Whether to add CF resource attributes to all events. |
+| `otel.javaagent.extension.sap.cf.binding.cloud-logging.label` or `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` or `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.user-provided.label` | `user-provided` | The label of a user-provided service binding to bind to. Note, this label is defined by the Cloud Foundry instance. |
+| `otel.javaagent.extension.sap.cf.resource.enabled` | `true` | Whether to add CF resource attributes to all events. |
+
+> The `otel.javaagent.extension.sap.*` properties are preferred over the `com.sap.otel.extension.*` properties, which are kept for compatibility. 
+Each `otel.javaagent.extension.sap.*` property can also be provided as environment variable `OTEL_JAVAAGENT_EXTENSION_SAP_*`.
 
 The extension will scan the environment variable `VCAP_SERVICES` for CF service bindings.
 User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" by default).
@@ -90,7 +140,7 @@ The [OpenTelemetry Java Instrumentation project](https://github.com/open-telemet
 
 ## Using User-Provided Service Instances
 
-The extension provides support not only for managed service instance of SAP Cloud Logging but also for user-provided service instances.
+The extension provides support not only for managed service instance of [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) but also for user-provided service instances.
 This helps to fine-tune the configuration, e.g. leave out or reconfigure the syslog drain.
 Furthermore, this helps on sharing service instances across CF orgs or landscapes.
 
@@ -103,7 +153,7 @@ The extension requires four fields in the user-provided service credentials and
 | `ingest-otlp-cert`| The mTLS client certificate in PEM format matching the client key. Line breaks as `\n`. |
 | `server-ca` | The trusted mTLS server certificate in PEM format. Line breaks as `\n`. |
 
-If you have a SAP Cloud Logging service key, you can generate the required JSON file with jq:
+If you have a [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) service key, you can generate the required JSON file with jq:
 
 ```bash
 cf service-key cls test \
@@ -119,5 +169,17 @@ Using this file, you can create the required user-provided service:
 ```
 
 Note, that you can easily feed arbitrary credentials to the extension.
-It does not need to be SAP Cloud Logging.
+It does not need to be [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 You can even change the tag using the configuration parameters of the extension.
+
+## Implementation Differences between Cloud-Logging and OTLP Exporter
+
+The `cloud-logging` exporter provided by this extension is a facade for the `OtlpGrpcExporter` provided by the OpenTelemetry Java Agent, just like the `otlp` exporter.
+The difference is just during the bootstrapping phase.
+The main differences are:
+
+* The `cloud-logging` exporter will send data to all found bindings to SAP Cloud Logging.
+The auto-instrumentation of the `otlp` exporter will just configure the first binding it finds priotizing user-provided services.
+* The `otlp` configuration will write the required certificates and keys to temporary files, which are deleted when the JVM is shut down. The `cloud-logging` exporter will keep the secrets in memory.
+* Since the `otlp` exporter is the default for traces and metrics, just attaching the extension and binding to Cloud Logging will result in metrics and traces being forwarded. 
+The `cloud-logging` exporter needs to be configured explictly as does the `otlp` exporter for logs.

sample-spring-boot/manifest-otel-javaagent.yml

@@ -15,7 +15,7 @@ applications:
       LOG_REFERER: false
       JBP_CONFIG_COMPONENTS: "jres: ['com.sap.xs.java.buildpack.jre.SAPMachineJRE']"
       JBP_CONFIG_SAP_MACHINE_JRE: '{ use_offline_repository: false, version: 17.+ }'
-      JBP_CONFIG_JAVA_OPTS: '[from_environment: false, java_opts: ''-javaagent:BOOT-INF/lib/opentelemetry-javaagent-1.31.0.jar -Dotel.javaagent.extensions=BOOT-INF/lib/cf-java-logging-support-opentelemetry-agent-extension-3.8.0.jar -Dotel.logs.exporter=otlp -Dotel.instrumentation.logback-appender.experimental.capture-mdc-attributes=* -Dotel.instrumentation.logback-appender.experimental.capture-key-value-pair-attributes=true -Dotel.instrumentation.logback-appender.experimental.capture-code-attributes=true -Dotel.instrumentation.logback-appender.experimental-log-attributes=true -Dotel.experimental.resource.disabled-keys=process.command_line,process.command_args,process.executable.path'']'
+      JBP_CONFIG_JAVA_OPTS: '[from_environment: false, java_opts: ''-javaagent:BOOT-INF/lib/opentelemetry-javaagent-1.31.0.jar -Dotel.javaagent.extensions=BOOT-INF/lib/cf-java-logging-support-opentelemetry-agent-extension-3.8.0.jar -Dotel.logs.exporter=cloud-logging,otlp -Dotel.instrumentation.logback-appender.experimental.capture-mdc-attributes=* -Dotel.instrumentation.logback-appender.experimental.capture-key-value-pair-attributes=true -Dotel.instrumentation.logback-appender.experimental.capture-code-attributes=true -Dotel.instrumentation.logback-appender.experimental-log-attributes=true -Dotel.experimental.resource.disabled-keys=process.command_line,process.command_args,process.executable.path'']'
     services:
       - cls