Demonstrate file based configuration of RuleBasedRoutingSampler

[jack-berg]

This demonstrates how file based configuration can be used to easily configure RuleBasedRoutingSampler to drop spans with match a particular pattern. It requires changes to several repos, but if the changes were adopted and file configuration and RuleBasedRoutingSampler were enabled in the java agent, this would effectively solve Exclude URLs from Tracing (the most upvoted open issue in opentelemetry-java-instrumentation).

Other required changes:

• Add support for loading SPIs to file configuration, like this commit which adds support for custom samplers
• Add RuleBasedRoutingSampler implementation of ConfigurableSamplerProvider SPI, like this commit. This requires that we define a configuration scheme, which becomes a lot more user friendly with file configuration with supports complex types.
• Extend ConfigProperties to support complex types, as prototyped here. This ensures that we RuleBasedRoutingSampler's configuration scheme doesn't need to be flattened.

If we do all this, we can reference RuleBasedRoutingSampler in file configuration, and specify its configuration with an ergonomic scheme that matches the programatic API:

# other configuration omitted for brevity
tracer_provider
  sampler:
    parent_based:
      root:
        rule_based_routing_sampler:
          fallback: always_on
          span_kind: SERVER
          drop_rules:
            - attribute: http.target
              pattern: "/actuator.*"
            - attribute: http.target
              pattern: "/foo"

I have all these changes made locally and published to maven local. I've updated the example to include spring boot, and the spring boot actuator which adds a health check endpoint (and various other /actuator.* endpoints) that users often want to drop.

If I run the app, and curl the health check endpoint we wish to drop (curl localhost:8080/actuator/health) and an application endpoint we wish to keep (curl localhost:8080/ping), we can see in the application output that we see the desired outcome:

11:22:36.459 [main] INFO io.opentelemetry.examples.fileconfig.Application - SDK config: OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=null, attributes={service.name="file-configuration-example", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.31.0-SNAPSHOT"}}, spanLimitsSupplier=SpanLimitsValue{maxNumberOfAttributes=128, maxNumberOfEvents=128, maxNumberOfLinks=128, maxNumberOfAttributesPerEvent=128, maxNumberOfAttributesPerLink=128, maxAttributeValueLength=2147483647}, sampler=ParentBased{root:RuleBasedRoutingSampler{rules=[SamplingRule{attributeKey=http.target, delegate=AlwaysOffSampler, pattern=/actuator.*}, SamplingRule{attributeKey=http.target, delegate=AlwaysOffSampler, pattern=/foo}], kind=SERVER, fallback=AlwaysOnSampler},remoteParentSampled:AlwaysOnSampler,remoteParentNotSampled:AlwaysOffSampler,localParentSampled:AlwaysOnSampler,localParentNotSampled:AlwaysOffSampler}, spanProcessor=BatchSpanProcessor{spanExporter=LoggingSpanExporter{}, scheduleDelayNanos=5000000000, maxExportBatchSize=512, exporterTimeoutNanos=30000000000}}, meterProvider=SdkMeterProvider{clock=SystemClock{}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.31.0-SNAPSHOT"}}, metricReaders=[], metricProducers=[], views=[]}, loggerProvider=SdkLoggerProvider{clock=SystemClock{}, resource=Resource{schemaUrl=null, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.31.0-SNAPSHOT"}}, logLimits=LogLimits{maxNumberOfAttributes=128, maxAttributeValueLength=2147483647}, logRecordProcessor=NoopLogRecordProcessor}, propagators=DefaultContextPropagators{textMapPropagator=MultiTextMapPropagator{textMapPropagators=[W3CTraceContextPropagator, W3CBaggagePropagator]}}}

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::               (v2.7.16)

2023-10-03 11:22:36.658  INFO 54858 --- [           main] i.o.examples.fileconfig.Application      : Starting Application using Java 17.0.3 on H6HJ6Q7NWV with PID 54858 (/Users/jberg/code/open-telemetry/opentelemetry-java-docs/file-configuration/build/classes/java/main started by jberg in /Users/jberg/code/open-telemetry/opentelemetry-java-docs/file-configuration)
2023-10-03 11:22:36.659  INFO 54858 --- [           main] i.o.examples.fileconfig.Application      : No active profile set, falling back to 1 default profile: "default"
2023-10-03 11:22:37.079  INFO 54858 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2023-10-03 11:22:37.084  INFO 54858 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2023-10-03 11:22:37.084  INFO 54858 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.80]
2023-10-03 11:22:37.135  INFO 54858 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2023-10-03 11:22:37.136  INFO 54858 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 462 ms
2023-10-03 11:22:37.373  INFO 54858 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2023-10-03 11:22:37.393  INFO 54858 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2023-10-03 11:22:37.401  INFO 54858 --- [           main] i.o.examples.fileconfig.Application      : Started Application in 0.909 seconds (JVM running for 1.239)
2023-10-03 11:22:57.574  INFO 54858 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-10-03 11:22:57.574  INFO 54858 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-10-03 11:22:57.574  INFO 54858 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 0 ms
2023-10-03 11:23:02.433  INFO 54858 --- [nio-8080-exec-3] i.o.examples.fileconfig.Controller       : A sample log message!
2023-10-03 11:23:06.466  INFO 54858 --- [_WorkerThread-1] i.o.e.logging.LoggingSpanExporter        : 'doWork' : 3bcf7160504bc63adb5cf95ef19b589a 82e0b9ff436898a6 INTERNAL [tracer: io.opentelemetry.examples.fileconfig.Application:] {}
2023-10-03 11:23:06.467  INFO 54858 --- [_WorkerThread-1] i.o.e.logging.LoggingSpanExporter        : 'GET /ping' : 3bcf7160504bc63adb5cf95ef19b589a d7f90c4a287ea4f7 SERVER [tracer: io.opentelemetry.spring-webmvc-5.3:1.29.0-alpha] AttributesMap{data={user_agent.original=curl/7.88.1, net.host.name=localhost, http.target=/ping, net.sock.peer.addr=127.0.0.1, http.status_code=200, net.protocol.name=http, http.method=GET, http.response_content_length=4, net.host.port=8080, net.protocol.version=1.1, http.scheme=http, net.sock.host.addr=127.0.0.1, net.sock.peer.port=50519, http.route=/ping}, capacity=128, totalAddedValues=14}

Notice I log the OpenTelemetrySdk#toString() at application start and we see RuleBasedRoutingSampler with our desired config.

Notice that we only see the LoggingSpanExporter print the span for http.target=/ping, and NOT for http.target=/actuator/health.

Let me know what you think! Hoping we can use this as a catalyst for landing these updates, and for ultimately packaging file based configuration and rule based routing sampler with the java agent.

[nedcerneckis]

Very interesting @jack-berg. Thanks for putting this together. I am one of many users of OTEL who wanted this feature implemented to exclude certain URL endpoints.

It requires changes to several repos

I am new to this project so apologies for my lack of knowledge but I'm eager to help contribute to the relevant repositories to support the integration of this feature. Could you briefly elaborate on which specific repositories require these changes to implement this feature, or any other guidance for a new contributor like myself. Thanks!

[jack-berg]

Could you briefly elaborate on which specific repositories require these changes to implement this feature, or any other guidance for a new contributor like myself.

Sure - I left a list of the required pieces in the PR description. A lot of the work is done, but we need review on the critical path. The critical path is something like:

1. Review and merge support for accessing complex types in ConfigProperties: Extended config properties opentelemetry-java#5912
2. Extend opentelemetry-java file configuration to load extension components (i.e. samplers, exporters, etc) via SPI. Pass ExtendedConfigProperties to the SPIs to give the opportunity for extension components to access complex types as provided in the YAML config file. I have this prototyped in this commit, but have been waiting to finish it until Extended config properties opentelemetry-java#5912 is merged.
3. Publish a new version of opentelemetry-java with these changes.
4. Update opentelemetry-java-contrib's rule based sampler to implement ConfigurableSamplerProvider, extracting config from ExtendedConfigProperties and using it to configure a RuleBasedRoutingSampler instance. I have this prototyped in this commit.

[nedcerneckis]

Sounds like there is a lot of work already done, thanks @jack-berg! Looks like there is not much to do code-wise.