[enhance][pulsar] add apache-pulsar client support

[tjiuming]

Fix: #2107

Motivation:
Support apache pulsar client from version 2.8.0 to lastest.

[tjiuming]

Tests to be completed.

[tjiuming]

server.log

Here is the server log.

[laurit]

@tjiuming

[otel.javaagent 2022-04-24 17:46:23:622 +0800] [main] WARN muzzleMatcher - Instrumentation skipped, mismatched references were found: apache-pulsar [class io.opentelemetry.javaagent.instrumentation.pulsar.PulsarInstrumentationModule] on jdk.internal.loader.ClassLoaders$AppClassLoader@73d16e93
[otel.javaagent 2022-04-24 17:46:23:628 +0800] [main] WARN muzzleMatcher - -- io.opentelemetry.javaagent.instrumentation.pulsar.ConsumerImplInstrumentation:51 Missing class io.opentelemetry.javaagent.extension.instrumentation.TypeTransformer
[otel.javaagent 2022-04-24 17:46:23:628 +0800] [main] WARN muzzleMatcher - -- io.opentelemetry.javaagent.instrumentation.pulsar.ConsumerImplInstrumentation:0 Missing class io.opentelemetry.javaagent.extension.instrumentation.TypeInstrumentation
[otel.javaagent 2022-04-24 17:46:23:628 +0800] [main] WARN muzzleMatcher - -- io.opentelemetry.javaagent.instrumentation.pulsar.ConsumerImplInstrumentation:44 Missing class net.bytebuddy.matcher.ElementMatchers
[otel.javaagent 2022-04-24 17:46:23:628 +0800] [main] WARN muzzleMatcher - -- io.opentelemetry.javaagent.instrumentation.pulsar.ConsumerImplInstrumentation:51 Missing class net.bytebuddy.matcher.ElementMatcher
[otel.javaagent 2022-04-24 17:46:23:628 +0800] [main] WARN muzzleMatcher - -- io.opentelemetry.javaagent.instrumentation.pulsar.ConsumerImplInstrumentation:44 Missing class net.bytebuddy.matcher.ElementMatcher$Junction

muzzle is our safety checker that prevents runtime failures caused by transformation using types, methods etc that are not present. Your main problem is

private static final Tracer TRACER = PulsarTelemetry.tracer();

in ConsumerImplInstrumentation. ConsumerImplInstrumentation depends on bytebuddy classes that are only available inside agent code. In your advice code you should not use the instrumentation class.
It would be best if you'd study some of our existing messaging instrumentations and instead of using Tracer directly used our instrumenter api.

[mateuszrzeszutek]

Hey @tjiuming ,

Please take a look at our contributing docs about adding a new instrumentation, writing an InstrumentationModule and using the Instrumenter API. You should be able to find answers to some of your questions there.
You can use the messaging package in from the Instrumenter API to implement telemetry for Pulsar - by using it you'll implement the OpenTelemetry messaging semantic conventions (well, for the most part) in your instrumentation. As Lauri has mentioned, you take a look at existing messaging instrumentations (e.g. spring-kafka, jms, rabbitmq, etc) and see how they're implemented.

[tjiuming]

@laurit @mateuszrzeszutek Many thanks for your reply. With your help, I’m already fixed these problems.
I‘m not familiar with byte-buddy api due to I was focus on skywalking agent plugin core mostly. My pulsar advices doesn't work because of incorrect byte-buddy annotation usages, thanks for your inspire.
Instrumentation api is more complex than Tracer api a lot for me, maybe I'll change it in the future, not currently.
Thanks for your help again!

[tjiuming]

[tjiuming]

@laurit Many thanks for your advice

[tjiuming]

Tests to be completed.

[tjiuming]

@trask @laurit Tests completed, PTAL

[mateuszrzeszutek]

Hey @tjiuming ,

Can you refactor your instrumentation so that is uses the Instrumenter API instead of plain Tracer? Currently all instrumentations in this repository make use of the Instrumenter API, as it's way easier for us to maintain them this way.

Also, please make sure that you implement appropriate OTel semantic convention -- in case of Pulsar I believe it's messaging semconv. If you use the Instrumenter API, there are several utilities that make implementing OTel semantic conventions way easier.

If you need to add custom attributes that are not present in the OTel messaging conventions then the usual pattern is to add a configuration flag, disabled by default, that controls whether they will be added to the produced telemetry or not. For example: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/hystrix-1.4/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/hystrix/HystrixSingletons.java#L24-L27

[trask]

hi @tjiuming! any objection to closing this (just trying to clean up the backlog a little), and you can re-open when you start working on it again? I think if I close it you won't be able to re-open it, but if you close it, I think you will be able to re-open yourself. And of course you can always ping on the closed PR and one of us will re-open it if that fails.

[tjiuming]

@trask @mateuszrzeszutek @laurit I‘ve improved and fixed the tests, and all the CI checks passed, cloud you please help review?

[KevinLiLu]

@tjiuming @trask @mateuszrzeszutek Any update on this? We are currently using OTel interceptors but this auto-instrumentation approach would be really awesome.

[tjiuming]

@KevinLiLu This PR is waiting for review

[trask]

hi @tjiuming sorry for the long delay, we will try to get to this in the next week

[tjiuming]

hi @tjiuming sorry for the long delay, we will try to get to this in the next week

Never mind, thanks!

[trask]

thx @tjiuming!

[tjiuming]

@trask I've fixed them, PTAL

[trask]

@tjiuming can you check the test failures when you have a chance? thx!

[trask]

👍

[trask]

did you try MessageListener and it didn't work? there is some magic that should enhance lambdas: #4182

hasSuperType is a more expensive matcher compared to named however, so the current implementation could still be better

[tjiuming]

I think so, too. The current implementation works fine, so it needn't to change.

[tjiuming]

Many thanks for your review!

[tjiuming]

@trask Could this PR be merged?