From ce369c783ac2eeb36a22794241bddb11c4dfa333 Mon Sep 17 00:00:00 2001 From: Kabir Khan Date: Fri, 18 Oct 2024 15:52:14 +0100 Subject: [PATCH 1/4] [WFLY-19835] Integrate OpenTracing with MicroProfile Reactive Messaging Kafka Connector --- ...le_reactive_messaging_otel_with_kafka.adoc | 179 ++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc diff --git a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc new file mode 100644 index 00000000..8fbb7a9c --- /dev/null +++ b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc @@ -0,0 +1,179 @@ +--- +// Add any category for this proposal as a yaml list, e.g. +// - core +// - management +// if missing, add it to _data/wildfly-categories and use its id +categories: +- microprofile +// Specify the stability level of the feature. +// Values can be one of: experimental preview community default +stability-level: default +// Specify the Feature Development tracker issue for the feature. +// This must be an issue tracked in https://github.com/orgs/wildfly/projects/7/views/1. +// To create a Feature Development tracker issue, go to https://github.com/wildfly/wildfly-proposals/issues/new/choose +// and select 'Feature Development' +issue: https://github.com/wildfly/wildfly-proposals/issues/636 +// Provide the github ids of the members of the feature team, organized by role. +// Provide a single id for the 'assignee' role. Use a yaml list for the 'sme' and +// 'outside-perspective' roles, even if there is only one person in a role. +feature-team: + developer: kabir + sme: + - + outside-perspective: + - +// If this issue tracks the promotion to a higher stability level of a previously +// completed feature, provide the URL of the https://github.com/wildfly/wildfly-proposals/issues +// issue that was used to track the previous feature. +promotes: +// This should be blank during initial development of a feature. It may be used +// after the feature is completed if a subsequent issue is field to track promotion +// of this feature to a higher stability level +promoted-by: +--- +// Other properties +:connector: Kafka +:connector-lower: kafka +:issue-id: WFLY-19835 + += Integrate OpenTelemetry with the MP Reactive Messaging {connector} connector +:author: Kabir Khan +:email: your.email@redhat.com +:toc: left +:icons: font +:idprefix: +:idseparator: - + + + +== Overview + +SmallRye Reactive Messaging's {connector} connector allows for integration with OpenTelemetry tracing so that we can track messages sent to and received from {connector} via this connector. + +This is controlled via the MicroProfile Config property `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled`. At present this is set to `false` in a `ConfigSource` set up by the integration with an `ordinal` value set to the maximum value. The result of this is that the user can never enable tracing to happen (since any attempt to set this property to `true` would be overridden by the mentioned `ConfigSource`). + + + +=== User Stories + +Users wish to be able to trace messages sent to/from {connector} via reactive messaging using MicroProfile Telemetry. + +== Issue Metadata + +* https://issues.redhat.com/browse/{issue-id}[{issue-id}] - Integrate OpenTelemetry with the MP Reactive Messaging {connector} connector + +=== Related Issues + +* https://issues.redhat.com/browse/WFLY-19846[WFLY-19846] - Promote MicroProfile Telemetry 2.0 to WildFly Standard + + +=== Affected Projects or Components +* https://github.com/wildfly/wildfly/ - All integration work is done here +* https://github.com/smallrye/smallrye-reactive-messaging - contains the implementation of the Reactive Messaging connectors. No work is expected to be needed here, unless bugs are found. + +=== Other Interested Projects +None to my knowledge + +=== Relevant Installation Types + +* Traditional standalone server (unzipped or provisioned by Galleon) +* OpenShift Source-to-Image (S2I) +* Bootable jar + +== Requirements + +It must be possible to turn the tracing on. To do this a resource called `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` is introduced. + +This resource contains an attribute called `{connector-lower}-connector`, which allows us to control if and how tracing is enabled. It is an enumeration with the following values: +* `NEVER` - This is the default value of the attribute, and also the value that will be used if the `opentelemetry-tracing=config` child resource is not present in the subsystem. This is the same behaviour that has been available today, i.e. `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is forced to be `false` in such a way that user's MicroProfile Config properties can never set it to `true`. +* `OFF` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is set to `false`, but a user can override it to set it to `true`. +* `ON` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is set to `true`, but a user can override it to set it to `false`. +* `ALWAYS` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is forced to be `true` in such a way that user's MicroProfile Config properties can never set it to `false` + +As mentioned in the link:#Overview[Overview], `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` controls whether tracing happens. It is a boolean, and its values have these meanings: + +* `false` - OpenTelemetry tracing is not enabled for the {connector} connector +* `true` - OpenTelemetry tracing is enabled for the {connector} connector + +When tracing is enabled, and the OpenTelemetry subsystem is configured to connect to a tracing collector, the traces should be available from there. + +=== Changed requirements + +N/A + +=== Non-Requirements + +N/A + +=== Future Work + +N/A + +== Backwards Compatibility + +If not configured, the default is the same as the existing behaviour, which is to not enable tracing, and to not allow users to enable it. + +=== Default Configuration + +No change + +=== Importing Existing Configuration + +Existing configurations will have the same behaviour as today. i.e. tracing is not enabled, and can not be turned on. + +=== Deployments + +No change. + +=== Interoperability + +N/A + +== Implementation Plan + +A single pull request to WildFLy will be created containing everything. + +== Admin Clients +It is a simple resource with an attribute, so no work is needed in JBoss CLI/HAL + +== Security Considerations + +None, beyond the consideration that the collector of the traces will receive a lot of the data, and so must be secured as well. + +[[test_plan]] +== Test Plan + +____ +//// +Depending on the stability level, the test plan required may vary. see below: +//// + +** Experimental - No test plan is required. Basic unit / integration tests should be added during development. + +** Preview - a brief high-level description of the testing approach should be added here, including types of tests added (unit, integration, smoke, component, subsystem, etc.) Note that not all test types are required for a particular feature, so include a description of what is being tested and the approach chosen to perform the testing. + +** Community - this level should include everything in the 'Preview' stability level, plus the following additional testing as relevant: +*** Manual tests: briefly describe checks to be performed during one-time exploratory testing. The purpose of this testing is to check corner cases and other cases that are not worth implementing as automated tests. Typical checks are: bad configurations are easy to reveal, attribute descriptions and error messages are clear, names are descriptive and consistent with similar resources, default values are reasonable. +*** Miscellaneous checks: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. Fully qualified test case names should be provided along with a brief description of what the test is doing. +*** Integration tests - at the 'Community' stability level, complete integration tests should be provided. +*** Compatibility tests - if backwards compatibility is relevant to the feature, then describe how the testing is performed. + +** Default - This stability level is reserved and requires approval by a professional Quality Engineer with subject matter expertise. + +Two main tests will be created in WildFly's https://github.com/wildfly/wildfly/tree/main/testsuite/integration/microprofile[testsuite/integration/microprofile] module: + +1. We test all the combinations of the `{connector-lower}-connector` in the new + `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` resource, and values of `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` set via the user's MicroProfile Config, and make sure that the resulting value of `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` in the final MicroProfile Config used for the deployment has the value specified in the link:#Requirements[Requirements] section. +2. Tesing tracing +a. Ensure that tracing does not happen when tracing is disabled +// This is currently a little bit vague, since I still have no idea how OpenTracing actually works, and have not been able to get my POC to work yet! +b. Ensure that tracing happens when tracing is enabled, and that the traces contain the expected values. + +== Community Documentation + +The current https://github.com/wildfly/wildfly/blob/main/docs/src/main/asciidoc/_admin-guide/subsystem-configuration/MicroProfile_Reactive_Messsaging_SmallRye.adoc[MicroProfile Reactive Messaging Document] will be enhanced to cover the new resource and properties, and what this means. + + +== Release Note Content + +You can now enable OpenTelemetry tracing for the MicroProfile Reactive Messaging {connector} connector. \ No newline at end of file From 986a022de858574f39c663838f3a8fd86c46fb65 Mon Sep 17 00:00:00 2001 From: Kabir Khan Date: Tue, 29 Oct 2024 19:24:47 +0000 Subject: [PATCH 2/4] [WFLY-19835] Cleanup --- ...le_reactive_messaging_otel_with_kafka.adoc | 21 ++----------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc index 8fbb7a9c..102b834e 100644 --- a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc +++ b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc @@ -85,6 +85,7 @@ None to my knowledge It must be possible to turn the tracing on. To do this a resource called `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` is introduced. This resource contains an attribute called `{connector-lower}-connector`, which allows us to control if and how tracing is enabled. It is an enumeration with the following values: + * `NEVER` - This is the default value of the attribute, and also the value that will be used if the `opentelemetry-tracing=config` child resource is not present in the subsystem. This is the same behaviour that has been available today, i.e. `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is forced to be `false` in such a way that user's MicroProfile Config properties can never set it to `true`. * `OFF` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is set to `false`, but a user can override it to set it to `true`. * `ON` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is set to `true`, but a user can override it to set it to `false`. @@ -131,7 +132,7 @@ N/A == Implementation Plan -A single pull request to WildFLy will be created containing everything. +A single pull request to WildFly will be created containing everything. == Admin Clients It is a simple resource with an attribute, so no work is needed in JBoss CLI/HAL @@ -142,24 +143,6 @@ None, beyond the consideration that the collector of the traces will receive a l [[test_plan]] == Test Plan - -____ -//// -Depending on the stability level, the test plan required may vary. see below: -//// - -** Experimental - No test plan is required. Basic unit / integration tests should be added during development. - -** Preview - a brief high-level description of the testing approach should be added here, including types of tests added (unit, integration, smoke, component, subsystem, etc.) Note that not all test types are required for a particular feature, so include a description of what is being tested and the approach chosen to perform the testing. - -** Community - this level should include everything in the 'Preview' stability level, plus the following additional testing as relevant: -*** Manual tests: briefly describe checks to be performed during one-time exploratory testing. The purpose of this testing is to check corner cases and other cases that are not worth implementing as automated tests. Typical checks are: bad configurations are easy to reveal, attribute descriptions and error messages are clear, names are descriptive and consistent with similar resources, default values are reasonable. -*** Miscellaneous checks: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. Fully qualified test case names should be provided along with a brief description of what the test is doing. -*** Integration tests - at the 'Community' stability level, complete integration tests should be provided. -*** Compatibility tests - if backwards compatibility is relevant to the feature, then describe how the testing is performed. - -** Default - This stability level is reserved and requires approval by a professional Quality Engineer with subject matter expertise. - Two main tests will be created in WildFly's https://github.com/wildfly/wildfly/tree/main/testsuite/integration/microprofile[testsuite/integration/microprofile] module: 1. We test all the combinations of the `{connector-lower}-connector` in the new From 4da86f27de2395f2022a4b709fe5f89d00335683 Mon Sep 17 00:00:00 2001 From: Kabir Khan Date: Tue, 5 Nov 2024 18:29:13 +0000 Subject: [PATCH 3/4] [WFLY-19835] Update for channel level properties --- ...le_reactive_messaging_otel_with_kafka.adoc | 47 ++++++++++++------- 1 file changed, 31 insertions(+), 16 deletions(-) diff --git a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc index 102b834e..99bf14df 100644 --- a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc +++ b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc @@ -50,8 +50,18 @@ promoted-by: SmallRye Reactive Messaging's {connector} connector allows for integration with OpenTelemetry tracing so that we can track messages sent to and received from {connector} via this connector. -This is controlled via the MicroProfile Config property `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled`. At present this is set to `false` in a `ConfigSource` set up by the integration with an `ordinal` value set to the maximum value. The result of this is that the user can never enable tracing to happen (since any attempt to set this property to `true` would be overridden by the mentioned `ConfigSource`). +In SmallRye Reactive Messaging, a set of MicroProfile Config properties are used to control whether tracing happens on a reactive messaging channel. These are: +* `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` - This is a connector level property. At present this is set to `false` in a `ConfigSource` set up by the integration with an `ordinal` value set to the maximum value. The result of this is that the user can never enable tracing to happen (since any attempt to set this property to `true` would be overridden by the mentioned `ConfigSource`). +* The above property may in turn be overridden by the following channel level properties. +** `mp.messaging.outgoing..tracing-enabled` - Channel level property, controlling the OpenTelemetry tracing integration for an outgoing channel. +** `mp.messaging.incoming..tracing-enabled` - Channel level property, controlling the OpenTelenetry integration for an incoming channel. + +The way the SmallRye Reactive Messaging configuration system works is that it will first check for relevant channel level properties. If those are not present, it will look for the related connector level property, and if that is not available it will use a default. + +Internally in SmallRye Reactive Messaging, the default value for the tracing integration is that it is always enabled for a channel if not specified with above properties. + +This RFE is about allowing users to enable tracing via OpenTelemetry in WildFly, and also to be able to continue to ban it. === User Stories @@ -77,27 +87,29 @@ None to my knowledge === Relevant Installation Types * Traditional standalone server (unzipped or provisioned by Galleon) +* Managed domain * OpenShift Source-to-Image (S2I) * Bootable jar == Requirements -It must be possible to turn the tracing on. To do this a resource called `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` is introduced. +It must be possible to turn the tracing on. To do this a resource called `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` is introduced. This adds a capability requirement on the MicroProfile Telemetry subsystem, which in turn requires the OpenTelemetry subsystem. The OpenTelemetry subsystem configures how tracing happens. This resource contains an attribute called `{connector-lower}-connector`, which allows us to control if and how tracing is enabled. It is an enumeration with the following values: -* `NEVER` - This is the default value of the attribute, and also the value that will be used if the `opentelemetry-tracing=config` child resource is not present in the subsystem. This is the same behaviour that has been available today, i.e. `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is forced to be `false` in such a way that user's MicroProfile Config properties can never set it to `true`. -* `OFF` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is set to `false`, but a user can override it to set it to `true`. -* `ON` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is set to `true`, but a user can override it to set it to `false`. -* `ALWAYS` - `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` is forced to be `true` in such a way that user's MicroProfile Config properties can never set it to `false` +* `NEVER` - This is the default value of the attribute, and also the value that will be used if the `opentelemetry-tracing=config` child resource is not present in the subsystem. This is the same behaviour that has been available today, i.e. the MicroProfile Config properties mentioned in the link:#overview[Overview] will not have any effect, and tracing will be disabled. +* `OFF` - By default tracing will be disabled, but a user can use the MicroProfile Config properties from the link:#overview[Overview] to turn on tracing for a connector or individual channel in an application. +* `ON` - Similar to `OFF` but the default is to enable tracing for all channels, unless overridden for the application with the MicroProfile Config properties from the link:#overview[Overview]. +* `ALWAYS` - Tracing is enabled for all channels, and can not be turned off with the MicroProfile Config properties from the link:#overview[Overview]. -As mentioned in the link:#Overview[Overview], `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` controls whether tracing happens. It is a boolean, and its values have these meanings: +The MicroProfile Config properties in the link:#Overview[Overview], are all booleans, and the values have these meanings: -* `false` - OpenTelemetry tracing is not enabled for the {connector} connector -* `true` - OpenTelemetry tracing is enabled for the {connector} connector +* `false` - OpenTelemetry tracing is not enabled +* `true` - OpenTelemetry tracing is enabled When tracing is enabled, and the OpenTelemetry subsystem is configured to connect to a tracing collector, the traces should be available from there. + === Changed requirements N/A @@ -116,7 +128,7 @@ If not configured, the default is the same as the existing behaviour, which is t === Default Configuration -No change +No change, and the behaviour is the same as before, which is to not enable tracing. === Importing Existing Configuration @@ -143,18 +155,21 @@ None, beyond the consideration that the collector of the traces will receive a l [[test_plan]] == Test Plan + +A unit test will be added in WildFly's `wildfly-microprofile-reactive-messaging-config` to make sure that the combinations of the new `{connector-lower}-connector` attribute and the MicroProfile Config properties controlling tracing yield the expected final value (e.g. if `{connector-lower}-connector=NEVER`, we check that the resulting value is always `false` no matter what we specify for the MicroProfile Config properties). + Two main tests will be created in WildFly's https://github.com/wildfly/wildfly/tree/main/testsuite/integration/microprofile[testsuite/integration/microprofile] module: 1. We test all the combinations of the `{connector-lower}-connector` in the new - `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` resource, and values of `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` set via the user's MicroProfile Config, and make sure that the resulting value of `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` in the final MicroProfile Config used for the deployment has the value specified in the link:#Requirements[Requirements] section. -2. Tesing tracing -a. Ensure that tracing does not happen when tracing is disabled -// This is currently a little bit vague, since I still have no idea how OpenTracing actually works, and have not been able to get my POC to work yet! -b. Ensure that tracing happens when tracing is enabled, and that the traces contain the expected values. + `subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config` resource, and values of `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` set via the user's MicroProfile Config, and make sure that the resulting value of `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` in the final MicroProfile Config used for the deployment has the resulting value specified in the link:#Requirements[Requirements] section. We only do this for the connector level attribute since the same code is used to handle the value of `{connector-lower}-connector` at runtime. The mentioned unit test has more coverage. +2. Testing tracing. The aim here is to test some combinations, and not all. +a. Ensure that tracing does not happen when tracing is disabled by setting `{connector-lower}-connector=NEVER` +b. Ensure that tracing happens when `{connector-lower}-connector=OFF` but is turned on for the deployment by specifying `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled=true`. Both sent and received messages will be traced. +c. When `{connector-lower}-connector=ON`, and `mp.messaging.incoming..tracing-enabled=false`, tracing will onlyt happen on the outgoing channel == Community Documentation -The current https://github.com/wildfly/wildfly/blob/main/docs/src/main/asciidoc/_admin-guide/subsystem-configuration/MicroProfile_Reactive_Messsaging_SmallRye.adoc[MicroProfile Reactive Messaging Document] will be enhanced to cover the new resource and properties, and what this means. +The current https://github.com/wildfly/wildfly/blob/main/docs/src/main/asciidoc/_admin-guide/subsystem-configuration/MicroProfile_Reactive_Messsaging_SmallRye.adoc[MicroProfile Reactive Messaging Document] will be enhanced to cover the new resource and its attributes, and how those values combine with the relevant MicroProfile Config Properties to control whether OpenTelemetry tracing happens. == Release Note Content From 9fed28fc129abcc6e56fa80996e52a4a05240c28 Mon Sep 17 00:00:00 2001 From: Kabir Khan Date: Wed, 4 Dec 2024 12:05:24 +0000 Subject: [PATCH 4/4] [WFLY-19835] Describe tracing resolution better --- ...ofile_reactive_messaging_otel_with_kafka.adoc | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc index 99bf14df..6b986374 100644 --- a/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc +++ b/microprofile/WFLY-19835_microprofile_reactive_messaging_otel_with_kafka.adoc @@ -98,8 +98,8 @@ It must be possible to turn the tracing on. To do this a resource called `subsys This resource contains an attribute called `{connector-lower}-connector`, which allows us to control if and how tracing is enabled. It is an enumeration with the following values: * `NEVER` - This is the default value of the attribute, and also the value that will be used if the `opentelemetry-tracing=config` child resource is not present in the subsystem. This is the same behaviour that has been available today, i.e. the MicroProfile Config properties mentioned in the link:#overview[Overview] will not have any effect, and tracing will be disabled. -* `OFF` - By default tracing will be disabled, but a user can use the MicroProfile Config properties from the link:#overview[Overview] to turn on tracing for a connector or individual channel in an application. -* `ON` - Similar to `OFF` but the default is to enable tracing for all channels, unless overridden for the application with the MicroProfile Config properties from the link:#overview[Overview]. +* `OFF` - By default tracing will be disabled, but a user can use the MicroProfile Config properties from the link:#overview[Overview] to turn on tracing for a connector or individual channel in an application. We will discuss how the properties are resolved below. +* `ON` - Similar to `OFF` but the default is to enable tracing for all channels, unless overridden for the application with the MicroProfile Config properties from the link:#overview[Overview]. We will discuss how the properties are resolved below. * `ALWAYS` - Tracing is enabled for all channels, and can not be turned off with the MicroProfile Config properties from the link:#overview[Overview]. The MicroProfile Config properties in the link:#Overview[Overview], are all booleans, and the values have these meanings: @@ -109,6 +109,18 @@ The MicroProfile Config properties in the link:#Overview[Overview], are all boo When tracing is enabled, and the OpenTelemetry subsystem is configured to connect to a tracing collector, the traces should be available from there. +=== Calculating whether tracing should happen + +If the `{connector-lower}-connector` attribute is `NEVER` or `ALWAYS`, tracing will always be totally disabled or enabled, and the MicroProfile Config properties from the link:#overview[Overview] will have no effect. + +Thus, they only work if the tracing is `{connector-lower}-connector` is `OFF` or `ON`. In this case the resolution works as follows for each channel: + +* If the channel is a {connector} channel it will check the channel level `mp.messaging.incoming.[channel-name].tracing-enabled` or `mp.messaging.outgoing.[channel-name].tracing-enabled` property, depending on if the channel is incoming or outgoing. If this has a value the value will be used. Otherwise, we go on to the next step. +* We then check the `mp.messaging.connector.smallrye-{connector-lower}.tracing-enabled` connector level property for the {connector} connector. If this has a value the value will be used. Otherwise, we go on to the next step. +* Finally, we check the value of the `{connector-lower}-connector` attribute, and use its value. `OFF` returns `false`, and `ON` returns `true`. + +In each of the above steps, if the found value is `false`, tracing is not enabled. If it is `true`, tracing is enabled. + === Changed requirements