[jaeger-v2] Rethink components configuration to align with OTEL

[yurishkuro]

Right now several PRs that are trying to add support for standard storage backends to v2 are reusing the same configuration for storage as we had in v1. Now is a good time to rethink that configuration, in particular:

• better align with standard components of OTEL Collector. For example, the gRPC storage ([jaeger-v2] Add support for GRPC storarge #5228) is conceptually very similar to OTLP Exporter - they both send data over gRPC to a remote service. Our configuration is pretty slim, while OTLP Exporter's configuration has a lot more features, there is no reason why we cannot support them (would be even better if we could simply reuse OTEL's configuration utilities for gRPC)
• our storage configs have been accumulating over time, we can see if there are better logical groupings. For example, ES config is a flat list of everything, we can group many flags into sub-types so that the YAML config looks a lot more structured than using long field names like priority_dependencies_template
• ensure that we have sensible defaults. For example, in [jaeger-v2] add cassandra e2e integration tests #5398 basic options had to be added to Cassandra configuration, even though in v1 many of them are already defaulted to these values, so we're clearly missing defaulting mechanism. It's somewhat complicated by the fact that the storage extension cannot actually define the defaults in CreateDefaultConfig because at that point we don't known which storage will be used.

Scope

• Cassandra storage config -- [jaeger-v2] Align Cassandra Storage Config With OTEL #5928
• Elasticsearch / OpenSearch storage config -- [jaeger-v2] Refactor ElasticSearch/OpenSearch Storage Configurations To Align With OTEL #6059
• Badger storage config -- [jaeger-v2] Align Badger Storage Config With OTEL #5926
• Memory storage config -- [jaeger-v2] Align Memory Storage Config With OTEL #5923
• Kafka config -- [jaeger-v2]: Fix Kafka E2E Test To Use Default Environment Variables #6027
• grpc storage (e.g. issue [grpc storage]: Propagate tenant to grpc backend #6030) -- [jaeger-v2] Consolidate v1 and v2 Configs For GRPC Storage #6041
• Adaptive sampling processor / extension config -- [jaeger-v2] Refactor Adaptive Sampling Processor Configurations #6021
• Query service config -- [jaeger-v2] Align Query Service Config With OTEL #5996
• Collector-specific config -- [jaeger-v2] Map Collector Specific Flags To Corresponding OTEL Configurations #6040
• Utilize env var defaults available since OTEL v0.110 e.g.

  jaeger/cmd/jaeger/internal/integration/kafka_test.go

  Line 24 in c655e56

  // we can change the config to use topic: "${KAFKA_TOPIC:-jaeger-spans}"

Tasks / outcomes

For each of the above components, we need to ensure

• All config fields are tagged with mapstructure such that they can be addressable from YAML via "good" names
• Configs implement Validate() -- example
• Configs via YAML support the same defaults as in jaeger-v1
• Configs reuse standard elements of OTEL configs whenever possible, e.g. TLS, clientcfg, etc.
• Configuration looks "logical" and properly grouped, not flattened with long weird names (good example [Jaeger-V2] Add configurable index data layout and rollover frequency #5790)
• We create a migration guide document with a table that lists v1 CLI flags and the corresponding v2 config fields (example for Kafka)

How to work on this meta-issue

• pick a component / area
• create a dedicated ticket for it
• copy tasks / outcomes from above
• use the ticket for design discussions, updates, links to PRs

[mahadzaryab1]

@yurishkuro What is the status of this issue? Is there anything I can help with?

[yurishkuro]

yes, plenty. I just updated the description.

[mahadzaryab1]

@yurishkuro are the configs for v1 supposed to be in plugin/storage/*/options.go and the configs for v2 supposed to be in pkg/*/config.go?

[yurishkuro]

yes, that's accurate

[mahadzaryab1]

@yurishkuro how come some stores still have the v2 configuration living in plugin/storage/*? do we want to move those over?

[yurishkuro]

We are in the transition period. Moving them doesn't solve anything, having configs side by side is actually convenient. The long term goal is to only have v2 configurations.

Wwe started v2 by just wrapping v1 configs. For some backends we did preliminary work of cleaning up and making v2 the "primary" config, i.e. the Factory uses v2 config directly, but v1 config still drives the v1 cli flags and can be transformed into v2. We could continue using this pattern. The reason we want v2 configs to be primary is to ensure that all their settings are actually used by the storage implementation - there's less guarantee of that if v1 is primary and v2 is translated into v1.

But overall this ticket is about what v2 config look like, rather than where they are located.

[mahadzaryab1]

@yurishkuro sounds good! and where should the migration doc be created? is any public google doc fine? Or do we want to have them in a README in this repo?

[yurishkuro]

yes a google doc is fine

[mahadzaryab1]

@yurishkuro PR for memory storage component at #5925

[mahadzaryab1]

@yurishkuro PR for badger storage component at #5927

[mahadzaryab1]

@yurishkuro PR for Cassandra component ready for review at #5949.

[mahadzaryab1]

@yurishkuro PR for Query Service ready for review at #5998

[mahadzaryab1]

@yurishkuro we can link the following PRs to the issue description for badger:

• [jaeger-v2] Add Validation And Comments to Badger Storage Config #5927
• [jaeger-v2] Consolidate Options And NamespaceConfig For Badger Storage #5937

[mahadzaryab1]

@yurishkuro what is the Collector-specific config checklist item referring to?

[yurishkuro]

what is the Collector-specific config checklist item referring to?

if you run SPAN_STORAGE_TYPE=memory go run ./cmd/collector help there are a bunch of collector-specific flags printed that are not related to storage but to the collector itself. Most of them are related to http/grpc endpoints, but some could control the overall collector's behavior. Ideally we need to create a gdoc mapping all these flags to the corresponding OTEL config locations and if any are not supported decide what to do about them.

[mahadzaryab1]

what is the Collector-specific config checklist item referring to?

if you run SPAN_STORAGE_TYPE=memory go run ./cmd/collector help there are a bunch of collector-specific flags printed that are not related to storage but to the collector itself. Most of them are related to http/grpc endpoints, but some could control the overall collector's behavior. Ideally we need to create a gdoc mapping all these flags to the corresponding OTEL config locations and if any are not supported decide what to do about them.

@yurishkuro Got it! So for this part we just need to create a migration guide? No code changes?

[yurishkuro]

Perhaps, depends on what the comparison would show.

[mahadzaryab1]

@yurishkuro Thanks! Would you also be able to help me understand how to solve Utilize env var defaults one OTEL supports them in 0.110 ? How do we determine which things need to be defaulted? As well, where do these defaults needs to be utilized? Is it just in the docker compose setups?

[yurishkuro]

We had a number of nasty workarounds because we couldn't use defaults. For instance, Kafka e2e test needs to physically rewrite the config because we can't just have a var for encoding without default (it makes the config invalid by default). I think it's sufficient to just fix that to close the item, since we can add more vars/defaults incrementally as the needs arise.

[mahadzaryab1]

@yurishkuro we can link the following issues to the issue description as they are complete / in-review:

• [jaeger-v2] Refactor Adaptive Sampling Processor Configurations #6021
• [jaeger-v2]: Fix Kafka E2E Test To Use Default Environment Variables #6027

[yurishkuro]

For Kafka, we have this document.

[mahadzaryab1]

For Kafka, we have this document.

@yurishkuro Did you want me to link this somewhere?

[yurishkuro]

no - I keep all links in the v2 RFC doc.

[mahadzaryab1]

@yurishkuro can link #6041 for grpc storage in the issue description

[mahadzaryab1]

@yurishkuro looks like we're pretty close to finishing this issue - do you know what the status is of #5790? any help needed there?

[yurishkuro]

do you know what the status is of #5790? any help needed there?

I had some comments that the author did not respond to. We can ping the author on the issue - if they don't have time to continue, we could take over that PR, it was nearly done iirc.

[JaredTan95]

do you know what the status is of #5790? any help needed there?

I had some comments that the author did not respond to. We can ping the author on the issue - if they don't have time to continue, we could take over that PR, it was nearly done iirc.

Sorry for the late reply. After @yurishkuro reivew and adjustment, I feel that the logic here is rather complicated for me. I can no longer make the changes at present, and hope to get more help from the maintainer or take over directly.

[mahadzaryab1]

do you know what the status is of #5790? any help needed there?

I had some comments that the author did not respond to. We can ping the author on the issue - if they don't have time to continue, we could take over that PR, it was nearly done iirc.

Sorry for the late reply. After @yurishkuro reivew and adjustment, I feel that the logic here is rather complicated for me. I can no longer make the changes at present, and hope to get more help from the maintainer or take over directly.

@JaredTan95 no worries! I can try picking up your PR and try addressing the feedback.

[mahadzaryab1]

do you know what the status is of #5790? any help needed there?

I had some comments that the author did not respond to. We can ping the author on the issue - if they don't have time to continue, we could take over that PR, it was nearly done iirc.

Sorry for the late reply. After @yurishkuro reivew and adjustment, I feel that the logic here is rather complicated for me. I can no longer make the changes at present, and hope to get more help from the maintainer or take over directly.

@JaredTan95 @yurishkuro I've opened a new PR at #6060 since I didn't have permission to modify the existing ones. I've addressed most of the feedback that was left on the original PR. Please take a look and let me know if you have any other feedback.

[mahadzaryab1]

@yurishkuro The elasticsearch configurations are now complete as well. Any other items we want to address here? Otherwise, I think we can close this issue out.

[yurishkuro]

Did you already update the google doc with the mapping?

[mahadzaryab1]

@yurishkuro Yes - the migration guide can be viewed here.

[yurishkuro]

I think this is completed - huge thanks to @mahadzaryab1 🚀 , this is a critical milestone for v2.