Configure OTEL Collector to observe Internal Telemetry

[Wise-Wizard]

Which problem is this PR solving?

This PR addresses a part of the issue #5633

Description of the changes
This is a Draft PR to achieve Observability Parity between V1 and V2 components by configuring OTEL Collector config files to initialise internal tracer and metrics
How was this change tested?

The changes were tested by running the following command:

make test

CI actions

Checklist

• I have read CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[yurishkuro]

The changes were tested by running the following command:
make test

How does doing that demonstrate that the changes result in desired behavior?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.66%. Comparing base (e437971) to head (9a3536d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5752   +/-   ##
=======================================
  Coverage   96.66%   96.66%           
=======================================
  Files         342      342           
  Lines       16515    16519    +4     
=======================================
+ Hits        15964    15968    +4     
  Misses        362      362           
  Partials      189      189           

Flag	Coverage Δ
badger_v1	8.05% <ø> (ø)
badger_v2	1.81% <ø> (ø)
cassandra-3.x-v1	16.61% <ø> (ø)
cassandra-3.x-v2	1.74% <ø> (ø)
cassandra-4.x-v1	16.61% <ø> (ø)
cassandra-4.x-v2	1.74% <ø> (ø)
elasticsearch-6.x-v1	18.78% <ø> (ø)
elasticsearch-7.x-v1	18.84% <ø> (ø)
elasticsearch-8.x-v1	19.02% <ø> (ø)
elasticsearch-8.x-v2	1.80% <ø> (ø)
grpc_v1	9.52% <ø> (+0.01%)	⬆️
grpc_v2	7.14% <ø> (+0.01%)	⬆️
kafka	9.74% <ø> (ø)
memory_v2	1.81% <ø> (ø)
opensearch-1.x-v1	18.89% <ø> (+0.01%)	⬆️
opensearch-2.x-v1	18.89% <ø> (ø)
opensearch-2.x-v2	1.80% <ø> (-0.02%)	⬇️
unittests	95.09% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Wise-Wizard]

The changes were tested by running the following command:
make test

How does doing that demonstrate that the changes result in desired behavior?

It dosen't, just created a PR to check for github actions

[Wise-Wizard]

@yurishkuro I think I have configured the tracer successfully as now when I am passing telset.TracerProvider instead of jTracer in Jaeger Query, the tests aren't failing anymore.

[yurishkuro]

Can you provide evidence that the changes are working? Just passing CI only means it didn't break things, but we don't explicitly test for internally generated traces (we should , it would be a good e2e test, e.g. as part of all-in-one e2e we could check that the internal traces are generated for API queries)

[Wise-Wizard]

Can you provide evidence that the changes are working? Just passing CI only means it didn't break things, but we don't explicitly test for internally generated traces (we should , it would be a good e2e test, e.g. as part of all-in-one e2e we could check that the internal traces are generated for API queries)

When I had configured telset with OTEL.TracerProvider instead of JTracer without any changes in configuration of yaml file the all-in-one V2 test getAPITrace fails because I think a no-op tracer is being provided but after the changes in the configuration file the test seems to be passing.
Apart from that I have no POC

[Wise-Wizard]

@Wise-Wizard a screenshot is good as anecdotal evidence, but we need to be more rigorous than that. Jaeger has a lot of different metrics, we cannot validate parity by eyeballing screenshots, we need a scripted diffing solution.

Yes, the screenshot was just a small evidence.
i have compared the metrics names emitted by Jaeger Query as mentioned in the Doc
All of them are emitted in V2 scope as well.

[yurishkuro]

i have compared the metrics names emitted by Jaeger Query ... All of them are emitted in V2 scope as well.

How did you compare them all?

[Wise-Wizard]

i have compared the metrics names emitted by Jaeger Query ... All of them are emitted in V2 scope as well.

How did you compare them all?

I just compared the names of the metrics side by side.
The difference is in the number of operations per metric name.

[yurishkuro]

I just compared the names of the metrics side by side.

Can we use something like this instead? https://github.com/prometheus/prom2json
I would like to have scripted diffs, not just eyeballing verification.

[Wise-Wizard]

I just compared the names of the metrics side by side.

Can we use something like this instead? https://github.com/prometheus/prom2json I would like to have scripted diffs, not just eyeballing verification.

Hi, I have created a tmp folder and added JSON file for each using prom2Json.
PTAL at it and lmk if I need to filter out and make changes in the JSON file.
After that I can delete the folder and store the JSON locally to use it later for the migration guide.

[yurishkuro]

We don't need to see the actual json files, but their diffs. Any thoughts on how to compare them? We don't care about the values, only names and labels. There are probably json diffing tools.

[Wise-Wizard]

We don't need to see the actual json files, but their diffs. Any thoughts on how to compare them? We don't care about the values, only names and labels. There are probably json diffing tools.

Ok, will research about that and look for some tool to see the differences.
By differences you also mean the jaeger_collector and jaeger_agent metrics names to be included right?

[Wise-Wizard]

Hi, so I wrote a Python script to read the difference in metrics names in both the files and written the differences in names in difference.json file for comparison.
Please do take a look at the file.

[yurishkuro]

I would like to merge this PR. The following changes are needed:

• commit your metrics comparison script
• remove differences.json
• fix unit test

[Wise-Wizard]

Will do the changes

[yurishkuro]

make sure to get the CI green, you have unit test failures, and I see new linter failures

[Wise-Wizard]

make sure to get the CI green, you have unit test failures, and I see new linter failures

Done.