Add Elasticsearch storage support for adaptive sampling

[Pushkarm029]

Which problem is this PR solving?

• Storage backends for adaptive sampling #3305

Description of the changes

• Implemented Elasticsearch storage for adaptive sampling

How was this change tested?

• not tested yet

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[Pushkarm029]

Raised PR to get to know from reviewers whether I am on the right track or not.

[Pushkarm029]

adding tests

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.56%. Comparing base (7a04461) to head (851f37c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5158      +/-   ##
==========================================
+ Coverage   94.49%   94.56%   +0.07%     
==========================================
  Files         334      336       +2     
  Lines       19303    19501     +198     
==========================================
+ Hits        18241    18442     +201     
+ Misses        875      871       -4     
- Partials      187      188       +1     

Flag	Coverage Δ
cassandra-3.x	24.94% <0.00%> (-0.59%)	⬇️
cassandra-4.x	24.94% <0.00%> (-0.59%)	⬇️
elasticsearch-5.x	21.65% <78.61%> (+1.83%)	⬆️
elasticsearch-6.x	21.65% <78.61%> (+1.83%)	⬆️
elasticsearch-7.x	21.79% <78.61%> (+1.83%)	⬆️
elasticsearch-8.x	21.88% <78.61%> (+1.84%)	⬆️
grpc-badger	19.11% <0.00%> (-0.36%)	⬇️
kafka	13.74% <0.00%> (-0.33%)	⬇️
opensearch-1.x	21.79% <78.61%> (+1.83%)	⬆️
opensearch-2.x	21.78% <78.61%> (+1.81%)	⬆️
unittests	92.39% <97.18%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yurishkuro]

• run make fmt
• please add tests to all new functionality
• enable integration tests (see plugin/storage/integration/integration.go:423)

[Pushkarm029]

Enabling integration tests

[Pushkarm029]

The Tests are failing because the GetLatestProbability is failing

    integration.go:466: 
        	Error Trace:	/home/runner/work/jaeger/jaeger/plugin/storage/integration/integration.go:466
        	Error:      	Should be true
        	Test:       	TestElasticsearchStorage/GetLatestProbability
    integration.go:467: 	 Expected: map[new-srv:map[op:0.1]]
    integration.go:468: 	 Actual  : map[new-srv3:map[op:0.123]]

[Pushkarm029]

any idea @yurishkuro, why it is failing on es v5 but not on v7 and v8.

[Pushkarm029]

added rollover support

[Pushkarm029]

Please review.

[yurishkuro]

almost there!

[Pushkarm029]

tests are also failing. looks like i have to add adaptive sampling in this cmd/es-index-cleaner/app/flags.go Config also????

[yurishkuro]

tests are also failing. looks like i have to add adaptive sampling in this cmd/es-index-cleaner/app/flags.go Config also????

heh, I left a comment about that, but why would have any impact on the tests? The cleaner is just over-eager in which indices it removes, but unless the integration tests are actually testing for this behavior I would expect to be a no-op (because either rollover will create extra indices and things would work, or it wouldn't create them and cleaner will be no-op).

are you able to run these integration tests locally?

[yurishkuro]

I see this error, maybe you're not passing extra flags inside the integration test to actually generate sampling-related index?

        	            	listA:
        	            	([]string) (len=3) {
        	            	 (string) (len=34) "integration-test-jaeger-span-write",
        	            	 (string) (len=37) "integration-test-jaeger-service-write",
        	            	 (string) (len=42) "integration-test-jaeger-dependencies-write"
        	            	}
        	            	
        	            	
        	            	listB:
        	            	([]string) (len=4) {
        	            	 (string) (len=37) "integration-test-jaeger-service-write",
        	            	 (string) (len=34) "integration-test-jaeger-span-write",
        	            	 (string) (len=42) "integration-test-jaeger-dependencies-write",
        	            	 (string) (len=38) "integration-test-jaeger-sampling-write"
        	            	}

[Pushkarm029]

are you able to run these integration tests locally?

YES, but sometimes it fails with a timeout, in that case, I rely on gh workflow

I see this error, maybe you're not passing extra flags inside the integration test to actually generate sampling-related index?

working on it

[Pushkarm029]

🤞everything seems good now.

[yurishkuro]

🎉