[jaeger-v2] Add `remotesampling` extension #5389

Pushkarm029 · 2024-04-25T17:11:09Z

Description of the changes

Part of [jaeger-v2] Support sampling strategies #5531
adds static sampling support for otel-based jaeger-v2.
supports hot reload
added unit tests

How was this change tested?

go run -tags=ui ./cmd/jaeger --config ./cmd/jaeger/config-badger.yaml
make test

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: yarn lint and yarn test

codecov · 2024-04-25T17:20:47Z

Codecov Report

Attention: Patch coverage is 86.15917% with 40 lines in your changes missing coverage. Please review.

Project coverage is 96.70%. Comparing base (882ae45) to head (ed06321).

Files	Patch %	Lines
...ger/internal/extension/remotesampling/extension.go	74.66%	19 Missing and 19 partials ⚠️
.../internal/processors/adaptivesampling/processor.go	94.59%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5389      +/-   ##
==========================================
- Coverage   96.88%   96.70%   -0.18%     
==========================================
  Files         335      341       +6     
  Lines       16156    16444     +288     
==========================================
+ Hits        15652    15902     +250     
- Misses        335      354      +19     
- Partials      169      188      +19

Flag	Coverage Δ
badger_v1	`8.05% <ø> (ø)`
badger_v2	`1.90% <0.00%> (ø)`
cassandra-3.x-v1	`16.62% <ø> (ø)`
cassandra-3.x-v2	`1.82% <0.00%> (ø)`
cassandra-4.x-v1	`16.62% <ø> (ø)`
cassandra-4.x-v2	`1.82% <0.00%> (ø)`
elasticsearch-6.x-v1	`18.78% <ø> (-0.02%)`	⬇️
elasticsearch-7.x-v1	`18.86% <ø> (ø)`
elasticsearch-8.x-v1	`19.05% <ø> (ø)`
elasticsearch-8.x-v2	`1.90% <0.00%> (ø)`
grpc_v1	`9.53% <ø> (+0.01%)`	⬆️
grpc_v2	`7.48% <0.00%> (ø)`
kafka	`9.75% <ø> (ø)`
opensearch-1.x-v1	`18.91% <ø> (+0.01%)`	⬆️
opensearch-2.x-v1	`18.91% <ø> (+0.01%)`	⬆️
opensearch-2.x-v2	`1.90% <0.00%> (+0.01%)`	⬆️
unittests	`95.12% <86.15%> (-0.15%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cmd/jaeger/internal/extension/jaegerstaticsampling/config.go

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md

Pushkarm029 · 2024-04-26T21:16:26Z

I am thinking of this approach.:

create a gRPC sampling server
then, register api_v2.RegisterSamplingManagerServer(server, sampling.NewGRPCHandler(SamplingStore))
similarly, create an HTTP server and register a sampling API.

That's how it works in the current Jaeger collector, but we are serving this API in the collector server. Here, we have to create a new one.

suggestions

Pushkarm029 · 2024-04-26T21:17:34Z

One more question:

How can I run hotrod with jaeger-v2 binary?
I tried this, but Jaeger UI was not able to detect Hotrod service.
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 go run ./main.go all
go run -tags=ui ./cmd/jaeger --config ./cmd/jaeger/config-elasticsearch.yaml

yurishkuro · 2024-04-27T01:15:44Z

We need a remotesampling extension which performs the following duties:

creates HTTP server that returns sampling strategies as JSON
allows user a choice of a sampling strategy store
- file: a static file with auto-reload
- adaptive: a backend storage that implements interfaces in sampling/strategystore package

We also need an adaptivesampling processor that

uses remotesampling to obtain storage backend (similar how jaegerexporter uses jaegerextension to get storage)
process spans and updates strategies in that storage (which then will be read by the extension's server)

All of this should require very minimal implementation, primarily just wiring up the existing components already implemented in app/collector/main

yurishkuro · 2024-04-29T15:59:38Z

The existing component (from jaeger v1) are in blue:

flowchart LR
    Receiver --> AdaptiveSamplingProcessor --> BatchProcessor --> Exporter
    Exporter -->|"(1) get storage"| JaegerStorageExension
    Exporter -->|"(2) write trace"| TraceStorage
    AdaptiveSamplingProcessor -->|"getStorage()"| StorageConfig

    OTEL_SDK[OTEL
             SDK]
    OTEL_SDK -->|"(1) GET /sampling"| HTTP_endpoint
    HTTP_endpoint -->|"(2) getStrategy()"| StrategiesProvider
    style HTTP_endpoint fill:blue,color:white

    subgraph Jaeger Collector
        Receiver
        BatchProcessor[Batch
                       Processor]
        Exporter        
        TraceStorage[(Trace
                      Storage)]
        AdaptiveSamplingProcessor[Adaptive
                                  Sampling
                                  Processor]
        AdaptiveSamplingProcessorV1[Adaptive
                                    Sampling
                                    Processor_v1]
        style AdaptiveSamplingProcessorV1 fill:blue,color:white
        AdaptiveSamplingProcessor -->|"[]*model.Span"| AdaptiveSamplingProcessorV1
        AdaptiveSamplingProcessorV1 ---|use| SamplingStorage

        subgraph JaegerStorageExension[Jaeger Storage Exension]
            Storage[[Storage
                     Config]]
        end
        subgraph RemoteSamplingExtension[Remote Sampling Extension]
            StrategiesProvider -->|"(3b) getStrategy()"| AdaptiveProvider
            StrategiesProvider -->|"(3a) getStrategy()"| FileProvider
            FileProvider --> FileConfig
            AdaptiveProvider --> StorageConfig
    
            HTTP_endpoint[HTTP
                          endpoint]
            StrategiesProvider[Strategies
                               Provider]
            FileProvider[File
                         Provider]
            AdaptiveProvider[Adaptive
                             Provider]
            style StrategiesProvider fill:blue,color:white
            style FileProvider fill:blue,color:white
            style AdaptiveProvider fill:blue,color:white
            subgraph Config
                FileConfig[[File Config]]
                StorageConfig[[Storage Config]]
            end
            StorageConfig --- SamplingStorage
            SamplingStorage[(Sampling
                             Storage)]
            style SamplingStorage fill:blue,color:white
        end
    end

cmd/jaeger/config-badger.yaml

Pushkarm029 · 2024-05-04T14:38:36Z

i haven;t pushed latest changes yet.

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md

pkg/clientcfg/clientcfghttp/handler.go

cmd/jaeger/internal/processors/adaptivesampling/processor.go

cmd/jaeger/internal/extension/remotesampling/extension_test.go

cmd/jaeger/config-badger.yaml

yurishkuro

looks good, in the right direction

cmd/jaeger/config-badger.yaml

cmd/jaeger/internal/extension/remotesampling/config.go

cmd/jaeger/internal/extension/remotesampling/factory.go

cmd/jaeger/internal/extension/remotesampling/internal/http.go

cmd/jaeger/internal/extension/remotesampling/testdata/strategy.json

cmd/jaeger/internal/extension/remotesampling/config.go

pkg/clientcfg/clientcfghttp/handler.go

cmd/jaeger/internal/extension/remotesampling/extension.go

cmd/jaeger/internal/processors/adaptivesampling/processor.go

cmd/jaeger/internal/extension/remotesampling/internal/http.go

cmd/jaeger/internal/extension/remotesampling/config.go

## Which problem is this PR solving? - part of #5389 ## Description of the changes - processor is co-located in strategy_store and aggregator. - In aggregator to run `generateStrategyResponses`, `runCalculationLoop`. - In strategy_store to run `loadProbabilities`, `runUpdateProbabilitiesLoop` ## How was this change tested? - `make test` ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` --------- Signed-off-by: Pushkar Mishra <[email protected]> Signed-off-by: pushkarm029 <[email protected]>

yurishkuro

how do you plan to test it?

cmd/jaeger/internal/extension/remotesampling/config.go

cmd/jaeger/internal/extension/remotesampling/factory.go

cmd/jaeger/internal/processors/adaptivesampling/config.go

plugin/sampling/strategystore/adaptive/aggregator.go

cmd/jaeger/internal/extension/remotesampling/extension.go

yurishkuro · 2024-05-25T21:34:54Z

cmd/jaeger/config-badger.yaml

+    # file:
+    #   path: ./cmd/jaeger/sampling-strategies.json


maybe we should default to file, not to more complex adaptive?

## Which problem is this PR solving? - #5389 (comment) ## Description of the changes - Refactored `handleRootSpan` logic into a helper method in aggregator.go.

cmd/jaeger/internal/extension/remotesampling/config.go

cmd/jaeger/internal/processors/adaptivesampling/processor.go

yurishkuro

please fix the DCO and keep it green

cmd/jaeger/internal/extension/remotesampling/extension.go

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro

@Pushkarm029 this looks to be in a good shape, we just need to bump test coverage. I verified that the server starts and serves sampling strategies over HTTP.

Signed-off-by: Yuri Shkuro <[email protected]>

Pushkarm029 · 2024-06-18T05:17:36Z

@Pushkarm029 this looks to be in a good shape, we just need to bump test coverage. I verified that the server starts and serves sampling strategies over HTTP.

thanks, I will start writing tests now.

Signed-off-by: pushkarm029 <[email protected]>

Pushkarm029 · 2024-06-30T09:17:45Z

Halfway🛣️

Signed-off-by: pushkarm029 <[email protected]>

Signed-off-by: Pushkar Mishra <[email protected]>

Signed-off-by: pushkarm029 <[email protected]>

Pushkarm029 · 2024-07-14T12:10:23Z

80% Done, few more to go

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro · 2024-07-14T20:45:08Z

@Pushkarm029 I added a few more test and am going to merge it, because the PR has been open for a long time.

The coverage is almost there, we can write extra tests (mostly for remotesampling/extension) separately.

@@            Coverage Diff             @@
##             main    #5389      +/-   ##
==========================================
- Coverage   96.88%   96.70%   -0.18%

## Description of the changes - Part of jaegertracing#5531 - adds static sampling support for otel-based jaeger-v2. - supports hot reload - added unit tests ## How was this change tested? - `go run -tags=ui ./cmd/jaeger --config ./cmd/jaeger/config-badger.yaml` - `make test` ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [ ] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` --------- Signed-off-by: pushkarm029 <[email protected]> Signed-off-by: Yuri Shkuro <[email protected]> Signed-off-by: Pushkar Mishra <[email protected]> Co-authored-by: Yuri Shkuro <[email protected]> Co-authored-by: Yuri Shkuro <[email protected]>

Pushkarm029 requested a review from a team as a code owner April 25, 2024 17:11

Pushkarm029 requested a review from pavolloffay April 25, 2024 17:11

yurishkuro reviewed Apr 25, 2024

View reviewed changes

cmd/jaeger/internal/extension/jaegerstaticsampling/config.go Outdated Show resolved Hide resolved

yurishkuro reviewed Apr 25, 2024

View reviewed changes

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md Outdated Show resolved Hide resolved

Pushkarm029 changed the title ~~[jaeger-v2] Add static sampling extension~~ [jaeger-v2] Add remotesampling extension May 4, 2024

Pushkarm029 marked this pull request as draft May 4, 2024 08:11

yurishkuro reviewed May 4, 2024

View reviewed changes

cmd/jaeger/config-badger.yaml Outdated Show resolved Hide resolved

yurishkuro reviewed May 4, 2024

View reviewed changes

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md Outdated Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

pkg/clientcfg/clientcfghttp/handler.go Outdated Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

cmd/jaeger/internal/processors/adaptivesampling/processor.go Outdated Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/extension_test.go Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

cmd/jaeger/config-badger.yaml Outdated Show resolved Hide resolved

yurishkuro added the changelog:new-feature Change that should be called out as new feature in CHANGELOG label May 7, 2024

yurishkuro reviewed May 7, 2024

View reviewed changes

yurishkuro reviewed May 10, 2024

View reviewed changes

Pushkarm029 mentioned this pull request May 11, 2024

Refactor Adaptive Sampling Aggregator & Strategy Store #5441

Merged

4 tasks

yurishkuro reviewed May 23, 2024

View reviewed changes

Pushkarm029 mentioned this pull request May 23, 2024

[refactor] move root span handler into aggregator #5478

Merged

4 tasks

yurishkuro reviewed May 25, 2024

View reviewed changes

yurishkuro reviewed May 26, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/config.go Show resolved Hide resolved

yurishkuro reviewed May 26, 2024

View reviewed changes

cmd/jaeger/internal/processors/adaptivesampling/processor.go Outdated Show resolved Hide resolved

yurishkuro reviewed Jun 2, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/extension.go Outdated Show resolved Hide resolved

add tests

88ec66f

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro reviewed Jun 18, 2024

View reviewed changes

fix

ff73c57

Signed-off-by: Yuri Shkuro <[email protected]>

Pushkarm029 added 2 commits June 23, 2024 10:56

Merge branch 'main' into sampling_extension

175e682

fix lint

1a2c505

Signed-off-by: pushkarm029 <[email protected]>

Pushkarm029 added 6 commits July 7, 2024 21:02

some prog

4c54c2e

Signed-off-by: pushkarm029 <[email protected]>

Merge branch 'main' into sampling_extension

be7b853

Signed-off-by: Pushkar Mishra <[email protected]>

Merge branch 'main' into sampling_extension

8b49933

fix

e313a74

Signed-off-by: pushkarm029 <[email protected]>

fix

b15f33e

Signed-off-by: pushkarm029 <[email protected]>

fix

08b71d2

Signed-off-by: pushkarm029 <[email protected]>

yurishkuro added 5 commits July 14, 2024 15:40

Increase coverage in cmd/jaeger/internal/processors/adaptivesampling

28d3198

Signed-off-by: Yuri Shkuro <[email protected]>

delint

cbcd5bb

Signed-off-by: Yuri Shkuro <[email protected]>

delint

9dba3b7

Signed-off-by: Yuri Shkuro <[email protected]>

more tests

7ac4f29

Signed-off-by: Yuri Shkuro <[email protected]>

add test

ed06321

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro marked this pull request as ready for review July 14, 2024 20:42

yurishkuro added changelog:exprimental Change to an experimental part of the code v2 and removed changelog:new-feature Change that should be called out as new feature in CHANGELOG labels Jul 14, 2024

dosubot bot added area/sampling go labels Jul 14, 2024

yurishkuro approved these changes Jul 14, 2024

View reviewed changes

yurishkuro merged commit 0616fba into jaegertracing:main Jul 14, 2024
47 of 48 checks passed

Pushkarm029 deleted the sampling_extension branch July 15, 2024 07:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jaeger-v2] Add `remotesampling` extension #5389

[jaeger-v2] Add `remotesampling` extension #5389

Pushkarm029 commented Apr 25, 2024 •

edited by yurishkuro

Loading

codecov bot commented Apr 25, 2024 •

edited

Loading

Pushkarm029 commented Apr 26, 2024

Pushkarm029 commented Apr 26, 2024

yurishkuro commented Apr 27, 2024

yurishkuro commented Apr 29, 2024

Pushkarm029 commented May 4, 2024

yurishkuro left a comment

yurishkuro left a comment

yurishkuro May 25, 2024

yurishkuro left a comment

yurishkuro left a comment

Pushkarm029 commented Jun 18, 2024

Pushkarm029 commented Jun 30, 2024

Pushkarm029 commented Jul 14, 2024

yurishkuro commented Jul 14, 2024

[jaeger-v2] Add remotesampling extension #5389

[jaeger-v2] Add remotesampling extension #5389

Conversation

Pushkarm029 commented Apr 25, 2024 • edited by yurishkuro Loading

Description of the changes

How was this change tested?

Checklist

codecov bot commented Apr 25, 2024 • edited Loading

Codecov Report

Pushkarm029 commented Apr 26, 2024

Pushkarm029 commented Apr 26, 2024

yurishkuro commented Apr 27, 2024

yurishkuro commented Apr 29, 2024

Pushkarm029 commented May 4, 2024

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro May 25, 2024

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

Pushkarm029 commented Jun 18, 2024

Pushkarm029 commented Jun 30, 2024

Pushkarm029 commented Jul 14, 2024

yurishkuro commented Jul 14, 2024

[jaeger-v2] Add `remotesampling` extension #5389

[jaeger-v2] Add `remotesampling` extension #5389

Pushkarm029 commented Apr 25, 2024 •

edited by yurishkuro

Loading

codecov bot commented Apr 25, 2024 •

edited

Loading