Add support for running EDOT inside of running Elastic Agent #5767

blakerouse · 2024-10-11T19:04:25Z

What does this PR do?

Adds the ability to run the EDOT along side the running Elastic Agent.

This connects the EDOT into the coordinator of the Elastic Agent. At any point if any of these top-level keys (receivers, processors, exporters, extensions, service) exist in the configuration or policy for the elastic-agent the EDOT is started. If all of those keys are removed from the configuration or policy then the EDOT is automatically stopped. If any configuration change occurs the updated configuration is passed along to the EDOT to handle.

Why is it important?

This allows EDOT configuration to exist inside of the configuration or policy and allow it to work as expected.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

This is an addition and doesn't affect the way the current Elastic Agent runs at all.

How to test this PR locally

Place OTel configuration into the elastic-agent.yml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:

exporters:
  otlp:
    endpoint: otelcol:4317

extensions:
  health_check:
  pprof:

service:
  extensions: [health_check, pprof]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Run elastic-agent run -e.

Related issues

Closes #5796

mergify · 2024-10-11T19:05:10Z

This pull request does not have a backport label. Could you fix it @blakerouse? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-10-11T19:05:10Z

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

mergify · 2024-10-15T16:19:45Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

mergify · 2024-10-15T19:22:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

mergify · 2024-10-25T10:57:03Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

mergify · 2024-11-07T08:26:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-hybrid upstream/otel-hybrid
git merge upstream/main
git push upstream otel-hybrid

blakerouse · 2024-11-12T17:54:57Z

This is ready for a review. Don't be scare too much by the size of this change, most of that comes from NOTICE.txt, the generate control protocol and the rename of Unpack to UnpackTo.

To answer the incoming question of why rename Unpack to UnpackTo is that the Unpacker interface used by go-ucfg has the function named Unpack so that collides. This required the rename for the configuration to work correctly.

elasticmachine · 2024-11-13T13:58:59Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

michalpristas

I think this looks good, the mentioned change makes this PR a bit uglier that it needs to be.

I imagined our first take on hybrid approach pretty much like this, i have no major things to point out besides missing keyword and verifying windows works.

Tested this locally, seems fine. Status reported nicely on component level

michalpristas · 2024-11-18T08:27:36Z

internal/pkg/agent/cmd/otel.go

@@ -2,8 +2,6 @@
 // or more contributor license agreements. Licensed under the Elastic License 2.0;
 // you may not use this file except in compliance with the Elastic License 2.0.

-//go:build !windows


have you verified Agent installed as a windows service starts properly without any timing issues? it was a bit difficult to verify last time

linking issue #4976 cc @leehinman

I am not seeing those issues in this PR. This PR is running on Windows with the integration testing framework and all of those are passing just fine. Those would not be passing if this was not working correctly.

michalpristas · 2024-11-18T08:30:08Z

internal/pkg/config/config.go

@@ -39,14 +51,20 @@ var DefaultOptions = []interface{}{
 	ucfg.VarExp,
 	VarSkipKeys("inputs"),
 	ucfg.IgnoreCommas,
+	OTelKeys("receivers", "processors", "exporters", "extensions", "service"),


missing connectors
refer to unmarshaller https://github.com/open-telemetry/opentelemetry-collector/blob/8e522ad950de6326a0841d7e1bef808bbc0d3537/otelcol/unmarshaler.go#L18

Added! Thanks for the catch.

michalpristas · 2024-11-18T08:30:55Z

internal/pkg/config/config.go

-// Unpack unpacks a struct to Config.
-func (c *Config) Unpack(to interface{}, opts ...interface{}) error {
+// UnpackTo unpacks this config into to with the given options.
+func (c *Config) UnpackTo(to interface{}, opts ...interface{}) error {


as said PR would be o simpler without this one, if we could extract this to refactoring PR we could get reviews for this one faster

Yeah, I probably should have split it out. Being that you already gave a +1 I would prefer to leave it as it. Splitting that out is going to take more work, then just getting this merged.

michalpristas · 2024-11-18T08:39:26Z

internal/pkg/otel/agentprovider/provider.go

+	return p.uri
+}
+
+func (p *Provider) replaceCanceller(replace context.CancelFunc) {


this looks wild

+1, why is this necessary? If there is no way to get rid of it, please put the explanation in the code as comments.

michalpristas · 2024-11-18T08:44:32Z

internal/pkg/otel/run.go

-				httpsprovider.NewFactory(),
-			},
-			ConverterFactories: []confmap.ConverterFactory{
-				expandconverter.NewFactory(),


👍 for dropping, we should pay more attention to upstream changelog

blakerouse · 2024-11-18T16:01:17Z

@michalpristas For some reason GitHub is not saying your approval counted, could you possibly approve again to see if that lets me have an approval.

pierrehilbert · 2024-11-18T16:44:30Z

You need more than @michalpristas's approval.
@MichaelKatsoulis / @gizas we will need your approval here please.

blakerouse · 2024-11-18T17:45:45Z

@pierrehilbert That is fine, but for some reason even @michalpristas is not showing as approved to GitHub.

cmacknz · 2024-11-18T20:04:59Z

I tried this locally and the first thing I noticed is that the format of the JSON output by elastic-agent status is different.

In isolation in the context of this PR, this isn't a problem. It will be a problem for Beats receivers though, as unless Fleet knows how to parse the new format we are going to lose the input health reporting features.

So we either have to:

Have Fleet understand collector status reporting and have it integrate with the UI similar to what we have today.
Have the agent translate the collector health into the existing components based health reporting structure.

I think I'd prefer 2 since the point of the Beats receivers project is to make the agent transparently use the OTel collector. Additional this has a better chance of getting the state reporting that is happening directly in Filebeat inputs working, see elastic/beats#39209 for where this was added.

I'll for this into a separate tracking issue unless we can resolve it in this PR.

sudo elastic-development-agent status --output=json

{
    "info": {
        "id": "8c884936-b6d7-4ff7-af1f-c15da3752384",
        "version": "9.0.0",
        "commit": "e787a19a6a6cb0f570cdf4f16c8839e20b0feb23",
        "build_time": "2024-11-18 19:12:02 +0000 UTC",
        "snapshot": true,
        "pid": 22023,
        "unprivileged": false,
        "is_managed": false
    },
    "state": 2,
    "message": "Running",
    "components": [],
    "FleetState": 6,
    "FleetMessage": "Not enrolled into Fleet",
    "collector": {
        "status": 2,
        "timestamp": "2024-11-18T14:52:56.483851-05:00",
        "components": {
            "extensions": {
                "status": 2,
                "timestamp": "2024-11-18T14:52:56.482935-05:00",
                "components": {
                    "extension:health_check": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.482525-05:00"
                    },
                    "extension:pprof": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.482935-05:00"
                    }
                }
            },
            "pipeline:logs": {
                "status": 2,
                "timestamp": "2024-11-18T14:52:56.483683-05:00",
                "components": {
                    "exporter:otlp": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483204-05:00"
                    },
                    "processor:batch": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483252-05:00"
                    },
                    "receiver:otlp": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483683-05:00"
                    }
                }
            },
            "pipeline:metrics": {
                "status": 2,
                "timestamp": "2024-11-18T14:52:56.483851-05:00",
                "components": {
                    "exporter:otlp": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483369-05:00"
                    },
                    "processor:batch": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483702-05:00"
                    },
                    "receiver:otlp": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483851-05:00"
                    }
                }
            },
            "pipeline:traces": {
                "status": 2,
                "timestamp": "2024-11-18T14:52:56.483829-05:00",
                "components": {
                    "exporter:otlp": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.4838-05:00"
                    },
                    "processor:batch": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483812-05:00"
                    },
                    "receiver:otlp": {
                        "status": 2,
                        "timestamp": "2024-11-18T14:52:56.483829-05:00"
                    }
                }
            }
        }
    }
}

I enrolled an agent running a collector configuration into Fleet, which was not interesting because the collector configuration was immediately replaced with the standard agent configuration from Fleet.

cmacknz · 2024-11-18T20:29:59Z

Looking in the diagnostics I see the otel configuration is separated out, and pre-config.yaml and computed-config.yaml are both mostly empty:

cat diag/computed-config.yaml
host:
    id: 48DA13D6-B83B-5C71-A4F3-494E674F9F37
path:
    config: /Library/Elastic/Agent-Development
    data: /Library/Elastic/Agent-Development/data
    home: /Library/Elastic/Agent-Development/data/elastic-agent-9.0.0-SNAPSHOT-e787a1
    logs: /Library/Elastic/Agent-Development
runtime:
    arch: arm64
    native_arch: arm64
    os: darwin
    osinfo:
        family: darwin
        major: 14
        minor: 7
        patch: 1
        type: macos
        version: 14.7.1

What would we expect computed-config to look like if we used variables in the otel configuration? Is that even supported here? If it is, how would we debug it?

I also see state.yaml is missing all of the collector status. This we need to address somehow for sure, the collector runtime state needs to be in diagnostics:

cat diag/state.yaml
components: []
fleet_message: Not enrolled into Fleet
fleet_state: 6
log_level: info
message: Running
state: 2

cmacknz · 2024-11-18T20:43:37Z

internal/pkg/agent/application/gateway/fleet/fleet_gateway.go

@@ -280,6 +287,36 @@ func (f *FleetGateway) convertToCheckinComponents(components []runtime.Component
 		checkinComponents = append(checkinComponents, checkinComponent)
 	}

+	// OTel status is placed as a component for each top-level component in OTel
+	// and each subcomponent is a unit.
+	if otelStatus != nil {


Can we do this in the status output as well? For the most part elastic-agent status --output=json mirrors what we would send to Fleet and right now it doesn't.

I am also worried that the mapping of inputs back to integrations might be broken by this remapping, thus breaking the health reporting UIs in Fleet. We'd probably need to let fleet send otel configurations. Perhaps we can do this with the override API to test this out since they are separate sections of the UI.

I don't understand how this would be broken in Fleet. This is sending it exactly in the same format that Fleet understand today, so this just works transparently without any changes to Fleet. Which is what we discussed and what was implemented here.

Fleet won't reject it, I am worried about how Fleet maps inputs back to integrations, since Fleet is integration focused and agent is only inputs.

We would need to test an integration implemented with Beats receivers and make sure the status reporting still works the same way.

I didn't know there was a link between those two. I was just under the impression that the UI just rendered the content of the components field that is shipped to Fleet. If it does some referencing between the two (which I don't think it does), then it would be better to just let Fleet understand the component field.

Being this is experimental that is something that can easily change in the future.

I think it looks into the meta information in the inputs, possibly it only needs the input ID to be what was in the agent policy for the component and it could figure it out.

I agree this isn't blocking and is something we could check easily with a test.

I don't love the elastic-agent status output=json format diverging from what gets sent to Fleet, I have used the fact that it is in the checkin format a few times for debugging, but I won't block the PR on it.

internal/pkg/agent/application/monitoring/liveness.go

cmacknz · 2024-11-18T20:54:27Z

internal/pkg/otel/manager/extension.go

+	// prevent aggregate statuses from flapping between StatusStarting and StatusOK
+	// as components are started individually by the service.


Could we just not do this filtering? Was it just annoying or causing an actual problem?

It causes the reporting to not really be in sync and looks very strange. This is also following the same pattern of the healthcheckv2extension.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/healthcheckv2extension/extension.go#L168

Ah, can you link to that code directly as a comment?

cmacknz · 2024-11-18T20:55:23Z

internal/pkg/otel/manager/force_extension_converter.go

+	name string
+}
+
+func (fe *forceExtension) Convert(ctx context.Context, conf *confmap.Conf) error {


What is this doing and why is it necessary? Please add the explanation as a comment in the code directly.

I will add explanation in the code.

internal/pkg/otel/manager/manager.go

cmacknz · 2024-11-18T21:22:25Z

Can we start some hybrid agent documentation in https://github.com/elastic/elastic-agent/tree/main/docs? Probably just an example configuration is enough to start, and then when the time comes we can give that to our tech writers to produce official documentation.

blakerouse · 2024-11-18T21:25:52Z

I tried this locally and the first thing I noticed is that the format of the JSON output by elastic-agent status is different.

It is only different as at the top-level there is a new field called collector that provides all the status information about the collector.

In isolation in the context of this PR, this isn't a problem. It will be a problem for Beats receivers though, as unless Fleet knows how to parse the new format we are going to lose the input health reporting features.

As you see later in your code review this is not sent to Fleet this way. The Fleet information is not in any way decoded from JSON marshalling.

So we either have to:

Have Fleet understand collector status reporting and have it integrate with the UI similar to what we have today.

Nope, this is already done in this PR.

Have the agent translate the collector health into the existing components based health reporting structure.

We could do that, but I don't think that is a good idea long term. The reason I made a new collector top-level key with the OTel status information, is that in the long run we will drop the other format and switch to the new format.

I think I'd prefer 2 since the point of the Beats receivers project is to make the agent transparently use the OTel collector. Additional this has a better chance of getting the state reporting that is happening directly in Filebeat inputs working, see elastic/beats#39209 for where this was added.

They should 100% only be reporting status using otel. The goal is to one day only run the collector, they should add status reporting through the collector.

I'll for this into a separate tracking issue unless we can resolve it in this PR.

sudo elastic-development-agent status --output=json

I don't see an issue. Adding new fields is not considered breaking. API's add new fields all the time and that is not considered breaking.

blakerouse · 2024-11-18T21:28:02Z

Looking in the diagnostics I see the otel configuration is separated out, and pre-config.yaml and computed-config.yaml are both mostly empty:

cat diag/computed-config.yaml
host:
    id: 48DA13D6-B83B-5C71-A4F3-494E674F9F37
path:
    config: /Library/Elastic/Agent-Development
    data: /Library/Elastic/Agent-Development/data
    home: /Library/Elastic/Agent-Development/data/elastic-agent-9.0.0-SNAPSHOT-e787a1
    logs: /Library/Elastic/Agent-Development
runtime:
    arch: arm64
    native_arch: arm64
    os: darwin
    osinfo:
        family: darwin
        major: 14
        minor: 7
        patch: 1
        type: macos
        version: 14.7.1

What would we expect computed-config to look like if we used variables in the otel configuration? Is that even supported here? If it is, how would we debug it?

Variables are not supported in otel configuration. That would be extending otel outside of the Elastic Agent which I don't think is a good idea. You will notice in this PR that otel configuration is extracted as early as possible. This is done to ensure that the Elastic Agent in no way affects the otel configuration.

I also see state.yaml is missing all of the collector status. This we need to address somehow for sure, the collector runtime state needs to be in diagnostics:
cat diag/state.yaml
components: []
fleet_message: Not enrolled into Fleet
fleet_state: 6
log_level: info
message: Running
state: 2

Good catch. I will add that now.

blakerouse · 2024-11-18T21:29:25Z

Can we start some hybrid agent documentation in https://github.com/elastic/elastic-agent/tree/main/docs? Probably just an example configuration is enough to start, and then when the time comes we can give that to our tech writers to produce official documentation.

Yes I can add that information.

cmacknz · 2024-11-18T21:51:52Z

They should 100% only be reporting status using otel. The goal is to one day only run the collector, they should add status reporting through the collector.

Long term yes, my actual goal is to discover the point where we need to commit engineering time from the UI team to rewriting or updating the input and component health implementation to account for the collector health status.

I don't think just Beats receivers is that point, it is probably the point at which there are integrations with OTel native configurations. That said, we could plan UI work to account for the status changes for Beats receivers, but it will block us shipping them.

blakerouse · 2024-11-18T22:02:20Z

They should 100% only be reporting status using otel. The goal is to one day only run the collector, they should add status reporting through the collector.

Long term yes, my actual goal is to discover the point where we need to commit engineering time from the UI team to rewriting or updating the input and component health implementation to account for the collector health status.

I don't think just Beats receivers is that point, it is probably the point at which there are integrations with OTel native configurations. That said, we could plan UI work to account for the status changes for Beats receivers, but it will block us shipping them.

Even short term they should be reporting the status through otel, because once something is running under the otel collector that is the only way we can get status information. They will not be connected over the control protocol and they will be in-process the only way to do that is report the status through otel.

If you mean more like transparent taking the otel status and translating it to a component status to report to Fleet, then that is something different and could be done. But that still requires that the component running under the collector to be reporting it status through componentstatus.

I think this highly depends on if there is really a link between what a component status reports and an integration in Fleet. I was under the impression that under the Fleet UI that only place it is used is on the status page for the Agent and there is not connection between the two. The only connection I know of is with Endpoint, and that won't be changing in the short term.

cmacknz · 2024-11-18T22:03:56Z

If you mean more like transparent taking the otel status and translating it to a component status to report to Fleet, then that is something different and could be done.

Yes the interface to the Fleet and the eventual content of the .fleet-agents datastream there is all I care about. It's our API contract with the UI team.

Everything happening inside the agent as far as getting status from the collector LGTM.

michalpristas

reapproving, probably force push did the magic

blakerouse · 2024-11-19T16:50:07Z

I need an approval from @elastic/ingest-eng-prod for the github lint action change.

It is required for lint to pass on Windows as some Windows only module uses a new golang 1.22 features. We are using olger version of golint action that used only go 1.21 for lint, it needs to understand 1.22.

elastic-sonarqube · 2024-11-19T18:57:51Z

Quality Gate passed

Issues
11 New issues
2 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
61.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

alexsapran

Reviewing .github/workflows/golangci-lint.yml LGTM

(cherry picked from commit b07566b) # Conflicts: # NOTICE.txt # go.mod # go.sum

…Elastic Agent (#6096) * Add support for running EDOT inside of running Elastic Agent (#5767) (cherry picked from commit b07566b) --------- Co-authored-by: Blake Rouse <[email protected]>

mergify bot assigned blakerouse Oct 11, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 11, 2024

cmacknz mentioned this pull request Oct 15, 2024

Don't run any agent service when running otel subcommand #5748

Merged

4 tasks

ycombinator mentioned this pull request Oct 15, 2024

Run Elastic Agent in otel mode as a service #4933

Closed

blakerouse marked this pull request as ready for review November 12, 2024 17:52

blakerouse requested review from a team as code owners November 12, 2024 17:52

blakerouse requested review from gizas, MichaelKatsoulis, michalpristas and swiatekm November 12, 2024 17:52

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Nov 13, 2024

pierrehilbert added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label Nov 13, 2024

michalpristas approved these changes Nov 18, 2024

View reviewed changes

Enable OTel Hybrid mode.

e787a19

blakerouse force-pushed the otel-hybrid branch from c212a8e to e787a19 Compare November 18, 2024 15:59

cmacknz reviewed Nov 18, 2024

View reviewed changes

internal/pkg/agent/application/monitoring/liveness.go Outdated Show resolved Hide resolved

cmacknz reviewed Nov 18, 2024

View reviewed changes

internal/pkg/otel/manager/manager.go Show resolved Hide resolved

blakerouse added 3 commits November 18, 2024 16:33

Rename otelStatus to collector.

c92dda0

Add coordinator state information to state.yml diagnostics.

f8263e8

Add comment to healthcheckv2extension.

2b9d859

michalpristas approved these changes Nov 19, 2024

View reviewed changes

MichaelKatsoulis approved these changes Nov 19, 2024

View reviewed changes

Merge branch 'main' into otel-hybrid

65f0717

Fix integration test.

ef72cc8

alexsapran approved these changes Nov 20, 2024

View reviewed changes

blakerouse merged commit b07566b into elastic:main Nov 20, 2024
14 checks passed

mergify bot pushed a commit that referenced this pull request Nov 20, 2024

Add support for running EDOT inside of running Elastic Agent (#5767)

7cfc0ba

(cherry picked from commit b07566b) # Conflicts: # NOTICE.txt # go.mod # go.sum

mergify bot mentioned this pull request Nov 20, 2024

[8.x](backport #5767) Add support for running EDOT inside of running Elastic Agent #6096

Merged

6 tasks

pchila mentioned this pull request Dec 18, 2024

Add small doc about running a standalone agent in otel mode elastic/ingest-docs#1572

Merged

mergify bot mentioned this pull request Dec 19, 2024

[8.x] Add small doc about running a standalone agent in otel mode (backport #1572) elastic/ingest-docs#1574

Merged

		// prevent aggregate statuses from flapping between StatusStarting and StatusOK
		// as components are started individually by the service.

Add support for running EDOT inside of running Elastic Agent #5767

Add support for running EDOT inside of running Elastic Agent #5767

Conversation

blakerouse commented Oct 11, 2024 • edited Loading

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

mergify bot commented Oct 11, 2024

mergify bot commented Oct 11, 2024

mergify bot commented Oct 15, 2024

mergify bot commented Oct 15, 2024

mergify bot commented Oct 25, 2024

mergify bot commented Nov 7, 2024

blakerouse commented Nov 12, 2024

elasticmachine commented Nov 13, 2024

michalpristas left a comment

Choose a reason for hiding this comment

michalpristas Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

michalpristas Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

blakerouse commented Nov 18, 2024

pierrehilbert commented Nov 18, 2024

blakerouse commented Nov 18, 2024 • edited Loading

cmacknz commented Nov 18, 2024

cmacknz commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz commented Nov 18, 2024

blakerouse commented Nov 18, 2024

blakerouse commented Nov 18, 2024

blakerouse commented Nov 18, 2024

cmacknz commented Nov 18, 2024

blakerouse commented Nov 18, 2024

cmacknz commented Nov 18, 2024

michalpristas left a comment

Choose a reason for hiding this comment

blakerouse commented Nov 19, 2024

elastic-sonarqube bot commented Nov 19, 2024

Quality Gate passed

alexsapran left a comment

Choose a reason for hiding this comment

blakerouse commented Oct 11, 2024 •

edited

Loading

michalpristas Nov 18, 2024 •

edited

Loading

michalpristas Nov 18, 2024 •

edited

Loading

michalpristas Nov 18, 2024 •

edited

Loading

blakerouse commented Nov 18, 2024 •

edited

Loading

cmacknz Nov 18, 2024 •

edited

Loading