[8.x](backport #5767) Add support for running EDOT inside of running Elastic Agent

[mergify]

What does this PR do?

Adds the ability to run the EDOT along side the running Elastic Agent.

This connects the EDOT into the coordinator of the Elastic Agent. At any point if any of these top-level keys (receivers, processors, exporters, extensions, service) exist in the configuration or policy for the elastic-agent the EDOT is started. If all of those keys are removed from the configuration or policy then the EDOT is automatically stopped. If any configuration change occurs the updated configuration is passed along to the EDOT to handle.

Why is it important?

This allows EDOT configuration to exist inside of the configuration or policy and allow it to work as expected.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

This is an addition and doesn't affect the way the current Elastic Agent runs at all.

How to test this PR locally

Place OTel configuration into the elastic-agent.yml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:

exporters:
  otlp:
    endpoint: otelcol:4317

extensions:
  health_check:
  pprof:

service:
  extensions: [health_check, pprof]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Run elastic-agent run -e.

Related issues

Closes #5796

---

This is an automatic backport of pull request #5767 done by Mergify.

[mergify]

Cherry-pick of b07566b has failed:

On branch mergify/bp/8.x/pr-5767
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit b07566b96a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .github/workflows/golangci-lint.yml
	new file:   changelog/fragments/1729011748-Add-EDOT-hybrid-mode.yaml
	modified:   control_v2.proto
	modified:   internal/pkg/agent/application/actions/handlers/handler_action_upgrade_test.go
	modified:   internal/pkg/agent/application/apm_config_modifier.go
	modified:   internal/pkg/agent/application/application.go
	modified:   internal/pkg/agent/application/config_test.go
	modified:   internal/pkg/agent/application/coordinator/coordinator.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_state.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_test.go
	modified:   internal/pkg/agent/application/coordinator/coordinator_unit_test.go
	modified:   internal/pkg/agent/application/coordinator/diagnostics_test.go
	modified:   internal/pkg/agent/application/gateway/fleet/fleet_gateway.go
	modified:   internal/pkg/agent/application/info/agent_id.go
	modified:   internal/pkg/agent/application/monitoring/liveness.go
	modified:   internal/pkg/agent/application/monitoring/liveness_test.go
	modified:   internal/pkg/agent/application/monitoring/reload/reload.go
	modified:   internal/pkg/agent/application/monitoring/v1_monitor.go
	modified:   internal/pkg/agent/application/upgrade/artifact/config.go
	modified:   internal/pkg/agent/application/upgrade/artifact/config_test.go
	modified:   internal/pkg/agent/application/upgrade/upgrade.go
	modified:   internal/pkg/agent/cmd/container.go
	modified:   internal/pkg/agent/cmd/container_test.go
	modified:   internal/pkg/agent/cmd/enroll_cmd.go
	modified:   internal/pkg/agent/cmd/enroll_cmd_test.go
	modified:   internal/pkg/agent/cmd/inspect.go
	modified:   internal/pkg/agent/cmd/otel.go
	modified:   internal/pkg/agent/cmd/otel_flags.go
	modified:   internal/pkg/agent/cmd/otel_flags_test.go
	deleted:    internal/pkg/agent/cmd/otel_windows.go
	deleted:    internal/pkg/agent/cmd/otel_windows_test.go
	modified:   internal/pkg/agent/cmd/status.go
	modified:   internal/pkg/agent/cmd/validate.go
	modified:   internal/pkg/agent/configuration/configuration.go
	modified:   internal/pkg/composable/controller.go
	modified:   internal/pkg/composable/providers/docker/docker.go
	modified:   internal/pkg/composable/providers/host/host.go
	modified:   internal/pkg/composable/providers/kubernetes/kubernetes.go
	modified:   internal/pkg/composable/providers/kubernetes/pod_test.go
	modified:   internal/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go
	modified:   internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets.go
	modified:   internal/pkg/composable/providers/local/local.go
	modified:   internal/pkg/composable/providers/localdynamic/localdynamic.go
	modified:   internal/pkg/config/config.go
	modified:   internal/pkg/config/config_test.go
	modified:   internal/pkg/config/loader.go
	modified:   internal/pkg/core/monitoring/config/config_test.go
	new file:   internal/pkg/otel/agentprovider/provider.go
	new file:   internal/pkg/otel/agentprovider/provider_test.go
	modified:   internal/pkg/otel/components.go
	deleted:    internal/pkg/otel/config_file_provider.go
	deleted:    internal/pkg/otel/config_file_provider_test.go
	deleted:    internal/pkg/otel/config_manager.go
	new file:   internal/pkg/otel/manager/extension.go
	new file:   internal/pkg/otel/manager/force_extension_converter.go
	new file:   internal/pkg/otel/manager/manager.go
	new file:   internal/pkg/otel/manager/manager_test.go
	new file:   internal/pkg/otel/otelhelpers/status.go
	new file:   internal/pkg/otel/otelhelpers/status_test.go
	modified:   internal/pkg/otel/run.go
	modified:   internal/pkg/otel/run_test.go
	modified:   internal/pkg/otel/validate.go
	modified:   internal/pkg/remote/client.go
	modified:   pkg/component/config.go
	modified:   pkg/control/v1/proto/control_v1.pb.go
	modified:   pkg/control/v1/proto/control_v1_grpc.pb.go
	modified:   pkg/control/v2/client/client.go
	modified:   pkg/control/v2/cproto/control_v2.pb.go
	modified:   pkg/control/v2/cproto/control_v2_grpc.pb.go
	modified:   pkg/control/v2/server/server.go
	modified:   pkg/control/v2/server/server_test.go
	modified:   pkg/features/features.go
	modified:   pkg/limits/limits.go
	modified:   pkg/testing/fixture.go
	modified:   testing/integration/diagnostics_test.go
	modified:   testing/integration/otel_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   NOTICE.txt
	both modified:   go.mod
	both modified:   go.sum

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.x/pr-5767 upstream/mergify/bp/8.x/pr-5767
git merge upstream/8.x
git push upstream mergify/bp/8.x/pr-5767

[mergify]

This pull request has not been merged yet. Could you please review and merge it @blakerouse? 🙏

[blakerouse]

@mergify rebase

[mergify]

rebase

☑️ Nothing to do

• -conflict [📌 rebase requirement]
• -closed [📌 rebase requirement]
• queue-position = -1 [📌 rebase requirement]
• any of:
  • #commits-behind > 0 [📌 rebase requirement]
  • #commits > 1 [📌 rebase requirement]
  • -linear-history [📌 rebase requirement]

[blakerouse]

@mergify update

[mergify]

update

☑️ Nothing to do

• -conflict [📌 update requirement]
• #commits-behind > 0 [📌 update requirement]
• -closed [📌 update requirement]
• queue-position = -1 [📌 update requirement]

[elastic-sonarqube]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube