[Kubernetes - Metricbeat] Providing config option to disable Kubeadm config api requests

[gizas]

• Enhancement

NOTE: Dont merge until elastic/elastic-agent-autodiscover#98 is also merged

The elastic-agent-autodoscover library updated to v0.8.2

Proposed commit message

WHAT: Provide a configuration option to disable /api/v1/namespaces/kube-system/configmaps/kubeadm-config . In more details user will disable the api requests to the modules he configures by using configuration use_kubeadm: false

WHY: https://github.com/elastic/enhancements/issues/21172

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

• Install a kubernetes cluster with audit-logs enabled
• Create a local elastic stack with elastic-package stack up -d -v --version=8.15.0-SNAPSHOT
• Update the go.mod file of this repo and point to the elastic-agent-autodiscovery folder, where related changes should also exist. Your go.mod file should have the following entry:

replace github.com/elastic/elastic-agent-autodiscover => /Users/andreasgkizas/elastic/elastic-agent-autodiscover

• Run go mod tidy
• Install Filebeat and collect audit logs
• Follow instructions of this page to build the metricbeat binary

 GOOS=linux GOARCH=arm64 go build
 kubectl apply -f module/kubernetes/_meta/test/docs/01_playground/metricbeat.yaml
 kubectl cp metricbeat `kubectl get pod -n kube-system -l k8s-app=metricbeat -o jsonpath='{.items[].metadata.name}'`:/usr/share/metricbeat/ -n kube-system\n
 kubectl exec `kubectl get pod -n kube-system -l k8s-app=metricbeat -o jsonpath='{.items[].metadata.name}'` -n kube-system -- bash -c "metricbeat -e -c /etc/metricbeat.yml"

• Use use_kubeadm: false flag in your enabled modules
• Open your local kibana cluster, navigate to Discovery and search inside kibana audit-logs for relevant api requests

Related issues

• Relates #https://github.com/elastic/enhancements/issues/21172
• Relates #[autodiscover] - Providing config option to disable Kubeadm config api requests elastic-agent-autodiscover#98

Screenshots

With 8.15.0 Image no changes:

We enable all metricsets (included events) and we have 85 API requests

With This repo code but the flag is disable_kubeadm: false

We enable all metricsets (included events) and we still have 85 API requests

With This repo code but the flag is disable_kubeadm: true`

We have no API requests after 10:04 that metrics collection established

{"log.level":"info","@timestamp":"2024-07-03T10:04:20.520Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://elasticsearch:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gizas? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kubeadm upstream/kubeadm
git merge upstream/main
git push upstream kubeadm

[gizas]

I have updated description and autodiscovery library to make use of use_kubeadm variable

See comment elastic/elastic-agent-autodiscover#98 (comment)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kubeadm upstream/kubeadm
git merge upstream/main
git push upstream kubeadm

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kubeadm upstream/kubeadm
git merge upstream/main
git push upstream kubeadm

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b kubeadm upstream/kubeadm
git merge upstream/main
git push upstream kubeadm

[gizas]

@MichaelKatsoulis I added again the use_kubeadm option here

Reason is that kubelet metricsets, proxy and scheduler make use of AddClusterECSMeta (like here) which is using finally the kubernetes config struct.

I have repeated the tests and now works ok!

[MichaelKatsoulis]

LGTM

[gizas]

@elastic/beats-tech-leads can I have a review here? The pr looks good and tested

[cmacknz]

Approving go.mod changes.

[tetianakravchenko]

LGTM!