[META Issue]Enhancement for being able to toggle the kubeadm-config api calls

[gizas]

This is the META issue that will summarise all the needed sub issues that are needed in order to be able to disable the /api/v1/namespaces/kube-system/configmaps/kubeadm-config API calls through configuration.

The relevant enhancement was raised here: https://github.com/elastic/enhancements/issues/21172
See this issue for more details

The enhancement finally needs to be available in the elastic agent when users install the Kubernetes Integration. Additional we note that the kubeadm config calls are being made both as part of log collection process start and as metrics collection starts.

So in more details the task list is:

Tasks

• [Metricbeat Enhancememt]: Issue to disable the kube-adm calls in metricbeat. [Kubernetes - Metricbeat] Providing config option to disable Kubeadm config api requests #40086
• [Filebeat autodiscover Enhancememt]: Issue to disable the kube-adm calls when autodiscover is configured in filebeat. This specific issue needs to be implemented on kubernetes provider. Respectively the pod, node and service have eg PodMetadataGenerator that finally calls the getClusterInfoFromKubeadmConfigMap that is responsible to make those calls Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301
• [Filebeat Input Enhancememt]: Issue to disable the kube-adm calls when filestream input is configured. In this case check if relevant api calls are made: Issue was covered in Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301
• [Elastic Agent Enhcanement]: Issue that is needed to make available in the Elastic Agent policy of kubernetes integration all the config options introduced from previous issues, from filebeat and metricbeat. We will need to make checks with metrics and logs ingestion enabled, to check whether the API calls are minimised: [Kubernetes] Include kubeadm parameter integrations#11187

NOTES:

• In elastic agent the add_kubernetes_metadata processor starts by default and is not configurable
• In case Fleet Elastic Agent needs to configure the kubernetes provider, we will use the following installation method: https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html#_step_2_create_a_new_configmap

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[gizas]

@cmacknz I was testing the above enhancement through the integration, but I could not observe any improvements in case of applying the use_kubeadm variable through the integration.

I think the reason is that the kubernetes provider is already started and we call the GetPodMetadataGen here

So to my questions:

DEV=true EXTERNAL=true SNAPSHOT=true PLATFORMS=linux/amd64 PACKAGES=tar.gz mage -v package
failed to run compiled magefile: signal: killed

I have at least 7GB of memory, any ideas how else I can unblock myself here?

• Do we have an API for the elastic-agent to configure the add_kubernetes_metadata processors in advance?
• I guess the work of Allow disabling and configuring the add_x_metadata default global processors elastic-agent#4670 is still in progress? I can not use it now somehow?

[cmacknz]

Do we have an API for the elastic-agent to configure the add_kubernetes_metadata processors in advance?

Not yet.

I guess the work of elastic/elastic-agent#4670 is still in progress? I can not use it now somehow?

It was paused because we are working on getting rid of the beats sub-processes which would eliminate this problem.

I would suggest if you need control over this for k8s to expose it as a env var as was done for add_cloud_metadata. There is unfortunately no other easy change to make here, the global processor configuration is unnecessarily hard to do in the current agent architecture.

[henrikno]

@gizas Could we add a way to toggle this via an ENV var?

Btw. failed to run compiled magefile: signal: killed went away from me when I upgraded go 1.22 to go 1.23.

[swiatekm]

It took me a bit of time to figure out where the kubeadm behaviour is configured in elastic-agent, so I'm going to leave the results here for posterity. To start with, this is the default of our internal autodiscovery library, defined here. Elastic-agent implements this default here.

If we wanted to disable this behaviour in agent, the config fragment would look like the following:

providers:
  kubernetes:
    add_resource_metadata:
      namespace:
        use_kubeadm: false
      node:
        use_kubeadm: false

[gizas]

Oh I see that we refer https://github.com/elastic/elastic-agent-autodiscover/blob/main/kubernetes/metadata/config.go#L63-L64 and this uses use_kubeadm: true!!!!

So question would be how to initialise the kubeadm in agent provider: I see only this option https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html#_step_2_create_a_new_configmap correct?
And also the users would need to configure the use_kubeadm in policy of elastic/integrations#11187

[swiatekm]

Oh I see that we refer https://github.com/elastic/elastic-agent-autodiscover/blob/main/kubernetes/metadata/config.go#L63-L64 and this uses use_kubeadm: true!!!!

So question would be how to initialise the kubeadm in agent provider: I see only this option https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html#_step_2_create_a_new_configmap correct? And also the users would need to configure the use_kubeadm in policy of elastic/integrations#11187

I think you can also set this via overrides in the fleet policy after elastic/elastic-agent#6169. @blakerouse please let me know if I'm off the mark here.