Consider not adding host.ip metadata to k8s container metrics by default #6674
Comments
|
An example of the list of IPs that I have observed in documents: |
|
These ip addresses are added by Beats / Elastic Agent AFAIK. Initially the idea was that 1 or 2 host ip addresses would be shipped. But k8s + ipv6 wreck havoc to the data we ship. What are all these ipv6 addresses? One for each container? For ipv6, should we skip all the @tommyers-elastic @gizas any thoughts on the above? |
|
A quick search rendered that these are IPv6 link-local addresses
AFAIK these are not critical for most Observability use cases. |
|
I did some search today to be sure where this ips come from: Those are the nodes ips, so in other words the networking of the underlying host. The metrics from an nginx pod Node's networking: 4: veth038d14aa@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether aa:07:91:89:6d:0f brd ff:ff:ff:ff:ff:ff link-netns cni-fefa3105-06d0-56f5-302c-f5223545f4d3
inet 10.244.0.1/32 scope global veth038d14aa
valid_lft forever preferred_lft forever
5: vethebc54a3e@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether f2:12:90:39:f7:21 brd ff:ff:ff:ff:ff:ff link-netns cni-3b66ba8b-35bc-8292-9679-60fb0dea2237
inet 10.244.0.1/32 scope global vethebc54a3e
valid_lft forever preferred_lft forever
6: vethc8cbf2a8@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 6e:3d:06:e8:08:97 brd ff:ff:ff:ff:ff:ff link-netns cni-320ccfbc-5113-88ae-96ce-36be83b6692b
inet 10.244.0.1/32 scope global vethc8cbf2a8
valid_lft forever preferred_lft forever
7: vetha8e1da93@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether d2:be:13:f5:f2:9c brd ff:ff:ff:ff:ff:ff link-netns cni-d3fe6667-7f55-ae5f-978e-e2306aa7603c
inet 10.244.0.1/32 scope global vetha8e1da93
valid_lft forever preferred_lft forever
12: eth1@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:14:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.20.0.2/16 brd 172.20.255.255 scope global eth1
valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fc00:f853:ccd:e793::3/64 scope global nodad
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe12:3/64 scope link
valid_lft forever preferred_lft forever
I can see the option of |
|
@cmacknz I see that add_host_metadata processor is responsible to add host.* fields and is enabled by default (https://www.elastic.co/guide/en/fleet/current/add_host_metadata-processor.html). I have not found a way to override it t the moment in the Agent. |
|
The default set of processors each Beat runs when they are started by agent is defined in the code, the Beats don't read their own default configuration files when agent starts them. Here is the definition for Metricbeat: This is equal to the set of default global processors that are enabled in the default Metricbeat configuration file: The problem right now is that an agent policy has no concept of a global processor today, so there is no place in the agent policy to expose these. This is something we plan to do, but there's no date set for it yet. https://github.com/elastic/ingest-dev/issues/2442 is the tracking issue. Even if we did have this, we'd want the change to the configuration here to be conditional on whether the agent is running on Kubernetes. This might actually be easier to do in code. If you can come up with an alternate configuration, we would only want to apply it when the agent is running on k8s. Since the default processors are defined in code, if you have a function that can accurately detect that the agent runs on Kubernetes when these processor configurations are generated you can use it to conditionally change the You could add an option to |
|
late to the party here but just chiming in that from where i'm standing these IPs for sure just look like noise. if there was some useful mapping from ip<->resource then maybe they would be more useful. hopefully we get something like that as part of the asset management work. |
|
To add more to the issue, I have repeated some more tests with a GKE cluster with 15 nodes and there you can see more host.ips added (I count sth like 64) 
I agree is noise because you can find the same information also in the kubernetes.state_node entries. I have tried to add processors inside the integration (in the module level): 
Also : - add_host_metadata:
netinfo.enabled: falseIt seems that they dont apply. I guess that global processors are applied at beats level so they apply last and this means that they override our configuration. So long story short, host.ip fields remain in the event |
|
Hey folks! Please have a look into elastic/elastic-agent#90. This seems to be the reason. We need a way to disable/tune the Beats' global level processors and that's what elastic/elastic-agent#90 is trying to address. EDIT: check also #6674 (comment) comment which explains the same. |
|
Could we adjust the add_host_metadata processor to just not ship local ip addresses in the first place by default, no config needed? |
As I suggested in #6674 (comment), yes but you need to make it conditional on detecting that the Beat is running on Kubernetes where this information is not useful. I would think that just removing the IP fields unconditionally would be a breaking change, technically it would be on k8s as well but it is highly unlikely anyone depends on these fields today. |
|
Please find a workaround here https://github.com/elastic/beats/blob/fixinghostips/x-pack/metricbeat/cmd/root.go#L76-L93 I am in the process of building the image and testing e2e so will report my findings. But let me know if workaround is ok. Not the most elegant solution but what do you think? |
|
@gizas I think you don't need to check first for a k8s environment and then for the env var. Checking just for the env var directly should be enough. The pros of checking just the env var is that we don't break anything for the existing users/configurations and then only users that want to use the I would be ok with adding this but only as a temporary solution which means that we will create GH issue to keep track of this and find a better and more generic way to implement this. To my mind we need a way to configure Beats through Agent and at the moment we are "locked" with the default configs which is really bad. @ruflin @cmacknz are there any plans to fix this? I thought that elastic/elastic-agent#90 and then https://github.com/elastic/ingest-dev/issues/2442 would address this but then #6674 (comment) mentions that https://github.com/elastic/ingest-dev/issues/2442 would not be enough. So in that case maybe we need to prioritize elastic/elastic-agent#90 directly? |
|
Just to clarify it is an or not and.
But yes we can change the logic. This is more to prove that it is working. |
This would be a breaking change for users that today run on k8s and actually collecting the data we want to skip here. |
|
I managed to build the image and in my local cluster this works for now: After Removal of IPs: So summary of above:
|
|
Using the disk usage API on both the system cpu and the kubernetes pod data stream (on edge-lite) revealed that half of the disk usage is due to the I think it's important that we find a solution where these fields don't have such an impact when using default configurations. To keep the risk of breaking users at a minimum, and to make the implementation simple, I suggest we investigate the approach proposed by @cmacknz and @ruflin and remove non-interesting (local?) ip and mac addresses directly in the metadata processor go code if it detects that it's running on k8s. |
|
Team I updated the tasklist of the story with latest updates. The fix is working and for now we enable/disable the netinfo only with related environmental variable NETINFO:false inside agent pod. I have not managed to find a way to pass from the kubernetes Integration a config option to add-host-metadata processor. The add-host-metadata is initialised even before kubernetes processor and this does not allow us to pass config options to it. So for now I propose only elastic/kibana#165700 as a mean to help users in managed mode. Any other ideas? |
|
@gizas have you considered the proposal in my last message to change the host metadata processor to omit link-local IP addresses by default or by default when running inside a container? This would only require changes in the |
|
@felixbarny thanks again for reminder, see my last udpate in PR, now all tests seem to work with changes in the add_host_metadata processor only |
## Summary This PR add the environmental veriable ELASTIC_NETINFO in the managed and standalone manifests of Elasitc agent. The variable has been introduced here elastic/elastic-agent#3354 The reason for the introduction of the new variable ELASTIC_NETINFO:false by default in the manifests, is related with the work done elastic/integrations#6674
|
The Kubernetes Manifests will set ELASTIC_NETINFO:false by default. I am closing this issue for now as related work is done |
|
@gizas in which cases is |
|
@felixbarny in both managed and standalone manifests (https://github.com/elastic/kibana/pull/166156/files) the variable is false. So host.ip should not be added nowhere by default. |
andrewkroh
added
Team:Cloudnative-Monitoring
@martijnvg found out that in a test dataset for k8s container metrics, there were 100+ IP addresses attached as metadata.
We'll need to find out why that's the case and if these IPs make sense to add to container metrics as metadata. If these represent the IP addresses of the k8s node, it doesn't seem useful to add them as metadata to the container metrics anyway.
Tasks
The text was updated successfully, but these errors were encountered: