-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhanncement for host.ip and host.mac] Disabling netinfo.enabled option of add-host-metadata processor #36506
Conversation
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
This pull request is now in conflicts. Could you fix it? 🙏
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- This change seems to be only done in metricbeat. Doesn't the same happen in Filebeat and any other Beat?
- Do we need additional documentation around this how a user could enable it (again) if they wanted to?
x-pack/metricbeat/cmd/root.go
Outdated
{"add_docker_metadata": nil}, | ||
{"add_kubernetes_metadata": nil}, | ||
|
||
// We check for the existance of environmental variable NETINFO. Related to https://github.com/elastic/integrations/issues/6674 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets have some more details here in the code docs. Jumping to the very long Github issue is not helpful. We can still leave the reference in, but lets add more details here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some comments in beb7e85
Indeed eg. filebeat has the same configuration see beats/x-pack/filebeat/cmd/root.go Line 41 in 1c5541e
My first thought here is that we need it only for metricbeat and filestream (if we decide to) for now as these are related to kubernetes. By a first look, auditbeat, osquerybeat and packetbeat initialise add_host_metadata processor as well, but dont need we need it for k8s. @martijnvg do you think we need it also for filebeat? Our tests were run only for metrics that is why we focused only in metricbeat I guess
For now I added it only in the manifests https://github.com/elastic/elastic-agent/pull/3354/files#diff-b85244108b202e9acc2f4eca1f918bb448bbd10a816a83123ec17f7cd3ca92b6 The other place I see that might feet is here: https://www.elastic.co/guide/en/fleet/current/agent-environment-variables.html |
I think this depends on whether local-link addresses add any value in the non metric use cases? I don't think so? But I think I'm not the person to answer this question. |
My understanding is that when any of these Beats / Elastic Agent are run under k8s, the problem would exist. Is there anything special here around metrics? Is the assumption that the other beats are not run in k8s? |
x-pack/metricbeat/cmd/root.go
Outdated
if valueNETINFO == "false" && status == true { | ||
return []mapstr.M{ | ||
{"add_host_metadata": mapstr.M{ | ||
"netinfo.enabled": "false", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why to enable/disable this at this level and not inside the processor's implementation?
Wouldn't that be more generic without the need to configure every single Beat (also covering Agent)? And that would cover @ruflin 's concern at #36506 (comment), right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 to do that inside the processor implementation. I think in the implementation we could even automatically check if this is running inside k8s or inside a container and either disable netinfo completely or filter out link-local addresses. See also elastic/integrations#6674 (comment).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you all team.
Reverted changes per beat module and made those inside the processor. Building a new image and retesting and let you know
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding the fix only to add-host-metadata processor (https://github.com/elastic/beats/pull/36506/files#diff-03b979565735e6707425275586e3215f44192f46e1ac3b456516d93c653eba13R38) seems that works only for metricbeat. Somehow seems we override it for filestream? Will keep looking
Lessons learned, you need to remove both builds in order to force agent image to build from local code: So all work by disabling netinfo in processor : For documentation please review: elastic/ingest-docs#463 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change LGTM. I left one comment about the env variable's name.
I think that skipping by default if k8s or containarized environment is detected would possibly be a breaking change at this point. Maybe some users want these data even in k8s envs.
So I think it's safer to go with the current approach.
// Setting environmental variable NETINFO:false in Elastic Agent pod will disable the netinfo.enabled option of add_host_metadata processor | ||
// This will result to events not being enhanced with host.ip and host.mac | ||
// Related to https://github.com/elastic/integrations/issues/6674 | ||
valueNETINFO, _ := os.LookupEnv("NETINFO") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My only question would be the naming we want to choose here. Maybe sth like ELASTIC_NETINFO
would be more explicit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree lets not introduce a breaking change (I was thinking an old comment you had done).
For the naming you are killing me :) but I was expecting the debate , so seeing the page again https://www.elastic.co/guide/en/fleet/current/agent-environment-variables.html, I would go for ELASTIC_AGENT_NETINFO as strickly speaking this is a change on the agent side.
I would wait until tomorrow to see if others have any opinion and I would need to change all prs
I agree that this can potentially be considered a breaking change, at the same I'm not sure the use case where all these ip addresses would be used. If we don't change the default, most users will not get the benefits and have to make active changes to get the performance and storage benefits. Do we agree |
// Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will disable the netinfo.enabled option of add_host_metadata processor | ||
// This will result to events not being enhanced with host.ip and host.mac | ||
// Related to https://github.com/elastic/integrations/issues/6674 | ||
valueNETINFO, _ := os.LookupEnv("ELASTIC_NETINFO") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets make sure we have this documented. Can you expand the changelog to contain a mention of this env variable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me know if this is ok, I added some info also there.
Also see elastic/ingest-docs#463
I believe we could just set
If we agree on this here we can even change the manifests in this PR, or change those in a follow-up if we want to discuss it more. Whatever that fits better. |
SGTM. The main thing that's important to me is that |
Thanks again all for the quick feedback. Default value in manifests is ELASTIC_NETINFO:false. So the host.ip and host.mac will be removed by default |
I that also true for logs or just for metrics? |
Both . See tests #36506 (comment) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The patch LGTM! Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Lets make sure we get this deployed directly in our test environments with the new configs and also get it into our cloud environments.
if valueNETINFO == "false" { | ||
return Config{ | ||
NetInfoEnabled: false, | ||
CacheTTL: 5 * time.Minute, | ||
ExpireUpdateTimeout: time.Second * 10, | ||
ReplaceFields: true, | ||
} | ||
} else { | ||
return Config{ | ||
NetInfoEnabled: true, | ||
CacheTTL: 5 * time.Minute, | ||
ExpireUpdateTimeout: time.Second * 10, | ||
ReplaceFields: true, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional suggestion to increase readability
if valueNETINFO == "false" { | |
return Config{ | |
NetInfoEnabled: false, | |
CacheTTL: 5 * time.Minute, | |
ExpireUpdateTimeout: time.Second * 10, | |
ReplaceFields: true, | |
} | |
} else { | |
return Config{ | |
NetInfoEnabled: true, | |
CacheTTL: 5 * time.Minute, | |
ExpireUpdateTimeout: time.Second * 10, | |
ReplaceFields: true, | |
} | |
return Config{ | |
NetInfoEnabled: valueNETINFO == "true", | |
CacheTTL: 5 * time.Minute, | |
ExpireUpdateTimeout: time.Second * 10, | |
ReplaceFields: true, | |
} |
…ion of add-host-metadata processor (elastic#36506) * adding check for host processor in case of K8s * Changing variable to ELASTIC_NETINFO * Updating changelog for variable ELASTIC_NETINFO
Proposed commit message
Relates to elastic/integrations#6674
Please explain:
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.How to test this PR locally
DEV=true SNAPSHOT=true PLATFORMS=linux/arm64 PACKAGES=docker mage package
kind load docker-image custom-agent-image:latest --name kind
elastic-package stack up -d -vvv --version=8.11.0-SNAPSHOT
elastic-agent-standalone-kubernetesHostIPs.yml.txt
elastic-agent-standalone-kubernetesHostIPs.yml.txt
This is particular the piece of code that our manifests will need to have from now on:
Related issues
Screens
Default 8.11.SNAPSHOT with host.ips and host.mac
hots
Standalone Agent with NETINNFO:false
Managed Agent with NETINNFO:false