-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor use of system.hostfs to fix cgroup metrics #24334
Refactor use of system.hostfs to fix cgroup metrics #24334
Conversation
Pinging @elastic/integrations (Team:Integrations) |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
Maybe to add this option to https://github.com/elastic/beats/blob/9735c6d51540b2fb72abfbbea0a165c033619542/libbeat/docs/shared-path-config.asciidoc too? Also in the manual testing section would be nice to check if setting the path through the configuration file works too. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice. Added some questions, mainly about the name of the option, and its availability for other beats.
@@ -28,6 +28,8 @@ | |||
// | |||
// path.config - Configuration files and Elasticsearch template default location | |||
// | |||
// system.hostfs - supplies an alternate filesystem root for containerized environments |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wdyt about calling it path.hostfs
now that it is another configurable path, and not only used by the system module?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yah, it might be worth it to have a second one for backwards compatibility?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, we would keep both at least till 8.0.
}{} | ||
|
||
if err := cfg.Unpack(&partialConfig); err != nil { | ||
return fmt.Errorf("error extracting default paths: %+v", err) | ||
} | ||
|
||
partialConfig.Path.Hostfs = partialConfig.Hostfs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this always safe? By introducing Hostfs
to the Path
struct we will allow users to overwrite the path in the configuration file already.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe I'm missing something, but is that any different from -E
flags?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Paths are configured twice I think. Once via --path.X
on startup + --system.hostfs
.
Then the config file is read and the path settings from the configuration file are applied again. The -E
flags for path settings only become effective during the second phase. That is why the shim bash scripts just use --path.X
CLI flags and we do not attempt to overwrite the defaults in the config.
The thing I was wondering: What if path.hostfs
is configured, but system.hostfs
is not? In that case you will set path.hostfs
to an empty string here. Is this on purpose? Why not:
if partialConfig.Path.Hostfs == "" {
partialConfig.Path.Hostfs = partialConfig.Hostfs
}
If the forced overwrite is on purpose a comment explaining the why would be helpful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, so, technically speaking, there is no path.hostfs
. There's only system.hostfs
, and we just expand the main Path
struct type with hostfs
, hence why that's being overwritten. Ideally it would only be path.hostfs
, but I'm mostly worried about breaking changes here, since I'm considering this to be primarily a bugfix so we can properly set hostfs
in the cgroup self-monitoring.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. It is still kinda "obscure" and I'm sure someone (me?, future you?) might change the logic on purpose or by accident. Can you please add a code comment about the why? Having the why in the PR review it will be lost.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, good point.
Bah, circular dependency with the |
Alright, I think I should probably put the hostfs flag declaration somewhere else. Just not sure where... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM to fix #23267 in Metricbeat.
I still wonder if the setting in the config file should be called path.hostfs
instead of system.hostfs
, specially if this can be used on other beats, where the path
prefix would make more sense to me than the system
one. But this could be done in a future change, and we could support both names during a time for compatibility with Metricbeat.
==== `system.hostfs` | ||
|
||
Specifies the mount point of the host's filesystem for use in monitoring a host. | ||
This can either be set in the config, or with the `--system.hostfs` CLI flag. This is used for cgroup self-monitoring. | ||
ifeval::["{beatname_lc}"=="metricbeat"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this setting is used for other beats too, I think we could add it as path.hostfs
for the config file setting, as other beats don't have a system
module. But we can leave it for a future change.
@@ -28,6 +28,8 @@ | |||
// | |||
// path.config - Configuration files and Elasticsearch template default location | |||
// | |||
// system.hostfs - supplies an alternate filesystem root for containerized environments |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, we would keep both at least till 8.0.
/test |
/test |
Yah, I have no idea what's wrong with CI here. |
* refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing (cherry picked from commit c70fe5c)
Hi! I noticed the issue has the v7.11.2 label but is not part of the release as far as I can tell. Is v7.13.0 going to be the next release after v7.11.x? If so, when is that to be expected? |
…p metrics (#24517) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing (cherry picked from commit c70fe5c) * fix changelog
* refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing
…reporting-changes * upstream/master: (23 commits) [Auditbeat] btmp offset check (elastic#24515) Clarify that the Tomcat module is for ingesting access logs (elastic#24543) [Ingest Manager] Move logging defaults to agent (elastic#24535) Update input-http-endpoint.asciidoc (elastic#24490) Fix typo in mqtt input docs (elastic#24509) [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) Add test for close.reader.after_interval to filestream input (elastic#24423) chore(ci): use beat_version instead of PR version (elastic#24446) Add syntax for multiple selector logging (elastic#24207) (elastic#24497) Update Golang to 1.15.9 (elastic#24442) [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) Add tests for encoding settings of filestream input (elastic#24426) [Ingest Manager] Sync on rename on windows (elastic#24504) Port four Harvester tests of log input to filestream in Golang (elastic#24250) [DOCS] Restructure content for SSL settings (elastic#24342) Move example to the correct location in reference docs (elastic#24455) Add unit tests for harvester.go of input-logfile package (elastic#24107) Fix type for uwsgi.status.worker.rss field (elastic#24468) [Ingest Manager] Logging to file disabled on enroll (elastic#24466) ...
* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (#24504) * Add tests for encoding settings of filestream input (#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (#24442) * Add syntax for multiple selector logging (#24207) (#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> * chore(ci): use beat_version instead of PR version (#24446) * Add test for close.reader.after_interval to filestream input (#24423) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (#24509) * Update input-http-endpoint.asciidoc (#24490) * [Ingest Manager] Move logging defaults to agent (#24535) [Ingest Manager] Move logging defaults to agent (#24535) * Clarify that the Tomcat module is for ingesting access logs (#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (#24404) Fixes #24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in #24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (#24571) * [CI] Add resilience when installing required tooling (#24542) * [CI] enable new flaky detector (#24464) * chore: do not pass beat version (#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Co-authored-by: Alex K <[email protected]> Co-authored-by: DeDe Morton <[email protected]> Co-authored-by: Lee Hinman <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: Andrew Pease <[email protected]> Co-authored-by: Victor Martinez <[email protected]>
* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (elastic#24504) * Add tests for encoding settings of filestream input (elastic#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (elastic#24442) * Add syntax for multiple selector logging (elastic#24207) (elastic#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> * chore(ci): use beat_version instead of PR version (elastic#24446) * Add test for close.reader.after_interval to filestream input (elastic#24423) * Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (elastic#24509) * Update input-http-endpoint.asciidoc (elastic#24490) * [Ingest Manager] Move logging defaults to agent (elastic#24535) [Ingest Manager] Move logging defaults to agent (elastic#24535) * Clarify that the Tomcat module is for ingesting access logs (elastic#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (elastic#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (elastic#24404) Fixes elastic#24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in elastic#24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (elastic#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (elastic#24571) * [CI] Add resilience when installing required tooling (elastic#24542) * [CI] enable new flaky detector (elastic#24464) * chore: do not pass beat version (elastic#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Co-authored-by: Alex K <[email protected]> Co-authored-by: DeDe Morton <[email protected]> Co-authored-by: Lee Hinman <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: Andrew Pease <[email protected]> Co-authored-by: Victor Martinez <[email protected]> (cherry picked from commit 5c6f1b6)
* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (elastic#24504) * Add tests for encoding settings of filestream input (elastic#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (elastic#24442) * Add syntax for multiple selector logging (elastic#24207) (elastic#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> * chore(ci): use beat_version instead of PR version (elastic#24446) * Add test for close.reader.after_interval to filestream input (elastic#24423) * Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (elastic#24509) * Update input-http-endpoint.asciidoc (elastic#24490) * [Ingest Manager] Move logging defaults to agent (elastic#24535) [Ingest Manager] Move logging defaults to agent (elastic#24535) * Clarify that the Tomcat module is for ingesting access logs (elastic#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (elastic#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (elastic#24404) Fixes elastic#24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in elastic#24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (elastic#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (elastic#24571) * [CI] Add resilience when installing required tooling (elastic#24542) * [CI] enable new flaky detector (elastic#24464) * chore: do not pass beat version (elastic#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Co-authored-by: Alex K <[email protected]> Co-authored-by: DeDe Morton <[email protected]> Co-authored-by: Lee Hinman <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: Andrew Pease <[email protected]> Co-authored-by: Victor Martinez <[email protected]> (cherry picked from commit 5c6f1b6)
* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (#24504) * Add tests for encoding settings of filestream input (#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (#24442) * Add syntax for multiple selector logging (#24207) (#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> * chore(ci): use beat_version instead of PR version (#24446) * Add test for close.reader.after_interval to filestream input (#24423) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (#24509) * Update input-http-endpoint.asciidoc (#24490) * [Ingest Manager] Move logging defaults to agent (#24535) [Ingest Manager] Move logging defaults to agent (#24535) * Clarify that the Tomcat module is for ingesting access logs (#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (#24404) Fixes #24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in #24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (#24571) * [CI] Add resilience when installing required tooling (#24542) * [CI] enable new flaky detector (#24464) * chore: do not pass beat version (#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Co-authored-by: Alex K <[email protected]> Co-authored-by: DeDe Morton <[email protected]> Co-authored-by: Lee Hinman <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: Andrew Pease <[email protected]> Co-authored-by: Victor Martinez <[email protected]> (cherry picked from commit 5c6f1b6)
* Update gosigar package after fix (#24502) * mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (#24504) * Add tests for encoding settings of filestream input (#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (#24442) * Add syntax for multiple selector logging (#24207) (#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> * chore(ci): use beat_version instead of PR version (#24446) * Add test for close.reader.after_interval to filestream input (#24423) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (#24509) * Update input-http-endpoint.asciidoc (#24490) * [Ingest Manager] Move logging defaults to agent (#24535) [Ingest Manager] Move logging defaults to agent (#24535) * Clarify that the Tomcat module is for ingesting access logs (#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (#24404) Fixes #24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in #24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (#24571) * [CI] Add resilience when installing required tooling (#24542) * [CI] enable new flaky detector (#24464) * chore: do not pass beat version (#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Co-authored-by: Alex K <[email protected]> Co-authored-by: DeDe Morton <[email protected]> Co-authored-by: Lee Hinman <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: Andrew Pease <[email protected]> Co-authored-by: Victor Martinez <[email protected]> (cherry picked from commit 5c6f1b6) * upadte changelog
Hi! Looks like we missed the 7.12.0 release. Is this still on schedule for 7.13.0? Any plans on backporting to the 7.12 branch? Thanks |
Bah, sorry, I expected the PR to get bumped by Steffen, so I missed it. |
* refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing (cherry picked from commit c70fe5c)
So, @schans we should be ready for 7.13, however, this is too major to backport to a future 7.12 release. |
|
||
Example: | ||
|
||
[source,yaml] | ||
------------------------------------------------------------------------------ | ||
path.logs: /var/log/beats | ||
------------------------------------------------------------------------------ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fearful-symmetry Is this the correct example? It seems to be a duplicate of the logs
example one section up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oops!
…lastic#24627) * Update gosigar package after fix (elastic#24502) * mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (elastic#24504) * Add tests for encoding settings of filestream input (elastic#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (elastic#24442) * Add syntax for multiple selector logging (elastic#24207) (elastic#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> * chore(ci): use beat_version instead of PR version (elastic#24446) * Add test for close.reader.after_interval to filestream input (elastic#24423) * Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (elastic#24509) * Update input-http-endpoint.asciidoc (elastic#24490) * [Ingest Manager] Move logging defaults to agent (elastic#24535) [Ingest Manager] Move logging defaults to agent (elastic#24535) * Clarify that the Tomcat module is for ingesting access logs (elastic#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (elastic#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (elastic#24404) Fixes elastic#24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in elastic#24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (elastic#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (elastic#24571) * [CI] Add resilience when installing required tooling (elastic#24542) * [CI] enable new flaky detector (elastic#24464) * chore: do not pass beat version (elastic#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <[email protected]> Co-authored-by: Michal Pristas <[email protected]> Co-authored-by: Noémi Ványi <[email protected]> Co-authored-by: Blake Rouse <[email protected]> Co-authored-by: EamonnTP <[email protected]> Co-authored-by: AndyHunt66 <[email protected]> Co-authored-by: Manuel de la Peña <[email protected]> Co-authored-by: Alex K <[email protected]> Co-authored-by: DeDe Morton <[email protected]> Co-authored-by: Lee Hinman <[email protected]> Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: Andrew Pease <[email protected]> Co-authored-by: Victor Martinez <[email protected]> (cherry picked from commit b5b66f5) * upadte changelog
What does this PR do?
This is a fix for #23267, wherein we weren't passing a
hostfs
to the metricbeat self-monitor code. In order to fix this in a somewhat clean way, I ended up migrating all of the hostfs handling tolibbeat/path
, and out of the system module entirely. This is a bit of a weird edge case forlibbeat/path
, so just tell me if this should be done somewhat differently.Why is it important?
As explained in the linked issue, this is causing problems for users setting
hostfs
.Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.How to test this PR locally
socket
,process
,entropy
anddiskio
are working correctly.system.hostfs
setting, such as in a container, like this:Insure that the above metricsets are still working, and that no errors like
instance/metrics.go:285 error getting group status: open /proc/33861/cgroup: no such file or directory
appear.Related issues
Logs
Tested, it looks like we're properly reporting cgroup metrics inside a docker container: