Monitoring data lost when Elasticsearch cluster is in maintenance mode

[eedugon]

When Elasticsearch cluster wants to prevent write operations for maintenance purposes (cluster in read_only mode or indices are), Filebeat drops the monitoring data (it looks the internal queue is very small), and this can be a real problem for some users who might consider monitoring data with the same importance and the main data.

For confirmed bugs, please report:

• Version: 6.8 and 7.3

• Operating System: Linux

• Related to: Index in read-only mode not handled properly by Filebeat #13844

• Steps to Reproduce: Set your cluster in read_only mode and let Filebeat works for a few minutes, during this period no data will be written at all (monitoring or main data from). Then remove the read_only status at cluster level and see how the main data is all written but monitoring data during that time is gone forever.

Notes:

• If this is not considered a bug, which I could agree, maybe we should document it properly to ensure users have clear expectations about how monitoring data is handled and what importance is given to that data by the Beat design.
• This is probably applicable to all beats.
• Probably providing monitoring data a bigger and reliable queue (on disk) might help here.

[elasticmachine]

Pinging @elastic/stack-monitoring

[jlind23]

Backlog grooming: Closing it for now until further activity, can still be reopened if needed.