Document include_fields filter usage #3340
Labels
Comments
|
I cant get any of these examples working, with either filter, or processors, surely.... something is wrong. The only thing that seems to happen is sometimes when I use filter, the entire mysql tree of data just disappears but not because I have written drop_fields fields ['mysql'] but for no obvious reason. Every time I try these I run -configtest and get a Config OK, and I see data in Kibana, its just never filtered! This is really annoying. What we need is just to be able to completely blacklist a whole tree and then whitelist specific metrics we want because the mysql.status.raw is too big. - module: mysql
enabled: true
period: 1s
metricsets: ["status"]
processors:
- drop_fields:
when:
range:
mysql.status.aborted.clients.gte: 0
fields: ["mysql.status.aborted.clients"]
raw: false
hosts: ["tcp(127.0.0.1:3306)/"]
username: root
password: {{ mysql_root_password }}
- module: mysql
enabled: true
period: 1s
metricsets: ["status"]
processors:
- include_fields:
fields: ["mysql.status"]
- drop_fields:
fields: ["mysql.status.aborted", "mysql.status.binlog"]
raw: false
hosts: ["tcp(127.0.0.1:3306)/"]
username: root
password: {{ mysql_root_password }}
- module: mysql
enabled: true
period: 1s
metricsets: ["status"]
processors:
- drop_fields:
fields: ["beat", "metricset"]
raw: false
hosts: ["tcp(127.0.0.1:3306)/"]
username: root
password: {{ mysql_root_password }}
- module: mysql
enabled: true
period: 1s
metricsets: ["status"]
processors:
- drop_fields:
fields: ["mysql.status.raw.Access_denied_errors"]
raw: true
hosts: ["tcp(127.0.0.1:3306)/"]
username: root
password: {{ mysql_root_password }} |
|
Inside the modules, you must be replacing |
|
As I said in my original post it doesnt work with either filter, or processors, I tried all that code with both. No cheese. Can someone show a working filter for mysql.status.raw? There are too many fields so it would be nice to blacklist ALL and then whitelist only a list of desired ones. |
|
@madhavajay As the filtering happens on the module level, the filter would have to be: |
|
Looks like it only works with ["raw.Access_denied_errors"]. I also cannot get a combination of include_fields and drop_fields to whitelist fields rather than blacklist. However this is working for me if anyone else needs this. This will blacklist every raw field except raw.Access_denied_errors as an example. |
|
@madhavajay You are right, the filtering happens on the metricset level which to be honest has some issues if multiple metricsets are used. Sorry for the false information above. I did a quick test with combining include_fields and drop_fields which seemed to work: The result is that I have only |
|
I have switched to mysqlbeat for several reasons:
It would be great if the mysql module of metricbeat had those two features since having less things installed is always good. |
|
@madhavajay Thank you for your feedback. Can you please open a feature request in Beats, so we can easily track it? Recently we added support in Metricbeat for dynamic metricsets and as an example, we had an attempt to allow custom MySQL queries here: #3170, but we decided to collect data from Prometheus instead. |
|
@madhavajay We switched to implementing dynamic metricsets on Prometheus first as it is simpler. MySQL brings in quite some complexity because of the different query and result structures. Can you share some of the queries you are running? Best in a new github issue as suggested by monica. |
|
@madhavajay Just found this issue here which is already open: #2987 Best add your details there. |
|
Closing the docs issue here because we now have better documentation for |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For confirmed bugs, please report:
Have very hard times to get Metricbeat mysql raw filtering working. The docs even state that "filters" are obsolete and will be replaced by a new mechanism. By which?
https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-metricbeat.html
It seems the Metricbeat filtering stuff is not documented for mysql raw field usage. We tried all sorts of patterns, but it simply does not work.
In my case, I want to filter on Innodb buffer status, and already tried many patterns, but none of them worked:
.. and so on.
I even didn't find any examples on GH in any project how to filter on mysql raw status values.
The text was updated successfully, but these errors were encountered: