Proposal: Dynamic Metricsets (MySQL) #3170
Conversation
Currently only Metricsets with a predefined query and data structure are possible. Some systems like MySQL allow dynamic queries for additional statistics or metrics to be queried (see Mysqlbeat as an example). JMX is an other example where dynamic queries for metrics are becoming critical. This PR proposes an generic implementation on how to handle dynamic metricsets based on the MySQL example.
A dynamic Metricset is based on an existing metricset. In the example here it is query. The configuration of the dynamic metricset will look as following:
```
- module: mysql
metricsets: ["query"]
namespace: "test"
query: "SHOW STATUS"
hosts: ["root:test@tcp(127.0.0.1:3306)/beat"]
```
A `query` must be defined for which the data should be fetched and a `namespace` to define under which namespace the data should be put. The namespace is required so the data does not conflict with an other metricset. The resulting event will look as following:
```
{
"@timestamp": "2016-12-12T08:35:34.711Z",
"beat": {
"hostname": "ruflin",
"name": "ruflin",
"version": "6.0.0-alpha1"
},
"metricset": {
"dynamic": "query",
"host": "127.0.0.1:3306",
"module": "mysql",
"name": "test",
"rtt": 16780
},
"mysql": {
"test": {
QUERY RESULT DATA
}
}
}
```
The data is put under the namespace `mysql.test` as the namespace defined was test. Ad additional fields `metricset.dynamic` is added with the original metricset name for easier debugging in case of errors, assuming there could be more then one metricset per module.
MySQL has different result types (SELECT *, COUNT(), SHOW DATABASES, ...). In Mysqlbeat this seems to be detected based on the result structure and based on this some magic happens. An alternative would be to have different dynamic metricsets in mysql and people would have to select the correct one. This would reduce the complexity on the code side but needs more user configuration. At first I would only support one type of queries and extend from there.
The field types of the events are dynamically generated at the moment. For future implementations we could allow a `fields` param in the config which can contain a yaml file with the fields definition out of which dynamically the template is generated and loaded. Alternative a `template` param could be used to point to a template in json format.
TODO:
* Currently errors do not take the event object into account. For the dynamic metricset this is required to make sure the name of the metricset is set correctly.
* Docs
* Code cleanup
| - module: mysql | ||
| metricsets: ["query"] | ||
| namespace: "test" | ||
| query: "SELECT * FROM example" |
There was a problem hiding this comment.
I see we have a bit of a config issue here, as the namespace and query apply only to the query metricset but structurally the config seems to indicate they apply to the whole module. But that's a more general Metricbeat issue, right?
There was a problem hiding this comment.
Yes, that is a more general problem we also have in other modules, for example ticks in the system module. To make it more obvious for the user we could namespace such configs with the metricset name, so it would be query.query which then probably needs a name change. Or for system, it would be cpu.ticks: true. Technically not really a difference, mainly a convention.
| query | ||
| fields: | ||
| - name: example | ||
| type: keyword |
There was a problem hiding this comment.
This is not quite right, I think, because the resulting object is mysql.test.example not mysql.query.example, right?
There was a problem hiding this comment.
yes, the PR is not complete yet and also not fully functional. Probably best would be to not ship a template in the first step at least.
|
Closing this in favor of #3213 |
ruflin
changed the title
Proposal: Dynamic Metricsets
Currently only Metricsets with a predefined query and data structure are possible. Some systems like MySQL allow dynamic queries for additional statistics or metrics to be queried (see Mysqlbeat as an example). JMX is an other example where dynamic queries for metrics are becoming critical. This PR proposes an generic implementation on how to handle dynamic metricsets based on the MySQL example.
A dynamic Metricset is based on an existing metricset. In the example here it is query. The configuration of the dynamic metricset will look as following:
A
querymust be defined for which the data should be fetched and anamespaceto define under which namespace the data should be put. The namespace is required so the data does not conflict with an other metricset. The resulting event will look as following:The data is put under the namespace
mysql.testas the namespace defined was test. Ad additional fieldsmetricset.dynamicis added with the original metricset name for easier debugging in case of errors, assuming there could be more then one metricset per module.MySQL has different result types (SELECT *, COUNT(), SHOW DATABASES, ...). In Mysqlbeat this seems to be detected based on the result structure and based on this some magic happens. An alternative would be to have different dynamic metricsets in mysql and people would have to select the correct one. This would reduce the complexity on the code side but needs more user configuration. At first I would only support one type of queries and extend from there.
The field types of the events are dynamically generated at the moment. For future implementations we could allow a
fieldsparam in the config which can contain a yaml file with the fields definition out of which dynamically the template is generated and loaded. Alternative atemplateparam could be used to point to a template in json format.Further potential dynamic metricsets are
jolokia.jmxandhttp.json. The json metricset could also be used as a foundation to make implementation of new metricsets easier as quite a few systems provide their metrics over http json.TODO: