Elasticsearch _xpack endpoint is getting deprecated

[ph]

We have a few things inside beats that depends on some _xpack's endpoint and that endpoint is deprecated

• monitoring
• functionbeat's license checker

cc @urso @ruflin

[ruflin]

Should we also deprecate the xpack prefix in our configs and move them to the top level?

[ph]

@ruflin Yes we need to do that and what about the x-pack directory in our repo?

[ruflin]

I would treat the x-pack directory differently in the dev repo as we need a way to separate the files and that is only for engineers and not user facing.

[ycombinator]

@ph @urso My reading of this issue is that it's not specific to the GET _xpack endpoint, but the general class comprising various _xpack/* ES endpoints, right?

Assuming that's the case, I recently did an audit of the entire Beats codebase for use of _xpack/* ES endpoints, as part of resolving #17880. I found that there are three types of uses of _xpack/* ES endpoints:

1. Calls to the specific GET _xpack ES API endpoint. AFAIK this endpoint is not going away, not even in 8.0.0 / master (doc | source).
2. Calls to the specific GET _xpack/usage ES API endpoint. AFAIK this endpoint is not going away, not even in 8.0.0 / master (doc | source)
3. Calls to all other _xpack/* endpoints. All of these are guarded by version checks so they will only be called if the Beat is talking to an ES < 7.0.0. Some examples:

   • beats/libbeat/esleg/eslegclient/bulkapi.go

     Lines 152 to 154 in a141e88

     	if esVersion.Major < 7 {
     	path, err = makePath("_xpack", "monitoring", "_bulk")
     	} else {

   • beats/metricbeat/module/elasticsearch/elasticsearch.go

     Lines 269 to 271 in a141e88

     	if info.Version.Number.Major < 7 {
     	licensePath = "_xpack/license"
     	} else {

   • beats/metricbeat/module/elasticsearch/ml_job/ml_job.go

     Lines 74 to 76 in a141e88

     	if info.Version.Number.Major < 7 {
     	m.SetServiceURI("/_xpack/ml" + jobPathSuffix)
     	} else {

So for (1) and (2), we need to keep those uses as-is. For (3), I think we can get rid of the guarded clauses for < 7.0.0 only in master / 8.0.0 now? In other words, an 8.0.0+ Beat talking to an 6.x Elasticsearch cluster may not be a supported use case?

---

As for xpack.* configuration settings, I found these:

• xpack.enabled in the elasticsearch, kibana, logstash, and beat Metricbeat modules. This setting toggles whether the data collected by these modules is sent to "regular" metricbeat-* indices vs. stack monitoring .monitoring-* indices. I think we could introduce a different setting for this toggling in, say, 7.9 and deprecate the xpack.enabled setting. Then in 8.0.0 / master we could remove the xpack.enabled setting altogether. WDYT?
• xpack.monitoring.* settings in Beat configuration. These settings indicate that the Beat's own monitoring data must be sent to the production Elasticsearch cluster (instead of directly to the monitoring Elasticsearch cluster). These settings have been deprecated since 7.2.0 and replaced by monitoring.* settings. I think we can remove the xpack.monitoring.* settings only in 8.0.0 / master. WDYT?

[urso]

Sounds like a plan :)

Will xpack monitoring functionality be gone in ES/Kibana with 8.0 as well?

[ycombinator]

Will xpack monitoring functionality [routing monitoring data via the production cluster] be gone in ES/Kibana with 8.0 as well?

I believe this is still the plan. @chrisronline @jakelandis can confirm.

[jakelandis]

It is planned for the /_xpack/monitoring/_bulk (currently removed from master) as well as /_monitoring/bulk will be going in away in 8.0 in favor of the clients writing directly to the _bulk end point with the correctly formatted monitoring documents.

[chrisronline]

Yes, legacy collection will be removed in 8.0

[ycombinator]

Thanks @jakelandis and @chrisronline for confirming that the legacy collection path is going away from Elasticsearch in 8.0.0.

[ycombinator]

As for xpack.* configuration settings, I found these:

• xpack.enabled in the elasticsearch, kibana, logstash, and beat Metricbeat modules. This setting toggles whether the data collected by these modules is sent to "regular" metricbeat-* indices vs. stack monitoring .monitoring-* indices. I think we could introduce a different setting for this toggling in, say, 7.9 and deprecate the xpack.enabled setting. Then in 8.0.0 / master we could remove the xpack.enabled setting altogether. WDYT?

Regarding deprecating and renaming the xpack.enabled setting currently being used by Metricbeat stack modules to toggle collection and indexing for Stack Monitoring, I had an off-issue chat with @igoristic and @chrisronline about a new name. We all like mode. Here's how that would relate to the current xpack.enabled setting:

	Collect and index like normal Metricbeat modules (default)	Collect and index for Stack Monitoring purposes
xpack.enabled (current name; to be deprecated)	false	true
mode (proposed new name)	normal	stack_monitoring

One advantage of using mode is that it allows for additional values, if needed, in the future. IOW, it's more extensible than a boolean setting.

@urso @exekias At this point I'd like to check with you if there is any precedent in other Metricbeat modules for such a "mode" concept — where the module's metricsets behave fundamentally differently (not just changing things like the period or hosts) depending on a configuration setting. If so, it might be good to try and be consistent with our naming here.

[urso]

One advantage of using mode is that it allows for additional values, if needed, in the future. IOW, it's more extensible than a boolean setting.

+1. I like using an enum with descriptive names over a boolean :)

What does 'normal' exactly mean? Can we come up with a more descriptive name (e.g. legacy)? Do we even need a mode after 8.0?

At this point I'd like to check with you if there is any precedent in other Metricbeat modules for such a "mode" concept — where the module's metricsets behave fundamentally differently (not just changing things like the period or hosts) depending on a configuration setting. If so, it might be good to try and be consistent with our naming here.

I'm not really aware of any.

[ycombinator]

What does 'normal' exactly mean? Can we come up with a more descriptive name (e.g. legacy)? Do we even need a mode after 8.0?

Yeah, "normal" is not a very descriptive name, so happy to replace this with something more descriptive. What this mode will do is:

• collect only some basic metrics about stack products, not the very rich set of metrics that are needed by the Stack Monitoring UI and Telemetry, and
• index data into metricbeat-* indices (or whatever the default/user configuration dictates) in a structure that is in line with Metricbeat field naming/structure conventions.

So maybe default is a better name?

[ycombinator]

Do we even need a mode after 8.0?

Yes, I think we will. There is some future in which the Stack Monitoring UI may be driven by data in metricbeat-* indices OR this UI is merged into the Metrics UI (which is already driven by metricbeat-* indices). In that case we would want to use the normal (or default or whatever better name we come up with) mode and deprecate the stack_monitoring mode).

[ycombinator]

BTW, one related change to changing the name of the xpack.enabled setting. For ease of use we provide users the ability to do ./metricbeat modules enable $STACK_PRODUCT or ./metricbeat modules enable $STACK_PRODUCT-xpack today. The latter corresponds to configuring the $STACK_PRODUCT module with xpack.enabled: true.

Since we're changing the name of the setting from xpack.enabled to mode, we should probably change the the suffix from -xpack to -stack or -stack-mon or -stack-monitoring (too long?) something like that?

[urso]

default or minimal work for me. We can also introduce 'aliases' in the future if we want to. e.g. stack-monitoring could be replaced with rich? Naming is hard :)

Since we're changing the name of the setting from xpack.enabled to mode, we should probably change the the suffix from -xpack to -stack or -stack-mon or -stack-monitoring (too long?) something like that?

:(
Now I'm wondering if we want to change the setting at all in 7.x. Renaming the module from -xpack to something different is a breaking change. E.g. this can mess with automation done via Ansible.
Given that these settings are normally hidden behind the modules + users are encouraged to use metricbeat modules enable ... I'm inclined not to change anything on the configuration side.

Any plans on moving stack monitoring to use fleet integrations instead (yet another index again :) )?

[ycombinator]

Now I'm wondering if we want to change the setting at all in 7.x.

Well, it wouldn't be done as a breaking change in 7.x. We'd introduce the new names in 7.x and deprecate the old ones. In 8.0, we'd remove the old ones.

[EDIT] To clarify, users in 7.x would be able to do either metricbeat modules enable $PRODUCT-xpack OR metricbeat modules enable $PRODUCT-stackmon (just using the -stackmon suffix for demonstration). In 8.0, they'd only be able to do metricbeat modules enable $PRODUCT-stackmon.

Likewise, users in 7.x would be able to specify either xpack.enabled: true or mode: stack_monitoring. In 8.0, they'd only be able to specify mode: stack_monitoring.

Any plans on moving stack monitoring to use fleet integrations instead (yet another index again :) )?

Pinging @elastic/stack-monitoring-ui for any thoughts/news on this.

[urso]

Well, it wouldn't be done as a breaking change in 7.x. We'd introduce the new names in 7.x and deprecate the old ones. In 8.0, we'd remove the old ones.

👍

[ycombinator]

xpack.monitoring.* settings in Beat configuration. These settings indicate that the Beat's own monitoring data must be sent to the production Elasticsearch cluster (instead of directly to the monitoring Elasticsearch cluster). These settings have been deprecated since 7.2.0 and replaced by monitoring.* settings. I think we can remove the xpack.monitoring.* settings only in 8.0.0 / master.

PR for this change is up: #18608.

[ycombinator]

Calls to all other _xpack/* endpoints. All of these are guarded by version checks so they will only be called if the Beat is talking to an ES < 7.0.0... I think we can get rid of the guarded clauses for < 7.0.0 only in master / 8.0.0 now? In other words, an 8.0.0+ Beat talking to an 6.x Elasticsearch cluster may not be a supported use case?

PR for this change is up: #18609.

[ycombinator]

xpack.enabled in the elasticsearch, kibana, logstash, and beat Metricbeat modules. This setting toggles whether the data collected by these modules is sent to "regular" metricbeat-* indices vs. stack monitoring .monitoring-* indices. I think we could introduce a different setting for this toggling in, say, 7.9 and deprecate the xpack.enabled setting.

PR for this change is up: #18610

[jlind23]

Backlog grooming: Closing it for now until further activity.