Replace deprecated _xpack endpoints #9656

ycombinator · 2018-12-19T01:37:26Z

In 7.0.0, Elasticsearch is deprecating most _xpack/* endpoints. See elastic/elasticsearch#35958.

This PR updates the Beats codebase, except test fixtures, with the appropriate replacements for the deprecated endpoints.

jsoriano

Will this break compatibility with older versions? I think we should support both paths at least during some versions.

ycombinator · 2018-12-20T16:32:35Z

Will this break compatibility with older versions? I think we should support both paths at least during some versions.

Yes, you are correct. Given that ES is deprecating the old endpoints only starting with 7.0.0, things would break for a user that ran Beats 7.0.0 (with the changes in this PR) against an ES 6.7.0 node. I'm assuming this combination will be supported, following from https://www.elastic.co/support/matrix#matrix_compatibility.

Now, if ES could perform the deprecation in 6.7.0 itself, then the changes in this PR would not cause a break. Note that the changes in this PR will go only into 7.0.0/master and not be backported.

Do you agree with my assessment above? If so, should I ask on elastic/elasticsearch#35958 if they would be willing to perform the deprecations starting in 6.7.0 itself?

ycombinator · 2019-01-03T13:40:06Z

@jsoriano Any thoughts about my last comment?

jsoriano · 2019-01-03T16:31:07Z

@ycombinator oh sorry, I didn't see this comment.

Yes, I guess that having both paths already in ES 6.7.0 can also be an option.

ycombinator · 2019-01-03T19:11:17Z

@jsoriano Looks like the new X-Pack APIs used in this PR all exist in ES 6.7.0 already, so there should be no breakage if a user tried to use Beats 7.0.0 (with this PR's changes in it) against ES 6.7.0.

Related: elastic/elasticsearch#35958 (comment)

ycombinator · 2019-01-03T20:33:05Z

@ph @kvch @urso Would be good to get one of your 👀 on this PR too, I think. Thanks!

urso

We want Beats 6.7 and ES 7.0 to work, but also Beats 7.0 and ES 6.7 to work correctly. In fact, it is recommended to update Elasticsearch first, but yet we're used to Beats users running against older ES versions.

The Connect on the beats ES client fetches the Elasticsearch version. You can access the vesion via client.GetVersion() I think. Maybe we can add some logic to take the version into account. E.g.

func apiCall(v *common.Version, api, path string) string {
  if v.Major >= 7 || (v.Major == 6 && v.Minor == 7) {
    return fmt.Sprintf("/_%v/%v", api, path)
  }
  return fmt.Sprintf("/_xpack/%v/%v", api, path)
}

and create the URL via:

client.Request("GET", apiCall(client.GetVersion(), "ml", "anomaly_detectors"), ...)

Or some ApiCall struct getting you the path by version:

var mlAnomalyCall = apiCall{"ml", "anomaly_detectors"}

...

client.Request("GET", mlAnonamlyCall.Path(client.GetVersion()), ...)

As _xpack namespace is still available in 7.0 I'd rather fallback to use _xpack if the version is unknown in the code (which is the behavior of the apiCall function).

WDYT?

urso

We want Beats 6.7 and ES 7.0 to work, but also Beats 7.0 and ES 6.7 to work correctly. In fact, it is recommended to update Elasticsearch first, but yet we're used to Beats users running against older ES versions.

The Connect on the beats ES client fetches the Elasticsearch version. You can access the vesion via client.GetVersion() I think. Maybe we can add some logic to take the version into account. E.g.

func apiCall(v *common.Version, api, path string) string {
  if v.Major >= 7 || (v.Major == 6 && v.Minor == 7) {
    return fmt.Sprintf("/_%v/%v", api, path)
  }
  return fmt.Sprintf("/_xpack/%v/%v", api, path)
}

and create the URL via:

client.Request("GET", apiCall(client.GetVersion(), "ml", "anomaly_detectors"), ...)

Or some ApiCall struct getting you the path by version:

var mlAnomalyCall = apiCall{"ml", "anomaly_detectors"}

...

client.Request("GET", mlAnonamlyCall.Path(client.GetVersion()), ...)

As _xpack namespace is still available in 7.0 I'd rather fallback to use _xpack if the version is unknown in the code (which is the behavior of the apiCall function).

WDYT?

jsoriano · 2019-01-04T10:04:43Z

I'd also prefer something like @urso proposal

ycombinator · 2019-01-04T12:26:07Z

I'm not sure we need the conditional logic, but maybe I'm missing something.

We want Beats 6.7 and ES 7.0 to work, but also Beats 7.0 and ES 6.7 to work correctly.

Let's break this statement into two, but in reverse order:

We want ... Beats 7.0 and ES 6.7 to work correctly.

This PR targets master, meaning the changes in this PR will go into Beats 7.0. As I mentioned in my earlier comment, the newer ES API endpoints (the ones without _xpack) are available in ES 6.7 as well. So the changes in this PR (i.e. calling the newer ES API endpoints) should work with ES 6.7.
We want Beats 6.7 and ES 7.0 to work.

I wasn't planning on backporting this PR to 6.x, meaning the changes in this PR will not go into Beats 6.7. If a user runs Beats 6.7 against ES 7.0, that will still work as the older (_xpack) API endpoints are still available in ES 7.0.

ycombinator · 2019-01-04T14:11:22Z

Chatted with @urso off-PR. He explained to me that we want to be even more liberal in our compatibility story than what's documented in the support matrix. So I'm going to implement his version check proposal in this PR now.

ph · 2019-01-04T14:22:43Z

@ycombinator do you mind also updating functionbeat's licenser for this change?
https://github.com/elastic/beats/blob/master/x-pack/functionbeat/licenser/elastic_fetcher.go#L114

ycombinator · 2019-01-04T14:35:13Z

@ph if I'm following that code correctly, it is calling the GET /_xpack API. AFAIK, that endpoint is not going away, even in ES 7.0.

ycombinator · 2019-02-04T13:07:10Z

jenkins, test this

ycombinator · 2019-02-04T17:19:12Z

jenkins, test this

ycombinator · 2019-02-04T17:35:38Z

jenkins, test this

ruflin

LGTM. One minor question.

Potentially worth a changelog even though users should not be impacted?

We should add this to our manual testing checklist (if we already have one). I didn't test it locally.

ruflin · 2019-02-04T21:15:47Z

metricbeat/module/elasticsearch/ml_job/ml_job.go

@@ -33,7 +33,7 @@ func init() {
 }

 const (
-	jobPath = "/_xpack/ml/anomaly_detectors/_all/_stats"
+	jobPathSuffix = "/anomaly_detectors/_all/_stats"


This has no ml prefix anymore?

In 7.0.0+, the preferred path is _ml/anomaly_detectors/_all/_stats (notice the underscore in _ml). Before that version, the path is _xpack/ml/anomaly_detectors/_all/_stats (notice that ml does not contain the underscore).

Given this difference I decided to only store the suffix, anomaly_detectors/_all/_stats, in the variable here. Further below I concatenate either /_ml or _xpack/ml to the suffix, depending on the ES version we're talking to.

ycombinator · 2019-02-04T22:36:04Z

One minor question.

I think the question was about the ML prefix? If so, I've answered it in the comment thread.

Potentially worth a changelog even though users should not be impacted?

If a user runs a Beat < 7.0.0 with ES >= 7.0.0, they will see deprecation logs in their Elasticsearch logs for Elasticsearch X-Pack API calls made from their Beat. So I added a CHANGELOG entry that says that they won't see such calls with this PR 😄.

We should add this to our manual testing checklist (if we already have one). I didn't test it locally.

++ where is this checklist kept? I can add this item to it.

ycombinator · 2019-02-04T23:49:17Z

jenkins, test this

ruflin · 2019-02-05T07:28:43Z

For the list, not sure if it exists yet.

ycombinator requested a review from a team as a code owner December 19, 2018 01:37

ycombinator added review v7.0.0-alpha1 Filebeat Filebeat libbeat Metricbeat Metricbeat labels Dec 19, 2018

ycombinator mentioned this pull request Dec 19, 2018

Upgrade test Docker images to latest versions #9198

Merged

ycombinator force-pushed the xpack-endpoints branch from 1c9ae30 to bda5db4 Compare December 20, 2018 14:24

ycombinator requested review from jsoriano and ph December 20, 2018 14:27

jsoriano requested changes Dec 20, 2018

View reviewed changes

ycombinator mentioned this pull request Jan 3, 2019

Deprecate X-Pack centric REST endpoints elastic/elasticsearch#35958

Closed

11 tasks

jsoriano approved these changes Jan 3, 2019

View reviewed changes

ycombinator force-pushed the xpack-endpoints branch from f755cad to 904b45f Compare January 3, 2019 20:03

ycombinator requested review from a team as code owners January 3, 2019 20:03

urso reviewed Jan 4, 2019

View reviewed changes

ycombinator added v7.0.0 and removed v7.0.0-alpha1 labels Feb 1, 2019

ycombinator force-pushed the xpack-endpoints branch from 9e6b4de to 1fd11c0 Compare February 1, 2019 23:07

ycombinator force-pushed the xpack-endpoints branch from b090923 to 885b74b Compare February 4, 2019 17:30

ruflin approved these changes Feb 4, 2019

View reviewed changes

ycombinator added 16 commits February 4, 2019 14:25

Replace deprecated _xpack endpoints

475a510

Fixing rebase issues

079be4f

Updating generated files

a432d62

Add version check to metricbeat es module integration test

4b6eddc

Regenerating generated files

331a462

Add version check to filebeat ML system test

29cedc3

Add version check to metricbeat es module system test

aa3e4b0

Add version check to libbeat ML importer integration test

5cc0212

Add version check to libbeat ML importer code

587b9a5

Make Hound happy

e9e444f

Add version checks to metricbeat elasticsearch module code

60c9738

Fixing formatting

eb369ef

Moving version-based service URI construction to Fetch()

e1e66e3

Fixing syntax

e3b830c

Regenerating generated files

0c40078

Adding CHANGELOG entry

d068660

ycombinator force-pushed the xpack-endpoints branch from 885b74b to d068660 Compare February 4, 2019 22:29

ycombinator merged commit fd98c11 into elastic:master Feb 5, 2019

ycombinator deleted the xpack-endpoints branch February 5, 2019 03:40

ycombinator mentioned this pull request Feb 14, 2019

Replace _xpack ES API endpoints with new bridge APIs #7909

Closed

ycombinator mentioned this pull request Mar 11, 2019

Replace _xpack/monitoring/* with _monitoring/* ES API calls #9480

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace deprecated _xpack endpoints #9656

Replace deprecated _xpack endpoints #9656

ycombinator commented Dec 19, 2018 •

edited

Loading

jsoriano left a comment

ycombinator commented Dec 20, 2018

ycombinator commented Jan 3, 2019

jsoriano commented Jan 3, 2019 •

edited

Loading

ycombinator commented Jan 3, 2019 •

edited

Loading

ycombinator commented Jan 3, 2019

urso left a comment

urso left a comment

jsoriano commented Jan 4, 2019

ycombinator commented Jan 4, 2019 •

edited

Loading

ycombinator commented Jan 4, 2019 •

edited

Loading

ph commented Jan 4, 2019

ycombinator commented Jan 4, 2019

ycombinator commented Feb 4, 2019

ycombinator commented Feb 4, 2019

ycombinator commented Feb 4, 2019

ruflin left a comment

ruflin Feb 4, 2019

ycombinator Feb 4, 2019 •

edited

Loading

ycombinator commented Feb 4, 2019 •

edited

Loading

ycombinator commented Feb 4, 2019

ruflin commented Feb 5, 2019

Replace deprecated _xpack endpoints #9656

Replace deprecated _xpack endpoints #9656

Conversation

ycombinator commented Dec 19, 2018 • edited Loading

jsoriano left a comment

Choose a reason for hiding this comment

ycombinator commented Dec 20, 2018

ycombinator commented Jan 3, 2019

jsoriano commented Jan 3, 2019 • edited Loading

ycombinator commented Jan 3, 2019 • edited Loading

ycombinator commented Jan 3, 2019

urso left a comment

Choose a reason for hiding this comment

urso left a comment

Choose a reason for hiding this comment

jsoriano commented Jan 4, 2019

ycombinator commented Jan 4, 2019 • edited Loading

ycombinator commented Jan 4, 2019 • edited Loading

ph commented Jan 4, 2019

ycombinator commented Jan 4, 2019

ycombinator commented Feb 4, 2019

ycombinator commented Feb 4, 2019

ycombinator commented Feb 4, 2019

ruflin left a comment

Choose a reason for hiding this comment

ruflin Feb 4, 2019

Choose a reason for hiding this comment

ycombinator Feb 4, 2019 • edited Loading

Choose a reason for hiding this comment

ycombinator commented Feb 4, 2019 • edited Loading

ycombinator commented Feb 4, 2019

ruflin commented Feb 5, 2019

ycombinator commented Dec 19, 2018 •

edited

Loading

jsoriano commented Jan 3, 2019 •

edited

Loading

ycombinator commented Jan 3, 2019 •

edited

Loading

ycombinator commented Jan 4, 2019 •

edited

Loading

ycombinator commented Jan 4, 2019 •

edited

Loading

ycombinator Feb 4, 2019 •

edited

Loading

ycombinator commented Feb 4, 2019 •

edited

Loading