Check license x-pack

[ph]

Early look on the way of doing it.

This strategy has some limitations, we cannot set a specific license for a specific call but for the current need, I think it's OK.

I am currently in the process or splittings the Elasticsearch output module into a pipeline client an elasticsearch client, I will try to revisit the strategy there.

Note: I've tried to use the normal callbacks, but I've decided against for two reasons:

• Callback execution does not respect the insertion order.
• Pipeline clients and load pipelines doesn't use the same part so they don't respect configured callbacks.

I am writing test but I wanted to have an early look.

[ph]

After a walk I will take another way, I am trying to find an OK way without a hack.

[tsg]

I did a bit of user testing:

• Filebeat default -> Elasticsearch OSS: Fails to connect, as expected. It doesn't actually bail, but logs the error. I think that is fine for now.

The error, however, is not very clear: Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license or no license endpoint: error from server, response code: 400. I think the error is technically correct becasue _xpack/license doesn't exist, but we should probably catch this somehow and provide a very clear error message. The exact wording should be discussed.

• Filebeat OSS -> Elasticsearch OSS: Works, as expected.

• Filebeat OSS -> Elasticsearch default: Works, as expected.

• Filebeat default -> Elasticsearch default: Works, as expected.

[tsg]

@ph IMO we can go with this solution for now, given the time constraints. It's probably the minimal diff to make it happen and the code is simple enough. I'd be worried about a larger refactoring this late into the game.

[ph]

@tsg @andrewkroh I've made the changes and tweak the error handling/messaging.

[ph]

@tsg, @andrewkroh I will do a followup on this, I didn't want to waste time on docker-compose environment variable and slow the merge. Since we need to add integration test for this new scenario we can fix it at the same time.

[ph]

Need backport to 6.7 and 7.0.

[ph]

Create an issue for the flaky metricbeat test #11336

[tsg]

LGTM.

[ph]

@graphaelli This might affects apm or not, I will do a followup to make the check cleaner.

[graphaelli]

It certainly will, thanks for the heads up. cc @elastic/apm-server

[alexjurkiewicz]

This has caused some issues with our AWS Elasticsearch Service cluster. /_xpack is forbidden. How can I disable this license check?

[andrewkroh]

Checkout my comment in #11607. There are separate releases of Beats that do not include any x-pack features and therefore do not need that /_xpack API.

[alexjurkiewicz]

You released this change with oblique release notes and error messages that obfuscate the true issue. It's painful and disappointing to be hurt by your fight against another company.

[andrewkroh]

What error message did you get? This is the message I suggested. I thought it was pretty clear, but if you have suggestions let us know.

#11296 (comment)

this Elastic licensed requires an Elasticsearch server with X-Pack and a free basic license. Please use the Elasticsearch distribution that includes X-Pack or download the Apache 2.0 licensed distribution that does not include X-Pack features

We should add more details to the release notes. There's not really enough detail that for someone unfamiliar to understand the impact of the change or how to handle it. We'll fix that.

[alexjurkiewicz]

That message is very good. I would have liked to receive it. Instead I see:

{"level":"error","timestamp":"2019-04-02T05:23:59.482Z","caller":"instance/beat.go:907","message":"Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://xyz.es.amazonaws.com:443: Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials]"}

One tweak to your message I suggest would be to use the words "default" and "OSS" which are how the two builds of filebeat are distinguished. For example:

This version of libbeats only supports connecting to an Elasticsearch service with X-Pack and a free basic license. Please use the default Elasticsearch distribution or download an OSS version of this utility that does not include X-Pack features.

Putting it together with the raw message I received:

Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://xyz.es.amazonaws.com:443: Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials. This version of libbeats only supports connecting to an Elasticsearch service with X-Pack and a free basic license. Please use the default Elasticsearch distribution or download an OSS version of this utility that does not include X-Pack features]

In other words, the message is added by the same part of code which adds "cannot retrieve the elasticsearch license" as further context.

[urso]

It looks like the reason you didn't get this error message is because the failure occurs before the actual license check, as the HTTP request was not authorized correctly: unauthorized access, could not connect to the xpack endpoint, verify your credentials.

Problem with the authorization error is that Beats can not tell if it is talking with AWS Elasticsearch, OSS Elasticsearch, or XPack Elasticsearch. Or maybe a proxy doing authentication. Therefore we can not do any license check and should not suggest the version a user is running, so to not cause more confusion.

I wonder about the raw HTTP response. Maybe we can find something in the HTTP response headers, so to create a more informative error message similar to what you suggested.
Running curl -v https://xyz.es.amazonaws.com:443/_xpack should print the headers.

Reading the code we will print a complete warning if 400 or 405 status code is received.

Have you tried the OSS version? Did you get any errors?

[alexjurkiewicz]

AWS ES response for this URL doesn't have many tells:

> GET /_xpack HTTP/1.1
> Host: xyz.es.amazonaws.com
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Wed, 03 Apr 2019 22:39:36 GMT
< Content-Type: application/json
< Content-Length: 53
< Connection: keep-alive
< x-amzn-RequestId: xyz
< Access-Control-Allow-Origin: *

Is this license check the first HTTP request made to the ES endpoint? If the entire cluster is protected with HTTP auth, libbeat would notice before getting to the license check.

[ph]

Looking at the discussion and the header output not sure I can give more information or details about that situation in that specific case. Let's say that you connect to _xpack with the right credentials what do you get?

[urso]

It seems like we always get a 401 from AWS, even if authentication is configured correctly. It's a little tricky, as we don't really want to hack around wrong status codes, making assumptions about the actual service in use.