Support CLI flag shortcuts for Cloud configuration

[tsg]

To simplify the usage of Beats with Elastic Cloud, this proposes the addition of two new CLI flags:

   -E cloud.id=<cloud-id>
   -E cloud.auth"<user:pass>

These options are CLI only, but they overwrite settings in the configuration file.

Starting with cloud.auth because it's simpler, the username and password are used to overwrite output.elasticsearch.username and output.elasticsearch.password. Using the new flag is equivalent with doing -E output.elasticsearch.username=<user> -E output.elasticsearch.password=<pass>, but shorter.

The cloud-id, to be passed to the cloud.id setting, looks something like this:

staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw==

This can be decomposed using the following algorithm:

1. Everything up to the first : is ignored. This is an optional user friendly token that can be used to more easily identify the cloud ID. If no : is present, then the IS is passed as is to the next step.

2. The rest of the ID is base64 decoded:

   us-east-1.aws.found.io$cec6f261a74bf24ce33bb8811b84294f$c6c2ca6d042249af0cc7d7a9e9625743

3. There are three parts, separated by $, in order:

   host: us-east-1.aws.found.io
   es-id: cec6f261a74bf24ce33bb8811b84294f
   kibana-id: c6c2ca6d042249af0cc7d7a9e9625743

4. The es-id and kibana-id are parsed as URLs, and host is appended to their host parts. The https scheme and port 443 are added:

   es-url: https://cec6f261a74bf24ce33bb8811b84294f.us-east-1.aws.found.io:443
   kibana-url: https://c6c2ca6d042249af0cc7d7a9e9625743.us-east-1.aws.found.io:443

The resulting es-url and kibana-url are used to overwrite the settings output.elasticsearch.hosts and setup.kibana.hosts.

TODOs:

• Add new config settings Added cloud.id and cloud.auth config settings #4964
• Create a "Using Beats with Elastic Cloud" guide

[suyograo]

@tsg is it cloud-auth or cloud.auth? Why not make it consistent?

[suyograo]

I see beats uses dots in long options, ignore my comment :)

[tsg]

A small change of plans, after discussing things in our team meeting, we decided to have a way to specify the cloud.id/cloud.auth in the CLI but also in the config file. Otherwise, there's no reasonable transition from "trying things out" to "this is my configuration and want to run it as a service".

Playing with possible implementations, most reasonable is to have cloud.id and cloud.auth as config settings that can be overwritten from the CLI. So the CLI command looks like something like this:

./metricbeat -e -E cloud.id="cloudidtest:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyQyNDlmM2FmMWY0ZWVlMjRhODRlM2I0MDFlNjhhMWIyYSRkNGFjNzU1OWQ0Njc0YjdjOTFhYmUxMDg1NmQ4NDMwNA==" -E 'cloud.auth=elastic:t5piSpmqYRTIGKQhPrJkvGyx'