Handling of TLS

[blakerouse]

Overview

Fleet Server needs to be bootstrapped by the Elastic Agent and be running with TLS. At the moment the bootstrap is all HTTP, which is not secure.

The goal is for in the default case that security is priority #1 followed by a good user experience. We would rather communication between a remote Elastic Agent and Fleet Server fail due to invalid TLS configuration versus being successful with insecure TLS communication.

Cloud Solution

When Fleet Server is bootstrapped by the Cloud then all the certificates will be provided to the bootstrap command allowing Fleet Server to start with the required certificates that the Cloud expects.

./elastic-agent enroll --fleet-server <connection_str> --enrollment-token <token> --cert <path_to_cert> --cert-key <path_to_cert_key>

On-prem Solution

In a customer deployment outside of Cloud they will have options.

Option 1 (Production Custom Certs)

They generate there own certificates that are verifiable by the other Elastic Agents in there organization, they pass these in the same way Cloud does.

./elastic-agent enroll --fleet-server <connection_str> --enrollment-token <token> --cert <path_to_cert> --cert-key <path_to_cert_key>

Option 2 (Auto-generated)

If no --cert* flags are passed to Elastic Agent then Elastic Agent will auto-generate a self-signed certificate with the hostname of the machine.

./elastic-agent enroll --fleet-server <connection_str> --enrollment-token <token>

This means that another Elastic Agent that is enrolling to this Fleet Server, needs to be explicit that it accepts the fact that the certificate is self-generated.

./elastic-agent enroll --url <url_to_fleet_server> --enrollment-token <token> --insecure

If they do not provide the --insecure flag then it will fail to actually connect to the Fleet Server to enroll. We should update this printed message to make it clear that is why it did not work.

Option 3 (HTTP-only BAD)

This is the final option in which they only want to run the Elastic Agent and Fleet Server with only HTTP. This is not recommended but is useful for development or in maybe special cases. In this case it is also best to ensure the Fleet Server is bound to the localhost which Elastic Agent will do by default with the --fleet-server-insecure-http flag.

./elastic-agent enroll --fleet-server <connection_str> --enrollment-token <token> --fleet-server-insecure-http

If they really want it to run in HTTP and not on localhost --fleet-server-bind 0.0.0.0 can be used.

[scunningham]

should we have an optional private key password, maybe pass by environment variable doesn't have to be on the command line?

[blakerouse]

@scunningham Yes we can add that but that will still need to be written to the fleet.yml so that password will still need to live on disk.

Maybe --cert-key-password? With --cert-key-password - pulling the value from stdin.

[scunningham]

If we have to persist, we should not include; we can't protect the yaml file. Worse to imply that its more secure with the password.

[ph]

@mostlyjason fyi

[scunningham]

@mostlyjason This is a product concern. The above proposal would place much of the onus on an on-prem customer to provide the cryptographic material in a production system. It will not be turnkey. However, it's not unreasonable to ask a customer to manage their own crypto; they often want to be able to control the key material.

For a on-prem customer that is just trying the system out, the --insecure mode of the agent will work but non-ideal. However, we've not come up with a better technical solution that isn't extremely complex.

I should note that in the above proposal, a cloud install should be turn-key for customers. The key material will be managed by the install process; and even if they allow self signed it will work due to the agent going through the proxy.

[mostlyjason]

Option 2 (Auto-generated) is a creative way to make it easier to get started and test it out.

If we have to persist, we should not include; we can't protect the yaml file. Worse to imply that its more secure with the password.

@scunningham you are saying that if the user wants to provide a password on their certificate, they should use option 1 instead? Option 2 will only auto-generate certs without a password?

As another option, the ES docs talk about storing a password in a keystore https://www.elastic.co/guide/en/elasticsearch/reference/7.11/configuring-tls.html#node-certificates. I'm not sure how similar it would be for fleet server or how much complication it adds.

If they really want it to run in HTTP and not on localhost --fleet-server-bind 0.0.0.0 can be used.

ES prevents this use case with its bootstrap checks https://www.elastic.co/guide/en/elasticsearch/reference/7.5/bootstrap-checks.html because it makes misconfiguring security easier. When the server runs on anything other than a loopback address, we should consider disabling fleet-server-insecure-http.

[scunningham]

@scunningham you are saying that if the user wants to provide a password on their certificate, they should use option 1 instead? Option 2 will only auto-generate certs without a password?

@mostlyjason I am saying that if user puts a password on their private key, they will need to pass it every time via environment variable in order to allow the agent to decrypt. If we only make them pass it in once, we have to persist either the password or the decrypted credential in an insecure way. We do not have a secure mechanism for protecting secrets in agent at the moment.

To minimize confusion, I was suggesting we don't allow the password option in the first release. We can add it later if customer's demand it.

[mostlyjason]

Thanks Sean that sounds good to me!

[mostlyjason]

One more use case to consider. If the user has multiple fleet servers and one or more is running with self-signed certs, and Elastic Agent does not have the --insecure parameter then it should only connect to the secure ones. If it has the parameter, then I imagine it could connect to insecure or secure servers?

[blakerouse]

Completed with elastic/beats#24142