Segment store TLS certificate discovery

[EronWright]

Problem description
It is rather difficult to assign certificates to segment stores, when deploying to a containerized environment such as Kubernetes or DCOS. The hostname of the segment store generally isn't known upfront, making it difficult to provision an appropriate certificate. As a result, the user may be tempted to disable hostname verification, which is an insecure practice.

A good alternative is for the client to trust only the specific certificates that are configured on the segment store, as opposed to relying on a chain-of-trust based on public root certificates. By constraining the truststore to the specific certificates of the segment store, one need not use hostname verification, and may use a self-signed certificate.

One could generate a truststore for the client that contained only the segment store certificates. But the set of segment stores is dynamic, and so the client's truststore should likewise be dynamic.

User confusion around hostname verification is wide considered to be a major security issue in non-browser web software. See "The Most Dangerous Code in the World".

Suggestions for an improvement
A relatively simple and effective solution is for the controller to serve the list of public keys that the associated segment stores are using. This information could be served piecemeal (as metadata associated with a segment server), or as a certificate bundle (containing all certificates that the controller considers to be needed). The client should obtain that bundle at runtime, and use it as the truststore when connecting to a segment store. The client needn't use hostname verification in such a situation.

[arvindkandhare]

A relatively simple and effective solution is for the controller to serve the list of public keys that the associated segment stores are using. This information could be served piecemeal (as metadata associated with a segment server), or as a certificate bundle (containing all certificates that the controller considers to be needed). The client should obtain that bundle at runtime, and use it as the truststore when connecting to a segment store. The client needn't use hostname verification in such a situation.

This looks like an elegant and workable solution.

But this scenario (setting up TLS with hostname verification enabled while the hosts are dynamically created) looks like a deployment technology issue. How does DC/OS or kubernetes solve these?

I am wary of allowing this to seep into Pravega. This does not look like Pravega's job to supply and mange certs. Can we not handle this in the deployment layer itself and keep Pravega out of this?

[EronWright]

Note: the Cert Manager product we intend to use in the platform is https://github.com/jetstack/cert-manager

[EronWright]

[EronWright]

Closing in favor of #3652