Add support for per-repository truststore

[DaveCTurner]

If we communicate with a repository over TLS then today we use the default (JVM-wide) truststore to validate the certificate chain that the repository presents to prove its identity. In most of the other contexts that use TLS we typically let users specify which truststore to use in order to support very fine-grained control of the certificates that are trusted in each context. I think we should permit users to configure a per-repository (or maybe per-repo-client) trust store to give the same level of control over traffic to snapshot repositories

This is particularly thinking of S3-compatible repositories, I'm not sure we need this for most of the other implementations right now.

Relates https://discuss.elastic.co/t/configure-certificate-for-repository-s3-with-ceph/282645

---

Workaround: Note that the JVM-wide truststore is under the user's control so for now they can work around this missing feature by adding a certificate for their own custom CAs there.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[tvernum]

@DaveCTurner, do you have an idea of how this should look in configuration?

Would it simply be a matter of adding support for s3.client.{client-name}.ssl.* settings in elasticsearch.yml / elasticsearch.keystore (as appropriate for the setting type)?

[DaveCTurner]

Yes that's what I'd envisioned. The various SDKs typically don't let you get access to the low-level details of the connection so we can only really customise things to the extend that the API permits it. For instance the S3 SDK lets us pass in a FileStoreTlsKeyManagersProvider that I believe does what we want, except it looks like it only supports single-file stores (i.e. it won't support a trust store that comprises a directory of PEM files).

[tvernum]

It's been a couple of years since I looked, but last time I investigated this, it looked like we might be able to add something like this to S3Service.buildConfiguration

SSLConnectionSocketFactory socketFactory = new SSLConnectionSocketFactory(sslContext, hostnameVerifier)
clientConfiguration.getApacheHttpClientConfig().setSslSocketFactory(socketFactory);

[ebuildy]

What about a global setting to configure all TLS CA on 1 shot? (inter node security, any web hook, snapshot etc...)

[tvernum]

What about a global setting to configure all TLS CA on 1 shot? (inter node security, any web hook, snapshot etc...)

We used to have that (though not for snapshots). We removed it because it caused more problems than it solved.

It is rarely the case that you actually want one CA for all of those different contexts. The CA you trust for internode security should be different than the CA you trust for webhooks. Attempting to configure them together is almost always a mistake.

We can continue this conversation on the forums: https://discuss.elastic.co/t/set-a-global-tls-ca-for-security-snapshot-reindex-etc/297389

[willem-dhaese]

Any news on this? We need to add our ca certs to /usr/share/elasticsearch/jdk/lib/security/cacerts after every upgrade... We were told by support that this issue would solve this.