[CI] :modules:reindex task can't run on FIPS JVMs

[jkakavas]

it looks like either #40564 or #40804 made it so some system properties (specifically javax.net.ssl.trustStorePassword and javax.net.ssl.keyStorePassword ) do not find their way to :modules:reindex:integTestCluster when running with a FIPS JVM. This causes the task to fail as the BCFKS keystore cannot be decrypted.

Example failures:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=immutable&&linux&&docker/116/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=immutable&&linux&&docker/340/console

Manifests as

Caused by: java.security.KeyStoreException: problem accessing trust storejava.io.IOException: BCFKS KeyStore corrupted: MAC calculation failed.
  	at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:74) ~[?:?]
  	at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_202]
  	at org.elasticsearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:151) ~[?:?]
  	at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:68) ~[?:?]
  	at org.elasticsearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:136) ~[?:?]
  	at org.elasticsearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:145) ~[?:?]
  	at org.elasticsearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:115) ~[?:?]
  	at org.elasticsearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:88) ~[?:?]
  	at org.elasticsearch.node.Node.lambda$new$9(Node.java:440) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_202]
  	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_202]
  	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_202]
  	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_202]
  	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_202]
  	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_202]
  	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_202]
  	at org.elasticsearch.node.Node.<init>(Node.java:443) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.node.Node.<init>(Node.java:253) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  	... 6 more

[elasticmachine]

Pinging @elastic/es-core-infra

[alpar-t]

@jkakavas is this something you plan to address yourself? Maybe the security label would be more appropriate than build here ?

[jkakavas]

@jkakavas is this something you plan to address yourself?

@atorok It wasn't in my to-do list but I have nothing against picking it up. It feels the build label was appropriate since this was running fine before #40564 or #40804 which makes me think something in there changed how system properties are passed ? I could be wrong though, I'll know more once I look into it.

I would appreciate it though if you have any quick ideas on what in these aforementioned PRs have changed the way system properties are picked up

[alpar-t]

Thanks for pointing that out it wasn't obvious to me this is a regression.
I probably missed something fips specific in test-clusters I will take a look