Monitoring Exporter attempts to read closed SecureSettings

[mellieA]

Using 6.2.3 and secure monitoring settings below causes bootstrap errors below:

java.lang.IllegalStateException: password has been cleared
        at java.security.KeyStore$PasswordProtection.getPassword(KeyStore.java:347) ~[?:1.8.0_161]
        at sun.security.pkcs12.PKCS12KeyStore.engineGetEntry(PKCS12KeyStore.java:1304) ~[?:?]
        at java.security.KeyStore.getEntry(KeyStore.java:1521) ~[?:1.8.0_161]
        at org.elasticsearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:351) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.settings.Settings$PrefixedSecureSettings.getString(Settings.java:1450) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:153) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:143) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.settings.SecureSetting.get(SecureSetting.java:94) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.xpack.core.ssl.CertUtils.createKeyConfig(CertUtils.java:216) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLConfiguration.createKeyConfig(SSLConfiguration.java:199) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLConfiguration.<init>(SSLConfiguration.java:82) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.sslConfiguration(SSLService.java:345) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.sslIOSessionStrategy(SSLService.java:141) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.configureSecurity(HttpExporter.java:461) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.createRestClient(HttpExporter.java:296) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.<init>(HttpExporter.java:229) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.http.HttpExporter.<init>(HttpExporter.java:216) ~[?:?]
        at org.elasticsearch.xpack.monitoring.Monitoring.lambda$createComponents$1(Monitoring.java:148) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.initExporters(Exporters.java:162) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.Exporters.doStart(Exporters.java:85) ~[?:?]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:66) ~[elasticsearch-6.2.3.jar:6.2.3]
        at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_161]
        at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) ~[?:1.8.0_161]
        at org.elasticsearch.node.Node.start(Node.java:598) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:262) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:332) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.3.jar:6.2.3]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.3.jar:6.2.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.3.jar:6.2.3]

Keystore settings:

sudo /usr/share/elasticsearch/bin/elasticsearch-keystore list 
keystore.seed 
xpack.monitoring.exporters.ppe-mon.ssl.keystore.secure_password 
xpack.security.http.ssl.keystore.secure_password 
xpack.security.transport.ssl.keystore.secure_password

Is there any workaround?

Related to: https://github.com/elastic/x-pack-elasticsearch/issues/3950

[mellieA]

@tvernum I moved it here :)

[elasticmachine]

Pinging @elastic/es-core-infra

[elasticmachine]

Pinging @elastic/es-security

[mellieA]

@tvernum I realize this is a similar but issue than #3950, any idea if there is any sort of workaround for a customer waiting to go into Prod for this?

[tvernum]

The only workaround that I'm aware of is to not use secure settings with monitoring.
For now, the key(store) passwords would need to be kept in elasticsearch.yml