[CI] IndexAuditTrail fails to start - cannot read security-version

[tvernum]

On 6.6:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+periodic/80/console

Seems to have failed twice (previous but the earlier failure may have been related to the underlying hardware.

Doesn't reproduce for me.
As a CI failure it seems pretty rare, but I wonder if there's some sort of timing/concurrency issue that could be an actual production problem.

Log:

00:03:38 |    [2019-01-02T00:01:44,802][ERROR][o.e.x.s.a.i.IndexAuditTrail] [upgraded-node-2] failed to start index audit trail services
00:03:38 |    java.lang.IllegalStateException: Cannot read security-version string in index .security_audit_log-2019.01.02
00:03:38 |    	at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.updateCurrentIndexMappingsIfNecessary(IndexAuditTrail.java:404) ~[x-pack-security-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.start(IndexAuditTrail.java:373) ~[x-pack-security-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail$2.doRun(IndexAuditTrail.java:235) [x-pack-security-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
00:03:38 |    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
00:03:38 |    	at java.lang.Thread.run(Thread.java:834) [?:?]
00:03:38 |    [2019-01-02T00:01:44,814][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-2] fatal error in thread [elasticsearch[upgraded-node-2][generic][T#1]], exiting
00:03:38 |    java.lang.AssertionError: security lifecycle services startup failed
00:03:38 |    	at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail$2.onFailure(IndexAuditTrail.java:230) ~[?:?]
00:03:38 |    	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:744) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 |    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
00:03:38 |    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
00:03:38 |    	at java.lang.Thread.run(Thread.java:834) [?:?]
00:03:38 |-----------------------------------------

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

@albertzaharovits can you take a look at this failure?

[romseygeek]

This happened again in a rolling upgrade test:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+release-tests/128/console

[jaymode]

@albertzaharovits merged the fix in #38286 but github did not close the issue since it was not targeted for master.