:x-pack:qa:rolling-upgrade:v8.1.3#oneThirdUpgradedTest fails on Windows builds

[Luegg]

CI Link

https://gradle-enterprise.elastic.co/s/hemwfnwir4mau/console-log?task=:x-pack:qa:rolling-upgrade:v8.1.3%23oneThirdUpgradedTest

Repro line

none

Does it reproduce?

Didn't try

Applicable branches

main, 8.2

Failure history

https://gradle-enterprise.elastic.co/scans/failures?failures.failureClassification=all_failures&failures.failureMessage=Execution%20failed%20for%20task%20%27:x-pack:qa:rolling-upgrade:v8.1.3%23oneThirdUpgradedTest%27*&search.relativeStartTime=P7D&search.timeZoneId=Europe/Zurich

Failure excerpt

completing election, {v8.1.3-2}{qBL8YWniQ4KftH0okOkxDw}{CyN3xYh-Se2kihAfXpVBtA}{127.0.0.1}{127.0.0.1:58563}{cdfhilmrstw} completing election, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [1]
»  org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
»  	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1718) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1625) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:78) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.Coordinator.cancelActivePublication(Coordinator.java:1380) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.Coordinator.becomeCandidate(Coordinator.java:693) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:534) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$3(JoinHelper.java:152) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:271) ~[?:?]
»  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:333) ~[?:?]
»  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$3.onResponse(SecurityServerTransportInterceptor.java:323) ~[?:?]
»  	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:610) ~[?:?]
»  	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:251) ~[?:?]
»  	at org.elasticsearch.xpack.security.transport.ServerTransportFilter.lambda$inbound$1(ServerTransportFilter.java:113) ~[?:?]
»  	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:99) ~[?:?]
»  	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:199) ~[?:?]
»  	at org.elasticsearch.xpack.security.transport.ServerTransportFilter.inbound(ServerTransportFilter.java:105) ~[?:?]
»  	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:344) ~[?:?]
»  	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:287) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
»  	at java.lang.Thread.run(Thread.java:833) [?:?]
»  Caused by: org.elasticsearch.ElasticsearchException: publication cancelled before committing: become candidate: joinLeaderInTerm
»  	at org.elasticsearch.cluster.coordination.Publication.cancel(Publication.java:75) ~[elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	... 24 more
» [2022-04-20T10:32:45,286][ERROR][o.e.i.g.GeoIpDownloader  ] [v8.1.3-0] exception during geoip databases update
»  java.net.UnknownHostException: invalid.endpoint
»  	at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:564) ~[?:?]
»  	at java.net.Socket.connect(Socket.java:633) ~[?:?]
»  	at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
»  	at sun.net.www.http.HttpClient.openServer(HttpClient.java:498) ~[?:?]
»  	at sun.net.www.http.HttpClient.openServer(HttpClient.java:603) ~[?:?]
»  	at sun.net.www.http.HttpClient.<init>(HttpClient.java:246) ~[?:?]
»  	at sun.net.www.http.HttpClient.New(HttpClient.java:351) ~[?:?]
»  	at sun.net.www.http.HttpClient.New(HttpClient.java:373) ~[?:?]
»  	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
»  	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]
»  	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
»  	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
»  	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) ~[?:?]
»  	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) ~[?:?]
»  	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
»  	at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
»  	at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:143) ~[ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:131) ~[ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:265) [ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776) [elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.1.3-SNAPSHOT.jar:8.1.3-SNAPSHOT]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
»  	at java.lang.Thread.run(Thread.java:833) [?:?]
» [2022-04-20T10:34:32,569][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v8.1.3-0] fatal error in thread [main], exiting
»  java.lang.AssertionError: expected file size 16777216 but was 0
»  	at org.elasticsearch.xpack.searchablesnapshots.cache.shared.SharedBytes.<init>(SharedBytes.java:64) ~[?:?]
»  	at org.elasticsearch.xpack.searchablesnapshots.cache.shared.FrozenCacheService.<init>(FrozenCacheService.java:294) ~[?:?]
»  	at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:339) ~[?:?]
»  	at org.elasticsearch.node.Node.lambda$new$16(Node.java:676) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
»  	at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
»  	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
»  	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
»  	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
»  	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
»  	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
»  	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
»  	at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
»  	at org.elasticsearch.node.Node.<init>(Node.java:690) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.node.Node.<init>(Node.java:292) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:165) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:156) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:53) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-cli-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.cli.Command.main(Command.java:87) ~[elasticsearch-cli-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-8.3.0-SNAPSHOT.jar:8.3.0-SNAPSHOT]
»  fatal error in thread [main], exiting
»  java.lang.AssertionError: expected file size 16777216 but was 0
»  	at org.elasticsearch.xpack.searchablesnapshots.cache.shared.SharedBytes.<init>(SharedBytes.java:64)
»  	at org.elasticsearch.xpack.searchablesnapshots.cache.shared.FrozenCacheService.<init>(FrozenCacheService.java:294)
»  	at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:339)
»  	at org.elasticsearch.node.Node.lambda$new$16(Node.java:676)
»  	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
»  	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
»  	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
»  	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
»  	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
»  	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
»  	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
»  	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
»  	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
»  	at org.elasticsearch.node.Node.<init>(Node.java:690)
»  	at org.elasticsearch.node.Node.<init>(Node.java:292)
»  	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
»  	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
»  	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358)
»  	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:165)
»  	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:156)
»  	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:53)
»  	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
»  	at org.elasticsearch.cli.Command.main(Command.java:87)
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
»  	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
»  ERROR: Elasticsearch did not exit normally - check the logs at C:\Users\jenkins\workspace\platform-support\60\x-pack\qa\rolling-upgrade\build\testclusters\v8.1.3-0\logs\v8.1.3.log
»   ↓ last 40 non error or warning messages from C:\Users\jenkins\workspace\platform-support\60\x-pack\qa\rolling-upgrade\build\testclusters\v8.1.3-0\logs\es.out ↓

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

This looks to be a duplicate of #85725