[CI] :qa:full-cluster-restart:v7.0.1#upgradedClusterTest failing

[pxsalehi]

CI Link

https://gradle-enterprise.elastic.co/s/ptwgurvvtzx5s

Repro line

Probably with ./gradlew :qa:full-cluster-restart:v7.0.1#upgradedClusterTest

Does it reproduce?

Didn't try

Applicable branches

main

Failure history

No response

Failure excerpt

See https://gradle-enterprise.elastic.co/s/ptwgurvvtzx5s/console-log/raw?task=:qa:full-cluster-restart:v7.0.1%23upgradedClusterTest

It also fails for other BWC versions: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+main+periodic+bwc/
(7.0.0 to 7.1.1)

» [2022-11-09T13:11:28,587][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.0.1-1] fatal error in thread [elasticsearch[v7.0.1-1][masterService#updateTask][T#1]], exiting java.lang.AssertionError: {[testclosedindices][0]=2}
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceReconciler.allocateUnassignedInvariant(DesiredBalanceReconciler.java:118)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceReconciler.run(DesiredBalanceReconciler.java:81)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator.recordTime(DesiredBalanceShardsAllocator.java:299)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator.reconcile(DesiredBalanceShardsAllocator.java:213)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator$ReconcileDesiredBalanceExecutor.lambda$applyBalance$1(DesiredBalanceShardsAllocator.java:277)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:518)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.AllocationService.executeWithRoutingAllocation(AllocationService.java:444)
»  	at [email protected]/org.elasticsearch.cluster.ClusterModule.reconcile(ClusterModule.java:145)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator$ReconcileDesiredBalanceExecutor.applyBalance(DesiredBalanceShardsAllocator.java:275)
»  	at [email protected]/org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator$ReconcileDesiredBalanceExecutor.execute(DesiredBalanceShardsAllocator.java:261)
»  	at [email protected]/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1052)
»  	at [email protected]/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1017)
»  	at [email protected]/org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:278)
»  	at [email protected]/org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170)
»  	at [email protected]/org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110)
»  	at [email protected]/org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148)
»  	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:850)
»  	at [email protected]/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:257)
»  	at [email protected]/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223)
»  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
»  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
»  	at java.base/java.lang.Thread.run(Thread.java:833)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[mark-vieira]

@DaveCTurner Given when this started failing and the stacktraces could #91343 be the culprit here? Looks like this is limited to upgrading from nodes earlier than 7.2.0.

[DaveCTurner]

Yeah, well at least #91343 introduced the assertion that's tripping here. Not yet clear why it would be tripping tho, but closed indices from pre-7.2 nodes are a special and interesting corner case (#33888). If I don't see an obvious problem soon I'll try and work out a way to mute these tests.