[CI] 6.8 BWC tests against many 5.6.x versions failing #48114
Labels
:Delivery/Build
Build or test infrastructure
:Distributed Indexing/Engine
Anything around managing Lucene and the Translog in an open shard.
Team:Delivery
Meta label for Delivery team
>test-failure
Triaged test failures from CI
Comments
droberts195
added
:Delivery/Build
|
Pinging @elastic/es-core-infra (:Core/Infra/Build) |
|
Pinging @elastic/es-distributed (:Distributed/Engine) |
#47895 in 6.8 should be the source here. We do not always have sequence numbers for the refresh optimization in a mixed cluster between 6.8 and 5.6. I have reverted #47895 in 6.8 in 97e64fe. @droberts195 Thanks for the ping. |
dnhatn
added a commit
that referenced
this issue
Oct 16, 2019
We might not have sequence numbers in a mixed cluster between 6.8 and 5.6. In this case, we should refresh unconditionally; otherwise, we can apply the refresh optimization. An alternative is to use translog locations instead of the local checkpoint for this optimization. Closes #48114
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
These failures started occurring on 15th October. After that many 6.8 BWC tests against 5.6.x versions have failed. For example, look at the failure pattern in:
Not all versions are affected. For example 5.6.13 has been fine:
The errors vary. For example:
java.lang.AssertionError: IndexVersionValue{version=1, seqNo=-2, term=1, location=null}java.lang.AssertionError: IndexVersionValue{version=1, seqNo=-2, term=1, location=null}Infra has also noticed that some of the workers are running out of disk space due to huge console logs. For example https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+default-distro+bwc/BWC_VERSION=5.6.6,nodes=centos-7&&immutable/210/consoleText is 352MB.
Was something changed recently that increased the parallelism of 6.8 BWC tests? If so then that probably explains it. But it seems that the level of parallelism is beyond what the currently configured workers can cope with.
I'm tagging the distributed team in case the
java.lang.AssertionError: IndexVersionValue{version=1, seqNo=-2, term=1, location=null}errors are a worry, and also the core build team in case this is all down to trying to run too much in parallel.The text was updated successfully, but these errors were encountered: