[CI] IndexingIT testIndexVersionPropagation failing #72735
Assignees
Labels
:Distributed Indexing/Distributed
A catch all label for anything in the Distributed Area. Please avoid if you can.
Team:Distributed (Obsolete)
Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
>test-failure
Triaged test failures from CI
Comments
iverase
added
:Distributed Indexing/Distributed
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
The failure to reach a green state was because the node that should have held the replica actually hit an |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
May 5, 2021
The assertion failure message in elastic#72735 did not indicate why it failed because `Translog.Delete#toString()` does not expose all the fields of the problematic operation. This commit adds the missing `id` field.
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
May 5, 2021
The assertion failure message in elastic#72735 did not indicate why it failed because `Translog.Delete#toString()` does not expose all the fields of the problematic operation. This commit adds the missing `id` and `type` fields. Backport of elastic#72752 to 7.x
|
The exception message is unhelpful: However the |
DaveCTurner
added a commit
that referenced
this issue
May 5, 2021
The assertion failure message in #72735 did not indicate why it failed because `Translog.Delete#toString()` does not expose all the fields of the problematic operation. This commit adds the missing `id` field.
|
@romseygeek on the assumption that it's the |
|
Oops, yes, that's a bad backport. I'll open a PR. |
|
Hm, except that the bad operation here is a Delete, and delete tombstones have a type. So maybe the type isn't getting passed around correctly? |
|
Here's two more example failures: |
|
Thanks Julie, the last one has the extra detail in the message which confirms that it's just the type that's wrong:
|
|
Failed two more times today: |
|
I think this is actually caused by changes in #72400, specifically the changes to LuceneChangesSnapshot, but I'm not sure how the type information is getting lost yet. |
|
Looking more closely at the timing, I think this is #72616, specifically I think we have a bug in |
romseygeek
added a commit
that referenced
this issue
May 6, 2021
#62616 refactored how we deal with building mapping updates for delete operations against an empty index. In 7x, delete operations include a type as part of the operation UID, and this type needs to be resolved against mappings. In particular, the type _doc can mean either a type called _doc (the general case), or it can mean whatever type this index actually has. There are two code paths in IndexShard.applyDeleteOperation, one for empty mappings and one for the standard case; #62616 was resolving the type for the first code path, but not the second, which meant that some delete operations could be written to the translog with an incorrect type. This commit adds type resolution to the second code path as well. Fixes #72735
|
Closed by #72809. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No sure what is going on here.
Build scan:
https://gradle-enterprise.elastic.co/s/q6hptfnh42dw4/tests/:qa:mixed-cluster:v7.13.0%23mixedClusterTest/org.elasticsearch.backwards.IndexingIT/testIndexVersionPropagation
Reproduction line:
./gradlew ':qa:mixed-cluster:v7.13.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.IndexingIT" -Dtests.method="testIndexVersionPropagation" -Dtests.seed=4C1C60629320C40 -Dtests.bwc=true -Dtests.locale=hr-HR -Dtests.timezone=Asia/Damascus -Druntime.java=11Applicable branches:
7.x
Reproduces locally?:
No
Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.IndexingIT&tests.test=testIndexVersionPropagation
Failure excerpt:
The text was updated successfully, but these errors were encountered: