Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MixedClusterClientYamlTestSuiteIT failing (crash due to assertion) #76552

Closed
henningandersen opened this issue Aug 16, 2021 · 2 comments
Closed

[CI] MixedClusterClientYamlTestSuiteIT failing (crash due to assertion) #76552

henningandersen opened this issue Aug 16, 2021 · 2 comments
Assignees
@original-brownbear
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@henningandersen
Copy link
Contributor

Has happened on two different PRs and locally on unmodified master

Build scan:
https://gradle-enterprise.elastic.co/s/3wwzm7vghnze4
https://gradle-enterprise.elastic.co/s/3jiec6et2wbbe

Repro line:

./gradlew ':qa:mixed-cluster:v7.15.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=snapshot.get/10_basic/Get snapshot info when verbose is false}" -Dtests.seed=489DEE34D256CCBA -Dtests.bwc=true -Dtests.locale=sl-SI -Dtests.timezone=America/Cayman -Druntime.java=11

Reproduces locally?:

Yes, using following against master:

./gradlew ':qa:mixed-cluster:check' -Dtests.seed=A5080EDD8FCF06C8 -Dtests.bwc=true -Dtests.locale=es-HN -Dtests.timezone=Etc/GMT+9 -Druntime.java=11

Applicable branches:

master (at least).

Failure history:

build-stats

started failing this morning. Likely the backport of #75501 in #76539.

Failure excerpt:

[2021-08-16T07:44:25,603][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.15.0-0] fatal error in thread [elasticsearch[v7.15.0-0][generic][T#4]], exiting
java.lang.AssertionError: Provided explict repo shard id statuses [[RepositoryShardId{[test_index/84YPuuceTUKHo8NGh3X_Xw]}{0}=>ShardSnapshotStatus[state=INIT, nodeId=qccWoxkyR7-hl0j9W2SRlA, reason=null, generation=_new, shardSnapshotResult=null]]] but no source
	at org.elasticsearch.cluster.SnapshotsInProgress$Entry.<init>(SnapshotsInProgress.java:559) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.SnapshotsInProgress$Entry.readFrom(SnapshotsInProgress.java:610) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.SnapshotsInProgress.lambda$new$0(SnapshotsInProgress.java:67) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1253) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.common.io.stream.StreamInput.readList(StreamInput.java:1203) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.cluster.SnapshotsInProgress.<init>(SnapshotsInProgress.java:67) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

Did not mute any tests, since this hits too widely.
@henningandersen henningandersen added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Aug 16, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Aug 16, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

This was referenced Aug 16, 2021
Merged
Merged
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Aug 16, 2021
@original-brownbear
Fix Broken Serialization of SnapshotsInProgress in 7.x
1493834 
We must not send the by-repo-shard-id map for normal
snapshots. The conditional got lost in `7.x` and `7.14`
because of a merge error. This aligns master and 7.x behavior.

closes elastic#76552
@original-brownbear original-brownbear mentioned this issue Aug 16, 2021
Merged
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Aug 16, 2021
@original-brownbear
Fix Broken Serialization of SnapshotsInProgress in 7.x
a820b48 
We must not send the by-repo-shard-id map for normal
snapshots. The conditional got lost in `7.x` and `7.14`
because of a merge error. This aligns master and 7.x behavior.

closes elastic#76552
@original-brownbear original-brownbear mentioned this issue Aug 16, 2021
Merged
original-brownbear added a commit that referenced this issue Aug 16, 2021
@original-brownbear
Fix Broken Serialization of SnapshotsInProgress in 7.x (#76557)
44bd97e 
We must not send the by-repo-shard-id map for normal
snapshots. The conditional got lost in `7.x` and `7.14`
because of a merge error. This aligns master and 7.x behavior.

closes #76552
original-brownbear added a commit that referenced this issue Aug 16, 2021
@original-brownbear
Fix Broken Serialization of SnapshotsInProgress in 7.x (#76553)
5ef2a8a 
We must not send the by-repo-shard-id map for normal
snapshots. The conditional got lost in `7.x` and `7.14`
because of a merge error. This aligns master and 7.x behavior.

closes #76552
@original-brownbear
Copy link
Member

Fixed via #76553

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@original-brownbear @elasticmachine @henningandersen