Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SharedClusterSnapshotRestoreIT#testGetSnapshotsRequest test failure #49048

Closed
hub-cap opened this issue Nov 13, 2019 · 2 comments · Fixed by #48944
Closed

[CI] SharedClusterSnapshotRestoreIT#testGetSnapshotsRequest test failure #49048

hub-cap opened this issue Nov 13, 2019 · 2 comments · Fixed by #48944
Assignees
@original-brownbear
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI

Comments

@hub-cap
Copy link
Contributor

hub-cap commented Nov 13, 2019

Reproduce (7.x) with:

./gradlew ':server:integTest' --tests "org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT.testGetSnapshotsRequest" \
  -Dtests.seed=14BDD5A1147CE32 \
  -Dtests.security.manager=true \
  -Dtests.locale=zh-TW \
  -Dtests.timezone=Europe/Skopje \
  -Dcompiler.java=12 \
  -Druntime.java=11

Stacktrace:

RepositoryException[[test-repo] Could not determine repository generation from root blobs]; nested: NoSuchFileException[/dev/shm/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA/openjdk12/ES_RUNTIME_JAVA/adoptopenjdk11/nodes/general-purpose/server/build/testrun/integTest/temp/org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT_14BDD5A1147CE32-001/tempDir-002/repos/qWSrDrNepA/index-2];
	at __randomizedtesting.SeedInfo.seed([14BDD5A1147CE32:80CBD44E913E7FFF]:0)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:911)
	at org.elasticsearch.snapshots.SnapshotsService.getRepositoryData(SnapshotsService.java:169)
	at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:99)
	at org.elasticsearch.action.admin.cluster.snapshots.get.TransportGetSnapshotsAction.masterOperation(TransportGetSnapshotsAction.java:56)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:169)
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.file.NoSuchFileException: /dev/shm/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA/openjdk12/ES_RUNTIME_JAVA/adoptopenjdk11/nodes/general-purpose/server/build/testrun/integTest/temp/org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT_14BDD5A1147CE32-001/tempDir-002/repos/qWSrDrNepA/index-2
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:145)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.readAttributes(FilterFileSystemProvider.java:177)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.readAttributes(FilterFileSystemProvider.java:177)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.readAttributes(FilterFileSystemProvider.java:177)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.readAttributes(FilterFileSystemProvider.java:177)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
	at org.elasticsearch.common.blobstore.fs.FsBlobContainer.listBlobsByPrefix(FsBlobContainer.java:100)
	at org.elasticsearch.snapshots.mockstore.BlobContainerWrapper.listBlobsByPrefix(BlobContainerWrapper.java:85)
	at org.elasticsearch.snapshots.mockstore.MockRepository$MockBlobStore$MockBlobContainer.listBlobsByPrefix(MockRepository.java:353)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:1017)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:999)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:909)
	... 11 more
@hub-cap hub-cap added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Nov 13, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@original-brownbear original-brownbear self-assigned this Nov 13, 2019
@original-brownbear
Copy link
Member

This will be fixed by #48944

@original-brownbear original-brownbear mentioned this issue Nov 13, 2019
Merged
original-brownbear added a commit that referenced this issue Nov 14, 2019
@original-brownbear
Track Repository Gen. in BlobStoreRepository (#48944)
37c58ca 
This is intended as a stop-gap solution/improvement to #38941 that
prevents repo modifications without an intermittent master failover
from causing inconsistent (outdated due to inconsistent listing of index-N blobs)
`RepositoryData` to be written.

Tracking the latest repository generation will move to the cluster state in a
separate pull request. This is intended as a low-risk change to be backported as
far as possible and motived by the recently increased chance of #38941 
causing trouble via SLM (see #47520).

Closes #47834
Closes #49048
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 14, 2019
@original-brownbear
Track Repository Gen. in BlobStoreRepository (elastic#48944)
edb5046 
This is intended as a stop-gap solution/improvement to elastic#38941 that
prevents repo modifications without an intermittent master failover
from causing inconsistent (outdated due to inconsistent listing of index-N blobs)
`RepositoryData` to be written.

Tracking the latest repository generation will move to the cluster state in a
separate pull request. This is intended as a low-risk change to be backported as
far as possible and motived by the recently increased chance of elastic#38941
causing trouble via SLM (see elastic#47520).

Closes elastic#47834
Closes elastic#49048
@original-brownbear original-brownbear mentioned this issue Nov 14, 2019
Merged
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Nov 14, 2019
@original-brownbear
Track Repository Gen. in BlobStoreRepository (elastic#48944)
870f8d4 
This is intended as a stop-gap solution/improvement to elastic#38941 that
prevents repo modifications without an intermittent master failover
from causing inconsistent (outdated due to inconsistent listing of index-N blobs)
`RepositoryData` to be written.

Tracking the latest repository generation will move to the cluster state in a
separate pull request. This is intended as a low-risk change to be backported as
far as possible and motived by the recently increased chance of elastic#38941
causing trouble via SLM (see elastic#47520).

Closes elastic#47834
Closes elastic#49048
@original-brownbear original-brownbear mentioned this issue Nov 14, 2019
Merged
original-brownbear added a commit that referenced this issue Nov 15, 2019
@original-brownbear
Track Repository Gen. in BlobStoreRepository (#48944) (#49119)
9ae57de 
This is intended as a stop-gap solution/improvement to #38941 that
prevents repo modifications without an intermittent master failover
from causing inconsistent (outdated due to inconsistent listing of index-N blobs)
`RepositoryData` to be written.

Tracking the latest repository generation will move to the cluster state in a
separate pull request. This is intended as a low-risk change to be backported as
far as possible and motived by the recently increased chance of #38941
causing trouble via SLM (see #47520).

Closes #47834
Closes #49048
original-brownbear added a commit that referenced this issue Nov 15, 2019
@original-brownbear
Track Repository Gen. in BlobStoreRepository (#48944) (#49116)
fc505aa 
This is intended as a stop-gap solution/improvement to #38941 that
prevents repo modifications without an intermittent master failover
from causing inconsistent (outdated due to inconsistent listing of index-N blobs)
`RepositoryData` to be written.

Tracking the latest repository generation will move to the cluster state in a
separate pull request. This is intended as a low-risk change to be backported as
far as possible and motived by the recently increased chance of #38941
causing trouble via SLM (see #47520).

Closes #47834
Closes #49048
This was referenced Feb 3, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Track Repository Gen. in BlobStoreRepository original-brownbear/elasticsearch
3 participants
@hub-cap @original-brownbear @elasticmachine