Fix inefficient (worst case exponential) loading of snapshot repository #24510

joachimdraeger · 2017-05-05T13:48:28Z

Ensure that getRepositoryData() is only called once during a list snapshots operation. Fixes #24509.

…ry data when checking for incompatible snapshots.

elasticmachine · 2017-05-05T13:48:30Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

joachimdraeger · 2017-05-05T13:52:29Z

core/src/main/java/org/elasticsearch/repositories/Repository.java

+     * @param snapshotId  snapshot id
+     * @return information about snapshot
+     */
+    SnapshotInfo getSnapshotInfo(RepositoryData repositoryData, SnapshotId snapshotId);


I appreciate that requiring to pass RepositoryData on getSnapshotInfo might not be the cleanest solution regarding Repository's API.

abeyad · 2017-05-05T16:26:16Z

@joachimdraeger thank you for this contribution, and great catch! Indeed this would help solve the performance issues introduced by incompatible-snapshots. However, as you stated, I don't think we want to alter the Repository interface to take the RepositoryData. Rather, I would propose a different solution: in TransportGetSnapshotsAction, we have already retrieved the RepositoryData, so we can pass the incompatible snapshots list from that RepositoryData instance into SnapshotsService#snapshots. The SnapshotsServce#snapshots method can return a SnapshotInfo.incompatible instance, instead of relying on BlobStoreRepository#getSnapshotInfo to do so.

Let me know if you have the time to do this, or if you prefer I take it on. We would want to get this in for 5.4.1 and 5.5.0.

Thanks again!

Regarding your question of running gradle check, what exactly was failing?

abeyad

I left feedback on a different approach

joachimdraeger · 2017-05-05T17:14:40Z

@abeyad thanks for your feedback. I see that the incompatible SnapshotInfo is only used for the list. In get and restore incompatible is determined twice at the moment: directly and in getSnapshotInfo.

I think it would be nicer in general if the source of truth would be getSnapshotInfo with the INCOMPATIBLE state in all cases. However, that would require a much bigger refactoring.

For now I agree that removing the check from 'getSnapshotInfo' and letting 'SnapshotsService#snapshots' check for incompatible snapshots itself seems to be the most viable solution.

I'm happy to give it a go on Monday.

abeyad · 2017-05-05T17:18:30Z

Thank you @joachimdraeger, don't hesitate to ping me on Monday if I can help in any way.

…repository data" This reverts commit 6eb9837.

…ry data

joachimdraeger · 2017-05-08T13:23:40Z

@abeyad I re-implemented the fix as discussed. I tested the incompatible snapshots listing manually and the performance improvement.

abeyad

I left a couple small comments. Once fixed, we can get this merged.

abeyad · 2017-05-08T13:45:47Z

core/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

-            // an incompatible snapshot - cannot read its snapshot metadata file, just return
-            // a SnapshotInfo indicating its incompatible
-            return SnapshotInfo.incompatible(snapshotId);
-        }


Since getSnapshotInfoInternal (just below) is only used by getSnapshotInfo, we can move the code in getSnapshotInfoInternal directly into getSnapshotInfo and get rid of getSnaphotInfoInternal

abeyad · 2017-05-08T13:47:01Z

core/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

@@ -196,6 +206,7 @@ public SnapshotInfo snapshot(final String repositoryName, final SnapshotId snaps
        return Collections.unmodifiableList(snapshotList);
    }

+


remove extra new line

abeyad

LGTM, I will run the tests locally as well and merge. Thank you for working on this @joachimdraeger!

abeyad · 2017-05-08T17:32:15Z

@elasticmachine test this please

This commit fixes inefficient (worst case exponential) loading of snapshot repository data when checking for incompatible snapshots, that was introduced in #22267. When getting snapshot information, getRepositoryData() was called on every snapshot, so if there are a large number of snapshots in the repository and _all snapshots were requested, the performance degraded exponentially. This commit fixes the issue by only calling getRepositoryData once and using the data from it in all subsequent calls to get snapshot information. Closes #24509

abeyad · 2017-05-08T21:44:24Z

5.x commit: 690250e
5.4 commit: 21da532

abeyad · 2017-05-08T21:44:38Z

@joachimdraeger the PR has been merged - thanks again!

talevy · 2017-05-08T22:29:54Z

@abeyad are you sure this change was made properly into 5.x?

I am seeing that there is a rogue reference to the getSnapshotInfoInternal method:

https://github.com/elastic/elasticsearch/blob/5.x/core/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java#L369

seems to be the only call to it... but the method definition is deleted.

abeyad · 2017-05-08T22:31:43Z

@talevy I pushed ccee1b0 and 6a0e070 to fix the issue on 5.x

* master: Increase compilation limit in ingest tests Mark 6.0.0-alpha1 as prerelease Updated release notes for 6.0.0-alpha1 Fix single shard scroll within a cluster with nodes in version >= 5.3 and <= 5.3 (elastic#24512) add option for _ingest.timestamp to use new ZonedDateTime (elastic#24030) Fixes inefficient loading of snapshot repository data (elastic#24510) Scripting: Deprecate file scripts (elastic#24552) Remove commented code from ESILRTC Ensure test replicas have valid recovery state Add global checkpoint assertion in index shard Improve bootstrap checks error messages Refactor UpdateHelper into unit-testable pieces Fix cache expire after access Document work-around for jar hell in idea_rt.jar file (elastic#24523) Move MockLogAppender to elasticsearch test (elastic#24542) Remove gap skipping when opening engine documentation of preserve existing settings remove duplicated import in AppendProcessor

elasticmachine · 2017-05-15T09:57:08Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Fix inefficient (worst case exponential) loading of snapshot reposito…

6eb9837

…ry data when checking for incompatible snapshots.

joachimdraeger commented May 5, 2017

View reviewed changes

jasontedor requested a review from abeyad May 5, 2017 14:24

jasontedor assigned abeyad May 5, 2017

abeyad suggested changes May 5, 2017

View reviewed changes

abeyad mentioned this pull request May 5, 2017

ElasticSearch Snapshot API is unresponsive for big indexes #24507

Closed

joachimdraeger added 2 commits May 8, 2017 09:58

Revert "Fix inefficient (worst case exponential) loading of snapshot …

7e3a859

…repository data" This reverts commit 6eb9837.

Fix inefficient (worst case exponential) loading of snapshot reposito…

6658c13

…ry data

abeyad suggested changes May 8, 2017

View reviewed changes

merged getSnapshotInfo and getSnapshotInfoInternal

e8ff88c

abeyad approved these changes May 8, 2017

View reviewed changes

abeyad added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >bug v5.4.1 v5.5.0 v6.0.0 labels May 8, 2017

abeyad merged commit fec1802 into elastic:master May 8, 2017

abeyad pushed a commit that referenced this pull request May 8, 2017

Fixes compilation error from backporting #24510

ccee1b0

abeyad pushed a commit that referenced this pull request May 8, 2017

Fixes compilation error from backporting #24510

6a0e070

clintongormley changed the title ~~Fix inefficient (worst case exponential) loading of snapshot reposito…~~ Fix inefficient (worst case exponential) loading of snapshot repository May 15, 2017

clintongormley added v6.0.0-alpha2 and removed v6.0.0 labels Jun 6, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix inefficient (worst case exponential) loading of snapshot repository #24510

Fix inefficient (worst case exponential) loading of snapshot repository #24510

joachimdraeger commented May 5, 2017

elasticmachine commented May 5, 2017

joachimdraeger May 5, 2017

abeyad commented May 5, 2017

abeyad left a comment

joachimdraeger commented May 5, 2017

abeyad commented May 5, 2017

joachimdraeger commented May 8, 2017

abeyad left a comment

abeyad May 8, 2017

abeyad May 8, 2017

abeyad left a comment

abeyad commented May 8, 2017

abeyad commented May 8, 2017

abeyad commented May 8, 2017

talevy commented May 8, 2017

abeyad commented May 8, 2017

elasticmachine commented May 15, 2017

		@@ -196,6 +206,7 @@ public SnapshotInfo snapshot(final String repositoryName, final SnapshotId snaps
		return Collections.unmodifiableList(snapshotList);
		}

Fix inefficient (worst case exponential) loading of snapshot repository #24510

Fix inefficient (worst case exponential) loading of snapshot repository #24510

Conversation

joachimdraeger commented May 5, 2017

elasticmachine commented May 5, 2017

joachimdraeger May 5, 2017

Choose a reason for hiding this comment

abeyad commented May 5, 2017

abeyad left a comment

Choose a reason for hiding this comment

joachimdraeger commented May 5, 2017

abeyad commented May 5, 2017

joachimdraeger commented May 8, 2017

abeyad left a comment

Choose a reason for hiding this comment

abeyad May 8, 2017

Choose a reason for hiding this comment

abeyad May 8, 2017

Choose a reason for hiding this comment

abeyad left a comment

Choose a reason for hiding this comment

abeyad commented May 8, 2017

abeyad commented May 8, 2017

abeyad commented May 8, 2017

talevy commented May 8, 2017

abeyad commented May 8, 2017

elasticmachine commented May 15, 2017