Fix Issue with Concurrent Snapshot Init + Delete #38518

original-brownbear · 2019-02-06T14:36:36Z

Ensures we're not finalizing a snapshot in the repository while it is initializing on another thread
Closes [CI] SharedClusterSnapshotRestoreIT.testAbortedSnapshotDuringInitDoesNotStart Fails #38489

elasticmachine · 2019-02-06T14:36:38Z

Pinging @elastic/es-distributed

original-brownbear · 2019-02-06T14:38:52Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

@@ -701,8 +714,8 @@ public void applyClusterState(ClusterChangedEvent event) {
                    // 3. Snapshots in any other state that have all their shard tasks completed
                    snapshotsInProgress.entries().stream().filter(
                        entry -> entry.state().completed()
-                            || entry.state() == State.INIT && initializingSnapshots.contains(entry.snapshot()) == false
-                            || entry.state() != State.INIT && completed(entry.shards().values())
+                            || initializingSnapshots.contains(entry.snapshot()) == false


Before we were finalising all snapshots that had all their shards completed here.
This caught the case where a snapshot had just went from INIT to ABORTED and was finalised in the repository when at the same time we were initialising it in https://github.com/elastic/elasticsearch/pull/38518/files#diff-a0853be4492c052f24917b5c1464003dR413 (this is what the failing test in #38489 forced by setting the on init block in the MockRepository).

With this change we will never finalise a snapshot here that is still being initialised.

original-brownbear · 2019-02-06T14:40:22Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

@@ -433,6 +434,8 @@ public ClusterState execute(ClusterState currentState) {

                            if (entry.state() == State.ABORTED) {
                                entries.add(entry);
+                                assert entry.shards().isEmpty();
+                                hadAbortedInitializations = true;


Since we now don't finalise this case in applyClusterState we need to end it when the cluster state for the init has been processed, so we set the flag here to mark that case.

original-brownbear · 2019-02-06T14:44:11Z

Jenkins run elasticsearch-ci/1

original-brownbear · 2019-02-06T15:54:32Z

Jenkins test this

original-brownbear · 2019-02-06T16:16:21Z

Jenkins test this

original-brownbear · 2019-02-06T17:36:18Z

Jenkins run elasticsearch-ci/1
Jenkins run elasticsearch-ci/2
Jenkins run elasticsearch-ci/default-distro

ywelsch · 2019-02-07T07:56:59Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

+
+                        if (hadAbortedInitializations) {
+                            final SnapshotsInProgress snapshotsInProgress = newState.custom(SnapshotsInProgress.TYPE);
+                            if (snapshotsInProgress != null) {


we should be able to assert != null here?

ywelsch · 2019-02-07T07:57:04Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

+                            final SnapshotsInProgress snapshotsInProgress = newState.custom(SnapshotsInProgress.TYPE);
+                            if (snapshotsInProgress != null) {
+                                final SnapshotsInProgress.Entry entry = snapshotsInProgress.snapshot(snapshot.snapshot());
+                                if (entry != null) {


we should be able to assert != null here?

I guess I was paranoid about potentially throwing an NPE in the cluster state thread :D but yea, assert should in theory be just fine => moving to assert :)

original-brownbear · 2019-02-07T20:06:04Z

Jenkins run elasticsearch-ci/1
Jenkins run elasticsearch-ci/2
Jenkins run elasticsearch-ci/default-distro

original-brownbear · 2019-02-07T21:33:59Z

Jenkins run elasticsearch-ci/1
Jenkins run elasticsearch-ci/2
Jenkins run elasticsearch-ci/default-distro

cbuescher · 2019-02-08T12:18:42Z

@original-brownbear I guess once this is fixed on master it is going to be backported to at least 7.0? Just asking because we got failures on that branch there today.

original-brownbear · 2019-02-08T12:40:14Z

@cbuescher yea should be possible to backport this to any affected branch and I'll do that. Added the labels :)

* master: Mute AnalysisModuleTests#testStandardFilterBWC (elastic#38636) add geotile_grid ref to asciidoc (elastic#38632) Enable Dockerfile from artifacts.elastic.co (elastic#38552) Mute FollowerFailOverIT testFailOverOnFollower (elastic#38634) Account for a possible rolled over file while reading the audit log file (elastic#34909) Mute failure in InternalEngineTests (elastic#38622) Fix Issue with Concurrent Snapshot Init + Delete (elastic#38518) Refactor ZonedDateTime.now in millis resolution (elastic#38577) Mute failing WatchStatusIntegrationTests (elastic#38621) Mute failing ApiKeyIntegTests (elastic#38614) [DOCS] Add warning about bypassing ML PUT APIs (elastic#38509) Add 7.1 and 8.0 version constants to master (elastic#38514)

* master: (1159 commits) Fix timezone fallback in ingest processor (elastic#38407) Avoid polluting download stats on builds (elastic#38660) SQL: Prevent grouping over grouping functions (elastic#38649) SQL: Relax StackOverflow circuit breaker for constants (elastic#38572) [DOCS] Fixes broken migration links (elastic#38655) Drop support for the low-level REST client on JDK 7 (elastic#38540) [DOCS] Adds placeholders for v8 highlights, breaking changes, release notes (elastic#38641) fix dissect doc "ip" --> "clientip" (elastic#38545) Concurrent file chunk fetching for CCR restore (elastic#38495) make DateMathIndexExpressionsIntegrationIT more resilient (elastic#38473) SQL: Replace joda with java time (elastic#38437) Add fuzziness example (elastic#37194) (elastic#38648) Mute AnalysisModuleTests#testStandardFilterBWC (elastic#38636) add geotile_grid ref to asciidoc (elastic#38632) Enable Dockerfile from artifacts.elastic.co (elastic#38552) Mute FollowerFailOverIT testFailOverOnFollower (elastic#38634) Account for a possible rolled over file while reading the audit log file (elastic#34909) Mute failure in InternalEngineTests (elastic#38622) Fix Issue with Concurrent Snapshot Init + Delete (elastic#38518) Refactor ZonedDateTime.now in millis resolution (elastic#38577) ...

* master: (27 commits) Mute AnalysisModuleTests#testStandardFilterBWC (elastic#38636) add geotile_grid ref to asciidoc (elastic#38632) Enable Dockerfile from artifacts.elastic.co (elastic#38552) Mute FollowerFailOverIT testFailOverOnFollower (elastic#38634) Account for a possible rolled over file while reading the audit log file (elastic#34909) Mute failure in InternalEngineTests (elastic#38622) Fix Issue with Concurrent Snapshot Init + Delete (elastic#38518) Refactor ZonedDateTime.now in millis resolution (elastic#38577) Mute failing WatchStatusIntegrationTests (elastic#38621) Mute failing ApiKeyIntegTests (elastic#38614) [DOCS] Add warning about bypassing ML PUT APIs (elastic#38509) Add 7.1 and 8.0 version constants to master (elastic#38514) ML: update set_upgrade_mode, add logging (elastic#38372) bad formatted JSON object (elastic#38515) (elastic#38525) Fix HistoryIntegrationTests timestamp comparsion (elastic#38505) SQL: Fix issue with IN not resolving to underlying keyword field (elastic#38440) Fix the clock resolution to millis in ScheduledEventTests (elastic#38506) Enable BWC after backport recovering leases (elastic#38485) Collapse retention lease integration tests (elastic#38483) TransportVerifyShardBeforeCloseAction should force a flush (elastic#38401) ...

* Fix Issue with Concurrent Snapshot Init + Delete by ensuring that we're not finalizing a snapshot in the repository while it is initializing on another thread * Closes elastic#38489

* Fix Issue with Concurrent Snapshot Init + Delete by ensuring that we're not finalizing a snapshot in the repository while it is initializing on another thread * Closes #38489

talevy · 2019-02-16T00:58:42Z

I went ahead and backported this since it was still causing problems in CI

Fix Issue with Concurrent Snapshot Init + Delete

6387014

* Closes elastic#38489

original-brownbear added >non-issue :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 labels Feb 6, 2019

original-brownbear commented Feb 6, 2019

View reviewed changes

Merge remote-tracking branch 'elastic/master' into 38489

3a95407

original-brownbear requested a review from ywelsch February 6, 2019 16:39

ywelsch approved these changes Feb 7, 2019

View reviewed changes

original-brownbear added 2 commits February 7, 2019 10:02

Merge remote-tracking branch 'elastic/master' into 38489

28128cd

assert instead of if

5aafa44

Merge remote-tracking branch 'elastic/master' into 38489

46bb7a3

original-brownbear added v7.2.0 v7.0.0-beta1 v7.0.0 and removed v7.0.0-beta1 labels Feb 8, 2019

original-brownbear merged commit b35d3f0 into elastic:master Feb 8, 2019

original-brownbear deleted the 38489 branch February 8, 2019 14:59

original-brownbear added the backport pending label Feb 8, 2019

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 12, 2019

talevy mentioned this pull request Feb 15, 2019

Fix Issue with Concurrent Snapshot Init + Delete (#38518) #38969

Merged

talevy added v7.0.0 and removed backport pending v7.0.0-beta1 labels Feb 16, 2019

michaelbaamonde added v7.0.0-rc1 and removed v7.0.0 labels Mar 25, 2019

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Issue with Concurrent Snapshot Init + Delete #38518

Fix Issue with Concurrent Snapshot Init + Delete #38518

original-brownbear commented Feb 6, 2019 •

edited

Loading

elasticmachine commented Feb 6, 2019

original-brownbear Feb 6, 2019 •

edited

Loading

original-brownbear Feb 6, 2019

original-brownbear commented Feb 6, 2019

original-brownbear commented Feb 6, 2019

original-brownbear commented Feb 6, 2019

original-brownbear commented Feb 6, 2019

ywelsch Feb 7, 2019

ywelsch Feb 7, 2019

original-brownbear Feb 7, 2019

original-brownbear commented Feb 7, 2019

original-brownbear commented Feb 7, 2019

cbuescher commented Feb 8, 2019

original-brownbear commented Feb 8, 2019

talevy commented Feb 16, 2019

Fix Issue with Concurrent Snapshot Init + Delete #38518

Fix Issue with Concurrent Snapshot Init + Delete #38518

Conversation

original-brownbear commented Feb 6, 2019 • edited Loading

elasticmachine commented Feb 6, 2019

original-brownbear Feb 6, 2019 • edited Loading

Choose a reason for hiding this comment

original-brownbear Feb 6, 2019

Choose a reason for hiding this comment

original-brownbear commented Feb 6, 2019

original-brownbear commented Feb 6, 2019

original-brownbear commented Feb 6, 2019

original-brownbear commented Feb 6, 2019

ywelsch Feb 7, 2019

Choose a reason for hiding this comment

ywelsch Feb 7, 2019

Choose a reason for hiding this comment

original-brownbear Feb 7, 2019

Choose a reason for hiding this comment

original-brownbear commented Feb 7, 2019

original-brownbear commented Feb 7, 2019

cbuescher commented Feb 8, 2019

original-brownbear commented Feb 8, 2019

talevy commented Feb 16, 2019

original-brownbear commented Feb 6, 2019 •

edited

Loading

original-brownbear Feb 6, 2019 •

edited

Loading