Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flatten Get Snapshots Response #74451

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
21665ae
step
original-brownbear Jun 22, 2021
3b094da
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 22, 2021
e7b70ac
fix it
original-brownbear Jun 22, 2021
fff9899
works
original-brownbear Jun 22, 2021
75bf116
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 23, 2021
c4cb98f
global sort order
original-brownbear Jun 23, 2021
6c59f0d
bwc and test
original-brownbear Jun 23, 2021
38d4780
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 23, 2021
588180e
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 23, 2021
902c259
fix test
original-brownbear Jun 23, 2021
52139d7
even moar test adjustments
original-brownbear Jun 23, 2021
2e1c9e9
reenable tests
original-brownbear Jun 23, 2021
d1529eb
reactivate more
original-brownbear Jun 23, 2021
3b4bf39
so many tests
original-brownbear Jun 23, 2021
719581c
woerks
original-brownbear Jun 23, 2021
731915a
fix all kinds of tests
original-brownbear Jun 23, 2021
fb687f1
and another one ...'
original-brownbear Jun 23, 2021
3f3c525
finally?
original-brownbear Jun 23, 2021
b6b218f
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 23, 2021
0393b0e
fix more
original-brownbear Jun 23, 2021
49a66e0
fix more
original-brownbear Jun 23, 2021
b8a36af
fix
original-brownbear Jun 23, 2021
f2da1ba
fix
original-brownbear Jun 23, 2021
0a0fe71
one more
original-brownbear Jun 23, 2021
a34b3e9
no more breaking change
original-brownbear Jun 23, 2021
c54d196
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 23, 2021
fa58d3d
yet another one
original-brownbear Jun 23, 2021
ab6df13
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 23, 2021
5332fda
yet another one
original-brownbear Jun 23, 2021
3a23d61
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 24, 2021
e5f9a6c
check not null
original-brownbear Jun 24, 2021
fc027a4
reenable tests + nits
original-brownbear Jun 24, 2021
5b30288
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 24, 2021
db1f4d6
Merge remote-tracking branch 'elastic/master' into flatten-get-snapsh…
original-brownbear Jun 24, 2021
bcade7d
include repository in responses by default
original-brownbear Jun 24, 2021
09b49f6
default include snapshot name, add test for it
original-brownbear Jun 24, 2021
617ad0f
CR: comments
original-brownbear Jun 24, 2021
14f88f3
CR: test changes
original-brownbear Jun 24, 2021
63b19e3
typo
original-brownbear Jun 24, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,14 @@
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.snapshots.AbstractSnapshotIntegTestCase;
import org.elasticsearch.snapshots.RestoreInfo;
import org.elasticsearch.snapshots.SnapshotInfo;
import org.mockito.internal.util.collections.Sets;

import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

import static org.elasticsearch.snapshots.SnapshotsService.NO_FEATURE_STATES_VALUE;
import static org.elasticsearch.tasks.TaskResultsService.TASKS_FEATURE_NAME;
Expand Down Expand Up @@ -209,14 +211,17 @@ public void testGetSnapshots() throws IOException {
GetSnapshotsResponse response = execute(request, highLevelClient().snapshot()::get, highLevelClient().snapshot()::getAsync);

assertThat(response.isFailed(), is(false));
assertThat(response.getRepositories(), equalTo(Sets.newSet(repository1, repository2)));

assertThat(response.getSnapshots(repository1), hasSize(1));
assertThat(response.getSnapshots(repository1).get(0).snapshotId().getName(), equalTo(snapshot1));
assertEquals(
Sets.newSet(repository1, repository2),
response.getSnapshots().stream().map(SnapshotInfo::repository).collect(Collectors.toSet())
);

assertThat(response.getSnapshots(repository2), hasSize(1));
assertThat(response.getSnapshots(repository2).get(0).snapshotId().getName(), equalTo(snapshot2));
assertThat(response.getSnapshots(repository2).get(0).userMetadata(), equalTo(originalMetadata));
assertThat(response.getSnapshots(), hasSize(2));
assertThat(response.getSnapshots().get(0).snapshotId().getName(), equalTo(snapshot1));
assertThat(response.getSnapshots().get(0).repository(), equalTo(repository1));
assertThat(response.getSnapshots().get(1).snapshotId().getName(), equalTo(snapshot2));
assertThat(response.getSnapshots().get(1).userMetadata(), equalTo(originalMetadata));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also validate that snapshot1 has repo1 and snapshot2 has repo2?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes definitely, added that assertion :)

assertThat(response.getSnapshots().get(1).repository(), equalTo(repository2));
}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1044,7 +1044,7 @@ private void assertSnapshotExists(final RestHighLevelClient client, final String
GetSnapshotsRequest getSnapshotsRequest = new GetSnapshotsRequest(new String[]{repo}, new String[]{snapshotName});
try {
final GetSnapshotsResponse snaps = client.snapshot().get(getSnapshotsRequest, RequestOptions.DEFAULT);
Optional<SnapshotInfo> info = snaps.getSnapshots(repo).stream().findFirst();
Optional<SnapshotInfo> info = snaps.getSnapshots().stream().findFirst();
if (info.isPresent()) {
info.ifPresent(si -> {
assertThat(si.snapshotId().getName(), equalTo(snapshotName));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -617,7 +617,7 @@ public void testSnapshotGetSnapshots() throws IOException {
// end::get-snapshots-execute

// tag::get-snapshots-response
List<SnapshotInfo> snapshotsInfos = response.getSnapshots(repositoryName);
List<SnapshotInfo> snapshotsInfos = response.getSnapshots();
SnapshotInfo snapshotInfo = snapshotsInfos.get(0);
RestStatus restStatus = snapshotInfo.status(); // <1>
SnapshotId snapshotId = snapshotInfo.snapshotId(); // <2>
Expand Down
58 changes: 0 additions & 58 deletions docs/reference/migration/migrate_8_0/snapshots.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,64 +9,6 @@

// end::notable-breaking-changes[]

.The get snapshot API's response format has changed.
[%collapsible]
====
*Details* +
It's possible to get snapshots from multiple repositories in one go. The response format has changed
and now contains separate response for each repository.

For example, requesting one snapshot from particular repository

[source,console]
-----------------------------------
GET _snapshot/repo1/snap1
-----------------------------------
// TEST[skip:no repo and snapshots are created]

produces the following response

[source,console-result]
-----------------------------------
{
"responses": [
{
"repository": "repo1",
"snapshots": [
{
"snapshot": "snap1",
"uuid": "cEzdqUKxQ5G6MyrJAcYwmA",
"version_id": 8000099,
"version": "8.0.0",
"indices": [],
"include_global_state": true,
"state": "SUCCESS",
"start_time": "2019-05-10T17:01:57.868Z",
"start_time_in_millis": 1557507717868,
"end_time": "2019-05-10T17:01:57.909Z",
"end_time_in_millis": 1557507717909,
"duration_in_millis": 41,
"failures": [],
"shards": {
"total": 0,
"failed": 0,
"successful": 0
}
}
]
}
]
}
-----------------------------------
// TESTRESPONSE[skip:no repo and snapshots are created]

See <<modules-snapshots>> for more information.

*Impact* +
Update your workflow and applications to use the get snapshot API's new response
format.
====

.The `repositories.fs.compress` node-level setting has been removed.
[%collapsible]
====
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ The API returns the following response:
"snapshot": {
"snapshot": "snapshot_2",
"uuid": "vdRctLCxSketdKb54xw67g",
"repository": "my_repository",
"version_id": <version_id>,
"version": <version>,
"indices": [],
Expand Down
62 changes: 29 additions & 33 deletions docs/reference/snapshot-restore/apis/get-snapshot-api.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -284,42 +284,38 @@ The API returns the following response:
[source,console-result]
----
{
"responses": [
"snapshots": [
{
"snapshot": "snapshot_2",
"uuid": "vdRctLCxSketdKb54xw67g",
"repository": "my_repository",
"snapshots": [
{
"snapshot": "snapshot_2",
"uuid": "vdRctLCxSketdKb54xw67g",
"version_id": <version_id>,
"version": <version>,
"indices": [],
"data_streams": [],
"feature_states": [],
"include_global_state": true,
"state": "SUCCESS",
"start_time": "2020-07-06T21:55:18.129Z",
"start_time_in_millis": 1593093628850,
"end_time": "2020-07-06T21:55:18.129Z",
"end_time_in_millis": 1593094752018,
"duration_in_millis": 0,
"failures": [],
"shards": {
"total": 0,
"failed": 0,
"successful": 0
}
}
]
"version_id": <version_id>,
"version": <version>,
"indices": [],
"data_streams": [],
"feature_states": [],
"include_global_state": true,
"state": "SUCCESS",
"start_time": "2020-07-06T21:55:18.129Z",
"start_time_in_millis": 1593093628850,
"end_time": "2020-07-06T21:55:18.129Z",
"end_time_in_millis": 1593094752018,
"duration_in_millis": 0,
"failures": [],
"shards": {
"total": 0,
"failed": 0,
"successful": 0
}
}
]
}
----
// TESTRESPONSE[s/"uuid": "vdRctLCxSketdKb54xw67g"/"uuid": $body.responses.0.snapshots.0.uuid/]
// TESTRESPONSE[s/"version_id": <version_id>/"version_id": $body.responses.0.snapshots.0.version_id/]
// TESTRESPONSE[s/"version": <version>/"version": $body.responses.0.snapshots.0.version/]
// TESTRESPONSE[s/"start_time": "2020-07-06T21:55:18.129Z"/"start_time": $body.responses.0.snapshots.0.start_time/]
// TESTRESPONSE[s/"start_time_in_millis": 1593093628850/"start_time_in_millis": $body.responses.0.snapshots.0.start_time_in_millis/]
// TESTRESPONSE[s/"end_time": "2020-07-06T21:55:18.129Z"/"end_time": $body.responses.0.snapshots.0.end_time/]
// TESTRESPONSE[s/"end_time_in_millis": 1593094752018/"end_time_in_millis": $body.responses.0.snapshots.0.end_time_in_millis/]
// TESTRESPONSE[s/"duration_in_millis": 0/"duration_in_millis": $body.responses.0.snapshots.0.duration_in_millis/]
// TESTRESPONSE[s/"uuid": "vdRctLCxSketdKb54xw67g"/"uuid": $body.snapshots.0.uuid/]
// TESTRESPONSE[s/"version_id": <version_id>/"version_id": $body.snapshots.0.version_id/]
// TESTRESPONSE[s/"version": <version>/"version": $body.snapshots.0.version/]
// TESTRESPONSE[s/"start_time": "2020-07-06T21:55:18.129Z"/"start_time": $body.snapshots.0.start_time/]
// TESTRESPONSE[s/"start_time_in_millis": 1593093628850/"start_time_in_millis": $body.snapshots.0.start_time_in_millis/]
// TESTRESPONSE[s/"end_time": "2020-07-06T21:55:18.129Z"/"end_time": $body.snapshots.0.end_time/]
// TESTRESPONSE[s/"end_time_in_millis": 1593094752018/"end_time_in_millis": $body.snapshots.0.end_time_in_millis/]
// TESTRESPONSE[s/"duration_in_millis": 0/"duration_in_millis": $body.snapshots.0.duration_in_millis/]
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ PUT /_snapshot/my_fs_backup
}

PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true

PUT /_snapshot/my_backup/some_other_snapshot?wait_for_completion=true
henningandersen marked this conversation as resolved.
Show resolved Hide resolved
-----------------------------------
// TESTSETUP

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.notNullValue;

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST)
public class URLSnapshotRestoreIT extends ESIntegTestCase {
Expand Down Expand Up @@ -77,7 +76,7 @@ public void testUrlRepository() throws Exception {
.prepareGetSnapshots("test-repo")
.setSnapshots("test-snap")
.get()
.getSnapshots("test-repo")
.getSnapshots()
.get(0)
.state();
assertThat(state, equalTo(SnapshotState.SUCCESS));
Expand Down Expand Up @@ -105,16 +104,14 @@ public void testUrlRepository() throws Exception {

logger.info("--> list available shapshots");
GetSnapshotsResponse getSnapshotsResponse = client.admin().cluster().prepareGetSnapshots("url-repo").get();
assertThat(getSnapshotsResponse.getSnapshots("url-repo"), notNullValue());
assertThat(getSnapshotsResponse.getSnapshots("url-repo").size(), equalTo(1));
assertThat(getSnapshotsResponse.getSnapshots().size(), equalTo(1));

logger.info("--> delete snapshot");
AcknowledgedResponse deleteSnapshotResponse = client.admin().cluster().prepareDeleteSnapshot("test-repo", "test-snap").get();
assertAcked(deleteSnapshotResponse);

logger.info("--> list available shapshot again, no snapshots should be returned");
getSnapshotsResponse = client.admin().cluster().prepareGetSnapshots("url-repo").get();
assertThat(getSnapshotsResponse.getSnapshots("url-repo"), notNullValue());
assertThat(getSnapshotsResponse.getSnapshots("url-repo").size(), equalTo(0));
assertThat(getSnapshotsResponse.getSnapshots().size(), equalTo(0));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -112,9 +112,6 @@ teardown:

---
"Restore with repository-url using http://":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

# Ensure that the URL repository is registered
- do:
Expand All @@ -129,9 +126,9 @@ teardown:
repository: repository-url
snapshot: snapshot-one,snapshot-two

- is_true: responses.0.snapshots
- match: { responses.0.snapshots.0.state : SUCCESS }
- match: { responses.0.snapshots.1.state : SUCCESS }
- is_true: snapshots
- match: { snapshots.0.state : SUCCESS }
- match: { snapshots.1.state : SUCCESS }

# Delete the index
- do:
Expand Down Expand Up @@ -177,9 +174,6 @@ teardown:

---
"Restore with repository-url using file://":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

# Ensure that the URL repository is registered
- do:
Expand All @@ -194,9 +188,9 @@ teardown:
repository: repository-file
snapshot: snapshot-one,snapshot-two

- is_true: responses.0.snapshots
- match: { responses.0.snapshots.0.state : SUCCESS }
- match: { responses.0.snapshots.1.state : SUCCESS }
- is_true: snapshots
- match: { snapshots.0.state : SUCCESS }
- match: { snapshots.1.state : SUCCESS }

# Delete the index
- do:
Expand Down Expand Up @@ -242,18 +236,13 @@ teardown:

---
"Get a non existing snapshot":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

- do:
catch: /snapshot_missing_exception/
snapshot.get:
repository: repository-url
snapshot: missing

- is_true: responses.0.error
- match: { responses.0.error.type: snapshot_missing_exception }

---
"Delete a non existing snapshot":

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,6 @@ setup:

---
"Snapshot/Restore with repository-azure":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

# Get repository
- do:
Expand Down Expand Up @@ -130,9 +127,9 @@ setup:
repository: repository
snapshot: snapshot-one,snapshot-two

- is_true: responses.0.snapshots
- match: { responses.0.snapshots.0.state: SUCCESS }
- match: { responses.0.snapshots.1.state: SUCCESS }
- is_true: snapshots
- match: { snapshots.0.state : SUCCESS }
- match: { snapshots.1.state : SUCCESS }

# Delete the index
- do:
Expand Down Expand Up @@ -212,18 +209,13 @@ setup:

---
"Get a non existing snapshot":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

- do:
catch: /snapshot_missing_exception/
snapshot.get:
repository: repository
snapshot: missing

- is_true: responses.0.error
- match: { responses.0.error.type: snapshot_missing_exception }

---
"Delete a non existing snapshot":

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,6 @@ setup:

---
"Snapshot/Restore with repository-gcs":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

# Get repository
- do:
Expand Down Expand Up @@ -130,9 +127,9 @@ setup:
repository: repository
snapshot: snapshot-one,snapshot-two

- is_true: responses.0.snapshots
- match: { responses.0.snapshots.0.state : SUCCESS }
- match: { responses.0.snapshots.1.state : SUCCESS }
- is_true: snapshots
- match: { snapshots.0.state : SUCCESS }
- match: { snapshots.1.state : SUCCESS }

# Delete the index
- do:
Expand Down Expand Up @@ -209,18 +206,13 @@ setup:

---
"Get a non existing snapshot":
- skip:
version: " - 7.99.99"
reason: "8.0 changes get snapshots response format"

- do:
catch: /snapshot_missing_exception/
snapshot.get:
repository: repository
snapshot: missing

- is_true: responses.0.error
- match: { responses.0.error.type: snapshot_missing_exception }

---
"Delete a non existing snapshot":

Expand Down
Loading