Allow date math for naming newly-created snapshots (#7939) #30479

vladimirdolzhenko · 2018-05-09T08:15:03Z

Use date math expression (same as used for indices naming) for naming newly-created snapshots

elasticmachine · 2018-05-09T08:15:05Z

Pinging @elastic/es-distributed

ywelsch

Can you add a description to your PR to explain what it does? Also remove the backport_pending label please, it should only be used if you've merged the PR into master but not backported the change (e.g. due to some complication)

ywelsch · 2018-05-09T12:34:58Z

docs/reference/modules/snapshots.asciidoc

@@ -282,7 +282,8 @@ PUT /_snapshot/my_backup/snapshot_2?wait_for_completion=true
 // TEST[continued]

 The list of indices that should be included into the snapshot can be specified using the `indices` parameter that
-supports <<multi-index,multi index syntax>>. The snapshot request also supports the
+supports <<multi-index,multi index syntax>>. It can be useful to use <<date-math-index-names,date math>> to name the


This links to docs on using date math for index names. This can be confusing. I think you need to clarify here that the same date math expression that is used for indices can be used for snapshot names. I think it would be better to put this info into a separate paragraph, together with an example. I wonder if we need to support date math not only for snapshot creation, but also for other operations.

ywelsch · 2018-05-09T12:36:49Z

server/src/test/java/org/elasticsearch/action/admin/cluster/snapshots/SnapshotBlocksIT.java

@@ -176,4 +177,30 @@ public void testSnapshotStatusWithBlocks() {
            setClusterReadOnly(false);
        }
    }
+
+    public void testSnapshotWithDateMath() {


I'm not sure what this is exactly testing. There are no assertions that check that the date math functionality has worked in a meaningful way. Also why is this testing blocks?

it tests that it is possible to create snapshot with name <snapshot-{now/d}> and it is resolved into smth like snapshot-2018.05.11 using nameExpressionResolver.resolveDateMathExpression(snapshotName) - I assume that nameExpressionResolver works.

Agree - it is worth to move test to DedicatedClusterSnapshotRestoreIT - is that ok ?

move test case to DedicatedClusterSnapshotRestoreIT

extended doc: example is added. referring to indices date math more clearly

ywelsch

I've left a few more comments. Can you also adapt the PR title so that it reflects more closely what's implemented here:

Allow date math for naming newly-created snapshots

and make it clear in the description that this only allows date math when creating new snapshots (it does not implement date math when resolving snapshot names, for exampling when asking for the snapshots that currently exist in the repo).

ywelsch · 2018-05-15T10:07:02Z

server/src/test/java/org/elasticsearch/action/admin/cluster/snapshots/SnapshotBlocksIT.java

@@ -176,4 +176,6 @@ public void testSnapshotStatusWithBlocks() {
            setClusterReadOnly(false);
        }
    }
+
+


can you revert this?

ywelsch · 2018-05-15T10:09:07Z

server/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java

+
+        final IndexNameExpressionResolver nameExpressionResolver = new IndexNameExpressionResolver(Settings.EMPTY);
+        final String snapshotName = "<snapshot-{now/d}>";
+        final String expression = nameExpressionResolver.resolveDateMathExpression(snapshotName);


I think that if there is a day rollover between resolving the expression here and taking the snapshot, the test will fail. A fix might be to resolve the expression a second time after the snapshot was taken, and then use both resolved expressions when asking for the snapshot status.

ywelsch · 2018-05-15T10:09:37Z

server/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java

+            admin.cluster().prepareCreateSnapshot(repo, snapshotName)
+                .setIncludeGlobalState(true)
+                .setWaitForCompletion(true)
+                .execute().actionGet();


just .get() will do. Also no need for setIncludeGlobalState(true), that's irrelevant for the test.

agreed, just copy-n-pasted from another place

ywelsch · 2018-05-15T10:23:50Z

docs/reference/modules/snapshots.asciidoc

@@ -289,6 +289,19 @@ By setting `include_global_state` to false it's possible to prevent the cluster
 the snapshot. By default, the entire snapshot will fail if one or more indices participating in the snapshot don't have
 all primary shards available. This behaviour can be changed by setting `partial` to `true`.

+Snapshot could be named according to the date that the snapshot made in a similar way as indices


How about the following:

Snapshot names can be automatically derived using <<date-math-index-names,date math expressions>>, similarly as when creating new indices. Note that special characters need to be URI encoded. For example, creating a snapshot with the current day in the name, like snapshot-2018.05.11, can be achieved with the following command:

...

vladimirdolzhenko · 2018-05-15T10:59:09Z

@ywelsch thank you for the feedback - I made relevant amendments.

ywelsch

Thanks @vladimirdolzhenko - LGTM. I've added version labels to the PR.

ywelsch · 2018-05-15T11:31:40Z

server/src/test/java/org/elasticsearch/snapshots/DedicatedClusterSnapshotRestoreIT.java

+        final String expression2 = nameExpressionResolver.resolveDateMathExpression(snapshotName);
+
+        SnapshotsStatusResponse response = admin.cluster().prepareSnapshotStatus(repo)
+            .setSnapshots(Sets.newHashSet(expression1, expression2).toArray(Strings.EMPTY_ARRAY))


this is a varargs method, so just .setSnapshots(expression1, expression2) is good enough.

w/o a day rollover it requests two statuses with two identical names - and the result is 2 snapshot statuses - that's why I used set here

jasontedor · 2018-05-15T17:34:38Z

test this please

(cherry picked from commit fe3e025)

…bad-mkay * elastic/6.x: [ML] Wait for ML indices in rolling upgrade tests (elastic#30615) Watcher: Ensure secrets integration tests also run triggered watch (elastic#30478) Move allocation awareness attributes to list setting (elastic#30626) [Docs] Update code snippet in has-child-query.asciidoc (elastic#30510) Allow date math for naming newly-created snapshots (elastic#7939) (elastic#30479) Awaits fix a failing test Switch many QA projects to use new style requests (elastic#30574) QA: System property to override distribution (elastic#30591)

vladimirdolzhenko · 2018-05-29T07:30:27Z

thanks @ywelsch

Automatic snapshot naming (elastic#7939)

4055c34

vladimirdolzhenko added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs backport pending labels May 9, 2018

vladimirdolzhenko requested a review from ywelsch May 9, 2018 08:15

ywelsch suggested changes May 9, 2018

View reviewed changes

Vladimir Dolzhenko added 2 commits May 11, 2018 09:38

Automatic snapshot naming elastic#7939

4b20260

move test case to DedicatedClusterSnapshotRestoreIT

Automatic snapshot naming elastic#7939

b168ec8

extended doc: example is added. referring to indices date math more clearly

vladimirdolzhenko requested a review from ywelsch May 11, 2018 07:56

vladimirdolzhenko removed the backport pending label May 11, 2018

ywelsch suggested changes May 15, 2018

View reviewed changes

vladimirdolzhenko changed the title ~~Automatic snapshot naming (#7939)~~ Allow date math for naming newly-created snapshots (#7939) May 15, 2018

Vladimir Dolzhenko added 4 commits May 15, 2018 12:39

elastic#7939 dropped unnecessary new lines in SnapshotBlocksIT

256c3c7

elastic#7939 use simple get() instead of execute().actionGet()

117495f

elastic#7939 fix test taking into account potential day rollover

ff244bd

elastic#7939 doc adjusted

4648145

vladimirdolzhenko requested a review from ywelsch May 15, 2018 10:57

ywelsch added v7.0.0 v6.4.0 labels May 15, 2018

ywelsch approved these changes May 15, 2018

View reviewed changes

Merge remote-tracking branch 'remotes/origin/master' into 7939

04404d0

vladimirdolzhenko merged commit fe3e025 into elastic:master May 16, 2018

vladimirdolzhenko deleted the 7939 branch May 16, 2018 05:23

vladimirdolzhenko pushed a commit that referenced this pull request May 16, 2018

Allow date math for naming newly-created snapshots (#7939) (#30479)

d212b30

(cherry picked from commit fe3e025)

colings86 added the v7.0.0-beta1 label Feb 7, 2019

colings86 removed the v7.0.0 label Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow date math for naming newly-created snapshots (#7939) #30479

Allow date math for naming newly-created snapshots (#7939) #30479

vladimirdolzhenko commented May 9, 2018 •

edited

Loading

elasticmachine commented May 9, 2018

ywelsch left a comment

ywelsch May 9, 2018

ywelsch May 9, 2018

vladimirdolzhenko May 11, 2018

ywelsch left a comment

ywelsch May 15, 2018

ywelsch May 15, 2018

ywelsch May 15, 2018

vladimirdolzhenko May 15, 2018

ywelsch May 15, 2018

vladimirdolzhenko commented May 15, 2018

ywelsch left a comment

ywelsch May 15, 2018

vladimirdolzhenko May 15, 2018

ywelsch May 15, 2018

jasontedor commented May 15, 2018

vladimirdolzhenko commented May 29, 2018

Allow date math for naming newly-created snapshots (#7939) #30479

Allow date math for naming newly-created snapshots (#7939) #30479

Conversation

vladimirdolzhenko commented May 9, 2018 • edited Loading

elasticmachine commented May 9, 2018

ywelsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ywelsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirdolzhenko commented May 15, 2018

ywelsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasontedor commented May 15, 2018

vladimirdolzhenko commented May 29, 2018

vladimirdolzhenko commented May 9, 2018 •

edited

Loading