Skip to content

Commit

Permalink
[DOCS] Clarify definition of max_size (#56561) (#56648)
Browse files Browse the repository at this point in the history
* [DOCS] Extract the cron docs from Watcher docs and add to the API conventions. (#56313)

* [DOCS] Promote cron expressions info from Watcher to a separate topic.

* Fix table error

* Fixed xref

* Apply suggestions from code review

Co-authored-by: James Rodewig <[email protected]>

* Incorporated review feedback

Co-authored-by: James Rodewig <[email protected]>

* [DOCS] Clarify definition of max_size (#56561)

Co-authored-by: James Rodewig <[email protected]>
  • Loading branch information
debadair and jrodewig authored May 12, 2020
1 parent af0e2fd commit ed31d0d
Show file tree
Hide file tree
Showing 10 changed files with 252 additions and 228 deletions.
3 changes: 3 additions & 0 deletions docs/reference/api-conventions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ API, unless otherwise specified.

* <<multi-index>>
* <<date-math-index-names>>
* <<cron-expressions>>
* <<common-options>>
* <<url-access-control>>

Expand Down Expand Up @@ -143,6 +144,8 @@ GET /%3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogs
// TEST[s/^/PUT logstash-2016.09.20\nPUT logstash-2016.09.19\nPUT logstash-2016.09.18\n/]
// TEST[s/now/2016.09.20||/]

include::rest-api/cron-expressions.asciidoc[]

[[common-options]]
=== Common options

Expand Down
9 changes: 4 additions & 5 deletions docs/reference/commands/croneval.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[[elasticsearch-croneval]]
== elasticsearch-croneval

Validates and evaluates a cron expression.
Validates and evaluates a <<cron-expressions,cron expression>>.

[discrete]
=== Synopsis
Expand All @@ -19,12 +19,11 @@ bin/elasticsearch-croneval <expression>
=== Description

This command enables you to verify that your
https://en.wikipedia.org/wiki/Cron[cron] expressions are valid for use with the
{es} {alert-features} and produce the expected results.
cron expressions are valid for use with
{es} and produce the expected results.

This command is provided in the `$ES_HOME/bin` directory.

[discrete]
=== Parameters

`-c, --count` <Integer>::
Expand All @@ -45,7 +44,7 @@ This command is provided in the `$ES_HOME/bin` directory.
Shows verbose output.

[discrete]
=== Examples
=== Example

If the cron expression is valid, the following command displays the next
20 times that the schedule will be triggered:
Expand Down
17 changes: 11 additions & 6 deletions docs/reference/ilm/actions/ilm-rollover.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,19 +43,24 @@ PUT my_index-000001
You must specify at least one rollover condition.
An empty rollover action is invalid.

`max_size`::
(Optional, <<byte-units, byte units>>)
Triggers roll over after the specified maximum primary shard index storage size is reached.
`max_age`::
(Optional, <<time-units, time units>>)
Triggers roll over after the maximum elapsed time from index creation is reached.

`max_docs`::
(Optional, integer)
Triggers roll over after the specified maximum number of documents is reached.
Documents added since the last refresh are not included in the document count.
The document count does *not* include documents in replica shards.

`max_age`::
(Optional, <<time-units, time units>>)
Triggers roll over after the maximum elapsed time from index creation is reached.
`max_size`::
(Optional, <<byte-units, byte units>>)
Triggers roll over when the index reaches a certain size.
This is the total size of all primary shards in the index.
Replicas are not counted toward the maximum index size.
+
TIP: To see the current index size, use the <<cat-indices, _cat indices>> API.
The `pri.store.size` value shows the combined size of all primary shards.

[[ilm-rollover-ex]]
==== Example
Expand Down
35 changes: 12 additions & 23 deletions docs/reference/ilm/using-policies-rollover.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,17 @@
== Configure rollover
[[using-policies-rollover]]
You control when the rollover action is triggered by specifying one or more
rollover parameters. The rollover is performed once any of the criteria are
met. Because the criteria are checked periodically, the index might grow
slightly beyond the specified threshold. To control how often the criteria are
checked, specify the `indices.lifecycle.poll_interval` cluster setting.
rollover criteria:

* Maximum size (the combined size of all primary shards in the index)
* Maximum document count
* Maximum age

The rollover is performed once any of the criteria are met.
Because the criteria are checked periodically, the index might grow
slightly beyond the specified threshold.
To control how often the criteria are checked,
specify the `indices.lifecycle.poll_interval` cluster setting.

IMPORTANT: New indices created via rollover will not automatically inherit the
policy used by the old index, and will not use any policy by default. Therefore,
Expand All @@ -16,24 +23,6 @@ it is highly recommended to apply the policy via
setting, for your indices which specifies the policy you wish to use for each
new index.

The rollover action takes the following parameters:

[[rollover-action-params]]
.`rollover` Action Parameters
[options="header"]
|===
|Name |Description
|max_size |The maximum estimated size the primary shard of the index is allowed
to grow to. Defaults to `null`. Optional.
|max_docs |The maximum number of document the index should
contain. Defaults to `null`. Optional.
|max_age |The maximum age of the index. Defaults to `null`. Optional.
|===

These parameters are used to determine when the index is considered "full" and
a rollover should be performed. Where multiple criteria are defined the
rollover operation will be performed once any of the criteria are met.

The following request defines a policy with a rollover action that triggers
when the index size reaches 25GB. The old index is subsequently deleted after
30 days.
Expand Down Expand Up @@ -127,7 +116,7 @@ the new index, enabling indexing to continue uninterrupted.
[[skipping-rollover]]
=== Skipping Rollover

The `index.lifecycle.indexing_complete` setting indicates to {ilm} whether this
The `index.lifecycle.indexing_complete` setting indicates to {ilm-init} whether this
index has already been rolled over. If it is set to `true`, that indicates that
this index has already been rolled over and does not need to be rolled over
again. Therefore, {ilm} will skip any Rollover Action configured in the
Expand Down
7 changes: 6 additions & 1 deletion docs/reference/indices/rollover-index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,12 @@ The document count does *not* include documents in replica shards.

`max_size`::
(Optional, <<byte-units, byte units>>)
Maximum estimated size of the primary shard of the index.
Maximum index size.
This is the total size of all primary shards in the index.
Replicas are not counted toward the maximum index size.

TIP: To see the current index size, use the <<cat-indices, _cat indices>> API.
The `pri.store.size` value shows the combined size of all primary shards.
--

include::{docdir}/rest-api/common-parms.asciidoc[tag=mappings]
Expand Down
182 changes: 182 additions & 0 deletions docs/reference/rest-api/cron-expressions.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
[[cron-expressions]]
=== Cron expressions

A cron expression is a string of the following form:

[source,txt]
------------------------------
<seconds> <minutes> <hours> <day_of_month> <month> <day_of_week> [year]
------------------------------

{es} uses the cron parser from the http://www.quartz-scheduler.org[Quartz Job Scheduler].
For more information about writing Quartz cron expressions, see the
http://www.quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/tutorial-lesson-06.html[Quartz CronTrigger Tutorial].

All schedule times are in coordinated universal time (UTC); other timezones are not supported.

TIP: You can use the <<elasticsearch-croneval>> command line tool to validate your cron expressions.


[[cron-elements]]
==== Cron expression elements

All elements are required except for `year`.
See <<cron-special-characters>> for information about the allowed special characters.

`<seconds>`::
(Required)
Valid values: `0`-`59` and the special characters `,` `-` `*` `/`

`<minutes>`::
(Required)
Valid values: `0`-`59` and the special characters `,` `-` `*` `/`

`<hours>`::
(Required)
Valid values: `0`-`23` and the special characters `,` `-` `*` `/`

`<day_of_month>`::
(Required)
Valid values: `1`-`31` and the special characters `,` `-` `*` `/` `?` `L` `W`

`<month>`::
(Required)
Valid values: `1`-`12`, `JAN`-`DEC`, `jan`-`dec`, and the special characters `,` `-` `*` `/`

`<day_of_week>`::
(Required)
Valid values: `1`-`7`, `SUN`-`SAT`, `sun`-`sat`, and the special characters `,` `-` `*` `/` `?` `L` `#`

`<year>`::
(Optional)
Valid values: `1970`-`2099` and the special characters `,` `-` `*` `/`

[[cron-special-characters]]
==== Cron special characters

`*`::
Selects every possible value for a field. For
example, `*` in the `hours` field means "every hour".

`?`::
No specific value. Use when you don't care what the value
is. For example, if you want the schedule to trigger on a
particular day of the month, but don't care what day of
the week that happens to be, you can specify `?` in the
`day_of_week` field.

`-`::
A range of values (inclusive). Use to separate a minimum
and maximum value. For example, if you want the schedule
to trigger every hour between 9:00 a.m. and 5:00 p.m., you
could specify `9-17` in the `hours` field.

`,`::
Multiple values. Use to separate multiple values for a
field. For example, if you want the schedule to trigger
every Tuesday and Thursday, you could specify `TUE,THU`
in the `day_of_week` field.

`/`::
Increment. Use to separate values when specifying a time
increment. The first value represents the starting point,
and the second value represents the interval. For example,
if you want the schedule to trigger every 20 minutes
starting at the top of the hour, you could specify `0/20`
in the `minutes` field. Similarly, specifying `1/5` in
`day_of_month` field will trigger every 5 days starting on
the first day of the month.

`L`::
Last. Use in the `day_of_month` field to mean the last day
of the month--day 31 for January, day 28 for February in
non-leap years, day 30 for April, and so on. Use alone in
the `day_of_week` field in place of `7` or `SAT`, or after
a particular day of the week to select the last day of that
type in the month. For example `6L` means the last Friday
of the month. You can specify `LW` in the `day_of_month`
field to specify the last weekday of the month. Avoid using
the `L` option when specifying lists or ranges of values,
as the results likely won't be what you expect.

`W`::
Weekday. Use to specify the weekday (Monday-Friday) nearest
the given day. As an example, if you specify `15W` in the
`day_of_month` field and the 15th is a Saturday, the
schedule will trigger on the 14th. If the 15th is a Sunday,
the schedule will trigger on Monday the 16th. If the 15th
is a Tuesday, the schedule will trigger on Tuesday the 15th.
However if you specify `1W` as the value for `day_of_month`,
and the 1st is a Saturday, the schedule will trigger on
Monday the 3rd--it won't jump over the month boundary. You
can specify `LW` in the `day_of_month` field to specify the
last weekday of the month. You can only use the `W` option
when the `day_of_month` is a single day--it is not valid
when specifying a range or list of days.

`#`::
Nth XXX day in a month. Use in the `day_of_week` field to
specify the nth XXX day of the month. For example, if you
specify `6#1`, the schedule will trigger on the first
Friday of the month. Note that if you specify `3#5` and
there are not 5 Tuesdays in a particular month, the
schedule won't trigger that month.

[[cron-expression-examples]]
==== Examples

[[cron-example-daily]]
===== Setting daily triggers

`0 5 9 * * ?`::
Trigger at 9:05 a.m. UTC every day.

`0 5 9 * * ? 2020`::
Trigger at 9:05 a.m. UTC every day during the year 2020.

[[cron-example-range]]
===== Restricting triggers to a range of days or times

`0 5 9 ? * MON-FRI`::
Trigger at 9:05 a.m. UTC Monday through Friday.

`0 0-5 9 * * ?`::
Trigger every minute starting at 9:00 a.m. UTC and ending at 9:05 a.m. UTC every day.

[[cron-example-interval]]
===== Setting interval triggers

`0 0/15 9 * * ?`::
Trigger every 15 minutes starting at 9:00 a.m. UTC and ending at 9:45 a.m. UTC every day.

`0 5 9 1/3 * ?`::
Trigger at 9:05 a.m. UTC every 3 days every month, starting on the first day of the month.

[[cron-example-day]]
===== Setting schedules that trigger on a particular day

`0 1 4 1 4 ?`::
Trigger every April 1st at 4:01 a.m. UTC.
`0 0,30 9 ? 4 WED`::
Trigger at 9:00 a.m. UTC and at 9:30 a.m. UTC every Wednesday in the month of April.

`0 5 9 15 * ?`::
Trigger at 9:05 a.m. UTC on the 15th day of every month.

`0 5 9 15W * ?`::
Trigger at 9:05 a.m. UTC on the nearest weekday to the 15th of every month.

`0 5 9 ? * 6#1`::
Trigger at 9:05 a.m. UTC on the first Friday of every month.

[[cron-example-last]]
===== Setting triggers using last

`0 5 9 L * ?`::
Trigger at 9:05 a.m. UTC on the last day of every month.

`0 5 9 ? * 2L`::
Trigger at 9:05 a.m. UTC on the last Monday of every month.

`0 5 9 LW * ?`::
Trigger at 9:05 a.m. UTC on the last weekday of every month.
2 changes: 1 addition & 1 deletion docs/reference/slm/apis/slm-put.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ Minimum number of snapshots to retain, even if the snapshots have expired.
====

`schedule`::
(Required, <<schedule-cron,Cron scheduler configuration>>)
(Required, <<cron-expressions,Cron syntax>>)
Periodic or absolute schedule at which the policy creates snapshots and deletes
expired snapshots. Schedule changes to existing policies are applied immediately.

Expand Down
27 changes: 17 additions & 10 deletions docs/reference/slm/getting-started-slm.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@ This is useful for taking snapshots before making a configuration change,
upgrading, or to test a new policy.
Manually executing a policy does not affect its configured schedule.

For example, the following request manually triggers the `nightly-snapshots` policy:
Instead of waiting for the policy to run, tell {slm-init} to take a snapshot
using the configuration right now instead of waiting for 1:30 a.m..

[source,console]
--------------------------------------------------
Expand Down Expand Up @@ -159,19 +160,25 @@ repository is lost while copying files.
"max_count": 50
}
},
"last_success": {
"snapshot_name": "nightly-snap-2019.04.24-tmtnyjtrsxkhbrrdcgg18a", <1>
"time_string": "2019-04-24T16:43:49.316Z", <2>
"last_success": { <1>
"snapshot_name": "nightly-snap-2019.04.24-tmtnyjtrsxkhbrrdcgg18a", <2>
"time_string": "2019-04-24T16:43:49.316Z",
"time": 1556124229316
} ,
"next_execution": "2019-04-24T01:30:00.000Z", <3>
"next_execution_millis": 1556048160000
"last_failure": { <3>
"snapshot_name": "nightly-snap-2019.04.02-lohisb5ith2n8hxacaq3mw",
"time_string": "2019-04-02T01:30:00.000Z",
"time": 1556042030000,
"details": "{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [important]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"important\",\"index_uuid\":\"_na_\",\"index\":\"important\",\"stack_trace\":\"[important] IndexNotFoundException[no such index [important]]\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:762)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:714)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:670)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:163)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:142)\\n\\tat org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:102)\\n\\tat org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:280)\\n\\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\\n\\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687)\\n\\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310)\\n\\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210)\\n\\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)\\n\\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\\n\\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\\n\\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688)\\n\\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\\n\\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\\n\\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\\n\\tat java.base/java.lang.Thread.run(Thread.java:834)\\n\"}"
} ,
"next_execution": "2019-04-24T01:30:00.000Z", <4>
"next_execution_millis": 1556048160000
}
}
--------------------------------------------------
// TESTRESPONSE[skip:the presence of last_failure and last_success is asynchronous and will be present for users, but is untestable]

<1> The name of the last snapshot that was succesfully initiated by the policy
<2> When the snapshot was initiated
<3> When the policy will initiate the next snapshot

<1> Information about the last time the policy successfully initated a snapshot
<2> The name of the snapshot that was successfully initiated
<3> Unformation about the last time the policy failed to initiate a snapshot
<4> The next time the policy will execute
Loading

0 comments on commit ed31d0d

Please sign in to comment.