Merge branch 'master' into zen2_replace_inmemorystate

CoordinatorTests now support restarts when GatewayMetaState is used

client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/RollupDocumentationIT.java

@@ -261,6 +261,14 @@ public void testStartRollupJob() throws Exception {
         } catch (Exception e) {
             // Swallow any exception, this test does not test actually cancelling.
         }
+        // stop job to prevent spamming exceptions on next start request
+        StopRollupJobRequest stopRequest = new StopRollupJobRequest(id);
+        stopRequest.waitForCompletion();
+        stopRequest.timeout(TimeValue.timeValueSeconds(10));
+
+        StopRollupJobResponse response = client.rollup().stopRollupJob(stopRequest, RequestOptions.DEFAULT);
+        assertTrue(response.isAcknowledged());
+
         // tag::rollup-start-job-execute-listener
         ActionListener<StartRollupJobResponse> listener = new ActionListener<StartRollupJobResponse>() {
             @Override
@@ -282,7 +290,8 @@ public void onFailure(Exception e) {
         assertTrue(latch.await(30L, TimeUnit.SECONDS));
 
         // stop job so it can correctly be deleted by the test teardown
-        rc.stopRollupJob(new StopRollupJobRequest(id), RequestOptions.DEFAULT);
+        response = rc.stopRollupJob(stopRequest, RequestOptions.DEFAULT);
+        assertTrue(response.isAcknowledged());
     }
 
     @SuppressWarnings("unused")

docs/plugins/analysis.asciidoc

@@ -58,6 +58,7 @@ A number of analysis plugins have been contributed by our community:
 * https://github.com/ofir123/elasticsearch-network-analysis[Network Addresses Analysis Plugin] (by Ofir123)
 * https://github.com/medcl/elasticsearch-analysis-string2int[String2Integer Analysis Plugin] (by Medcl)
 * https://github.com/ZarHenry96/elasticsearch-dandelion-plugin[Dandelion Analysis Plugin] (by ZarHenry96)
+* https://github.com/medcl/elasticsearch-analysis-stconvert[STConvert Analysis Plugin] (by Medcl)
 
 include::analysis-icu.asciidoc[]
 

docs/reference/docs.asciidoc

@@ -50,3 +50,5 @@ include::docs/termvectors.asciidoc[]
 include::docs/multi-termvectors.asciidoc[]
 
 include::docs/refresh.asciidoc[]
+
+include::docs/concurrency-control.asciidoc[]

docs/reference/docs/bulk.asciidoc

@@ -197,6 +197,17 @@ size for your particular workload.
 If using the HTTP API, make sure that the client does not send HTTP
 chunks, as this will slow things down.
 
+[float]
+[[bulk-optimistic-concurrency-control]]
+=== Optimistic Concurrency Control
+
+Each `index` and `delete` action within a bulk API call may include the
+`if_seq_no` and `if_primary_term` parameters in their respective action
+and meta data lines. The `if_seq_no` and `if_primary_term` parameters control
+how operations are executed, based on the last modification to existing
+documents. See <<optimistic-concurrency-control>> for more details.
+
+
 [float]
 [[bulk-versioning]]
 === Versioning

docs/reference/docs/concurrency-control.asciidoc

@@ -0,0 +1,114 @@
+[[optimistic-concurrency-control]]
+== Optimistic concurrency control
+
+Elasticsearch is distributed. When documents are created, updated, or deleted,
+the new version of the document has to be replicated to other nodes in the cluster. 
+Elasticsearch is also asynchronous and concurrent, meaning that these replication
+requests are sent in parallel, and may arrive at their destination out of sequence. 
+Elasticsearch needs a way of ensuring that an older version of a document never 
+overwrites a newer version.
+
+
+To ensure an older version of a document doesn't overwrite a newer version, every
+operation performed to a document is assigned a sequence number by the primary
+shard that coordinates that change. The sequence number is increased with each
+operation and thus newer operations are guaranteed to have a higher sequence
+number than older operations. Elasticsearch can then use the sequence number of
+operations to make sure a newer document version is never overridden by
+a change that has a smaller sequence number assigned to it.
+
+For example, the following indexing command will create a document and assign it
+an initial sequence number and primary term:
+
+[source,js]
+--------------------------------------------------
+PUT products/_doc/1567
+{
+    "product" : "r2d2",
+    "details" : "A resourceful astromech droid"
+}
+--------------------------------------------------
+// CONSOLE
+
+You can see the assigned sequence number and primary term in the 
+the `_seq_no` and `_primary_term` fields of the response:
+
+[source,js]
+--------------------------------------------------
+{
+    "_shards" : {
+        "total" : 2,
+        "failed" : 0,
+        "successful" : 1
+    },
+    "_index" : "products",
+    "_type" : "_doc",
+    "_id" : "1567",
+    "_version" : 1,
+    "_seq_no" : 362,
+    "_primary_term" : 2,
+    "result" : "created"
+}
+--------------------------------------------------
+// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 2/"_primary_term" : $body._primary_term/]
+
+
+Elasticsearch keeps tracks of the sequence number and primary of the last
+operation to have changed each of the document it stores. The sequence number
+and primary term are returned in the `_seq_no` and `_primary_term` fields in
+the response of the <<docs-get,GET API>>:
+
+[source,js]
+--------------------------------------------------
+GET products/_doc/1567
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+returns:
+
+[source,js]
+--------------------------------------------------
+{
+    "_index" : "products",
+    "_type" : "_doc",
+    "_id" : "1567",
+    "_version" : 1,
+    "_seq_no" : 362,
+    "_primary_term" : 2,
+    "found": true,
+    "_source" : {
+        "product" : "r2d2",
+        "details" : "A resourceful astromech droid"
+    }
+}
+--------------------------------------------------
+// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 2/"_primary_term" : $body._primary_term/]
+
+
+Note: The <<search-search,Search API>> can return the `_seq_no` and `_primary_term`
+for each search hit by requesting the `_seq_no` and `_primary_term` <<search-request-docvalue-fields,Doc Value Fields>>.
+
+The sequence number and the primary term uniquely identify a change. By noting down 
+the sequence number and primary term returned, you can make sure to only change the
+document if no other change was made to it since you retrieved it. This
+is done by setting the `if_seq_no` and `if_primary_term` parameters of either the
+<<docs-index_,Index API>> or the <<docs-delete,Delete API>>. 
+
+For example, the following indexing call will make sure to add a tag to the
+document without losing any potential change to the description or an addition
+of another tag by another API:
+
+[source,js]
+--------------------------------------------------
+PUT products/_doc/1567?if_seq_no=362&if_primary_term=2
+{
+    "product" : "r2d2",
+    "details" : "A resourceful astromech droid",
+    "tags": ["droid"]
+}
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+// TEST[catch: conflict]
+

docs/reference/docs/delete.asciidoc

@@ -35,6 +35,16 @@ The result of the above delete operation is:
 // TESTRESPONSE[s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]
 // TESTRESPONSE[s/"_seq_no" : 5/"_seq_no" : $body._seq_no/]
 
+[float]
+[[optimistic-concurrency-control-delete]]
+=== Optimistic concurrency control
+
+Delete operations can be made optional and only be performed if the last
+modification to the document was assigned the sequence number and primary 
+term specified by the `if_seq_no` and `if_primary_term` parameters. If a
+mismatch is detected, the operation will result in a `VersionConflictException`
+and a status code of 409. See <<optimistic-concurrency-control>> for more details. 
+
 [float]
 [[delete-versioning]]
 === Versioning

docs/reference/docs/index_.asciidoc

@@ -79,89 +79,6 @@ Automatic index creation can include a pattern based white/black list,
 for example, set `action.auto_create_index` to `+aaa*,-bbb*,+ccc*,-*` (+
 meaning allowed, and - meaning disallowed).
 
-[float]
-[[index-versioning]]
-=== Versioning
-
-Each indexed document is given a version number. The associated
-`version` number is returned as part of the response to the index API
-request. The index API optionally allows for
-http://en.wikipedia.org/wiki/Optimistic_concurrency_control[optimistic
-concurrency control] when the `version` parameter is specified. This
-will control the version of the document the operation is intended to be
-executed against. A good example of a use case for versioning is
-performing a transactional read-then-update. Specifying a `version` from
-the document initially read ensures no changes have happened in the
-meantime. For example:
-
-[source,js]
---------------------------------------------------
-PUT twitter/_doc/1?version=2
-{
-    "message" : "elasticsearch now has versioning support, double cool!"
-}
---------------------------------------------------
-// CONSOLE
-// TEST[continued]
-// TEST[catch: conflict]
-
-*NOTE:* versioning is completely real time, and is not affected by the
-near real time aspects of search operations. If no version is provided,
-then the operation is executed without any version checks.
-
-By default, internal versioning is used that starts at 1 and increments
-with each update, deletes included. Optionally, the version number can be
-supplemented with an external value (for example, if maintained in a
-database). To enable this functionality, `version_type` should be set to
-`external`. The value provided must be a numeric, long value greater or equal to 0,
-and less than around 9.2e+18. When using the external version type, instead
-of checking for a matching version number, the system checks to see if
-the version number passed to the index request is greater than the
-version of the currently stored document. If true, the document will be
-indexed and the new version number used. If the value provided is less
-than or equal to the stored document's version number, a version
-conflict will occur and the index operation will fail.
-
-WARNING: External versioning supports the value 0 as a valid version number.
-This allows the version to be in sync with an external versioning system
-where version numbers start from zero instead of one. It has the side effect
-that documents with version number equal to zero cannot neither be updated
-using the <<docs-update-by-query,Update-By-Query API>> nor be deleted
-using the <<docs-delete-by-query,Delete By Query API>> as long as their
-version number is equal to zero.
-
-A nice side effect is that there is no need to maintain strict ordering
-of async indexing operations executed as a result of changes to a source
-database, as long as version numbers from the source database are used.
-Even the simple case of updating the Elasticsearch index using data from
-a database is simplified if external versioning is used, as only the
-latest version will be used if the index operations are out of order for
-whatever reason.
-
-[float]
-==== Version types
-
-Next to the `internal` & `external` version types explained above, Elasticsearch
-also supports other types for specific use cases. Here is an overview of
-the different version types and their semantics.
-
-`internal`:: only index the document if the given version is identical to the version
-of the stored document.
-
-`external` or `external_gt`:: only index the document if the given version is strictly higher
-than the version of the stored document *or* if there is no existing document. The given
-version will be used as the new version and will be stored with the new document. The supplied
-version must be a non-negative long number.
-
-`external_gte`:: only index the document if the given version is *equal* or higher
-than the version of the stored document. If there is no existing document
-the operation will succeed as well. The given version will be used as the new version
-and will be stored with the new document. The supplied version must be a non-negative long number.
-
-*NOTE*: The `external_gte` version type is meant for special use cases and
-should be used with care. If used incorrectly, it can result in loss of data.
-There is another option, `force`, which is deprecated because it can cause
-primary and replica shards to diverge.
 
 [float]
 [[operation-type]]
@@ -238,6 +155,16 @@ The result of the above index operation is:
 --------------------------------------------------
 // TESTRESPONSE[s/W0tpsmIBdwcYyG50zbta/$body._id/ s/"successful" : 2/"successful" : 1/]
 
+[float]
+[[optimistic-concurrency-control-index]]
+=== Optimistic concurrency control
+
+Index operations can be made optional and only be performed if the last
+modification to the document was assigned the sequence number and primary 
+term specified by the `if_seq_no` and `if_primary_term` parameters. If a
+mismatch is detected, the operation will result in a `VersionConflictException`
+and a status code of 409. See <<optimistic-concurrency-control>> for more details. 
+
 [float]
 [[index-routing]]
 === Routing
@@ -380,3 +307,83 @@ PUT twitter/_doc/1?timeout=5m
 }
 --------------------------------------------------
 // CONSOLE
+
+[float]
+[[index-versioning]]
+=== Versioning
+
+Each indexed document is given a version number. By default, 
+internal versioning is used that starts at 1 and increments
+with each update, deletes included. Optionally, the version number can be
+set to an external value (for example, if maintained in a
+database). To enable this functionality, `version_type` should be set to
+`external`. The value provided must be a numeric, long value greater or equal to 0,
+and less than around 9.2e+18. 
+
+When using the external version type, the system checks to see if
+the version number passed to the index request is greater than the
+version of the currently stored document. If true, the document will be
+indexed and the new version number used. If the value provided is less
+than or equal to the stored document's version number, a version
+conflict will occur and the index operation will fail. For example:
+
+[source,js]
+--------------------------------------------------
+PUT twitter/_doc/1?version=2&version_type=external
+{
+    "message" : "elasticsearch now has versioning support, double cool!"
+}
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+*NOTE:* versioning is completely real time, and is not affected by the
+near real time aspects of search operations. If no version is provided,
+then the operation is executed without any version checks.
+
+The above will succeed since the the supplied version of 2 is higher than
+the current document version of 1. If the document was already updated
+and its version was set to 2 or higher, the indexing command will fail
+and result in a conflict (409 http status code).
+
+WARNING: External versioning supports the value 0 as a valid version number.
+This allows the version to be in sync with an external versioning system
+where version numbers start from zero instead of one. It has the side effect
+that documents with version number equal to zero can neither be updated
+using the <<docs-update-by-query,Update-By-Query API>> nor be deleted
+using the <<docs-delete-by-query,Delete By Query API>> as long as their
+version number is equal to zero.
+
+A nice side effect is that there is no need to maintain strict ordering
+of async indexing operations executed as a result of changes to a source
+database, as long as version numbers from the source database are used.
+Even the simple case of updating the Elasticsearch index using data from
+a database is simplified if external versioning is used, as only the
+latest version will be used if the index operations arrive out of order for
+whatever reason.
+
+[float]
+==== Version types
+
+Next to the `external` version type explained above, Elasticsearch
+also supports other types for specific use cases. Here is an overview of
+the different version types and their semantics.
+
+`internal`:: only index the document if the given version is identical to the version
+of the stored document.
+
+`external` or `external_gt`:: only index the document if the given version is strictly higher
+than the version of the stored document *or* if there is no existing document. The given
+version will be used as the new version and will be stored with the new document. The supplied
+version must be a non-negative long number.
+
+`external_gte`:: only index the document if the given version is *equal* or higher
+than the version of the stored document. If there is no existing document
+the operation will succeed as well. The given version will be used as the new version
+and will be stored with the new document. The supplied version must be a non-negative long number.
+
+*NOTE*: The `external_gte` version type is meant for special use cases and
+should be used with care. If used incorrectly, it can result in loss of data.
+There is another option, `force`, which is deprecated because it can cause
+primary and replica shards to diverge.
+

docs/reference/ilm/policy-definitions.asciidoc

@@ -93,6 +93,7 @@ The below list shows the actions which are available in each phase.
   - <<ilm-shrink-action,Shrink>>
 * Cold
   - <<ilm-allocate-action,Allocate>>
+  - <<ilm-freeze-action,Freeze>>
 * Delete
   - <<ilm-delete-action,Delete>>