Merge remote-tracking branch 'origin/main' into updates_with_new_inte…

…rface

docs/changelog/110928.yaml

@@ -0,0 +1,5 @@
+pr: 110928
+summary: Dense vector field types updatable for int4
+area: Vector Search
+type: enhancement
+issues: []

docs/changelog/111015.yaml

@@ -0,0 +1,15 @@
+pr: 111015
+summary: Always allow rebalancing by default
+area: Allocation
+type: enhancement
+issues: []
+highlight:
+  title: Always allow rebalancing by default
+  body: |-
+    In earlier versions of {es} the `cluster.routing.allocation.allow_rebalance` setting defaults to
+    `indices_all_active` which blocks all rebalancing moves while the cluster is in `yellow` or `red` health. This was
+    appropriate for the legacy allocator which might do too many rebalancing moves otherwise. Today's allocator has
+    better support for rebalancing a cluster that is not in `green` health, and expects to be able to rebalance some
+    shards away from over-full nodes to avoid allocating shards to undesirable locations in the first place. From
+    version 8.16 `allow_rebalance` setting defaults to `always` unless the legacy allocator is explicitly enabled.
+  notable: true

docs/reference/modules/cluster/shards_allocation.asciidoc

@@ -98,9 +98,9 @@ the cluster:
 Specify when shard rebalancing is allowed:
 
 
-* `always` -                    Always allow rebalancing.
+* `always` -                    (default) Always allow rebalancing.
 * `indices_primaries_active` -  Only when all primaries in the cluster are allocated.
-* `indices_all_active` -        (default) Only when all shards (primaries and replicas) in the cluster are allocated.
+* `indices_all_active` -        Only when all shards (primaries and replicas) in the cluster are allocated.
 --
 
 `cluster.routing.rebalance.enable`::

docs/reference/release-notes/8.12.0.asciidoc

@@ -14,6 +14,13 @@ there are deleted documents in the segments, quantiles may fail to build and pre
 
 This issue is fixed in 8.12.1.
 
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
+information about the new functionality of these upgraded nodes may not be registered properly with the master node.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
+If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
+To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
+are upgraded. This issue is fixed in 8.15.0.
+
 [[breaking-8.12.0]]
 [float]
 === Breaking changes

docs/reference/release-notes/8.12.1.asciidoc

@@ -3,6 +3,16 @@
 
 Also see <<breaking-changes-8.12,Breaking changes in 8.12>>.
 
+[[known-issues-8.12.1]]
+[float]
+=== Known issues
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
+information about the new functionality of these upgraded nodes may not be registered properly with the master node.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
+If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
+To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
+are upgraded. This issue is fixed in 8.15.0.
+
 [[bug-8.12.1]]
 [float]
 === Bug fixes

docs/reference/release-notes/8.12.2.asciidoc

@@ -3,6 +3,16 @@
 
 Also see <<breaking-changes-8.12,Breaking changes in 8.12>>.
 
+[[known-issues-8.12.2]]
+[float]
+=== Known issues
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
+information about the new functionality of these upgraded nodes may not be registered properly with the master node.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
+If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
+To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
+are upgraded. This issue is fixed in 8.15.0.
+
 [[bug-8.12.2]]
 [float]
 === Bug fixes

docs/reference/release-notes/8.13.0.asciidoc

@@ -21,12 +21,12 @@ This affects clusters running version 8.10 or later, with an active downsampling
 https://www.elastic.co/guide/en/elasticsearch/reference/current/downsampling-ilm.html[configuration]
 or a configuration that was activated at some point since upgrading to version 8.10 or later.
 
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.13.1.asciidoc

@@ -6,12 +6,12 @@ Also see <<breaking-changes-8.13,Breaking changes in 8.13>>.
 [[known-issues-8.13.1]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.13.2.asciidoc

@@ -6,12 +6,12 @@ Also see <<breaking-changes-8.13,Breaking changes in 8.13>>.
 [[known-issues-8.13.2]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.13.3.asciidoc

@@ -13,12 +13,12 @@ SQL::
 [[known-issues-8.13.3]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.13.4.asciidoc

@@ -6,12 +6,12 @@ Also see <<breaking-changes-8.13,Breaking changes in 8.13>>.
 [[known-issues-8.13.4]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.14.0.asciidoc

@@ -15,12 +15,12 @@ Security::
 [[known-issues-8.14.0]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.14.1.asciidoc

@@ -7,12 +7,12 @@ Also see <<breaking-changes-8.14,Breaking changes in 8.14>>.
 [[known-issues-8.14.1]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.14.2.asciidoc

@@ -6,12 +6,12 @@ Also see <<breaking-changes-8.14,Breaking changes in 8.14>>.
 [[known-issues-8.14.2]]
 [float]
 === Known issues
-* When upgrading clusters from version 8.12.2 or earlier, if your cluster contains non-master-eligible nodes,
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
 information about the new functionality of these upgraded nodes may not be registered properly with the master node.
-This can lead to some new functionality added since 8.13.0 not being accessible on the upgraded cluster.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
 If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
 To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
-are upgraded.
+are upgraded. This issue is fixed in 8.15.0.
 
 * The `pytorch_inference` process used to run Machine Learning models can consume large amounts of memory.
 In environments where the available memory is limited, the OS Out of Memory Killer will kill the `pytorch_inference`

docs/reference/release-notes/8.14.3.asciidoc

@@ -3,6 +3,16 @@
 
 Also see <<breaking-changes-8.14,Breaking changes in 8.14>>.
 
+[[known-issues-8.14.3]]
+[float]
+=== Known issues
+* When upgrading clusters from version 8.11.4 or earlier, if your cluster contains non-master-eligible nodes,
+information about the new functionality of these upgraded nodes may not be registered properly with the master node.
+This can lead to some new functionality added since 8.12.0 not being accessible on the upgraded cluster.
+If your cluster is running on ECK 2.12.1 and above, this may cause problems with finalizing the upgrade.
+To resolve this issue, perform a rolling restart on the non-master-eligible nodes once all Elasticsearch nodes
+are upgraded. This issue is fixed in 8.15.0.
+
 [[bug-8.14.3]]
 [float]
 === Bug fixes

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/logsdb/10_settings.yml

@@ -114,42 +114,44 @@ using default timestamp field mapping:
               message:
                 type: text
 
----
-missing hostname field:
-  - requires:
-      test_runner_features: [ capabilities ]
-      capabilities:
-        - method: PUT
-          path: /{index}
-          capabilities: [ logs_index_mode ]
-      reason: "Support for 'logs' index mode capability required"
-
-  - do:
-      catch: bad_request
-      indices.create:
-        index: test-hostname-missing
-        body:
-          settings:
-            index:
-              mode: logs
-              number_of_replicas: 0
-              number_of_shards: 2
-          mappings:
-            properties:
-              "@timestamp":
-                type: date
-              agent_id:
-                type: keyword
-              process_id:
-                type: integer
-              http_method:
-                type: keyword
-              message:
-                type: text
-
-  - match: { error.root_cause.0.type: "illegal_argument_exception" }
-  - match: { error.type: "illegal_argument_exception" }
-  - match: { error.reason: "unknown index sort field:[host.name]" }
+#---
+#missing hostname field:
+#  - requires:
+#      test_runner_features: [ capabilities ]
+#      capabilities:
+#        - method: PUT
+#          path: /{index}
+#          capabilities: [ logs_index_mode ]
+#      reason: "Support for 'logs' index mode capability required"
+#
+#  - do:
+#      indices.create:
+#        index: test-hostname-missing
+#        body:
+#          settings:
+#            index:
+#              mode: logs
+#              number_of_replicas: 0
+#              number_of_shards: 2
+#          mappings:
+#            properties:
+#              "@timestamp":
+#                type: date
+#              agent_id:
+#                type: keyword
+#              process_id:
+#                type: integer
+#              http_method:
+#                type: keyword
+#              message:
+#                type: text
+#
+#  - do:
+#      indices.get_settings:
+#        index: test-hostname-missing
+#
+#  - is_true: test-hostname-missing
+#  - match: { test-hostname-missing.settings.index.mode: "logs" }
 
 ---
 missing sort field: