Merge branch 'main' into BytesRefBlockHash

.buildkite/pull-requests.json

@@ -8,6 +8,7 @@
         "admin",
         "write"
       ],
+      "allowed_list": ["elastic-renovate-prod[bot]"],
       "set_commit_status": false,
       "build_on_commit": true,
       "build_on_comment": true,

build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/DockerBase.java

@@ -27,7 +27,7 @@ public enum DockerBase {
     // Chainguard based wolfi image with latest jdk
     // This is usually updated via renovatebot
     // spotless:off
-    WOLFI("docker.elastic.co/wolfi/chainguard-base:latest@sha256:90888b190da54062f67f3fef1372eb0ae7d81ea55f5a1f56d748b13e4853d984",
+    WOLFI("docker.elastic.co/wolfi/chainguard-base:latest@sha256:277ebb42c458ef39cb4028f9204f0b3d51d8cd628ea737a65696a1143c3e42fe",
         "-wolfi",
         "apk"
     ),

docs/changelog/113374.yaml

@@ -0,0 +1,5 @@
+pr: 113374
+summary: Add ESQL match function
+area: ES|QL
+type: feature
+issues: []

docs/changelog/113735.yaml

@@ -0,0 +1,28 @@
+pr: 113735
+summary: "ESQL: Introduce per agg filter"
+area: ES|QL
+type: feature
+issues: []
+highlight:
+  title: "ESQL: Introduce per agg filter"
+  body: |-
+    Add support for aggregation scoped filters that work dynamically on the
+    data in each group.
+    
+    [source,esql]
+    ----
+    | STATS success = COUNT(*) WHERE 200 <= code AND code < 300,
+            redirect = COUNT(*) WHERE 300 <= code AND code < 400,
+            client_err = COUNT(*) WHERE 400 <= code AND code < 500,
+            server_err = COUNT(*) WHERE 500 <= code AND code < 600,
+            total_count = COUNT(*)
+    ----
+    
+    Implementation wise, the base AggregateFunction has been extended to 
+    allow a filter to be passed on. This is required to incorporate the
+    filter as part of the aggregate equality/identity which would fail with 
+    the filter as an external component.
+    As part of the process, the serialization for the existing aggregations
+    had to be fixed so AggregateFunction implementations so that it
+    delegates to their parent first.
+  notable: true

docs/changelog/114168.yaml

@@ -0,0 +1,5 @@
+pr: 114168
+summary: Add a query rules tester API call
+area: Relevance
+type: enhancement
+issues: []

docs/changelog/114407.yaml

@@ -0,0 +1,6 @@
+pr: 114407
+summary: Fix synthetic source handling for `bit` type in `dense_vector` field
+area: Search
+type: bug
+issues:
+  - 114402

docs/changelog/114439.yaml

@@ -0,0 +1,5 @@
+pr: 114439
+summary: Adding new bbq index types behind a feature flag
+area: Vector Search
+type: feature
+issues: []

docs/changelog/114453.yaml

@@ -0,0 +1,5 @@
+pr: 114453
+summary: Switch default chunking strategy to sentence
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/114482.yaml

@@ -0,0 +1,5 @@
+pr: 114482
+summary: Remove snapshot build restriction for match and qstr functions
+area: ES|QL
+type: feature
+issues: []

docs/changelog/114549.yaml

@@ -0,0 +1,5 @@
+pr: 114549
+summary: Send mid-stream errors to users
+area: Machine Learning
+type: bug
+issues: []

docs/changelog/114596.yaml

@@ -0,0 +1,5 @@
+pr: 114596
+summary: Stream Google Completion
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/114623.yaml

@@ -0,0 +1,5 @@
+pr: 114623
+summary: Preserve thread context when waiting for segment generation in RTG
+area: CRUD
+type: bug
+issues: []

docs/changelog/114636.yaml

@@ -0,0 +1,5 @@
+pr: 114636
+summary: Dynamically get of num allocations
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/114638.yaml

@@ -0,0 +1,7 @@
+pr: 114638
+summary: "ES|QL: Restrict sorting for `_source` and counter field types"
+area: ES|QL
+type: bug
+issues:
+ - 114423
+ - 111976

docs/changelog/114683.yaml

@@ -0,0 +1,5 @@
+pr: 114683
+summary: Default inference endpoint for the multilingual-e5-small model
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/114715.yaml

@@ -0,0 +1,5 @@
+pr: 114715
+summary: Ignore unrecognized openai sse fields
+area: Machine Learning
+type: bug
+issues: []

docs/changelog/114719.yaml

@@ -0,0 +1,5 @@
+pr: 114719
+summary: Wait for allocation on scale up
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/114732.yaml

@@ -0,0 +1,5 @@
+pr: 114732
+summary: Stream Bedrock Completion
+area: Machine Learning
+type: enhancement
+issues: []

docs/reference/data-streams/set-up-tsds.asciidoc

@@ -121,7 +121,8 @@ naming scheme].
 * Specify a mapping that defines your dimensions and metrics:
 
 ** One or more <<time-series-dimension,dimension fields>> with a `time_series_dimension` value of `true`.
-   At least one of these dimensions must be a plain `keyword` field.
+   Alternatively, one or more <<passthrough-dimensions, pass-through>> fields configured as dimension containers,
+   provided that they will contain at least one sub-field (mapped statically or dynamically).
 
 ** One or more <<time-series-metric,metric fields>>, marked using the `time_series_metric` mapping parameter.
 
@@ -203,10 +204,9 @@ DELETE _ilm/policy/my-weather-sensor-lifecycle-policy
 Documents in a TSDS must include:
 
 * A `@timestamp` field
-* One or more dimension fields. At least one dimension must be a `keyword` field
-that matches the `index.routing_path` index setting, if specified. If not specified
-explicitly, `index.routing_path` is set automatically to whichever mappings have
- `time_series_dimension` set to `true`.
+* One or more dimension fields. At least one dimension must match the `index.routing_path` index setting,
+if specified. If not specified explicitly, `index.routing_path` is set automatically to whichever mappings have
+`time_series_dimension` set to `true`.
 
 To automatically create your TSDS, submit an indexing request that
 targets the TSDS's name. This name must match one of your index template's
@@ -285,13 +285,12 @@ POST metrics-weather_sensors-dev/_rollover
 
 Configuring a TSDS via an index template that uses component templates is a bit more complicated.
 Typically with component templates mappings and settings get scattered across multiple component templates.
-When configuring the `index.mode` setting in a component template, the `index.routing_path` setting needs to
-be defined in the same component template. Additionally the fields mentioned in the `index.routing_path`
-also need to be defined in the same component template with the `time_series_dimension` attribute enabled.
+If the `index.routing_path` is defined, the fields it references need to be defined in the same component
+template with the `time_series_dimension` attribute enabled.
 
-The reasons for this is that each component template needs to be valid on its own and the time series index mode
-requires the `index.routing_path` setting. When configuring the  `index.mode` setting in an index template, the `index.routing_path` setting is configured automatically. It is derived  from
-the field mappings with `time_series_dimension` attribute enabled.
+The reasons for this is that each component template needs to be valid on its own. When configuring the
+`index.mode` setting in an index template, the `index.routing_path` setting is configured automatically.
+It is derived from the field mappings with `time_series_dimension` attribute enabled.
 
 [discrete]
 [[set-up-tsds-whats-next]]

docs/reference/data-streams/tsds.asciidoc

@@ -109,7 +109,10 @@ parameter:
 * <<number,`unsigned_long`>>
 * <<boolean,`boolean`>>
 
-For a flattened field, use the `time_series_dimensions` parameter to configure an array of fields as dimensions. For details refer to <<flattened-params,`flattened`>>.
+For a flattened field, use the `time_series_dimensions` parameter to configure an array of fields as dimensions.
+For details refer to <<flattened-params,`flattened`>>.
+
+Dimension definitions can be simplified through <<passthrough-dimensions, pass-through>> fields.
 
 [discrete]
 [[time-series-metric]]
@@ -294,12 +297,15 @@ When you create the matching index template for a TSDS, you must specify one or
 more dimensions in the `index.routing_path` setting. Each document in a TSDS
 must contain one or more dimensions that match the `index.routing_path` setting.
 
-Dimensions in the `index.routing_path` setting must be plain `keyword` fields.
 The `index.routing_path` setting accepts wildcard patterns (for example `dim.*`)
 and can dynamically match new fields. However, {es} will reject any mapping
-updates that add scripted, runtime, or non-dimension, non-`keyword` fields that
+updates that add scripted, runtime, or non-dimension fields that
 match the `index.routing_path` value.
 
+<<passthrough-dimensions, Pass-through>> fields may be configured
+as dimension containers. In this case, their sub-fields get included to the
+routing path automatically.
+
 TSDS documents don't support a custom `_routing` value. Similarly, you can't
 require a `_routing` value in mappings for a TSDS.
 

docs/reference/esql/processing-commands/where.asciidoc

@@ -5,6 +5,13 @@
 The `WHERE` processing command produces a table that contains all the rows from
 the input table for which the provided condition evaluates to `true`.
 
+[TIP]
+====
+In case of value exclusions, fields with `null` values will be excluded from search results. 
+In this context a `null` means either there is an explicit `null` value in the document or there is no value at all.
+For example: `WHERE field != "value"` will be interpreted as `WHERE field != "value" AND field IS NOT NULL`.
+====
+
 **Syntax**
 
 [source,esql]