…10472)

) (elastic#110489)

…c#110492)

This PR was reviewed in elastic#109275

Block and Vector use a non-thread-safe RefCounted. Threads that increase
or decrease the references must have a happen-before relationship.
However, this order is not guaranteed in the enrich lookup for the
reference of selectedPositions. The driver can complete the
MergePositionsOperator, which decreases the reference count of
selectedPositions, while the finally block may also decrease it in a
separate thread. These actions occur without a defined happen-before
relationship.

Closes elastic#108532

These are the docs changes in relation to
elastic#108733

…cription of the index.codec setting (elastic#110468) (elastic#110515)

…lastic#110498)

* Always write empty role descriptor fields to index

* Add audit logging for bulk put role

…notes index. (elastic#110522)

Bit vector tests were failing in cases where an index has more than 1 shards.
For error cases when we expected a failure of the whole request, shards
with empty documents returned success and the whoel request unexpectedly returned 200.

Ensuring that index contains only 1 shard fixes these failures.

Closes elastic#110290, elastic#110291

This change makes the three-parameter constructor of ToPartial public so 
that EsqlNodeSubclassTests can pick it up properly.

Closes elastic#110310

…c#110550)

Currently, read_slm privilege grants access to get the ILM status,
and manage_slm grants access to start/stop ILM. This access will
be removed in the future, but needs to be deprecated before
removal. Add deprecation warning to the read_slm and manage_slm docs.

…110248) (elastic#110545)

* Initial commit; setup Gradle; start service

* initial commit

* minor cleanups, builds green; needs tests

* bug fixes; tested working embeddings & completion

* use custom json builder for embeddings request

* Ensure auto-close; fix forbidden API

* start of adding unit tests; abstraction layers

* adding additional tests; cleanups

* add requests unit tests

* all tests created

* fix cohere embeddings response

* fix cohere embeddings response

* fix lint

* better test coverage for secrets; inference client

* update thread-safe syncs; make dims/tokens + int

* add tests for dims and max tokens positive integer

* use requireNonNull;override settings type;cleanups

* use r/w lock for client cache

* remove client reference counting

* update locking in cache; client errors; noop doc

* remove extra block in internalGetOrCreateClient

* remove duplicate dependencies; cleanup

* add fxn to get default embeddings similarity

* use async calls to Amazon Bedrock; cleanups

* use Clock in cache; simplify locking; cleanups

* cleanups around executor; remove some instanceof

* cleanups; use EmbeddingRequestChunker

* move max chunk size to constants

* oof - swapped transport vers w/ master node req

* use XContent instead of Jackson JsonFactory

* remove gradle versions; do not allow dimensions

Currently, if a child request fails, we automatically trigger  cancellation
for ES|QL requests. This can result in TaskCancelledException being
collected by the RefCountingListener first, which then returns that
exception to the caller. For example, if we encounter a
CircuitBreakingException (429), we might incorrectly return a
TaskCancelledException (400) instead.

This change introduces the ComputeListener, a variant of
RefCountingListener, which selects the most appropriate exception to return
to the caller. I also integrated the following features into ComputeListener to
simplify ComputeService:

- Automatic cancellation of sub-tasks on failure.
- Collection of profiles from sub-tasks.
- Collection of response headers from sub-tasks.

…10567)

Adding `parent task was cancelled [test cancel]` to the list 
of allowed cancellation messages.

…sue (elastic#110523) (elastic#110575)

…10553) (elastic#110588)

…10589) (elastic#110608)

closes elastic#110357

With the loosening of what is considered a unit vector, we need to
ensure we only normalize for equality checking if the query vector is
indeed not a unit vector.

(cherry picked from commit fd790ff)

ComputeResponse from old nodes may have a null value instead of an empty
list for profiles.

Relates elastic#110400 Closes elastic#110591

Currently, we incorrectly remove the `@timestamp` attribute from
the EsRelation when translating metric aggregates.

… (elastic#110622)

(cherry picked from commit fbcde9c)

…#110592) (elastic#110624)

Fixes error using the Amazon Bedrock service with a large 
input that was chunked.

Backport of:  * elastic#110599

CCS tests could split the vectors over any number of shards. Through
empirical testing, I determined this commits values work to provide the
expected order, even if they are not all part of the same shard.

quantization can have weird behaviors when there are uniform values,
just like this test does.

closes elastic#109978

(cherry picked from commit 9dbe97b)

Partial backport from elastic#110586

Just the Max fix and an extra test for it.

(cherry picked from commit 1b6d44b)

Co-authored-by: David Kyle <[email protected]>

….xcore (elastic#110565) (elastic#110642)

The types from com.nimbusds.jwt are almost not needed in x-pack/plugin/core.
They're only needed in module org.elasticsearch.security, x-pack:plugin:security project.

…0670)

We've already removed Windows-specific BWC jobs in our periodic
pipelines. They shouldn't behave differently and are very prone to
timeouts so let's just remove them from pull requests when the
`test-windows` label is added.

…r engine is paused (elastic#110061) (elastic#110674)

elastic#110675)

Co-authored-by: Joe Gallo <[email protected]>

…elastic#110666) (elastic#110679)

…#110651) (elastic#110683)

This is a follow-up from
elastic#102456

…lastic#110656) (elastic#110684)

…xt actions have been consumed before tearDown (elastic#110685)

… (elastic#110703)

…ic#110730) (elastic#110731)

Ensures that the value provided by the `runtime.java` system property,
or `RUNTIME_JAVA_HOME` environment variable is respected, even when it
is the same as the current `JAVA_HOME`. The previous logic had
`isRuntimeJavaHomeSet` return `false` when the "requested" Java was the
same as the current one. This isn't strictly correct, as the behavior
when Java home is unset is to use the bundled JDK. The result was that
passing `-Druntime.java=17` when the Gradle daemon was also using Java
17 was to execute tests with the bundled JDK, which could be something
entirely different.

…10611)

* Document new settings
* Mention agg allow list in scripting security doc

…lastic#110707) (elastic#110756)

* Fix issue with noMatchSize for plain highlighter

* Update docs/changelog/110707.yaml

…astic#110760)

👋 howdy, team! Mini doc update to include default settings for `cluster.blocks.*` settings.

) (elastic#110745)

(cherry picked from commit 816cedc)

# Conflicts:
#	qa/packaging/build.gradle

…astic#110749) (elastic#110775)

… (elastic#110758)

Resolves elastic#104323

This fixes and adds tests for the first of the two bullets in the linked
issue.  `ExpressionBuilder#visitIntegerValue` will attempt to parse a
string as an integral value, and return a Literal of the appropriate
type.  The actual parsing happens in `StringUtils#parseIntegral`.  That
function has special handling for values that are larger than
`Long.MAX_VALUE` where it attempts to turn them into unsigned longs, and
if the number is still out of range, throw `InvalidArgumentException`.
`ExpressionBuilder` catches that `InvalidArgumentException` and tries to
parse a `double` instead.  If, on the other hand, the value is smaller
than `Long.MIN_VALUE`, `StringUtils` never enters the unsigned long path
and just calls `intValueExact`, which throws `ArithmeticException`.
This PR solves the issue by catching that `ArithmeticException` and
rethrowing it as an `InvalidArgumentException`.

elastic#110352) (elastic#110782)

Min/max range for the event.ingested timestamp field (part of Elastic Common
Schema) was added to IndexMetadata in cluster state for searchable snapshots
in elastic#106252.

This commit modifies the search coordinator to rewrite searches to MatchNone
if the query searches a range of event.ingested that, from the min/max range
in cluster state, is known to not overlap. This is the same behavior we currently
have for the @timestamp field.

…astic#110812)

We updated to use develocity plugin a while ago but still used
a deprecated api in the build complete logic we have running on ci

… API (elastic#110594) (elastic#110832)

* Add Amazon Bedrock Inference API to docs

* fix example errors

* update semantic search tutorial; add changelog

* fix typo

* fix error; accept suggestions

…lDependencies (elastic#110828) (elastic#110834)

…lastic#110839)

…ce API` (elastic#110835) (elastic#110865)

* [DOCS] Add note about ml model 502 timeout

* Add note to API ref

)

So that the only expected disk write at the point of the assertion is from the bulk request. And not from the asynchronous runnable of updateDanglingIndicesInfo().

Fixes elastic#110551

…110682)

Co-authored-by: Elastic Machine <[email protected]>

* [ESQL] Count_distinct(_source) should return a 400 (elastic#110824)

Resolves
[elastic#105240](elastic#105240)

Count_distinct doesn't work on source, but the type resolution was
allowing that through.  This resulted in a 500 layer deeper in the
aggregations code.  This PR fixes the 500 error by correctly failing
during type resolution.

* Even hand-backporting, I messed up the capabilities file

* one more

… rewrites (elastic#110352) (elastic#110782)" (elastic#110883)

This reverts commit 9092394.

elastic#110889)

The runtime classpath of `ThirdPartyAuditTask` was not being tracked as
an input which can cause issues with task output caching and build
avoidance.

Native preallocation has several issues, introduced in a refactoring for
8.13. First, the native allocator is never even tried, it always decides
to fall back to the Java setLength method. Second, the stat method did
not work correctly on all systems, see elastic#110807. This commit fixes
native preallocate to properly execute on Linux, as well as MacOS. It
also adds direct tests of preallocation.

Note that this is meant as a bugfix for 8.15, so as minimal a change as
possible is made here. The code has completely changed in main. Some
things like the new test and fixes for macos will be forward ported to
main, but I did not want to make larger changes in a bugfix.

* Fix bug in union-types with type-casting in grouping key of STATS (elastic#110476)

* Allow auto-generated type-cast fields in CsvTests

This allows, for example, a csv-spec test result header like `client_ip::ip:ip`, which is generated with a command like `STATS count=count(*) BY client_ip::ip`

It is also a small cleanup of the header parsing code, since it was using Strings.split() in an odd way.

* Fix bug in union-types with type-casting in grouping key of STATS

* Update docs/changelog/110476.yaml

* Added casting_operator required capability

Using the new `::` syntax requires disabling support for older versions in multi-cluster tests.

* Added more tests for inline stats over long/datetime

* Trying to fix the STATS...STATS bug

This makes two changes:

* Keeps the Alias in the aggs.aggregates from the grouping key, so that ReplaceStatsNestedExpressionWithEval still works
* Adds explicit support for union-types conversion at grouping key loading in the ordinalGroupingOperatorFactory

Neither fix the particular edge case, but do seem correct

* Added EsqlCapability for this change

So that mixed cluster tests don't fail these new queries.

* Fix InsertFieldExtract for union types

Union types require a FieldExtractExec to be performed first thing at
the bottom of local physical plans.

In queries like
```
  from testidx*
  | eval x = to_string(client_ip)
  | stats c = count(*) by x
  | keep c
```
The `stats` has the grouping `x` but the aggregates get pruned to just
`c`. In cases like this, we did not insert a FieldExtractExec, which
this fixes.

* Revert query that previously failed

With Alex's fix, this query now passes.

* Revert integration of union-types to ordinals aggregator

This is because we have not found a test case that actually demonstrates this is necessary.

* More tests that would fail without the latest fix

* Correct code style

* Fix failing case when aggregating on union-type with invalid grouping key

* Capabilities restrictions on the new YML tests

* Update docs/changelog/110476.yaml

---------

Co-authored-by: Alexander Spies <[email protected]>

* An alternative approach to supporting union-types on stats grouping field (elastic#110600)

* Added union-types field extration to ordinals aggregation

* Revert previous approach to getting union-types working in aggregations

Where the grouping field is erased by later commands, like a subsequent stats.
Instead we include union-type supports in the ordinals aggregation and mark the block loader as not supporting ordinals.

* Fix union-types when aggregating on inline conversion function (elastic#110652)

A query like:

```
FROM sample_data, sample_data_str
| STATS count=count(*) BY client_ip = TO_IP(client_ip)
| SORT count DESC, client_ip ASC
| KEEP count, client_ip
```

Failed due to unresolved aggregates from the union-type in the grouping key

* Fix for union-types for multiple columns with the same name (elastic#110793)

* Make union types use unique attribute names

* Cleanup leftover

* Added failing test and final fix to EsRelation

* Implement FieldAttribute.fieldName()

* Fix tests

* Refactor

* Do not ignore union typed field's parent

* Fix important typo

D'oh

* Mute unrelated (part of) test

* Move capability to better location

* Fix analyzer tests

* multi-node tests with an earlier version of union-types (before this change) fail

* Add capability to remaining failing tests

* Remove variable

* Add more complex test

* Consolidate union type cleanup rules

* Add 3 more required_capability's to make CI happy

* Update caps for union type subfield yaml tests

* Update docs/changelog/110793.yaml

* Refined changelog text

* Mute BWC for 8.15.0 for failing YAML tests

* union_types_remove_fields for all 160_union_types tests

The tests fail spordically, so safer to mute the entire suite.

---------

Co-authored-by: Alexander Spies <[email protected]>

---------

Co-authored-by: Alexander Spies <[email protected]>

…#110449) (elastic#110910)

* (Doc+) Error "number of documents in the index can't exceed"

👋 howdy, team! 

This adds resolution outline for error ... which induces ongoing, lowkey support
```
Number of documents in the index can't exceed [2147483519]
```

* feedback

* feedback



* feedback




* feedback

* feedback

* Test change to address docs check failure

* Revert test change

* Test docs check

---------

Co-authored-by: David Turner <[email protected]>
Co-authored-by: Liam Thompson <[email protected]>

* Union types documentation

* Try remove asciidoc error

* Another attempt

* Using literal block

* Nicer formatting

* Remove partintro

* Small refinements

* Edits for clarity and style

---------

Co-authored-by: Marci W <[email protected]>

Backports the docs changes from elastic#110911

…lastic#110710) (elastic#110924)

Non-master-eligible nodes that are already part of a cluster when the master is upgraded don't re-join the cluster, so their cluster features never get updated. This adds a cluster listener that spots this occurring, and manually gets the node's features with a new transport action and updates the cluster state after the fact.

Fixing this case for now by enforcing unpooled to plug the leak, this
needs a little more work to function well pooled.

…lastic#110948

…lastic#110966)

* ESQL: Validate unique plan attribute names (elastic#110488)

* Enforce an invariant in our dependency checker so that logical plans never have duplicate output attribute names or ids.
* Fix ROW to not produce columns with duplicate names.
* Fix ResolveUnionTypes to not create multiple synthetic field attributes for the same union type.
* Add tests for commands using the same column name more than once.
* Update docs w.r.t. how commands behave if they are used with duplicate column names.

(cherry picked from commit da53921)

# Conflicts:
#	x-pack/plugin/esql/qa/testFixtures/src/main/resources/stats.csv-spec
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/OptimizerRules.java
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Rename.java

* Remove unrelated csv tests

These slipped in via merge conflicts.

)

Calling Rename.output() previously returned wrong results.

Since elastic#110488, instead it throws an IllegalStateException. That leads to test failures in the EsqlNodeSubclassTests because e.g. MvExpandExec and FieldExtractExec eagerly calls .output() on its child when it's being constructed, and the child can be a fragment containing a Rename.

(cherry picked from commit 7df1b06)

# Conflicts:
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Rename.java

…lastic#110988)

This adds minimal docs  around how to the new logs index mode for data
streams (most common use case). This is minimal because logs index mode
is still in tech preview. Minimal docs should allow any interested users
to experiment with the new logs index mode.

…elastic#111000)

Co-authored-by: Adam Demjen <[email protected]>

…tes (elastic#110938) (elastic#111031)

* Inject `host.name` field without relying on (component) templates (elastic#110938)

We do not want to rely on templates or component templates to include
the host.name field in indices using LogsDB. The host.name field is a field
we sort on by default when LogsDB is used. As a result, we just inject it
by default, the same way we do for the @timestamp field. This prevents
sorting errors due to missing host.name field in mappings.

The host.name is a keyword field and depending on the value of subobjects it will
be mapped as a name keyword nested inside a host or as a flat host.name keyword.
We also include ignore_above as we normally do for keywords in observability mappings.

* Enable missing hostname test

Closes elastic#111023

… (elastic#110687)

Update stack monitoring template for .monitoring-beats-mb to include latest apm-server monitoring metrics. All stack monitoring apm-server metrics references in kibana should be intact. To avoid breaking stack monitoring UI, although beat.stats.apm_server.server.response.errors.concurrency is unused and is not present in apm-server stats, it is manually kept in the mapping.

(cherry picked from commit 2fb6c80)

elastic#110983)

Co-authored-by: Najwa Harif <[email protected]>

…tic#111057)

Use segment ordinals when possible to collect zero document buckets

…viders (elastic#110844) (elastic#111077)

Co-authored-by: Keith Massey <[email protected]>

…#111087)

Correct force merge disk space requirements

(cherry picked from commit 548aea5)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/esql/qa/testFixtures/src/main/resources/stats.csv-spec
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

(cherry picked from commit cb0ecb8)

The preallocate tests assumed that preallocation was using the fallback implementation which calls setLength on Windows. However, that fallback only happens inside the SharedBytes class, so windows doesn't actually do anything when tryPreallocate is called. This commit skips the test on windows.

closes elastic#110948

…#111098)

… (elastic#111110)

With elastic#110109 i accidentally added local files to the repo.
This removes them again.

Co-authored-by: Keith Massey <[email protected]>

…elastic#110343

…de.RollupIT elastic#111142

…asticsearch.multi_node.GlobalCheckpointSyncActionIT elastic#111124