Put together some example script contexts for scripted fields. This is
almost certainly wrong, but it gets us something to think about.

This reworks the script contexts to support multiple values for each
hit. Just like before, these are almost certainly not the final
implementation, but they give us something to iterate on.

Painless extensions work differently now.

Uses a temporary hack to allow us to use painless in unit tests.

This is the first draft implementation of scripted field of type keyword. Still lots of TODOs to address, this PR targets a feature branch.

This adds what I think is just about the simplest possible `term` query
implementation for `keyword` script fields and wires it into the field
mapper that we build for them.

Relates to #59332

Adds tests for the `keyword` typed `script` field's fielddata
implementation. One unit test which runs quickly and an integretion test
for fetching and an integration test for the `terms` agg.

This adds the `exits` and `terms` query for the `keyword` style of
scripted field.

Adds the remaining queries for scripted keyword fields and handles a few
leftover TODOs, mostly around `hashCode` and `equals`.

This updates a test so that runtime fields depend on one another and
fixes support for it.

Instead of relying no createComponents to get the script service, we rely on the query shard context supplier that's available in the parser context, yet we only carry around as callback that allows us to compile the script

This will cause them to fail if you don't have permission to execute
expensive queries.

There seems to be no longer a way to require a field in mapper paramaters. This needs to be fixed upstream.

The error was calling toString on the parameter object. we need to instead print out the returned value by calling getValue

…utput (#59672)

Relates to #59332

This addresses a TODO around using the script params, which are now parsed from the mappings. It also expand existing tests to verify that params are carried around and accessible in script for both fielddata and queries.

Relates to #59332

This adds the `long` runtime type to `script` fields and implements doc
values, field data, the `term` and `exists` query.

This fixes the rendering of the `script` field in the `script` field
type. Originally we weren't rendering the `params` and such because we
couldn't parse them. But we can parse them now so we should render them.

This updates the integration tests to include params and adds an
integration test that fetches the mapping.

Relates to #59724

Adds the `terms` and `range` queries for `long` typed `script` fields.
It also fixes a bug in a few of `toString` tests for `keyword` typed
`script` fields.

Relates to #59332

Relates to #59332

The keyword scripted field should apply the keyword analyzer at all times, by providing it as part of TextSearchInfo to the base mapped field type class. That is because the field does not support normalization, and if we don't force the keyword analyzer the default search analyzer will get picked up which will lowercase the query etc.

Relates to #59332

This adds support for `double` typed `script` fields.

This replaces the fancy unit tests that we were doing to test our
painless integration with an extension to our existing integration
tests. The integration tests are "more real", shorter, and are a nice
step down the road of dropping our dependency on the painless plugin in
favor of a dependency only on it's SPI.

We have unit tests for all of these things and the other yml tests offer
good proof that these things work.

This standardizes the exception type thrown when attempting to build a
query on a script field that it doesn't support. It also replaces the
field type `[script]` in the error message with whatever the runtime type
is.

Now that we have good yml tests for runtime_script fields we don't need
painless in our unit tests at all. This replaces painless with a mock
script which runs more quickly and isolates us from painless's
internals.

Runtime fields depend on SearchLookup to be able to look up other fields and manipulate them. With this change we add a SearchLookup argument to the existing MappedFieldType#fielddataBuilder method. Additionally the use of runtime fields in index sorting is disallowed.

Relates to #59332

Adds support for `runtime_script` fields with `runtime_type: date`.
Doesn't add support for the `format` parameter to the mapper but *does*
support it on aggregations and the like. Its a start!

Most core types throw back an error if the `relation` on a `range` query
is `disjoint`. This make runtime fields do the same.

Adds `format` and `locale` support to `runtime_script` fields,
specifically those with the `runtime_type` of `date`. Others
`runtime_type`s will return an error if provided with those parameters.

Multi fields and copy_to are not supported by the runtime script field mapper. They are now rejected when provided.

Looks like we lost a method that I was using in a test when merging
master. Oops! This fixes the build and lets us move forward.

This implements the `ip` typed runtime fields. They share a fair bit
with `string` runtime fields but we represent them as a `BytesRef`
containing 128 bits so that the comparisons all happen in the same way
as Lucene's `InetAddressPoint`.

This makes parsing runtime fields much simpler.

We'd initially resisted doing this but all of our other integration options
proved more troublesome.

Adds the `x-pack/plugin/runtime-fields/qa/rest` project to run core's
integration tests against an index with `runtime_scipt` fields. This
works by modifying the test configuration on load to replace supported
field types with `runtime_script`.

Relates to #59332

This breaks apart the a test for the `terms` aggregation into one that
work for runtime fields and one that doesn't.

This implements the distance_feature for `date` valued `runtime_script`s.
This produces the same numbers running against an indexed date, but it
doesn't have the same performance characteristics at all. Which is normal
for `runtime_script`s`. But `distance_feature` against an indexes fields
does a lot of work to refine the query as it goes, limiting the number of
documents that it has to visit. We can't do that because we don't have an
index. So we just spit out the same numbers and hope it is good enough.

This "borrows" 150 more tests from core for runtime fields, extending
the work done in #60931. More precisely, it adds a template to every
test that forces dynamic mapping updates to build runtime fields where
possible. In particular, `long` and `double` field are created as
runtime fields. `string`-typed fields are mimick the out of the box
behavior and create a top level `text` field with a `.keyword`
multi-field, but this `keyword` multi-field executes a script and loads
from source.

Add `boost` to a scripted-`date`'s `distance_feature` query's equals
method.

This adds `boolean` values `runtime_script` fields.

This adds a `parse` method to `date` valued `runtime_script`s that
converts any `Object` into a `long` using the same mechanism that
indexing a `date` uses.

We also use this new method to expand our test that run's core's
integration tests against runtime fields to `date` values fields.
Previously we couldn't do that because we didn't have a consistent way
to parse the field.

Fixes `doc['ip'].value` when `ip` is a script field. It didn't work
properly before because it wasn't whitelisted. We can't whitelist it
easilly because it was over in a module rather than core. This fixes
that by sharing more code with core that *is* whitelisted.

This excuses script fields from the script compilation rate limiting.
It'd be fairly sad to prevent mapping updates because of the rate limit.
And we don't expect folks to add a zillion fields. On the other hand,
once we allow these scripts on the `_search` request we might indeed
want them to be considered in the limit. But we don't support that yet
and we can deal with that when we get there.

This opts all of the script contexts used by `runtime_script` our of the
compilation rate limiting because it'd be lame to prevent a mapping
update because we can't compile the script.

* First crack at rewriting the CCR introduction.

* Emphasizing Kibana in configuring CCR (part one).

* Many more edits, plus new files.

* Fixing test case.

* Removing overview page and consolidating that information in the main page.

* Adding redirects for moved and deleted pages.

* Removing, consolidating, and adding redirects.

* Fixing duplicate ID in redirects and removing outdated reference.

* Adding test case and steps for recreating a follower index.

* Adding steps for managing CCR tasks in Kibana.

* Adding tasks for managing auto-follow patterns.

* Fixing glossary link.

* Fixing glossary link, again.

* Updating the upgrade information and other stuff.

* Apply suggestions from code review

* Incorporating review feedback.

* Adding more edits.

* Fixing link reference.

* Adding use cases for #59812.

* Incorporating feedback from reviewers.

* Apply suggestions from code review

* Incorporating more review comments.

* Condensing some of the steps for accessing Kibana.

* Incorporating small changes from reviewers.

Co-authored-by: debadair <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>

There is a corner case here in which during partial snapshot the index is
deleted right between starting the snapshot in the CS and the data node getting to work
on it, causing the data node the fail that shard snapshot and making the snapshot `PARTIAL`.

Closes #61208

We only work with heap byte buffers at this point and those we can and do unwrap the
`byte[]` ourselves and use `BytesArray` instead of a needless level of indirection via `ByteBuffer`.

…lity. (#61248)

This PR resurrects support for building Docker images based on one of Red Hat's UBI images. It also adds support for
running the existing Docker tests against the image. The image is named `elasticsearch-ubi8:<version>`.

I also changed the Docker build file uses enums instead strings in a lot of places, for added rigour.

This can remain package private

This replaces the `ESSingleNodeTestCase` with `ESTestCase` in runtime
fields, proving that we can use `ESTestCase` to test field mappers.

This replaces the value collection method in `double` valued runtime
script fields with simply returning a `double[]`. Painless has some
"convert" features that allow us to define automatic conversions from
things like `double` and `Collection` into `double[]` so we use those.

This runs the async search and graph tests against runtime fields.

This reverts commit 6d8170c. We've
decided to stick with the `value` style functions for runtime fields.

This standardizes that name for all of the "emit value" methods used by
runtime field scripts to `emitValue`. To make dates possible this adds a
a `toEpochMilli` method to convert from `TemporalAccessor` into millis
since epoch.

Possilby after #61457, the priority we use for the index template that converts core yaml tests to use runtime fields is already taken, which causes an error as there ma be multiple matching templates with same priority.