From 303b0883f95888738d12fb1122e7bcc43c09f960 Mon Sep 17 00:00:00 2001 From: Craig Taverner Date: Fri, 3 Jun 2022 10:06:43 +0200 Subject: [PATCH] Generate release notes for v8.3.0 (#87294) * Generate release notes for v8.3.0 * [DOCS] Add 8.3 migration file to index * Fixed version number * Fix formatting of deprecation in 85326 * Use asciidoc format for deprecations * Extract static content from migration/index * This was just an enhancement * Nope, it was an upgrade * Added migration/index.asciidoc generation support (#87318) Including extracting static content from migration/index, so the template would be as light as possible. The reason for this work is because the gradle task `generateReleaseNotes` was not correctly adding new links and imports to the migrations/index and that caused documentation to fail building for 8.3.0. * [DOCS] Add ml-cpp PRs to release notes * Added back incorrectly deleted changlog * Added missing highlight * Fixed spelling of StackOverflowError Co-authored-by: lcawl --- docs/changelog/86261.yaml | 17 + docs/reference/migration/index.asciidoc | 4 +- docs/reference/migration/migrate_8_3.asciidoc | 62 ++++ docs/reference/release-notes.asciidoc | 2 + docs/reference/release-notes/8.3.0.asciidoc | 319 ++++++++++++++++++ .../release-notes/highlights.asciidoc | 97 +++++- 6 files changed, 490 insertions(+), 11 deletions(-) create mode 100644 docs/changelog/86261.yaml create mode 100644 docs/reference/migration/migrate_8_3.asciidoc create mode 100644 docs/reference/release-notes/8.3.0.asciidoc diff --git a/docs/changelog/86261.yaml b/docs/changelog/86261.yaml new file mode 100644 index 0000000000000..29616aafafd95 --- /dev/null +++ b/docs/changelog/86261.yaml @@ -0,0 +1,17 @@ +pr: 86261 +summary: Snapshots as simple archives +area: Search +type: feature +issues: + - 81210 +highlight: + title: Reading indices from older Elasticsearch versions + body: |- + Elasticsearch has full query and write support for indices created in the previous major + version. If you have indices created in Elasticsearch versions 5 or 6, you can now use + the archive functionality to import and query these indices as well. + The archive functionality provides slower read-only access to older data, + for compliance or regulatory reasons, the occasional lookback or investigation, + or to rehydrate parts of it. Access to the data is expected to be infrequent, + and can therefore happen with limited performance and query capabilities. + notable: true diff --git a/docs/reference/migration/index.asciidoc b/docs/reference/migration/index.asciidoc index 0d72d22bc1f62..496a666e875d0 100644 --- a/docs/reference/migration/index.asciidoc +++ b/docs/reference/migration/index.asciidoc @@ -1,9 +1,11 @@ include::migration_intro.asciidoc[] -* <> +* <> +* <> * <> * <> +include::migrate_8_3.asciidoc[] include::migrate_8_2.asciidoc[] include::migrate_8_1.asciidoc[] include::migrate_8_0.asciidoc[] diff --git a/docs/reference/migration/migrate_8_3.asciidoc b/docs/reference/migration/migrate_8_3.asciidoc new file mode 100644 index 0000000000000..b69422dc5f4df --- /dev/null +++ b/docs/reference/migration/migrate_8_3.asciidoc @@ -0,0 +1,62 @@ +[[migrating-8.3]] +== Migrating to 8.3 +++++ +8.3 +++++ + +This section discusses the changes that you need to be aware of when migrating +your application to {es} 8.3. + +See also <> and <>. + +coming::[8.3.0-SNAPSHOT] + + +[discrete] +[[breaking-changes-8.3]] +=== Breaking changes + +There are no breaking changes in {es} 8.3. + + +[discrete] +[[deprecated-8.3]] +=== Deprecations + +The following functionality has been deprecated in {es} 8.3 +and will be removed in a future version. +While this won't have an immediate impact on your applications, +we strongly encourage you take the described steps to update your code +after upgrading to 8.3. + +To find out if you are using any deprecated functionality, +enable <>. + + +[discrete] +[[deprecations_83_cluster_and_node_setting]] +==== Cluster and node setting deprecations + +[[configuring_bind_dn_in_an_ldap_or_active_directory_ad_realm_without_corresponding_bind_password_deprecated]] +.Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password is deprecated +[%collapsible] +==== +*Details* + +For LDAP or AD authentication realms, setting a bind DN (via the +`xpack.security.authc.realms.ldap.*.bind_dn` realm setting) without a +bind password is a misconfiguration that may prevent successful +authentication to the node. In the next major release, nodes will fail +to start if a bind DN is specified without a password. + +*Impact* + +If you have a bind DN configured for an LDAP or AD authentication +realm, set a bind password for {ref}/ldap-realm.html#ldap-realm-configuration[LDAP] +or {ref}/active-directory-realm.html#ad-realm-configuration[Active Directory]. +Configuring a bind DN without a password generates a warning in the +deprecation logs. + +*Note:* This deprecation only applies if your current LDAP or AD +configuration specifies a bind DN without a password. This scenario is +unlikely, but might impact a small subset of users. +==== + diff --git a/docs/reference/release-notes.asciidoc b/docs/reference/release-notes.asciidoc index fd8aa061f3bc6..112d795eb7949 100644 --- a/docs/reference/release-notes.asciidoc +++ b/docs/reference/release-notes.asciidoc @@ -6,6 +6,7 @@ This section summarizes the changes in each release. +* <> * <> * <> * <> @@ -24,6 +25,7 @@ This section summarizes the changes in each release. -- +include::release-notes/8.3.0.asciidoc[] include::release-notes/8.2.3.asciidoc[] include::release-notes/8.2.2.asciidoc[] include::release-notes/8.2.1.asciidoc[] diff --git a/docs/reference/release-notes/8.3.0.asciidoc b/docs/reference/release-notes/8.3.0.asciidoc new file mode 100644 index 0000000000000..459113a15d2d5 --- /dev/null +++ b/docs/reference/release-notes/8.3.0.asciidoc @@ -0,0 +1,319 @@ +[[release-notes-8.3.0]] +== {es} version 8.3.0 + +coming[8.3.0] + +Also see <>. + +[[bug-8.3.0]] +[float] +=== Bug fixes + +Aggregations:: +* Allow `serial_diff` under `min_doc_count` aggs {es-pull}86401[#86401] +* Allow bucket paths to specify `_count` within a bucket {es-pull}85720[#85720] +* Fix flaky `top_metrics` test {es-pull}86582[#86582] (issue: {es-issue}86377[#86377]) +* Fix: check field existence before trying to merge running stats {es-pull}86926[#86926] +* Fix: ordering terms aggregation on top metrics null values {es-pull}85774[#85774] +* Serialize interval in auto date histogram aggregation {es-pull}85473[#85473] + +Audit:: +* Fix audit logging to consistently include port number in `origin.address` {es-pull}86732[#86732] + +Authentication:: +* An authorized user can disable a user with same name but different realm {es-pull}86473[#86473] +* Fix clearing of `lastSuccessfulAuthCache` when clear all realm cache API is called {es-pull}86909[#86909] (issue: {es-issue}86650[#86650]) + +CAT APIs:: +* Get hidden indices stats in `GET _cat/shards` {es-pull}86601[#86601] (issue: {es-issue}84656[#84656]) + +CCR:: +* Prevent invalid datastream metadata when CCR follows a datastream with closed indices on the follower {es-pull}87076[#87076] (issue: {es-issue}87048[#87048]) + +Cluster Coordination:: +* Add `master_timeout` support to voting config exclusions APIs {es-pull}86670[#86670] + +Discovery-Plugins:: +* [discovery-gce] Fix initialisation of transport in FIPS mode {es-pull}85817[#85817] (issue: {es-issue}85803[#85803]) + +Distributed:: +* Enforce external id uniqueness during `DesiredNode` construction {es-pull}84227[#84227] + +Engine:: +* Removing Blocking Wait for Close in `RecoverySourceHandler` {es-pull}86127[#86127] (issue: {es-issue}85839[#85839]) + +Features:: +* Fix 'autoGeneratedTimestamp should not be set externally' error when retrying IndexRequest {es-pull}86184[#86184] (issue: {es-issue}83927[#83927]) + +Geo:: +* Fix Geotile aggregations on `geo_shapes` for precision 0 {es-pull}87202[#87202] (issue: {es-issue}87201[#87201]) +* Fix `null_value` for array-valued `geo_point` fields {es-pull}85959[#85959] +* Guard for adding null value tags to vector tiles {es-pull}87051[#87051] +* Quantize geo queries to remove true negatives from search results {es-pull}85441[#85441] (issue: {es-issue}40891[#40891]) + +Highlighting:: +* `FastVectorHighlighter` should use `ValueFetchers` to load source data {es-pull}85815[#85815] (issues: {es-issue}75011[#75011], {es-issue}84690[#84690], {es-issue}82458[#82458], {es-issue}80895[#80895]) + +ILM+SLM:: +* Make the ILM Move to Error Step Batched {es-pull}85565[#85565] (issue: {es-issue}81880[#81880]) + +Infra/Core:: +* Adjust osprobe assertion for burst cpu {es-pull}86990[#86990] +* Clean up `DeflateCompressor` after exception {es-pull}87163[#87163] (issue: {es-issue}87160[#87160]) +* Error on direct creation of non-primary system index {es-pull}86707[#86707] +* Fix null message in output {es-pull}86981[#86981] +* Fix using `FilterOutputStream` without overriding bulk write {es-pull}86304[#86304] +* Hide system indices and their aliases in upgraded clusters {es-pull}87125[#87125] +* Refactor code to avoid JDK bug: JDK-8285835 {es-pull}86614[#86614] + +Infra/Logging:: +* Temporarily provide `SystemPropertiesPropertySource` {es-pull}87149[#87149] + +Infra/Plugins:: +* Use Windows newlines when listing plugin information on Windows {es-pull}86408[#86408] (issue: {es-issue}86352[#86352]) + +Infra/REST API:: +* Fix min node version before state recovery {es-pull}86482[#86482] + +Infra/Scripting:: +* Allow to sort by script value using `SemVer` semantics {es-pull}85990[#85990] (issues: {es-issue}85989[#85989], {es-issue}82287[#82287]) +* Script: Fix setter shortcut for unbridged setters {es-pull}86868[#86868] + +Infra/Settings:: +* Permit removal of archived index settings {es-pull}86107[#86107] + +Ingest:: +* Execute self-reference checks once per pipeline {es-pull}85926[#85926] (issue: {es-issue}85790[#85790]) + +Java Low Level REST Client:: +* Do not retry client requests when failing with `ContentTooLargeException` {es-pull}87248[#87248] (issue: {es-issue}86041[#86041]) + +License:: +* Consistent response for starting basic license {es-pull}86272[#86272] (issue: {es-issue}86244[#86244]) + +Machine Learning:: +* Fix ML task auditor exception early in cluster lifecycle {es-pull}87023[#87023] (issue: {es-issue}87002[#87002]) +* Fix distribution change check for `change_point` aggregation {es-pull}86423[#86423] +* Correct logic for restart from failover fine tuning hyperparameters for training classification and regression models {ml-pull}2251[#2251] +* Fix possible source of "x = NaN, distribution = class boost::math::normal_distribution<..." log errors training classification and regression models {ml-pull}2249[#2249] +* Fix some bugs affecting decision to stop optimizing hyperparameters for training classification and regression models {ml-pull}2259[#2259] +* Fix cause of "Must provide points at which to evaluate function" log error training classification and regression models {ml-pull}2268[#2268] +* Fix a source of "Discarding sample = nan, weights = ..." log errors for time series anomaly detection {ml-pull}2286[#2286] + +Network:: +* Reject `openConnection` attempt while closing {es-pull}86315[#86315] (issue: {es-issue}86249[#86249]) + +SQL:: +* Fix FORMAT function to comply with Microsoft SQL Server specification {es-pull}86225[#86225] (issue: {es-issue}66560[#66560]) +* Implement binary format support for SQL clear cursor {es-pull}84230[#84230] (issue: {es-issue}53359[#53359]) + +Search:: +* Add status field to Multi Search Template Responses {es-pull}85496[#85496] (issue: {es-issue}83029[#83029]) +* Fix `_terms_enum` on unconfigured `constant_keyword` {es-pull}86191[#86191] (issues: {es-issue}86187[#86187], {es-issue}86267[#86267]) +* Fix status code when open point in time without `keep_alive` {es-pull}87011[#87011] (issue: {es-issue}87003[#87003]) +* Make sure to rewrite explain query on coordinator {es-pull}87013[#87013] (issue: {es-issue}64281[#64281]) + +Security:: +* Make user and role name constraint consistent with max document ID {es-pull}86728[#86728] (issue: {es-issue}66020[#66020]) + +Snapshot/Restore:: +* DONE should mean fully processed in snapshot status {es-pull}86414[#86414] +* Distinguish missing and invalid repositories {es-pull}85551[#85551] (issue: {es-issue}85550[#85550]) +* Fork after calling `getRepositoryData` from `StoreRecovery` {es-pull}87264[#87264] (issue: {es-issue}87237[#87237]) +* Fork after calling `getRepositoryData` from `StoreRecovery` {es-pull}87254[#87254] (issue: {es-issue}87237[#87237]) +* Upgrade Azure SDK to 12.16.0 {es-pull}86135[#86135] + +TSDB:: +* TSDB: fix the time_series in order collect priority {es-pull}85526[#85526] +* TSDB: fix wrong initial value of tsidOrd in TimeSeriesIndexSearcher {es-pull}85713[#85713] (issue: {es-issue}85711[#85711]) + +Transform:: +* Fix transform `_start` permissions to use stored headers in the config {es-pull}86802[#86802] + +[[deprecation-8.3.0]] +[float] +=== Deprecations + +Authentication:: +* Configuring a bind DN in an LDAP or Active Directory (AD) realm without a corresponding bind password is deprecated {es-pull}85326[#85326] (issue: {es-issue}47191[#47191]) + +[[enhancement-8.3.0]] +[float] +=== Enhancements + +Aggregations:: +* Improve min and max performance while in a `random_sampler` aggregation {es-pull}85118[#85118] + +Authentication:: +* Support configurable claims in JWT Realm Tokens {es-pull}86533[#86533] +* Warn on user roles disabled due to licensing requirements for document or field level security {es-pull}85393[#85393] (issue: {es-issue}79207[#79207]) +* `TokenService` decode JWTs, change warn to debug {es-pull}86498[#86498] + +Authorization:: +* Add delete privilege to `kibana_system` for Synthetics {es-pull}85844[#85844] +* Authorize painless execute as index action when an index is specified {es-pull}85512[#85512] (issue: {es-issue}86428[#86428]) +* Better error message for run-as denials {es-pull}85501[#85501] (issue: {es-issue}72904[#72904]) +* Improve "Has Privilege" performance for boolean-only response {es-pull}86685[#86685] +* Relax restrictions for role names in roles API {es-pull}86604[#86604] (issue: {es-issue}86480[#86480]) +* [Osquery] Extend `kibana_system` role with an access to osquery_manager… {es-pull}86609[#86609] + +Autoscaling:: +* Add support for CPU ranges in desired nodes {es-pull}86434[#86434] + +Cluster Coordination:: +* Block joins while applier is busy {es-pull}84919[#84919] +* Compute master task batch summary lazily {es-pull}86210[#86210] +* Log `cluster.initial_master_nodes` at startup {es-pull}86101[#86101] +* Reduce resource needs of join validation {es-pull}85380[#85380] (issue: {es-issue}83204[#83204]) +* Report pending joins in `ClusterFormationFailureHelper` {es-pull}85635[#85635] +* Speed up map diffing (2) {es-pull}86375[#86375] + +Distributed:: +* Keep track of desired nodes cluster membership {es-pull}84165[#84165] + +Engine:: +* Cache immutable translog lastModifiedTime {es-pull}82721[#82721] (issue: {es-issue}82720[#82720]) +* Increase `force_merge` threadpool size based on the allocated processors {es-pull}87082[#87082] (issue: {es-issue}84943[#84943]) +* More optimal forced merges when max_num_segments is greater than 1 {es-pull}85065[#85065] + +Geo:: +* Support 'GeoJSON' in CartesianPoint for 'point' {es-pull}85442[#85442] +* Support geo label position as runtime field {es-pull}86154[#86154] +* Support geo label position through REST vector tiles API {es-pull}86458[#86458] (issue: {es-issue}86044[#86044]) + +Health:: +* Add a basic check for tier preference and allocation filter clashing {es-pull}85071[#85071] +* Add preflight checks to Health API to ensure health is obtainable {es-pull}86404[#86404] +* Health api add indicator doc links {es-pull}86904[#86904] (issue: {es-issue}86892[#86892]) +* Health api copy editing {es-pull}87010[#87010] +* Return a default user action if no actions could be determined {es-pull}87079[#87079] + +ILM+SLM:: +* Make the ILM and SLM `history_index_enabled` settings dynamic {es-pull}86493[#86493] + +Indices APIs:: +* Batch execute template and pipeline cluster state operations {es-pull}86017[#86017] + +Infra/Core:: +* Add mapping for tags for the elastic agent {es-pull}86298[#86298] +* Expand jar hell to include modules {es-pull}86622[#86622] +* Faster GET _cluster/settings API {es-pull}86405[#86405] (issue: {es-issue}82342[#82342]) +* Faster string writes by saving stream flushes {es-pull}86114[#86114] +* Fleet: Add `start_time` and `minimum_execution_duration` attributes to actions {es-pull}86167[#86167] +* Refactor array part into a `BytesRefArray` which can be serialized and … {es-pull}85826[#85826] +* Speed up ip v4 parser {es-pull}86253[#86253] +* Use varhandles for primitive type conversion in more places {es-pull}85577[#85577] (issue: {es-issue}78823[#78823]) + +Infra/Scripting:: +* Script: add ability to alias classes in whitelist {es-pull}86899[#86899] + +Ingest:: +* Iteratively execute synchronous ingest processors {es-pull}84250[#84250] (issue: {es-issue}84274[#84274]) + +License:: +* Initialize active realms without logging a message {es-pull}86134[#86134] (issue: {es-issue}81380[#81380]) + +Machine Learning:: +* A text categorization aggregation that works like ML categorization {es-pull}80867[#80867] +* Add new _infer endpoint for all supervised models and deprecate deployment infer api {es-pull}86361[#86361] +* Adds new `question_answering` NLP task for extracting answers to questions from a document {es-pull}85958[#85958] +* Adds start and end params to `_preview` and excludes cold/frozen tiers from unbounded previews {es-pull}86989[#86989] +* Adjust automatic JVM heap sizing for dedicated ML nodes {es-pull}86399[#86399] +* Replace the implementation of the `categorize_text` aggregation {es-pull}85872[#85872] +* Upgrade PyTorch to version 1.11 {ml-pull}2233[#2233], {ml-pull}2235[#2235],{ml-pull}2238[#2238] +* Upgrade zlib to version 1.2.12 on Windows {ml-pull}2253[#2253] +* Upgrade libxml2 to version 2.9.14 on Linux and Windows {ml-pull}2287[#2287] +* Improve time series model stability and anomaly scoring consistency for data + for which many buckets are empty {ml-pull}2267[#2267] +* Address root cause for actual equals typical equals zero anomalies {ml-pull}2270[#2270] +* Better handling of outliers in update immediately after detecting changes in time series {ml-pull}2280[#2280] + +Mapping:: +* Intern field names in Mappers {es-pull}86301[#86301] +* Replace BYTE_BLOCK_SIZE - 2 with indexWriter#MAX_TERM_LENGTH {es-pull}85518[#85518] + +Network:: +* Log node identity at startup {es-pull}85773[#85773] + +Search:: +* GeoBoundingBox query should work on bounding box with equal latitude or longitude {es-pull}85788[#85788] (issue: {es-issue}77717[#77717]) +* Improve error message for search API url parameters {es-pull}86984[#86984] (issue: {es-issue}79719[#79719]) + +Security:: +* Add run-as support for OAuth2 tokens {es-pull}86680[#86680] +* Relax username restrictions for User APIs {es-pull}86398[#86398] (issue: {es-issue}86326[#86326]) +* User Profile - Add hint support to SuggestProfiles API {es-pull}85890[#85890] +* User Profile - Add new action origin and internal user {es-pull}86026[#86026] +* User Profile - Support request cancellation on HTTP disconnect {es-pull}86332[#86332] +* User Profile - add caching for `hasPrivileges` check {es-pull}86543[#86543] + +Snapshot/Restore:: +* Add parameter to exclude indices in a snapshot from response {es-pull}86269[#86269] (issue: {es-issue}82937[#82937]) + +Stats:: +* Add documentation for "io_time_in_millis" {es-pull}84911[#84911] + +TLS:: +* Set `serverAuth` extended key usage for generated certificates and CSRs {es-pull}86311[#86311] (issue: {es-issue}81067[#81067]) + +TSDB:: +* Aggregation Execution Context add timestamp provider {es-pull}85850[#85850] + +Transform:: +* Prefer secondary auth headers for transforms {es-pull}86757[#86757] + +[[feature-8.3.0]] +[float] +=== New features + +Authorization:: +* Has privileges API for profiles {es-pull}85898[#85898] + +Health:: +* Add support for `impact_areas` to health impacts {es-pull}85830[#85830] (issue: {es-issue}85829[#85829]) +* Add troubleshooting guides to shards allocation actions {es-pull}87078[#87078] +* Adding potential impacts to remaining health indicators {es-pull}86197[#86197] +* Health api drill down {es-pull}85234[#85234] (issue: {es-issue}84793[#84793]) +* New service to keep track of the master history as seen from each node {es-pull}85941[#85941] +* Sorting impact index names by index priority {es-pull}85347[#85347] + +Mapping:: +* Add support for dots in field names for metrics usecases {es-pull}86166[#86166] (issue: {es-issue}63530[#63530]) +* Synthetic source {es-pull}85649[#85649] + +SQL:: +* SQ: Allow partial results in SQL queries {es-pull}85897[#85897] (issue: {es-issue}33148[#33148]) + +Search:: +* Snapshots as simple archives {es-pull}86261[#86261] (issue: {es-issue}81210[#81210]) + +TSDB:: +* TSDB: Implement downsampling on time-series indices {es-pull}85708[#85708] (issues: {es-issue}69799[#69799], {es-issue}65769[#65769]) + +[[upgrade-8.3.0]] +[float] +=== Upgrades + +Infra/CLI:: +* Upgrade procrun executables to 1.3.1 {es-pull}86710[#86710] + +Infra/Core:: +* Upgrade jackson to 2.13.2 {es-pull}86051[#86051] + +Ingest:: +* Upgrading to tika 2.4 {es-pull}86015[#86015] + +Network:: +* Upgrade to Netty 4.1.76 {es-pull}86252[#86252] + +Packaging:: +* Update Iron Bank base image to 8.6 {es-pull}86796[#86796] + +SQL:: +* Update dependency - JLine - to v 3.21.0 {es-pull}83767[#83767] (issue: {es-issue}83575[#83575]) + +Search:: +* Upgrade to Lucene 9.2 snapshot {es-pull}86227[#86227] +* Upgrade to lucene snapshot 978eef5459c {es-pull}86852[#86852] + + diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index 25b0d6b322e6e..8deb96bd98601 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -1,6 +1,8 @@ [[release-highlights]] == What's new in {minor-version} +coming::[{minor-version}] + Here are the highlights of what's new and improved in {es} {minor-version}! ifeval::[\{release-state}\"!=\"unreleased\"] For detailed information about this release, see the <> and @@ -14,18 +16,93 @@ Other versions: | {ref-bare}/8.1/release-highlights.html[8.1] | {ref-bare}/8.0/release-highlights.html[8.0] -// The notable-highlights tag marks entries that -// should be featured in the Stack Installation and Upgrade Guide: // tag::notable-highlights[] -// [discrete] -// === Heading -// -// Description. -// end::notable-highlights[] +[discrete] +[[reading_indices_from_older_elasticsearch_versions]] +=== Reading indices from older Elasticsearch versions +Elasticsearch has full query and write support for indices created in the previous major +version. If you have indices created in Elasticsearch versions 5 or 6, you can now use +the archive functionality to import and query these indices as well. +The archive functionality provides slower read-only access to older data, +for compliance or regulatory reasons, the occasional lookback or investigation, +or to rehydrate parts of it. Access to the data is expected to be infrequent, +and can therefore happen with limited performance and query capabilities. + +[discrete] +[[add_support_for_dots_in_field_names_for_metrics_usecases]] +=== Add support for dots in field names for metrics usecases +Metrics data can often be made of several fields with dots in their names, +sharing common prefixes, like in the following example: + +``` +{ + "metrics.time" : 10, + "metrics.time.min" : 1, + "metrics.time.max" : 500 +} +``` + +Such format causes a mapping conflict as the `metrics.time` holds a value, +but it also needs to be mapped as an object in order to hold the `min` and +`max` leaf fields. + +A new object mapping parameter called `subobjects`, which defaults to `true`, +has been introduced to preserve dots in field names. An object with `subobjects` +set to `false` can only ever hold leaf sub-fields and no further objects. The +following example shows how it can be configured in the mappings for the +`metrics` object: + +``` +{ + "mappings": { + "properties" : { + "metrics" : { + "type" : "object", + "subobjects" : false + } + } + } +} +``` + +With this configuration any child of `metrics` will be mapped unchanged, +without expanding dots in field names to the corresponding object structure. +That makes it possible to store the metrics document above. [discrete] -[[integrate_filtering_support_for_approximate_nearest_neighbor_search]] -=== Integrate filtering support for approximate nearest neighbor search -The {ref}/knn-search-api.html[_knn_search endpoint] now has a "filter" option that allows to return only the nearest documents that satisfy the provided filter +[[execute_self_reference_checks_once_per_pipeline]] +=== Execute self-reference checks once per pipeline +Ingest pipelines have a mechanism to prevent circular references in the records +they process, so that they are serializable. Prior to this change, this check was +performed after each `script` processor execution, and was ignorable. + +Because of this check, a script processor configured with +``` +"source": """ + def x = ctx; + ctx.x = x; +""" +``` + +...would error with `"type" : "illegal_argument_exception", "reason" : "Iterable +object is self-referencing itself (ingest script)"`. + +If the script processor also had +``` +"ignore_failure" true +``` + +...then the handling thread would actually crash from an unrecoverable +StackOverflowError when trying to serialize the resulting event. + +Now, this check is performed once per pipeline, remediating the potential for +a StackOverflowError. There are some side effects also: + +- The resulting error message specifies which pipeline is causing the issue. +- There is a slight speed up for pipelines with multiple scripts, and a slight +slowdown for pipelines without scripts. + +// end::notable-highlights[] +