diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index 95a5cc2909aeb..1d85b3a64c1b5 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -12,6 +12,8 @@ include::{elasticsearch-root}/docs/Versions.asciidoc[] include::intro.asciidoc[] +include::release-notes/highlights.asciidoc[] + include::getting-started.asciidoc[] include::setup.asciidoc[] @@ -70,8 +72,6 @@ include::glossary.asciidoc[] include::rest-api/index.asciidoc[] -include::release-notes/highlights.asciidoc[] - include::migration/index.asciidoc[] include::release-notes.asciidoc[] diff --git a/docs/reference/intro.asciidoc b/docs/reference/intro.asciidoc index 2591487bd04db..bf103ee26e1d9 100644 --- a/docs/reference/intro.asciidoc +++ b/docs/reference/intro.asciidoc @@ -1,7 +1,5 @@ [[elasticsearch-intro]] -= Elasticsearch introduction -[partintro] --- +== What is {es}? _**You know, for search (and analysis)**_ {es} is the distributed search and analytics engine at the heart of @@ -35,10 +33,9 @@ We’re continually amazed by the novel ways people use search. But whether your use case is similar to one of these, or you're using {es} to tackle a new problem, the way you work with your data, documents, and indices in {es} is the same. --- [[documents-indices]] -== Data in: documents and indices +=== Data in: documents and indices {es} is a distributed document store. Instead of storing information as rows of columnar data, {es} stores complex data structures that have been serialized @@ -92,7 +89,7 @@ used at search time. When you query a full-text field, the query text undergoes the same analysis before the terms are looked up in the index. [[search-analyze]] -== Information out: search and analyze +=== Information out: search and analyze While you can use {es} as a document store and retrieve documents and their metadata, the real power comes from being able to easily access the full suite @@ -108,7 +105,7 @@ or Ruby. [float] [[search-data]] -=== Searching your data +==== Searching your data The {es} REST APIs support structured queries, full text queries, and complex queries that combine the two. Structured queries are @@ -133,7 +130,7 @@ third-party applications to interact with {es} via SQL. [float] [[analyze-data]] -=== Analyzing your data +==== Analyzing your data {es} aggregations enable you to build complex summaries of your data and gain insight into key metrics, patterns, and trends. Instead of just finding the @@ -165,7 +162,7 @@ embroidery_ needles. [float] [[more-features]] -==== But wait, there’s more +===== But wait, there’s more Want to automate the analysis of your time-series data? You can use {ml-docs}/ml-overview.html[machine learning] features to create accurate @@ -180,7 +177,7 @@ And the best part? You can do this without having to specify algorithms, models, or other data science-related configurations. [[scalability]] -== Scalability and resilience: clusters, nodes, and shards +=== Scalability and resilience: clusters, nodes, and shards ++++ Scalability and resilience ++++ @@ -212,7 +209,7 @@ interrupting indexing or query operations. [float] [[it-depends]] -=== It depends... +==== It depends... There are a number of performance considerations and trade offs with respect to shard size and the number of primary shards configured for an index. The more @@ -240,7 +237,7 @@ testing with your own data and queries]. [float] [[disaster-ccr]] -=== In case of disaster +==== In case of disaster For performance reasons, the nodes within a cluster need to be on the same network. Balancing shards in a cluster across nodes in different data centers @@ -260,7 +257,7 @@ secondary clusters are read-only followers. [float] [[admin]] -=== Care and feeding +==== Care and feeding As with any enterprise system, you need tools to secure, manage, and monitor your {es} clusters. Security, monitoring, and administrative features diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 6dd7b2d24c879..0d048386c2ce1 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -891,6 +891,10 @@ See <>. See <>. +[[role="exclude", id="release-highlights-7.7.0"] +=== 7.7.0 release highlights + +See <>. //// [role="exclude",id="search-request-body"] diff --git a/docs/reference/release-notes/7.1.asciidoc b/docs/reference/release-notes/7.1.asciidoc index 615bade10c94d..a2e8a9dc113a7 100644 --- a/docs/reference/release-notes/7.1.asciidoc +++ b/docs/reference/release-notes/7.1.asciidoc @@ -40,7 +40,7 @@ Also see <>. === Enhancements Security:: -* Moved some security features to basic. See <> +* Moved some security features to basic. See {ref-bare}/7.1/release-highlights-7.1.0.html[7.1.0 Release highlights]. Authentication:: * Log warning when unlicensed realms are skipped {pull}41778[#41778] diff --git a/docs/reference/release-notes/highlights.asciidoc b/docs/reference/release-notes/highlights.asciidoc index a22f3d9d4ca8e..4def7304e156e 100644 --- a/docs/reference/release-notes/highlights.asciidoc +++ b/docs/reference/release-notes/highlights.asciidoc @@ -1,27 +1,200 @@ [[release-highlights]] -= Release highlights - -[partintro] --- -This section summarizes the most important changes in each release. For the -full list, see <> and <>. - -* <> -* <> -* <> -* <> -* <> -* <> -* <> -* <> - --- - -include::highlights-7.7.0.asciidoc[] -include::highlights-7.6.0.asciidoc[] -include::highlights-7.5.0.asciidoc[] -include::highlights-7.4.0.asciidoc[] -include::highlights-7.3.0.asciidoc[] -include::highlights-7.2.0.asciidoc[] -include::highlights-7.1.0.asciidoc[] -include::highlights-7.0.0.asciidoc[] +== What's new in {minor-version} + +Here are the highlights of what's new and improved in {es} {minor-version}! +For detailed information about this release, see the +<> and +<>. + +// Add previous release to the list +Other versions: +{ref-bare}/7.7/release-highlights.html[7.6] +| {ref-bare}/7.5/release-highlights-7.5.0.html[7.5] +| {ref-bare}/7.4/release-highlights-7.4.0.html[7.4] +| {ref-bare}/7.3/release-highlights-7.3.0.html[7.3] +| {ref-bare}/7.2/release-highlights-7.2.0.html[7.2] +| {ref-bare}/7.1/release-highlights-7.1.0.html[7.1] +| {ref-bare}/7.0/release-highlights-7.0.0.html[7.0] + +// tag::notable-highlights[] +[discrete] +=== Fixed index corruption on shrunk indices + +Applying deletes or updates on an index after it had been shrunk would likely +corrupt the index. We advise users of Elasticsearch 6.x who opt in for soft +deletes on some of their indices and all users of Elasticsearch 7.x to upgrade +to 7.7 as soon as possible to no longer be subject to this corruption bug. In +case upgrading in the near future is not an option, we recommend to completely +stop using `_shrink` on read-write indices and to do a force-merge right after +shrinking on read-only indices, which significantly reduces the likeliness of +being affected by this bug in case deletes or updates get applied by mistake. +This bug is fixed as of {es} 7.7.0. Low-level details can be found on the +https://issues.apache.org/jira/browse/LUCENE-9300[corresponding issue]. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== Significant reduction of heap usage of segments + +This release of Elasticsearch significantly reduces the amount of heap memory +that is needed to keep Lucene segments open. In addition to helping with cluster +stability, this helps reduce costs by storing much more data per node before +hitting memory limits. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== {transforms-cap} – now in GA! + +In 7.7, we move {transforms} from beta to general availability. + +{ref}/transforms.html[{transforms-cap}] enable you to pivot existing {es} +indices using group-by and aggregations into a destination feature index, which +provides opportunities for new insights and analytics. For example, you can use +{transforms} to pivot your data into entity-centric indices that summarize the +behavior of users or sessions or other entities in your data. + +{transforms-cap} now include support for cross-cluster search. Allowing you to +create your destination feature index on a separate cluster from the source +indices. + +Aggregation support has been expanded within {transforms} to include support for +{ref}/search-aggregations-metrics-percentile-aggregation.html[multi-value (percentiles)] +and +{ref}/search-aggregations-bucket-filter-aggregation.html[filter aggregations]. +We also optimized the performance of the +{ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram aggregations]. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== Introducing multiclass {classification} + +{ml-docs}/dfa-classification.html[{classification-cap}] using multiple classes +is now available in {dfanalytics}. {classification-cap} is a supervised {ml} +technique which has been already available as a binary process in the previous +release. Multiclass {classification} works well with up to 30 distinct +categories. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== {feat-imp-cap} at {infer} time + +{feat-imp-cap} now can be calculated at {infer} time. This value provides +further insight into the results of a {classification} or {regression} job and +therefore helps interpret these results. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== Finer memory control for bucket aggregations + +While building buckets, aggregations will now periodically check the +real-memory circuit breaker before continuing to allocate more buckets. This +allows better responsivity to memory pressure and avoids `OutOfMemory` +situations due to allocating more buckets than the node can handle. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== A new way of searching: asynchronously + +You can now submit {ref}/async-search-intro.html[long-running searches] using +the new {ref}/async-search.html[`_async_search` API]. The new API accepts the +same parameters and request body as the {ref}/search-search.html[Search API]. +However, instead of blocking and returning the final response only when it's +entirely finished, you can retrieve results from an async search as they become +available. + +The request takes a parameter, `wait_for_completion`, which controls how long +the server will wait until it sends back a response. The first response +contains among others a search unique ID, a response version, an indication if +this response is partial or not, plus the usual metadata (shards involved, +number of hits etc) and potentially results. If the response is not complete +and final, the client can continue polling for results, issuing a new request +using the provided search ID. If new results are available, the returned +version is incremented and the new batch of results are returned. This can +continue until all the results are fetched. + +Unless deleted earlier by the user, the asynchronous searches are kept alive +for a given interval. This defaults to 5 days and can be controlled by another +request parameter, `keep_alive`. +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== Password protection for the keystore + +{es} uses a custom on-disk {ref}/secure-settings.html[keystore] for secure settings such as +passwords and SSL certificates. Up until now, this prevented users with +{ref}/elasticsearch-keystore.html[command-line access] from viewing secure files by listing commands, but nothing +prevented such users from changing values in the keystore, or removing values +from it. Furthermore, the values were only obfuscated by a hash; no +user-specific secret protected the secure settings. + +This new feature changes all of that by adding password-protection to the +keystore. This is not be a breaking change: if a keystore has no password, +there won’t be any new prompts. A user must choose to password-protect their +keystore in order to benefit from the new behavior. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== A new aggregation: `top_metrics` + +The new {ref}//search-aggregations-metrics-top-metrics.html[`top_metrics` aggregation] "selects" a metric from a document according +to a criteria on a given, different field. That criteria is currently the +largest or smallest "sort" value. It is fairly similar to `top_hits` in spirit, +but because it is more limited, `top_metrics` uses less memory and +is often faster. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== Query speed-up for sorted queries on time-based indices + +We've optimized sorted, top-documents-only queries run on time-based indices. +The optimization stems from the fact that the ranges of (document) timestamps +in the shards don't overlap. It is implemented by rewriting the shard search +requests based on the partial results already available from other shards, if +it can be determined that the query will not yield any result from the current +shard; i.e. we know in advance that the bottom entry of the (sorted) result set +after a partial merge is better than the values contained in this current +shard. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== A new aggregation: `boxplot` + +The https://en.wikipedia.org/wiki/Interquartile_range[interquartile range (IQR)] is a common robust measure of statistical dispersion. +Compared to the standard deviation, the IQR is less sensitive to outliers in +the data, with a breakdown point of 0.25. Along with the median, it is often +used in creating a box plot, a simple yet common way to summarize data and +identify potential outliers. + +The new {ref}/search-aggregations-metrics-boxplot-aggregation.html[`boxplot` +aggregation] calculates the min, max, and medium as well as the first and third +quartiles of a given data set. + +// end::notable-highlights[] + +// tag::notable-highlights[] +[discrete] +=== AArch64 support + +{es} now provides AArch64 packaging, including bundling an AArch64 JDK +distribution. There are some restrictions in place, namely no {ml} support and +depending on underlying page sizes, class data sharing is disabled. + +// end::notable-highlights[]