Feature/geo shape parameter deprecation #76626
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…) (elastic#75766) CacheIteratorHelper requires lock acquisition for any mutation to the underlying cache. This means it is incorrect to manipulate the cache without invocation of CacheIteratorHelper#acquireUpdateLock. This is OK for caches of simple values, but feels excessive for caches of ListenableFuture. This PR update the cache invalidation code to use cache.forEach instead of CacheInvalidator. It simplifies the code by removing any explicit lockings. The tradeoff is that it needs to build a list of keys to delete in memory. Overall it is a better tradeoff since no explicit locking is required and better leverage of Cache's own methods. Co-authored-by: Yang Wang <[email protected]>
…lastic#75748) _mvt end point gives priority to geometries with larger area
In a mixed 6.x and 7.x cluster, a search that uses dfs_query_then_fetch can cause a transport serialization errors. This is related to https://issues.apache.org/jira/browse/LUCENE-8007, which was introduced in Lucene 8 and adds stricter checks to TermStatistics and CollectionStatistics, and https://issues.apache.org/jira/browse/LUCENE-8020, which was introduced in Lucene 8 and avoids bogus term stats (e.g. docfreq=0). Co-authored-by: Julie Tibshirani [email protected] Closes elastic#75349
…tic#75740) this has been deprecated and will be removed in Gradle 8.0. We use JavaPluginExtension instead from now on. Co-authored-by: Rene Groeschke <[email protected]>
These assertions were simply wrong, but took time to catch as they were very rarely triggered. Fixes elastic#75736.
… (elastic#75772) * Make PublishPlugin gradle configuration cache compliant for using --configuration-cache we need to remove some usages of the mutable project instance * Polishing
Part of elastic#67335. Add tasks for generating release notes, using information stored in files in the repository: * `generateReleaseNotes` - generates new release notes, release highlights and breaking changes * `validateChangelogs` - validates that all the changelog YAML files are well-formed (confirm to schema, have required fields depending on the `type` value) I also changed `Version` to allow a `v` prefix in relaxed mode
Part of elastic#70219. Split the install and remove plugin CLI commands into `Command` and `Action` classes, so that the main implementations can be reused in a later PR to create a `Sync` command.
Backporting _nodes/stats feature This commit cherry-picks the commit from elastic#75433, and changes it to apply to 7.15 and later.
…port PR (elastic#75788) This is related to elastic#75052
…lastic#75797) The bwc-test plugin automatically wires up BWC tasks to the appropriate lifecycle tasks for us. There's no need to do this explicitly. Running 'check' should only run BWC tests against unreleased versions as-per convention. Released version tests are run via periodic jobs in CI. Co-authored-by: Mark Vieira <[email protected]>
…c#75809) Co-authored-by: Lisa Cawley <[email protected]>
…astic#75823) In addition to read, access to this API is now also granted by view_index_metadata and manage. Resolves: elastic#74779
…75790) (elastic#75827) Add xref Co-authored-by: James Rodewig <[email protected]> Co-authored-by: Andrei Dan <[email protected]> Co-authored-by: James Rodewig <[email protected]>
The ML plugin should not try to respond to shutdown API calls when disabled.
…c#75459) (elastic#75829) Remove an unnecessary indirection and refactor progress tracking. Both rollup and transform process documents as stream, however in the AsyncTwoPhaseIndexer takes a List of index requests. This change removes the unnecessary temporary container and makes upcoming transform enhancements easier.
elastic#75826) This is a follow up on elastic#75569 and should fix installation problems in FIPS enabled environments.
…5814) * [DOCS] Steps for updating TLS certificates * Updates for changing CA * Updates for rotating certs with a new CA * Add instructions for generating HTTP certs with a new CA * Add steps for creating HTTP certs with new CA * Clarify note about cluser restart and other edits * Clarifying scenarios * Apply suggestions from code review Co-authored-by: Ioannis Kakavas <[email protected]> * Incorporating review feedback and making necessary changes * Clarifications and changes regarding restarts * Remove errant --pem in basic security setup * Incorporate suggestions from code review Co-authored-by: Ioannis Kakavas <[email protected]> * Many, many updates. But good ones. * Add languages for snippets * Reorder steps to reference rolling restart throughout for consistency * Add clarifying what's next steps * Add instructions for updating Kibana certificate * Apply suggestions from Ioannis' stellar code review Co-authored-by: Ioannis Kakavas <[email protected]> * Update instructions to use a single keystore, plus other review changes * Incorporating another round of review comments * Minor updates from reviewer feedback * Clarifying examples and fixing numbering * Skip tests that are creating unnecessary noise * Quieting other tests Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Ioannis Kakavas <[email protected]> Co-authored-by: Adam Locke <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Ioannis Kakavas <[email protected]>
…5662) (elastic#75833) Same as elastic#75252 pretty much just continuing to make this logic a little simpler for easier profiling and (very) maybe performance through saving some allocations/indirection.
…lastic#75835) testKsTest_LowerTailedValues is failing due to distribution issues. The random variables in the sparse case are too uniform and causing two-sided alternative tests to not be consistent. closes elastic#74909 Co-authored-by: Benjamin Trent <[email protected]>
…uckets (elastic#75815) (elastic#75837) In a consistently delayed environment, it is possible that many buckets in a row have delayed data detected. This can flood the audit message page, causing noise that detracts from other useful information This commit reduces the number of audit messages written as the number of consecutive buckets with delayed data increases. closes elastic#73679 Co-authored-by: Benjamin Trent <[email protected]>
…apshot (elastic#75855) * Properly apply `system` flag on data streams when restoring a snapshot (elastic#75819) This commit modifies the restore process to ensure that the `system` flag is properly applied to restored data streams. Otherwise, this flag is lost when restoring system data streams, which causes errors and/or assertion failures as the backing indices are properly marked as system indices, but the restored data stream is no longer a system data stream. Also adds a test to ensure this flag survives a round trip through the snapshot/restore process. * Fix compilation for backport
In elastic#74260 we disabled the "filter by filter" aggregations optimizations when we couldn't be 100% sure that they were both faster and *safe* for the cluster. Before that, mostly in the 7.13 line, we would aggressively enable the optimization whenever we could and there were lots of cases where the well meaning optimization was way way slower than the original implementation. There were also cases where it'd fill up memory and knock over the cluster! Disaster. With elastic#74260 we're safe. The optimization still kicks in when its faster - like when the top level query is empty - but otherwise it get's out of the way and lets all the normal aggs run.
…astic#75868) This commit replaces use of "blacklist" with "failed" to better reflect the behaviour of the test
…c#75850) (elastic#75872) * [ML] Ignore exceptions while opening job after SIGTERM to JVM We observed that some jobs failed during a rolling upgrade in Elastic Cloud. This happened because steps of the job open sequence failed with exceptions after core Elasticsearch services shut down in response to the SIGTERM. This change makes the persistent task executor for anomaly detection jobs ignore exceptions received after the JVM has received a shutdown signal, for example a SIGTERM. By doing nothing in response to such exceptions the persistent task remains in cluster state and will get assigned to a different node after the current node leaves the cluster. * Address review comment Co-authored-by: David Roberts <[email protected]>
…elastic#75501) (elastic#76539) This refactors the snapshots-in-progress logic to work from `RepositoryShardId` when working out what parts of the repository are in-use by writes for snapshot concurrency safety. This change does not go all the way yet on this topic and there are a number of possible follow-up further improvements to simplify the logic that I'd work through over time. But for now this allows fixing the remaining known issues that snapshot stress testing surfaced when combined with the fix in elastic#75530. These issues all come from the fact that `ShardId` is not a stable key across multiple snapshots if snapshots are partial. The scenarios that are broken are all roughly this: * snapshot-1 for index-A with uuid-A runs and is partial * index-A is deleted and re-created and now has uuid-B * snapshot-2 for index-A is started and we now have it queued up behind snapshot-1 for the index * snapshot-1 finishes and the logic tries to start the next snapshot for the same shard-id * this fails because the shard-id is not the same, we can't compare index uuids, just index name + shard id * this change fixes all these spots by always taking the round trip via `RepositoryShardId` planned follow-ups here are: * dry up logic across cloning and snapshotting more as both now essentially run the same code in many state-machine steps * serialize snapshots-in-progress efficiently instead of re-computing the index and by-repository-shard-id lookups in the constructor every time * refactor the logic in snapshots-in-progress away from maps keyed by shard-id in almost all spots to this end, just keep an index name to `Index` map to work out what exactly is being snapshotted * refactoring snapshots-in-progress to be a map of list of operations keyed by repository shard id instead of a list of maps as it currently is to make the concurrency simpler and more obviously correct closes elastic#75423 relates (elastic#75339 ... should also fix this, but I have to verify by testing with a backport to 7.x)
…lastic#76546) We kept wrapping the collection over and over again which in extreme corner cases could lead to a SOE. Closes elastic#76490 Co-authored-by: hanbj <[email protected]>
…after Failing Operations (elastic#75733) (elastic#76548) The node executing a shard level operation would in many cases communicate `null` for the shard state update, leading to follow-up operations incorrectly assuming an empty shard snapshot directory and starting from scratch. closes elastic#75598
We must not send the by-repo-shard-id map for normal snapshots. The conditional got lost in `7.x` and `7.14` because of a merge error. This aligns master and 7.x behavior. closes elastic#76552
backport of elastic#76476 Licensed feature tracking utilizes the existing license level checks to track when a feature is used. However, some features check the license level at the start of an operation or when enabling a feature, but then the tracking only captures the beginning time. This commit reworks the licensed feature framework to use a new LicensedFeature class which will eventually replace XPackLicenseState.Feature values. There are two LicensedFeature implementations, one for "momentary" features that are tracked just at the moment they are used, and "persistent" features that are considered "on" until the feature is untracked. The usage map of tracked features is cleaned up every hour, and those features that have not been used in the last 24 hours are removed from tracking. Not all features are converted to LicensedFeature yet. Instead, a few features have been converted to demonstrate how it can be done, so that the rest can be done in parallel at a future time. Co-authored-by: Tim Vernum <[email protected]>
elastic#76561) Adding check that shard level index metadata actually contains the snapshots it's supposed to contain. This would have caught a number of recent bugs.
…n of ILM (elastic#74219) (elastic#76565) Add validation of the number_of_shards parameter in Shrink Action of ILM (cherry picked from commit 58feb4e) Signed-off-by: Andrei Dan <[email protected]> Co-authored-by: bellengao <[email protected]>
…76512) (elastic#76566) This fixes the error message in Painless for when an instance-style method isn't found. It now prints the number of arguments instead of the memory location of the nodes data structure.
…lastic#76568) * Add system data streams to feature state snapshots (elastic#75902) Add system data streams to the "snapshot feature state" code block, so that if we're snapshotting a feature by name we grab that feature's system data streams too. Handle these data streams on the restore side as well. * Add system data streams to feature state snapshots * Don't pass system data streams through index name resolution * Don't add no-op features to snapshots * Hook in system data streams for snapshot restoration
…ons (elastic#43874) (elastic#76475) When performing incremental reductions, 0 value of docCountError may mean that the error was not previously calculated, or that the error was indeed previously calculated and its value was 0. We end up rejecting true values set to 0 this way. This may lead to wrong upper bound of error in result. To fix it, this PR makes docCountError nullable. null values mean that error was not calculated yet. Fixes elastic#40005, elastic#75667 Co-authored-by: Nikita Glashenko <[email protected]>
…m interval for better performance (elastic#74004) (elastic#76570)
…#76499) (elastic#76572) * Keep track of data recovered from snapshots in RecoveryState (elastic#76499) Adds new field to recovery API to keep track of amount of data recovered from snapshots. The normal recovered_bytes field remains and is also increased for recovery from snapshot but can go backwards in the unlikely case that recovery from snapshot fails to download a file. Relates elastic#73496 * one less space Co-authored-by: Francisco Fernández Castaño <[email protected]>
…elastic#75942) (elastic#76574) We have to run the loop checking for completed snapshots if we see an index disappearing. Otherwise, we never get around to finalizing a queued snapshot stuck after a clone if the index is deleted during cloning.
) * Fix compiler warnings in :server - part 4 (elastic#76302) Closes elastic#40366. Fix the last remaining javac issues when linting is enforced in `server/`. * Further fixes after backport
Currently the cluster.remote.*.transport.compression_scheme setting defaults to the transport.compression_scheme value. This commit modifies this to default to deflate (the existing compression scheme prior to 7.14) when cluster.remote.*.transport.compress is explicitly set. This will ensure that users do not accidentally change their compression scheme for 7.x.
This PR adds Rest API spec and related yaml tests for the new Query API key API. Relates: elastic#71023
We must never write generations extracted out of failed shard snapshot status values to the repository as these can not be trusted in all cases. Instead we must always put a `null` for these into the generations object to write to the repo so that the existing generations are not changed.
…) (elastic#76596) * [Test] Improve assertions for better messages on failure (elastic#76543) This PR adjusts assertions in testPagination so that more detailed messages would be made available on failures. For example, if the sorted keys do not match the expected order, it now shows all keys for better context instead of just the single mismatched key. Relates: elastic#76542 * fix for 7.x quirks
This refactors the signature of snapshot finalization. For one it allows removing a TODO about being dependent on mutable `SnapshotInfo` which was not great but more importantly this sets up a follow-up where state can be shared between the cluster state update at the end of finalization and subsequent old-shard-generation cleanup so that we can resolve another open TODO about leaking shard generation files in some cases.
Add test to verify that concurrently indexing together with recover from snapshot works. Relates elastic#76237
elastic#76603) With recent fixes it is never correct to simply remove a snapshot from the cluster state without updating other snapshot entries if an entry contains any successful shards due to possible dependencies. This change reproduces two issues resulting from simply removing snapshot without regard for other queued operations and fixes them by having all removal of snapshot from the cluster state go through the same code path. Also, this change moves the tracking of a snapshot as "ending" up a few lines to fix an assertion about finishing snapshots that forces them to be in this collection.
Selectively muting parts of the rolling upgrade test for recover from snapshot. Relates elastic#76595
…astic#76607) (elastic#76608) This should help track down problems with the GlobalInfoPlugin people reported
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following properties have been removed from the geo_shape data type in 8.0: "strategy", "tree", "tree_levels", "precision", "distance_error_pct", "points_only". This PR adds a deprecation info API check for indexes and templates that have any of these properties in their mappings. Relates to #42404 and #70850.