Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Elasticsearch version to 9.0.0 #112570

Merged
merged 44 commits into from
Sep 11, 2024
Merged

Bump Elasticsearch version to 9.0.0 #112570

merged 44 commits into from
Sep 11, 2024

Conversation

mark-vieira
Copy link
Contributor

@mark-vieira mark-vieira commented Sep 5, 2024

This is the initial work to bump the Elasticsearch version to 9.0 in preparation for our next major version release.

What this pull request includes

  • The stack version has been bumped to 9.0.0.
  • Backward compatibility testing logic has been updated to account for the new major version and branching changes. That is, once this PR is merged to main, the current main will be branched to 8.x. Back compat testing will now checkout the 8.x branch for testing that snapshot version.
  • The minimum wire compatibility version has been updated to 8.16.0. This is now determined dynamically based on known versions so no further changes should be necessary when 8.x is bumped to 8.17.
  • Rest compatibility tests are running using tests sourced from the 8.x branch. These are effectively identical to those in main so all existing rest compat overrides and transforms have been removed.
  • Our rest compatibility plugins have been simplified as well. Task names no longer include the previous major version as there will only ever be one of them. This will make future major version bumps slightly easier.
  • BWC test versions have been updated to remove all 7.x versions since there is now no level of backward compatibility with that version of Elasticsearch.
  • The minimum index version has been bumped to V_8_0_0 and all references in tests to index versions earlier to this have either been removed, or the test muted. This should mean that we can start wiping out production references to these versions, or replace them with placeholders, without affecting tests.
  • Any tests that were explicitly testing upgrades or migrations from 7.x index versions have been removed or muted. If it was unclear whether the test was strictly related to 7.x (since it likely was using a 7.x version constant) it was muted and annotated with @UpdateForV9 for later investigation.
  • The EQL correctness tests have been muted since they rely on restoring a snapshot of test data. This snapshot unfortunately is from Elasticsearch 7.x, so cannot be restored in a 9.0 cluster. A new 9.0-compatible snapshot will need to be created for these tests to be re-enabled. See EsEQLCorrectnessIT fails on Elasticsearch 9.0 #112572.

What this pull request does not include

  • Production code references to pre-8.0 index versions have not yet been removed. This will be done in a follow up.
  • The Lucene version remains 9.x. The upgrade to Lucene 10 is a separate effort.
  • The minimum transport version remains V_7_17_0. This will be bumped in a follow up.
  • The REST API version remains V_8. A constant has been added for V_9 but is not used. Also, the bootstrap check to ensure the current REST API version matches the current major version has been disabled. This will need to be re-enabled once the REST API version is incremented.
  • No functional changes. For all intents and purposes this behaves exactly like 8.x, with the exception of the removal of support for pre-8.0 indexes. Anything deprecated for removal in 9.0 still remains. This will be done in a follow up.
    Native ML code is still version 8.16.0. We do not yet have 9.0 snapshot builds of ml-cpp, so this will need to be updated in a follow up. (done in this PR)

# Conflicts:
#	build-tools-internal/src/integTest/groovy/org/elasticsearch/gradle/internal/test/rest/LegacyYamlRestCompatTestPluginFuncTest.groovy
# Conflicts:
#	.buildkite/pipelines/intake.yml
#	.buildkite/pipelines/periodic.yml
#	.ci/snapshotBwcVersions
@mark-vieira mark-vieira added :Delivery/Build Build or test infrastructure :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts test-windows Trigger CI checks on Windows labels Sep 5, 2024
Copy link
Contributor

@jakelandis jakelandis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -286,7 +266,17 @@ public List<Version> getUnreleasedWireCompatible() {
}

public Version getMinimumWireCompatibleVersion() {
return MINIMUM_WIRE_COMPATIBLE_VERSION;
// Determine minimum wire compatible version from list of known versions.
// Current BWC policy states the minimum wire compatible version is the last minor release or the previous major version.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo or -> of

.getUnreleased()
.stream()
.filter(v -> v.getMajor() == currentMajor - 1)
.min(Comparator.reverseOrder())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

max(Comparator.naturalOrder()) is more understandable

(v('8.1.0')): new UnreleasedVersionInfo(v('8.1.0'), 'main', ':distribution')
]
bwc.wireCompatible == [v('7.17.0'), v('8.0.0'), v('8.1.0')]
bwc.indexCompatible == osFiltered([v('7.14.0'), v('7.14.1'), v('7.14.2'), v('7.15.0'), v('7.15.1'), v('7.15.2'), v('7.16.0'), v('7.16.1'), v('7.16.2'), v('7.17.0'), v('8.0.0'), v('8.1.0')])
bwc.indexCompatible == [v('7.14.0'), v('7.14.1'), v('7.14.2'), v('7.15.0'), v('7.15.1'), v('7.15.2'), v('7.16.0'), v('7.16.1'), v('7.16.2'), v('7.17.0'), v('8.0.0'), v('8.1.0')]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what's going on here, but I assume it's deliberate that indexCompatible /minWireCompatible is still referring to v7 versions

@brianseeders
Copy link
Contributor

@elasticmachine merge upstream

@jakelandis
Copy link
Contributor

@elasticmachine merge upstream

@jakelandis
Copy link
Contributor

@elasticmachine merge upstream

@elasticmachine
Copy link
Collaborator

merge conflict between base and head

@mark-vieira mark-vieira merged commit 4ce661c into main Sep 11, 2024
20 checks passed
@mark-vieira mark-vieira deleted the es-9.0-bump branch September 11, 2024 16:40
v1v added a commit to v1v/elasticsearch that referenced this pull request Sep 12, 2024
…tion-ironbank-ubi

* upstream/main: (302 commits)
  Deduplicate BucketOrder when deserializing (elastic#112707)
  Introduce test utils for ingest pipelines (elastic#112733)
  [Test] Account for auto-repairing for shard gen file (elastic#112778)
  Do not throw in task enqueued by CancellableRunner (elastic#112780)
  Mute org.elasticsearch.script.StatsSummaryTests testEqualsAndHashCode elastic#112439
  Mute org.elasticsearch.repositories.blobstore.testkit.integrity.RepositoryVerifyIntegrityIT testTransportException elastic#112779
  Use a dedicated test executor in MockTransportService (elastic#112748)
  Estimate segment field usages (elastic#112760)
  (Doc+) Inference Pipeline ignores Mapping Analyzers (elastic#112522)
  Fix verifyVersions task (elastic#112765)
  (Doc+) Terminating Exit Codes (elastic#112530)
  (Doc+) CAT Nodes default columns (elastic#112715)
  [DOCS] Augment installation warnings (elastic#112756)
  Mute org.elasticsearch.repositories.blobstore.testkit.integrity.RepositoryVerifyIntegrityIT testCorruption elastic#112769
  Bump Elasticsearch to a minimum of JDK 21 (elastic#112252)
  ESQL: Compute support for filtering ungrouped aggs (elastic#112717)
  Bump Elasticsearch version to 9.0.0 (elastic#112570)
  add CDR related data streams to kibana_system priviliges (elastic#112655)
  Support widening of numeric types in union-types (elastic#112610)
  Introduce data stream options and failure store configuration classes (elastic#109515)
  ...
cbuescher added a commit that referenced this pull request Dec 19, 2024
This test has been removed with #112570 but with
the coming read-only support for v7 we need it back.
cbuescher added a commit to cbuescher/elasticsearch that referenced this pull request Dec 19, 2024
This test has been removed with elastic#112570 but with
the coming read-only support for v7 we need it back.
cbuescher added a commit that referenced this pull request Dec 19, 2024
This test has been removed with #112570 but with
the coming read-only support for v7 we need it back.
cbuescher added a commit to cbuescher/elasticsearch that referenced this pull request Dec 19, 2024
This test was removed with elastic#112570 but needs to be added back with the
read-only support for v7 indices.
cbuescher added a commit that referenced this pull request Dec 19, 2024
This test was removed with #112570 but needs to be added back with the
read-only support for v7 indices.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Delivery/Build Build or test infrastructure :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts >refactoring Team:Delivery Meta label for Delivery team test-full-bwc Trigger full BWC version matrix tests test-update-serverless v8.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants