Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement VersionCollection in Java #34050

Merged
merged 25 commits into from
Nov 1, 2018
Merged

Implement VersionCollection in Java #34050

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
0e475b0
Add test for 7.0.0 index compatability
alpar-t Sep 21, 2018
17b9734
Add java unused implementation of VersionCollection
alpar-t Sep 24, 2018
8ab7df8
Start using Java implementation of version collection
alpar-t Sep 25, 2018
b08b3b6
Self review
alpar-t Sep 25, 2018
bd03af5
Determine unreleased versions by gruping major and minor versions
alpar-t Sep 27, 2018
f0ce80f
Check all versions not just the index compatible ones
alpar-t Sep 27, 2018
4f5ef6d
remove unused inport
alpar-t Sep 27, 2018
ca79ab9
rename version collection
alpar-t Sep 28, 2018
fbf8f3e
Move version check core logic in version collection
alpar-t Sep 28, 2018
c72cc95
improove docs
alpar-t Sep 28, 2018
ff69345
Remove unused import
alpar-t Sep 28, 2018
4d54ac8
remove unused index property
alpar-t Sep 28, 2018
d01c914
Merge remote-tracking branch 'origin/master' into remove-version-qual…
alpar-t Oct 1, 2018
6962565
Remove TreeSet
alpar-t Oct 1, 2018
555ef78
Fix javadoc
alpar-t Oct 1, 2018
a27505f
Merge branch 'master' into remove-version-qualifier-snapshot-build
alpar-t Oct 4, 2018
c2041d2
Merge remote-tracking branch 'origin/master' into remove-version-qual…
alpar-t Oct 5, 2018
5d11cd4
PR review
alpar-t Oct 30, 2018
dc328a7
Change indent so git diff is less confused
alpar-t Oct 30, 2018
179e54c
PR review
alpar-t Oct 30, 2018
7d4a144
Merge remote-tracking branch 'origin/master' into remove-version-qual…
alpar-t Oct 30, 2018
7252607
fix javadoc
alpar-t Oct 30, 2018
8032307
unused imports
alpar-t Oct 30, 2018
c576684
pull out changes to ASCIDOC
alpar-t Oct 31, 2018
47a7b79
PR review
alpar-t Oct 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 9 additions & 36 deletions TESTING.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,34 +246,6 @@ Pass arbitrary jvm arguments.
./gradlew test -Dtests.jvm.argline="-Djava.security.debug=access,failure"
------------------------------

== Backwards Compatibility Tests
alpar-t marked this conversation as resolved.
Show resolved Hide resolved

Running backwards compatibility tests is disabled by default since it
requires a release version of elasticsearch to be present on the test system.
To run backwards compatibility tests untar or unzip a release and run the tests
with the following command:

---------------------------------------------------------------------------
./gradlew test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.bwc.path=/path/to/elasticsearch -Dtests.security.manager=false
---------------------------------------------------------------------------

Note that backwards tests must be run with security manager disabled.
If the elasticsearch release is placed under `./backwards/elasticsearch-x.y.z` the path
can be omitted:

---------------------------------------------------------------------------
./gradlew test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.security.manager=false
---------------------------------------------------------------------------

To setup the bwc test environment execute the following steps (provided you are
already in your elasticsearch clone):

---------------------------------------------------------------------------
$ mkdir backwards && cd backwards
$ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.tar.gz
$ tar -xzf elasticsearch-1.2.1.tar.gz
---------------------------------------------------------------------------

== Running verification tasks

To run all verification tasks, including static checks, unit tests, and integration tests:
Expand Down Expand Up @@ -571,25 +543,26 @@ environment variable.
== Testing backwards compatibility

Backwards compatibility tests exist to test upgrading from each supported version
to the current version. To run all backcompat tests use:
to the current version. To run them all use:

-------------------------------------------------
./gradlew bwcTest
-------------------------------------------------

A specific version can be tested as well. For example, to test backcompat with
A specific version can be tested as well. For example, to test bwc with
version 5.3.2 run:

-------------------------------------------------
./gradlew v5.3.2#bwcTest
-------------------------------------------------

When running `./gradlew check`, some minimal backcompat checks are run. Which version
is tested depends on the branch. On master, this will test against the current
stable branch. On the stable branch, it will test against the latest release
branch. Finally, on a release branch, it will test against the most recent release.
Tests are ran for versions that are not yet released but with which the current version will be compatible with.
These are automatically checked out and build from source.
See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection] for more information.

When running `./gradlew check`, minimal bwc checks are also run against compatible versions that are not yet released.

=== BWC Testing against a specific remote/branch
==== BWC Testing against a specific remote/branch

Sometimes a backward compatibility change spans two versions. A common case is a new functionality
that needs a BWC bridge in an unreleased versioned of a release branch (for example, 5.x).
Expand All @@ -614,7 +587,7 @@ will contain your change.
. Push both branches to your remote repository.
. Run the tests with `./gradlew check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec.5.x=index_req_bwc_5.x`.

== Skip fetching latest
==== Skip fetching latest

For some BWC testing scenarios, you want to use the local clone of the
repository without fetching latest. For these use cases, you can set the system
Expand Down
53 changes: 17 additions & 36 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,6 @@ subprojects {
* in a branch if there are only betas and rcs in the branch so we have
* *something* to test against. */
VersionCollection versions = new VersionCollection(file('server/src/main/java/org/elasticsearch/Version.java').readLines('UTF-8'))
if (versions.currentVersion != VersionProperties.elasticsearch) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check is moved into version collections.
I tried to move checks and decisions there for better visibility.

throw new GradleException("The last version in Versions.java [${versions.currentVersion}] does not match " +
"VersionProperties.elasticsearch [${VersionProperties.elasticsearch}]")
}

// build metadata from previous build, contains eg hashes for bwc builds
String buildMetadataValue = System.getenv('BUILD_METADATA')
Expand Down Expand Up @@ -141,25 +137,13 @@ task verifyVersions {
throw new GradleException("Must run in online mode to verify versions")
}
// Read the list from maven central
Node xml
new URL('https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch/maven-metadata.xml').openStream().withStream { s ->
xml = new XmlParser().parse(s)
}
Set<Version> knownVersions = new TreeSet<>(xml.versioning.versions.version.collect { it.text() }.findAll { it ==~ /\d+\.\d+\.\d+/ }.collect { Version.fromString(it) })

// Limit the known versions to those that should be index compatible, and are not future versions
knownVersions = knownVersions.findAll { it.major >= bwcVersions.currentVersion.major - 1 && it.before(VersionProperties.elasticsearch) }

/* Limit the listed versions to those that have been marked as released.
* Versions not marked as released don't get the same testing and we want
* to make sure that we flip all unreleased versions to released as soon
* as possible after release. */
Set<Version> actualVersions = new TreeSet<>(bwcVersions.indexCompatible.findAll { false == it.snapshot })

// Finally, compare!
if (knownVersions.equals(actualVersions) == false) {
throw new GradleException("out-of-date released versions\nActual :" + actualVersions + "\nExpected:" + knownVersions +
"\nUpdate Version.java. Note that Version.CURRENT doesn't count because it is not released.")
bwcVersions.compareToAuthoritative(
alpar-t marked this conversation as resolved.
Show resolved Hide resolved
new XmlParser().parse(s)
.versioning.versions.version
.collect { it.text() }.findAll { it ==~ /\d+\.\d+\.\d+/ }
.collect { Version.fromString(it) }
)
}
}
}
Expand Down Expand Up @@ -251,20 +235,17 @@ subprojects {
"org.elasticsearch.plugin:percolator-client:${version}": ':modules:percolator',
"org.elasticsearch.plugin:rank-eval-client:${version}": ':modules:rank-eval',
]

bwcVersions.snapshotProjectNames.each { snapshotName ->
Version snapshot = bwcVersions.getSnapshotForProject(snapshotName)
if (snapshot != null ) {
String snapshotProject = ":distribution:bwc:${snapshotName}"
project(snapshotProject).ext.bwcVersion = snapshot
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch:${snapshot}"] = snapshotProject
if (snapshot.onOrAfter('6.3.0')) {
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch-oss:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch-oss:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch-oss:${snapshot}"] = snapshotProject
}
// substitute unreleased versions with projects that check out and build locally
bwcVersions.forPreviousUnreleased { VersionCollection.UnreleasedVersionDescription unreleasedVersion ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a foreach closure? I'm fine with the name change to "unreleased" instead of "snapshot", but can't we just have bwcVersions.unreleased as a collection that we can iterate here as before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to avoid creating a collection and returning it when the calling code is really interested in iteration. It could have to be a collection of UnreleasedVersionDescription built each time, or build once cached. I taught this is simpler and easier to read and fits well with Gradle DSL.

We could simply call it unreleased if you think the extra detail in the name makes it actually more confusing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the collection should be created once in the constructor, so there shouldn't be any concern about returning it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel that having the collections as different views of the same data makes the class harder to understand and in this particular case, it won't make a difference in performance large enough to be worthwhile. Do you see other advantages in doing so besides not having to compute it multiple times ?

Version unreleased = unreleasedVersion.version
String snapshotProject = ":distribution:bwc:${unreleasedVersion.gradleProjectName}"
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch:${unreleased}"] = snapshotProject
if (unreleased.onOrAfter('6.3.0')) {
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch-oss:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch-oss:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch-oss:${unreleased}"] = snapshotProject
}
}

Expand Down
Loading