Skip to content

Commit

Permalink
Merge branch 'master' into breadth_first-support-for-significant-term…
Browse files Browse the repository at this point in the history 
…-agg

* master: (528 commits)
  Register Azure max_retries setting (elastic#35286)
  add version 6.4.4
  [Docs] Add painless context details for bucket_script (elastic#35142)
  Upgrade jline to 3.8.2 (elastic#35288)
  SQL: new SQL CLI logo (elastic#35261)
  Logger: Merge ESLoggerFactory into Loggers (elastic#35146)
  Docs: Add section about range query for range type (elastic#35222)
  [ILM] change remove-policy-from-index http method from DELETE to POST (elastic#35268)
  [CCR] Forgot missing return statement,
  SQL: Fix null handling for AND and OR in SELECT (elastic#35277)
  [TEST] Mute ChangePolicyForIndexIT#testChangePolicyForIndex
  Serialize ignore_throttled also to 6.6 after backport
  Check for java 11 in buildSrc (elastic#35260)
  [TEST] increase await timeout in RemoteClusterConnectionTests
  Add missing up-to-date configuration (elastic#35255)
  Adapt Lucene BWC version
  SQL: Introduce Coalesce function (elastic#35253)
  Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (elastic#35224)
  Fix failing ICU tests (elastic#35207)
  Prevent throttled indices to be searched through wildcards by default (elastic#34354)
  ...
  • Loading branch information
@matarrese
matarrese committed Nov 6, 2018
2 parents a9f5e3e + a4b26fe commit e77b9ec
Show file tree
Hide file tree
Showing 2,248 changed files with 57,412 additions and 14,465 deletions.
1 change: 1 addition & 0 deletions .ci/java-versions.properties
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ ES_BUILD_JAVA=java11
ES_RUNTIME_JAVA=java8
GRADLE_TASK=build

GRADLE_EXTRA_ARGS=-Dtests.bwc.refspec=elastic/index-lifecycle-6.x
47 changes: 11 additions & 36 deletions TESTING.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -229,34 +229,6 @@ Pass arbitrary jvm arguments.
./gradlew test -Dtests.jvm.argline="-Djava.security.debug=access,failure"
------------------------------

== Backwards Compatibility Tests

Running backwards compatibility tests is disabled by default since it
requires a release version of elasticsearch to be present on the test system.
To run backwards compatibility tests untar or unzip a release and run the tests
with the following command:

---------------------------------------------------------------------------
./gradlew test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.bwc.path=/path/to/elasticsearch -Dtests.security.manager=false
---------------------------------------------------------------------------

Note that backwards tests must be run with security manager disabled.
If the elasticsearch release is placed under `./backwards/elasticsearch-x.y.z` the path
can be omitted:

---------------------------------------------------------------------------
./gradlew test -Dtests.filter="@backwards" -Dtests.bwc.version=x.y.z -Dtests.security.manager=false
---------------------------------------------------------------------------

To setup the bwc test environment execute the following steps (provided you are
already in your elasticsearch clone):

---------------------------------------------------------------------------
$ mkdir backwards && cd backwards
$ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.tar.gz
$ tar -xzf elasticsearch-1.2.1.tar.gz
---------------------------------------------------------------------------

== Running verification tasks

To run all verification tasks, including static checks, unit tests, and integration tests:
Expand Down Expand Up @@ -554,25 +526,28 @@ environment variable.
== Testing backwards compatibility

Backwards compatibility tests exist to test upgrading from each supported version
to the current version. To run all backcompat tests use:
to the current version. To run them all use:

-------------------------------------------------
./gradlew bwcTest
-------------------------------------------------

A specific version can be tested as well. For example, to test backcompat with
A specific version can be tested as well. For example, to test bwc with
version 5.3.2 run:

-------------------------------------------------
./gradlew v5.3.2#bwcTest
-------------------------------------------------

When running `./gradlew check`, some minimal backcompat checks are run. Which version
is tested depends on the branch. On master, this will test against the current
stable branch. On the stable branch, it will test against the latest release
branch. Finally, on a release branch, it will test against the most recent release.
Tests are ran for versions that are not yet released but with which the current version will be compatible with.
These are automatically checked out and built from source.
See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection]
and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle]
for more information.

When running `./gradlew check`, minimal bwc checks are also run against compatible versions that are not yet released.

=== BWC Testing against a specific remote/branch
==== BWC Testing against a specific remote/branch

Sometimes a backward compatibility change spans two versions. A common case is a new functionality
that needs a BWC bridge in an unreleased versioned of a release branch (for example, 5.x).
Expand All @@ -597,7 +572,7 @@ will contain your change.
. Push both branches to your remote repository.
. Run the tests with `./gradlew check -Dtests.bwc.remote=${remote} -Dtests.bwc.refspec.5.x=index_req_bwc_5.x`.

== Skip fetching latest
==== Skip fetching latest

For some BWC testing scenarios, you want to use the local clone of the
repository without fetching latest. For these use cases, you can set the system
Expand Down
8 changes: 2 additions & 6 deletions benchmarks/src/main/java/org/elasticsearch/benchmark/routing/allocation/Allocators.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,6 @@ public final class Allocators {
private static class NoopGatewayAllocator extends GatewayAllocator {
public static final NoopGatewayAllocator INSTANCE = new NoopGatewayAllocator();

protected NoopGatewayAllocator() {
super(Settings.EMPTY);
}

@Override
public void applyStartedShards(RoutingAllocation allocation, List<ShardRouting> startedShards) {
// noop
Expand Down Expand Up @@ -79,7 +75,7 @@ public static AllocationService createAllocationService(Settings settings) throw

public static AllocationService createAllocationService(Settings settings, ClusterSettings clusterSettings) throws
InvocationTargetException, NoSuchMethodException, InstantiationException, IllegalAccessException {
return new AllocationService(settings,
return new AllocationService(
defaultAllocationDeciders(settings, clusterSettings),
NoopGatewayAllocator.INSTANCE, new BalancedShardsAllocator(settings), EmptyClusterInfoService.INSTANCE);
}
Expand All @@ -88,7 +84,7 @@ public static AllocationDeciders defaultAllocationDeciders(Settings settings, Cl
IllegalAccessException, InvocationTargetException, InstantiationException, NoSuchMethodException {
Collection<AllocationDecider> deciders =
ClusterModule.createAllocationDeciders(settings, clusterSettings, Collections.emptyList());
return new AllocationDeciders(settings, deciders);
return new AllocationDeciders(deciders);

}

Expand Down
69 changes: 27 additions & 42 deletions build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ if (properties.get("org.elasticsearch.acceptScanTOS", "false") == "true") {
// common maven publishing configuration
subprojects {
group = 'org.elasticsearch'
version = VersionProperties.elasticsearch.toString()
version = VersionProperties.elasticsearch
description = "Elasticsearch subproject ${project.path}"
}

Expand Down Expand Up @@ -103,10 +103,6 @@ subprojects {
* in a branch if there are only betas and rcs in the branch so we have
* *something* to test against. */
VersionCollection versions = new VersionCollection(file('server/src/main/java/org/elasticsearch/Version.java').readLines('UTF-8'))
if (versions.currentVersion != VersionProperties.elasticsearch) {
throw new GradleException("The last version in Versions.java [${versions.currentVersion}] does not match " +
"VersionProperties.elasticsearch [${VersionProperties.elasticsearch}]")
}

// build metadata from previous build, contains eg hashes for bwc builds
String buildMetadataValue = System.getenv('BUILD_METADATA')
Expand Down Expand Up @@ -140,26 +136,16 @@ task verifyVersions {
if (gradle.startParameter.isOffline()) {
throw new GradleException("Must run in online mode to verify versions")
}
// Read the list from maven central
Node xml
// Read the list from maven central.
// Fetch the metadata an parse the xml into Version instances because it's more straight forward here
// rather than bwcVersion ( VersionCollection ).
new URL('https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch/maven-metadata.xml').openStream().withStream { s ->
xml = new XmlParser().parse(s)
}
Set<Version> knownVersions = new TreeSet<>(xml.versioning.versions.version.collect { it.text() }.findAll { it ==~ /\d+\.\d+\.\d+/ }.collect { Version.fromString(it) })

// Limit the known versions to those that should be index compatible, and are not future versions
knownVersions = knownVersions.findAll { it.major >= bwcVersions.currentVersion.major - 1 && it.before(VersionProperties.elasticsearch) }

/* Limit the listed versions to those that have been marked as released.
* Versions not marked as released don't get the same testing and we want
* to make sure that we flip all unreleased versions to released as soon
* as possible after release. */
Set<Version> actualVersions = new TreeSet<>(bwcVersions.indexCompatible.findAll { false == it.snapshot })

// Finally, compare!
if (knownVersions.equals(actualVersions) == false) {
throw new GradleException("out-of-date released versions\nActual :" + actualVersions + "\nExpected:" + knownVersions +
"\nUpdate Version.java. Note that Version.CURRENT doesn't count because it is not released.")
bwcVersions.compareToAuthoritative(
new XmlParser().parse(s)
.versioning.versions.version
.collect { it.text() }.findAll { it ==~ /\d+\.\d+\.\d+/ }
.collect { Version.fromString(it) }
)
}
}
}
Expand Down Expand Up @@ -251,20 +237,17 @@ subprojects {
"org.elasticsearch.plugin:percolator-client:${version}": ':modules:percolator',
"org.elasticsearch.plugin:rank-eval-client:${version}": ':modules:rank-eval',
]

bwcVersions.snapshotProjectNames.each { snapshotName ->
Version snapshot = bwcVersions.getSnapshotForProject(snapshotName)
if (snapshot != null ) {
String snapshotProject = ":distribution:bwc:${snapshotName}"
project(snapshotProject).ext.bwcVersion = snapshot
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch:${snapshot}"] = snapshotProject
if (snapshot.onOrAfter('6.3.0')) {
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch-oss:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch-oss:${snapshot}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch-oss:${snapshot}"] = snapshotProject
}
// substitute unreleased versions with projects that check out and build locally
bwcVersions.forPreviousUnreleased { VersionCollection.UnreleasedVersionInfo unreleasedVersion ->
Version unreleased = unreleasedVersion.version
String snapshotProject = ":distribution:bwc:${unreleasedVersion.gradleProjectName}"
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch:${unreleased}"] = snapshotProject
if (unreleased.onOrAfter('6.3.0')) {
ext.projectSubstitutions["org.elasticsearch.distribution.deb:elasticsearch-oss:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.rpm:elasticsearch-oss:${unreleased}"] = snapshotProject
ext.projectSubstitutions["org.elasticsearch.distribution.zip:elasticsearch-oss:${unreleased}"] = snapshotProject
}
}

Expand Down Expand Up @@ -299,7 +282,7 @@ subprojects {
// other packages (e.g org.elasticsearch.client) will point to server rather than
// their own artifacts.
if (project.plugins.hasPlugin(BuildPlugin) || project.plugins.hasPlugin(PluginBuildPlugin)) {
String artifactsHost = VersionProperties.elasticsearch.isSnapshot() ? "https://snapshots.elastic.co" : "https://artifacts.elastic.co"
String artifactsHost = VersionProperties.elasticsearch.endsWith("-SNAPSHOT") ? "https://snapshots.elastic.co" : "https://artifacts.elastic.co"
Closure sortClosure = { a, b -> b.group <=> a.group }
Closure depJavadocClosure = { shadowed, dep ->
if (dep.group == null || false == dep.group.startsWith('org.elasticsearch')) {
Expand Down Expand Up @@ -572,11 +555,13 @@ wrapper {
}
}

/* Remove assemble/dependenciesInfo on all qa projects because we don't need to publish
* artifacts for them. */
gradle.projectsEvaluated {
subprojects {
if (project.path.startsWith(':qa')) {
/*
* Remove assemble/dependenciesInfo on all qa projects because we don't
* need to publish artifacts for them.
*/
if (project.name.equals('qa') || project.path.contains(':qa:')) {
Task assemble = project.tasks.findByName('assemble')
if (assemble) {
assemble.enabled = false
Expand Down
84 changes: 55 additions & 29 deletions buildSrc/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,46 +41,33 @@ if (project == rootProject) {
* Propagating version.properties to the rest of the build *
*****************************************************************************/

Properties props = new Properties()
props.load(project.file('version.properties').newDataInputStream())
version = props.getProperty('elasticsearch')
boolean snapshot = "true".equals(System.getProperty("build.snapshot", "true"));
if (snapshot) {
// we update the version property to reflect if we are building a snapshot or a release build
// we write this back out below to load it in the Build.java which will be shown in rest main action
// to indicate this being a snapshot build or a release build.
version += "-SNAPSHOT"
props.put("elasticsearch", version);
}

File tempPropertiesFile = new File(project.buildDir, "version.properties")
task writeVersionProperties {
inputs.properties(props)
outputs.file(tempPropertiesFile)
// we update the version property to reflect if we are building a snapshot or a release build
// we write this back out below to load it in the Build.java which will be shown in rest main action
// to indicate this being a snapshot build or a release build.
File propsFile = project.file('version.properties')
Properties props = VersionPropertiesLoader.loadBuildSrcVersion(propsFile)
version = props.getProperty("elasticsearch")
processResources {
inputs.file(propsFile)
// We need to be explicit with the version because we add snapshot and qualifier to it based on properties
inputs.property("dynamic_elasticsearch_version", props.getProperty("elasticsearch"))
doLast {
OutputStream stream = Files.newOutputStream(tempPropertiesFile.toPath());
Writer writer = file("$destinationDir/version.properties").newWriter()
try {
props.store(stream, "UTF-8");
props.store(writer, "Generated version properties")
} finally {
stream.close();
writer.close()
}
}
}

processResources {
dependsOn writeVersionProperties
from tempPropertiesFile
}


if (JavaVersion.current() < JavaVersion.VERSION_1_10) {
throw new GradleException('At least Java 10 is required to build elasticsearch gradle tools')
}

/*****************************************************************************
* Java version *
*****************************************************************************/

if (JavaVersion.current() < JavaVersion.VERSION_11) {
throw new GradleException('At least Java 11 is required to build elasticsearch gradle tools')
}
// Gradle 4.10 does not support setting this to 11 yet
targetCompatibility = "10"
sourceCompatibility = "10"
Expand Down Expand Up @@ -232,3 +219,42 @@ if (project != rootProject) {
generatePomFileForPluginMavenPublication.enabled = false
}
}

// Define this here because we need it early.
class VersionPropertiesLoader {
static Properties loadBuildSrcVersion(File input) throws IOException {
Properties props = new Properties();
InputStream is = new FileInputStream(input)
try {
props.load(is)
} finally {
is.close()
}
loadBuildSrcVersion(props, System.getProperties())
return props
}

protected static void loadBuildSrcVersion(Properties loadedProps, Properties systemProperties) {
String elasticsearch = loadedProps.getProperty("elasticsearch")
if (elasticsearch == null) {
throw new IllegalStateException("Elasticsearch version is missing from properties.")
}
if (elasticsearch.matches("[0-9]+\\.[0-9]+\\.[0-9]+") == false) {
throw new IllegalStateException(
"Expected elasticsearch version to be numbers only of the form X.Y.Z but it was: " +
elasticsearch
)
}
String qualifier = systemProperties.getProperty("build.version_qualifier", "alpha1");
if (qualifier.isEmpty() == false) {
if (qualifier.matches("(alpha|beta|rc)\\d+") == false) {
throw new IllegalStateException("Invalid qualifier: " + qualifier)
}
elasticsearch += "-" + qualifier
}
if ("true".equals(systemProperties.getProperty("build.snapshot", "true"))) {
elasticsearch += "-SNAPSHOT"
}
loadedProps.put("elasticsearch", elasticsearch)
}
}
10 changes: 2 additions & 8 deletions buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -696,18 +696,12 @@ class BuildPlugin implements Plugin<Project> {
jarTask.destinationDir = new File(project.buildDir, 'distributions')
// fixup the jar manifest
jarTask.doFirst {
final Version versionWithoutSnapshot = new Version(
VersionProperties.elasticsearch.major,
VersionProperties.elasticsearch.minor,
VersionProperties.elasticsearch.revision,
VersionProperties.elasticsearch.suffix,
false)
// this doFirst is added before the info plugin, therefore it will run
// after the doFirst added by the info plugin, and we can override attributes
jarTask.manifest.attributes(
'X-Compile-Elasticsearch-Version': versionWithoutSnapshot,
'X-Compile-Elasticsearch-Version': VersionProperties.elasticsearch.replace("-SNAPSHOT", ""),
'X-Compile-Lucene-Version': VersionProperties.lucene,
'X-Compile-Elasticsearch-Snapshot': VersionProperties.elasticsearch.isSnapshot(),
'X-Compile-Elasticsearch-Snapshot': VersionProperties.isElasticsearchSnapshot(),
'Build-Date': ZonedDateTime.now(ZoneOffset.UTC),
'Build-Java-Version': project.compilerJavaVersion)
if (jarTask.manifest.attributes.containsKey('Change') == false) {
Expand Down
Loading

0 comments on commit e77b9ec

Please sign in to comment.