Skip to content

Commit

Permalink
Merge branch 'master' into feature/reader-context
Browse files Browse the repository at this point in the history
  • Loading branch information
dnhatn committed Feb 28, 2020
2 parents aade8b6 + 1d4a9bb commit cdbfb67
Show file tree
Hide file tree
Showing 347 changed files with 7,122 additions and 2,900 deletions.
10 changes: 10 additions & 0 deletions .github/workflows/gradle-wrapper-validation.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: "Validate Gradle Wrapper"
on: [push]

jobs:
validation:
name: "Validation"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: gradle/wrapper-validation-action@v1
60 changes: 42 additions & 18 deletions TESTING.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -270,34 +270,49 @@ code 137 (128 + 9, where 9 means SIGKILL).

== Testing the REST layer

The available integration tests make use of the java API to communicate with
the elasticsearch nodes, using the internal binary transport (port 9300 by
default).
The REST layer is tested through specific tests that are shared between all
the elasticsearch official clients and consist of YAML files that describe the
The REST layer is tested through specific tests that are executed against
a cluster that is configured and initialized via Gradle. The tests
themselves can be written in either Java or with a YAML based DSL.

YAML based REST tests should be preferred since these are shared between all
the elasticsearch official clients. The YAML based tests describe the
operations to be executed and the obtained results that need to be tested.

The YAML files support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification]
The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification]
In order to run the the YAML tests, the relevant API specification needs
to be on the test classpath. Any gradle project that has support for REST
tests will get the primary API on it's class path. However, to better support
Gradle incremental builds, it is recommended to explicitly declare which
parts of the API the tests depend upon.

For example:
---------------------------------------------------------------------------
restResources {
restApi {
includeCore '_common', 'indices', 'index', 'cluster', 'nodes', 'get', 'ingest'
}
}
---------------------------------------------------------------------------

YAML tests that include x-pack specific APIs need to explicitly declare
which APIs are required through a similar `includeXpack` configuration.

The REST tests are run automatically when executing the "./gradlew check" command. To run only the
REST tests use the following command:

---------------------------------------------------------------------------
./gradlew :distribution:archives:integ-test-zip:integTest \
-Dtests.class="org.elasticsearch.test.rest.*Yaml*IT"
./gradlew :distribution:archives:integ-test-zip:integTestRunner \
--tests "org.elasticsearch.test.rest.IntegTestZipClientYamlTestSuiteIT"
---------------------------------------------------------------------------

A specific test case can be run with
A specific test case can be run with the following command:

---------------------------------------------------------------------------
./gradlew :distribution:archives:integ-test-zip:integTest \
-Dtests.class="org.elasticsearch.test.rest.*Yaml*IT" \
-Dtests.method="test {p0=cat.shards/10_basic/Help}"
./gradlew ':distribution:archives:integ-test-zip:integTestRunner' \
--tests "org.elasticsearch.test.rest.IntegTestZipClientYamlTestSuiteIT" \
-Dtests.method="test {p0=cat.segments/10_basic/Help}"
---------------------------------------------------------------------------

`*Yaml*IT` are the executable test classes that runs all the
yaml suites available within the `rest-api-spec` folder.

The REST tests support all the options provided by the randomized runner, plus the following:

* `tests.rest[true|false]`: determines whether the REST tests need to be run (default) or not.
Expand Down Expand Up @@ -459,6 +474,14 @@ version 5.3.2 run:
./gradlew v5.3.2#bwcTest
-------------------------------------------------

Use -Dtest.class and -Dtests.method to run a specific bwcTest test.
For example to run a specific tests from the x-pack rolling upgrade from 7.7.0:
-------------------------------------------------
./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \
-Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
-Dtests.method="test {p0=*/40_ml_datafeed_crud/*}"
-------------------------------------------------

Tests are ran for versions that are not yet released but with which the current version will be compatible with.
These are automatically checked out and built from source.
See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection]
Expand Down Expand Up @@ -512,8 +535,9 @@ There are multiple base classes for tests:
* **`ESIntegTestCase`**: An integration test case that creates a cluster that
might have multiple nodes.
* **`ESRestTestCase`**: An integration tests that interacts with an external
cluster via the REST API. For instance, YAML tests run via sub classes of
`ESRestTestCase`.
cluster via the REST API. This is used for Java based REST tests.
* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run
YAML based REST tests.

=== Good practices

Expand Down Expand Up @@ -643,4 +667,4 @@ please see https://esrally.readthedocs.io/en/stable/[Rally's documentation].

The Elasticsearch docs are in AsciiDoc format. You can test and build the docs
locally using the Elasticsearch documentation build process. See
https://github.com/elastic/docs.
https://github.com/elastic/docs.
4 changes: 2 additions & 2 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ subprojects {
// Projects that should be formatted and checked with Spotless are
// listed here, by project path. Once the number of formatted projects
// is greater than the number of unformatted projects, this can be
// switched to an exclude list, and eventualy removed completely.
// switched to an exclude list, and eventually removed completely.
def projectPathsToFormat = [
':benchmarks',
':build-tools',
Expand Down Expand Up @@ -193,7 +193,7 @@ task verifyVersions {
throw new GradleException("Must run in online mode to verify versions")
}
// Read the list from maven central.
// Fetch the metadata an parse the xml into Version instances because it's more straight forward here
// Fetch the metadata and parse the xml into Version instances because it's more straight forward here
// rather than bwcVersion ( VersionCollection ).
new URL('https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch/maven-metadata.xml').openStream().withStream { s ->
bwcVersions.compareToAuthoritative(
Expand Down
15 changes: 15 additions & 0 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,9 @@ sourceSets {
compileMinimumRuntimeJava {
targetCompatibility = 8
sourceCompatibility = 8
tasks.withType(JavaCompile).configureEach {
options.encoding = 'UTF-8'
}
}

jar {
Expand Down Expand Up @@ -196,6 +199,18 @@ if (project != rootProject) {
}
}

// Track reaper jar as a test input using runtime classpath normalization strategy
tasks.withType(Test).configureEach {
inputs.files(configurations.reaper).withNormalizer(ClasspathNormalizer)
}

normalization {
runtimeClasspath {
// We already include the reaper jar as part of our runtime classpath. Ignore the copy in META-INF.
ignore('META-INF/reaper.jar')
}
}

// TODO: re-enable once randomizedtesting gradle code is published and removed from here
licenseHeaders.enabled = false

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
package org.elasticsearch.gradle.docker;

import org.elasticsearch.gradle.LoggedExec;
import org.gradle.api.DefaultTask;
import org.gradle.api.file.DirectoryProperty;
import org.gradle.api.file.RegularFileProperty;
import org.gradle.api.provider.ListProperty;
import org.gradle.api.provider.Property;
import org.gradle.api.tasks.Input;
import org.gradle.api.tasks.InputDirectory;
import org.gradle.api.tasks.OutputFile;
import org.gradle.api.tasks.PathSensitive;
import org.gradle.api.tasks.PathSensitivity;
import org.gradle.api.tasks.TaskAction;
import org.gradle.process.ExecOperations;
import org.gradle.workers.WorkAction;
import org.gradle.workers.WorkParameters;
import org.gradle.workers.WorkerExecutor;

import javax.inject.Inject;
import java.io.IOException;
import java.util.Arrays;

public class DockerBuildTask extends DefaultTask {
private final WorkerExecutor workerExecutor;
private final RegularFileProperty markerFile = getProject().getObjects().fileProperty();
private final DirectoryProperty dockerContext = getProject().getObjects().directoryProperty();

private String[] tags;
private boolean pull = true;
private boolean noCache = true;

@Inject
public DockerBuildTask(WorkerExecutor workerExecutor) {
this.workerExecutor = workerExecutor;
this.markerFile.set(getProject().getLayout().getBuildDirectory().file("markers/" + this.getName() + ".marker"));
}

@TaskAction
public void build() {
workerExecutor.noIsolation().submit(DockerBuildAction.class, params -> {
params.getDockerContext().set(dockerContext);
params.getMarkerFile().set(markerFile);
params.getTags().set(Arrays.asList(tags));
params.getPull().set(pull);
params.getNoCache().set(noCache);
});
}

@InputDirectory
@PathSensitive(PathSensitivity.RELATIVE)
public DirectoryProperty getDockerContext() {
return dockerContext;
}

@Input
public String[] getTags() {
return tags;
}

public void setTags(String[] tags) {
this.tags = tags;
}

@Input
public boolean isPull() {
return pull;
}

public void setPull(boolean pull) {
this.pull = pull;
}

@Input
public boolean isNoCache() {
return noCache;
}

public void setNoCache(boolean noCache) {
this.noCache = noCache;
}

@OutputFile
public RegularFileProperty getMarkerFile() {
return markerFile;
}

public abstract static class DockerBuildAction implements WorkAction<Parameters> {
private final ExecOperations execOperations;

@Inject
public DockerBuildAction(ExecOperations execOperations) {
this.execOperations = execOperations;
}

@Override
public void execute() {
LoggedExec.exec(execOperations, spec -> {
spec.executable("docker");

spec.args("build", getParameters().getDockerContext().get().getAsFile().getAbsolutePath());

if (getParameters().getPull().get()) {
spec.args("--pull");
}

if (getParameters().getNoCache().get()) {
spec.args("--no-cache");
}

getParameters().getTags().get().forEach(tag -> spec.args("--tag", tag));
});

try {
getParameters().getMarkerFile().getAsFile().get().createNewFile();
} catch (IOException e) {
throw new RuntimeException("Failed to create marker file", e);
}
}
}

interface Parameters extends WorkParameters {
DirectoryProperty getDockerContext();

RegularFileProperty getMarkerFile();

ListProperty<String> getTags();

Property<Boolean> getPull();

Property<Boolean> getNoCache();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
import org.gradle.api.Plugin;
import org.gradle.api.Project;
import org.gradle.api.Task;
import org.gradle.api.plugins.ExtraPropertiesExtension;
import org.gradle.api.provider.Provider;

import java.io.File;
Expand All @@ -13,21 +12,10 @@
/**
* Plugin providing {@link DockerSupportService} for detecting Docker installations and determining requirements for Docker-based
* Elasticsearch build tasks.
* <p>
* Additionally registers a task graph listener used to assert a compatible Docker installation exists when task requiring Docker are
* scheduled for execution. Tasks may declare a Docker requirement via an extra property. If a compatible Docker installation is not
* available on the build system an exception will be thrown prior to task execution.
*
* <pre>
* task myDockerTask {
* ext.requiresDocker = true
* }
* </pre>
*/
public class DockerSupportPlugin implements Plugin<Project> {
public static final String DOCKER_SUPPORT_SERVICE_NAME = "dockerSupportService";
public static final String DOCKER_ON_LINUX_EXCLUSIONS_FILE = ".ci/dockerOnLinuxExclusions";
public static final String REQUIRES_DOCKER_ATTRIBUTE = "requiresDocker";

@Override
public void apply(Project project) {
Expand All @@ -45,12 +33,13 @@ public void apply(Project project) {
)
);

// Ensure that if any tasks declare they require docker, we assert an available Docker installation exists
// Ensure that if we are trying to run any DockerBuildTask tasks, we assert an available Docker installation exists
project.getGradle().getTaskGraph().whenReady(graph -> {
List<String> dockerTasks = graph.getAllTasks().stream().filter(task -> {
ExtraPropertiesExtension ext = task.getExtensions().getExtraProperties();
return ext.has(REQUIRES_DOCKER_ATTRIBUTE) && (boolean) ext.get(REQUIRES_DOCKER_ATTRIBUTE);
}).map(Task::getPath).collect(Collectors.toList());
List<String> dockerTasks = graph.getAllTasks()
.stream()
.filter(task -> task instanceof DockerBuildTask)
.map(Task::getPath)
.collect(Collectors.toList());

if (dockerTasks.isEmpty() == false) {
dockerSupportServiceProvider.get().failIfDockerUnavailable(dockerTasks);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,16 +84,25 @@ public ListProperty<String> getIncludeXpack() {
@SkipWhenEmpty
@InputFiles
public FileTree getInputDir() {
xpackPatternSet.setIncludes(includeXpack.get().stream().map(prefix -> prefix + "*/**").collect(Collectors.toList()));
ConfigurableFileCollection fileCollection = getProject().files(xpackConfig.getAsFileTree().matching(xpackPatternSet));
if (BuildParams.isInternal()) {
corePatternSet.setIncludes(includeCore.get().stream().map(prefix -> prefix + "*/**").collect(Collectors.toList()));
fileCollection.plus(coreConfig.getAsFileTree().matching(corePatternSet));
} else {
fileCollection.plus(coreConfig);
FileTree coreFileTree = null;
FileTree xpackFileTree = null;
if (includeXpack.get().isEmpty() == false) {
xpackPatternSet.setIncludes(includeXpack.get().stream().map(prefix -> prefix + "*/**").collect(Collectors.toList()));
xpackFileTree = xpackConfig.getAsFileTree().matching(xpackPatternSet);
}
boolean projectHasYamlRestTests = projectHasYamlRestTests();
if (includeCore.get().isEmpty() == false || projectHasYamlRestTests) {
if (BuildParams.isInternal()) {
corePatternSet.setIncludes(includeCore.get().stream().map(prefix -> prefix + "*/**").collect(Collectors.toList()));
coreFileTree = coreConfig.getAsFileTree().matching(corePatternSet); // directory on disk
} else {
coreFileTree = coreConfig.getAsFileTree(); // jar file
}
}
ConfigurableFileCollection fileCollection = getProject().files(coreFileTree, xpackFileTree);

// if project has rest tests or the includes are explicitly configured execute the task, else NO-SOURCE due to the null input
return projectHasYamlRestTests() || includeCore.get().isEmpty() == false || includeXpack.get().isEmpty() == false
return projectHasYamlRestTests || includeCore.get().isEmpty() == false || includeXpack.get().isEmpty() == false
? fileCollection.getAsFileTree()
: null;
}
Expand Down Expand Up @@ -148,15 +157,13 @@ private boolean projectHasYamlRestTests() {
return false;
}
try {
if (testSourceResourceDir != null) {
return new File(testSourceResourceDir, "rest-api-spec/test").exists() == false
|| Files.walk(testSourceResourceDir.toPath().resolve("rest-api-spec/test"))
.anyMatch(p -> p.getFileName().toString().endsWith("yml"));
if (testSourceResourceDir != null && new File(testSourceResourceDir, "rest-api-spec/test").exists()) {
return Files.walk(testSourceResourceDir.toPath().resolve("rest-api-spec/test"))
.anyMatch(p -> p.getFileName().toString().endsWith("yml"));
}
if (testOutputResourceDir != null) {
return new File(testOutputResourceDir, "rest-api-spec/test").exists() == false
|| Files.walk(testOutputResourceDir.toPath().resolve("rest-api-spec/test"))
.anyMatch(p -> p.getFileName().toString().endsWith("yml"));
if (testOutputResourceDir != null && new File(testOutputResourceDir, "rest-api-spec/test").exists()) {
return Files.walk(testOutputResourceDir.toPath().resolve("rest-api-spec/test"))
.anyMatch(p -> p.getFileName().toString().endsWith("yml"));
}
} catch (IOException e) {
throw new IllegalStateException(String.format("Error determining if this project [%s] has rest tests.", getProject()), e);
Expand Down
Loading

0 comments on commit cdbfb67

Please sign in to comment.