Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #60630

jakelandis · 2020-08-03T22:03:08Z

For 1/2 the plugins in x-pack, the integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.

This includes the following projects:
async-search, autoscaling, ccr, enrich, eql, frozen-indicies,
data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml

A few of the more specialized qa projects within these plugins
have not been changed with this PR due to additional complexity which should
be addressed separately.

A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is).

related: #61802
related: #56841
related: #59939
related: #55896

…k_plugins

jakelandis · 2020-08-05T22:22:58Z

x-pack/plugin/eql/qa/common/build.gradle

@@ -9,3 +9,17 @@ dependencies {
  // TOML parser for EqlActionIT tests
  api 'io.ous:jtoml:2.0.0'
 }
+


I moved some test only dependencies out of the main source set and into the test source set and added expose it as a test artifact.

What's the reasoning behind that? If we share code with other projects doing so w/o sharing the entire tests jar is definitely preferable. While "main" is not the best descriport (we should be using the gradle testfixture plugin) it's still better then slamming everything in "test" and then sharing that.

I can move it back... it just seemed odd to have test only code in the main source set, even for a qa project.

Yeah, it's not really "main" code, but its the most convenient place to put it.

jakelandis · 2020-08-05T22:24:19Z

...ernalClusterTest/java/org/elasticsearch/xpack/idp/saml/sp/SamlServiceProviderIndexTests.java

@@ -242,59 +240,6 @@ private void refresh() {
        future.actionGet();
    }

-    public static SamlServiceProviderDocument randomDocument() {


moved this to utility class (in the the test sourceset) so it can be shared easier.

jakelandis · 2020-08-05T22:24:58Z

...provider/src/test/java/org/elasticsearch/xpack/idp/saml/sp/SamlServiceProviderTestUtils.java

+
+    private SamlServiceProviderTestUtils(){} //utility class
+
+    public static SamlServiceProviderDocument randomDocument() {


no functional changes here ... just copy pasta to allow re-use across source sets.

jakelandis · 2020-08-05T22:27:22Z

...ti-node/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/MlBasicMultiNodeIT.java

@@ -29,6 +28,8 @@

 public class MlBasicMultiNodeIT extends ESRestTestCase {

+    private static final String BASE_PATH = "/_ml/";


Quite a few of the ML REST tests had a dependency on the ML main code for ONLY this base path reference in the tests. Adding to each test like here is a bit redundant, but allows us to break that class path dependency and the REST path for ml is unlikely to change.

Good call. And if it does, that's a breaking change anyhow.

jakelandis · 2020-08-05T22:29:18Z

x-pack/plugin/ml/qa/single-node-tests/build.gradle


-dependencies {
-  testImplementation project(":x-pack:plugin:core")
-  testImplementation project(path: xpackModule('ml'))


here is an example where duplicating the BASE_PATH allows us to remove the dependency.

…k_plugins

jakelandis · 2020-09-01T20:24:25Z

@elasticmachine run elasticsearch-ci/2

…Test or internalClusterTest (#61802) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: security, spatial, stack, transform, vecotrs, voting-only-node, and watcher. A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. related: #60630 related: #56841 related: #59939 related: #55896

…est or internalClusterTest (elastic#60630) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: async-search, autoscaling, ccr, enrich, eql, frozen-indicies, data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is). related: elastic#61802 related: elastic#56841 related: elastic#59939 related: elastic#55896 # Conflicts: # x-pack/plugin/identity-provider/src/internalClusterTest/java/org/elasticsearch/xpack/idp/saml/sp/SamlServiceProviderIndexTests.java # x-pack/plugin/ilm/qa/with-security/build.gradle

…Test or internalClusterTest (elastic#61802) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: security, spatial, stack, transform, vecotrs, voting-only-node, and watcher. A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. related: elastic#60630 related: elastic#56841 related: elastic#59939 related: elastic#55896 # Conflicts: # x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/integration/IndexPrivilegeIntegTests.java # x-pack/plugin/security/src/test/java/org/elasticsearch/integration/IndexPrivilegeIntegTests.java # x-pack/plugin/security/src/test/java/org/elasticsearch/integration/IndexPrivilegeTests.java # x-pack/plugin/transform/qa/multi-node-tests/build.gradle # x-pack/plugin/transform/qa/single-node-tests/build.gradle # x-pack/plugin/watcher/build.gradle

…]RestTest or internalClusterTest (#60630) (#61855) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: async-search, autoscaling, ccr, enrich, eql, frozen-indicies, data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is). related: #61802 related: #56841 related: #59939 related: #55896

…a]RestTest or internalClusterTest (#61802) (#61856) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: security, spatial, stack, transform, vecotrs, voting-only-node, and watcher. A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. related: #60630 related: #56841 related: #59939 related: #55896

javanna · 2020-09-04T11:08:01Z

heya, not sure if you are already aware, but after this change (I think), I get a compile error on IntelliJ on both master and 7.x.

/home/javanna/elastic/elasticsearch/x-pack/plugin/ml/qa/ml-with-security/src/yamlRestTest/java/org/elasticsearch/smoketest/MlWithSecurityIT.java:19:66
java: cannot find symbol
  symbol:   class UsernamePasswordToken
  location: package org.elasticsearch.xpack.core.security.authc.support

Compiling from command line works ok, I tried to reimport the project in IntelliJ but that did not fix it.

This commit removes `integTest` task from all es-plugins. Most relevant projects have been converted to use yamlRestTest, javaRestTest, or internalClusterTest in prior PRs. A few projects needed to be adjusted to allow complete removal of this task * x-pack/plugin - converted to use yamlRestTest and javaRestTest * plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task * qa/die-with-dignity - convert to javaRestTest * x-pack/qa/security-example-spi-extension - convert to javaRestTest * multiple projects - remove the integTest.enabled = false (yay!) related: #61802 related: #60630 related: #59444 related: #59089 related: #56841 related: #59939 related: #55896

This commit removes `integTest` task from all es-plugins. Most relevant projects have been converted to use yamlRestTest, javaRestTest, or internalClusterTest in prior PRs. A few projects needed to be adjusted to allow complete removal of this task * x-pack/plugin - converted to use yamlRestTest and javaRestTest * plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task * qa/die-with-dignity - convert to javaRestTest * x-pack/qa/security-example-spi-extension - convert to javaRestTest * multiple projects - remove the integTest.enabled = false (yay!) related: elastic#61802 related: elastic#60630 related: elastic#59444 related: elastic#59089 related: elastic#56841 related: elastic#59939 related: elastic#55896 # Conflicts: # qa/die-with-dignity/src/javaRestTest/java/org/elasticsearch/qa/die_with_dignity/DieWithDignityIT.java # x-pack/qa/security-example-spi-extension/build.gradle

This commit removes `integTest` task from all es-plugins. Most relevant projects have been converted to use yamlRestTest, javaRestTest, or internalClusterTest in prior PRs. A few projects needed to be adjusted to allow complete removal of this task * x-pack/plugin - converted to use yamlRestTest and javaRestTest * plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task * qa/die-with-dignity - convert to javaRestTest * x-pack/qa/security-example-spi-extension - convert to javaRestTest * multiple projects - remove the integTest.enabled = false (yay!) related: #61802 related: #60630 related: #59444 related: #59089 related: #56841 related: #59939 related: #55896

jakelandis added 22 commits August 3, 2020 14:41

async-search

92809d6

address java rest tests

0dc6caa

autoscaling

2b6ff9f

ccr:rest

54446e2

enrich

537fe05

eql

06dcaf3

enrich fixes

4b823b6

data stream fixes

7f1dff1

ccr:rest fixes

9e8b72e

autoscale fix

3ff3316

frozen-indices

13c8176

graph

27f34ce

fix autoscaling testing conventions

d5509df

identity-provider

0caf4b3

ilm

48163d5

mapper-constant-keyword

3f43102

mapper-flattened

78ae27d

ml - part 1 (not all working)

61abbd2

ml - part 2

9cfb7cc

ml part 3

ed0186f

fix ILM tests

bceb48c

Merge remote-tracking branch 'upstream/master' into yamlRestTest_xpac…

a19c812

…k_plugins

jakelandis changed the title ~~Convert most x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest~~ Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest Aug 5, 2020

jakelandis commented Aug 5, 2020

View reviewed changes

jakelandis marked this pull request as ready for review August 5, 2020 22:29

jakelandis added the :Delivery/Build Build or test infrastructure label Aug 5, 2020

mark-vieira approved these changes Aug 10, 2020

View reviewed changes

jakelandis added 4 commits August 24, 2020 11:29

Merge remote-tracking branch 'upstream/master' into yamlRestTest_xpac…

f803842

…k_plugins

preserve main source set

29bb22f

Merge branch 'master' into yamlRestTest_xpack_plugins

43cbc3e

fix merge

9ec969b

jakelandis mentioned this pull request Sep 1, 2020

Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #61802

Merged

jakelandis added 2 commits September 1, 2020 12:31

fix available processors

9f15419

fix the merge fix

666ae84

jakelandis mentioned this pull request Sep 1, 2020

update gradle config for new testing plugins pugnascotia/elasticsearch#3

Merged

jakelandis merged commit 51d0bca into elastic:master Sep 2, 2020

jakelandis deleted the yamlRestTest_xpack_plugins branch September 2, 2020 14:22

jakelandis added the backport pending label Sep 2, 2020

jakelandis mentioned this pull request Sep 2, 2020

[7.x] Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest (#60630) #61855

Merged

jakelandis removed the backport pending label Sep 2, 2020

jakelandis mentioned this pull request Sep 2, 2020

Remove integTest task from PluginBuildPlugin #61879

Merged

benwtrent mentioned this pull request Sep 3, 2020

[ML] fix integration test command with es elastic/ml-cpp#1470

Merged

mark-vieira added Team:Delivery Meta label for Delivery team and removed Team:Core/Infra Meta label for core/infra team labels Nov 11, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #60630

Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #60630

jakelandis commented Aug 3, 2020 •

edited

Loading

jakelandis Aug 5, 2020

mark-vieira Aug 6, 2020

jakelandis Aug 6, 2020

mark-vieira Aug 10, 2020

jakelandis Aug 5, 2020

jakelandis Aug 5, 2020

jakelandis Aug 5, 2020

mark-vieira Aug 6, 2020

jakelandis Aug 5, 2020

jakelandis commented Sep 1, 2020

javanna commented Sep 4, 2020


		private SamlServiceProviderTestUtils(){} //utility class

		public static SamlServiceProviderDocument randomDocument() {

		@@ -29,6 +28,8 @@

		public class MlBasicMultiNodeIT extends ESRestTestCase {

		private static final String BASE_PATH = "/_ml/";

Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #60630

Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #60630

Conversation

jakelandis commented Aug 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakelandis commented Sep 1, 2020

javanna commented Sep 4, 2020

jakelandis commented Aug 3, 2020 •

edited

Loading