Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #61802

jakelandis · 2020-09-01T15:56:09Z

For 1/2 the plugins in x-pack, the integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.

This includes the following projects:
security, spatial, stack, transform, vecotrs, voting-only-node, and watcher.

A few of the more specialized qa projects within these plugins
have not been changed with this PR due to additional complexity which should
be addressed separately.

related: #60630
related: #56841
related: #59939
related: #55896

…k_plugins_part2

…stTest_xpack_plugins_part2

jakelandis · 2020-09-01T20:09:44Z

@elasticmachine run elasticsearch-ci/packaging-sample-windows

elasticmachine · 2020-09-01T20:10:42Z

Pinging @elastic/es-core-infra (:Core/Infra/Build)

...internalClusterTest/java/org/elasticsearch/transport/filter/IpFilteringIntegrationTests.java

mark-vieira

Left a few comments but otherwise LGTM.

mark-vieira · 2020-09-01T20:56:53Z

x-pack/plugin/security/qa/basic-enable-security/build.gradle

+task javaRestTestWithSecurity(type: StandaloneRestIntegTestTask) {
+  description = "Run tests against a cluster that has security enabled"
+  useCluster testClusters.javaRestTest
+  dependsOn javaRestTest


This whole thing looks gnarly. Why don't we use two different clusters for this?

yeah... that tripped me up at first sight too. This is testing that enabling security where it was not enabled prior works. e.g. start without security run some tests, enable security restart then run the same tests. There is a boolean read by the tests that controls what to assert with/without security enabled.

mark-vieira · 2020-09-01T21:00:28Z

x-pack/plugin/security/build.gradle

@@ -476,6 +484,11 @@ test {
  systemProperty 'es.transport.buffer.size', '256k'
 }

+internalClusterTest {


I suspect this probably isn't needed for unit tests. Can we ditch the block above?

yup: 1f4ca9a

mark-vieira · 2020-09-01T21:03:08Z

x-pack/plugin/watcher/build.gradle

@@ -86,6 +87,10 @@ test {
  systemProperty 'es.set.netty.runtime.available.processors', 'false'
 }

+internalClusterTest {
+  systemProperty 'es.set.netty.runtime.available.processors', 'false'


Same, I suspect this is only needed for integration tests.

indeed: 1f4ca9a

jakelandis · 2020-09-01T22:31:02Z

thanks @mark-vieira - I addressed your comments.

Also, I held off merging the first half #60630 since I didn't want to merge 1/2 a major refactoring then go on vacation :) ... so I will merge these two PRs back to back.

…est or internalClusterTest (#60630) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: async-search, autoscaling, ccr, enrich, eql, frozen-indicies, data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is). related: #61802 related: #56841 related: #59939 related: #55896

…est or internalClusterTest (elastic#60630) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: async-search, autoscaling, ccr, enrich, eql, frozen-indicies, data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is). related: elastic#61802 related: elastic#56841 related: elastic#59939 related: elastic#55896 # Conflicts: # x-pack/plugin/identity-provider/src/internalClusterTest/java/org/elasticsearch/xpack/idp/saml/sp/SamlServiceProviderIndexTests.java # x-pack/plugin/ilm/qa/with-security/build.gradle

…Test or internalClusterTest (elastic#61802) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: security, spatial, stack, transform, vecotrs, voting-only-node, and watcher. A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. related: elastic#60630 related: elastic#56841 related: elastic#59939 related: elastic#55896 # Conflicts: # x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/integration/IndexPrivilegeIntegTests.java # x-pack/plugin/security/src/test/java/org/elasticsearch/integration/IndexPrivilegeIntegTests.java # x-pack/plugin/security/src/test/java/org/elasticsearch/integration/IndexPrivilegeTests.java # x-pack/plugin/transform/qa/multi-node-tests/build.gradle # x-pack/plugin/transform/qa/single-node-tests/build.gradle # x-pack/plugin/watcher/build.gradle

…]RestTest or internalClusterTest (#60630) (#61855) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: async-search, autoscaling, ccr, enrich, eql, frozen-indicies, data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is). related: #61802 related: #56841 related: #59939 related: #55896

…a]RestTest or internalClusterTest (#61802) (#61856) For 1/2 the plugins in x-pack, the integTest task is now a no-op and all of the tests are now executed via a test, yamlRestTest, javaRestTest, or internalClusterTest. This includes the following projects: security, spatial, stack, transform, vecotrs, voting-only-node, and watcher. A few of the more specialized qa projects within these plugins have not been changed with this PR due to additional complexity which should be addressed separately. related: #60630 related: #56841 related: #59939 related: #55896

This commit reenables the searchable snapshot feature for integration tests after #61802 which changed some build plugins.

…1965) This commit reenables the searchable snapshot feature for integration tests after #61802 which changed some build plugins.

This commit removes `integTest` task from all es-plugins. Most relevant projects have been converted to use yamlRestTest, javaRestTest, or internalClusterTest in prior PRs. A few projects needed to be adjusted to allow complete removal of this task * x-pack/plugin - converted to use yamlRestTest and javaRestTest * plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task * qa/die-with-dignity - convert to javaRestTest * x-pack/qa/security-example-spi-extension - convert to javaRestTest * multiple projects - remove the integTest.enabled = false (yay!) related: #61802 related: #60630 related: #59444 related: #59089 related: #56841 related: #59939 related: #55896

This commit removes `integTest` task from all es-plugins. Most relevant projects have been converted to use yamlRestTest, javaRestTest, or internalClusterTest in prior PRs. A few projects needed to be adjusted to allow complete removal of this task * x-pack/plugin - converted to use yamlRestTest and javaRestTest * plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task * qa/die-with-dignity - convert to javaRestTest * x-pack/qa/security-example-spi-extension - convert to javaRestTest * multiple projects - remove the integTest.enabled = false (yay!) related: elastic#61802 related: elastic#60630 related: elastic#59444 related: elastic#59089 related: elastic#56841 related: elastic#59939 related: elastic#55896 # Conflicts: # qa/die-with-dignity/src/javaRestTest/java/org/elasticsearch/qa/die_with_dignity/DieWithDignityIT.java # x-pack/qa/security-example-spi-extension/build.gradle

This commit removes `integTest` task from all es-plugins. Most relevant projects have been converted to use yamlRestTest, javaRestTest, or internalClusterTest in prior PRs. A few projects needed to be adjusted to allow complete removal of this task * x-pack/plugin - converted to use yamlRestTest and javaRestTest * plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task * qa/die-with-dignity - convert to javaRestTest * x-pack/qa/security-example-spi-extension - convert to javaRestTest * multiple projects - remove the integTest.enabled = false (yay!) related: #61802 related: #60630 related: #59444 related: #59089 related: #56841 related: #59939 related: #55896

jakelandis added 13 commits August 31, 2020 13:53

most of security

c61dbb1

security internalCluster

ec7a4f6

finish up security

ef58dfb

security:qa

92519db

spatial

5051538

stack

3660ab0

transform

351cb26

vectors

c4d39dd

voting-only-node

e62708c

watcher - internalClusterTest

5c325e5

watcher qa

5a65706

Merge remote-tracking branch 'upstream/master' into yamlRestTest_xpac…

0a94232

…k_plugins_part2

Merge branch 'master' of github.com:elastic/elasticsearch into yamlRe…

9313528

…stTest_xpack_plugins_part2

jakelandis changed the title ~~Yaml rest test xpack plugins part2~~ Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest Sep 1, 2020

jakelandis mentioned this pull request Sep 1, 2020

Convert first 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #60630

Merged

jakelandis added 3 commits September 1, 2020 12:33

remove commented out line

745f762

spotless

254b20d

fix precommit

3faa914

jakelandis marked this pull request as ready for review September 1, 2020 20:09

jakelandis added >non-issue v7.10.0 v8.0.0 :Delivery/Build Build or test infrastructure >refactoring and removed >non-issue labels Sep 1, 2020

elasticmachine added the Team:Core/Infra Meta label for core/infra team label Sep 1, 2020

jakelandis requested review from mark-vieira and rjernst September 1, 2020 20:11

jakelandis commented Sep 1, 2020

View reviewed changes

...internalClusterTest/java/org/elasticsearch/transport/filter/IpFilteringIntegrationTests.java Outdated Show resolved Hide resolved

jakelandis added 2 commits September 1, 2020 15:20

fix package location

f3dee8b

fix package location

f84d203

mark-vieira approved these changes Sep 1, 2020

View reviewed changes

jakelandis mentioned this pull request Sep 1, 2020

update gradle config for new testing plugins pugnascotia/elasticsearch#3

Merged

jakelandis added 2 commits September 1, 2020 17:12

remove unecessary watcher test artifact

667ea2a

remove unnecessary config

1f4ca9a

jakelandis merged commit 3049e55 into elastic:master Sep 2, 2020

jakelandis deleted the yamlRestTest_xpack_plugins_part2 branch September 2, 2020 14:23

jakelandis added the backport pending label Sep 2, 2020

jakelandis mentioned this pull request Sep 2, 2020

[7.x] Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest (#61802) #61856

Merged

jakelandis removed the backport pending label Sep 2, 2020

jakelandis mentioned this pull request Sep 2, 2020

Remove integTest task from PluginBuildPlugin #61879

Merged

tlrx mentioned this pull request Sep 3, 2020

Enable searchable snapshot feature for integTest clusters #61888

Merged

tlrx added a commit that referenced this pull request Sep 4, 2020

Enable searchable snapshot feature for all test clusters (#61888)

0b7fde8

This commit reenables the searchable snapshot feature for integration tests after #61802 which changed some build plugins.

tlrx mentioned this pull request Sep 4, 2020

Enable searchable snapshot feature for all test clusters #61965

Merged

tlrx added a commit that referenced this pull request Sep 4, 2020

Enable searchable snapshot feature for all test clusters (#61888) (#6…

10d14ce

…1965) This commit reenables the searchable snapshot feature for integration tests after #61802 which changed some build plugins.

mark-vieira added Team:Delivery Meta label for Delivery team and removed Team:Core/Infra Meta label for core/infra team labels Nov 11, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #61802

Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #61802

jakelandis commented Sep 1, 2020 •

edited

Loading

jakelandis commented Sep 1, 2020

elasticmachine commented Sep 1, 2020

mark-vieira left a comment

mark-vieira Sep 1, 2020

jakelandis Sep 1, 2020

mark-vieira Sep 1, 2020

jakelandis Sep 1, 2020

mark-vieira Sep 1, 2020

jakelandis Sep 1, 2020

jakelandis commented Sep 1, 2020

Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #61802

Convert second 1/2 x-pack plugins from integTest to [yaml | java]RestTest or internalClusterTest #61802

Conversation

jakelandis commented Sep 1, 2020 • edited Loading

jakelandis commented Sep 1, 2020

elasticmachine commented Sep 1, 2020

mark-vieira left a comment

Choose a reason for hiding this comment

mark-vieira Sep 1, 2020

Choose a reason for hiding this comment

jakelandis Sep 1, 2020

Choose a reason for hiding this comment

mark-vieira Sep 1, 2020

Choose a reason for hiding this comment

jakelandis Sep 1, 2020

Choose a reason for hiding this comment

mark-vieira Sep 1, 2020

Choose a reason for hiding this comment

jakelandis Sep 1, 2020

Choose a reason for hiding this comment

jakelandis commented Sep 1, 2020

jakelandis commented Sep 1, 2020 •

edited

Loading