From 20ce7dd812798386ea2bdfe7927b6293f44b7c58 Mon Sep 17 00:00:00 2001 From: William Brafford Date: Mon, 14 Oct 2019 16:51:47 -0400 Subject: [PATCH 1/4] Update testing document for new packaging tests The TESTING.asciidoc document had gotten out of date due to some new and wonderful changes in our vagrant testing code. I've taken a first cut at some updates here. --- TESTING.asciidoc | 177 ++++++++++++++++++----------------------------- 1 file changed, 69 insertions(+), 108 deletions(-) diff --git a/TESTING.asciidoc b/TESTING.asciidoc index cb94ad4a8a962..3192ef44771ba 100644 --- a/TESTING.asciidoc +++ b/TESTING.asciidoc @@ -152,7 +152,7 @@ Default value provided below in [brackets]. === Load balancing and caches. -By default the tests run on multiple processes using all the available cores on all +By default the tests run on multiple processes using all the available cores on all available CPUs. Not including hyper-threading. If you want to explicitly specify the number of JVMs you can do so on the command line: @@ -300,9 +300,17 @@ e.g. -Dtests.rest.blacklist=index/*/Index document,get/10_basic/* == Testing packaging -The packaging tests use Vagrant virtual machines to verify that installing -and running elasticsearch distributions works correctly on supported operating systems. -These tests should really only be run in vagrant vms because they're destructive. +The packaging tests use Vagrant virtual machines or cloud instances to verify +that installing and running elasticsearch distributions works correctly on +supported operating systems. These tests should really only be run on ephemeral +systems because they're destructive; that is, these tests install and remove +packages and freely modify system settings, so you will probably regret it if +you execute them on your development machine. + +When you run a packaging test, Gradle will set up the target VM and mount your +repo directory in the VM. Once this is done, a Gradle task will issue a Vagrant +command to run a *nested* Gradle task on the VM. This nested Gradle runs the +actual "destructive" test classes. . Install Virtual Box and Vagrant. + @@ -313,37 +321,25 @@ a bit more performance out of the process: vagrant plugin install vagrant-cachier -------------------------------------- + -. Validate your installed dependencies: -+ -------------------------------------- -./gradlew :qa:vagrant:vagrantCheckVersion -------------------------------------- -+ -. Download and smoke test the VMs with `./gradlew vagrantSmokeTest` or -`./gradlew -Pvagrant.boxes=all vagrantSmokeTest`. The first time you run this it will -download the base images and provision the boxes and immediately quit. Downloading all -the images may take a long time. After the images are already on your machine, they won't -be downloaded again unless they have been updated to a new version. -+ -. Run the tests with `./gradlew packagingTest`. This will cause Gradle to build -the tar, zip, and deb packages and all the plugins. It will then run the tests -on ubuntu-1604 and centos-7. We chose those two distributions as the default -because they cover deb and rpm packaging and SyvVinit and systemd. - -You can choose which boxes to test by setting the `-Pvagrant.boxes` project property. All of -the valid options for this property are: - -* `sample` - The default, only chooses ubuntu-1604 and centos-7 -* List of box names, comma separated (e.g. `oel-7,fedora-28`) - Chooses exactly the boxes listed. -* `linux-all` - All linux boxes. -* `windows-all` - All Windows boxes. If there are any Windows boxes which do not -have images available when this value is provided, the build will fail. -* `all` - All boxes we test. If there are any boxes (e.g. Windows) which do not have images -available when this value is provided, the build will fail. - -For a complete list of boxes on which tests can be run, run `./gradlew :qa:vagrant:listAllBoxes`. -For a list of boxes that have images available from your configuration, run -`./gradlew :qa:vagrant:listAvailableBoxes` + +. You can run all of the OS packaging tests with `./gradlew distroTest`. This +will cause Gradle to build the tar, zip, and deb packages and all the plugins. +It will then run the tests on every available system. This will take some time. + +The various systems under test have their own Gradle tasks under `qa/os`. To find +out what packiging combinations can be tested on a system, run the `tasks` task. +For example: + +---------------------------------- +./gradlew :qa:os:ubuntu-1804:tasks +---------------------------------- + +If you want a quick test of the tarball and RPM packagings for Centos 7, you +would run: + +------------------------------------------------------------------------------------------------- +./gradlew :qa:os:centos-7:distroTest.default-rpm :qa:os:centos-7:distroTest.default-linux-archive +------------------------------------------------------------------------------------------------- Note that if you interrupt gradle in the middle of running these tasks, any boxes started will remain running and you'll have to stop them manually with `./gradlew stop` or @@ -393,9 +389,7 @@ or passed on the command line like `-Pvagrant.windows-2012r2.id=my-image-id` `-Pvagrant.windows-2016=another-image-id` These properties are required for Windows support in all gradle tasks that -handle packaging tests. Either or both may be specified. Remember that to run tests -on these boxes, the project property `vagrant.boxes` still needs to be set to a -value that will include them. +handle packaging tests. Either or both may be specified. If you're running vagrant commands outside of gradle, specify the Windows boxes with the environment variables @@ -438,84 +432,51 @@ that'd consume a ton of ram. === Iterating on packaging tests -Running the packaging tests through gradle can take a while because it will start -and stop the VM each time. You can iterate faster by keeping the VM up and running -the tests directly. - -The packaging tests use a random seed to determine which past version to use for -testing upgrades. To use a single past version fix the test seed when running -the commands below (see <>) - -First build the packaging tests and their dependencies - --------------------------------------------- -./gradlew :qa:vagrant:setupPackagingTest --------------------------------------------- - -Then choose the VM you want to test on and bring it up. For example, to bring -up Debian 9 use the gradle command below. Bringing the box up with vagrant directly -may not mount the packaging test project in the right place. Once the VM is up, ssh -into it - --------------------------------------------- -./gradlew :qa:vagrant:vagrantDebian9#up -vagrant ssh debian-9 --------------------------------------------- - -Now inside the VM, start the packaging tests from the terminal. There are two packaging -test projects. The old ones are written with https://github.com/sstephenson/bats[bats] -and only run on linux. To run them do - --------------------------------------------- -cd $PACKAGING_ARCHIVES - -# runs all bats tests -sudo bats $BATS_TESTS/*.bats - -# you can also pass specific test files -sudo bats $BATS_TESTS/20_tar_package.bats $BATS_TESTS/25_tar_plugins.bats --------------------------------------------- +Because our packaging tests are capable of testing many combinations of OS (Windows +or Linux), package type (zip file or RPM), Elasticsearch distribution type (default +or OSS), and so forth, it's faster to develop against smaller subsets of the tests. +For example, to run tests for the default archive distribution on Fedora 28: -The new packaging tests are written in Java and run on both linux and windows. On -linux (again, inside the VM) +----------------------------------------------------------- +./gradlew :qa:os:fedora-28:distroTest.default-linux-archive +----------------------------------------------------------- --------------------------------------------- -# run the full suite -sudo bash $PACKAGING_TESTS/run-tests.sh +These test tasks can use the `--tests`, `--info`, and `--debug` parameters just like +non-OS tests can. For example: -# run specific test cases -sudo bash $PACKAGING_TESTS/run-tests.sh \ - org.elasticsearch.packaging.test.DefaultWindowsZipTests \ - org.elasticsearch.packaging.test.OssWindowsZipTests --------------------------------------------- +----------------------------------------------------------- +./gradlew :qa:os:fedora-28:distroTest.default-linux-archive \ + --tests "com.elasticsearch.packaging.test.ArchiveTests" +----------------------------------------------------------- -or on Windows, from a terminal running as Administrator +If you need to work on the packaging test framework or run Gradle tests from +the VM's command line, you might have to trace the operations of the gradle +plugin; we don't have a single command that will spin up and prepare a Vagrant +VM but stop just short of running the tests. --------------------------------------------- -# run the full suite -powershell -File $Env:PACKAGING_TESTS/run-tests.ps1 +However, this sequence of commands runs the default-rpm tests on the command +line of a centos-7 VM, and it might provide some clues for test development: -# run specific test cases -powershell -File $Env:PACKAGING_TESTS/run-tests.ps1 ` - org.elasticsearch.packaging.test.DefaultWindowsZipTests ` - org.elasticsearch.packaging.test.OssWindowsZipTests --------------------------------------------- +-------- +# set up JDKs needed for Gradle testing on the VM +./gradlew :extractLinuxJdk-openjdk-11.0.2+9 +./gradlew :extractLinuxJdk-openjdk-12.0.1+12@69cfe15208a647278a19ef0990eea691 -Note that on Windows boxes when running from inside the GUI, you may have to log out and -back in to the `vagrant` user (password `vagrant`) for the environment variables that -locate the packaging tests and distributions to take effect, due to how vagrant provisions -Windows machines. +vagrant up --provision centos-7 +vagrant ssh centos-7 -When you've made changes you want to test, keep the VM up and reload the tests and -distributions inside by running (on the host) +# from here, we are running on the gradle command line +export PATH=/elasticsearch/build/jdks/openjdk-12.0.1_linux/bin:$PATH +export SYSTEM_JAVA_HOME-/elasticsearch/build/jdks/openjdk-11.0.2_linux --------------------------------------------- -./gradlew :qa:vagrant:clean :qa:vagrant:setupPackagingTest --------------------------------------------- +cd /project -Note: Starting vagrant VM outside of the elasticsearch folder requires to -indicates the folder that contains the Vagrantfile using the VAGRANT_CWD -environment variable. +# If running gradle with sudo scares you, it should! These tests are +# destructive. VM use only! +sudo --preserve-env=PATH,SYSTEM_JAVA_HOME ./gradlew --Dtests.inVM \ + --project-cache-dir=/project/gradle-cache \ + :qa:os:destructiveDistroTest.default-rpm +-------- == Testing backwards compatibility @@ -536,7 +497,7 @@ version 5.3.2 run: Tests are ran for versions that are not yet released but with which the current version will be compatible with. These are automatically checked out and built from source. See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection] -and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle] +and link:./distribution/bwc/build.gradle[distribution/bwc/build.gradle] for more information. When running `./gradlew check`, minimal bwc checks are also run against compatible versions that are not yet released. @@ -657,7 +618,7 @@ care. == Test coverage analysis Generating test coverage reports for Elasticsearch is currently not possible through Gradle. -However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage +However, it _is_ possible to gain insight in code coverage using IntelliJ's built-in coverage analysis tool that can measure coverage upon executing specific tests. Eclipse may also be able to do the same using the EclEmma plugin. From 374860c71958a7a65458d0d53ce177bc6ba909d5 Mon Sep 17 00:00:00 2001 From: William Brafford Date: Tue, 15 Oct 2019 10:56:50 -0400 Subject: [PATCH 2/4] Correct indentation, spelling, and capitalization --- TESTING.asciidoc | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/TESTING.asciidoc b/TESTING.asciidoc index 3192ef44771ba..3ac21fb5f08bf 100644 --- a/TESTING.asciidoc +++ b/TESTING.asciidoc @@ -301,16 +301,16 @@ e.g. -Dtests.rest.blacklist=index/*/Index document,get/10_basic/* == Testing packaging The packaging tests use Vagrant virtual machines or cloud instances to verify -that installing and running elasticsearch distributions works correctly on +that installing and running Elasticsearch distributions works correctly on supported operating systems. These tests should really only be run on ephemeral systems because they're destructive; that is, these tests install and remove packages and freely modify system settings, so you will probably regret it if you execute them on your development machine. When you run a packaging test, Gradle will set up the target VM and mount your -repo directory in the VM. Once this is done, a Gradle task will issue a Vagrant -command to run a *nested* Gradle task on the VM. This nested Gradle runs the -actual "destructive" test classes. +repository directory in the VM. Once this is done, a Gradle task will issue a +Vagrant command to run a *nested* Gradle task on the VM. This nested Gradle +runs the actual "destructive" test classes. . Install Virtual Box and Vagrant. + @@ -324,24 +324,25 @@ vagrant plugin install vagrant-cachier . You can run all of the OS packaging tests with `./gradlew distroTest`. This will cause Gradle to build the tar, zip, and deb packages and all the plugins. -It will then run the tests on every available system. This will take some time. - -The various systems under test have their own Gradle tasks under `qa/os`. To find -out what packiging combinations can be tested on a system, run the `tasks` task. -For example: - +It will then run the tests on every available system. This will take a very +long time. ++ +Fortunately, the various systems under test have their own Gradle tasks under +`qa/os`. To find out what packaging combinations can be tested on a system, run +the `tasks` task. For example: ++ ---------------------------------- ./gradlew :qa:os:ubuntu-1804:tasks ---------------------------------- - ++ If you want a quick test of the tarball and RPM packagings for Centos 7, you would run: - ++ ------------------------------------------------------------------------------------------------- ./gradlew :qa:os:centos-7:distroTest.default-rpm :qa:os:centos-7:distroTest.default-linux-archive ------------------------------------------------------------------------------------------------- -Note that if you interrupt gradle in the middle of running these tasks, any boxes started +Note that if you interrupt Gradle in the middle of running these tasks, any boxes started will remain running and you'll have to stop them manually with `./gradlew stop` or `vagrant halt`. @@ -450,9 +451,9 @@ non-OS tests can. For example: ----------------------------------------------------------- If you need to work on the packaging test framework or run Gradle tests from -the VM's command line, you might have to trace the operations of the gradle -plugin; we don't have a single command that will spin up and prepare a Vagrant -VM but stop just short of running the tests. +the VM's command line, you might have to trace the operations of the Gradle +`DistroTestPlugin` plugin; we don't have a single command that will spin up and +prepare a Vagrant VM but stop just short of running the tests. However, this sequence of commands runs the default-rpm tests on the command line of a centos-7 VM, and it might provide some clues for test development: @@ -465,13 +466,13 @@ line of a centos-7 VM, and it might provide some clues for test development: vagrant up --provision centos-7 vagrant ssh centos-7 -# from here, we are running on the gradle command line +# from here, we are running on the Gradle command line export PATH=/elasticsearch/build/jdks/openjdk-12.0.1_linux/bin:$PATH export SYSTEM_JAVA_HOME-/elasticsearch/build/jdks/openjdk-11.0.2_linux cd /project -# If running gradle with sudo scares you, it should! These tests are +# If running Gradle with sudo scares you, it should! These tests are # destructive. VM use only! sudo --preserve-env=PATH,SYSTEM_JAVA_HOME ./gradlew --Dtests.inVM \ --project-cache-dir=/project/gradle-cache \ From ea40b27d5773d98c9bcee0186bcdf7814b60b039 Mon Sep 17 00:00:00 2001 From: William Brafford Date: Tue, 15 Oct 2019 16:58:04 -0400 Subject: [PATCH 3/4] Respond to PR feedback I've edited a few instructions for style, clarity, and correctness. Furthermore, I've removed the unwieldy instructions for mimicking the Gradle plugin using shell commands. --- TESTING.asciidoc | 50 +++++++++++------------------------------------- 1 file changed, 11 insertions(+), 39 deletions(-) diff --git a/TESTING.asciidoc b/TESTING.asciidoc index 3ac21fb5f08bf..73c79f76a8f3e 100644 --- a/TESTING.asciidoc +++ b/TESTING.asciidoc @@ -321,11 +321,11 @@ a bit more performance out of the process: vagrant plugin install vagrant-cachier -------------------------------------- + - -. You can run all of the OS packaging tests with `./gradlew distroTest`. This -will cause Gradle to build the tar, zip, and deb packages and all the plugins. -It will then run the tests on every available system. This will take a very -long time. +. You can run all of the OS packaging tests with `./gradlew packagingTest`. +This task includes our legacy `bats` tests. To run only the OS tests that are +written in Java, run `.gradlew distroTest`, will cause Gradle to build the tar, +zip, and deb packages and all the plugins. It will then run the tests on every +available system. This will take a very long time. + Fortunately, the various systems under test have their own Gradle tasks under `qa/os`. To find out what packaging combinations can be tested on a system, run @@ -343,7 +343,7 @@ would run: ------------------------------------------------------------------------------------------------- Note that if you interrupt Gradle in the middle of running these tasks, any boxes started -will remain running and you'll have to stop them manually with `./gradlew stop` or +will remain running and you'll have to stop them manually with `./gradlew --stop` or `vagrant halt`. All the regular vagrant commands should just work so you can get a shell in a @@ -433,10 +433,11 @@ that'd consume a ton of ram. === Iterating on packaging tests -Because our packaging tests are capable of testing many combinations of OS (Windows -or Linux), package type (zip file or RPM), Elasticsearch distribution type (default -or OSS), and so forth, it's faster to develop against smaller subsets of the tests. -For example, to run tests for the default archive distribution on Fedora 28: +Because our packaging tests are capable of testing many combinations of OS +(e.g., Windows or Linux), package type (e.g., zip file or RPM), Elasticsearch +distribution type (e.g., default or OSS), and so forth, it's faster to develop +against smaller subsets of the tests. For example, to run tests for the default +archive distribution on Fedora 28: ----------------------------------------------------------- ./gradlew :qa:os:fedora-28:distroTest.default-linux-archive @@ -450,35 +451,6 @@ non-OS tests can. For example: --tests "com.elasticsearch.packaging.test.ArchiveTests" ----------------------------------------------------------- -If you need to work on the packaging test framework or run Gradle tests from -the VM's command line, you might have to trace the operations of the Gradle -`DistroTestPlugin` plugin; we don't have a single command that will spin up and -prepare a Vagrant VM but stop just short of running the tests. - -However, this sequence of commands runs the default-rpm tests on the command -line of a centos-7 VM, and it might provide some clues for test development: - --------- -# set up JDKs needed for Gradle testing on the VM -./gradlew :extractLinuxJdk-openjdk-11.0.2+9 -./gradlew :extractLinuxJdk-openjdk-12.0.1+12@69cfe15208a647278a19ef0990eea691 - -vagrant up --provision centos-7 -vagrant ssh centos-7 - -# from here, we are running on the Gradle command line -export PATH=/elasticsearch/build/jdks/openjdk-12.0.1_linux/bin:$PATH -export SYSTEM_JAVA_HOME-/elasticsearch/build/jdks/openjdk-11.0.2_linux - -cd /project - -# If running Gradle with sudo scares you, it should! These tests are -# destructive. VM use only! -sudo --preserve-env=PATH,SYSTEM_JAVA_HOME ./gradlew --Dtests.inVM \ - --project-cache-dir=/project/gradle-cache \ - :qa:os:destructiveDistroTest.default-rpm --------- - == Testing backwards compatibility Backwards compatibility tests exist to test upgrading from each supported version From 2d8f21328fa0c1b38fd54bdc1d3e6d5feb13104c Mon Sep 17 00:00:00 2001 From: William Brafford Date: Tue, 15 Oct 2019 17:01:19 -0400 Subject: [PATCH 4/4] Add some etc's to examples --- TESTING.asciidoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/TESTING.asciidoc b/TESTING.asciidoc index 73c79f76a8f3e..e9279c3c7ce41 100644 --- a/TESTING.asciidoc +++ b/TESTING.asciidoc @@ -434,10 +434,10 @@ that'd consume a ton of ram. === Iterating on packaging tests Because our packaging tests are capable of testing many combinations of OS -(e.g., Windows or Linux), package type (e.g., zip file or RPM), Elasticsearch -distribution type (e.g., default or OSS), and so forth, it's faster to develop -against smaller subsets of the tests. For example, to run tests for the default -archive distribution on Fedora 28: +(e.g., Windows, Linux, etc.), package type (e.g., zip file, RPM, etc.), +Elasticsearch distribution type (e.g., default or OSS), and so forth, it's +faster to develop against smaller subsets of the tests. For example, to run +tests for the default archive distribution on Fedora 28: ----------------------------------------------------------- ./gradlew :qa:os:fedora-28:distroTest.default-linux-archive