From f9502256350bb753f0e7b326b7012e53994eb933 Mon Sep 17 00:00:00 2001 From: Benjamin Trent <4357155+benwtrent@users.noreply.github.com> Date: Wed, 29 Jun 2022 16:59:09 -0400 Subject: [PATCH 1/2] Update CONTRIBUTING.md to include how to run failing tests --- CONTRIBUTING.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d10cbc65426f0..9787a5bd5aeee 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -698,6 +698,27 @@ If your changes affect only the documentation, run: For more information about testing code examples in the documentation, see https://github.com/elastic/elasticsearch/blob/master/docs/README.asciidoc +### Only running failed tests + +When you open your pull-request it may be approved for review. When it is the full +test suite is ran within Elasticsearch's CI environment. If a test fails, +you can see how to run that particular test by searching for the `REPRODUCE` +string in the CI's console output. + +Elasticsearch's testing suite takes advantage of randomized testing. Consequently, +a test that passes locally, may actually fail later due to random settings +or data input. To make tests repeatable, a `REPRODUCE` line in CI will also include +the `-Dtests.seed` parameter. + +When running locally, gradle does its best to take advantage of cached results. +So, if the code is unchanged, running the same test with the same `-Dtests.seed` +repeatedly may not actually run the test if it has passed with that seed + in the previous execution. A way around this is to pass a separate parameter +to adjust the command options seen by gradle. +A simple option may be to add the parameter `-Dtests.timestamp=$(date +%s)` +which will give the current time stamp as a parameter, thus making the parameters +sent to gradle unique and bypassing the cache. + ### Project layout This repository is split into many top level directories. The most important From ac9fc5d79860653133e5d30f2d9e1eb89bb8e86f Mon Sep 17 00:00:00 2001 From: Benjamin Trent Date: Thu, 30 Jun 2022 07:38:18 -0400 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Rory Hunter --- CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9787a5bd5aeee..91c9fd1cc6c76 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -700,8 +700,8 @@ https://github.com/elastic/elasticsearch/blob/master/docs/README.asciidoc ### Only running failed tests -When you open your pull-request it may be approved for review. When it is the full -test suite is ran within Elasticsearch's CI environment. If a test fails, +When you open your pull-request it may be approved for review. If so, the full +test suite is run within Elasticsearch's CI environment. If a test fails, you can see how to run that particular test by searching for the `REPRODUCE` string in the CI's console output.