Update Github pull request template to have a task for inspecting fai…

…ling checks (#10751) (#10794)

* Update Github pull request template to have a task for inspecting failing checks



* Add failing builds troubleshooting section to CONTRIBUTING.md



* Address review comments



---------


(cherry picked from commit ffe9371)

Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

.github/pull_request_template.md

@@ -9,7 +9,8 @@
   - [ ] All tests pass
 - [ ] New functionality has been documented.
   - [ ] New functionality has javadoc added
-- [ ] Commits are signed per the DCO using --signoff 
+- [ ] Failing checks are inspected and point to the corresponding known issue(s) (See: [Troubleshooting Failing Builds](../blob/main/CONTRIBUTING.md#troubleshooting-failing-builds))
+- [ ] Commits are signed per the DCO using --signoff
 
 By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
 For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

CONTRIBUTING.md

@@ -14,6 +14,7 @@ OpenSearch is a community project that is built and maintained by people just li
     - [Contributing Code](#contributing-code)
   - [Developer Certificate of Origin](#developer-certificate-of-origin)
   - [Review Process](#review-process)
+  - [Troubleshooting Failing Builds](#troubleshooting-failing-builds)
 
 
 ## First Things First
@@ -120,3 +121,14 @@ During the PR process, expect that there will be some back-and-forth.  Please tr
 If we accept the PR, a [maintainer](MAINTAINERS.md) will merge your change and usually take care of backporting it to appropriate branches ourselves.
 
 If we reject the PR, we will close the pull request with a comment explaining why. This decision isn't always final: if you feel we have misunderstood your intended change or otherwise think that we should reconsider then please continue the conversation with a comment on the PR and we'll do our best to address any further points you raise.
+
+## Troubleshooting Failing Builds
+
+The OpenSearch testing framework offers many capabilities but exhibits significant complexity (it does lot of randomization internally to cover as many edge cases and variations as possible). Unfortunately, this posses a challenge by making it harder to discover important issues/bugs in straightforward way and may lead to so called flaky tests - the tests which flip randomly from success to failure without any code changes. 
+
+If your pull request reports a failing test(s) on one of the checks, please:
+ - look if there is an existing [issue](https://github.com/opensearch-project/OpenSearch/issues) reported for the test in question 
+ - if not, please make sure this is not caused by your changes, run the failing test(s) locally for some time
+ - if you are sure the failure is not related, please open a new [bug](https://github.com/opensearch-project/OpenSearch/issues/new?assignees=&labels=bug%2C+untriaged&projects=&template=bug_template.md&title=%5BBUG%5D) with `flaky-test` label
+ - add a comment referencing the issue(s) or bug report(s) to your pull request explaining the failing build(s)
+ - as a bonus point, try to contribute by fixing the flaky test(s)