Centralize and update test documentation - Jenkins features reference #4690
Conversation
Ndacyayisenga-droid
changed the title
| @@ -0,0 +1,129 @@ | |||
| # Jenkins Features Reference | |||
| The "Jenkins Features Reference" is a comprehensive documentation outlining the various [configuration parameters](https://ci.adoptium.net/view/Test_grinder/job/Grinder/build?delay=0sec) and options available for running [Grinder jobs](https://ci.adoptium.net/view/Test_grinder/job/Grinder/) on the Jenkins server. | |||
| It serves as a guide for testers and developers using AQAvit Grinder to perform testing and triage on the ci.adoptium.net Jenkins server and the OpenJ9 Jenkins server (and internal servers). | |||
There was a problem hiding this comment.
Can you convert ci.adoptium.net to a link and add a link to the OpenJ9 Jenkins server?
doc/pages/JenkinFeatures.md
Outdated
| ### Test Repositories Parameters | ||
| Repositories where we pull test material from. Unless you are testing test code, these do not need to be changed. | ||
|
|
||
| - `ADOPTOPENJDK_REPO`: optional: your fork of aqa-tests. |
There was a problem hiding this comment.
I assume a table would make sense (see https://docs.github.com/de/get-started/writing-on-github/working-with-advanced-formatting/organizing-information-with-tables)
Next to this I would really prefer to have a sample for each property to see in what form an repo need to be specified. Is it "foo/bar" or "https://github.com/foo/bar" for example
| Additional test repositories that can be overlaid onto existing AQA test material for extra testing. | ||
| This is used for laying down smoke tests in the temurin-build repository and installer tests in the installer repository, along with any private vendor tests that cannot be run in the open. | ||
|
|
||
| - `VENDOR_TEST_REPOS`: optional, comma-separated list of repositories containing additional test material. |
There was a problem hiding this comment.
again: table + samples
Co-authored-by: Hendrik Ebbers <[email protected]>
| ## Access Permissions | ||
| For access permissions to the Grinder job at ci.adoptium.net, you need to be part of the `test-triage` Github team (requires [2FA](https://docs.github.com/en/authentication/securing-your-account-with-two-factor-authentication-2fa/configuring-two-factor-authentication) on your Github account). |
There was a problem hiding this comment.
Let's remove this section, as it refers to details on how to access a particular Jenkins server, versus what the features of the Jenkins pipeline code includes.
| ## Grouping and Granularity | ||
| We have many tests, so it is important to be able to slice and dice them in different ways to be efficient with testing, debugging, and triage. | ||
| Tests are organized into groups typically based on where they came from and what type of test they are: `openjdk`, `perf`, `system`, `functional`, `external`. |
There was a problem hiding this comment.
We can remove this section as its answering "why" and is covered under other documentation versus being a reference on implementation details of how we run tests in a Jenkins environment.
| | Parameter | Description | | ||
| |--------------------------------------|-----------------------------------------------------------------------------------------------------------| | ||
| | `ADOPTOPENJDK_REPO` | Optional. Your fork of `aqa-tests`. | | ||
| | `ADOPTOPENJDK_BRANCH` | Optional. Your branch off of your fork of `aqa-tests`. | | ||
| | `OPENJ9_REPO` | Optional. Your fork of `openj9`. | | ||
| | `OPENJ9_BRANCH` | Optional. Your branch of your fork of `openj9`. | | ||
| | `OPENJ9_SHA` | Optional. Pin to a specific SHA of `openj9`. | | ||
| | `JDK_REPO` | Optional. Use test material from a specific OpenJDK repository. | | ||
| | `JDK_BRANCH` | Optional. Use test material from a specific OpenJDK branch. | | ||
| | `OPENJDK_SHA` | Optional. Pin to a specific OpenJDK SHA. | | ||
| | `TKG_OWNER_BRANCH` | Optional. Use a specific `adoptium/TKG` fork/branch. | | ||
| | `ADOPTOPENJDK_SYSTEMTEST_OWNER_BRANCH`| Optional. Use a specific `adoptium/aqa-systemtest` fork/branch. | | ||
| | `OPENJ9_SYSTEMTEST_OWNER_BRANCH` | Optional. Use a specific `openj9/openj9-systemtest` fork/branch. | | ||
| | `STF_OWNER_BRANCH` | Optional. Use a specific `adoptium/STF` fork/branch. | | ||
| | `JCK_GIT_REPO` | Optional. Use a specific private repository for JCK test material supplied under OCTLA. | |
There was a problem hiding this comment.
We should add a column to the table to include the suggested default settings for parameters. Some of the descriptions need updating. Some parameters are not optional, but rather there are default values, so one does not need to fill in the value, because a default value is set.
| ## Grinder Etiquette | ||
| - When possible, avoid running top-level targets (all of sanity.openjdk tests, etc., since we have nightly runs for those), or runs with 100x+ ITERATIONS. | ||
| These can take hours and will block the regular testing, especially on platforms where we have few machines. | ||
| - Do not mark Grinder runs as “keep forever,” there is a limited amount of space on Jenkins master. | ||
| Download any artifacts you need from your run as soon as it finishes to attach to issues, etc., and consider the Jenkins job transient. | ||
| - When reporting failures in issues, understand that Grinder jobs are transient (not kept for long on Jenkins). | ||
| Sharing a link to the Grinder is only useful for a short time. | ||
| Share the "Rerun in Grinder" link which has details of the impl/version/test target, output statistics & test artifacts in the issue; those are more concrete items when reproducing failures. | ||
| For openjdk test failures, here are some useful triage [instructions](https://github.com/adoptium/aqa-tests/wiki/Guidance-for-Creating-OpenJDK-Test-Defects). | ||
| - Proactively delete Grinder runs that you do not need (setup fails or mistake in target, etc.). | ||
| We keep the last 50 runs, and if you delete your unneeded runs immediately, other jobs will stay a little bit longer, giving others more time to grab artifacts. | ||
|
|
||
| **Note**: The above documentation provides a comprehensive reference for configuring Jenkins Test Jobs for AQAvit Grinder testing at ci.adoptium.net and OpenJ9 Jenkins server. | ||
| Make sure to follow the guidelines and etiquette to ensure efficient and effective testing processes. |
There was a problem hiding this comment.
This section can be removed from the reference documentation. It can go into a different doc that is not talking about the details of how the Jenkins pipeline function, but rather a request for how the community is expected to behave.
|
Thanks @Ndacyayisenga-droid - we will plan to merge this PR as is and make further updates on this documentation once its in. @sophia-guo - can I get a review to get this in, and I will make the next phase of improvements shortly. |
Co-authored-by: Shelley Lambert <[email protected]>
Add Jenkins Features Reference
fixes part of #1558
cc @ShelleyLambert @hendrikebbers