Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed parallel script for cypress tests in QA and buildkite #169311

Merged
merged 49 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
7d2104c
Fixed parallel script for cypress and buildkite
dkirchan Oct 18, 2023
9e43643
Fixed API_KEY target to be more accurate to which API_KEY it refers to
dkirchan Oct 19, 2023
bf05e41
Fixed error codes to avoid logging them. Logging status code and mess…
dkirchan Oct 19, 2023
950cf06
Wrapped vault read to a retry
dkirchan Oct 19, 2023
2d7e9bb
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Oct 19, 2023
93bb326
Added parallelism - Disabled Essentials non-functioning test - Run al…
dkirchan Oct 23, 2023
023b18f
fixes linting issue
MadameSheema Oct 26, 2023
c22ec0e
reuses existing types
MadameSheema Oct 31, 2023
fd3e26c
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Oct 31, 2023
60270f9
Revert "reuses existing types"
MadameSheema Oct 31, 2023
56c4e9c
reusing existing types
MadameSheema Oct 31, 2023
dcd01d8
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Oct 31, 2023
fb90107
Revert "reusing existing types"
MadameSheema Oct 31, 2023
f68e7ad
extracts retrieve integrations to utils
MadameSheema Oct 31, 2023
e4c169e
removing unnecessary methods
MadameSheema Nov 2, 2023
9e33e81
Fixed comment with product type to get an already defined one
dkirchan Nov 2, 2023
47653cb
Changed the encode method to the default btoa()
dkirchan Nov 2, 2023
1934c67
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Nov 2, 2023
5535a0e
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Nov 2, 2023
597d8c1
Fixing return types and API_KEY check
dkirchan Nov 2, 2023
255c0c5
Merge branch 'security/dkirchan-create-envs' of github.com:elastic/ki…
dkirchan Nov 2, 2023
9d646ca
Reverted Product type change
dkirchan Nov 2, 2023
57d39ba
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Nov 2, 2023
596f4d3
Removed return undefined
dkirchan Nov 2, 2023
f4d61e8
Merge branch 'security/dkirchan-create-envs' of github.com:elastic/ki…
dkirchan Nov 2, 2023
267c0b9
stabilizing tests in production
MadameSheema Nov 2, 2023
eb293e4
Merge branch 'main' into security/dkirchan-create-envs
MadameSheema Nov 2, 2023
d761a4a
fixes merge conflict issue
MadameSheema Nov 2, 2023
5917580
Changed poll method with pRetry
dkirchan Nov 2, 2023
3cb02fb
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Nov 2, 2023
d7613c8
Fixed comments for envrionments errors and types
dkirchan Nov 3, 2023
3006b32
Fixed process exit to correctly map the exit code in order to fail th…
dkirchan Nov 3, 2023
1f9eb5a
Removed then from pMap Promise
dkirchan Nov 3, 2023
73bb92d
stabilizing more tests in production
MadameSheema Nov 3, 2023
010e4fd
Merge branch 'main' into security/dkirchan-create-envs
MadameSheema Nov 3, 2023
7b9a157
simplify waitForEsStatusGreen function
maximpn Nov 3, 2023
3077982
Fixed wait for kibana status available.
dkirchan Nov 3, 2023
cda8ecd
Enabled tests to run on proper kibana nodes with parallelism
dkirchan Nov 3, 2023
2482326
Removed parallel count as it is defined in the yaml file
dkirchan Nov 3, 2023
fb18ec6
Introducing a long timeout in and we will be able to redefine it when…
dkirchan Nov 3, 2023
954c08f
Fixed the ts script that is targeted in the preparation of the build
dkirchan Nov 3, 2023
691d8c7
Fixed delay before environment ready
dkirchan Nov 3, 2023
d25ab4e
Addressed comments around errors and burn number
dkirchan Nov 3, 2023
52a7b74
Merge branch 'main' into security/dkirchan-create-envs
dkirchan Nov 3, 2023
756665e
Merge branch 'main' into security/dkirchan-create-envs
MadameSheema Nov 6, 2023
b73cf3c
Merge branch 'main' into security/dkirchan-create-envs
MadameSheema Nov 6, 2023
46649ed
Split all tests to non investigations/explore
dkirchan Nov 6, 2023
91708d6
Removed dependencies in Buildkite jobs
dkirchan Nov 6, 2023
f10e5ff
Wait for 8 mins
dkirchan Nov 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ echo "--- Serverless Security Second Quality Gate"
cd x-pack/test/security_solution_cypress
set +e

VAULT_DEC_KEY=$(vault read -field=key secret/kibana-issues/dev/security-solution-qg-enc-key)
ENV_PWD=$(echo $TEST_ENV_PWD | openssl aes-256-cbc -d -a -pass pass:$VAULT_DEC_KEY)
QA_API_KEY=$(retry 5 5 vault read -field=qa_api_key secret/kibana-issues/dev/security-solution-qg-enc-key)

CYPRESS_ELASTICSEARCH_URL=$TEST_ENV_ES_URL CYPRESS_BASE_URL=$TEST_ENV_KB_URL CYPRESS_ELASTICSEARCH_USERNAME=$TEST_ENV_USERNAME CYPRESS_ELASTICSEARCH_PASSWORD=$ENV_PWD CYPRESS_KIBANA_URL=$CYPRESS_BASE_URL yarn cypress:run:qa:serverless; status=$?; yarn junit:merge || :; exit $status
PARALLEL_COUNT=4 CLOUD_QA_API_KEY=$QA_API_KEY yarn cypress:run:qa:serverless:parallel; status=$?; yarn junit:merge || :; exit $status
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain why was 4 selected as parallel_count?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the time being the specific PR runs the tests in a default Buildkite agent. It is extremely difficult to guess how many threads can be running in this agent. So a minimum acceptable number is hard coded here.

In a next PR for the relevant ticket this will be addressed and a totally similar way like the base.yml will be followed.

31 changes: 1 addition & 30 deletions x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,38 +28,9 @@ import {
import { createFailError } from '@kbn/dev-cli-errors';
import pRetry from 'p-retry';
import { renderSummaryTable } from './print_run';
import { isSkipped, parseTestFileConfig } from './utils';
import { parseTestFileConfig, retrieveIntegrations } from './utils';
import { getFTRConfig } from './get_ftr_config';

/**
* Retrieve test files using a glob pattern.
* If process.env.RUN_ALL_TESTS is true, returns all matching files, otherwise, return files that should be run by this job based on process.env.BUILDKITE_PARALLEL_JOB_COUNT and process.env.BUILDKITE_PARALLEL_JOB
*/
const retrieveIntegrations = (integrationsPaths: string[]) => {
const nonSkippedSpecs = integrationsPaths.filter((filePath) => !isSkipped(filePath));

if (process.env.RUN_ALL_TESTS === 'true') {
return nonSkippedSpecs;
} else {
// The number of instances of this job were created
const chunksTotal: number = process.env.BUILDKITE_PARALLEL_JOB_COUNT
? parseInt(process.env.BUILDKITE_PARALLEL_JOB_COUNT, 10)
: 1;
// An index which uniquely identifies this instance of the job
const chunkIndex: number = process.env.BUILDKITE_PARALLEL_JOB
? parseInt(process.env.BUILDKITE_PARALLEL_JOB, 10)
: 0;

const nonSkippedSpecsForChunk: string[] = [];

for (let i = chunkIndex; i < nonSkippedSpecs.length; i += chunksTotal) {
nonSkippedSpecsForChunk.push(nonSkippedSpecs[i]);
}

return nonSkippedSpecsForChunk;
}
};

export const cli = () => {
run(
async () => {
Expand Down
Loading