Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: windows smoke test list #11797

Merged
merged 75 commits into from
Jan 18, 2023
Merged
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
52007b9
ci: migrated to new test strategy
awsluja Dec 29, 2022
52979de
chore: add other tests and migration too
awsluja Dec 29, 2022
dc39171
chore: cleanup
awsluja Dec 29, 2022
d560079
chore: cleanup
awsluja Dec 29, 2022
0a19d69
chore: cleanup unused e2e tests
awsluja Dec 29, 2022
00ea815
chore: remove generated file
awsluja Dec 29, 2022
c38f04c
chore: remove parallelization blocker
awsluja Dec 29, 2022
72bc406
chore: force parallel runs
awsluja Dec 29, 2022
8886c1e
chore: cleanup test list
awsluja Dec 29, 2022
f85c2ff
chore: disable retries and cleanup exclusions
awsluja Dec 29, 2022
e1c82af
chore: bump test count to fill gaps
awsluja Dec 29, 2022
5c7a51c
chore: move random id earlier in project name
awsluja Dec 29, 2022
1c408b7
chore: update yarn version
awsluja Dec 29, 2022
e93880e
chore: fix typo
awsluja Dec 29, 2022
a5d077e
chore: try to fix delete issues
awsluja Dec 29, 2022
dcb75f7
chore: fix teardown on delete error
awsluja Dec 29, 2022
f0f7787
Merge branch 'dev' into e2e-multi-core-optimizations
awsluja Dec 29, 2022
a9e08cb
ci: create custom file system mutex
awsluja Dec 30, 2022
82fed6e
chore: cleanup logs
awsluja Dec 30, 2022
e1b32af
ci: increase randomness of initial lock creation
awsluja Dec 30, 2022
7afc0ad
chore: move migration tests to small isolated vms
awsluja Dec 30, 2022
55111a8
chore: adjust concurrency
awsluja Dec 30, 2022
eed73be
chore: update naming
awsluja Dec 30, 2022
ebd0be2
ci: update reporter to better handle concurrent tests
awsluja Dec 30, 2022
b38e298
ci: add use parent account exceptions
awsluja Dec 30, 2022
4842214
chore: cleanup naming, fix windows artifacts
awsluja Dec 30, 2022
52094aa
chore: use data to optimize test scheduling
awsluja Dec 31, 2022
4341105
chore: cleanup
awsluja Dec 31, 2022
11d2e94
chore: cleanup
awsluja Dec 31, 2022
3b7efb3
chore: fix call flow
awsluja Dec 31, 2022
62b9f9e
chore: workaround duplicate spinner bugs
awsluja Dec 31, 2022
c1678e1
ci: split api_2 test
awsluja Dec 31, 2022
304c345
chore: increase push timeout
awsluja Dec 31, 2022
c1c2ef9
ci: add method to run solo tests
awsluja Dec 31, 2022
bcde99e
ci: configure retries compatible with worker pools
awsluja Dec 31, 2022
9482eb4
ci: workaround concurrent polling limits
awsluja Dec 31, 2022
ae166d6
chore: fix bug with solo tests
awsluja Dec 31, 2022
31c0f81
chore: move import & geo tests to run solo
awsluja Dec 31, 2022
1d110fe
chore: add artifact scripts
awsluja Dec 31, 2022
ce925d0
chore: trigger workflow results
awsluja Dec 31, 2022
e64f8b1
ci: fix artifact path
awsluja Dec 31, 2022
c0b086e
Merge branch 'dev' into e2e-multi-core-optimizations
awsluja Jan 4, 2023
3673c22
Merge branch 'dev' into e2e-multi-core-optimizations
awsluja Jan 10, 2023
bdd4eea
chore: cleanup
awsluja Jan 10, 2023
817c60e
chore: add timer spy
awsluja Jan 10, 2023
c0e2df5
Merge branch 'dev' into e2e-multi-core-optimizations
awsluja Jan 10, 2023
1c3ad43
ci: update test lists
awsluja Jan 10, 2023
565fe8f
chore: cleanup
awsluja Jan 10, 2023
fe810ab
ci: fix retries
awsluja Jan 10, 2023
a6645fe
ci: move delays to createProjectDir
awsluja Jan 10, 2023
6f93e9a
ci: use same script for e2e as linux on windows
awsluja Jan 10, 2023
d6fafa1
chore: split storage-1 test
awsluja Jan 10, 2023
6e1df49
ci: fix config
awsluja Jan 10, 2023
5f0919b
ci: add bash env
awsluja Jan 10, 2023
b02b80e
ci: reduce timeout on push
awsluja Jan 11, 2023
0a136ca
ci: update test lists
awsluja Jan 11, 2023
3ec4e50
ci: update test lists
awsluja Jan 11, 2023
70ff919
ci: lower timeout and enable concurrent migration testing
awsluja Jan 11, 2023
be2e50f
ci: update test list
awsluja Jan 11, 2023
214fd0f
ci: restructure migration test setups
awsluja Jan 11, 2023
538a33f
ci: cleanup to use file names instead of old job names
awsluja Jan 11, 2023
d3054fc
chore: refactor
awsluja Jan 11, 2023
6fb874e
ci: fix dependencies
awsluja Jan 11, 2023
fb590c5
ci: attempt to fix retry on windows
awsluja Jan 12, 2023
fa1b2d0
chore: revert retry changes
awsluja Jan 12, 2023
c31f7e7
chore: undo changes to addCircleCiTags
awsluja Jan 13, 2023
01f46b0
chore: address PR feedback
awsluja Jan 13, 2023
8370b71
chore: undo changes to spinners
awsluja Jan 14, 2023
ae885d2
chore: cleanup unused files
awsluja Jan 17, 2023
a88c2b4
chore: lint & formatting
awsluja Jan 17, 2023
06d60f7
chore: lint, remove unused import
awsluja Jan 17, 2023
b9d7daa
chore: undo changes to initialize-env.ts
awsluja Jan 17, 2023
ef7ffb8
Merge branch 'dev' into e2e-multi-core-optimizations
awsluja Jan 17, 2023
bce4eff
ci: windows smoke test list for non-dev branches
awsluja Jan 17, 2023
07357b6
Merge branch 'dev' into windows-smoke-test-list
awsluja Jan 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions scripts/split-e2e-tests-v2.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,66 @@ const RUN_SOLO = [
'src/__tests__/schema-auth-15.test.ts',
'src/__tests__/schema-connection-1.test.ts',
];
/**
* Most Windows tests only run on 'dev', except for this list of smoke tests.
* This list should contain ~30 high quality tests that provide good overall coverage.
*
* All other tests will eventually run on 'dev'.
*/
const WINDOWS_SMOKE_TESTS = [
// api, lambda, multi-env
'src/__tests__/api_lambda_auth_1.test.ts',
// auth with api, trigger, and function that depends on api
'src/__tests__/auth_2e.test.ts',
// js with with all auth options
'src/__tests__/auth_3c.test.ts',
// headless auth with totp & sms
'src/__tests__/auth_5b.test.ts',
// headless auth update resource
'src/__tests__/auth_5e.test.ts',
// js with all auth options and front end config
'src/__tests__/auth_6.test.ts',
// auth import
'src/__tests__/auth_7b.test.ts',
// flutter with auth
'src/__tests__/auth_8b.test.ts',
// android
'src/__tests__/auth_11.test.ts',
// auth with multiple triggers
'src/__tests__/auth-trigger.test.ts',
// configure
'src/__tests__/congifure-project.test.ts',
// api with containers and secrets
'src/__tests__/containers-api-secrets.test.ts',
// api with dynamodb & lambda, custom policies
'src/__tests__/custom_policies_function.test.ts',
// env
'src/__tests__/env-1.test.ts',
// export and pull
'src/__tests__/export-pull-a.test.ts',
// functions
'src/__tests__/function_10.test.ts',
// notifications with function permissions
'src/__tests__/function-permissions.test.ts',
// geo
'src/__tests__/geo-add-a.test.ts',
// global sandbox
'src/__tests__/global_sandbox-b.test.ts',
// hooks
'src/__tests__/hooks-b.test.ts',
// hosting
'src/__tests__/hostingPROD.test.ts',
// import, s3
'src/__tests__/import_s3_2b.test.ts',
// interactions
'src/__tests__/interactions.test.ts',
// schema auth test
'src/__tests__/schema-auth-1a.test.ts',
// schema model test
'src/__tests__/schema-model-e.test.ts',
// interative deployments
'src/__tests__/schema-iterative-update-1.test.ts',
];
const TEST_EXCLUSIONS: { l: string[]; w: string[] } = {
l: [],
w: [
Expand Down Expand Up @@ -170,6 +230,11 @@ export const splitTestsV2 = function splitTests(
if (TEST_EXCLUSIONS[os].find(excluded => test === excluded)) {
continue;
}
// when we are not running E2E on 'dev', we only run a subset of tests on Windows
const isNotDevBranch = process.env.CIRCLE_BRANCH !== 'dev';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (isNotDevBranch && os === 'w' && !WINDOWS_SMOKE_TESTS.includes(test)) {
Comment on lines +234 to +235
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const isNotDevBranch = process.env.CIRCLE_BRANCH !== 'dev';
if (isNotDevBranch && os === 'w' && !WINDOWS_SMOKE_TESTS.includes(test)) {
const isDevBranch = process.env.CIRCLE_BRANCH === 'dev';
if (!isDevBranch && os === 'w' && !WINDOWS_SMOKE_TESTS.includes(test)) {

continue; // skip this test
}
const US_WEST_2 = FORCE_US_WEST_2.find(t => test.startsWith(t));
const USE_PARENT = USE_PARENT_ACCOUNT.some(usesParent => test.startsWith(usesParent));

Expand Down