-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Run E2E tests on suites #5025
Conversation
f61027b
to
bb886ed
Compare
remaining=$(expr $i + $double) | ||
|
||
# if it can't be divided exactly so the last test will include the remaining tests. | ||
if [[ $remaining -lt $total_tests ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that some tests should always be in the same suite (i.e. executed in parallel). For example, all pubsub tests should be executed in parallel to avoid overloading Service Bus
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case all pubsub tests should be executed sequentially rather than parallel, is that you meant?
If yes, indeed, as you can see the test suite division is not explicit, I was wondering if it really needs to be. Thoughts?
An option might be prefixing the folder name with the desired test suite, something like:
1-pubsub_test_b
1-pubsub_test_a
Folders that not contains the prefix wouldn't have any restriction, how about that?
Another option is having subfolders
/pubsub/test_a
/pubsub/test_b
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, sorry, I meant sequentially.
Any implementation would be fine.
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
bb886ed
to
dc7c0be
Compare
This pull request has been automatically marked as stale because it has not had activity in the last 60 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions! |
This pull request has been automatically closed because it has not had activity in the last 67 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions! |
Description
This is an attempt to make our e2e test stability better. The general idea is to divide our tests in N suites that could be configurable by just setting up an environment variable "E2E_TEST_SUITE_COUNT". The test chunks are arbitrarily chosen, following the lexicographical order of the folders' name. If we intentionally want some tests to execute in same suite one option is just name the folder as 1-testA and 1-testB.
We expect the following improvements:
Issue reference
Please reference the issue this PR will close: #4962
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list: