Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Fix flakey test #202

Merged
merged 47 commits into from
Aug 19, 2024
Merged

chore: Fix flakey test #202

merged 47 commits into from
Aug 19, 2024

Conversation

juliev0
Copy link
Collaborator

@juliev0 juliev0 commented Aug 15, 2024 •
edited
Loading

Fixes #199

Modifications

  1. this fixes the flakey e2e test. Root cause was because the cluster for the e2e test testing "dataLossPrevention=true" wasn't clean, and still had a lingering Job running from the Pipeline of the previous test. Have described the issue that occurs here.
  2. Refactored a bit to subdivide the test which was previously in one big long file into multiple files. (would prefer to later maybe move these new files to a util subdirectory, and then they could be used by multiple tests)

Verification

Have run the e2e test 8 times in a row with no failures.

@juliev0
fix: if Pipeline is paused in order to update its spec, make sure it'…
29f85de 
…s left paused until new spec is reconciled

Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
logging
87dc0be 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
a358ec7 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
log
eef5fc1 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
merge main
8a1ec3e 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
add debug info to e2e
9479e08 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0 juliev0 changed the title Test e2e pending chore: Test e2e pending Aug 15, 2024
@juliev0
fix: empty commit
26a85bd 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
fd30def 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0 juliev0 mentioned this pull request Aug 15, 2024
Closed
@juliev0 juliev0 changed the title chore: Test e2e pending chore: Troubleshoot flakey test Aug 15, 2024
@juliev0
fix: empty commit
5ce88d1 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
4fcecad 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
614476c 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
4f8bef0 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
divide the e2e test into multiple files
d82c313 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
print Conditions for Pending Pod instead of Reason which isn't showin…
a0a592a 
…g up

Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
1aa8a0f 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
edfd0cd 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
269fd23 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
print init container log
fb53995 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
print init container log
cdb1d6a 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
bb49013 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
6fe61a8 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
75862fe 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
25e166a 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
8ce80be 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
try rearranging the order of testing
d3257da 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
69bd861 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
f4e16cd 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0 juliev0 mentioned this pull request Aug 17, 2024
Closed
@juliev0
fix: empty commit
92e2323 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
7bf3f81 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
56fdefb 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
create a new cluster for 2nd e2e test
0fbc374 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
1d02b2c 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
ab0aea3 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
d0ba912 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
fb312d5 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
Merge remote-tracking branch 'origin/main' into test-e2e-pending
17889ed
@juliev0
fix: empty commit
8c4c065 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
don't assume 'Namespace' value
417aceb 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
637cadd 
Signed-off-by: Julie Vogelman <[email protected]>
@juliev0
fix: empty commit
53498e0 
Signed-off-by: Julie Vogelman <[email protected]>
juliev0
juliev0 commented Aug 18, 2024
View reviewed changes
tests/e2e/common.go
isbServiceRolloutClient planepkg.ISBServiceRolloutInterface
numaflowControllerRolloutClient planepkg.NumaflowControllerRolloutInterface
kubeClient clientgo.Interface
)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all these variables need to be in a file not including "_test.go" in the name in order to be referenced from the other files

@juliev0
add namespace parameter
c2b163a 
Signed-off-by: Julie Vogelman <[email protected]>
juliev0
juliev0 commented Aug 18, 2024
View reviewed changes
.github/workflows/ci.yaml
echo '127.0.0.1 localhost' | sudo tee -a /etc/hosts
echo 'Waiting for the cluster to be ready...'
until kubectl cluster-info; do sleep 1; done

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this resolves the CI flakey test by creating a clean cluster before the second e2e test runs

@juliev0 juliev0 marked this pull request as ready for review August 18, 2024 06:11
@juliev0 juliev0 changed the title chore: Troubleshoot flakey test chore: Fix flakey test Aug 19, 2024
dpadhiar
dpadhiar approved these changes Aug 19, 2024
View reviewed changes
Copy link
Contributor

@dpadhiar dpadhiar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xdevxy xdevxy merged commit 4fa0054 into main Aug 19, 2024
8 checks passed
@xdevxy xdevxy deleted the test-e2e-pending branch August 19, 2024 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xdevxy xdevxy xdevxy approved these changes

@dpadhiar dpadhiar dpadhiar approved these changes

@afugazzotto afugazzotto Awaiting requested review from afugazzotto afugazzotto is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Flakey e2e test - "dataLossPrevention=true" - Numaflow Vertex Pods stuck in Pending state
3 participants
@juliev0 @dpadhiar @xdevxy