-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[core] Migrate many_nodes_actor_tests to new cloud. #31863
Conversation
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
572ce8b
to
2a86f62
Compare
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
Signed-off-by: Yi Cheng <[email protected]>
Given there is regression in both infra and oss side, I'll adjust the number later. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what do you mean by the regression btw?
@@ -85,7 +87,12 @@ def run_prepare_command( | |||
Command runners may choose to run this differently than the | |||
test command. | |||
""" | |||
return self.run_command(command, env, timeout) | |||
return exponential_backoff_retry( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is this change for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Somehow in the new stack it needs waiting for a while (1-3s) before it's ready to be used
The infra can't start 2k nodes and hangs and oss ray somehow GCS disappear. :( |
can you run the release test before merging it? |
@rkooo567 I triggered the test hours ago (https://buildkite.com/ray-project/release-tests-pr/builds/26327#0185e53f-d293-4e2a-a34b-b6aa149e30ab) but still pending :( |
passed |
Why are these changes needed?
This PR make the test run with the new cloud to prevent regression.
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.