added more fallback tests + using a file #873
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
xTRam1
commented
Jun 16, 2023
xTRam1
commented
- Modified the tests so that they load from an actual quota file instead of me defining a dictionary.
- Modified planner so that it can accept a file name for the quota limits (default to the skyplane config quota files)
- Added more tests for error conditions (no quota file is provided + quota file is provided but the requested region is not included in the quota file)
…is actually exists
Remove duplicated planners
Instead of storing the vcpu limits as a hardcoded variable inside Planner, created a new .csv file that includes that information. When Planner is created, we read this csv file. Wrote test cases that include fake quota limits.
This allows for testing transfers locally, both for synthetic data and real object store data.
…ne-project#865) S3 currently does not have proper bucket cleanup after testing leading to `TooManyBuckets` errors on tests after a certain point. This adds the cleanup logic to every integration test. Furthermore, the hadoop jdk installs were removed as they should not be required anymore.
…kyplane-project#868) This fixes two bugs: * During chunk dispatching for CLI transfers, Ctrl-C could not be used to exit out of the transfer due to threads not being properly filled. This is fixed by adding exit signals to the tracker and stopped the multipart threads when the tracker is exited. * Previous `self.auth` in the `Server` instance was modified to contain credentials for all clouds that are initialized, however this was not properly updated for child classes.
xTRam1
commented
Jun 16, 2023
| if os.path.exists(gcp_quota_path): | ||
| with gcp_quota_path.open("r") as f: | ||
| self.quota_limits["gcp"] = json.load(f) | ||
| quota_limits = {} |
There was a problem hiding this comment.
Planner can take custom quota limit files for better testing
xTRam1
commented
Jun 16, 2023
| @@ -121,7 +125,9 @@ def _calculate_vm_types(self, region_tag: str) -> Optional[Tuple[str, int]]: | |||
|
|
|||
| # No quota limits (quota limits weren't initialized properly during skyplane init) | |||
| if quota_limit is None: | |||
| logger.warning(f"Quota limit file not found for {region_tag}") | |||
| logger.warning( | |||
There was a problem hiding this comment.
Improved the warning message
sarahwooders
added a commit
that referenced
this pull request
Jun 22, 2023
* Modified the tests so that they load from an actual quota file instead of me defining a dictionary. * Modified planner so that it can accept a file name for the quota limits (default to the skyplane config quota files) * Added more tests for error conditions (no quota file is provided + quota file is provided but the requested region is not included in the quota file) --------- Co-authored-by: Sarah Wooders <[email protected]> Co-authored-by: Asim Biswal <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.