-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS firewall configuration via security groups #239
Conversation
Follow up from today's meeting: we should reuse the same VPC + SG and not clear it (except in |
@parasj Can you list out the exact scenario you are thinking for which this would be useful? I can then reason about the best way to handle it. |
@ShishirPatil Setup:
This will allow concurrent transfers to occur. |
In a new issue: we should remove the client from the SG and use port 22 to copy a JSON to each instance containing the initial gateway ChunkRequests using SFTP (already implemented) then call curl via SSH to actually start the transfer. |
We cannot support concurrent transfers with the current architecture. This is because - on the source bucket side we will first check if there are any instances up. If there are, then we will just start re-using those instances. However, we don't check if those instances are busy, already engaged in a transfer. |
* Switch to ILock from oslo.concurrency's lockutils * Change behavior of remove_ip_from_security_group to instead remove * Remove make_vpc from add_ip_to_security_group and instead call it explicitly * Firewall rules also called old init jobs, fixed by redefining jobs * Remove add_ip_to_security_group and remove_ip_from_security_group from GCP/Azure since those two clouds have different terminology
* Clean up instance profiles in deprovision * Fix bug in #239 * Fix pytype * Update deprovisoin logic * Cache pytype * Fix pytype * Disable pylint
Skyplane now supports concurrent transfers in a secure manner. Every instance's ip is manually added to the SG at the start of transfer, and removed from the SG at the end of a transfer.
* Clean up instance profiles in deprovision * Fix bug in #239 * Fix pytype * Update deprovisoin logic * Cache pytype * Fix pytype * Disable pylint
…ation (#323) * Fix #318 by passing project_id during setup to region config serialization * Try another fix * Fig * Fix project ID * Fix bug with GCS transfers * Add internal CLI commands * Exit replicator client upon any errors (#324) * AWS firewall configuration via security groups (#239) Skyplane now supports concurrent transfers in a secure manner. Every instance's ip is manually added to the SG at the start of transfer, and removed from the SG at the end of a transfer. * Fix bug in #239 * Add CLI option to use BBR for transfers (#331) * Add option to use BBR in skylark cli * format * Update * Fix pytype * Open tunnel * Bold font * Update * Update * Update * Increase connections * Clean up instance profiles in deprovision (#334) * Clean up instance profiles in deprovision * Fix bug in #239 * Fix pytype * Update deprovisoin logic * Cache pytype * Fix pytype * Disable pylint * Fix issue Co-authored-by: Shishir Patil <[email protected]>
Adds firewalls
Fixes #272