-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Skyplane flytekit plugin #2860
base: master
Are you sure you want to change the base?
Skyplane flytekit plugin #2860
Conversation
Signed-off-by: 10sharmashivam <[email protected]>
Signed-off-by: 10sharmashivam <[email protected]>
Signed-off-by: 10sharmashivam <[email protected]>
Signed-off-by: 10sharmashivam <[email protected]>
Signed-off-by: 10sharmashivam <[email protected]>
Signed-off-by: 10sharmashivam <[email protected]>
Thank you for opening this pull request! 🙌 These tips will help get your PR across the finish line:
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2860 +/- ##
===========================================
+ Coverage 45.53% 95.08% +49.55%
===========================================
Files 196 38 -158
Lines 20418 1527 -18891
Branches 2647 0 -2647
===========================================
- Hits 9298 1452 -7846
+ Misses 10658 75 -10583
+ Partials 462 0 -462 ☔ View full report in Codecov by Sentry. |
Signed-off-by: 10sharmashivam <[email protected]>
Quick update: I’ve tried implementing the error handling and logging as planned. This should make debugging and future maintenance easier. Let me know if there’s anything specific you’d like me to adjust! |
@10sharmashivam thank you! as per testing, looks like AWS Free Tier includes 5GB of S3 standard storage, would you be able to try it? |
Tracking issue
Reference Issue- Skyplane flytekit plugin
Why are the changes needed?
Skyplane plugin adds the capability of transferring data using Skyplane to Flyte. Skyplane allows for significantly faster and cheaper data transfers compared to traditional methods, enhancing the overall efficiency of data workflows.
What changes were proposed in this pull request?
How was this patch tested?
I have not been able to run a full end-to-end test as I currently do not have access to cloud storage locations. However, I validated the general structure of the plugin, and I’d appreciate any recommendations for accessible cloud storage locations for testing purposes.
Additional Comments
I’m submitting this PR as an initial implementation of the Skyplane plugin integration. At this stage, I’ve structured the plugin, created the SkyplaneJob and SkyplaneFunctionTask classes, and registered the plugin with Flyte.
And as mentioned in some old comments on parent issue for this PR, about launching machines and apache arrow based file system (I believe that’s what arrow FS means there, please let me know if I am correct), how can I go about it?
I’d greatly appreciate any feedback or guidance to ensure that I’m on the right track with this implementation, especially regarding the integration with Skyplane’s data transfer features. I’ll continue to refine the code and address any changes based on the team’s suggestions.
Looking forward to your insights!
Next Steps-
I plan to enhance error handling and logging in the upcoming iterations of this Skyplane plugin.
And likewise, this plugin will mature.
Check all the applicable boxes