Preliminary support for on-prem transfers with HDFS and POSIX path support #735

ShishirPatil · 2023-01-19T05:30:24Z

Skyplane for on-prem to cloud object store transfers. This is focused on HDFS using GCP's Dataproc.

Rebase to upstream
Tune hyper-parameters
Testing
- Test HDFS on GCP Dataproc

This is the HDFS interface using PyArrow. In order to run this, it assumes that you have a working hadoop, jdk installed, and proper environmental variables set in the VM. (https://arrow.apache.org/docs/python/filesystems.html#hadoop-distributed-file-system-hdfs) Co-authored-by: Shishir Patil <[email protected]>

Reverts #684

Continued from previous PR Co-authored-by: Shishir Patil <[email protected]>

Debugging/Fixing Test_HDFS - Making it so that Test_HDFS deprovisions and terminates whenever there is an exception raised in the interface testing

skyplane/gateway/gateway_daemon.py

skyplane/gateway/gateway_sender.py

skyplane/obj_store/file_system_interface.py

skyplane/obj_store/object_store_interface.py

skyplane/utils/networking_tools.py

skyplane/cli/cli_impl/cp_replicate.py

JasonDing0401 · 2023-01-25T21:03:32Z

For the readthedocs error, some files (e.g. skyplane_cli_internal.rst) in doc/ in this branch are not updated with the main branch. Maybe merge from main will help?

skyplane/gateway/gateway_daemon.py

skyplane/cli/cli_transfer.py

skyplane/obj_store/object_store_interface.py

Co-authored-by: Zizhong Zhang <[email protected]>

Co-authored-by: Hailey Jang <Hailey Jang>

skyplane/cli/cli_transfer.py

skyplane/compute/cloud_provider.py

skyplane/compute/gcp/gcp_cloud_provider.py

skyplane/obj_store/hdfs_interface.py

tests/unit_gcs/test_dataproc.py

ShishirPatil and others added 17 commits November 16, 2022 01:53

File system interface

42aa7a9

Moving away from mnt

3b1e8b2

Iterator return object of type file

a5ee402

Merge branch 'dev/gateway' into 'dev/shishir/on-prem'

a103950

Revert "[onprem] HDFS Interface " (#716)

e9d333c

Reverts #684

[onprem] HDFS Interface (#719)

ee4fa78

Continued from previous PR Co-authored-by: Shishir Patil <[email protected]>

Fix test_hdfs workflow to terminate the cluster whenever

a1aa397

Upstream POSIX file system interface

295b37a

[On-prem] Fix Test_HDFS (#729)

353964e

Debugging/Fixing Test_HDFS - Making it so that Test_HDFS deprovisions and terminates whenever there is an exception raised in the interface testing

Catch multipart for hdfs

8357a77

Fix: test_hdfs bugs

6035bd6

Use gateways

dedb577

Fix: File system interface

2bbd728

port from aws to gcp

b16267e

List object fixed

75aae41

Fix: test hdfs for aws, posix file system

987b5b3

parasj reviewed Jan 25, 2023

View reviewed changes

skyplane/cli/cli_impl/cp_replicate.py Outdated Show resolved Hide resolved

HaileyJang reviewed Jan 26, 2023

View reviewed changes

skyplane/gateway/gateway_daemon.py Outdated Show resolved Hide resolved

HaileyJang added 2 commits January 26, 2023 19:30

Fix Merge to main merge conflict

131c5eb

Adding local, nfs, hdfs portion to CLI

f5a3241

HaileyJang reviewed Jan 27, 2023

View reviewed changes

skyplane/cli/cli_transfer.py Show resolved Hide resolved

Fix dependency issues

c21bd05

parasj changed the title ~~Dev/shishir/on prem~~ Preliminary support for on-prem transfers with HDFS and POSIX path support Jan 28, 2023

zizhong reviewed Jan 30, 2023

View reviewed changes

skyplane/obj_store/object_store_interface.py Outdated Show resolved Hide resolved

ShishirPatil and others added 2 commits January 30, 2023 19:17

Minor mods:Resolving comments for PR

9174d82

Merge branch 'main' into dev/shishir/on-prem

c99ca36

ShishirPatil mentioned this pull request Jan 30, 2023

[request] Enable multipart for on-prem transfers #741

Open

zizhong reviewed Feb 1, 2023

View reviewed changes

skyplane/obj_store/object_store_interface.py Outdated Show resolved Hide resolved

HaileyJang and others added 2 commits February 1, 2023 00:19

Fix upto gateway creation

5601cf3

Update skyplane/obj_store/object_store_interface.py

0497e5d

Co-authored-by: Zizhong Zhang <[email protected]>

ShishirPatil mentioned this pull request Feb 4, 2023

[request] Add unit tests for GCP and Azure for on-prem #745

Open

ShishirPatil and others added 3 commits February 14, 2023 19:23

Merge branch 'main' into dev/shishir/on-prem

f9042e3

fix: lint

a00829c

End-to-End Integration for HDFS (#758)

b8d766c

Co-authored-by: Hailey Jang <Hailey Jang>

ShishirPatil commented Feb 27, 2023

View reviewed changes

Fix legacy codes from previous runs

b1c0bfe

HaileyJang reviewed Feb 28, 2023

View reviewed changes

tests/unit_gcs/test_dataproc.py Outdated Show resolved Hide resolved

HaileyJang added 7 commits March 9, 2023 14:42

Fix all the comments

a66073c

Fix black pytype issue

9b9c445

Update Pyproject for dataproc

e05c323

Add readme

21e834d

Delete hostname

4cadac8

Fix merge conflict with main

b039b35

GCP test should pass now:

f3052d0

ShishirPatil merged commit e82cc5e into main Mar 13, 2023

ShishirPatil deleted the dev/shishir/on-prem branch March 13, 2023 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preliminary support for on-prem transfers with HDFS and POSIX path support #735

Preliminary support for on-prem transfers with HDFS and POSIX path support #735

ShishirPatil commented Jan 19, 2023 •

edited

Loading

JasonDing0401 commented Jan 25, 2023 •

edited

Loading

Preliminary support for on-prem transfers with HDFS and POSIX path support #735

Preliminary support for on-prem transfers with HDFS and POSIX path support #735

Conversation

ShishirPatil commented Jan 19, 2023 • edited Loading

JasonDing0401 commented Jan 25, 2023 • edited Loading

ShishirPatil commented Jan 19, 2023 •

edited

Loading

JasonDing0401 commented Jan 25, 2023 •

edited

Loading