[#2059] feat(client-python): Support Gravitino Virtual FileSystem in Python #3528

xloya · 2024-05-23T08:57:34Z

What changes were proposed in this pull request?

Support Gravitino Virtual File System in Python so that we can read and write Fileset storage data. The first PR only supports HDFS.

After research, the following popular cloud storages or companies have implemented their own FileSystem based on fsspec(https://filesystem-spec.readthedocs.io/en/latest/index.html):

S3(https://github.com/fsspec/s3fs)
Azure(https://github.com/fsspec/adlfs)
Gcs(https://github.com/fsspec/gcsfs)
OSS(https://github.com/fsspec/ossfs)
Databricks(https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/dbfs.py)
Snowflake(https://github.com/snowflakedb/snowflake-ml-python),

So this PR will implement GVFS based on the fsspec interface.

Why are the changes needed?

Fix: #2059

How was this patch tested?

Add some UTs and ITs.

clients/client-python/gravitino/filesystem/gvfs.py

noidname01 · 2024-05-28T00:34:22Z

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

xloya · 2024-05-28T01:44:20Z

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Yeah, actually current implementation is customized based on the ArrowWrapper provided by fsspec and HadoopFileSystem provided by PyArrow.

noidname01 · 2024-05-28T02:36:20Z

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Yeah, actually current implementation is customized based on the ArrowWrapper provided by fsspec and HadoopFileSystem provided by PyArrow.

Yeah, but why not just use their implementation and add our features (ex. lock, custom handlers) on them?

xloya · 2024-05-28T03:08:32Z

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Yeah, actually current implementation is customized based on the ArrowWrapper provided by fsspec and HadoopFileSystem provided by PyArrow.

Yeah, but why not just use their implementation and add our features (ex. lock, custom handlers) on them?

ArrowWrapper some logics will call the self methods, but our GVFS need check the path in the args if valid (such as start with gvfs://), this may cause a path exception, so we need customize the code.

xloya · 2024-05-28T03:37:29Z

Btw, the code is not complete yet, I need to do more tests about the interface compatibility.

noidname01 · 2024-05-28T03:39:51Z

Btw, the code is not complete yet, I need to do more tests about the interface compatibility.

Sure, would love to help if needed.

clients/client-python/gravitino/filesystem/gvfs.py

jerryshao · 2024-05-28T10:15:22Z

Does it support read/write now? I would like to do a local test to see the current status.

xloya · 2024-05-28T11:24:22Z

Does it support read/write now? I would like to do a local test to see the current status.

Currently I have tested the following interfaces and they can all be executed normally with the fileset mounts on HDFS:
Configure these in the env:

export HADOOP_HOME={HADOOP_HOME}/pack
export HADOOP_CONF_DIR={HADOOP_HOME}/conf
export CLASSPATH=`$HADOOP_HOME/bin/hdfs classpath --glob`

The python script:

from gravitino import gvfs

if __name__ == "__main__":
    server_uri = "http://localhost:8090"
    metalake_name = "test_metalake"
    fs = gvfs.GravitinoVirtualFileSystem(server_uri=server_uri, metalake_name=metalake_name)
    print(fs.ls(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset"))
    print(fs.info(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset"))
    print(fs.exists(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset"))
    with fs.open(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset/test3.txt", mode="wb") as output:
        print(output.writable())
        output.write(b"hello world")

    with fs.open(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset/test3.txt", mode="rb") as input:
        print(input.readable())
        print(input.read(1024))

xloya · 2024-06-04T11:15:33Z

Currently, Python integration testings lacks the support for some Docker environments, so I am considering not adding HDFS integration testings to the current PR, but supporting it separately in another PR (#3760) to reduce the difficulty of code review.

xloya · 2024-06-04T11:16:17Z

@jerryshao @xunliu @shaofengshi @noidname01 This PR is ready for review, could you take a look when you have time? Thanks.

clients/client-python/gravitino/filesystem/gvfs.py

jerryshao · 2024-06-05T21:27:07Z

I will finish the review before this weekend, thanks a lot @xloya for your work.

jerryshao · 2024-06-07T05:35:39Z

@noidname01 can you please also help to review?

noidname01 · 2024-06-07T05:39:17Z

@noidname01 can you please also help to review?

Sure, I will review on this weekend.

xloya · 2024-06-12T11:30:11Z

clients/client-python/tests/unittests/test_gvfs_with_local.py

+            "test_pandas", f"{_fileset_dir}/test_pandas"
+        ),
+    )
+    def test_pandas(self, mock_method1, mock_method2, mock_method3, mock_method4):


Add some unit tests for engines' apis like pandas, pyarrow, llama_index.

xloya · 2024-06-13T12:14:32Z

@jerryshao @noidname01 Have addressed the comments, could you take a look again? Thanks.

clients/client-python/gravitino/filesystem/gvfs.py

noidname01 · 2024-06-16T08:18:38Z

LGTM, @jerryshao WDYT?

jerryshao · 2024-06-17T04:07:03Z

Sorry for late response, I need to take another round of review, will finish this before Tuesday.

clients/client-python/gravitino/filesystem/gvfs.py

licenses/cachetools.txt

clients/client-python/gravitino/filesystem/gvfs.py

jerryshao · 2024-06-18T06:28:14Z

clients/client-python/requirements.txt

+readerwriterlock==1.0.9
+fsspec==2024.3.1
+pyarrow


Do we need to specify the version of pyarrow here?

The pyarrow version is already specified in requirement-dev.txt, which I think is no longer needed here.

jerryshao · 2024-06-18T06:28:40Z

clients/client-python/requirements-dev.txt

@@ -6,3 +6,7 @@ pylint==3.2.2
 black==24.4.2
 twine==5.1.0
 coverage==7.5.1
+pandas==2.0.3
+pyarrow==15.0.2
+llama-index==0.10.40


Why do we need package like pandas, llama-index?

This is because unit tests and integration tests will include tests with these third-party libs' APIs. In order to ensure the stability of unit tests and integration tests, the corresponding feasible versions are specified here. I'm not sure if we need to add tests for integration with third-party libs.

LICENSE

jerryshao

LGTM, thanks @xloya for your contribution.

…em in Python (apache#3528) ### What changes were proposed in this pull request? Support Gravitino Virtual File System in Python so that we can read and write Fileset storage data. The first PR only supports HDFS. After research, the following popular cloud storages or companies have implemented their own FileSystem based on fsspec(https://filesystem-spec.readthedocs.io/en/latest/index.html): 1. S3(https://github.com/fsspec/s3fs) 2. Azure(https://github.com/fsspec/adlfs) 3. Gcs(https://github.com/fsspec/gcsfs) 4. OSS(https://github.com/fsspec/ossfs) 5. Databricks(https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/dbfs.py) 6. Snowflake(https://github.com/snowflakedb/snowflake-ml-python), So this PR will implement GVFS based on the fsspec interface. ### Why are the changes needed? Fix: apache#2059 ### How was this patch tested? Add some UTs and ITs. --------- Co-authored-by: xiaojiebao <[email protected]>

…tion tests (#3876) ### What changes were proposed in this pull request? Add Hive Docker env for client-python, and add integration tests for PyGVFS + HDFS. Depends on #3528. ### Why are the changes needed? Fix: #3760 ### How was this patch tested? Add some ITs. --------- Co-authored-by: xiaojiebao <[email protected]>

…Python (#3528) ### What changes were proposed in this pull request? Support Gravitino Virtual File System in Python so that we can read and write Fileset storage data. The first PR only supports HDFS. After research, the following popular cloud storages or companies have implemented their own FileSystem based on fsspec(https://filesystem-spec.readthedocs.io/en/latest/index.html): 1. S3(https://github.com/fsspec/s3fs) 2. Azure(https://github.com/fsspec/adlfs) 3. Gcs(https://github.com/fsspec/gcsfs) 4. OSS(https://github.com/fsspec/ossfs) 5. Databricks(https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/dbfs.py) 6. Snowflake(https://github.com/snowflakedb/snowflake-ml-python), So this PR will implement GVFS based on the fsspec interface. ### Why are the changes needed? Fix: #2059 ### How was this patch tested? Add some UTs and ITs. --------- Co-authored-by: xiaojiebao <[email protected]> (cherry picked from commit cddef88)

xloya force-pushed the support-python-virtual-filesystem branch from 2420ab8 to 18dcf58 Compare May 23, 2024 09:05

xloya mentioned this pull request May 23, 2024

[Subtask] Add Python Arrow FileSystem implementation for fileset. #2059

Closed

xloya force-pushed the support-python-virtual-filesystem branch 4 times, most recently from 52b038a to 230b4d4 Compare May 24, 2024 07:55

shaofengshi reviewed May 24, 2024

View reviewed changes

clients/client-python/gravitino/filesystem/gvfs.py Outdated Show resolved Hide resolved

xloya force-pushed the support-python-virtual-filesystem branch 2 times, most recently from 08f8477 to c25726c Compare May 27, 2024 07:30

jerryshao reviewed May 28, 2024

View reviewed changes

clients/client-python/gravitino/filesystem/gvfs.py Outdated Show resolved Hide resolved

jerryshao reviewed May 28, 2024

View reviewed changes

clients/client-python/gravitino/filesystem/gvfs.py Outdated Show resolved Hide resolved

xloya force-pushed the support-python-virtual-filesystem branch 4 times, most recently from a9abe92 to 8156fe6 Compare June 4, 2024 09:29

xloya marked this pull request as ready for review June 4, 2024 11:13

xloya commented Jun 4, 2024

View reviewed changes

clients/client-python/gravitino/filesystem/gvfs.py Show resolved Hide resolved

xiaojiebao added 7 commits June 12, 2024 09:35

remove useless code

92e640e

fix style

7e7df75

add uts

77fb792

add its

fc9ced8

remove its

21051f2

fix

5a88b7b

fix comments

7c87f33

xloya force-pushed the support-python-virtual-filesystem branch from 3881a31 to 7c87f33 Compare June 12, 2024 10:23

fix

9f85276

xloya commented Jun 12, 2024

View reviewed changes

noidname01 reviewed Jun 14, 2024

View reviewed changes

clients/client-python/gravitino/filesystem/gvfs.py Show resolved Hide resolved

xiaojiebao added 3 commits June 14, 2024 09:55

Merge branch 'main' into support-python-virtual-filesystem

63385aa

using cachetools.TTLCache

e8dab11

fix

9041575

Merge branch 'main' into support-python-virtual-filesystem

33ebdef

xloya mentioned this pull request Jun 17, 2024

[#3760] improvement(client-python): Add Docker env and PyGVFS Integration tests #3876

Merged

xiaojiebao added 2 commits June 17, 2024 16:25

fix uts

e51bb78

align cache default expired time

c3bf70d

jerryshao reviewed Jun 18, 2024

View reviewed changes

xiaojiebao added 2 commits June 18, 2024 15:20

fix comments

21da01f

fix

10866b4

jerryshao assigned xloya Jun 18, 2024

jerryshao approved these changes Jun 18, 2024

View reviewed changes

jerryshao merged commit cddef88 into apache:main Jun 18, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#2059] feat(client-python): Support Gravitino Virtual FileSystem in Python #3528

[#2059] feat(client-python): Support Gravitino Virtual FileSystem in Python #3528

xloya commented May 23, 2024 •

edited

Loading

noidname01 commented May 28, 2024

xloya commented May 28, 2024

noidname01 commented May 28, 2024

xloya commented May 28, 2024 •

edited

Loading

xloya commented May 28, 2024

noidname01 commented May 28, 2024

jerryshao commented May 28, 2024

xloya commented May 28, 2024 •

edited

Loading

xloya commented Jun 4, 2024 •

edited

Loading

xloya commented Jun 4, 2024

jerryshao commented Jun 5, 2024

jerryshao commented Jun 7, 2024

noidname01 commented Jun 7, 2024

xloya Jun 12, 2024

xloya commented Jun 13, 2024

noidname01 commented Jun 16, 2024

jerryshao commented Jun 17, 2024

jerryshao Jun 18, 2024

xloya Jun 18, 2024

jerryshao Jun 18, 2024

xloya Jun 18, 2024

jerryshao left a comment

[#2059] feat(client-python): Support Gravitino Virtual FileSystem in Python #3528

[#2059] feat(client-python): Support Gravitino Virtual FileSystem in Python #3528

Conversation

xloya commented May 23, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

noidname01 commented May 28, 2024

xloya commented May 28, 2024

noidname01 commented May 28, 2024

xloya commented May 28, 2024 • edited Loading

xloya commented May 28, 2024

noidname01 commented May 28, 2024

jerryshao commented May 28, 2024

xloya commented May 28, 2024 • edited Loading

xloya commented Jun 4, 2024 • edited Loading

xloya commented Jun 4, 2024

jerryshao commented Jun 5, 2024

jerryshao commented Jun 7, 2024

noidname01 commented Jun 7, 2024

xloya Jun 12, 2024

Choose a reason for hiding this comment

xloya commented Jun 13, 2024

noidname01 commented Jun 16, 2024

jerryshao commented Jun 17, 2024

jerryshao Jun 18, 2024

Choose a reason for hiding this comment

xloya Jun 18, 2024

Choose a reason for hiding this comment

jerryshao Jun 18, 2024

Choose a reason for hiding this comment

xloya Jun 18, 2024

Choose a reason for hiding this comment

jerryshao left a comment

Choose a reason for hiding this comment

xloya commented May 23, 2024 •

edited

Loading

xloya commented May 28, 2024 •

edited

Loading

xloya commented May 28, 2024 •

edited

Loading

xloya commented Jun 4, 2024 •

edited

Loading