Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding support for reading table into ParquetFile #163

Merged
merged 3 commits into from
Aug 1, 2023
Merged

Adding support for reading table into ParquetFile #163

merged 3 commits into from
Aug 1, 2023

Commits on Jul 31, 2023

  1. Adding support for reading just the parquet meta by leveraging Parque… 

    …tFile

[Description] Right now we read the entire table using read_parquet method of pyarrow. However, we lose the parquet metadata in the process and this will eagerly load all the data. In this commit we will use ParquetFile (https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetFile.html)

[Motivation] #162
    @raghumdani
    raghumdani committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    e04bc19 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Using pyarrow.fs.S3FileSystem instead of s3fs

    @raghumdani
    raghumdani committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    b9b4997 View commit details
    Browse the repository at this point in the history

  2. Adding s3 adaptive retries and validate encoding

    @raghumdani
    raghumdani committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    68e9b76 View commit details
    Browse the repository at this point in the history