Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0. #21070

Closed
wants to merge 11 commits into from
Closed

[SPARK-23972][BUILD][SQL] Update Parquet to 1.10.0. #21070

wants to merge 11 commits into from

Commits on May 7, 2018

  1. SPARK-23972: Update Parquet to 1.10.0. 

    This updates the vectorized path for changes in Parquet 1.10.0, which
uses ByteBufferInputStream instead of byte arrays in encoders. This
allows Parquet to break allocations into smaller chunks that are better
for garbage collection.
    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    26336a9 View commit details
    Browse the repository at this point in the history

  2. SPARK-23972: Update dependencies for Parquet 1.10.0.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    a599515 View commit details
    Browse the repository at this point in the history

  3. SPARK-23972: Fix method signatures for IOExceptions.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    3e1f84c View commit details
    Browse the repository at this point in the history

  4. SPAKR-23972: Bulk write to ColumnVector when Parquet data is on the h… 

    …eap.
    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    93b8ee8 View commit details
    Browse the repository at this point in the history

  5. SPARK-23972: Fix test failures and review comments.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    93f2a92 View commit details
    Browse the repository at this point in the history

  6. SPARK-23972: Add lz4, brotli, and zstd compression for Parquet.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    64907bc View commit details
    Browse the repository at this point in the history

  7. SPARK-23972: Update InMemoryColumnarQuerySuite for new Parquet version. 

    The new Parquet version adds some metadata so files are slightly larger.
This updates the test case that reports the file size for the size of an
in-memory relation.
    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    66df4b6 View commit details
    Browse the repository at this point in the history

  8. SPARK-23972: Regenerate golden outputs for Parquet size changes.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    ee2ffdd View commit details
    Browse the repository at this point in the history

  9. SPARK-23972: Update Hadoop 3.1 dependency profile.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    270540f View commit details
    Browse the repository at this point in the history

  10. SPARK-23972: Update sql-programming-guide.md.

    @rdblue
    rdblue committed May 7, 2018
    Configuration menu
    Copy the full SHA
    6c9d47b View commit details
    Browse the repository at this point in the history

Commits on May 8, 2018

  1. Fix buffer unpacking offset in VectorizedRleValuesReader.

    @rdblue
    rdblue committed May 8, 2018
    Configuration menu
    Copy the full SHA
    95ecde0 View commit details
    Browse the repository at this point in the history