Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-5341]Fix test write parquet with compression codec #5424

Merged
merged 3 commits into from
Apr 17, 2024

Conversation

ayushi-agarwal
Copy link
Contributor

@ayushi-agarwal ayushi-agarwal commented Apr 16, 2024

What changes were proposed in this pull request?

LZ4_RAW is currently not supported in velox, so fallback to non-native write when user specifies this compression codec.
It got introduced in spark 3.5 https://issues.apache.org/jira/browse/SPARK-43273

(Partially Fixes: #5341)

How was this patch tested?

Ran UTs

Copy link

#5341

Copy link

Run Gluten Clickhouse CI

@ayushi-agarwal
Copy link
Contributor Author

@JkSelf Could you please review this PR

Copy link

Run Gluten Clickhouse CI

@ayushi-agarwal
Copy link
Contributor Author

ayushi-agarwal commented Apr 16, 2024

The tests won't pass till issue #5427 gets fixed as reading this codec is not supported in velox. I will disable these tests.

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link
Contributor

@Yohahaha Yohahaha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Thanks.

@ayushi-agarwal
Copy link
Contributor Author

@JkSelf @Yohahaha Thanks for reviewing. Could you please merge it if everything looks good.

@zhli1142015 zhli1142015 merged commit bd2ca45 into apache:main Apr 17, 2024
39 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_5424_time.csv log/native_master_04_16_2024_33d183203_time.csv difference percentage
q1 36.07 38.05 1.973 105.47%
q2 28.32 24.39 -3.930 86.12%
q3 38.51 37.50 -1.011 97.37%
q4 41.14 38.79 -2.352 94.28%
q5 69.30 70.62 1.325 101.91%
q6 6.95 9.68 2.727 139.22%
q7 86.12 83.76 -2.360 97.26%
q8 84.42 86.47 2.052 102.43%
q9 124.35 124.25 -0.098 99.92%
q10 44.56 47.10 2.541 105.70%
q11 20.63 20.01 -0.618 97.00%
q12 26.40 30.02 3.619 113.71%
q13 55.10 53.95 -1.147 97.92%
q14 18.93 17.11 -1.815 90.41%
q15 29.43 29.43 -0.000 100.00%
q16 13.83 13.85 0.016 100.12%
q17 103.15 101.18 -1.967 98.09%
q18 144.02 144.50 0.477 100.33%
q19 13.57 16.75 3.175 123.39%
q20 27.72 26.88 -0.843 96.96%
q21 290.26 287.81 -2.456 99.15%
q22 14.57 16.10 1.525 110.47%
total 1317.36 1318.19 0.833 100.06%

Preetesh2110 pushed a commit to Preetesh2110/incubator-gluten that referenced this pull request Apr 25, 2024
[GLUTEN-5341] Fix test write parquet with compression codec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

UT failures for Spark3.5
5 participants