Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable parallel option in Delta Lake product tests #18750

Closed
wants to merge 7 commits into from

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Aug 21, 2023

Description

Delta Product tests sometimes take 2 hours in total.
Relates to #18805

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@wendigo
Copy link
Contributor

wendigo commented Aug 21, 2023

Does it work as expected? How can I verify that test indeed was executed in parallel?

@ebyhr ebyhr changed the title Enable parallel option in TestDeltaLakeColumnMappingMode Enable parallel option in Delta Lake product tests Aug 21, 2023
@findinpath findinpath requested a review from pajaks August 21, 2023 11:56
@@ -164,7 +164,7 @@
<dep.jsonwebtoken.version>0.11.5</dep.jsonwebtoken.version>
<dep.oracle.version>21.9.0.0</dep.oracle.version>
<dep.drift.version>1.21</dep.drift.version>
<dep.tempto.version>200</dep.tempto.version>
<dep.tempto.version>201</dep.tempto.version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes for other reviewers: relates to trinodb/tempto#122

@ebyhr
Copy link
Member Author

ebyhr commented Aug 23, 2023

How can I verify that test indeed was executed in parallel?

The log indicates tests ran in parallel. However, suite-delta-lake-oss faced concurrency issues. I will investigate the cause.

@ebyhr ebyhr force-pushed the ebi/delta-pt-speed branch 2 times, most recently from 5f47a00 to 82f97d5 Compare August 25, 2023 04:20
@ebyhr ebyhr marked this pull request as draft August 25, 2023 06:30
@ebyhr ebyhr force-pushed the ebi/delta-pt-speed branch 2 times, most recently from fb550db to e50229f Compare August 28, 2023 09:36
@ebyhr
Copy link
Member Author

ebyhr commented Sep 5, 2023

Closing as we're trying a different approach.

@ebyhr ebyhr closed this Sep 5, 2023
@ebyhr ebyhr deleted the ebi/delta-pt-speed branch September 5, 2023 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

3 participants