Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI #13920

Merged
merged 6 commits into from
Aug 23, 2022
Merged

ARROW-17451: [CI][Java] Use manylinux2014 image for JNI #13920

merged 6 commits into from
Aug 23, 2022

Conversation

kou
Copy link
Member

@kou kou commented Aug 19, 2022

Because our official .jar packages are built in manylinux2014 image.

@github-actions
Copy link

https://issues.apache.org/jira/browse/ARROW-17451

@github-actions
Copy link

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@kou
Copy link
Member Author

kou commented Aug 19, 2022

Let's confirm CI time of https://github.com/apache/arrow/runs/7914727219?check_suite_focus=true .

@kou kou force-pushed the ci-java-jni-use-manylinux2014 branch from 585aaff to cd20ca6 Compare August 19, 2022 12:13
@kou
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI
b319661 
Because our official .jar packages are built in manylinux2014 image.
@kou kou force-pushed the ci-java-jni-use-manylinux2014 branch from cd20ca6 to b319661 Compare August 19, 2022 12:34
@kou
Copy link
Member Author

kou commented Aug 19, 2022

@pitrou manylinux2014 based image isn't slower than Debian 9 based image:

pitrou
pitrou reviewed Aug 19, 2022
View reviewed changes
docker-compose.yml
<<: *ccache
ARROW_BUILD_TESTS: "OFF"
ARROW_S3: "OFF"
ARROW_SUBSTRAIT: "OFF"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reuse these environment variables in the java-manylinux build?

Copy link
Member Author

@kou kou Aug 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use ARROW_BUILD_TESTS=OFF and ARROW_SUBSTRAIT=OFF and they are already used.
(But I want to enable ARROW_BUILD_TESTS in ARROW-17081 / #13911 to run a Dataset JNI test that uses arrow_testing.)

We can't use ARROW_S3=OFF because our JNI packages enable S3 recently: ARROW-16584 / #13157

@pitrou
Copy link
Member

pitrou commented Aug 19, 2022

@lwhite1 @lidavidm What do you think about removing Debian-based CI for Java JNI?

@kou
Copy link
Member Author

kou commented Aug 19, 2022

@github-actions crossbow submit java-jars

@github-actions
Copy link

Revision: b319661

Submitted crossbow builds: ursacomputing/crossbow @ actions-f9f22f6178

Task Status
java-jars Github Actions

@lidavidm
Copy link
Member

I think it's fine to drop Debian if we have manylinux instead.

@pitrou
Copy link
Member

pitrou commented Aug 19, 2022

@kou Can you also display ccache statistics just like is done in cpp_build.sh?

pitrou
pitrou reviewed Aug 19, 2022
View reviewed changes
ci/scripts/java_jni_macos_build.sh
@@ -41,9 +41,15 @@ install_dir=${build_dir}/cpp-install
: ${ARROW_PLASMA:=ON}
: ${ARROW_PYTHON:=OFF}
: ${ARROW_S3:=ON}
: ${ARROW_USE_CCACHE:=OFF}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, but you should perhaps also pass it to CMake below?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I changed wrong file... I should have changed java_jni_manylinux_build.sh...

I've passed it to CMake arguments. I've updated java_jni_manylinux_build.sh too.

I chose ARROW_USE_CCACHE:=OFF here because cpp_build.sh uses it. I set ARROW_USE_CCACHE=ON in caller like other jobs.

@kou
Copy link
Member Author

kou commented Aug 20, 2022

@github-actions crossbow submit java-jars

@github-actions
Copy link

Revision: 3e7930d

Submitted crossbow builds: ursacomputing/crossbow @ actions-14ab753fc3

Task Status
java-jars Github Actions

@kou
Copy link
Member Author

kou commented Aug 20, 2022

@github-actions crossbow submit java-jars

@github-actions
Copy link

Revision: 64cf4c7

Submitted crossbow builds: ursacomputing/crossbow @ actions-afee36cc35

Task Status
java-jars Github Actions

@kou
Copy link
Member Author

kou commented Aug 20, 2022

ccache statistics:

https://github.com/apache/arrow/runs/7929238651?check_suite_focus=true#step:6:2308a

=== ccache statistics before build ===
cache directory                     /ccache
primary config                      /ccache/ccache.conf
secondary config (readonly)         /usr/local/etc/ccache.conf
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
cache hit rate                      0.00 %
cleanups performed                     0
files in cache                         0
cache size                           0.0 kB
max cache size                       1.0 GB

https://github.com/apache/arrow/runs/7929238651?check_suite_focus=true#step:6:3472

=== ccache statistics after build ===
cache directory                     /ccache
primary config                      /ccache/ccache.conf
secondary config (readonly)         /usr/local/etc/ccache.conf
stats updated                       Sat Aug 20 05:03:33 2022
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                            98
cache hit rate                      0.00 %
cleanups performed                     0
files in cache                       190
cache size                          12.2 MB
max cache size                       1.0 GB

Cache will be used after we merge this.

@kou
Copy link
Member Author

kou commented Aug 23, 2022

It seems that there is no objection.
I'll merge this.

@kou kou merged commit 9882df5 into apache:master Aug 23, 2022
@kou kou deleted the ci-java-jni-use-manylinux2014 branch August 23, 2022 00:06
@ursabot
Copy link

ursabot commented Aug 23, 2022

Benchmark runs are scheduled for baseline = 0600bb2 and contender = 9882df5. 9882df5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.27% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.55% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.28% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 9882df58 ec2-t3-xlarge-us-east-2
[Failed] 9882df58 test-mac-arm
[Failed] 9882df58 ursa-i9-9960x
[Finished] 9882df58 ursa-thinkcentre-m75q
[Finished] 0600bb27 ec2-t3-xlarge-us-east-2
[Failed] 0600bb27 test-mac-arm
[Failed] 0600bb27 ursa-i9-9960x
[Finished] 0600bb27 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

anjakefala pushed a commit to anjakefala/arrow that referenced this pull request Aug 31, 2022
@kou @anjakefala
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI (apache#13920)
b8056b9 
Because our official .jar packages are built in manylinux2014 image.

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
zagto pushed a commit to zagto/arrow that referenced this pull request Oct 7, 2022
@kou @zagto
ARROW-17451: [CI][Java] Use manylinux2014 image for JNI (apache#13920)
bd02afa 
Because our official .jar packages are built in manylinux2014 image.

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pitrou pitrou pitrou left review comments

Assignees
No one assigned
Labels
Component: C++
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kou @pitrou @lidavidm @ursabot