GH-40964: [CI][Archery] Archery linking should also check for undefined symbols Linux

[vibhatha]

Rationale for this change

This PR includes a check to find out if there are undefined symbols associated with the libraries.

What changes are included in this PR?

Includes a util functions which uses nm and ldconfig tools to find out libraries and their symbols and determine
if there are undefined symbols excluding the symbols in the allowed dependencies.

Are these changes tested?

Locally tested with a faulty shared library, but this needs validation on versions as there is a version mismatch in the local libraries.

Are there any user-facing changes?

No

• GitHub Issue: [CI][Archery] Archery linking should also check for undefined symbols Linux #40964
• GitHub Issue: [CI][Archery] Archery linking should also check for undefined symbols Linux #40964

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 090275e

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ff12b2218

Task	Status
java-jars

[github-actions]

⚠️ GitHub issue #40018 has been automatically assigned in GitHub to PR creator.

[vibhatha]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 090275e

Submitted crossbow builds: ursacomputing/crossbow @ actions-e8a43da78d

Task	Status
java-jars

[vibhatha]

seems like nm version issue: https://github.com/ursacomputing/crossbow/actions/runs/8274012442/job/22638708916#step:8:4124
I am looking into this.

[vibhatha]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 803932a

Submitted crossbow builds: ursacomputing/crossbow @ actions-cf41726ea3

Task	Status
java-jars

[vibhatha]

@github-actions crossbow submit java-jars

[github-actions]

Revision: b9fb4f2

Submitted crossbow builds: ursacomputing/crossbow @ actions-2164940a0a

Task	Status
java-jars

[vibhatha]

@github-actions crossbow submit java-jars

[vibhatha]

@github-actions crossbow submit java-jars

[vibhatha]

Locally I have the following nm

nm --version
GNU nm (GNU Binutils) 2.40
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

but in the CI we have

nm: just-symbols: invalid output format
GNU nm version 2.35-5.el7.4
Copyright (C) 2020 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

Would it be possible to upgrade the nm version in the CI containers?

cc @raulcd

[kou]

If we run archery linking check-dependencies in package-jars job not build-cpp-ubuntu, we can use newer nm.
But we can't change nm version for java-jni-manylinux-2014. We need to use CentOS 7 based image for java-jni-manylinux-2014.
If we use manylinux_2_28 not manylinux-2014, we can use newer nm. (Can we drop support for CentOS 7?)

[vibhatha]

I see, it seems a bit complicated approach to update nm. Then should we target on more work on Python rather than nm update?

[kou]

Yes. Could you try the approach?

[vibhatha]

@kou I will try this.

[vibhatha]

@github-actions crossbow submit java-jars

[vibhatha]

@github-actions crossbow submit -g java

[vibhatha]

@kou, I just checked with the https://repo1.maven.org/maven2/org/apache/arrow/arrow-dataset/15.0.2/arrow-dataset-15.0.2.jar

Error: Undefined symbols found in /home/vibhatha/Documents/Work/Apache_Arrow/issues/gh-40018/arrow-dataset-15.0.2/x86_64/libarrow_dataset_jni.so:
EVP_MD_CTX_create
EVP_MD_CTX_destroy
HMAC_CTX_cleanup
HMAC_CTX_init

This PR fixes the protobuf issue, but it seems like we still do have openssl related stuff here, which is not in the allowed list right?

[vibhatha]

@pitrou I updated the PR, but I have one question and noted here

[vibhatha]

@pitrou appreciate your feedback for this PR.