Build breaks with libszip-2.1.1: undefined reference due to --no-allow-shlib-undefined: SZ_encoder_enabled

[yurivict]

Describe the bug

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_encoder_enabled
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffCompress
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffDecompress
>>> referenced by ../bin/libhdf5.so.310.5.0
cc: error: linker command failed with exit code 1 (use -v to see invocation)
--- java/src/hdf/hdf5lib/CMakeFiles/jarhdf5.dir/all ---
--- java/src/hdf/hdf5lib/jarhdf5-1.14.5.jar ---
--- test/CMakeFiles/testhdf5.dir/all ---
[ 73%] Building C object test/CMakeFiles/testhdf5.dir/testhdf5.c.o
cd /usr/ports/science/hdf5/work/.build/test && /usr/local/libexec/ccache/cc -DH5_BUILT_AS_DYNAMIC_LIB -DH5_HAVE_TEST_API -I/usr/ports/science/hdf5/work/hdf5-1.14.5/src -I/usr/ports/science/hdf5/work/hdf5-1.14.5/src/H5FDsubfiling -I/usr/ports/science/hdf5/work/.build/src -I/usr/ports/science/hdf5/work/.build/test -I/usr/ports/science/hdf5/work/hdf5-1.14.5/test -isystem /usr/local/include/json-c -isystem /usr/local/include/gcc13 -std=c99 -O2 -pipe  -D__LONG_LONG_SUPPORTED -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -O2 -pipe  -D__LONG_LONG_SUPPORTED -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -DNDEBUG -std=gnu99 -fPIE -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -MD -MT test/CMakeFiles/testhdf5.dir/testhdf5.c.o -MF CMakeFiles/testhdf5.dir/testhdf5.c.o.d -o CMakeFiles/testhdf5.dir/testhdf5.c.o -c /usr/ports/science/hdf5/work/hdf5-1.14.5/test/testhdf5.c
--- test/CMakeFiles/chunk_info.dir/all ---
*** [bin/chunk_info] Error code 1

make[2]: stopped in /usr/ports/science/hdf5/work/.build
1 error

make[2]: stopped in /usr/ports/science/hdf5/work/.build
*** [test/CMakeFiles/chunk_info.dir/all] Error code 2

make[1]: stopped in /usr/ports/science/hdf5/work/.build
--- test/CMakeFiles/testhdf5.dir/all ---
*** [test/CMakeFiles/testhdf5.dir/all] Error code 6

make[1]: stopped in /usr/ports/science/hdf5/work/.build
--- java/src/hdf/hdf5lib/CMakeFiles/jarhdf5.dir/all ---
[ 73%] Creating Java archive jarhdf5-1.14.5.jar
cd /usr/ports/science/hdf5/work/.build/java/src/hdf/hdf5lib/CMakeFiles/jarhdf5.dir && /usr/local/bin/jar -cfm /usr/ports/science/hdf5/work/.build/java/src/hdf/hdf5lib/jarhdf5-1.14.5.jar /usr/ports/science/hdf5/work/.build/java/src/hdf/hdf5lib/Manifest.txt @java_class_filelist
--- test/CMakeFiles/direct_chunk.dir/all ---
ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_encoder_enabled
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffCompress
>>> referenced by ../bin/libhdf5.so.310.5.0

ld: error: undefined reference due to --no-allow-shlib-undefined: SZ_BufftoBuffDecompress
>>> referenced by ../bin/libhdf5.so.310.5.0
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [bin/direct_chunk] Error code 1

make[2]: stopped in /usr/ports/science/hdf5/work/.build
1 error

make[2]: stopped in /usr/ports/science/hdf5/work/.build
*** [test/CMakeFiles/direct_chunk.dir/all] Error code 2

Expected behavior
n/a

Platform (please complete the following information)

• HDF5 version: 1.14.5
• OS and version: FreeBSD 14.1
• Compiler and version: clang-18
• Build system (e.g. CMake, Autotools) and version: cmake-3.30.4
• Any configure options you specified: n/a

Additional context
libszip is discovered by cmake, but then this error occurs.

[hyoklee]

@yurivict , thank you so much for supporting HDF5 on FreeBSD!

It seems that libszip is a new port package.

Port Added: 2024-10-10 21:24:07
Last Update: 2024-10-10 21:20:23

Are you adding the new port because
HDF5 CMake does not find libaec package automatically as an SZIP alternative?

How can I install the new libszip port on FreeBSD so that I can reproduce error?
I tried sudo·pkg·install·-y·libszip but I got an error:
pkg: No packages available to install matching 'libszip' have been found in the repositories

Regards,

[yurivict]

Are you adding the new port because
HDF5 CMake does not find libaec package automatically as an SZIP alternative?

Yes.

I wasn't sure what libaec is, and added libszip.
But libaec wasn't found by cmake silently.

How can I install the new libszip port on FreeBSD so that I can reproduce error?
I tried sudo·pkg·install·-y·libszip but I got an error:
pkg: No packages available to install matching 'libszip' have been found in the repositories

This is a very new port, and its packages aren't yet built.

In order to install it you need to run this command on the updated ports tree:
cd /usr/ports/archivers/libszip && make install clean

[hyoklee]

Than you, @yurivict , for answering my questions!

I installed the new port and could duplicate your error:

https://my.cdash.org/viewBuildError.php?buildid=2688292

Yes, I also noticed that libaec can't be found silently by cmake.

[hyoklee]

Here are a few hacks for FreeBSD community.

For libszip package, run

$export LDFLAGS="-L/usr/local/lib -lszip"

For libaec package, run

$export LDFLAGS="-L/usr/local/lib -lsz"
$sudo ln -s /usr/local/cmake/libaec-config-version.cmake /usr/local/cmake/szip-config-version.cmake
$sudo ln -s /usr/local/cmake/libaec-config.cmake /usr/local/cmake/szip-config.cmake

Run the above before you run cmake.

[cho-m]

Yes, I also noticed that libaec can't be found silently by cmake.

This appears to be caused by #4567 (included in 1.14.5 via 95fa8f1) which dropped support for libaec. Also saw failure in Homebrew Homebrew/homebrew-core#193283 (for us HDF5_ENABLE_SZIP_SUPPORT gets disabled due to not being detected).

---

EDIT: Noting that restoring find_package(libaec CONFIG) fixed detection, but build failed afterward at

CMake Error at src/CMakeLists.txt:1204 (get_target_property):
  get_target_property() called with non-existent target
  "/opt/homebrew/lib/libsz.dylib".

EDIT 2: Trying some workarounds but pkg-config file ends up broken:

Libs.private:   -lzlib-static -llibname-NOTFOUND

[yurivict]

Maybe the best solution is if hdf5 would either auto-detect libaec and libszip, or to detect each of them based on special cmake variables instructing cmake to look for one or the other library?

[hyoklee]

@cho-m , thank you for identifying the root cause related to libaec.

Your comment helped me to find another hack for FreeBSD community:

cmake -DLIBAEC_PACKAGE_NAME:STRING=libaec ..

This can eliminate:

$sudo ln -s /usr/local/cmake/libaec-config-version.cmake /usr/local/cmake/szip-config-version.cmake
$sudo ln -s /usr/local/cmake/libaec-config.cmake /usr/local/cmake/szip-config.cmake

However, users will get a warning:

CMake Warning (dev) at /usr/local/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:441 (message):
  The package name passed to `find_package_handle_standard_args` (libaec)
  does not match the name of the calling package (SZIP).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.

Yet, build and test succeeds:

https://my.cdash.org/viewTest.php?onlypassed&buildid=2689761

[cho-m]

Personally would recommend re-adding find_package (libaec CONFIG).

I'm guessing this change was an untested breakage as CI doesn't run this combination. Especially so given CI runs on Ubuntu which doesn't have CMake files https://packages.ubuntu.com/jammy/amd64/libaec-dev/filelist

hdf5/.github/workflows/main-auto.yml

Line 42 in 97420ea

sudo apt install libaec0 libaec-dev

hdf5/.github/workflows/main-cmake-par.yml

Line 34 in 97420ea

sudo apt install libaec0 libaec-dev

---

Build issue (failing to find libsz) should be reproducible with other Linux distros as most will not provide SZip (due to license, e.g. https://fedoraproject.org/wiki/Forbidden_items#szip) and libaec is installed in standard manner with libaec-config.cmake:

• Fedora - https://packages.fedoraproject.org/pkgs/libaec/libaec-devel/fedora-rawhide.html#files
• Arch Linux - https://archlinux.org/packages/extra/x86_64/libaec/
• Alpine - https://pkgs.alpinelinux.org/contents?name=libaec-dev&repo=community&branch=edge&arch=x86_64

Only thing for Linux distros is that they use Autotools to build HDF5 so won't see problem in CMake.

[yurivict]

Is my understanding correct that libaec provides a superset of functionality compared to libszip?

[cho-m]

Is my understanding correct that libaec provides a superset of functionality compared to libszip?

The most commonly used API is drop-in compatible but libszip does expose a few more functions as part of its public API (various SZ_Decompress* and SZ_Compress*) so might not be considered a proper superset. HDF5 uses the common ones so either library should work.

[yurivict]

Any hope to fix this and to make a release with the fix?
We have the SZIP option broken in the FreeBSD port because of this problem.

[cho-m]

In Homebrew, I've been trying out a patch. Opened PR #5010 which works for me with libaec.

Note that libaec uses a Windows-only directory layout for installing CMake config files (<prefix>/cmake)¹ thus CMake requires help finding them, e.g. via libaec_DIR. Would be better if they allowed installing into Unix compatible directory too. Can easily be moved, which is common approach used in Linux distros.

Footnotes

1. https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-search-procedure ↩

[hyoklee]

@cho-m , thank you for your hard effort fixing HDF5 CMake for szip.

You may find these also helpful for making your patch portable & usable for Windows vcpkg users:

1. https://github.com/microsoft/vcpkg/blob/master/ports/hdf5/szip.patch
2. [hdf5] Disable szip by default microsoft/vcpkg#38073

I don't know when libaec will be official for Microsoft vcpkg.
In the mean time, I think it's good to continue supporting szip.

[yurivict]

The package doesn't have to have .cmake or .pc files to be discoverable by cmake.

hdf5 can use cmake's find_library to discover libaec.

[cho-m]

The package doesn't have to have .cmake or .pc files to be discoverable by cmake.

libaec should have CMake files as part of default build, but in a poor location. Can fixup like e.g.

• Arch moves them https://gitlab.archlinux.org/archlinux/packaging/packages/libaec/-/blob/main/PKGBUILD?ref_type=heads#L25
• I added symlinks for them in Homebrew, https://github.com/Homebrew/homebrew-core/blob/master/Formula/lib/libaec.rb#L37-L44

EDIT: I checked FreeBSD and they are available: https://github.com/freebsd/freebsd-ports/blob/main/science/libaec/pkg-plist

cmake/libaec-config-version.cmake
cmake/libaec-config.cmake

[yurivict]

Yes, libaec's cmake scripts work on FreeBSD.
find_package(libaec REQUIRED) should just be added in hdf5.

[cho-m]

find_package(libaec REQUIRED) should just be added in hdf5.

Problem looks like Windows support. I do think my changes are backwards compatible with whatever was previously running as I retained all fallbacks.

However, the previously running logic may already be broken on 1.14.5 for vcpkg. Don't have a setup readily available to test this.

[cho-m]

You may find these also helpful for making your patch portable & usable for Windows vcpkg users:

1. https://github.com/microsoft/vcpkg/blob/master/ports/hdf5/szip.patch

Looks like this patch is related to root cause of regression. Seems like only Windows was tested and the patched-behavior was assumed to be the default.

Also noted this in #5010 (comment).