Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDF5 1.14.4.2/3 is breaking GDAL build #4527

Closed
imincik opened this issue May 28, 2024 · 13 comments · Fixed by OSGeo/gdal#10052
Closed

HDF5 1.14.4.2/3 is breaking GDAL build #4527

imincik opened this issue May 28, 2024 · 13 comments · Fixed by OSGeo/gdal#10052
Assignees
@jhendersonHDF
Labels
Component - Misc Anything else (CODEOWNERS, etc.) Priority - 1. High 🔼 These are important issues that should be resolved in the next release Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub
Milestone
1.14.5

Comments

@imincik
Copy link

imincik commented May 28, 2024

Describe the bug

GDAL build has started to fail since HDF5 version 1.14.4.2 and it keeps failing with 1.14.4.3 as well.

See: NixOS/nixpkgs#309595

Expected behavior
I want to see GDAL building successfully.

Platform (please complete the following information)

  • HDF5 version (if building from a maintenance branch, please include the commit hash): 1.14.4.2 and 1.14.4.3
  • OS and version: Nixpkgs master
  • Compiler and version
  • Build system (e.g. CMake, Autotools) and version
  • Any configure options you specified
  • MPI library and version (parallel HDF5)

Additional context
GDAL build log:

gdrivers/gxf.py ..ssssss                                                 [ 61%]
gdrivers/hdf5.py ..........ssFE..Fatal Python error: Floating point exception

Current thread 0x00007ffff7696740 (most recent call first):
  File "/nix/store/5jd0c9xi99hviq92h3xvm0nmzlvmzam7-gdal-3.8.5/lib/python3.11/site-packages/osgeo/gdal.py", line 5538 in Checksum
  File "/build/source/build/autotest/gdrivers/hdf5.py", line 454 in test_hdf5_17
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/python.py", line 1772 in runtest
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/runner.py", line 172 in pytest_runtest_call
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/runner.py", line 240 in <lambda>
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/runner.py", line 340 in from_call
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/runner.py", line 239 in call_and_report
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/runner.py", line 134 in runtestprotocol
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/runner.py", line 115 in pytest_runtest_protocol
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/main.py", line 339 in _main
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/config/__init__.py", line 174 in main
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/config/__init__.py", line 197 in console_main
  File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/pytest/__main__.py", line 7 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: osgeo._gdal, osgeo._gdalconst, osgeo._ogr, osgeo._osr, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, osgeo._gdal_array, osgeo._gnm, lxml._elementpath, lxml.etree, _gdal_python_driver (total: 22)
/nix/store/0hh52fb06c14cnzyhxk0nlhr9ws4d8lx-pytest-check-hook/nix-support/setup-hook: line 53: 15425 Floating point exception(core dumped) /nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/bin/python3.11 -m pytest -k "not test_jp2openjpeg_45 and not test_transformer_dem_overrride_srs and not test_osr_ct_options_area_of_interest and not test_sentinel2_zipped and not test_SetPROJAuxDbPaths and not test_visoss_6 and not test_osr_esri_28" --ignore="gcore/vsis3.py" --ignore="gdrivers/gdalhttp.py" --ignore="gdrivers/wms.py" --benchmark-disable
/nix/store/wq046wv3s3h4m2n9ak3j534di2pzra93-stdenv-linux/setup: line 1579: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/xdinkla57lwm3x60dk6hvqn3rmzh2iih-gdal-3.8.5.drv' failed with exit code 136;
       last 10 log lines:
       >   File "/nix/store/725kzy1ql7a7vkngmnr3r0s87ag820p8-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py", line 501 in __call__
       >   File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/config/__init__.py", line 174 in main
       >   File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/_pytest/config/__init__.py", line 197 in console_main
       >   File "/nix/store/4w47qwmh7j66pf7qls7ri088zz9hv3pk-python3.11-pytest-8.1.1/lib/python3.11/site-packages/pytest/__main__.py", line 7 in <module>
       >   File "<frozen runpy>", line 88 in _run_code
       >   File "<frozen runpy>", line 198 in _run_module_as_main
       >
       > Extension modules: osgeo._gdal, osgeo._gdalconst, osgeo._ogr, osgeo._osr, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, osgeo._gdal_array, osgeo._gnm, lxml._elementpath, lxml.etree, _gdal_python_driver (total: 22)
       > /nix/store/0hh52fb06c14cnzyhxk0nlhr9ws4d8lx-pytest-check-hook/nix-support/setup-hook: line 53: 15425 Floating point exception(core dumped) /nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/bin/python3.11 -m pytest -k "not test_jp2openjpeg_45 and not test_transformer_dem_overrride_srs and not test_osr_ct_options_area_of_interest and not test_sentinel2_zipped and not test_SetPROJAuxDbPaths and not test_visoss_6 and not test_osr_esri_28" --ignore="gcore/vsis3.py" --ignore="gdrivers/gdalhttp.py" --ignore="gdrivers/wms.py" --benchmark-disable
       > /nix/store/wq046wv3s3h4m2n9ak3j534di2pzra93-stdenv-linux/setup: line 1579: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/xdinkla57lwm3x60dk6hvqn3rmzh2iih-gdal-3.8.5.drv'.
@imincik imincik mentioned this issue May 28, 2024
Merged
13 tasks
@hyoklee
Copy link
Member

hyoklee commented May 28, 2024

Hi, @imincik !

Thank you for testing the latest HDF5 with GDAL.
Does it also fail with GDAL 3.9.0 which was released 3 weeks ago by @rouault?
I would recommend you to report this to gdal-dev mailing list as well.

You can also try

hdf5/release_docs/RELEASE.txt

Line 2136 in 162e7d6

CMake: HDF5_ENABLE_NONSTANDARD_FEATURE_FLOAT16=OFF

@jhendersonHDF jhendersonHDF self-assigned this May 28, 2024
@imincik
Copy link
Author

imincik commented May 28, 2024

Does it also fail with GDAL 3.9.0 which was released 3 weeks ago by @rouault?
I would recommend you to report this to gdal-dev mailing list as well.

Yes, unfortunately it fails with GDAL 3.9.0 as well.

@imincik
Copy link
Author

imincik commented May 28, 2024

I confirm that both GDAL 3.8.5 and GDAL 3.9.0 are building successfully when HDF5 1.14.4.3 is built with HDF5_ENABLE_NONSTANDARD_FEATURE_FLOAT16=OFF . Thank you very much for suggestion.

@jhendersonHDF
Copy link
Collaborator

I confirm that both GDAL 3.8.5 and GDAL 3.9.0 are building successfully when HDF5 1.14.4.3 is built with HDF5_ENABLE_NONSTANDARD_FEATURE_FLOAT16=OFF . Thank you very much for suggestion.

Hi @imincik,

it would be interesting to see what it is about the _Float16 type that's causing issues with GDAL. Does GDAL happen to use any fortran code? We've seen issues with HDF5 Fortran when float16 support is enabled.

@imincik
Copy link
Author

imincik commented May 28, 2024

GDAL code search for float16 shows some results

@rouault
Copy link

rouault commented May 28, 2024

Does GDAL happen to use any fortran code?

I'm on it. The availability of the H5T_NATIVE_FLOAT16 breaks assumption in GDAL when it reads dataset with those types, since previously such types were seen as Float32.

rouault added a commit to rouault/gdal that referenced this issue May 28, 2024
@rouault
HDF5: add support for libhdf5 >= 1.14.4.2 when built with Float16
b95f9df 
Fixes HDFGroup/hdf5#4527
@rouault rouault mentioned this issue May 28, 2024
Merged
@rouault
Copy link

rouault commented May 28, 2024

GDAL fix queued in OSGeo/gdal#10052

By the way, I struggled a lot to manage building HDF5 1.14.4.3 with Float16 support, using CMake, with Alpine edge (using "alpine:edge" Docker image). I had to patch ConfigureChecks.cmake to force set(${HDF_PREFIX}_SIZEOF__FLOAT16 2) and set (${HDF_PREFIX}_HAVE__FLOAT16 1) since auto-detection failed.

and I had to paste the following (extracted from gcc' float.h) on top of H5Tconv.c:

#ifdef __FLT16_MANT_DIG__
#undef FLT16_MANT_DIG
#define FLT16_MANT_DIG		__FLT16_MANT_DIG__
#undef FLT16_DIG
#define FLT16_DIG		__FLT16_DIG__
#undef FLT16_MIN_EXP
#define FLT16_MIN_EXP		__FLT16_MIN_EXP__
#undef FLT16_MIN_10_EXP
#define FLT16_MIN_10_EXP	__FLT16_MIN_10_EXP__
#undef FLT16_MAX_EXP
#define FLT16_MAX_EXP		__FLT16_MAX_EXP__
#undef FLT16_MAX_10_EXP
#define FLT16_MAX_10_EXP	__FLT16_MAX_10_EXP__
#undef FLT16_MAX
#define FLT16_MAX		__FLT16_MAX__
#undef FLT16_EPSILON
#define FLT16_EPSILON		__FLT16_EPSILON__
#undef FLT16_MIN
#define FLT16_MIN		__FLT16_MIN__
#undef FLT16_DECIMAL_DIG
#define FLT16_DECIMAL_DIG	__FLT16_DECIMAL_DIG__
#undef FLT16_TRUE_MIN
#define FLT16_TRUE_MIN		__FLT16_DENORM_MIN__
#if defined __STDC_VERSION__ && __STDC_VERSION__ > 201710L
#undef FLT16_SNAN
#define FLT16_SNAN	(__builtin_nansf16 (""))
#endif /* C2X */
#endif /* __FLT16_MANT_DIG__.  */

rouault added a commit to rouault/gdal that referenced this issue May 29, 2024
@rouault
HDF5: add support for libhdf5 >= 1.14.4.2 when built with Float16
16ade82 
Fixes HDFGroup/hdf5#4527
@jhendersonHDF
Copy link
Collaborator

GDAL fix queued in OSGeo/gdal#10052

By the way, I struggled a lot to manage building HDF5 1.14.4.3 with Float16 support, using CMake, with Alpine edge (using "alpine:edge" Docker image). I had to patch ConfigureChecks.cmake to force set(${HDF_PREFIX}_SIZEOF__FLOAT16 2) and set (${HDF_PREFIX}_HAVE__FLOAT16 1) since auto-detection failed.

and I had to paste the following (extracted from gcc' float.h) on top of H5Tconv.c:

#ifdef __FLT16_MANT_DIG__
#undef FLT16_MANT_DIG
#define FLT16_MANT_DIG		__FLT16_MANT_DIG__
#undef FLT16_DIG
#define FLT16_DIG		__FLT16_DIG__
#undef FLT16_MIN_EXP
#define FLT16_MIN_EXP		__FLT16_MIN_EXP__
#undef FLT16_MIN_10_EXP
#define FLT16_MIN_10_EXP	__FLT16_MIN_10_EXP__
#undef FLT16_MAX_EXP
#define FLT16_MAX_EXP		__FLT16_MAX_EXP__
#undef FLT16_MAX_10_EXP
#define FLT16_MAX_10_EXP	__FLT16_MAX_10_EXP__
#undef FLT16_MAX
#define FLT16_MAX		__FLT16_MAX__
#undef FLT16_EPSILON
#define FLT16_EPSILON		__FLT16_EPSILON__
#undef FLT16_MIN
#define FLT16_MIN		__FLT16_MIN__
#undef FLT16_DECIMAL_DIG
#define FLT16_DECIMAL_DIG	__FLT16_DECIMAL_DIG__
#undef FLT16_TRUE_MIN
#define FLT16_TRUE_MIN		__FLT16_DENORM_MIN__
#if defined __STDC_VERSION__ && __STDC_VERSION__ > 201710L
#undef FLT16_SNAN
#define FLT16_SNAN	(__builtin_nansf16 (""))
#endif /* C2X */
#endif /* __FLT16_MANT_DIG__.  */

Thanks for looking into this @rouault! If you happen to get a chance, could you post the output from the CMake configuration process and also possibly the <build_dir>/CMakeFiles/CMakeConfigureLog.yaml and <build_dir>/CMakeFiles/CMakeError.log files? It would be good to see why CMake isn't picking up the types
on Alpine to see if we're doing something wrong and can make that work without any patching needed.

@rouault
Copy link

rouault commented May 29, 2024

It would be good to see why CMake isn't picking up the types on Alpine to see if we're doing something wrong and can make that work without any patching needed.

here's the logs logs.zip

and here's the recipe to reproduce:

docker run --rm -it -v /path/to/hdf5_source:/build alpine:edge

cd /build
apk add g++ cmake make zlib-dev libsz openssl
mkdir build
cd build
cmake .. -DHDF5_ENABLE_NONSTANDARD_FEATURES=ON

@jhendersonHDF
Copy link
Collaborator

It would be good to see why CMake isn't picking up the types on Alpine to see if we're doing something wrong and can make that work without any patching needed.

here's the logs logs.zip

and here's the recipe to reproduce:

docker run --rm -it -v /path/to/hdf5_source:/build alpine:edge

cd /build
apk add g++ cmake make zlib-dev libsz openssl
mkdir build
cd build
cmake .. -DHDF5_ENABLE_NONSTANDARD_FEATURES=ON

Thanks! It looks like CMake didn't even try to run the configuration checks for the type since there aren't any references to "FLOAT16" in the log files. But when I get a chance I'll try to reproduce with your docker commands and see what the issue is.

@derobins derobins added this to the 1.14.5 milestone Jun 2, 2024
@derobins derobins added Priority - 1. High 🔼 These are important issues that should be resolved in the next release Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub Component - Misc Anything else (CODEOWNERS, etc.) labels Jun 2, 2024
rouault added a commit to OSGeo/gdal that referenced this issue Jun 3, 2024
@rouault @github-actions
HDF5: add support for libhdf5 >= 1.14.4.2 when built with Float16
e2b4f25 
Fixes HDFGroup/hdf5#4527
@derobins
Copy link
Member

derobins commented Jun 6, 2024

Can this issue be closed or are there still things we need to look into on our side?

@imincik
Copy link
Author

imincik commented Jun 7, 2024

Can this issue be closed or are there still things we need to look into on our side?

I believe that OSGeo/gdal@e2b4f25 has fixed the problem . Thank you very much for support. This issue can be closed now.

@imincik imincik closed this as completed Jun 7, 2024
@Manamama
Copy link

Manamama commented Nov 3, 2024 •
edited
Loading

FYI, also in prooted Debian and standard Termux I had to use this:
cmake -DHDF5_ENABLE_NONSTANDARD_FEATURE_FLOAT16=OFF ..

My box:

Environment at local 🎋 prooted system: Linux localhost 6.2.1-PRoot-Distro #1 SMP PREEMPT Thu Mar 17 16:28:22 CST 2022 aarch64 GNU/Linux PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/system/bin:/system/xbin:/data/data/com.termux/files/home/.local/bin/:/data/data/com.termux/files/usr/bin/ LD_LIBRARY_PATH: CFLAGS: LDFLAGS: CPPFLAGS: C_INCLUDE_PATH: :/data/data/com.termux/files/usr/include/flang/Evaluate/:/data/data/com.termux/files/usr/include/:/data/data/com.termux/files/usr/opt/libllvm-16/include/llvm/Frontend/OpenMP CPLUS_INCLUDE_PATH: USE_VULKAN: OFF

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jhendersonHDF jhendersonHDF

Labels
Component - Misc Anything else (CODEOWNERS, etc.) Priority - 1. High 🔼 These are important issues that should be resolved in the next release Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub
Projects
None yet
Milestone
1.14.5
Development

Successfully merging a pull request may close this issue.

HDF5: add support for libhdf5 >= 1.14.4.2 when built with Float16 rouault/gdal
6 participants
@derobins @imincik @rouault @hyoklee @jhendersonHDF @Manamama