Enhance generic `Bundle` easyblock to transfer module requirements of components #3472

Thyre · 2024-10-07T16:11:00Z

This PR implements the make_module_req_guess method for the generic Bundle EasyBlock. With this, all the requirements of the components in a bundle are transferred correctly to the final module. Previously, this could lead to missing environment variables, letting the build succeed but still resulting in a broken module, for example because PATH is not set.

Fixes #2733

Thyre · 2024-10-07T16:15:51Z

I've only tested the changes with easybuilders/easybuild-easyconfigs#21582 so far. I'll try to gather the EasyConfigs using Bundle in some way to make sure that I did not break them, since this is a change on a widely used EasyBlock.

Unfortunately, I can only test GCC 12.3.0 and newer easily and are somewhat limited with my compute resources. So this might take a while..

Thyre · 2024-10-07T16:17:01Z

Example with the changed EasyBlock:

easybuilders/easybuild-easyconfigs#21582 (comment)

Thyre · 2024-10-07T16:56:49Z

Test report by @Thyre

Overview of tested easyconfigs (in order)

SUCCESS FlexiBLAS-3.4.4-GCC-13.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/a33ae670ab1932bdcfff62bae82c3f53 for a full test report.

Thyre · 2024-10-07T18:44:45Z

Test report by @Thyre

Overview of tested easyconfigs (in order)

SUCCESS GCC-13.3.0.eb
SUCCESS Autotools-20231222-GCCcore-13.3.0.eb
SUCCESS cffi-1.16.0-GCCcore-13.3.0.eb
SUCCESS Julia-1.9.2-linux-x86_64.eb
SUCCESS groff-1.23.0-GCCcore-13.3.0.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS expat-2.6.2-GCCcore-13.3.0.eb
SUCCESS libiconv-1.17-GCCcore-13.3.0.eb
SUCCESS patchelf-0.18.0-GCCcore-13.3.0.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS gettext-0.22.5-GCCcore-13.3.0.eb
SUCCESS Rust-1.78.0-GCCcore-13.3.0.eb
SUCCESS intltool-0.51.0-GCCcore-13.3.0.eb
SUCCESS Doxygen-1.11.0-GCCcore-13.3.0.eb
SUCCESS hatchling-1.24.2-GCCcore-13.3.0.eb
SUCCESS setuptools-rust-1.9.0-GCCcore-13.3.0.eb
SUCCESS maturin-1.6.0-GCCcore-13.3.0.eb
SUCCESS typing-extensions-4.11.0-GCCcore-13.3.0.eb
FAIL (build issue) pydantic-2.9.1-GCCcore-13.3.0.eb (partial log available at https://gist.github.com/Thyre/abd4413683d3801209f2daa0a6d77ed8)
SUCCESS gperf-3.1-GCCcore-13.3.0.eb
SUCCESS util-linux-2.40-GCCcore-13.3.0.eb
SUCCESS libpng-1.6.43-GCCcore-13.3.0.eb
SUCCESS Brotli-1.1.0-GCCcore-13.3.0.eb
SUCCESS freetype-2.13.2-GCCcore-13.3.0.eb
SUCCESS fontconfig-2.15.0-GCCcore-13.3.0.eb
SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 25 out of 26 (7 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/3b8848b109d905611283a882db8bb378 for a full test report.

The failed one doesn't look related:

Building a mixed python/rust project
       Found pyo3 bindings
       maturin failed
        Caused by: Python interpreter should be a kind of interpreter (e.g. 'python3.8' or 'pypy3.9') when cross-compiling, got path to interpreter: /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/eb-1tc417di/pip-modern-metadata-s3zbqxe6', '--interpreter', '/tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/eb-1tc417di/pip-modern-metadata-s3zbqxe6 --interpreter /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.12.3-GCCcore-13.3.0/bin/python`
      [end of output]

The tested ones include normal Bundle (e.g. GCC & Autotools), PerlBundle (Perl-bundle-CPAN), PythonBundle (cffi) and JuliaBundle (Circuitscape). CargoPythonBundle failed due to the failure shown above.

Thyre · 2024-10-11T06:02:12Z

Changed PR so that the EasyBlock for each component is stored and not initialized in multiple steps.

Thyre · 2024-10-11T06:35:46Z

Based on https://docs.easybuild.io/version-specific/easyblocks/?h=bundle, this change should affect the following EasyBlocks:

Bundle
BuildEnv
CrayToolchain
EB_Clang_minus_AOMP
EB_OpenSSL_wrapper
JuliaBundle
PerlBundle
PythonBundle
CargoPythonBundle
SystemCompiler
SystemMPI
Toolchain

I will not be able to test all of them (e.g. CrayToolchain), but I'll try to get one EasyConfig for all of these EasyBlocks to make sure that nothing broke and compare their modules.

Thyre · 2024-10-25T09:14:16Z

I've chosen a few EasyConfigs to compare with this MR and the current develop.

Bundle -- X11-20240607-GCCcore-13.3.0.eb
BuildEnv -- buildenv-default-foss-2023b.eb
CrayToolchain -- Not possible to test for me (no Cray system)
EB_Clang_minus_AOMP -- Clang-AOMP-4.5.0-GCCcore-11.2.0.eb
EB_OpenSSL_wrapper -- OpenSSL-3.eb
JuliaBundle -- Circuitscape-5.12.3-Julia-1.9.2.eb
PerlBundle -- Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
PythonBundle -- crypt4gh-1.7-GCC-12.3.0.eb
CargoPythonBundle -- jiter-0.4.1-GCCcore-12.3.0.eb
SystemCompiler -- GCC-system.eb
SystemMPI -- OpenMPI-system-GCC-system-2.29.eb
Toolchain -- gfbf-2024a.eb

Building will take quite some time. I'll post the resulting module diffs later.

Thyre · 2024-10-26T16:34:20Z

Now that all modules are finally installed (but I had to partially select other modules due to general installation issues), I found one bug with the proposed changes:

When checking crypt4gh-1.7-GCC-12.3.0.eb, the module does a sanity check for an executable in /bin. This path is correctly added in develop:

help([==[

Description
===========
crypt4gh is a Python tool to encrypt, decrypt or re-encrypt files, 
according to the GA4GH encryption file format.


More information
================
 - Homepage: https://github.com/EGA-archive/crypt4gh


Included extensions
===================
crypt4gh-1.7
]==])

whatis([==[Description: crypt4gh is a Python tool to encrypt, decrypt or re-encrypt files, 
according to the GA4GH encryption file format.]==])
whatis([==[Homepage: https://github.com/EGA-archive/crypt4gh]==])
whatis([==[URL: https://github.com/EGA-archive/crypt4gh]==])
whatis([==[Extensions: crypt4gh-1.7]==])

local root = "/tank/Programs/Linux/EasyBuild/datenlager/Software/software/crypt4gh/1.7-GCC-12.3.0"

conflict("crypt4gh")

if not ( isloaded("GCC/12.3.0") ) then
    load("GCC/12.3.0")
end

if not ( isloaded("Python/3.11.3-GCCcore-12.3.0") ) then
    load("Python/3.11.3-GCCcore-12.3.0")
end

if not ( isloaded("Python-bundle-PyPI/2023.06-GCCcore-12.3.0") ) then
    load("Python-bundle-PyPI/2023.06-GCCcore-12.3.0")
end

if not ( isloaded("PyYAML/6.0-GCCcore-12.3.0") ) then
    load("PyYAML/6.0-GCCcore-12.3.0")
end

if not ( isloaded("cryptography/41.0.1-GCCcore-12.3.0") ) then
    load("cryptography/41.0.1-GCCcore-12.3.0")
end

if not ( isloaded("bcrypt/4.0.1-GCCcore-12.3.0") ) then
    load("bcrypt/4.0.1-GCCcore-12.3.0")
end

prepend_path("CMAKE_PREFIX_PATH", root)
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("PATH", pathJoin(root, "bin"))
setenv("EBROOTCRYPT4GH", root)
setenv("EBVERSIONCRYPT4GH", "1.7")
setenv("EBDEVELCRYPT4GH", pathJoin(root, "easybuild/crypt4gh-1.7-GCC-12.3.0-easybuild-devel"))

prepend_path("PYTHONPATH", pathJoin(root, "lib/python3.11/site-packages"))
-- Built with EasyBuild version 4.9.5.dev0
setenv("EBEXTSLISTCRYPT4GH", "crypt4gh-1.7")

but this fails with the proposed changes. CMAKE_PREFIX_PATH, LIBRARY_PATH and PATH are missing. I suspect that adding make_module_req_guess to Bundle causes this to fail.

Thyre · 2024-10-26T18:35:38Z

Test report by @Thyre

Overview of tested easyconfigs (in order)

SUCCESS buildenv-default-foss-2023b.eb
SUCCESS OpenSSL-3.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS gfbf-2024a.eb
SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
SUCCESS GCC-system.eb
SUCCESS OpenMPI-system-GCC-system-2.29.eb
SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/75aa472a40750fd854cf2d3ec2ede6e3 for a full test report.

I'm not able to test jiter-0.4.1-GCCcore-12.3.0.eb due to the following error caused by maturin:

  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

   Preparing metadata (pyproject.toml) did not run successfully.
   exit code: 1
  > [6 lines of output]
       Found pyo3 bindings
       maturin failed
        Caused by: Python interpreter should be a kind of interpreter (e.g. 'python3.8' or 'pypy3.9') when cross-compiling, got path to interpreter: /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python
      Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/eb-ibucduig/pip-modern-metadata-debc3bzs', '--interpreter', '/tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python']' returned non-zero exit status 1.
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/eb-ibucduig/pip-modern-metadata-debc3bzs --interpreter /tank/Programs/Linux/EasyBuild/datenlager/Software/software/Python/3.11.3-GCCcore-12.3.0/bin/python`
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

 Encountered error while generating package metadata.
> See above for output.

I've also checked if the module files of the EasyConfigs above changed. They did not.

Thyre · 2024-10-26T20:50:09Z

Full (re-)installation

Test report by @Thyre

Overview of tested easyconfigs (in order)

SUCCESS buildenv-default-foss-2023b.eb
SUCCESS OpenSSL-3.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS gfbf-2024a.eb
SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
SUCCESS GCC-system.eb
SUCCESS OpenMPI-system-GCC-system-2.29.eb
SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/bbc3468bc740b0c511c9f1e665d32910 for a full test report.

Thyre · 2024-10-26T21:02:29Z

Module only

Test report by @Thyre

Overview of tested easyconfigs (in order)

SUCCESS buildenv-default-foss-2023b.eb
SUCCESS OpenSSL-3.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS gfbf-2024a.eb
SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
SUCCESS GCC-system.eb
SUCCESS OpenMPI-system-GCC-system-2.29.eb
SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/c78da935cdd54f14102e4d932295777e for a full test report.

Thyre · 2024-10-27T17:20:26Z

Testing a few of the Stages 2025 EasyConfigs using Bundle/PythonBundle/... of the JSC repo

Test report by @Thyre

Overview of tested easyconfigs (in order)

SUCCESS Autotools-20231222-GCCcore-13.3.0.eb
SUCCESS OpenSSL-3.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS boto3-1.35.36-GCCcore-13.3.0.eb
SUCCESS npsmpic-2024a.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
jrlogin05.jureca - Linux Rocky Linux 9.4, x86_64, AMD EPYC 7742 64-Core Processor, 2 x NVIDIA Quadro RTX 8000, 560.35.03, Python 3.9.18
See https://gist.github.com/Thyre/81e26fbe93c25adabffd33e5664ddc0b for a full test report.

Thyre · 2024-11-06T10:20:37Z

Bi-weekly EasyBuild conf call 2024-11-06: While these changes continue to work with EasyBuild 5.0.x, the make_module_req_guess method will be deprecated.

My current idea would be to continue with this PR as it is. Once easybuilders/easybuild-framework#4653 is merged, I'll work on porting these changes to the new method for EasyBuild 5.0.x.

Thyre · 2024-11-08T09:52:15Z

@boegelbot please test @ jsc-zen3
EB_ARGS="X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"
CORE_CNT=16

boegelbot · 2024-11-08T09:53:09Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 5224

Test results coming soon (I hope)...

- notification for comment with ID 2464270943 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-11-08T11:00:05Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS X11-20240607-GCCcore-13.3.0.eb
SUCCESS buildenv-default-foss-2023b.eb
SUCCESS OpenSSL-3.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
SUCCESS GCC-system.eb
SUCCESS gfbf-2024a.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/f0ee3fa8737148ccb5fae372b743d256 for a full test report.

Thyre · 2024-11-08T11:04:48Z

@boegelbot please test @ jsc-zen3
EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"

boegelbot · 2024-11-08T11:13:09Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 5226

Test results coming soon (I hope)...

- notification for comment with ID 2464420612 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-11-08T11:29:22Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS X11-20240607-GCCcore-13.3.0.eb
SUCCESS buildenv-default-foss-2023b.eb
SUCCESS OpenSSL-3.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
SUCCESS GCC-system.eb
SUCCESS gfbf-2024a.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/b951173c327e4bb0f8b38422bdf91456 for a full test report.

lexming

Looks good, I only have a few comments to improve the code a bit. See below

easybuild/easyblocks/generic/bundle.py

Thyre · 2024-11-08T12:44:41Z

Thanks a lot for the suggestions. I'll add them to the commit in a moment 😄

…omponents This commit implements the `make_module_req_guess` method for the generic Bundle EasyBlock. With this, all the requirements of the components in a bundle are transferred correctly to the final module. Previously, this could lead to missing environment variables, letting the build succeed but still resulting in a broken module, for example because `PATH` is not set. Signed-off-by: Jan André Reuter <[email protected]>

Instead of zipping bundles and configs every time, use a single instance list. Also, do not check for the type when building the module requirements and catch the exception instead. Signed-off-by: Jan André Reuter <[email protected]>

Thyre · 2024-11-08T14:07:49Z

@boegelbot please test @ jsc-zen3
EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb"

boegelbot · 2024-11-08T14:13:09Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3472 EB_ARGS="--module-only X11-20240607-GCCcore-13.3.0.eb buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb GCC-system.eb gfbf-2024a.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3472 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 5235

Test results coming soon (I hope)...

- notification for comment with ID 2464855921 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-11-08T14:30:35Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS X11-20240607-GCCcore-13.3.0.eb
SUCCESS buildenv-default-foss-2023b.eb
SUCCESS OpenSSL-3.eb
SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
SUCCESS GCC-system.eb
SUCCESS gfbf-2024a.eb

Build succeeded for 9 out of 9 (9 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/2df274b1d2731957ac6b0e5be8d36daf for a full test report.

lexming · 2024-11-08T14:50:54Z

@Thyre thanks for the quick update! do you want to add your name to the list of contributors to this easyblock? If you do, there is no need to send new tests again 🙂

Signed-off-by: Jan André Reuter <[email protected]>

Thyre · 2024-11-08T14:56:46Z

@Thyre thanks for the quick update! do you want to add your name to the list of contributors to this easyblock?

Done. Took longer than I want to admit to find the authors comment 😅

If you do, there is no need to send new tests again 🙂

Agreed, changing the comment shouldn't break anything. Just wanted to make sure nothing broke during the last changes.

lexming

LGTM

lexming · 2024-11-09T18:18:43Z

Merging, thanks @Thyre !

boegel · 2024-11-12T17:41:59Z

easybuild/easyblocks/generic/bundle.py

-        # list of EasyConfig instances for components
-        self.comp_cfgs = []
+        # list of EasyConfig instances and their EasyBlocks for components
+        self.comp_instances = []


@Thyre renaming this will have some fallout, see the clang_aomp.py where self.comp_cfgs is used...

In that case, we should keep the old one, yeah. With the PR reverted, I revise the changes.

Yeah... I missed that EasyBlock since it is unused. However, there is only a single usage of self.comp_cfgs.
When I open a new PR for this, I'll update the clang_aomp.py EasyBlock as well. Haven't found other usages.

boegel · 2024-11-12T17:42:53Z

@Thyre @lexming I think the changes here may lead to an infinite loop, the easyconfigs test suite is failing due to generating too many open files, and it seems to coincide with this PR getting merged...

boegel · 2024-11-12T17:43:33Z

easybuild/easyblocks/generic/bundle.py

+
+        for cfg, comp in self.comp_instances:
+            self.log.info("Gathering module paths for component %s v%s", cfg['name'], cfg['version'])
+            reqs = comp.make_module_req_guess()


What if one of the component itself is a Bundle instance, that may result in an infinite loop I think?

Hmm... a Bundle instance should gather the requirements of the components. If one of the components is also a Bundle, we should collect all of them recursively.

An infinite loop can be achieved if one EasyBlock tries to call this function via super. Can we prevent that?

@boegel I don't see why there would be an infinite loop here, the list of components is finite and equal to whatever the easyconfig defines. Even if there is a bundle in the component list, it is also defined in the easyconfig.
This is not like looping over dependencies, with multiple layers of depth.

Having a Bundle as component of another Bundle is explicitly forbidden. See

easybuild-easyblocks/easybuild/easyblocks/generic/bundle.py

Line 142 in 28b2084

raise EasyBuildError("The Bundle easyblock can not be used to install components in a bundle")

So this is not the cause of the issue (unless some unit test does some weird stuff and hijacks that check in Bundle)

Apparently there is a general bug in this check, as it does not cover easyblocks based on Bundle, like PythonBundle or OpenSSL_wrapper. So that is one thing that can improve.
However, it is already impossible to make a nested bundle in practice because the resulting construct always errors out. If the bundle component has sources, it results in error about "bundles cannot have sources"; and if it does not have sources it results in error about "missing sources".

…nfinite loop in easyconfigs test suite?

boegel · 2024-11-12T17:47:18Z

@Thyre @lexming I think the changes here may lead to an infinite loop, the easyconfigs test suite is failing due to generating too many open files, and it seems to coincide with this PR getting merged...

I've opened #3504 to revert the changes that were done here.
Purely based on timing, it seems like the changes here are the culprit for the trouble we're currently seeing in the easyconfigs test suite...

revert changes from PR #3472, seems to be leading to an infinite loop in easyconfigs test suite?

Thyre · 2024-11-12T19:04:45Z

easybuild/easyblocks/generic/bundle.py

@@ -198,7 +199,7 @@ def __init__(self, *args, **kwargs):
            if comp_cfg['patches']:
                self.cfg.update('patches', comp_cfg['patches'])

-            self.comp_cfgs.append(comp_cfg)
+            self.comp_instances.append((comp_cfg, comp_cfg.easyblock(comp_cfg)))


@boegel, @lexming This may be the issue causing infinite loop. Changing this to:

-self.comp_instances.append((comp_cfg, comp_cfg.easyblock(comp_cfg))) +self.comp_instances.append((comp_cfg, comp_cfg.easyblock))

certainly improves the situation.
I'll continue checking if the test passes locally now.

The test suite is now stuck for QuantumESPRESSO. That's a huge progress (since we were previously stuck even for the first few EasyConfigs with A), but still not a complete fix.

The QuantumESPRESSO check is not stuck. Instead, the logging files are not closed and all (?) tests after that are logged as well, causing around 11GB logs on my system. This also happens with develop as far as I can tell.

total 11G drwx------ 2 jreuter jreuter 800 12. Nov 20:44 . drwxrwxrwt 24 root root 540 12. Nov 20:36 .. -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-5.3.0-20241112.203827.pyJFM.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-5.4.0-20241112.203827.FyduX.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-5.4.0-20241112.203827.hzkuy.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.0-20241112.203827.FzleV.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.1-20241112.203827.FZgWV.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.2.1-20241112.203827.xMxaO.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.2-20241112.203827.fABEs.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.3-20241112.203827.kaPJh.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.3-20241112.203827.nZhTm.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.4.1-20241112.203827.juhTP.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.5-20241112.203827.pfDnY.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.5-20241112.203827.VBbnI.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.bRTCU.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.ewZNc.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.IzdWy.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.PgGZc.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.6-20241112.203827.vwIEO.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.KUUtU.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.piJmd.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.rQAjF.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.sbJUP.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.ZBRpm.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.7-20241112.203827.ZSOpP.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.8-20241112.203827.cKErR.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.8-20241112.203827.wwDpe.log -rw-r--r-- 1 jreuter jreuter 299M 12. Nov 20:45 easybuild-QuantumESPRESSO-6.8-20241112.203827.XATBJ.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.0-20241112.203827.KuImR.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.0-20241112.203827.SXBiL.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.1-20241112.203827.BWcVK.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.1-20241112.203827.tpQDP.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.2-20241112.203827.GrtNk.log -rw-r--r-- 1 jreuter jreuter 298M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.2-20241112.203827.MArZe.log -rw-r--r-- 1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.2-20241112.203828.qrsQj.log -rw-r--r-- 1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3.1-20241112.203828.gMpKu.log -rw-r--r-- 1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3.1-20241112.203828.kpFyU.log -rw-r--r-- 1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3.1-20241112.203828.lrPYN.log -rw-r--r-- 1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3-20241112.203828.TOcVA.log -rw-r--r-- 1 jreuter jreuter 283M 12. Nov 20:45 easybuild-QuantumESPRESSO-7.3-20241112.203828.wnnQg.log

Could this be related to the way the EasyBlock is written?

lexming · 2024-11-12T22:38:55Z

@Thyre the changes in this PR are still alive in 5.0.x branch (https://github.com/easybuilders/easybuild-easyblocks/blob/5.0.x/easybuild/easyblocks/generic/bundle.py). We can focus on that branch for trobleshooting and fixing.

Thyre · 2024-11-12T22:52:17Z

@Thyre the changes in this PR are still alive in 5.0.x branch (https://github.com/easybuilders/easybuild-easyblocks/blob/5.0.x/easybuild/easyblocks/generic/bundle.py). We can focus on that branch for trobleshooting and fixing.

@lexming We found the issue, see easybuilders/easybuild-easyconfigs#21841. It basically just boils down to loggers not being closed during the EasyConfig tests. I'll prepare an updated PR with a fix for this (and the comment regarding clang_aomp.py) tomorrow.

Signed-off-by: Jan André Reuter <[email protected]>

Thyre mentioned this pull request Oct 7, 2024

{compiler}[system/system] Add CodePlay CUDA SYCL plug-in variants for oneAPI 2023.0.0 and newer easybuilders/easybuild-easyconfigs#21582

Open

4 tasks

Thyre mentioned this pull request Oct 7, 2024

Bundle easyblock ignores make_module_req_guess() from components #2733

Closed

SebastianAchilles added this to the release after 4.9.4 milestone Oct 7, 2024

SebastianAchilles added the enhancement label Oct 7, 2024

Thyre force-pushed the generic-bundle-include-module-requirements branch from 07b4490 to 24cec43 Compare October 26, 2024 17:14

Thyre force-pushed the generic-bundle-include-module-requirements branch from 24cec43 to 5e3d7e4 Compare October 26, 2024 18:47

boegel changed the title ~~Enhance generic Bundle EasyBlock to transfer module requirements of components~~ Enhance generic Bundle easyblock to transfer module requirements of components Nov 6, 2024

Thyre mentioned this pull request Nov 6, 2024

Building CargoPythonBundle EasyConfigs fails because of maturin error easybuilders/easybuild-easyconfigs#21789

Open

lexming requested changes Nov 8, 2024

View reviewed changes

easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved

easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved

Thyre force-pushed the generic-bundle-include-module-requirements branch 2 times, most recently from 8247d63 to 957ee53 Compare November 8, 2024 13:31

Bundle: make implementation more efficient

eda3ca7

Instead of zipping bundles and configs every time, use a single instance list. Also, do not check for the type when building the module requirements and catch the exception instead. Signed-off-by: Jan André Reuter <[email protected]>

Thyre force-pushed the generic-bundle-include-module-requirements branch from 957ee53 to eda3ca7 Compare November 8, 2024 13:31

Bundle: Update authors

e696150

Signed-off-by: Jan André Reuter <[email protected]>

lexming approved these changes Nov 9, 2024

View reviewed changes

lexming merged commit 27c471c into easybuilders:develop Nov 9, 2024
41 checks passed

boegel reviewed Nov 12, 2024

View reviewed changes

boegel added a commit to boegel/easybuild-easyblocks that referenced this pull request Nov 12, 2024

revert changes from PR easybuilders#3472, seems to be leading to an i…

80c0e96

…nfinite loop in easyconfigs test suite?

jfgrimm added a commit that referenced this pull request Nov 12, 2024

Merge pull request #3504 from boegel/20241112184552_new_pr_bundle

28b2084

revert changes from PR #3472, seems to be leading to an infinite loop in easyconfigs test suite?

Thyre commented Nov 12, 2024

View reviewed changes

Thyre mentioned this pull request Nov 12, 2024

test: Test suite doesn't remove log files for QuantumESPRESSO (nested EasyBlocks?) easybuilders/easybuild-easyconfigs#21841

Open

Thyre mentioned this pull request Nov 14, 2024

QuantumESPRESSO: Let internal EasyBlock not create a log file #3505

Open

1 task

Thyre added a commit to Thyre/easybuild-easyblocks that referenced this pull request Nov 15, 2024

Reintroduce easybuilders#3472 after framework changes

a755c94

Signed-off-by: Jan André Reuter <[email protected]>

Thyre mentioned this pull request Nov 15, 2024

Reintroduce #3472 (transfer Bundle requirements to module) but do not create logfile in components #3509

Open

1 task

Enhance generic Bundle easyblock to transfer module requirements of components #3472

Enhance generic Bundle easyblock to transfer module requirements of components #3472

Conversation

Thyre commented Oct 7, 2024 • edited Loading

Thyre commented Oct 7, 2024 • edited Loading

Thyre commented Oct 7, 2024

Thyre commented Oct 7, 2024

Overview of tested easyconfigs (in order)

Thyre commented Oct 7, 2024 • edited Loading

Overview of tested easyconfigs (in order)

Thyre commented Oct 11, 2024 • edited Loading

Thyre commented Oct 11, 2024 • edited Loading

Thyre commented Oct 25, 2024 • edited Loading

Thyre commented Oct 26, 2024 • edited Loading

Thyre commented Oct 26, 2024 • edited Loading

Overview of tested easyconfigs (in order)

Thyre commented Oct 26, 2024 • edited Loading

Overview of tested easyconfigs (in order)

Thyre commented Oct 26, 2024 • edited Loading

Overview of tested easyconfigs (in order)

Thyre commented Oct 27, 2024 • edited Loading

Overview of tested easyconfigs (in order)

Thyre commented Nov 6, 2024

Thyre commented Nov 8, 2024

boegelbot commented Nov 8, 2024

boegelbot commented Nov 8, 2024

Overview of tested easyconfigs (in order)

Thyre commented Nov 8, 2024

boegelbot commented Nov 8, 2024

boegelbot commented Nov 8, 2024

Overview of tested easyconfigs (in order)

lexming left a comment

Choose a reason for hiding this comment

Thyre commented Nov 8, 2024 • edited Loading

Thyre commented Nov 8, 2024

boegelbot commented Nov 8, 2024

boegelbot commented Nov 8, 2024

Overview of tested easyconfigs (in order)

lexming commented Nov 8, 2024 • edited Loading

Thyre commented Nov 8, 2024 • edited Loading

lexming left a comment

Choose a reason for hiding this comment

lexming commented Nov 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Nov 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boegel commented Nov 12, 2024

Thyre Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Thyre Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Thyre Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

lexming commented Nov 12, 2024

Thyre commented Nov 12, 2024

Enhance generic `Bundle` easyblock to transfer module requirements of components #3472

Enhance generic `Bundle` easyblock to transfer module requirements of components #3472

Thyre commented Oct 7, 2024 •

edited

Loading

Thyre commented Oct 7, 2024 •

edited

Loading

Thyre commented Oct 7, 2024 •

edited

Loading

Thyre commented Oct 11, 2024 •

edited

Loading

Thyre commented Oct 11, 2024 •

edited

Loading

Thyre commented Oct 25, 2024 •

edited

Loading

Thyre commented Oct 26, 2024 •

edited

Loading

Thyre commented Oct 26, 2024 •

edited

Loading

Thyre commented Oct 26, 2024 •

edited

Loading

Thyre commented Oct 26, 2024 •

edited

Loading

Thyre commented Oct 27, 2024 •

edited

Loading

Thyre commented Nov 8, 2024 •

edited

Loading

lexming commented Nov 8, 2024 •

edited

Loading

Thyre commented Nov 8, 2024 •

edited

Loading

Thyre Nov 12, 2024 •

edited

Loading

Thyre Nov 12, 2024 •

edited

Loading

Thyre Nov 12, 2024 •

edited

Loading