Skip to content

Conference call notes 20241009

Jump to bottom
Kenneth Hoste edited this page Oct 9, 2024 · 1 revision

(back to Conference calls)

Notes on the 256th EasyBuild conference call, Wednesday 9 Oct 2024 (10:00 UTC)

Attendees

List of attendees (18):

  • Simon Branford (Univ. of Birmingham, UK)
  • Bibek Chapagain (Barcelona Supercomputing Centre, Spain)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Davide Grassano (CECAM, Switzerland)
  • Jasper Grimm (University of York, UK)
  • Jorge Guerra (Universidad Politécnica de Madrid, Spain)
  • Alexander Grund (ZIH, Dresden, Germany)
  • Leonardo Honfi Camilo (Univ. of Wageningen, Netherlands)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Sébastien Moretti (SIB, Switzerland)
  • Alan O'Cais (CECAM, Univ. of Barcelona)
  • Jan Reuter (JSC, Germany)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Åke Sandgren (Umeå University, Sweden)
  • Alexandre Strube (JSC, Germany)
  • Alain van Hoof (TU Eindhoven, Netherlands
  • Cintia Willemyns (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a/2024b update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.4 (22 Sept 2024)
  • next EasyBuild release:
    • very likely EasyBuild v5.0.0 🔥
    • additional EasyBuild 4.9.x versions could still be done via 4.9.x branches, but none are planned currently
  • EasyBuild v5.0.0
    • project board: https://github.com/orgs/easybuilders/projects/18/views/2
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version
  • collapsing of 5.0.x branches in develop post-poned
    • impact on people using EasyBuild 4.x considered too big, so we want to limit the time window in which problems will arise as much as possible
    • impact:
      • --from-pr will be broken with EasyBuild 4.x due to major version mismatch
        • --from-commit can be used as a workaround
      • open easyconfig PRs should be synced with develop branch before they get merged
        • due to changes in easyconfigs test suite implemented in 5.0.x
        • for example use_pip = True should no longer be used
        • same goes for framework & easyblocks

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • [develop] resolve symlink when making log dir writable (PR #4658)
      • [5.0.x] also define functions in interactive shell session created by cmd.sh script that is produced by run_shell_cmd (PR #4662)
      • [5.0.x] add flushing and possible closing of stdin on run_shell_cmd (PR #4664)
      • [develop] Fix typo in veryloose toolchain option for RISC-V (PR #4668)
    • enhancements
      • [5.0.x] prepend to $PYTHONPATH or $EBPYTHONPREFIXES in generated module files by automatically scanning for python site package directories (PR #4539)
      • [develop] improve error message when missing commit message for --new-pr (PR #4640)
    • changes
      • [5.0.x] enable module-depends-on by default (PR #4500)
      • [5.0.x] use Slurm as the default job backend + deprecate support for GC3Pie as job backend (PR #4659)
    • code cleanup
      • ...
    • tests
      • [5.0.x] also run tests with Python 3.13 (PR #4674)

  • easyblocks (merged PRs)

    • bug fixes
      • [5.0.x] Make CMakeMake respect the toolchainopts when selecting build type (PR #3452)
      • [develop] strip leading local/ from pylibdir where appropriate (PR #3464)
      • [develop] Remove X11 flag for GROMACS 2023+ (PR #3466)
    • enhancements
      • [5.0.x] update easyblock to let EasyBuild framework take care of prepend to $PYTHONPATH or $EBPYTHONPREFIXES + make PythonBundle and PythonPackage aware of --prefer-python-search-path EasyBuild configuration option (PR #3343)
      • [develop] Update ROOT easyblock to support sysroot (PR #3467)
    • updates
      • [develop] enhance AOCC EasyBlock to work with LLVM 16 and newer (PR #3458)
      • [develop] Update VTune EasyBlock to work with v2024 and newer (PR #3465)
    • changes
      • ...
    • new easyblocks
      • ...
    • code cleanup
      • ...

  • easyconfigs (merged PRs)

    • ~75 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • [develop] replace buggy ESIpy v20240709 with v20240731 (PR #21095)
      • [develop] add patch to fix compilation of HPDBSCAN 20210826 (PR #21467)
      • [develop] add patch for ROOT v6.24.06, v6.26.06, v6.26.10, v6.30.6 to disable sysroot for cling at runtime (PR #21526)
      • [develop] link libOpenCL.so.1 to the lib directory for NextGenMap (required when using RPATH linking) (PR #21528)
      • [develop] add patch for bug/typo in RISC-V toolchain options to EasyBuild 4.9.4 easyconfig (PR #21547)
      • [develop] strip out '-minsize 1' option from umi_binning.sh in longread_umi 0.3.2, since it's not supported by VSEARCH (PR #21557)
    • enhancements
      • [develop] add missing (optional) dependency cp2k-input-tools to phonopy 2.20.0 (PR #21550)
      • [develop] add goldilocks + gsDesign extensions to R-bundle-CRAN v2024.06 (PR #21583)
    • (noteworthy) new software
    • noteworthy software updates
      • ...
    • cleanup
      • [5.0.x] stop using modextrapaths to update $PYTHONPATH with standard path to installed Python packages (lib/python%(pyshortver)s/site-packages) (PR #20960)
    • changes
      • [5.0.x] use pycodestyle for code style check + stop using l in list comprehensions (PR #21502)
    • EasyBuild 5.0
      • [5.0.x] sync with develop (20240930) (PR #21497)

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

    • bug fixes
      • [5.0.x] Reverse order for parsing files in $XDG_CONFIG_DIRS (PR #4630)
      • [5.0.x] improve portability of reproducible tarballs by replacing external tar command with tarfile module (PR #4660)
        • to fix issue #4657: reproducible tarballs have unexpected checksums on mac
        • tarfile module in Python allows creating tarballs in a reproducible way across systems (Linux + macOS)
          • but there's a breaking change in Python 3.9, so we get different tarballs across different Python versions :man-facepalming:
          • two options to deal with this:
            • only support creating reproducible tarballs with Python 3.9+
              • people using Python < 3.9 would have to use --ignore-checksums
              • EasyBuild 5.0 could print a warning when using Python < 3.9 that checksums can not be verified?
              • this seems like the best way forward?
              • we should probably consider deprecating support for Python < 3.9 in EasyBuild 5.0
            • always include two checksums for source tarballs created via git_config: one for Python < 3.9, one for Python >= 3.9
          • binary patching generated tarballs with older Python versions would be very difficult
      • get_os_name should be updated to include Rocky and Alma (PR #4663)
    • enhancements
      • [5.0.x] disable strict RPATH sanity check by default, allow re-enabling it via --strict-rpath-sanity-check configuration option (PR #4475)
      • [develop] Copy build log and artifacts to a permanent location after failures (WIP PR #4601)
      • [develop] allow extensions to add text to the module file (PR #4652)
      • Problem using $CPATH in modulefiles overwriting system paths (issue #3331)
        • [5.0.x] add --search-path-cpp-headers configuration option to control how EasyBuild sets paths to headers at build time (PR #4645)
        • [5.0.x] refactor generation of required environment variables in module files (PR #4653)
          • test was added to verify that making ModuleLoadEnvironment a singleton causes trouble
          • needs more love, additional test is broken
          • easyblocks will need to be updated accordingly to not use make_module_req_guess anymore
        • [5.0.x] add module-search-path-headers configuration option to control how modules set search paths to header files (PR #4655)
      • [develop] add support for injecting checksums for cargo crates (PR #4661)
      • [develop] add support for %(rpath)s template value (PR #4670)
        • this makes it even more important that we merge the PR that ensures that templates are resolved
    • changes
      • [5.0.x] Rename SOURCE_STEP to EXTRACT_STEP (PR #4629)
      • [5.0.x] Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)
      • [5.0.x] Deprecate post_install_step, rename it to post_processing_step (issue #4656)
      • [5.0.x] proposed changest to add a timeout to run_shell_cmd (PR #4665)
      • [5.0.x] include path to cmd.sh script in output generated by run_shell_cmd when a command fails + use colors: red for ERROR line, yellow for path to output files + cmd.sh script (PR #4666)
      • [develop] Added possibility to call amend/try-amend multiple times from easystack (PR #4667)
      • [5.0.x] Let jobs retweak easyconfigs themselves (draft PR #4669)
      • [5.0.x] don't raise error when required extensions are not found when installing extensions in parallel (draft PR #4671)
      • [5.0.x] mark support for installing extensions in parallel as being mature, since it's no longer experimental (PR #4672)
      • [5.0.x] mark easystack support as being mature, since it's no longer experimental (PR #4673)

  • easyblocks (open PRs + issues)

    • bug fixes
      • [develop] update Kokkos CPU mapping for LAMMPS >= 2Aug2023 to use icx on Intel Icelake systems #3321 (PR #3321)
      • [5.0.x] run pip check only once for PythonBundle (PR #3432)
      • [5.0.x] Make MesonNinja respect the toolchainopts with buildtype as well as --debug and --optimization flags (PR #3454)
      • [develop] Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set (PR #3460)
      • [develop] let CMakeMake easyblock also set Python_EXECUTABLE option, as well as Python3_EXECUTABLE and Python2_EXECUTABLE derivatives (when appropriate) (PR #3463)
      • [develop] take into account that f90wrap_version may be None in MetalWalls easyblock (PR #3468)
      • [5.0.x] take into account that 'pic' toolchain option may not be defined in CMakeMake easyblock (PR #3470)
      • [develop] fix setting of CMAKE_OPTIONS for SuiteSparse versions between 5.1.2 and 6.0.0 (PR #3471)
      • [5.0.x] add missing return for customised install_extension_async methods in Rserve and Rmpi easyblocks (PR #3474)
    • enhancements
      • [develop] Set CMake hints for path to Python installation if it's direct or indirect dependency (when using CMake >= 3.12) (PR #3282)
      • [develop -> 5.0.x] enhance LLVM easyblock for compilation of clang/flang + other llvm-projects (PR #3373)
      • [5.0.x] enhance CMakeMake easyblock to check whether correct Python installation was picked up by CMake (PR #3399)
      • [develop] ignore user python packages by using -s option when running Python code via "python -c" (PR #3440)
      • [develop] enhance generic Bundle EasyBlock to transfer module requirements of components (PR #3472)
      • [develop] enhance cuDNN easyblock to verify that EULA is accepted before installing it ((PR #3473)
    • updates
      • [develop] update custom easyblock for CP2K >= v2024 (PR #3433)
    • changes
      • ...
    • new
      • [develop] custom easyblock for DeepSpeed (PR #3450)

  • easyconfigs (open PRs + issues)

    • bug fixes/reports
      • [develop -> 5.0.x] switch to CargoPythonBundle easyblock for orjson 3.9.15 (PR #21392)
        • requires being able to produce reproducible source tarballs from Git repository (WIP in 5.0.x branch of EasyBuild framework)
      • [5.0.x] promote pybind11 to runtime dependency for CPPE (to fix pip check failure) (PR #21564)
      • [develop] add patch to SciPy-bundle 2024.05 that fixes test failure on aarch64 (PR #21559)
    • enhancements
      • [develop] ignore user-installed packages when running python in sanitycheck (PR #21362)
      • [develop] also install w90spn2spn.x + w90chk2chk.x for Wannier90 3.1.0 (PR #21383)
    • (noteworthy) new software
      • ...
    • software updates
    • changes
      • [5.0.x] Replace parallel by maxparallel (PR #19375)
      • [develop] GTDB-Tk v2.4.0 should use skani and not FastANI (PR #21518)
        • (Åke) make sure that CPU boost is working on your systems if people are using this...

2024b common toolchains

  • GCC 14.2 as a base (see easyconfigs PR #21114)
  • we should define candidate toolchains using latest version of all components
    • are newer versions of Intel components compared to intel/2024a already available?
  • should we consider replacing intel with Clang-based toolchain as 2nd common toolchain next to foss?
    • Fortran frontend in Clang is becoming mature enough?

Q&A / others

  • NVHPC
    • missing library path for QC libraries, should be added to NVHPC easyblock
  • trouble with building Wannier90 with NVHPC (Fortran code)
  • "funny" case: infinite loop in spaln configure script (see https://github.com/ogotoh/spaln/issues/74)
  • Implement sanity check in EasyBuild for CUDA compute capabilities (see EESSI support issue #92)
    • should probably be implemented in EasyBuild, but as opt-in?
    • needs to be taken into account in easyblocks like LAMMPS that pick one of the specified CUDA compute capabilities
      • easyblock should "register" which CUDA compute capability it's actually using
      • which one is picked is inconsistent across easyblocks
  • (Alexandre) why are we still using Bazel 6.x (not 7.x)
    • there are several open PRs for Bazel 7.x
    • TensorFlow & friends may require specific (older) Bazel versions
  • (Jan) PR to implement make_module_req_guess in Bundle (see PR #3472)
    • should try and avoid re-creating easyblock instances
    • required for CodePlay SYCL, see easyconfigs PR #21582
  • (Alexander) upcoming PR for Cargo easyblock to change how it deal with git_config, would be good to test it throroughly
Clone this wiki locally