Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[email protected] #84

Merged
merged 18 commits into from
Dec 2, 2023
Merged

[email protected] #84

merged 18 commits into from
Dec 2, 2023

Conversation

jaelynlitz
Copy link
Contributor

@jaelynlitz jaelynlitz commented Nov 21, 2023

Merge request type

  • New feature
  • Resolves bug
  • Documentation
  • Other

Relates to

  • OPFLOW
  • SOPFLOW
  • SCOPFLOW
  • TCOPFLOW
  • CMake build system
  • Spack configuration
  • Manual
  • Web docs
  • Other - hiop interface

This MR updates

  • Header files
  • Source code
  • CMake build system
  • Spack configuration
  • Web docs
  • Manual
  • Other

Summary

Resolve issues interfacing with [email protected] onwards. Some function signature changes resulted in changes in the interface, other bugs TBD

Linked Issue(s)

@cameronrutherford
Copy link
Contributor

Turns out you have to mention the "closing keywords" for each issue individually. Don't have the time at the moment, but would be good to explore prettier issue templates like spack has

@jaelynlitz
Copy link
Contributor Author

jaelynlitz commented Nov 30, 2023

trying [email protected] on incline and am getting this error? have you seen this before @cameronrutherford?

https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/137810#L128

  1 ==> petsc: Executing phase: 'configure'
  2 ==> [2023-11-29-17:25:41.538594] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  3 ==> [2023-11-29-17:25:41.542749] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  4 ==> [2023-11-29-17:25:41.543214] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  5 ==> [2023-11-29-17:25:41.543359] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  6 ==> [2023-11-29-17:25:41.545360] '/share/apps/python/3.11.4/bin/python3.11' 'configure' '--prefix=/qfs/projects/exasgd/src/ci-incline/install/linux-    centos7-zen/clang-15.0.0-rocm5.3.0/petsc-3.20.1-qwd245ym3rxespvefk3rvqivxpv4xtqx' '--with-ssl=0' '--download-c2html=0' '--download-sowing=0' '--down    load-hwloc=0' '--with-make-exec=make' 'CXXFLAGS=--gcc-toolchain=/share/apps/gcc/8.4.0/' 'CXXOPTFLAGS=' '--with-cc=/share/apps/openmpi/4.1.4/gcc/8.4.    0/bin/mpicc' '--with-cxx=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpic++' '--with-fc=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpif90' '--with-precisio    n=double' '--with-scalar-type=real' '--with-shared-libraries=1' '--with-debugging=0' '--with-openmp=0' '--with-64-bit-indices=0' '--with-blas-lapack    -lib=/qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ahtbucwlxhwv4lkwnavfui/lib/libopenblas.so' '-    -with-x=0' '--with-sycl=0' '--with-clanguage=C' '--with-cuda=0' '--with-hip=0' '--with-metis=0' '--with-hypre=0' '--with-parmetis=0' '--with-kokkos=    0' '--with-kokkos-kernels=0' '--with-superlu_dist=0' '--with-ptscotch=0' '--with-suitesparse=0' '--with-hdf5=0' '--with-zlib=0' '--with-mumps=0' '--    with-trilinos=0' '--with-fftw=0' '--with-valgrind=0' '--with-gmp=0' '--with-libpng=0' '--with-giflib=0' '--with-mpfr=0' '--with-netcdf=0' '--with-pn    etcdf=0' '--with-moab=0' '--with-random123=0' '--with-exodusii=0' '--with-cgns=0' '--with-memkind=0' '--with-p4est=0' '--with-saws=0' '--with-yaml=0    ' '--with-hwloc=0' '--with-libjpeg=0' '--with-scalapack=0' '--with-strumpack=0' '--with-mmg=0' '--with-parmmg=0' '--with-tetgen=0'
  7 =============================================================================================
  8                          Configuring PETSc to compile on your system
  9 =============================================================================================
 10 =============================================================================================
 11                                      ***** WARNING *****
 12   Found environment variable:
 13   CC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang.
 14   Ignoring it, since its also set on command line
 15 =============================================================================================
 16 =============================================================================================
 17                                      ***** WARNING *****
 18   Found environment variable:
 19   CXX=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang++.
 20   Ignoring it, since its also set on command line
 21 =============================================================================================
 22 =============================================================================================
 23                                      ***** WARNING *****
 24   Found environment variable:
 25   FC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 26   Ignoring it, since its also set on command line
 27 =============================================================================================
 28 =============================================================================================
 29                                      ***** WARNING *****
 30   Found environment variable:
 31   F77=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 32   Ignoring it! Use "./configure F77=$F77" if you really want to use this value
 33 =============================================================================================
 34 =============================================================================================
 35                                      ***** WARNING *****
 36   Found environment variable: F90=/share/apps/gcc/8.4.0/bin/gfortran. Ignoring it! Use
 37   "./configure F90=$F90" if you really want to use this value
 38 =============================================================================================
 39 =============================================================================================
 40                                      ***** WARNING *****
 41   Using default C optimization flags "-g -O3". You might consider manually setting optimal
 42   optimization flags for your system with COPTFLAGS="optimization flags" see
 43   config/examples/arch-*-opt.py for examples
 44 =============================================================================================
 45 =============================================================================================
 46                                      ***** WARNING *****
 47   Using default FC optimization flags "-g -O". You might consider manually setting optimal
 48   optimization flags for your system with FOPTFLAGS="optimization flags" see
 49   config/examples/arch-*-opt.py for examples
 50 =============================================================================================
 51
 52 *********************************************************************************************
 53            UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
 54 ---------------------------------------------------------------------------------------------
 55   Timeout: Unable to run MPI program with /share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpiexec
 56   --oversubscribe -n 1
 57   (1) make sure this is the correct program to run MPI jobs
 58   (2) your network may be misconfigured; see
 59   https://petsc.org/release/faq/#mpi-network-misconfigure
 60   (3) you may have VPN running whose network settings may not play nice with MPI
 61 *********************************************************************************************

@cameronrutherford
Copy link
Contributor

trying [email protected] on incline and am getting this error? have you seen this before @cameronrutherford?

  1 ==> petsc: Executing phase: 'configure'
  2 ==> [2023-11-29-17:25:41.538594] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  3 ==> [2023-11-29-17:25:41.542749] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  4 ==> [2023-11-29-17:25:41.543214] Find (not recursive): /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ck    x5653ahtbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  5 ==> [2023-11-29-17:25:41.543359] Find complete: /qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ah    tbucwlxhwv4lkwnavfui/lib ['libopenblas.so', 'openblas.so']
  6 ==> [2023-11-29-17:25:41.545360] '/share/apps/python/3.11.4/bin/python3.11' 'configure' '--prefix=/qfs/projects/exasgd/src/ci-incline/install/linux-    centos7-zen/clang-15.0.0-rocm5.3.0/petsc-3.20.1-qwd245ym3rxespvefk3rvqivxpv4xtqx' '--with-ssl=0' '--download-c2html=0' '--download-sowing=0' '--down    load-hwloc=0' '--with-make-exec=make' 'CXXFLAGS=--gcc-toolchain=/share/apps/gcc/8.4.0/' 'CXXOPTFLAGS=' '--with-cc=/share/apps/openmpi/4.1.4/gcc/8.4.    0/bin/mpicc' '--with-cxx=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpic++' '--with-fc=/share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpif90' '--with-precisio    n=double' '--with-scalar-type=real' '--with-shared-libraries=1' '--with-debugging=0' '--with-openmp=0' '--with-64-bit-indices=0' '--with-blas-lapack    -lib=/qfs/projects/exasgd/src/ci-incline/install/linux-centos7-zen/gcc-8.4.0/openblas-0.3.20-ii6ckx5653ahtbucwlxhwv4lkwnavfui/lib/libopenblas.so' '-    -with-x=0' '--with-sycl=0' '--with-clanguage=C' '--with-cuda=0' '--with-hip=0' '--with-metis=0' '--with-hypre=0' '--with-parmetis=0' '--with-kokkos=    0' '--with-kokkos-kernels=0' '--with-superlu_dist=0' '--with-ptscotch=0' '--with-suitesparse=0' '--with-hdf5=0' '--with-zlib=0' '--with-mumps=0' '--    with-trilinos=0' '--with-fftw=0' '--with-valgrind=0' '--with-gmp=0' '--with-libpng=0' '--with-giflib=0' '--with-mpfr=0' '--with-netcdf=0' '--with-pn    etcdf=0' '--with-moab=0' '--with-random123=0' '--with-exodusii=0' '--with-cgns=0' '--with-memkind=0' '--with-p4est=0' '--with-saws=0' '--with-yaml=0    ' '--with-hwloc=0' '--with-libjpeg=0' '--with-scalapack=0' '--with-strumpack=0' '--with-mmg=0' '--with-parmmg=0' '--with-tetgen=0'
  7 =============================================================================================
  8                          Configuring PETSc to compile on your system
  9 =============================================================================================
 10 =============================================================================================
 11                                      ***** WARNING *****
 12   Found environment variable:
 13   CC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang.
 14   Ignoring it, since its also set on command line
 15 =============================================================================================
 16 =============================================================================================
 17                                      ***** WARNING *****
 18   Found environment variable:
 19   CXX=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/clang++.
 20   Ignoring it, since its also set on command line
 21 =============================================================================================
 22 =============================================================================================
 23                                      ***** WARNING *****
 24   Found environment variable:
 25   FC=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 26   Ignoring it, since its also set on command line
 27 =============================================================================================
 28 =============================================================================================
 29                                      ***** WARNING *****
 30   Found environment variable:
 31   F77=/qfs/people/svcexasgd/gitlab/26194/spack_incline/tpl/spack/lib/spack/env/clang/gfortran.
 32   Ignoring it! Use "./configure F77=$F77" if you really want to use this value
 33 =============================================================================================
 34 =============================================================================================
 35                                      ***** WARNING *****
 36   Found environment variable: F90=/share/apps/gcc/8.4.0/bin/gfortran. Ignoring it! Use
 37   "./configure F90=$F90" if you really want to use this value
 38 =============================================================================================
 39 =============================================================================================
 40                                      ***** WARNING *****
 41   Using default C optimization flags "-g -O3". You might consider manually setting optimal
 42   optimization flags for your system with COPTFLAGS="optimization flags" see
 43   config/examples/arch-*-opt.py for examples
 44 =============================================================================================
 45 =============================================================================================
 46                                      ***** WARNING *****
 47   Using default FC optimization flags "-g -O". You might consider manually setting optimal
 48   optimization flags for your system with FOPTFLAGS="optimization flags" see
 49   config/examples/arch-*-opt.py for examples
 50 =============================================================================================
 51
 52 *********************************************************************************************
 53            UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
 54 ---------------------------------------------------------------------------------------------
 55   Timeout: Unable to run MPI program with /share/apps/openmpi/4.1.4/gcc/8.4.0/bin/mpiexec
 56   --oversubscribe -n 1
 57   (1) make sure this is the correct program to run MPI jobs
 58   (2) your network may be misconfigured; see
 59   https://petsc.org/release/faq/#mpi-network-misconfigure
 60   (3) you may have VPN running whose network settings may not play nice with MPI
 61 *********************************************************************************************

Maybe this isn't the right thing to do, but one idea is to rebuild ROCm 5.6.0 and a new clang version like we have already done for ReSolve https://github.com/ORNL/ReSolve/pull/53/files.

#89 and @nkoukpaizan was mentioning test failures anyway, so maybe that's a bad idea.

I don't know what could be going wrong here, but there should be some better logs being printed in configure.log that you need to look at that will say more of what is going wrong.

I would suggest maybe you look at the incline PR and review tricky SLURM configuration issues that might be tripping you up.

@cameronrutherford
Copy link
Contributor

cameronrutherford commented Nov 30, 2023

If you don't have any ideas, we can also ping Satish and xSDK folks for advice. You can quickly try PETSc 3.19 as well since that should also work in theory

@jaelynlitz
Copy link
Contributor Author

If you don't have any ideas, we can also ping Satish and xSDK folks for advice. You can quickly try PETSc 3.19 as well since that should also work in theory

I am very confused bc the first pipeline petsc 3.19.6 failed, but on the retry it built? https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/138077 I'm stumped but won't question it...

@cameronrutherford
Copy link
Contributor

cameronrutherford commented Nov 30, 2023

If you don't have any ideas, we can also ping Satish and xSDK folks for advice. You can quickly try PETSc 3.19 as well since that should also work in theory

I am very confused bc the first pipeline petsc 3.19.6 failed, but on the retry it built? https://gitlab.pnnl.gov/exasgd/frameworks/exago-github-mirror/-/jobs/138077 I'm stumped but won't question it...

Glad 3.19 builds, but 3.20 should work.

None of the pipelines or error messages you have linked are at the right level for us to diagnose the problem. Please get the right configure.log or error message that describes an actual issue occurring. Currently we are blindly speculating and praying, and 3.20 should still work just fine.

tests/functionality/opflow/CMakeLists.txt Show resolved Hide resolved
@jaelynlitz
Copy link
Contributor Author

Waiting for the ascent rebuild to run - then this is good to go! I just opened a spack PR, so when that is merged, we can move the spack submodule back to spack proper instead of my fork.

@cameronrutherford cameronrutherford merged commit 2bdd5b7 into develop Dec 2, 2023
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants