Override mache yaml file for Anvil with Gnu and OpenMPI

[xylar]

This merge adds a yaml file for Anvil's Gnu and OpenMPI environment that uses a more recent gcc. This appears to be necessary on Anvil to avoid problems when compiling Albany with gcc 8.2.0, see:
#523 (comment)
E3SM-Project/spack#12

Checklist

• Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

[xylar]

Testing

I successfully ran the full_integration suite with this branch (which is based off of #523 ) on Anvil with Gnu and OpenMPI. With the previous spack configuration, Albany could not be built.

See the directory:

/lcrc/group/e3sm/ac.xylar/compass_1.2/anvil/test_20230219/landice_full_new_yaml

[xylar]

@matthewhoffman, I'm leaving this in draft mode unil #523 gets merged but you can test this whenever you have time.

[matthewhoffman]

@xylar , I tested this on Anvil with:

./conda/configure_compass_env.py     --conda ${CONDA_BASE}     --mache_fork E3SM-Pro==> Error: Keyboard interrupt.main     --update_spack      --spack /lcrc/group/e3sm/ac.mhoffman/tmp/spack  --tmpdir /lcrc/group/e3sm/ac.mhoffman/tmp/spacktmp     --compiler gnu     --mpi openmpi --recreate --with_albany

and got this error:

1 error found in build log:
  >> 3    CMake Error at CMakeLists.txt:62 (CMAKE_MINIMUM_REQUIRED):
     4      CMake 3.23.0 or higher is required.  You are running version 3.20.3
     5
     6
     7    -- Configuring incomplete, errors occurred!

It is possible I didn't do something correctly though. I realize now that I probably should not be including the mache branch options, because they may be interfering with what this PR is meant to do. I'm trying again without them.

[xylar]

@matthewhoffman, yep, that's the moving target we were talking about. In the meantime, you will also need:
#549

[xylar]

I can rebase this onto #549 since it won't work without it.

[xylar]

@matthewhoffman, please try again when you have time.

[xylar]

@matthewhoffman, oh, dear! My ocean pr testing on Anvil with this configuration shows that nothing happens on steps that try to use multiple nodes. I don't think we can build our own MPI like this :-( :-(

[matthewhoffman]

I might be doing something wrong, but when I use this command on Anvil:

./conda/configure_compass_env.py     --conda ${CONDA_BASE}     --mache_fork E3SM-Project/mache     --mache_branch main     --update_spack      --spack /lcrc/group/e3sm/ac.mhoffman/tmp/spack  --tmpdir /lcrc/group/e3sm/ac.mhoffman/tmp/spacktmp     --compiler gnu     --mpi openmpi --recreate --with_albany

I get this error:

==> Error: No version for 'cmake' satisfies '@3.20.3' and '@3.24.3'. Consider setting `concretizer:unify` to `when_possible` or `false` to relax the concretizer strictness.

But I think that makes sense, because the spack spec specifies 3.20.3, but we need >3.23. I checked and 3.20.3 is the newest cmake the system provides.

I tried letting spack build cmake by removing the machine cmake from the spec:

diff --git a/conda/spack/anvil_gnu_openmpi.yaml b/conda/spack/anvil_gnu_openmpi.yaml
index 660bc46..9ccdc87 100644
--- a/conda/spack/anvil_gnu_openmpi.yaml
+++ b/conda/spack/anvil_gnu_openmpi.yaml
@@ -80,13 +80,6 @@ spack:
       - spec: [email protected]
         prefix: /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/zlib-1.2.11-dudhhig
       buildable: false
-    cmake:
-      externals:
-      - spec: [email protected]
-        prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/cmake-3.20.3-vedypwm
-        modules:
-        - cmake/3.20.3-vedypwm
-      buildable: false
     perl:
       externals:
       - spec: [email protected]

but then got this error that I don't know what to make of:

==> Error: Couldn't find patch for package builtin.rhash with sha256: 35dd7ee429a92fbe388152d009c3ab524abb9ac79ee78dfff8f3d8b747b6c568

[xylar]

Yes, I fixed that in #555 but didn't update this branch with the fix. Please use that branch instead.

[xylar]

However, as I noted, it doesn't work for me when running on multiple nodes so I don't know that it's worth testing.

[xylar]

Closing this to avoid confusion.