Update config_machines.xml and corresponding files

[altheaden]

In this PR, I updated the config_machines.xml file, and likewise updated the module names and versions in chrysalis_gnu_openmpi.yaml. I also added some missing environment variables in the various .csh and .sh files for chrysalis, pm-cpu, and pm-gpu.

Checklist

• Testing comment in the PR documents testing used to verify the changes

[xylar]

A few things to change for now.

[altheaden]

@xylar Sorry about the comment confusion, I actually had several that I wanted your feedback on. I just re-wrote them so you could see them (instead of including them, accidentally, in a review). Would you mind looking at the other questions when you get a chance?

[altheaden]

Little progress update: Just added what I think is the last of the changes from the E3SM update, and I am about to start adding all the other missing environment variables.

[xylar]

Everything looks good. Let's see how testing with the pr suite goes on Chrysalis and Perlmutter with these changes.

[altheaden]

@xylar I forgot to finish and push the last bit of the pm files. These are basically the same changes, except I did add what appeared to be a missing NETCDF env var - I can remove this if necessary but I figured I'd add it for now since I was in the files anyways. I figured since there were other NETCDF env vars that it might be OK to add the extra one. I can start testing now.

[altheaden]

@xylar Just set up the conda env on pm and it appears to have been successful! These are the flags I tested with for reference:
./configure_polaris_envs.py --conda ~/miniforge3/ --env_name polaris_test --verbose --update_spack --spack $SCRATCH/spack_test --tmpdir $SCRATCH/spack_tmp --compiler gnu --machine pm-cpu --mache_fork altheaden/mache --mache_branch update-config-machines

[altheaden]

@xylar gnu run of the pr suite on pm passed all tests, intel run failed the 10km threads test but passed all others. I will take a look at the environment variables again in the morning and see if there's anything that might have screwed up threading in particular. Here is the log file from the failed run:
threads-out.log

[xylar]

@altheaden, great! It seems like it might be worth having me explore whether commenting back in one or more of these flags makes things pass for that thread test:

## purposefully omitting OMP variables that cause trouble in ESMF
# export OMP_STACKSIZE=128M
# export OMP_PROC_BIND=spread
# export OMP_PLACES=threads

In the meantime, I think we can proceed. We will just want to make a Polaris issue reporting that that particular test on that particular machine and compiler is not producing bit-for-bit results as expected. It's a little too early to add that report but if we see the same thing right before we're ready to merge the branch that updates to Polaris 0.4.0-alpha.1, we will post an issue and link to it in the comments about testing that version of Polaris.

[xylar]

@altheaden, in my testing today I noticed 2 things that we need to fix.

[altheaden]

@xylar After the if-statement change, the intel test had the same results as I got last night.

[xylar]

@altheaden, yes, that's what I saw, too. I also reran after commenting in the OMP* environment variables and I still see small differences. So this is a bug we're going to have to report to MPAS-Ocean developers to investigate.

[xylar]

Testing

@altheaden and I have used this branch to build Polaris' spack environment for:

• Chrysalis
  • intel
  • gnu
• Perlmutter-CPU
  • gnu
  • intel
  • nvidia
• Perlmutter-GPU
  • gnugpu
  • nvidiagpu (not quite done yet)

We have run the Polaris pr suite with the first 4 configurations (the last 3 are not supported). The one test thread test failed with intel on Perlmutter-CPU but we don't think that's related to these changes and will be reported for later investigation.

We ran the Omega CTests on Chrysalis with intel and gnu, and on Perlmutter-GPU with gnugpu. This required #174 and some modifications to Polaris that will be included in the next update.

[xylar]

This is amazing, @altheaden. Very nice work figuring all of this out and being so thorough.