Issues running/testing on Izumi

[ekluzek]

We've had trouble running on Izumi, where it will work for some -- but not others. Personally I had things working until ctsm5.2.007 where the CESM build went from python/3.7.0 to python/3.11.5. I did have to fiddle with things before that, but had it working. I also had tested a few cases when that change was coming and saw things working, but I found later that other tests would fail. It also started failing for @glemieux at this point as well. Others such as @samsrabin and @slevis-lmwg had things working. And I actually had both of them run tests for me during this time.

Below I reference some issues filed with NRIT CGD IT support regarding this (form HELP-????? for the ticket number). I worked with Joseph O'Neil and Garth D'Attilo on these things.

The problems I saw throughout this:

• I couldn't run CESM tests (once we moved to ctsm5.2.007)
• The ctsm_pylib environment created with python/3.11.5 was invalid (HELP-29204)
• Some tests failed for me because it couldn't find "env" (HELP-27216)

We talked about this at our CTSM SE meeting yesterday. Here's a link to the notes, which includes some about people's setup that they had working:

https://docs.google.com/document/d/1tgDRocjAFF1svWjt0mvjjPaPLFuDZEHLpf4vlGIQs5o/edit#heading=h.yotzlypez1l7

This relates to #2622

@briandobbins

[ekluzek]

Joseph figured out that module environments interact with conda environments -- so you can only really have one or the other working at the same time. So this means a couple things

1. Don't add module loads to ANY startup files
2. Don't do any module loads before running CESM cases
3. If you want to load a conda environment created with python/3.11.5 -- start a fresh shell and only load the conda environment

Once, I did that I could create a new ctsm_pylib environment with python/3.11.5. You create the environment with modules loaded and then only activate it in a new login shell where modules were NOT loaded.

[glemieux]

Thanks for this @ekluzek. I can confirm that following this cleared up the issues I was seeing.