v3.NARRM PE-layouts on Chrysalis

[amametjanov]

v3.NARRM WCYCL PE-layouts on Chrysalis:

• Tiny: 10 nodes -- ~1 sypd
• XSmall: 20 nodes -- ~2 sypd
• Small: 30 nodes -- ~3 sypd
• SMedium: 40 nodes -- ~4 sypd
• Medium: 50 nodes -- ~5 sypd
• XMedium: 64 nodes -- ~6 sypd

Todo:

• Large: 100 nodes --- ~10sypd

[BFB]

[github-actions]

PR Preview Action v1.4.8
🚀 Deployed preview to https://E3SM-Project.github.io/E3SM/pr-preview/pr-6700/
on branch gh-pages at 2024-10-29 04:31 UTC

[tangq]

These two PE layouts are great for short tests. Are we going to have larger layouts, e.g., S, M, and L, for longer runs? Or those will be added in a separate PR?

[tangq]

Hi @amametjanov , I notice that the XMedium (64 nodes) layout gives ~6 SYPD. This is very close to what I am having (5.92 SYPD) with all components equally assigned to 60 nodes (https://pace.ornl.gov/exp-details/200339). Is it because atm uses most of the time and we don't gain much by putting other components to different nodes from atm?

[amametjanov]

Yes, ATM uses ~30 seconds/mday out of total ~39 seconds/mday.
In your runs, OCN uses ~3.7 seconds/mday.
I saw 6.1 SYPD in Ld10-long runs.
I think you'll observe more than 6.1 SYPD in lengthier Ly5-long jobs.
This can be tried out with a few xmlchange to match this pelayout:

$ ./pelayout
Comp  NTASKS  NTHRDS  ROOTPE PSTRIDE
CPL :   3840/     1;      0      1
ATM :   3840/     1;      0      1
LND :   2304/     1;   1536      1
ICE :   1536/     1;      0      1
OCN :    256/     1;   3840      1
ROF :   2304/     1;   1536      1
GLC :      1/     1;      0      1
WAV :      1/     1;      0      1
IAC :      1/     1;      0      1
ESP :      1/     1;      0      1

and

$ ./case.setup -r && ./xmlchange BUILD_COMPLETE=TRUE && ./preview_run
$ ./case.submit

[tangq]

Yes, we can try it with the ongoing Ly-5 test run.

@amametjanov , could you paste the xmlchange commands here? (I am not quite sure about these commands.)

Do I have to run ./case.submit again after ./case.setup -r && ./xmlchange BUILD_COMPLETE=TRUE && ./preview_run? It will lose the existing place in the queue (The waiting time is 1-2 days.) I'd prefer keeping the place in the queue if we can skip .case.submit. Thanks.

[amametjanov]

Due to 60->64 node number change, ./case.submit is necessary (queue priority accumulation is slow though).
Here is a way to change PEs:

$ ./xmlchange NTASKS_CPL=3840,NTASKS_ATM=3840,NTASKS_ICE=1536,NTASKS_OCN=256,NTASKS_LND=2304,NTASKS_ROF=2304,NTASKS_GLC=1,NTASKS_WAV=1,NTASKS_IAC=1,NTASKS_ESP=1
$ ./xmlchange ROOTPE_OCN=3840,ROOTPE_LND=1536,ROOTPE_ROF=1536

[tangq]

OK, I cancelled the job and submitted a new one with this 64-node layout (jobid = 618254).