How to switch from NetCDF output to ADIOS2 output?

[kiraduwe]

Hi,
I have a question regarding changing the PIO_IOTYPE . How do I change the IO from netcdf to something else, in this case ADIOS2? In CIME it is mentioned that one should not modify the *_modelio.nml files in the CaseDocs directory. So, where am I supposed to change the used format?
In the run_e3sm.template.csh there are PIO settings. However, the following does not work:
xmlchange_exe -file env_run.xml -id PIO_TYPENAME -val adios

The error I get does not include any option related to ADIOS2:
ERROR: Did not find adios in valid values for PIO_TYPENAME: ['netcdf', 'pnetcdf', 'netcdf4p', 'netcdf4c', 'default']

Many thanks in advance!

[philipwjones]

Tagging @jayeshkrishna for latest status.

[jayeshkrishna]

@Bella42 : Please let us know the machine you are trying to use ADIOS, @dqwu can provide you instructions on how to run the simulations using ADIOS (The ADIOS installation paths are currently missing in E3SM master and might be the source of your issue. However you should be able to run your simulation with ADIOS using @dqwu 's branch and instructions.).

[kiraduwe]

@jayeshkrishna Thank you very much for your fast reply and for pointing me towards @dqwu ' s branches! I think, a combination of the changes in dqwu/test_pio2_adios2_anlworkstation and dqwu/test_scorpio_adios2 should work for me :-)
Currently, I am trying to use E3SM with ADIOS on my laptop as a first test.

[dqwu]

@Bella42
Please note that dqwu/test_pio2_adios2_anlworkstation branch is out of date, so you should just refer to dqwu/test_scorpio_adios2 branch. You also need dqwu/adios2_support branch of ESMCI/cime, which is
now a submodule of E3SM.

If you want to run E3SM cases in your laptop, you might add a new entry for your machine, probably based on
"linux-generic" (or "mac"). For example, "anlworkstation" is extended from "linux-generic".
You also need to checkout input files to your laptop (which could be huge), and build required software stacks
like hdf5/netcdf/pnetcdf/aidos

So it is highly recommended to test E3SM with one of the machines listed in cime_config/machines/config_machines.xml.
Currently, ADIOS 2.5 is only installed on the following machines:
anlworkstation
cori-knl
summit
theta
bebop
anvil
compy

Let's first decide the machine you want to run some E3SM cases. If you want to use your laptop, you will need some setups mentioned above.

[kiraduwe]

@dwqu Thank you very much for your detailed response!
I think I need to clarify some points. Sorry if I made them not clear enough.
I already modified the necessary files (e.g. added an entry to config_machines.xml etc.) so that a smaller test case is already running on my laptop. Took a while to get it all together but it works now.
My questions really only were about the possibilities to change the output from NetCDF to ADIOS.

I only referred to the dqwu/test_pio2_adios2_anlworkstation branch as I thought I probably need to make similar changes to buildlib.pio as you did, in addition to those in dqwu/test_scorpio_adios2 to actually have a parameter to pass to xmlchange_exe.
Thanks to your information about your branches I now know that these are in the dqwu/adios2_support branch of ESMCI/cime.

[dqwu]

@Bella42
Did you install ADIOS 2.5 on your laptop before?

If you have not done so, you can install it with following steps (change CMAKE_INSTALL_PREFIX):

wget https://github.com/ornladios/ADIOS2/archive/v2.5.0.tar.gz
tar zxf v2.5.0.tar.gz
cd ADIOS2-2.5.0
mkdir build
cd build
CC=mpicc CXX=mpicxx FC=mpifort CFLAGS="-g -O2" CPPFLAGS="-g -O2" CXXFLAGS="-g -O2" FCFLAGS="-g -O2" cmake -DCMAKE_INSTALL_PREFIX=your_adios_installation_directory -DBUILD_SHARED_LIBS=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_BZip2=OFF -DADIOS2_USE_HDF5=OFF ..
make -j16
make install

Then you can refer to dqwu/test_scorpio_adios2 branch to add ADIOS2_DIR to config_machines.xml and config_compilers.xml
for your laptop.

[dqwu]

@Bella42
Here are steps to run ne4 F case on anlworkstation.
To run your small E3SM case on your laptop, the steps should be quite similar.

1. Check out latest E3SM master branch, and merge dqwu/test_scorpio_adios2 to support ADIOS type.

git clone https://github.com/E3SM-Project/E3SM.git E3SM_ADIOS
cd E3SM_ADIOS
git merge --no-edit origin/dqwu/test_scorpio_adios2
git submodule update --init --recursive

2. Update some cime files to support ADIOS type.

cd cime
git merge --no-edit origin/dqwu/adios2_support

3. Create a new ne4 F case under cime/scripts directory.
   For create_newcase command, you might specify the machine name (your laptop)
   explicitly with --machine option, if it could not automatically detect it.

cd scripts
./create_newcase --case FC5AV1C-H01B_ne4_ne4_ADIOS --compset FC5AV1C-H01B --res ne4_ne4
cd FC5AV1C-H01B_ne4_ne4_ADIOS

4. Create a new CAM namelist file for this ne4 F case

cat <<EOF >> user_nl_cam
mfilt = 1,48
nhtfrq = -120,-1
fincl2 ='OMEGA500:I','OMEGA850:I','FLNT:I','TMQ:I','PRECT:I','PS:I'
'TS:I',,'T850:I','U250:I',,'U850:I','UBOT:I','V250:I',,'V850:I','VBOT:I',
,'Z500:I','SWCF:I','LWCF:I','CLDLOW:I','CLDMED:I','CLDHGH:I',
'U:I'
EOF

5. Some xml settings for testing this ne4 F case. Set ADIOS type to ATM, ICE and LND.
   Note: For F case, CPL could not use ADIOS type so far (reading back ADIOS type output file cpl.r.0001-01-02-00000.nc not supported yet), so we just use default PnetCDF type instead.

./xmlchange STOP_OPTION=ndays
./xmlchange STOP_N=1
./xmlchange PIO_TYPENAME_ATM=adios
./xmlchange PIO_TYPENAME_ICE=adios
./xmlchange PIO_TYPENAME_LND=adios

6. Set up, preview, build and run ne4 F case with ADIOS type.

./case.setup

./preview_run

./case.build

./case.submit

[kiraduwe]

@Bella42
Did you install ADIOS 2.5 on your laptop before?

@dqwu Thank you for the explanation. Yes, I did. I am currently working on my own ADIOS2 engine. So from all the libraries and components in and around E3SM ADIOS2 is the most familiar one to me.

[kiraduwe]

Here are steps to run ne4 F case on anlworkstation.
To run your small E3SM case on your laptop, the steps should be quite similar.

@dqwu Thank you so much for this very helpful instruction! This should be a great help to get everything running.