Add 1 degree BGC configurations

[aidanheerdegen]

Overview

The full ACCESS-OM2 release requires the 1 degree configurations, IAF and RYF atmospheric forcing with bgc.

The goal is to have both COSIMA 1 degree BGC configurations as specially named branches in this repository, versioned (2.0) and tagged:

• release-1deg_jra55_ryf_bgc (repo)
• release-1deg_jra55_iaf_bgc (repo)

For each configuration this requires the following steps:

1. Add configuration to this repository in a dev-* branch
2. git checkout an update branch from this dev-* branch, modify it to use input paths from vk83, ACCESS-NRI deployed model versions and update some metadata and configuration options
3. Make a PR to update the dev-* branch created above. Ensure QA checks pass, then merge
4. Create a release-* branch if one doesn't already exist from the dev-* branch and run a special workflow to add reproducibility checksums, version and tag.

The release-1deg_jra55_ryf branch is the template to follow for RYF configurations. Similarly the release-1deg_jra55_iaf branch is the template to follow for IAF configurations.

There is a Contributing Guide which covers in more detail how to add and change configurations, but below are specific instructions for this specific task of porting COSIMA configurations to this repo.

The ACCESS-OM2-BGC spack.location file gives the paths to the BGC installed components (note only the MOM5 build differs from the physics only ACCESS-OM2):

[email protected]=2023.10.19         /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo
[email protected]=2023.10.26  /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo
[email protected]=2023.11.09          /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33

Steps to follow to add a COSIMA configuration to this repo:

1. Clone this repo, e.g. gh repo clone ACCESS-NRI/access-om2-configs
2. Clone configuration repo, e.g. gh repo clone COSIMA/1deg_jra55_ryf
3. cd access-om2-configs
4. Add the COSIMA repo as a remote, e.g. remote add 1deg_jra55_ryf ../1deg_jra55_ryf
5. Create branch from config repo, e.g. git checkout -b dev-1deg_jra55_ryf 1deg_jra55_ryf/master+bgc
6. Add configuration to GitHub repository, e.g. `git push dev-1deg_jra55_ryf

Steps to modify a configuration:

Because this is modifying an existing COSIMA configuration we would like to make sure the changes we're making are not introducing any source of error. There is a slow way (test bit reproducibility) or a fast way (use manifest checksums). Both options are outlined below. Choose which you're comfortable using. If you do choose the slow way, it probably isn't necessary to do it for all configurations, as it is possible to compare the branches and use that as a sanity check, as they should differ in small but deterministic ways.

1. Checkout update branch (don't choose a name that begins with dev- or release-), e.g. git checkout update-1deg_jra55_ryf

---

Either Option 1: make changes and rely on manifests for error checking (fast)

1. Modify config.yaml input paths to point to the new structure in vk83 (see 1 degree config). Note paths to qv56 for IAF configs remains unchanged.
2. Run payu setup to generate new input manifests. Run git diff manifests/input.yaml and confirm that only fullpath and binhash entries modified. All md5 hashes should be unchanged. This confirms the inputs are identical to the COSIMA configuration.
3. Modify model executable paths to those in vk83 (see above for paths, and note there are different CICE executables for the different resolutions)
4. Run payu setup to generate new executable manifests. Run git diff manifests/input.yaml and confirm that fullpath, binhash and md5 entries are all modified. All md5 hashes should be changed.

Or Option 2: make changes and use model runs to check reproducibility (or close to it)

1. Set runlog: false to prevent unnecessary git commits during testing
2. Change the run length to something short for testing purposes (maybe 1 day for 0.25 degree)
3. Create a ground-truth from the COSIMA configuration to compare against. Make sure you have diagnostic output at a high enough frequency that fields are output for the shortened run length. Good fields to choose are prognostic variables like salt, temp and velocities u, v (alternatively can compare the restart files the model outputs).
4. Modify config.yaml input paths to point to the new structure in vk83 (see 1 degree config). Note paths to qv56 for IAF configs remains unchanged.
5. Run short test run and check for reproducibility (see below). This confirms we have the same inputs as the COSIMA configuration.
6. Modify model executable paths to those in vk83 (see above for paths, and note there are different CICE executables for the different resolutions)
7. Do short test run and check differences are small (should just be some emerging noise depending on run length)
8. Change run length back, and turn runlog back on

---

2. Create a tools subdirectory and git mv resub.sh tools/.. Update config.yaml as required to point to new locations.
3. Change collate exe to point to /g/data/vk83/apps/mppnccombine-fast/0.2/bin/mppnccombine-fast
4. Update README (see 1 degree config example)
5. Add summary of testing done to this issue
6. git push update branch to repository and create a pull request to the dev-* branch you created and pushed in first section.
7. Fix all QA tests which fail in the CI testing. See Contributing Guide for an explanation of some the tests and what is typically required for a conforming configuration.
8. Once all QA tests are passing, request a review. When PR is approved merge.
9. Create a release branch and add reproducibility checksums following the instructions. In this case specify the initial version to be 2.0 to indicate that this is no longer bit reproducible with the COSIMA (1.0) versions.

As a guide this GitHub comparison shows the changes required for the 1 degree RYF configuration.

224b125...3f4cd6e

Bit reproducibility comparisons

We no longer have bit reproducibility with historical (COSIMA) versions of ACCESS-OM2 because we have opted to turn on compile options to ensure we have reproducibility across modified restart boundaries.

However the outputs should remain bit reproducible until the point where executables are modified.

Regardless, even when the executables are modified it is a good idea to run the model to check the differences after a short time are what we'd expect: just some noise in the fields.

For comparing model outputs I recommend a few different tools:

• nccmp (/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/nccmp)

nccmp is silent when files are equivalent, otherwise it will echo to STDERR whether metadata or a specific variable differs. By default, comparing stops after the first difference.

• ncdiff (/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/ncdiff)

ncdiff subtracts one netcdf file from another

• ncview (/apps/ncview/2.1.7/bin/ncview)

Simple Xwindows based netCDF file viewer. When subtracting one file from another it is a good idea to make sure the files contain actual data. If they're empty you may get a false negative for bit reproducibility as they will be identical.

6/4: Instructions updated

[minghangli-uni]

For the BGC configuration, a few essential executables and input files are required, such as,

1. exe of MOM,
2. input files of MOM, including bgc_param.nc, co2_iaf.nc, co2_ryf.nc, csiro_bgc.res.nc, csiro_bgc_sediment.res.nc and dust.nc.
3. input files of CICE

As far as I am aware, these files are not located in vk83.

[aidanheerdegen]

For the BGC configuration, a few essential executables and input files are required, such as,

1. exe of MOM,

Correct. BGC builds need to be added to the automated deployment. See ACCESS-NRI/spack-packages#71

This is discussed in #16 (comment)

2. input files of MOM, including bgc_param.nc, co2_iaf.nc, co2_ryf.nc, csiro_bgc.res.nc, csiro_bgc_sediment.res.nc and dust.nc.

I have copied over what I believe to be the correct versions of these files

ls -l /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.1deg/2022.02.24/
total 189680
-rw-r-----+ 1 aph502 vk83    152148 Mar  1 14:18 bgc_param.nc
-rw-r-----+ 1 aph502 vk83     80176 Mar  1 14:18 co2_iaf.nc
-rw-r-----+ 1 aph502 vk83     25596 Mar  1 14:18 co2_ryf.nc
-rw-r-----+ 1 aph502 vk83 187201997 Mar  1 14:18 csiro_bgc.res.nc
-rw-r-----+ 1 aph502 vk83     32065 Mar  1 14:18 csiro_bgc_sediment.res.nc
-rw-r-----+ 1 aph502 vk83   6724003 Mar  1 14:18 dust.nc

but I welcome any checks to ensure those are the correct files.

3. input files of CICE

Again, these should be the correct files. They're the same for IAF and RYF it seems, which is logical.

$ ls -l /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.1deg/2022.02.24/
total 24476
-rw-r-----+ 1 aph502 vk83 17282364 Mar  1 15:38 i2o.nc
-rw-r-----+ 1 aph502 vk83  7776964 Mar  1 15:38 o2i.nc

These are coupling fields. The model needs these from start-up. They're the same name as the physics only experiment, they just have more fields in them I assume. The way payu works the input file that is mentioned first is the one that is used, subsequent attempts to link the same filename into the work directory fail.

We won't have to rely on that, as we're linking directly to the full path for input files where possible.

[aidanheerdegen]

I am trying to determine what should be in the notes field of the metadata.yaml. There is a late issue with licensing I need to clear up.

[dougiesquire]

Testing of 1deg_jra55_ryf_bgc configuration:

Followed Option 1 above:

• inputs: no md5 hash changed, a few binhashhashes changed
• exes: all md5 and binhash hashes changed
• successfully ran configuration for 1 month

[dougiesquire]

Testing of 1deg_jra55_iaf_bgc configuration:

Followed Option 1 above:

• inputs: no md5 hash changed, a few binhash hashes changed
• exes: all md5 and binhash hashes changed
• successfully ran configuration for 1 month

[dougiesquire]

I think this can now be closed, but I'll leave that to someone else in case there's a reason to leave it open