Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a list of test types to the users guide #2062

Merged
merged 4 commits into from
Nov 15, 2017
Merged

Add a list of test types to the users guide #2062

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
aae8c78
Add list of test types to users guide.
fischer-ncar Nov 14, 2017
2f7b948
Add a list of the different test types to section 4 of
fischer-ncar Nov 14, 2017
42e7527
Test suite:
fischer-ncar Nov 15, 2017
fb7dc2d
Changes requested by bertinia
fischer-ncar Nov 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 122 additions & 10 deletions doc/source/users_guide/testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,11 @@
Testing with create_test
**************************

.. todo ::

add list and descriptions on functionality tests (ERS, ERI, ect)

create_test is the tool we use to test cime-driven models. It can be used as a easy way to
run a single basic test or can be used to run an entire suite of tests. It has parallelism
built-in on many levels to make full use of the underlying system. create_test is the driver
behind the automated nightly testing of cime-driven models.
**create_test** is the tool we use to test cime-driven models. It can be used as an easy way to
run a single basic test or an entire suite of tests. create_test runs a test suite in parallel
for improved performance. create_test is the driver behind the automated nightly testing of
cime-driven models.

Tests come in form::

Expand All @@ -22,10 +19,125 @@ A common test type would be ERS which stands for exact restart, which means you
grid and compset to see if they reproduce the same result upon restart as they would if no restart
took place. GRID and COMPSET are self-explanatory. MACHINE_COMPILER is optional; create_test will probe
the underlying machine and use the default compiler for that machine if this value is not
supplied. TESTMODS is also optional and can be used to make some env XML changes prior to doing
the test.
supplied. TESTMODS are optional and can be used to make some env XML changes prior to running
a test.

======================= =====================================================================================
TESTTYPE Description
======================= =====================================================================================
ERS Exact restart from startup (default 6 days + 5 days)
Do an 11 day initial test - write a restart at day 6. (file suffix: base)

Do a 5 day restart test, starting from restart at day 6. (file suffix: rest)

Compare component history files '.base' and '.rest' at day 11.
ERS2 Exact restart from startup (default 6 days + 5 days).
Do an 11 day initial test without making restarts. (file suffix: base)

Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (file suffix: rest)

Compare component history files ".base" and ".rest" at day 11.
ERT Exact restart from startup, default 2 month + 1 month (ERS with info DBUG = 1).
IRT Exact restart from startup, (default 4 days + 7 days) with restart from interim file.
ERIO Exact restart from startup with different PIO methods, (default 6 days + 5 days).
ERR Exact restart from startup with resubmit, (default 4 days + 3 days).
ERRI Exact restart from startup with resubmit, (default 4 days + 3 days). Tests incomplete logs option for st_archive.
ERI hybrid/branch/exact restart test, default (by default STOP_N is 22 days)

(1) ref1case

Do an initial run for 3 days writing restarts at day 3.

ref1case is a clone of the main case.

Short term archiving is on.

(2) ref2case (Suffix hybrid)

Do a hybrid run for default 19 days running with ref1 restarts from day 3,

and writing restarts at day 10.

ref2case is a clone of the main case.

Short term archiving is on.

(3) case

Do a branch run, starting from restarts written in ref2case,

for 9 days and writing restarts at day 5.

Short term archiving is off.

(4) case (Suffix base)

Do a restart run from the branch run restarts for 4 days.

Compare component history files '.base' and '.hybrid' at day 19.

Short term archiving is off.
ERP PES counts hybrid (OPENMP/MPI) restart bit for bit test from startup, (default 6 days + 5 days).
Initial PES set up out of the box

Do an 11 day initial test - write a restart at day 6. (file suffix base)

Half the number of tasks and threads for each component.

Do a 5 day restart test starting from restart at day 6. (file suffix rest)

Compare component history files '.base' and '.rest' at day 11.

This is just like an ERS test but the tasks/threading counts are modified on restart
PEA Single PE bit for bit test (default 5 days)
Do an initial run on 1 PE with mpi library. (file suffix: base)

Do the same run on 1 PE with mpiserial library. (file suffix: mpiserial)

Compare base and mpiserial.
PEM Modified PE counts for MPI(NTASKS) bit for bit test (default 5 days)
Do an initial run with default PE layout (file suffix: base)

Do another initial run with modified PE layout (NTASKS_XXX => NTASKS_XXX/2) (file suffix: modpes)

Compare base and modpes
PET Modified threading OPENMP bit for bit test (default 5 days)
Do an initial run where all components are threaded by default. (file suffix: base)

Do another initial run with NTHRDS=1 for all components. (file suffix: single_thread)

Compare base and single_thread.

PFS Performance test setup. (default 20 days)
ICP CICE performance test.
OCP POP performance test. (default 10 days)

MCC Multi-driver validation vs single-instance. (default 5 days)
NCK Multi-instance validation vs single instance - sequential PE for instances (default length)
Do an initial run test with NINST 1. (file suffix: base)

Do an initial run test with NINST 2. (file suffix: multiinst for both _0001 and _0002)

Compare base and _0001 and _0002.

REP Reproducibility: Two identical runs are bit for bit. (default 5 days)
SBN Smoke build-namelist test (just run preview_namelist and check_input_data).
SMS Smoke startup test (default 5 days)
Do a 5 day initial test. (file suffix: base)
SEQ Different sequencing bit for bit test. (default 10 days)
Do an initial run test with out-of-box PE-layout. (file suffix: base)

Do a second run where all root pes are at pe-0. (file suffix: seq)

Compare base and seq.
DAE Data assimilation test, default 1 day, two DA cycles, no data modification.
PRE Pause-resume test: by default a bit for bit test of pause-resume cycling.
Default 5 hours, five pause/resume cycles, no data modification.

======================= =====================================================================================


Each test run by create test will be put through the following mandatory phases:
Each test run by create_test includes the following mandatory steps:

* CREATE_NEWCASE: creating the create
* XML: xml changes to case based on test settings
Expand Down