diff --git a/doc/source/users_guide/testing.rst b/doc/source/users_guide/testing.rst index e7f8b0cec72..de3d6ecf4cb 100644 --- a/doc/source/users_guide/testing.rst +++ b/doc/source/users_guide/testing.rst @@ -4,14 +4,11 @@ Testing with create_test ************************** -.. todo :: - add list and descriptions on functionality tests (ERS, ERI, ect) - -create_test is the tool we use to test cime-driven models. It can be used as a easy way to -run a single basic test or can be used to run an entire suite of tests. It has parallelism -built-in on many levels to make full use of the underlying system. create_test is the driver -behind the automated nightly testing of cime-driven models. +**create_test** is the tool we use to test cime-driven models. It can be used as an easy way to +run a single basic test or an entire suite of tests. create_test runs a test suite in parallel +for improved performance. create_test is the driver behind the automated nightly testing of +cime-driven models. Tests come in form:: @@ -22,10 +19,125 @@ A common test type would be ERS which stands for exact restart, which means you grid and compset to see if they reproduce the same result upon restart as they would if no restart took place. GRID and COMPSET are self-explanatory. MACHINE_COMPILER is optional; create_test will probe the underlying machine and use the default compiler for that machine if this value is not -supplied. TESTMODS is also optional and can be used to make some env XML changes prior to doing -the test. +supplied. TESTMODS are optional and can be used to make some env XML changes prior to running +a test. + +======================= ===================================================================================== +TESTTYPE Description +======================= ===================================================================================== + ERS Exact restart from startup (default 6 days + 5 days) + Do an 11 day initial test - write a restart at day 6. (file suffix: base) + + Do a 5 day restart test, starting from restart at day 6. (file suffix: rest) + + Compare component history files '.base' and '.rest' at day 11. + ERS2 Exact restart from startup (default 6 days + 5 days). + Do an 11 day initial test without making restarts. (file suffix: base) + + Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (file suffix: rest) + + Compare component history files ".base" and ".rest" at day 11. + ERT Exact restart from startup, default 2 month + 1 month (ERS with info DBUG = 1). + IRT Exact restart from startup, (default 4 days + 7 days) with restart from interim file. + ERIO Exact restart from startup with different PIO methods, (default 6 days + 5 days). + ERR Exact restart from startup with resubmit, (default 4 days + 3 days). + ERRI Exact restart from startup with resubmit, (default 4 days + 3 days). Tests incomplete logs option for st_archive. + ERI hybrid/branch/exact restart test, default (by default STOP_N is 22 days) + + (1) ref1case + + Do an initial run for 3 days writing restarts at day 3. + + ref1case is a clone of the main case. + + Short term archiving is on. + + (2) ref2case (Suffix hybrid) + + Do a hybrid run for default 19 days running with ref1 restarts from day 3, + + and writing restarts at day 10. + + ref2case is a clone of the main case. + + Short term archiving is on. + + (3) case + + Do a branch run, starting from restarts written in ref2case, + + for 9 days and writing restarts at day 5. + + Short term archiving is off. + + (4) case (Suffix base) + + Do a restart run from the branch run restarts for 4 days. + + Compare component history files '.base' and '.hybrid' at day 19. + + Short term archiving is off. + ERP PES counts hybrid (OPENMP/MPI) restart bit for bit test from startup, (default 6 days + 5 days). + Initial PES set up out of the box + + Do an 11 day initial test - write a restart at day 6. (file suffix base) + + Half the number of tasks and threads for each component. + + Do a 5 day restart test starting from restart at day 6. (file suffix rest) + + Compare component history files '.base' and '.rest' at day 11. + + This is just like an ERS test but the tasks/threading counts are modified on restart + PEA Single PE bit for bit test (default 5 days) + Do an initial run on 1 PE with mpi library. (file suffix: base) + + Do the same run on 1 PE with mpiserial library. (file suffix: mpiserial) + + Compare base and mpiserial. + PEM Modified PE counts for MPI(NTASKS) bit for bit test (default 5 days) + Do an initial run with default PE layout (file suffix: base) + + Do another initial run with modified PE layout (NTASKS_XXX => NTASKS_XXX/2) (file suffix: modpes) + + Compare base and modpes + PET Modified threading OPENMP bit for bit test (default 5 days) + Do an initial run where all components are threaded by default. (file suffix: base) + + Do another initial run with NTHRDS=1 for all components. (file suffix: single_thread) + + Compare base and single_thread. + + PFS Performance test setup. (default 20 days) + ICP CICE performance test. + OCP POP performance test. (default 10 days) + + MCC Multi-driver validation vs single-instance. (default 5 days) + NCK Multi-instance validation vs single instance - sequential PE for instances (default length) + Do an initial run test with NINST 1. (file suffix: base) + + Do an initial run test with NINST 2. (file suffix: multiinst for both _0001 and _0002) + + Compare base and _0001 and _0002. + + REP Reproducibility: Two identical runs are bit for bit. (default 5 days) + SBN Smoke build-namelist test (just run preview_namelist and check_input_data). + SMS Smoke startup test (default 5 days) + Do a 5 day initial test. (file suffix: base) + SEQ Different sequencing bit for bit test. (default 10 days) + Do an initial run test with out-of-box PE-layout. (file suffix: base) + + Do a second run where all root pes are at pe-0. (file suffix: seq) + + Compare base and seq. + DAE Data assimilation test, default 1 day, two DA cycles, no data modification. + PRE Pause-resume test: by default a bit for bit test of pause-resume cycling. + Default 5 hours, five pause/resume cycles, no data modification. + +======================= ===================================================================================== + -Each test run by create test will be put through the following mandatory phases: +Each test run by create_test includes the following mandatory steps: * CREATE_NEWCASE: creating the create * XML: xml changes to case based on test settings