From aae8c7841184de62718d7a10a6566d89727886e6 Mon Sep 17 00:00:00 2001 From: Chris Fischer Date: Tue, 14 Nov 2017 14:41:10 -0700 Subject: [PATCH 1/4] Add list of test types to users guide. Test suite: Test baseline: Test namelist changes: Test status: [bit for bit, roundoff, climate changing] Fixes [CIME Github issue #] User interface changes?: Update gh-pages html (Y/N)?: Y Code review: --- doc/source/users_guide/testing.rst | 125 +++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/doc/source/users_guide/testing.rst b/doc/source/users_guide/testing.rst index e7f8b0cec72..b5a377e896c 100644 --- a/doc/source/users_guide/testing.rst +++ b/doc/source/users_guide/testing.rst @@ -8,6 +8,10 @@ Testing with create_test add list and descriptions on functionality tests (ERS, ERI, ect) + + + + create_test is the tool we use to test cime-driven models. It can be used as a easy way to run a single basic test or can be used to run an entire suite of tests. It has parallelism built-in on many levels to make full use of the underlying system. create_test is the driver @@ -25,6 +29,127 @@ the underlying machine and use the default compiler for that machine if this val supplied. TESTMODS is also optional and can be used to make some env XML changes prior to doing the test. +======================= ===================================================================================== +TESTTYPE Description +======================= ===================================================================================== + DAE Data assimilation test, default 1 day, two DA cycles, no data modification. + ERI hybrid/branch/exact restart test, default (by default STOP_N is 22 days) + + (1) ref1case + + Do an initial run for 3 days writing restarts at day 3. + + ref1case is a clone of the main case. + + Short term archiving is on. + + (2) ref2case (Suffix hybrid) + + Do a hybrid run for default 19 days running with ref1 restarts from day 3, + + and writing restarts at day 10. + + ref2case is a clone of the main case. + + Short term archiving is on. + + (3) case + + Do a branch run, starting from restarts written in ref2case, + + for 9 days and writing restarts at day 5. + + Short term archiving is off. + + (4) case (Suffix base) + + Do a restart run from the branch run restarts for 4 days. + + Compare component history files '.base' and '.hybrid' at day 19. + + Short term archiving is off. + + ERP PES counts hybrid (OPENMP/MPI) restart bit for bit test from startup, (default 6 days + 5 days). + Initial PES set up out of the box + + Do an 11 day initial test - write a restart at day 6. (suffix base) + + Half the number of tasks and threads for each component. + + Do a 5 day restart test starting from restart at day 6. (suffix rest) + + Compare component history files '.base' and '.rest' at day 11. + + This is just like an ERS test but the tasks/threading counts are modified on restart + + ERS Exact restart from startup (default 6 days + 5 days) + Do an 11 day initial test - write a restart at day 6. (suffix: base) + + Do a 5 day restart test, starting from restart at day 6. (suffix: rest) + + Compare component history files '.base' and '.rest' at day 11. + + ERS2 Exact restart from startup (default 6 days + 5 days). + Do an 11 day initial test without making restarts. (suffix: base) + + Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (suffix: rest) + + Compare component history files ".base" and ".rest" at day 11. + + IRT Exact restart from startup, (default 4 days + 7 days) with restart from interim file. + ERIO Exact restart from startup with different PIO methods, (default 6 days + 5 days). + ERR Exact restart from startup with resubmit, (default 4 days + 3 days). + ERRI Exact restart from startup with resubmit, (default 4 days + 3 days). Tests incomplete logs option for st_archive. + ERT Exact restart from startup, default 2 month + 1 month (ERS with info DBUG = 1). + PRE Pause-resume test: by default a bit for bit test of pause-resume cycling. + Default 5 hours, five pause/resume cycles, no data modification. + + ICP CICE performance test. + PEA Single PE bit for bit test (default 5 days) + Do an initial run on 1 PE with mpi library. (suffix: base) + + Do the same run on 1 PE with mpiserial library. (suffix: mpiserial) + + Compare base and mpiserial. + + PEM Modified PE counts for MPI(NTASKS) bit for bit test (default 5 days) + Do an initial run with default PE layout (suffix: base) + + Do another initial run with modified PE layout (NTASKS_XXX => NTASKS_XXX/2) (suffix: modpes) + + Compare base and modpes + + PET Modified threading OPENMP bit for bit test (default 5 days) + Do an initial run where all components are threaded by default. (suffix: base) + + Do another initial run with NTHRDS=1 for all components. (suffix: single_thread) + + Compare base and single_thread. + + PFS Performance test setup. (default 20 days) + MCC Multi-driver validation vs single-instance. (default 5 days) + NCK Multi-instance validation vs single instance - sequential PE for instances (default length) + Do an initial run test with NINST 1. (suffix: base) + + Do an initial run test with NINST 2. (suffix: multiinst for both _0001 and _0002) + + Compare base and _0001 and _0002. + + OCP POP performance test. (default 10 days) + REP Reproducibility: Two identical runs are bit for bit. (default 5 days) + SBN Smoke build-namelist test (just run preview_namelist and check_input_data). + SEQ Different sequencing bit for bit test. (default 10 days) + Do an initial run test with out-of-box PE-layout. (suffix: base) + + Do a second run where all root pes are at pe-0. (suffix: seq) + + Compare base and seq. + + SMS Smoke startup test (default 5 days) + Do a 5 day initial test. (suffix: base) +======================= ===================================================================================== + + Each test run by create test will be put through the following mandatory phases: * CREATE_NEWCASE: creating the create From 2f7b9487e0386909117cd31a455df100a953e3ec Mon Sep 17 00:00:00 2001 From: Chris Fischer Date: Tue, 14 Nov 2017 14:46:46 -0700 Subject: [PATCH 2/4] Add a list of the different test types to section 4 of the users guide. Test suite: Test baseline: Test namelist changes: Test status: [bit for bit, roundoff, climate changing] Fixes [CIME Github issue #] User interface changes?: Update gh-pages html (Y/N)?: Y Code review: --- doc/source/users_guide/testing.rst | 7 ------- 1 file changed, 7 deletions(-) diff --git a/doc/source/users_guide/testing.rst b/doc/source/users_guide/testing.rst index b5a377e896c..7e38c6b2262 100644 --- a/doc/source/users_guide/testing.rst +++ b/doc/source/users_guide/testing.rst @@ -4,13 +4,6 @@ Testing with create_test ************************** -.. todo :: - - add list and descriptions on functionality tests (ERS, ERI, ect) - - - - create_test is the tool we use to test cime-driven models. It can be used as a easy way to run a single basic test or can be used to run an entire suite of tests. It has parallelism From 42e7527b77dd16797f55cf38f9851e498e30af9e Mon Sep 17 00:00:00 2001 From: Chris Fischer Date: Wed, 15 Nov 2017 11:19:11 -0700 Subject: [PATCH 3/4] Test suite: Test baseline: Test namelist changes: Test status: [bit for bit, roundoff, climate changing] Fixes [CIME Github issue #] User interface changes?: Update gh-pages html (Y/N)?: Y Code review: --- doc/source/users_guide/testing.rst | 56 +++++++++++++----------------- 1 file changed, 25 insertions(+), 31 deletions(-) diff --git a/doc/source/users_guide/testing.rst b/doc/source/users_guide/testing.rst index 7e38c6b2262..21e58a5c75c 100644 --- a/doc/source/users_guide/testing.rst +++ b/doc/source/users_guide/testing.rst @@ -25,7 +25,23 @@ the test. ======================= ===================================================================================== TESTTYPE Description ======================= ===================================================================================== - DAE Data assimilation test, default 1 day, two DA cycles, no data modification. + ERS Exact restart from startup (default 6 days + 5 days) + Do an 11 day initial test - write a restart at day 6. (suffix: base) + + Do a 5 day restart test, starting from restart at day 6. (suffix: rest) + + Compare component history files '.base' and '.rest' at day 11. + ERS2 Exact restart from startup (default 6 days + 5 days). + Do an 11 day initial test without making restarts. (suffix: base) + + Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (suffix: rest) + + Compare component history files ".base" and ".rest" at day 11. + ERT Exact restart from startup, default 2 month + 1 month (ERS with info DBUG = 1). + IRT Exact restart from startup, (default 4 days + 7 days) with restart from interim file. + ERIO Exact restart from startup with different PIO methods, (default 6 days + 5 days). + ERR Exact restart from startup with resubmit, (default 4 days + 3 days). + ERRI Exact restart from startup with resubmit, (default 4 days + 3 days). Tests incomplete logs option for st_archive. ERI hybrid/branch/exact restart test, default (by default STOP_N is 22 days) (1) ref1case @@ -61,7 +77,6 @@ TESTTYPE Description Compare component history files '.base' and '.hybrid' at day 19. Short term archiving is off. - ERP PES counts hybrid (OPENMP/MPI) restart bit for bit test from startup, (default 6 days + 5 days). Initial PES set up out of the box @@ -74,44 +89,18 @@ TESTTYPE Description Compare component history files '.base' and '.rest' at day 11. This is just like an ERS test but the tasks/threading counts are modified on restart - - ERS Exact restart from startup (default 6 days + 5 days) - Do an 11 day initial test - write a restart at day 6. (suffix: base) - - Do a 5 day restart test, starting from restart at day 6. (suffix: rest) - - Compare component history files '.base' and '.rest' at day 11. - - ERS2 Exact restart from startup (default 6 days + 5 days). - Do an 11 day initial test without making restarts. (suffix: base) - - Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (suffix: rest) - - Compare component history files ".base" and ".rest" at day 11. - - IRT Exact restart from startup, (default 4 days + 7 days) with restart from interim file. - ERIO Exact restart from startup with different PIO methods, (default 6 days + 5 days). - ERR Exact restart from startup with resubmit, (default 4 days + 3 days). - ERRI Exact restart from startup with resubmit, (default 4 days + 3 days). Tests incomplete logs option for st_archive. - ERT Exact restart from startup, default 2 month + 1 month (ERS with info DBUG = 1). - PRE Pause-resume test: by default a bit for bit test of pause-resume cycling. - Default 5 hours, five pause/resume cycles, no data modification. - - ICP CICE performance test. PEA Single PE bit for bit test (default 5 days) Do an initial run on 1 PE with mpi library. (suffix: base) Do the same run on 1 PE with mpiserial library. (suffix: mpiserial) Compare base and mpiserial. - PEM Modified PE counts for MPI(NTASKS) bit for bit test (default 5 days) Do an initial run with default PE layout (suffix: base) Do another initial run with modified PE layout (NTASKS_XXX => NTASKS_XXX/2) (suffix: modpes) Compare base and modpes - PET Modified threading OPENMP bit for bit test (default 5 days) Do an initial run where all components are threaded by default. (suffix: base) @@ -120,6 +109,9 @@ TESTTYPE Description Compare base and single_thread. PFS Performance test setup. (default 20 days) + ICP CICE performance test. + OCP POP performance test. (default 10 days) + MCC Multi-driver validation vs single-instance. (default 5 days) NCK Multi-instance validation vs single instance - sequential PE for instances (default length) Do an initial run test with NINST 1. (suffix: base) @@ -128,18 +120,20 @@ TESTTYPE Description Compare base and _0001 and _0002. - OCP POP performance test. (default 10 days) REP Reproducibility: Two identical runs are bit for bit. (default 5 days) SBN Smoke build-namelist test (just run preview_namelist and check_input_data). + SMS Smoke startup test (default 5 days) + Do a 5 day initial test. (suffix: base) SEQ Different sequencing bit for bit test. (default 10 days) Do an initial run test with out-of-box PE-layout. (suffix: base) Do a second run where all root pes are at pe-0. (suffix: seq) Compare base and seq. + DAE Data assimilation test, default 1 day, two DA cycles, no data modification. + PRE Pause-resume test: by default a bit for bit test of pause-resume cycling. + Default 5 hours, five pause/resume cycles, no data modification. - SMS Smoke startup test (default 5 days) - Do a 5 day initial test. (suffix: base) ======================= ===================================================================================== From fb7dc2d875bcc4ba5e105783178fb23444abdcc1 Mon Sep 17 00:00:00 2001 From: Chris Fischer Date: Wed, 15 Nov 2017 14:49:22 -0700 Subject: [PATCH 4/4] Changes requested by bertinia Test suite: Test baseline: Test namelist changes: Test status: [bit for bit, roundoff, climate changing] Fixes [CIME Github issue #] User interface changes?: Update gh-pages html (Y/N)?: Y Code review: --- doc/source/users_guide/testing.rst | 48 +++++++++++++++--------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/doc/source/users_guide/testing.rst b/doc/source/users_guide/testing.rst index 21e58a5c75c..de3d6ecf4cb 100644 --- a/doc/source/users_guide/testing.rst +++ b/doc/source/users_guide/testing.rst @@ -5,10 +5,10 @@ Testing with create_test ************************** -create_test is the tool we use to test cime-driven models. It can be used as a easy way to -run a single basic test or can be used to run an entire suite of tests. It has parallelism -built-in on many levels to make full use of the underlying system. create_test is the driver -behind the automated nightly testing of cime-driven models. +**create_test** is the tool we use to test cime-driven models. It can be used as an easy way to +run a single basic test or an entire suite of tests. create_test runs a test suite in parallel +for improved performance. create_test is the driver behind the automated nightly testing of +cime-driven models. Tests come in form:: @@ -19,22 +19,22 @@ A common test type would be ERS which stands for exact restart, which means you grid and compset to see if they reproduce the same result upon restart as they would if no restart took place. GRID and COMPSET are self-explanatory. MACHINE_COMPILER is optional; create_test will probe the underlying machine and use the default compiler for that machine if this value is not -supplied. TESTMODS is also optional and can be used to make some env XML changes prior to doing -the test. +supplied. TESTMODS are optional and can be used to make some env XML changes prior to running +a test. ======================= ===================================================================================== TESTTYPE Description ======================= ===================================================================================== ERS Exact restart from startup (default 6 days + 5 days) - Do an 11 day initial test - write a restart at day 6. (suffix: base) + Do an 11 day initial test - write a restart at day 6. (file suffix: base) - Do a 5 day restart test, starting from restart at day 6. (suffix: rest) + Do a 5 day restart test, starting from restart at day 6. (file suffix: rest) Compare component history files '.base' and '.rest' at day 11. ERS2 Exact restart from startup (default 6 days + 5 days). - Do an 11 day initial test without making restarts. (suffix: base) + Do an 11 day initial test without making restarts. (file suffix: base) - Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (suffix: rest) + Do an 11 day restart test stopping at day 6 with a restart, then resuming from restart at day 6. (file suffix: rest) Compare component history files ".base" and ".rest" at day 11. ERT Exact restart from startup, default 2 month + 1 month (ERS with info DBUG = 1). @@ -80,31 +80,31 @@ TESTTYPE Description ERP PES counts hybrid (OPENMP/MPI) restart bit for bit test from startup, (default 6 days + 5 days). Initial PES set up out of the box - Do an 11 day initial test - write a restart at day 6. (suffix base) + Do an 11 day initial test - write a restart at day 6. (file suffix base) Half the number of tasks and threads for each component. - Do a 5 day restart test starting from restart at day 6. (suffix rest) + Do a 5 day restart test starting from restart at day 6. (file suffix rest) Compare component history files '.base' and '.rest' at day 11. This is just like an ERS test but the tasks/threading counts are modified on restart PEA Single PE bit for bit test (default 5 days) - Do an initial run on 1 PE with mpi library. (suffix: base) + Do an initial run on 1 PE with mpi library. (file suffix: base) - Do the same run on 1 PE with mpiserial library. (suffix: mpiserial) + Do the same run on 1 PE with mpiserial library. (file suffix: mpiserial) Compare base and mpiserial. PEM Modified PE counts for MPI(NTASKS) bit for bit test (default 5 days) - Do an initial run with default PE layout (suffix: base) + Do an initial run with default PE layout (file suffix: base) - Do another initial run with modified PE layout (NTASKS_XXX => NTASKS_XXX/2) (suffix: modpes) + Do another initial run with modified PE layout (NTASKS_XXX => NTASKS_XXX/2) (file suffix: modpes) Compare base and modpes PET Modified threading OPENMP bit for bit test (default 5 days) - Do an initial run where all components are threaded by default. (suffix: base) + Do an initial run where all components are threaded by default. (file suffix: base) - Do another initial run with NTHRDS=1 for all components. (suffix: single_thread) + Do another initial run with NTHRDS=1 for all components. (file suffix: single_thread) Compare base and single_thread. @@ -114,20 +114,20 @@ TESTTYPE Description MCC Multi-driver validation vs single-instance. (default 5 days) NCK Multi-instance validation vs single instance - sequential PE for instances (default length) - Do an initial run test with NINST 1. (suffix: base) + Do an initial run test with NINST 1. (file suffix: base) - Do an initial run test with NINST 2. (suffix: multiinst for both _0001 and _0002) + Do an initial run test with NINST 2. (file suffix: multiinst for both _0001 and _0002) Compare base and _0001 and _0002. REP Reproducibility: Two identical runs are bit for bit. (default 5 days) SBN Smoke build-namelist test (just run preview_namelist and check_input_data). SMS Smoke startup test (default 5 days) - Do a 5 day initial test. (suffix: base) + Do a 5 day initial test. (file suffix: base) SEQ Different sequencing bit for bit test. (default 10 days) - Do an initial run test with out-of-box PE-layout. (suffix: base) + Do an initial run test with out-of-box PE-layout. (file suffix: base) - Do a second run where all root pes are at pe-0. (suffix: seq) + Do a second run where all root pes are at pe-0. (file suffix: seq) Compare base and seq. DAE Data assimilation test, default 1 day, two DA cycles, no data modification. @@ -137,7 +137,7 @@ TESTTYPE Description ======================= ===================================================================================== -Each test run by create test will be put through the following mandatory phases: +Each test run by create_test includes the following mandatory steps: * CREATE_NEWCASE: creating the create * XML: xml changes to case based on test settings