Add option to define test types in query_testlists

[billsacks]

As far as I could tell, we didn't have any way to query the different
test types and their meaning (ERS, ERP, etc.). This PR adds this
capability to query_testlists.

I'm not sure if query_testlists is exactly the right place for this, but
I couldn't think of anywhere better. I thought about query_config, but I
didn't want to clutter that tool with this option that is more for
developers than users. So I have implemented this as an optional header
that can be output with query_testlists. I'm open to suggestions for a
better user interface.

The output looks like this:

$ ./query_testlists --define-testtypes
########################################################################
Test types
----------
DAE: data assimilation test, default 1 day, two DA cycles, no data modification
ERI: hybrid/branch/exact restart test, default 3+19/10+9/5+4 days
ERP: pes counts hybrid (open-MP/MPI) restart bfb test from startup, default 6 days + 5 days
ERS: exact restart from startup, default 6 days + 5 days
ERS2: exact restart from startup, default 6 days + 5 days
IRT: exact restart from startup, default 4 days + 7 days with restart from interim file
ERIO: exact restart from startup with different PIO methods, default 6 days + 5 days
ERR: exact restart from startup with resubmit, default 4 days + 3 days
ERRI: exact restart from startup with resubmit, default 4 days + 3 days. Tests incomplete logs option for st_archive
ERT: exact restart from startup, default 2 month + 1 month (ERS with info dbug = 1)
HOMME: Run homme tests.
PRE: pause/resume test, default 5 hours, five pause/resume cycles, no data modification
ICP: cice performance test
PEA: single pe bfb test (default length)
PEM: pes counts mpi bfb test (seq tests; default length)
PET: openmp bfb test (seq tests; default length)
PFS: performance test setup
MCC: multi-driver validation vs single-instance (default length)
NCK: multi-instance validation vs single instance (default length)
NCR: multi-instance validation sequential vs concurrent (default length)
OCP: pop performance test
REP: reproducibility test: do two runs give the same answers?
SBN: smoke build-namelist test (just run preview_namelist and check_input_data)
SEQ: sequencing bfb test (10 day seq,conc tests)
SMS: smoke startup test (default length)
########################################################################
jim        : SMS.f19_g16.X.cheyenne_intel
jim        : SMS.f19_g16.X.cheyenne_gnu
jim        : SMS.f19_g16.X.cheyenne_pgi
pauseresume: PRE.f09_f09.ADESP.hobart_nag
pauseresume: PRE.f09_f09.ADESP_TEST.hobart_nag
prealpha   : ERS_C3.f19_g17_rx1.A.cheyenne_pgi

(and then a lot more tests following that, since this lists out all the
tests defined in cime).

Test suite: scripts_regression_tests on cheyenne
Test baseline: N/A
Test namelist changes: none
Test status: bit for bit

Fixes #1929

User interface changes?: adds --define-testtypes option to query_testlists

Update gh-pages html (Y/N)?: N

Code review:

[billsacks]

I don't know if this was the right way to do this. I wanted some way to exclude these infrastructure-only tests, because otherwise they clutter the output. I'm open to suggestions of a better way to denote this.

[jgfouca]

Seems like a decent approach to me. The other way would be check if the test is a subclass of FakeTest.

[jedwards4b]

Nearly all of the tests except "homme" work with stand alone cime. "homme" in particular only works with the acme version of the atmosphere component. Can we make that clear in this output?

[jgfouca]

See inline.

[jgfouca]

Seems like a decent approach to me. The other way would be check if the test is a subclass of FakeTest.

[jgfouca]

This might be cleaner to achieve with ArgParse's mutually exclusive group (see case.build).

[billsacks]

That's a useful-looking feature. But I'm struggling with how to do that here. I see how I can replace this with a mutually exclusive group:

    expect(not(args.count and args.list_type),
           "Cannot specify both --count and --list arguments.")

But I can't see how to express the rest of the logic with that mechanism: Although it's not allowed to combine either --show-options or --define-testtypes with either --list or --count, it is allowed to combine --show-options with --define-testtypes. Can you see how to express this with a mutually exclusive group?

[jgfouca]

OK, if it's not obvious then don't worry about it.

[billsacks]

Nearly all of the tests except "homme" work with stand alone cime. "homme" in particular only works with the acme version of the atmosphere component. Can we make that clear in this output?

Would the right thing to do here be to move the homme test into ACME's CAM's cime_config? In the meantime, I'll just edit the DESC string for that test to add, "Only works with the ACME version of the atmosphere component."

[billsacks]

I just rediscovered #1685 after submitting this PR. I still feel like it's useful to have this information available via query_testlists, but maybe we want it in create_test, too? @jedwards4b @jgfouca @fischer-ncar if it's okay with you, I'll leave the extension to create_test up to @fischer-ncar as part of resolving #1685 . You should be able to reuse much of what I did here.