Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add option to define test types in query_testlists #1930

Merged
merged 2 commits into from
Oct 2, 2017
Merged

Add option to define test types in query_testlists #1930

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5ba0c67
Add option to define test types in query_testlists
billsacks Sep 27, 2017
948a290
Clarify that HOMME tests only work with the ACME version of the atmos…
billsacks Sep 27, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions config/config_tests.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ The following are the test functionality categories:
11) other component-specific tests

NOTES:
- specifying the attribute INFRASTRUCTURE_TEST="TRUE" declares the given
test type to be an infrastructure-only test that is only meant to be
run by scripts_regression_tests.
- unless otherwise noted everything is run in one executable directory
- suffix: denotes the component history file suffixes that are added as part of the test
- IOP test is done along with regular tests - not as a separate test
Expand Down Expand Up @@ -336,7 +339,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
</test>

<test NAME="HOMME">
<DESC>Run homme tests.</DESC>
<DESC>Run homme tests. Only works with the ACME version of the atmosphere component.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
<STOP_N>11</STOP_N>
Expand All @@ -354,7 +357,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTBUILDFAIL">
<test NAME="TESTBUILDFAIL" INFRASTRUCTURE_TEST="TRUE">
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this was the right way to do this. I wanted some way to exclude these infrastructure-only tests, because otherwise they clutter the output. I'm open to suggestions of a better way to denote this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a decent approach to me. The other way would be check if the test is a subclass of FakeTest.

<DESC>For testing infra only. Insta-fail build step.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -363,7 +366,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTBUILDFAILEXC">
<test NAME="TESTBUILDFAILEXC" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Insta-fail build step by failing to init.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -372,7 +375,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTRUNFAIL">
<test NAME="TESTRUNFAIL" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Insta-fail run step.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -381,7 +384,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTRUNFAILEXC">
<test NAME="TESTRUNFAILEXC" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Insta-fail run step via exception.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -390,7 +393,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTRUNPASS">
<test NAME="TESTRUNPASS" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Insta-pass run step.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -399,7 +402,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTMEMLEAKFAIL">
<test NAME="TESTMEMLEAKFAIL" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Insta-fail memleak step.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -408,7 +411,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTMEMLEAKPASS">
<test NAME="TESTMEMLEAKPASS" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Insta-pass memleak step.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -417,7 +420,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTRUNDIFF">
<test NAME="TESTRUNDIFF" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Produces a canned hist file.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -426,7 +429,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTTESTDIFF">
<test NAME="TESTTESTDIFF" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Simulates internal test diff (non baseline)</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -435,7 +438,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="TESTRUNSLOWPASS">
<test NAME="TESTRUNSLOWPASS" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. After 5 minutes of sleep, pass run step.</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>ndays</STOP_OPTION>
Expand All @@ -444,7 +447,7 @@ NODEFAIL Tests restart upon detected node failure. Generates fake failu
<DOUT_S>FALSE</DOUT_S>
</test>

<test NAME="NODEFAIL">
<test NAME="NODEFAIL" INFRASTRUCTURE_TEST="TRUE">
<DESC>For testing infra only. Tests restart upon detected node failure</DESC>
<INFO_DBUG>1</INFO_DBUG>
<STOP_OPTION>nsteps</STOP_OPTION>
Expand Down
19 changes: 19 additions & 0 deletions scripts/lib/CIME/XML/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,22 @@ def get_test_node(self, testname):
node = self.get_node("test",{"NAME":testname})
logger.debug("Found {}".format(node.text))
return node

def print_values(self, skip_infrastructure_tests=True):
"""
Print each test type and its description.

If skip_infrastructure_tests is True, then this does not write
information for tests with the attribute
INFRASTRUCTURE_TEST="TRUE".
"""
all_tests = self.get_nodes(nodename="test")
for one_test in all_tests:
if skip_infrastructure_tests:
infrastructure_test = one_test.get("INFRASTRUCTURE_TEST")
if (infrastructure_test is not None and
infrastructure_test.upper() == "TRUE"):
continue
name = one_test.get("NAME")
desc = one_test.find("DESC").text
print("{}: {}".format(name, desc))
47 changes: 42 additions & 5 deletions scripts/query_testlists
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Script to query xml test lists, displaying all tests in human-readable form.

Usage:
./query_testlists [--show-options]
./query_testlists [--show-options] [--define-testtypes]
Display a list of tests
./query_testlists --count
Count tests by category/machine/compiler
Expand All @@ -16,6 +16,7 @@ Usage:
from __future__ import print_function
from Tools.standard_script_setup import *
from CIME.test_utils import get_tests_from_xml, test_to_string
from CIME.XML.tests import Tests
from CIME.utils import expect

logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -46,6 +47,11 @@ def parse_command_line(args, description):
"(wallclock time, memory leak tolerance, etc.). "
"(Has no effect with --list or --count options.)")

parser.add_argument("--define-testtypes", action="store_true",
help="At the top of the list of tests, define "
"all of the possible test types. "
"(Has no effect with --list or --count options.)")

parser.add_argument("--xml-category",
help="Only include tests in this category; default is all categories")

Expand All @@ -61,14 +67,37 @@ def parse_command_line(args, description):

args = CIME.utils.parse_args_and_handle_standard_logging_options(args, parser)

expect(not(args.count and args.list_type),
"Cannot specify both --count and --list arguments.")
_check_argument_compatibility(args)

if args.list_type:
_process_list_type(args)

return args

###############################################################################
def _check_argument_compatibility(args):
###############################################################################
"""Ensures there are no incompatible arguments

If incompatible arguments are found, aborts with a helpful error
message.
"""

expect(not(args.count and args.list_type),
"Cannot specify both --count and --list arguments.")

if args.count:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be cleaner to achieve with ArgParse's mutually exclusive group (see case.build).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a useful-looking feature. But I'm struggling with how to do that here. I see how I can replace this with a mutually exclusive group:

    expect(not(args.count and args.list_type),
           "Cannot specify both --count and --list arguments.")

But I can't see how to express the rest of the logic with that mechanism: Although it's not allowed to combine either --show-options or --define-testtypes with either --list or --count, it is allowed to combine --show-options with --define-testtypes. Can you see how to express this with a mutually exclusive group?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, if it's not obvious then don't worry about it.

expect(not args.show_options,
"--show-options is incompatible with --count")
expect(not args.define_testtypes,
"--define-testtypes is incompatible with --count")

if args.list_type:
expect(not args.show_options,
"--show-options is incompatible with --list")
expect(not args.define_testtypes,
"--define-testtypes is incompatible with --list")

###############################################################################
def _process_list_type(args):
###############################################################################
Expand All @@ -87,7 +116,7 @@ def _process_list_type(args):
args.list_type = 'compiler'

###############################################################################
def print_test_data(test_data, show_options):
def print_test_data(test_data, show_options, define_testtypes):
###############################################################################
"""
Args:
Expand All @@ -96,6 +125,14 @@ def print_test_data(test_data, show_options):
- category: test category
"""

if define_testtypes:
print("#"*72)
print("Test types")
print("----------")
test_definitions = Tests()
test_definitions.print_values(skip_infrastructure_tests=True)
print("#"*72)

categories = sorted(set([item['category'] for item in test_data]))
max_category_len = max([len(category) for category in categories])
max_test_len = max([len(item['name']) for item in test_data])
Expand Down Expand Up @@ -178,7 +215,7 @@ def _main_func(description):
elif args.list_type:
list_test_data(test_data, args.list_type)
else:
print_test_data(test_data, args.show_options)
print_test_data(test_data, args.show_options, args.define_testtypes)

if __name__ == "__main__":
_main_func(__doc__)
8 changes: 6 additions & 2 deletions scripts/tests/scripts_regression_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2279,8 +2279,8 @@ def _run_and_assert_query_testlist(self, extra_args=""):
files = Files()
testlist_drv = files.get_value("TESTS_SPEC_FILE", {"component":"drv"})

run_cmd_assert_result(self, "%s/query_testlists --xml-testlist %s %s"%
(SCRIPT_DIR, testlist_drv, extra_args))
run_cmd_assert_result(self, "{}/query_testlists --xml-testlist {} {}".format(
SCRIPT_DIR, testlist_drv, extra_args))

def test_query_testlists_runs(self):
"""Make sure that query_testlists runs successfully
Expand All @@ -2291,6 +2291,10 @@ def test_query_testlists_runs(self):
"""
self._run_and_assert_query_testlist(extra_args="--show-options")

def test_query_testlists_define_testtypes_runs(self):
"""Make sure that query_testlists runs successfully with the --define-testtypes argument"""
self._run_and_assert_query_testlist(extra_args="--define-testtypes")

def test_query_testlists_count_runs(self):
"""Make sure that query_testlists runs successfully with the --count argument"""
self._run_and_assert_query_testlist(extra_args="--count")
Expand Down