Automatically add TEST_SUITE labels to discovered tests

[shivupa]

The PR is against dev, and I have read the CONTRIBUTING.md. I understand that usually a issue is opened for discussion, but I figured the draft PR would illustrate the changes better.

Description

What:
When using doctest_discover_tests, the LABELS property is set for the generated test based on the TEST_SUITE.

Why:
In a downstream project using CMake, I found it useful to group each TEST_CASE in a TEST_SUITE. I found myself wanting to selectively run tests based on what TEST_SUITE they belong to, e.g, ctest -L TEST_SUITE_NAME. The test discovery returns the list of test, but modifying each one would be tedious.

GitHub Issues

#302 is related, but the discussion doesn't seem to indicate that this is the desired solution.

[onqtam]

Hi,

In a world where we had native XML parsing support in CMake it would have been better to switch to parsing the XML output in doctestAddTests.cmake instead of the console output - there you'll be able to get all test cases and their test suites (as an attribute to the test cases - see here) with a single execute_process call (as compared to the current approach - with one for each test case). But we don't live in that world...

My concern with the current approach is that if there are thousands of test cases there would be thousands of invocations of the binary just to discover the test suite of the test cases. Perhaps this labeling shouldn't be on by default and should be guarded in an if()...

I'll leave this PR open for now (hopefully will get more feedback from others as I'm not a user of these CMake scripts) but you should be free to use these changes locally for now.

[shivupa]

Hi,

In a world where we had native XML parsing support in CMake it would have been better to switch to parsing the XML output in doctestAddTests.cmake instead of the console output - there you'll be able to get all test cases and their test suites (as an attribute to the test cases - see here) with a single execute_process call (as compared to the current approach - with one for each test case). But we don't live in that world...

My concern with the current approach is that if there are thousands of test cases there would be thousands of invocations of the binary just to discover the test suite of the test cases. Perhaps this labeling shouldn't be on by default and should be guarded in an if()...

Sorry for the delay I had some deadlines. I also was worried about this. In my use case there are only a few (<100) tests so there is no perceptible overhead, but this clearly could vary. I hope this method can serve as a stand in until there is a cleaner XML based method. I wrapped this in an if() so now doctest_discover_tests(main_test ADD_LABELS 1) will add the labels.

I'll leave this PR open for now (hopefully will get more feedback from others as I'm not a user of these CMake scripts) but you should be free to use these changes locally for now.

I'm fine with leaving this open for as long as you'd like. I am working from the branch for the time being so it doesn't hold me up!

[onqtam]

thanks for the contribution!

[jecassis]

Seeing a failure in test discovery when upgrading to 2.4.6 due to this feature when labels are not explicitly set. Example:

doctest_discover_tests(MyTests)

This is the error:

 CMake Error at .../scripts/cmake/doctestAddTests.cmake:60 (if):
    if given arguments:
  
      "EQUAL" "1"
  
    Unknown arguments specified

To workaround, need to explicitly set ADD_LABELS to 0:

doctest_discover_tests(MyTests ADD_LABELS 0)

Please take care of the scenario where ADD_LABELS is an empty string so that the original and non-argumented test discovery directive maintains its original behavior.

[shivupa]

doctest_discover_tests(MyTests)

Hi Thanks for reporting this I opened #489. I'll update there once the issue is fixed.

[shivupa]

Seeing a failure in test discovery when upgrading to 2.4.6 due to this feature when labels are not explicitly set. Example:

doctest_discover_tests(MyTests)

This is the error:

 CMake Error at .../scripts/cmake/doctestAddTests.cmake:60 (if):
    if given arguments:
  
      "EQUAL" "1"
  
    Unknown arguments specified

To workaround, need to explicitly set ADD_LABELS to 0:

doctest_discover_tests(MyTests ADD_LABELS 0)

Please take care of the scenario where ADD_LABELS is an empty string so that the original and non-argumented test discovery directive maintains its original behavior.

#490 fixes this.

[jharmer95]

This is still broken, it can be fixed by changing it to the CMake-preferred syntax if(${add_labels}) rather than compare a string to 1. See PR #527 (from issue #524) for the fix