Many tests are being skipped by the CI

[RaulPPelaez]

The CI runs this to test:

ctest --verbose --exclude-regex TestCuda

Running this on my machine results in 11 tests. But the CI only runs 3.
I cannot find out why.

[mikemhenry]

It looks like we run 5 on the GPU CI:
https://github.com/openmm/NNPOps/actions/runs/5063145679/jobs/9089458222#step:9:75
which just calls ctest --verbose

[RaulPPelaez]

I think I know whats going on:

NNPOps/CMakeLists.txt

Lines 60 to 77 in 054d487

	add_custom_target(copy_test ALL
	COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/test
	COMMAND ${CMAKE_COMMAND} -E copy
	${CMAKE_SOURCE_DIR}/src/pytorch/Test*.py
	${CMAKE_SOURCE_DIR}/src/pytorch/neighbors/Test*.py
	${CMAKE_SOURCE_DIR}/src/pytorch/neighbors/getNeighborPairs.py
	${CMAKE_BINARY_DIR}/test
	COMMAND ${CMAKE_COMMAND} -E copy_directory
	${CMAKE_SOURCE_DIR}/src/pytorch/molecules
	${CMAKE_BINARY_DIR}/test/molecules
	)
	# Add tests for all scripts in the test directory
	file(GLOB_RECURSE PYTHON_TEST_PATHS ${CMAKE_BINARY_DIR}/test/Test*.py)
	foreach(TEST_PATH ${PYTHON_TEST_PATHS})
	cmake_path(GET TEST_PATH STEM TEST_NAME)
	add_test(${TEST_NAME} pytest -v ${CMAKE_BINARY_DIR}/test/${TEST_NAME}.py)
	endforeach()

This assumes that the custom target runs before the loop in line 74. Since this is not the case, the first time one runs cmake there is no file under CMAKE_BINARY_DIR/test and thus the tests are skipped.
I just tried and confirmed this is the case, run cmake once and you get 3 tests, run twice and you get 11.

[RaulPPelaez]

I just want to point out that I gave the above snippet to gpt4 without any context and the thing answered this:

There seems to be an issue here, though. The add_test command is run at configure time, but the files that you want to glob won't exist until build time, after the custom command is run. Therefore, file(GLOB_RECURSE PYTHON_TEST_PATHS ${CMAKE_BINARY_DIR}/test/Test*.py) will return an empty list.

To fix this issue, you could move the test discovery and registration into a separate script that you call using add_test, or consider configuring the tests at build time. This is a known limitation of CMake's test discovery.

Dang...
I opened #104 to work on this.