Files required in test.source_files are not found during container test

[mfansler]

Looks like the biocontainer build section does not properly find the testthat files that are loaded through the test.source_files key. Is this a known bug?

Log Snippet (link)

12:41:03 BIOCONDA INFO (ERR) [Nov 19 12:41:03] SERR Error: `path` does not exist
12:41:03 BIOCONDA INFO (ERR) [Nov 19 12:41:03] SERR Execution halted
12:41:03 BIOCONDA INFO (ERR) [Nov 19 12:41:03] ERRO Task processing failed: Unexpected exit code [1] of container [fe3a4865cc0f step-cf14308070], container preserved
12:41:03 BIOCONDA INFO (OUT) ...
12:41:03 BIOCONDA ERROR COMMAND FAILED (exited with 1): mulled-build build-and-test r-archr=1.0.3--r43hdbdd923_0 -n biocontainers --test bash -c '/usr/local/env-execute true && . /usr/local/env-activate.sh && Rscript -e "library('"'"'ArchR'"'"')" && Rscript -e "testthat::test_file('"'"'tests/testthat.R'"'"', stop_on_failure=TRUE)"' --channels conda-forge,file:///opt/mambaforge/envs/bioconda/conda-bld,bioconda --involucro-path /opt/mambaforge/envs/bioconda/lib/python3.10/site-packages/bioconda_utils/involucro

12:41:03 BIOCONDA ERROR TEST FAILED: recipes/r-archr

Originally posted by @mfansler in #52121 (comment)

[mfansler]

Note that this is following the practice for testthat tests established for Conda Forge, where we don't use --install-tests, but copy testthat tests during test time. I believe this is the first time I'm trying testthat tests here. The only other recipes I found that run the testthat tests make use of --install-tests, so that they are part of the Conda artifact (see r-sew and r-quilt).

[mfansler]

Maybe even more prohibitive is that tests that need additional dependencies specified in test.requires will fail because they are not - and presumably, cannot be - installed in the container.

At this point, it seems rather that I require a mechanism to skip certain tests just when testing the container environment. Is there any way to do this?

[aliciaaevans]

I think to disable tests just for the container, you'd have to do some bash-related (or R?) check in the test line itself. You might be able to use an architecture selector to skip a test for linux-64, but then it wouldn't run them at all for that arch. (I haven't tried this myself.)

Usually it's been discouraged to run a full suite of unit tests because of CI runner limitations. Not sure if that's a factor here or how extensive the tests you're running are.

[aliciaaevans]

(just to add details here for future reference since you figured this out already) It looks like the code that gets the tests for the mulled build is pretty simple and doesn't support test.requirements or test.source_files. Instead it's just using requirements.run https://github.com/bioconda/bioconda-utils/blob/6240d31e706b91b4b217ab10305fb84aabd8a147/bioconda_utils/pkg_test.py#L40

[mfansler]

(just to add details here for future reference since you figured this out already) It looks like the code that gets the tests for the mulled build is pretty simple and doesn't support test.requirements or test.source_files. Instead it's just using requirements.run https://github.com/bioconda/bioconda-utils/blob/6240d31e706b91b4b217ab10305fb84aabd8a147/bioconda_utils/pkg_test.py#L40

Thanks for tracking that down 😊 - yes, that's what I expected might be happening under-the-hood. Seems like it might not be too hard to add functionality to also copy the test.source_files, but since that doesn't solve my ultimate issue (and I can work around that part), then maybe not worth the effort.

I may just keep the tests only for OSX as you suggested. It's mostly about checking for version incompatibilities with dependencies and seems I've already gotten out of it what I needed.

[mfansler]

I ended up conditioning on the presence of one of the test.requires dependencies to distinguish whether it's the container test. Otherwise, maybe a flag could be set that would more deliberately signal the container is being tested.