in-tree tests

[maxnoe]

Is there a reason why this guide (only) recommends out-of-tree tests?

I think having in-tree tests, especially for scientific python projects that have compiled components provide benefits, mainly that you can run the tests of the installed module to check if everything is working fine using:

pytest --pyargs <package>

[willingc]

Hi @maxnoe. I don't think there was any specific reason that it wasn't included. It was likely because it is a less common workflow though valuable for the reasons you state. Would you be willing to submit a brief PR to add this? ☀️

[lwasser]

Hey @maxnoe

I'll summarize our decision about tests while recognizing the importance of in-tree tests for packages like numpy that have compiled components. Please let me know if I need to reread your question.

This structure (assuming this is what you mean by out-of-tree tests) ensures that the user is always running tests on the installed version (rather than the flat files) of a package. This is particularly important for packaging beginners who may need help understanding that this is happening (until something goes wrong and they can't figure out why tests pass locally but not in CI, for example). This is why PyPA suggests that tests live outside the package code dir.

my_project/
├── my_package/
│   ├── __init__.py
│   ├── module1.py
│   ├── module2.py
├── tests/
│   ├── test_module1.py
│   ├── test_module2.py
│   └── test_integration.py
├── setup.py
└── README.md

In-tree test structure

In-tree tests are excellent because they will automatically ship with a package, allowing users to ensure that the build is building as expected locally. This is good for troubleshooting. However, for a beginner, you could run into the issue of running tests and inadvertently running them on the "flat" files rather than the installed version. We ran into this funny enough at our packaging guide sprint when creating a compiled package example to add to our guide. The code being run at a Python prompt was trying to access C files that weren't yet compiled (funny story).

my_project/
├── my_package/
│   ├── __init__.py
│   ├── module1.py
│   ├── module2.py
│   └── tests/
│       ├── test_module1.py
│       └── test_module2.py
├── setup.py
└── README.md

I see a use for both approaches depending on the package structure and type. Still, we decided to take the beginner-friendly approach suggested by PyPA following the assumption that folks newer to packaging won't yet have compiled code but also that we do want to expand our guide over time to capture more complex packaging topics that the scientific ecosystem often faces (as you point out).

As you point out, it's still important for many scientific packages to have tests close to the code.

If I understand what you are asking, adding a section on tests for packages that have compiled code would be an excellent middle ground. If you are open to working with us to create such a section, we'd welcome help / a pr on it, as @willingc mentions above!!

i hope that helps clarify! We are starting at the basics and will slowly build out our content over time. We appreciate help when it is available!! 🚀