Improve our testing infrastructure

[ocelotl]

I see several issues with our infrastructure:

1. Lint is very slow. This mostly happens because we install every single package to create a virtual environment for linting.
2. We are using tox.ini files instead of a pure python alternative.
3. We use several bash scripts, instead of python scripts
4. We use a script (eachdist.py) to run a few actions, which I think they can be moved into noxfile.py files
5. We may be missing better tools to specify dependencies (like pipfile.lock files and such)
6. We install several unnecessary dependencies every time we want to test our packages
7. We may not be using the best tools or combination of tools for linting
8. tox.ini is very long. It is easy to overlook testing errors in this huge file, it has happened before.

In more high-level, architectural terms, I feel like we have a huge monorepo where lots of different packages exist and our infrastructure is designed in the same way. For example, we have a single tox.ini file where everything needed to test any package is added, instead of having different tox.ini files, one for every package. The latter approach is better in my opinion since it keeps these files smaller and relevant only for the package they would be in. I have noticed several errors in our tox.ini file that have been there for a long time, since they are very hard to detect just because of how big and complex this file is.

The main reasons we have this monorepo approach is because we test every commit in the core repo against the test cases in these repo and we test the packages in the contrib repo against a particular commit of the core repo.

To be clear, I am not suggesting a change in this testing approach. The issues and possible solutions I mention in this issue are separate and independent from this approach.

My intention with this particular issue is just to find a solution for the points above. Even if we fix them all, we can continue with the cloning-the-core-repo testing approach.

[ocelotl]

• Lint is very slow #2419
• Use nox #2445
• Use python scripts #2446
• Move eachdist.py actions into noxfile.py files #2447
• Investigate better ways to specify test dependencies #2448
• Do not install unnecessary packages #2423
• Investigate better tools for linting #2449
• Separate tox.ini into multiple files #2478
• Duplicate test running information in tox.ini and CI workflows #2686