[ci] Optimize resource usage

[johnkerl]

As we've got more folks pushing code, and more OS/version combinations being tested, we're often hitting CI concurrency limits during the workday.

GitHub docs initially look optimistic in terms of how we can bump spending limits but it turns out it's a moot point for our use-case. Our number of minutes per month isn't the thing binding peak usage -- it's the number of concurrent jobs (at peak). The max number of MacOS concurrent runners is 5 in our current tier (Free) as well as for Pro and Team. Only for Enterprise level does it jump all the way to 50 and that's a bit of overkill.

Some things we can do though:

• Do some kind of path-detection on modified files -- don't run Python CI on R commits or vice versa
• Don't run the full matrix of OS and Python/R version on each PR push -- perhaps a subset on PR pushes, and the full set on merge to main
• Can we build wheels less often? [ci/python] Don't build wheels on each PR push #697, but we should also add back in a way to run this nightly

[mlin]

Scribbling a random idea to speed up the python wheel builds: I think when building each python version x OS combo, it builds libtiledbsoma using the default CMake settings which ends up spending time building the unit test executable (and the Catch2 library supporting it). This is a waste because (in this context) we don't run those unit tests. We can turn that off. The only reason this change will probably take 30 minutes and not 2 is figuring out the right route to thread the flag from py-packaging.yml -> cibuildwheel -> apis/python/setup.py -> scripts/bld -> CMake.

[thetorpedodog]

I feel like we should also be able to cache the builds of the tiledbsoma native library using GitHub caching (https://github.com/actions/cache) and avoid having to do them every time.

[thetorpedodog]

I think #760 and #756 are the two most important pieces of this—pre-commit caching (#756) should drop 40ish seconds off that and binary caching (#760) will save entire minutes on changes where the binary doesn’t change. Between those, that should cut down runtime by 60% (and even more on Mac, where compilation takes far longer than on Linux).

[atolopko-czi]

I think #760 and #756 are the two most important pieces of this—pre-commit caching (#756) should drop 40ish seconds off that and binary caching (#760) will save entire minutes on changes where the binary doesn’t change. Between those, that should cut down runtime by 60% (and even more on Mac, where compilation takes far longer than on Linux).

Makes sense, approved both. (Saw the earlier comment on this, but didn't understand the size of the gain!).

Still worth incorporating the minimal-vs-full work on #750? We'll still use fewer GHA worker slots, which will allow more workflows overall to run in parallel, reducing contention amongst multiple (Python) develoers.

[johnkerl]

#750 #756 #760 are all merged 🎉