Integration test using downstream PyJulia packages

[MilesCranmer]

This is a draft GitHub action to test a downstream Python package which depends on the conda-forge version of Julia. This should help with catching issues before a new Julia version is released on conda-forge. For example, we have seen issues with updating from 1.7.1 to 1.7.2 version of conda-forge's julia: conda-forge/pysr-feedstock#41. With a CI action like this it should help flag such issues early.

For the downstream package example I'm using PySR which has its conda-forge feedstock here. PySR connects to Julia through PyJulia (feedstock here).

I think this will be a nice test for all facets of the conda-forge version of julia. For it to succeed it needs to:

• Install Julia
• Create a new Julia shared env
• Install Julia packages in that shared env
• Activate that env, and import those Julia packages from a PyJulia runtime
• Run a smoke test of the PyJulia package, which requires many different julia dependencies

cc @ngam

Edit: didn't mean to request reviews from everybody, I guess vscode's pull request feature adds everybody listed as an owner...

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[MilesCranmer]

I'm not sure how to use build-locally.py with locally-built conda packages. We would want this workflow to first build the Julia and PyJulia packages, using the version in main, and then use those to build the PySR package. @ngam any idea?

[ngam]

We can likely integrate that as a test post-build. We will do it on azure. I will think of a way, but basically, something like the following is doable as a standalone line in the test section:
git clone repo && cd repo && build pysr && instantiate pysr

[MilesCranmer]

A quick way to do this would be to install the pip version of PyJulia and PySR. In principle those should interact the same with the base Julia install as the conda versions. But perhaps that misses some key things we would want to test as part of this.

[mkitti]

@ngam do you have further thoughts here?

[ngam]

Sorry this slipped my radar. There is really no need for this in the context of what's going on in the pysr-feedstock right now. Generally, we would test things downstream (in pysr-feedstock) not upstream (julia-feedstock).

For future reference btw, depending on how far we get while we still have momentum, we may want an ecosystem-wide pin for julia.

A good illustration of this is how we deal with pytorch. We actually pin pytorch globally on the minimum (currently 1.12) and because pytorch is quite extensible and has its own fledging ecosystem, the pin moves with every minor and everything gets rebuilt. For example, see how torchvision got rebuilt for pytorch 1.12 (conda-forge/torchvision-feedstock#58).

A global pin basically represents the "default" package of a certain dep should it be required. We obviously have global pins for all sorts of things around here, e.g. libhdf5

We could do the same with julia in the future. Then, any minor bump in julia (e.g. 1.8 -> 1.9) will trigger an ecosystem-wide rebuild of packages depending on julia (e.g. pysr). This will streamline the process and update the ecosystem accordingly. That's actually the beauty of conda-forge, all these things come together nicely, when you think about it.

We will need to break julia into smaller parts at point so that we can more effectively isolate it and package it, but that's a different story. Building the sys-img, libjulia, etc. etc. can be the way to go, but anyway, that's a different topic for now.

We can probably close this once we get the stuff in pysr-feedstock merged