CEP: support for ABI3 packages

[isuruf]

Description

[wolfv]

I think for V1 recipes we would use:

build:
  python:
    version_independent: true/false

[wshanks]

Is version_independent better than abi3? It seems possibly misleading to me because a package can still set a minimum Python version and so be Python version-dependent.

[isuruf]

abi3 packages with package_metadata_version = 2 should not be that different from any other package. It just has more things in info/link.json.

[mariusvniekerk]

i think something like

python:
   abi: abi3

leaves open enough space to support future expansion of other abis that may require special behavior

for example

python:
   abi: hpy

[isuruf]

@mariusvniekerk, both those options need the exact same thing. (B1-B4)

[isuruf]

Here's what a conda-build PR would look like conda/conda-build#5456

[wshanks]

Is it planned that the conda-forge tooling would check for this option and then set up Windows/Mac/Linux builds even though noarch: python is specified where it would otherwise only build on one arch?

[isuruf]

Yes, we can do that.

[wshanks]

Should there be a suggestion for a new way to specify abi3 packages in this proposal? I understand how noarch: python almost works already and so can be used to get abi3 packages working with existing tools, but it is a misleading label since these packages are still architecture dependent.

Also, I had tried to test with post-link.sh and pre-unlink.sh scripts previously here but I didn't try noarch: python and got stuck on the run exports of Python. Maybe I just didn't know the right syntax, but it seemed like when you don't use noarch: python conda-build pins the Python version in a way that is hard to override by setting the Python version in the recipe requirements. That might be worth mentioning as another important behavior of noarch: python (if I was not missing something).

[isuruf]

Should there be a suggestion for a new way to specify abi3 packages in this proposal? I understand how noarch: python almost works already and so can be used to get abi3 packages working with existing tools, but it is a misleading label since these packages are still architecture dependent.

This is package_metadata_version = 1 which is for existing tool compatibility. Please see the next section for new ways.

Also, I had tried to test with post-link.sh and pre-unlink.sh scripts previously conda-forge/qiskit-terra-feedstock#41 but I didn't try noarch: python and got stuck on the run exports of Python. Maybe I just didn't know the right syntax, but it seemed like when you don't use noarch: python conda-build pins the Python version in a way that is hard to override by setting the Python version in the recipe requirements. That might be worth mentioning as another important behavior of noarch: python (if I was not missing something).

That should be fixable by using ignore_run_exports. This is only specific to conda-forge which has run_exports on python package and defaults does not AFAIK. I wanted to keep the CEP more generic.

[baszalmstra]

For reference, rattler follows the same behavior as micromamba.

[jezdez]

I'm interested what your suggestion is for this mismatch in behavior, should conda be adapted to follow mamba and rattler in this case?

[isuruf]

My suggestion is to keep the existing mismatched behaviour in package_metadata_version = 1 and make this uniform in package_metadata_version = 2

[baszalmstra]

Doesnt this imply that older python versions available on for instance conda-forge will never have the ability to use abi3 packages? If we are modifying the installer we could just as well move the Python files to the python specific site-packages directory.

[isuruf]

Sure, but we can also have the site.py in a separate conda package as well. I'm trying to reduce all these special things that a installer has to do. With the files in lib/python/site-packages, the only thing that a installer has to do for an abi3 package is to compile the pyc files which is an optional thing anyway.

[baszalmstra]

But relying on a special package might also be problematic if you are not using conda-forge though.

[isuruf]

I'm hoping we can add support in defaults too.

[baszalmstra]

I would not prefer this because it means the proper functioning of the installer is dependent on a specific package. I'd rather encode the behavior in the installer itself.

[baszalmstra]

I agree that the installer should do less special case stuff. But I also don't think that it should depend on specific packages to function properly.

Even with this proposal the installer does already have special case behavior. However, without the dependency on a package that introduces a site.py file it will leave the package in a broken state. To me that adds complexity for the package maintainer.

However, one could argue that the same holds for depending on python itself..

It would be ideal if a recipe author doesn't have to care about these details.

[isuruf]

We could make the build tool add this package, then the installer does not have to care about it.

[baszalmstra]

Do we do this for other packages? Because then we tie the build-tool to specific package names which might also not be ideal.

[isuruf]

We take the C compiler name from c_compiler variable in the config. We could do the same.

[isuruf]

Oh I forgot about run_exports: noarch. We can extend that in the python recipe.

[jezdez]

Should we hard-code this into the CEP which may prove to not work anymore at some point?

[isuruf]

No, I'll remove it