unbundle libcrypt

[beckermr]

There is an ABI change from SONAME 1 to SONAME 2 from alma/rocky 8 to alma/rocky 9 for libxcrypt. Some of our packages don't run out of the box on alma 9 without extra an extra compatibility package installed on the runtime OS.

We should

• remove libcrypt.so/libcrypt.a from our current sysroots
  • ENH no libcrypt on cos6 #54
  • ENH no libcrypt on cos7 #55
  • ENH no libcrypt on cos8 #56
• remove the remaining libcrypt*.so* from our current sysroots
• possibly rename the libxcrypt package to libxcrypt2 to reflect the ABI

cc @jakirkham @isuruf

[mbargull]

SGTM!
Though, I wouldn't change libxcrypt to libxcrypt2.
We already have a libxcrypt1 compatibility package for libcrypt.so.1 and the x in libxcrypt depicts the changed ABI anyway, AFAIK.

[mbargull]

(For reference, these changes would be very similar to what we did for libnsl in #40 .)

[mbargull]

I've updated the issue description with links to @beckermr's PRs and added an item for removing the library files that current packages link to in the future (this is mostly just for the case of us using the sysroot package for QEMU_LD_PREFIX; see #54 (comment) ).

I'd say removing the remaining files in a few month before we switch to CentOS 7 as the default linking target would make sense.

[AaronOpfer]

Hi, My company is using conda-forge to build C extensions against Python 3.6.15 and 3.7.12. Our compilations now fail because we cannot include a crypt.h, and our investigation eventually led us to this changeset. I see that it was necessary for conda-forge to rebuild Python 3.8 and newer with a new recipe in order to work with the latest sysroot, and it makes sense to me that this would not occur for Python 3.6.15 and 3.7.12 since they are no longer being built by conda-forge.

My question is: what do you believe my company's path forward should be to retaining our ability to build Python 3.6 and 3.7 C extensions using current day conda-forge? Obviously we shouldn't use old Pythons, but that is something that we're actively working on, bridging our ecosystem into Conda, and is why we encountered this problem. We also could fix the sysroot to the old version, but this is a difficult fix to implement in a large number of conda recipes, and we're worried about how we can document this process and publicize it fast enough. We could try to proxy conda-forge and remove this new sysroot, but that feels like it might be missing the whole point of this changeset. It's not clear to me that a repodata patch for the existing Python 3.6 and Python 3.7 builds would work either. So, our current inclination is to try and revive the old Python recipes and rebuild them against libxcrypt and put them into our third party package channel, but we have no idea how successful this would actually be. I'd be happy to hear your thoughts as we try to work through the problem.

EDIT: could we repodata patch Python 3.6 and Python 3.7 to depend on the compatibility package libxcrypt1?

[beckermr]

You should be able to include the libxcrypt package in your env and it should build against that.

[AaronOpfer]

My builds compile if I add libxcrypt to the host section. However, I am not sure that this is actually proper from an ABI standpoint, since it implies we're using different header files in the compilation process than the ones that were used to build Python (CentOS6). I am not an expert on ABI though, and could be wrong about this. FWIW, I tried to build with the libxcrypt1 compatibility package mentioned above, but I found that it does not actually have any header files: is that intentional?

[jakirkham]

If this can be solved by rebuilding the old python's with build/number bumps, would suggest creating PRs to do that. It's probably the simplest solution we can offer

[beckermr]

So ABI is about runtime linking whereas the problem of possibly different headers is an API question. I only point this out to ensure that we keep the discussion straight.

[AaronOpfer]

I uploaded a PR that I think would address the issue for Python 3.7. Linux and OSX builds are working, however, the Windows build is failing, and I have no ability to determine what is wrong (I haven't debugged a Windows build in almost 10 years), and the fault appears to be in some of the other recipe changes from conda-smithy. I would appreciate any assistance in resolving it.

Assuming that we can overcome that issue, I can put up a Python 3.6 build afterward.

[jakirkham]

Thanks Aaron! 🙏

Added a suggestion on that PR. It's possible we will need to adjust that depending on how it goes

Edit: In the worst case, we can always ignore Windows. CI failing just means it won't produce Windows packages, but we don't really need them to solve this issue anyways

[isuruf]

Unbundled now