Unvendor libmpdec sources

[zware]

To facilitate cleaner updates of the externally-maintained libmpdec required by the _decimal module, we should migrate away from the bundled copy in Modules/_decimal/libmpdec and towards an "external" in cpython-source-deps for Windows and --with-system-libmpdec by default elsewhere.

My tentative plan is as follows:

• In 3.13:
  • Add mpdecimal-2.5.1 and mpdecimal-4.0.0 to cpython-source-deps:
    • Sort list of downloads, and add link to mpdecimal sources cpython-source-deps#32
    • https://github.com/python/cpython-source-deps/tree/mpdecimal
  • Switch Windows build to mpdecimal-2.5.1 external (gh-115119: Switch Windows build to mpdecimal external #115182)
  • gh-115119: Detect _decimal dependencies using pkg-config #115406
  • Update deps and build instructions wrt. libmpdecimal devguide#1313
  • gh-115119: Update macOS installer to build with libmpdec 4.0.0 #118382
  • gh-115119: Update Windows installer to mpdecimal 4.0.0 #118506
  • Update macOS build to mpdecimal-4.0.0
  • gh-115119: Default to --with-system-libmpdec=yes #118539
  • gh-115119: Bump CI to use Ubuntu 22.04 #118631
  • Post #118539: amend Ubuntu CI to use system installed mpdecimal
  • Update the docs; see Unvendor libmpdec sources #115119 (comment)
  • gh-115119: Fall back to bundled libmpdec if system libmpdec is not found #119196
• In 3.14:
  • Remove the implicit fallback to the bundled copy
  • Keep support for explicit --without-system-libmpdec (with warning)
• In 3.15:
  • Remove --with[out]-system-libmpdec configure options and Modules/_decimal/libmpdec/

Linked PRs

• gh-115119: Switch Windows build to mpdecimal external #115182
• gh-115119: Detect _decimal dependencies using pkg-config #115406
• gh-115119: Update macOS installer to build with libmpdec 4.0.0 #118382
• gh-115119: Update Windows installer to mpdecimal 4.0.0 #118506
• gh-115119: Default to --with-system-libmpdec=yes #118539
• gh-115119: Bump CI to use Ubuntu 22.04 #118631
• [3.12] gh-115119: Bump CI to use Ubuntu 22.04 (#118631) #118634
• gh-115119: Fall back to bundled libmpdec if system libmpdec is not found #119196
• [3.13] gh-115119: Fall back to bundled libmpdec if system libmpdec is not found (GH-119196) #119217

[erlend-aasland]

[...] with a fallback to the bundled copy (maybe with a warning?)

+1 for an autoconf warning.

[ned-deily]

Adding an external libmpdec to the macOS installer build shouldn't be a big issue. Assign it to me if you proceed with this.

[erlend-aasland]

FTR: the source deps repo is now updated.

[erlend-aasland]

@ned-deily: for build-installer.py, is there more to do than adding a configuration for mpdecimal? I've used the following configure options: ['--disable-cxx', 'MACHINE=universal']

[ned-deily]

Likely yes, I'll take care of it. Thanks.

[ned-deily]

I took a quick look at this and be aware that this change proposal will likely impact everyone building Python 3.13 on macOS. macOS does not ship a copy of libmpdec so, if the bundled version is removed, anyone building Python on macOS will now need to provide a local copy, either building from source or using a version from one of the widely-used third-party package managers, like Homebrew or MacPorts. Both currently have mpdecimal packages at the 4.0.0 level. (Recall that, unlike Windows builds, we do not currently provide non-vendored third-party library binaries for macOS users, other than those built for and linked into the Pythons provided by python.org installers for macOS.) In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected. And, in any case, the devguide instructions for installing dependencies will need to be updated to cover libmpdec. We should add all this and get this working before moving on to changing the installer.

[erlend-aasland]

In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

Last time I checked, libmpdec did not provide pkgconfig files:

$ ls $(brew --prefix libmpdec)/lib/pkgconfig
ls: /opt/homebrew/opt/mpdecimal/lib/pkgconfig: No such file or directory
$ cd mpdecimal-2.5.1; find . | grep pkg
$

[ned-deily]

With the current 4.0.0 version, it appears both Homebrew and MacPorts ship them:

$ pkg-config libmpdec --cflags
-I/opt/homebrew/Cellar/mpdecimal/4.0.0/include
$ pkg-config libmpdec --libs
-L/opt/homebrew/Cellar/mpdecimal/4.0.0/lib -lmpdec -lm

...

$ pkg-config libmpdec --libs
-L/opt/macports/lib -lmpdec -lm

[erlend-aasland]

Yes, the 4.0.0 version does indeed provide pkg-config configuration files. I'll work on an Autoconf patch.

[erlend-aasland]

@zware are we still aiming for 3.13 for the first batch of items?

[erlend-aasland]

In a quick test, it appeared that with ./configure --with-system-libmpdec today, pkg-config info for libmpdec was not being used. If so, that should be corrected.

Resolved by:

• gh-115119: Detect _decimal dependencies using pkg-config #115406

[freakboy3742]

can you use the copy we provide in the cpython-sources-deps repo?

For iOS: almost certainly not without some patching. It's a configure-based project, so at a minimum it will need a patch to support the iOS and iOS-simulator host tags. That might be as simple as updating config.sub (the current version from CPython would do the job), but I'd need to dig in to be certain.

I've tried multiple times to get the iOS tags (especially the simulator ones) upstream into autoconf; I was partially successful with the 2024-01-01 release, but it's still missing the simulator tags. The autoconf contribution process appears to be a black hole... you email them a patch, and maybe they apply that patch at some point in the future, so you watch the repo to see if your patch has been applied. I haven't even received acknowledgement that I've sent a patch, let alone feedback on the patch.

[erlend-aasland]

[...] The autoconf contribution process appears to be a black hole... you email them a patch, and maybe they apply that patch at some point in the future, so you watch the repo to see if your patch has been applied. I haven't even received acknowledgement that I've sent a patch, let alone feedback on the patch.

We can solve this with https://discuss.python.org/t/pinning-gnu-autotools-using-the-github-container-registry/47437
I'm aiming to complete that build container during the PyCon sprints.

[freakboy3742]

[...] The autoconf contribution process appears to be a black hole... you email them a patch, and maybe they apply that patch at some point in the future, so you watch the repo to see if your patch has been applied. I haven't even received acknowledgement that I've sent a patch, let alone feedback on the patch.

We can solve this with https://discuss.python.org/t/pinning-gnu-autotools-using-the-github-container-registry/47437 I'm aiming to complete that build container during the PyCon sprints.

While I agree the devcontainer approach is worth pursing, unless I'm missing something, devcontainers won't address what I'm talking about here.

The underlying issue is that third party libraries that have autoconf configuration won't support iOS out of the box, because autoconf doesn't have full support for iOS. As a result, any autoconf library that you want to use under iOS will require a patch of the config.sub file in that library's sources to add that support. We're talking about libmpdec here, but I currently have to patch xz, freetype, libpng, and libjpeg in order to support CPython builds and the limited set of binary wheels that I'm currently maintaining. Every binary library using autoconf will have the same problem.

This is a problem that needs autoconf to accept a patch upstream to add iOS simulator support, so that third-party packages can update their config.sub to an official version that has simulator support, rather than needing to apply the same custom patch to every config.sub that has been vendored into a library's sources.

[erlend-aasland]

[...] Every binary library using autoconf will have the same problem.

Got it; thanks for the clarification! Well, let's hope your upstream patch makes it to the next GNU Autoconf release 🤞

[skirpichev]

It seems, that due to https://bugs.debian.org/1056785 - more projects now use _pydecimal implementation, that gets battle-tested. Here is an example of a failure: https://github.com/aleaxit/gmpy/actions/runs/9153743194/job/25163188162 due to difference in the tp_name: "decimal.Decimal" vs "Decimal".

[we-are-all-individuals]

Reaching out to the Debian CPython maintainer @doko42. It would be very nice to make mpdecimal available again in Debian. According to https://www.bytereef.org/mpdecimal/changelog.html the Sphinx issues no longer exist in version 4.0.0.