python3 spkg: Work around errors building/importing the _crypt module #29012

mkoeppe · 2020-01-14T20:53:08Z

Building the spkg python3 3.7.3.p1 fails on Debian with libc 2.28-10 when using the c-compiler package from conda-forge, as reported in #28745.

The _crypt extension module of Python is deprecated in 3.8 and will be removed in 3.10 according to https://www.python.org/dev/peps/pep-0594/

This ticket proposes to remove the module from our build of Python or to ignore errors found during the build.

CC: @isuruf @dimpase @jdemeyer

Component: packages: standard

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29012

The text was updated successfully, but these errors were encountered:

mkoeppe · 2020-01-14T21:14:21Z

comment:1

More information on the failure:
Python3's configure script correctly determines that crypt_r is available if linking -lcrypt.
But this information is not used when building the extension. Instead it links to crypt only if find_library_file can find libcrypt - which fails.
This flawed logic is still present in CPython master (3.9.0 alpha 2) at https://github.com/python/cpython/blob/master/setup.py (https://github.com/python/cpython/blob/7d6378051feeadf45b4ce45b4b406b65df255648/setup.py#L1035)

  def detect_crypt(self):
        # crypt module.
       [...]

        if self.compiler.find_library_file(self.lib_dirs, 'crypt'):
            libs = ['crypt']
        else:
            libs = []

        self.add(Extension('_crypt', ['_cryptmodule.c'],
                               libraries=libs))
}}

mkoeppe · 2020-01-14T21:15:52Z

comment:2

conda-forge has a patch for python3 that seems related, but does not fix this issue: https://github.com/conda-forge/python-feedstock/blob/3.8/recipe/patches/0012-Fix-find_library-so-that-it-looks-in-sys.prefix-lib-.patch

dimpase · 2020-01-14T22:25:57Z

comment:3

isn't it easy to patch cpython to avoid said find_library_file ?
E.g., make sure pyconfig.h.in gets a variable which is #defined if
r_crypt is available from libcrypt, and call sysconfig.get_config_var() in
setup.py. I did something like this in (rejected, for unrelated reasons) python/cpython#12825

we can do a cpython PR for this issue...

mkoeppe · 2020-01-14T22:30:38Z

comment:4

My guess is that this patch will be rejected because upstream is already phasing out crypt (and also nis, which is hitting a similar problem).

mkoeppe · 2020-01-14T22:33:17Z

comment:5

Update: It seems that installing the Debian package dpkg-dev (a dependency of build-essential), which provides dpkg-architecture, makes Python recognize the multiarch structure on Debian, and the problem seems to go away.

mkoeppe · 2020-01-14T22:47:29Z

comment:6

This is the code in https://github.com/python/cpython/blob/9f220e4968cf73fa60440120ee46881e7974e47d/setup.py#L548

mkoeppe · 2020-01-15T00:30:51Z

comment:7

Replying to @mkoeppe:

... and the problem seems to go away.

Actually, no.

embray · 2020-01-17T14:12:06Z

comment:8

What is the actual error from Python when building the crypt module? I've never had a problem with it before.

mkoeppe · 2020-01-17T16:42:06Z

comment:9

Replying to @embray:

What is the actual error from Python when building the crypt module? I've never had a problem with it before.

#28745 has the Dockerfile that can be used to reproduce it, based on continuumio/miniconda3:latest (Debian)

The errors after adding dpkg-dev look like this:

[python3-3.7.3.p1] LD_LIBRARY_PATH=/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src CC='/opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread' LDSHARED='/opt/conda/envs/sage/bin/x86_64-
conda_cos6-linux-gnu-cc -pthread -shared -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-
rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib -L. -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,re
lro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib  ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'
    _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python -E ./setup.py  build
[python3-3.7.3.p1] running build
[python3-3.7.3.p1] running build_ext
[python3-3.7.3.p1] INFO: Can't locate Tcl/Tk libs and/or headers
[python3-3.7.3.p1] building '_struct' extension
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src
[python3-3.7.3.p1] creating build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules
[python3-3.7.3.p1] /opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -march=nocona -mtune=haswell -ftr
ee-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/conda/envs/sage/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-i
nitializers -Werror=implicit-function-declaration -I./Include -I/sage/local/include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Incl
ude -I/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src -c /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c -o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3
-3.7.3.p1/src/Modules/_struct.o
[python3-3.7.3.p1] In file included from /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/features.h:356:0,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/limits.h:27,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/limits.h:194,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/syslimits.h:7,
[python3-3.7.3.p1]                  from /opt/conda/envs/sage/lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/include-fixed/limits.h:34,
[python3-3.7.3.p1]                  from ./Include/Python.h:11,
[python3-3.7.3.p1]                  from /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c:8:
[python3-3.7.3.p1] /usr/include/x86_64-linux-gnu/sys/cdefs.h:482:49: error: missing binary operator before token "("
[python3-3.7.3.p1]  #if __GNUC_PREREQ (4,8) || __glibc_clang_prereq (3,5)
[python3-3.7.3.p1]                                                  ^
[python3-3.7.3.p1] In file included from ./Include/Python.h:25:0,
[python3-3.7.3.p1]                  from /sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_struct.c:8:
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:49:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'typedef'
[python3-3.7.3.p1]  typedef struct _IO_FILE FILE;
[python3-3.7.3.p1]  ^~~~~~~
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:50:1: error: unknown type name '__END_NAMESPACE_STD'
[python3-3.7.3.p1]  __END_NAMESPACE_STD
[python3-3.7.3.p1]  ^~~~~~~~~~~~~~~~~~~
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h: In function '__USING_NAMESPACE_STD':
[python3-3.7.3.p1] /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/stdio.h:65:25: error: storage class specified for parameter '__FILE'

followed by another 366MB of error messages

embray · 2020-01-17T17:26:50Z

comment:10

Yikes! I take it this is clang, not gcc?

isuruf · 2020-01-17T17:36:45Z

comment:11

This is mixing the sysroot in conda's gcc /opt/conda/envs/sage/x86_64-conda_cos6-linux-gnu/sysroot/usr/include/ and the system headers in /usr/include/x86_64-linux-gnu/ which won't work.

mkoeppe · 2020-01-28T02:08:23Z

comment:12

Replying to @embray:

What is the actual error from Python when building the crypt module? I've never had a problem with it before.

See https://github.com/mkoeppe/sage/runs/412042067
for the logs

mkoeppe · 2020-01-30T03:07:32Z

comment:13

isuruf's comment on #29053:
Looks like python's setup.py doesn't find the conda compiler's sysroot. https://github.com/python/cpython/blob/v3.7.3/setup.py#L867

mkoeppe · 2020-03-13T01:18:59Z

comment:14

isuruf's comment on #29053: This patch might help https://github.com/conda-forge/python-feedstock/blob/master/recipe/patches/0014-Fix-cross-compilation-on-Debian-based-distros.patch

mkoeppe · 2020-03-13T01:59:13Z

comment:15

I don't think this patch can help -- we are not attempting a cross compile.

mkoeppe · 2020-05-01T04:28:42Z

comment:16

Moving tickets to milestone sage-9.2 based on a review of last modification date, branch status, and severity.

mkoeppe · 2021-03-27T02:13:38Z

comment:18

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

mkoeppe · 2021-12-21T05:58:18Z

comment:20

Outdated, should close

dimpase · 2021-12-22T09:48:23Z

Reviewer: Dima Pasechnik

mkoeppe added this to the sage-9.1 milestone Jan 14, 2020

mkoeppe added c: packages: standard labels Jan 14, 2020

mkoeppe modified the milestones: sage-9.1, sage-9.2 May 1, 2020

mkoeppe modified the milestones: sage-9.2, sage-9.3 Oct 24, 2020

mkoeppe modified the milestones: sage-9.3, sage-9.4 Mar 27, 2021

mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021

mkoeppe added the s: needs review label Dec 21, 2021

mkoeppe removed this from the sage-9.5 milestone Dec 21, 2021

dimpase added s: positive review and removed s: needs review labels Dec 22, 2021

mkoeppe removed the s: positive review label Jan 8, 2022

mkoeppe closed this as completed Jan 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3 spkg: Work around errors building/importing the _crypt module #29012

python3 spkg: Work around errors building/importing the _crypt module #29012

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

dimpase commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 15, 2020

embray commented Jan 17, 2020

mkoeppe commented Jan 17, 2020

embray commented Jan 17, 2020

isuruf commented Jan 17, 2020

mkoeppe commented Jan 28, 2020

mkoeppe commented Jan 30, 2020

mkoeppe commented Mar 13, 2020

mkoeppe commented Mar 13, 2020

mkoeppe commented May 1, 2020

mkoeppe commented Mar 27, 2021

mkoeppe commented Dec 21, 2021

dimpase commented Dec 22, 2021

python3 spkg: Work around errors building/importing the _crypt module #29012

python3 spkg: Work around errors building/importing the _crypt module #29012

Comments

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

dimpase commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 14, 2020

mkoeppe commented Jan 15, 2020

embray commented Jan 17, 2020

mkoeppe commented Jan 17, 2020

embray commented Jan 17, 2020

isuruf commented Jan 17, 2020

mkoeppe commented Jan 28, 2020

mkoeppe commented Jan 30, 2020

mkoeppe commented Mar 13, 2020

mkoeppe commented Mar 13, 2020

mkoeppe commented May 1, 2020

mkoeppe commented Mar 27, 2021

mkoeppe commented Dec 21, 2021

dimpase commented Dec 22, 2021