Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImportError: No module named _ctypes #572

Closed
ConnorRules opened this issue Oct 2, 2018 · 11 comments
Closed

ImportError: No module named _ctypes #572

ConnorRules opened this issue Oct 2, 2018 · 11 comments
Assignees
@Laevos
Labels
bug enhancement

Comments

@ConnorRules
Copy link

It looks like the ctypes library is required by the pex bootstrapper. Our current python environment does not have that library. Is there any way to package ctypes in my pex build?
@jsirois
Copy link
Member

jsirois commented Oct 3, 2018

There isn't. The actual ctypes library is part of the python stdlib, but it relies on the native library libffi. See here for some details: https://bugs.python.org/issue31652

This is unfortunate - we only use ctypes on linux and in the context of determining if the current linux host is manylinux1 compatible. We could spend some effort to make importing lazy such that we only trigger this error if you're actually running pex on linux and you want manylinux support.

@ConnorRules
Copy link
Author

Interesting. During initial debugging of this error I did install libffi to no avail. The only thing that worked was copying the ctypes.so and ctypes library directory into my python environment.

Trying to find a good workaround. The python environment I'm working with is prepackaged with some enterprise software. So manipulating the modules in it is not preferred and for some reason they didn't pack ctypes with it even though that's standard.

Thanks for the reply

@jsirois
Copy link
Member

jsirois commented Oct 3, 2018

Interesting. During initial debugging of this error I did install libffi to no avail.

I should have been more clear: you need to have libffi(-dev) installed to have libffi headers available before you build python. Unfortunately python will happily build without ctypes support. This is why you had to monkey with copying.

If you don't actually need manylinux1 support, then I think this issue stands as a bug/enhancement. We definitely could be lazy accessing the tiny bit of code that needs to consult ctypes. If you then built pexes with --no-manylinux, you should be good.

@Laevos
Copy link

Laevos commented Oct 3, 2018

Hey @jsirois, I've been working with @ConnorRules on this issue. A --no-manylinux build option in pants would definitely solve this for us. Alternatively, would it be at all possible to just wrap the import of ctypes in glibc.py#L4 in a try block? Thanks!

@jsirois
Copy link
Member

jsirois commented Oct 3, 2018

To be clear --no-manylinux already exists. I'm just suggesting that its only valid for us to lazily import the ctypes code (or try/except as you suggest) if --no-manylinux.

@Laevos
Copy link

Laevos commented Oct 3, 2018

Right, okay, that makes sense. I saw and tried to use --no-manylinux with PEX, but ran into the same issue. Just trying to think about how we would send the status of --no-manylinux to the glibc module. As far as I can tell, glibc.py is only pulled in by pep425tags.py, which is in turn imported in package.py, platforms.py, translator.py and interpreter.py, so just being able to pass the flag to the glibc module seems ideal.

@jsirois
Copy link
Member

jsirois commented Oct 4, 2018

There is no support for global flags so the information will need to be threaded through those callsites unfortunately. Is this something you're willing to work on @Laevos?

@Laevos
Copy link

Laevos commented Oct 4, 2018

Yeah, I'd be willing to spend some time on it. Right now to get things working we're just making a wrapper script that runs the PEX in a separate Python environment that does have ctypes available, but eventually it would be nice to switch to a solution that is less dependent on a hack like that.

@jsirois
Copy link
Member

jsirois commented Oct 4, 2018

Excellent @Laevos - thank you! I've invited you to the contributors team. If you accept I can assign this issue to you. If not, that's fine too.

@Laevos
Copy link

Laevos commented Oct 4, 2018

Accepted, go right ahead, thanks!

Laevos pushed a commit to Laevos/pex that referenced this issue Oct 9, 2018
# This is a combination of 8 commits.
e338eb3 
# The first commit's message is:
pex-tool#572: Allow import of ctypes to be skipped if use_manylinux is false

# This is the 2nd commit message:

Narrow the env marker test. (pex-tool#578)

The jupyter dist is just a meta-dist with fully unconstrained deps on ~6
other dists. This test was added to test environment marker support in
pex, which ipython - not jupyter - leverages heavily.

# This is the 3rd commit message:

Fix resolve regressions introduced by the 1.4.8. (pex-tool#580)

PR pex-tool#571 regressed the half-broken state of having
`--interpreter_constraint` selected interpreters not setup to also
having `--python` selected interpreters also not setup. In addition,
PR pex-tool#568 incorrectly classified the current Platform passed by
`resolve_multi` as a user-specified extended platform specification
breaking custom interpreter resolution. Fix both and add tests that
failed prior to this combination of fixes.

A more comprehensive fix is tracked in part by pex-tool#579.

# This is the 4th commit message:

Cleanup `PexInfo` and `PythonInterpreter`. (pex-tool#581)

Kill an unused type in `PexInfo` as well as our last remaining use of
`pkg_resources.get_platform`. Also kill unused `COMPATIBLE_SETUPTOOLS`
constants in `PythonInterpreter`.

# This is the 5th commit message:

Support environment markers during pex activation. (pex-tool#582)

We've had support for environment markers on the resolve side for a
while and with just a little plumbing we can now support multi-python
pexes with environment-specific requirements.

Fixes pex-tool#456

# This is the 6th commit message:

Revert "Support environment markers during pex activation. (pex-tool#582)"

This reverts commit 5f1f00f.

We want to do a 1.4.9 bugfix release before this ~API change.

# This is the 7th commit message:

Prepare the 1.4.9 release. (pex-tool#588)

Work towards pex-tool#583

# This is the 8th commit message:

Revert "Revert "Support environment markers during pex activation. (pex-tool#582)""

This reverts commit 44ff463.

This restores pex-tool#582 for the 1.5.0 release tracked by pex-tool#585.
@Laevos Laevos mentioned this issue Oct 9, 2018
Closed
@jsirois
Copy link
Member

jsirois commented Jul 11, 2022

As noted in #591 this issue has been solved by switching to using a vendored version of PyPA's packaging which forgives missing ctypes.

@jsirois jsirois closed this as completed Jul 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Laevos Laevos

Labels
bug enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jsirois @ConnorRules @Laevos