Generating lockfiles fails with: unknown error (_ssl.c:3161) #20467
Comments
|
The source is free to read. My reading says the underlying SSL lib CPython is linking against is not supported (you're probably on the right track): https://github.com/python/cpython/blob/8fc8c45b6717be58ad927def1bf3ea05c83cab8c/Modules/_ssl.c#L3161 I'd |
|
I'm experiencing the same issues, Also using Fedora. I'm only ever able to replicate the issue when using what's in the sandbox. I can't:
Using distrobox to try and run in older versions of fedora seems to have the same issue, but it doesn't seem to be entirely isolating things from the rest of the system. |
|
The sandbox blanks out env vars and that can be important. @xlevus can you |
the original when changing |
|
Further: Unpacking
The Buuuuut, changing the
|
|
@xlevus it looks like you have everything at hand you need to dig. If you can come up with a docker-based repro, perhaps someone can help out, but as it stands you have the files, paths, etc. |
|
I've created a docker-based reproduction here: https://github.com/xlevus/pants-issue-20467 and it seems to work in a Fedora VM, trying to get an ubuntu VM up to confirm it works in one of those too. Will poke around more tonight. But i'm a little stumped tbqh. It appears to only be an issue when specifically combining the distributed python3.9 venv for pants, and |
|
Great - thanks. I'll try to poke around with the repro case. That said, this is crazy-making "It appears to only be an issue when specifically combining the distributed python3.9" since your OP is this: That is definitely not python3.9 let alone the scie-pants hermetic python3.9. |
The error message is misleading. The 'failed to spawn a job' is from Pex's Job runner. I'm Here's my hacked up sandbox with a pdb.breakpoint stuck right before the failing ssl call: |
|
Ok, thanks for the repro case @xlevus - super helpful. I have not figured out why PBS Python 3.9 is different here, and apparently only different in a Fedora context to boot, but the issue is related to threading. If you use a PBS 3.9 repl to [root@3d2dd3ceaa5c pants-sandbox-qAIWax]# diff -u .deps/pex-2.1.137-py2.py3-none-any.whl/pex/fetcher.py pex-venv/lib/python3.9/site-packages/pex/fetcher.py
--- .deps/pex-2.1.137-py2.py3-none-any.whl/pex/fetcher.py 1980-01-01 00:00:00.000000000 +0000
+++ pex-venv/lib/python3.9/site-packages/pex/fetcher.py 2024-01-28 21:18:01.662789434 +0000
@@ -4,6 +4,8 @@
from __future__ import absolute_import
import ssl
+ssl.create_default_context()
+
import time
from contextlib import closing, contextmanager
[root@3d2dd3ceaa5c pants-sandbox-qAIWax]#There the diff represents some sandbox mucking about, but the upshot is trying to grab the context on import of The remaining work to do is to see what is buggy here. Is this a PBS Python build buggy somehow? Is it a bug in Pex code - should SSLContext only ever be created in the application main thread? Is this a Fedora glibc modern (which includes libpthread) vs libpthread.so.0 which PBS links to (unlike the system Python 3.9)? I have no clue at the moment. |
|
I'll note that I'm dropping work for the evening and I'm AFK likely until the 1st. |
|
Further Investigation:
Possible key change between the two is:
|
|
@xlevus I'm working on a short-term fix in pex-tool/pex#2355. I'd still love to know what's really going on here, but 1st to stop the bleeding. |
|
I've flipped this back to a bug - apologies @mjimlittle, you ended up being right there. With @xlevus's help debugging, a fix for this issue in Pex is now released in 2.1.163: https://github.com/pantsbuild/pex/releases/tag/v2.1.163 A Pants maintainer will take it from here and upgrade Pants / instruct you how to do so for your Pants version. |
|
Hey @jsirois thanks for the update. Also, I am sorry I could not help out in tracing the source of the issue. As a workaround, I have Dockerized pants using an Ubuntu base image and I can successfully generate needed lock files. |
Changelogs: * https://github.com/pantsbuild/pex/releases/tag/v2.1.163 ``` Lockfile diff: 3rdparty/python/user_reqs.lock [python-default] == Upgraded dependencies == pex 2.1.162 --> 2.1.163 ``` Fixes pantsbuild#20467
|
To use the new version of Pex without waiting on a Pants release (That's the sha256 and size of the |
|
Thanks @cburroughs works fine now! |
Changelogs: * https://github.com/pantsbuild/pex/releases/tag/v2.1.163 ``` Lockfile diff: 3rdparty/python/user_reqs.lock [python-default] == Upgraded dependencies == pex 2.1.162 --> 2.1.163 ``` Fixes #20467
|
@xlevus it turns out the issue is the custom RedHat OpenSSL option "rh-allow-sha1-signatures", seen here for example: https://gitlab.com/redhat/centos-stream/rpms/openssl/-/blob/c9s/0049-Selectively-disallow-SHA1-signatures.patch If I do this on a fedora:37 image: [root@d13f087cea45 /]# diff -u /etc/crypto-policies/back-ends/opensslcnf.config.orig /etc/crypto-policies/back-ends/opensslcnf.config
--- /etc/crypto-policies/back-ends/opensslcnf.config.orig 2024-02-09 00:54:33.569271689 +0000
+++ /etc/crypto-policies/back-ends/opensslcnf.config 2024-02-09 00:54:54.309267497 +0000
@@ -6,8 +6,3 @@
DTLS.MaxProtocol = DTLSv1.2
SignatureAlgorithms = ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ed25519:ed448:rsa_pss_pss_sha256:rsa_pss_pss_sha384:rsa_pss_pss_sha512:rsa_pss_rsae_sha256:rsa_pss_rsae_sha384:rsa_pss_rsae_sha512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
-[openssl_init]
-alg_section = evp_properties
-
-[evp_properties]
-rh-allow-sha1-signatures = yesThen a test rig works without main thread vs non shenanigans. As to why the thread makes a difference I have no clue yet, but a custom PBS build that enables openssl debug symbols and many gdb sessions later, I was able to narrow in on reading I'll update indygreg/python-build-standalone#207 with all the details of the debug session later tonight. This is not Gregory's problem, but others may bump into RedHat shenanigans and need the ~FAQ on what goes on when vanilla openssl in PBS tries to read RedHat custom config. |
|
The explanation is contained in a comment in pex-tool/pex#2358 which I've pinged folks in this thread on. |
Point readers to the result of the investigation in pantsbuild/pants#20467 and pex-tool/pex#2355 to indygreg/python-build-standalone#207.
Point readers to the result of the investigation in pantsbuild/pants#20467 and pex-tool/pex#2355 to indygreg/python-build-standalone#207.
Describe the bug
When trying to generate lockfiles command fails with the following error:
Failed to spawn a job for /home/manos/Workspace/pants-repo/.conda/bin/python3.9: unknown error (_ssl.c:3161)Pants version
Tested with versions:
(same result for all tested versions)
OS
Tested with
(same result for all tested versions)
Additional info
I think this issue started happening after a kernel update from Fedora. Has anyone else run into this issue before?
Any suggestions on how to resolve this would be very appreciated!
The text was updated successfully, but these errors were encountered: