Support system certificate store

[datamoc]

Description

Hello,

We don’t have direct Internet access in my company main network. But we have local pip repository.
Sadly pip on windows does not recognized the company certificates.
I was able to use pip with a ca-bundle.crt created with a very simple script:
import ssl

context = ssl.create_default_context()
der_certs = context.get_ca_certs(binary_form=True)
pem_certs = [ssl.DER_cert_to_PEM_cert(der) for der in der_certs]

with open('C:\Users\user\certs\ca-bundle.crt', 'w') as outfile:
for pem in pem_certs:
outfile.write(pem + '\n')

In _ssl.py file it’s written that is equivalent to ssl.create_default_context() but it’s not the case.
I was not able to change the _ssl.py file in order to have the expected behavior.

Regards

Michel

Expected behavior

pip should be able to use the windows certificates without any user action

pip version

21.3.1

Python version

3.6.6

OS

Windows 10 1909

How to Reproduce

add ini C:\Users\User\pip\pip.ini

[global]
index-url = htts://locarepo.company/pypi/simple

(you have to create a local server)
in Powershell

PS C:\Users\user\Documents\My Apps> & 'C:\Program Files\Python36\python.exe' -m pip install toto --user

result:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/

Output

PS C:\Users\user\Documents\My Apps> & 'C:\Program Files\Python36\python.exe' -m pip install toto --user

Looking in indexes: https://repos.company/api/pypi/python_pypi/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
Could not fetch URL https:// repos.company/api/pypi/python_pypi/simple/toto/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host= 'repos.company', port=443): Max retries exceeded with url: /api/pypi/python_pypi/simple/toto/ (Caused by SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)) - skipping
ERROR: Could not find a version that satisfies the requirement toto (from versions: none)
ERROR: No matching distribution found for toto

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

I seem to recall another entry on this but couldn’t find it, so I changed the title to more match the thing we want. There’s no active effort toward this though, so feel free to try your hands on it.

[pradyunsg]

Closing as a duplicate of #10961.

That issue has notes on what the appropriate changes would be, to make this happen.

[notatallshaw]

@uranusjr @pradyunsg I have a question on this, is it possible to add something as a non-supported option?

As discussed in #10961 adding an option for the requests ssl context with load_default_certs would definitely help most cases where the certs are in the system store but not in certifi. But there is an edge cases where Windows has not yet lazily loaded the certs in to system store and calling this method does not trigger that. However in most cases the user will have already triggered this by opening a web browser or using any other tool which uses Schannel to make an HTTPS request.

So would it be possible to add this feature optionally but add a warning that says it is not supported? Or is this too nuanced for user support for pip maintainers?

[pradyunsg]

That's... odd. I don't think it makes sense to add a feature only to call it unsupported. I'm definitely not comfortable adding such an option, since it does seem nuanced in a way that increases the support burden.

That said, let's move the discussion to that other issue, and see if other maintainers feel differently. :)

[uranusjr]

Is “that other issue” #10961? Should we unlock it then?

Another thing I feel is it’s difficult to gauge the impact (both directions) without the code. If this could be done similarly to keyring support, I’d probably be fine adding it; but if it needs to take the form of a hidden option or environment variable, probably less so.

[pfmoore]

Agreed I don't like the idea of adding a deliberately unsupported feature. We have enough trouble with existing unsupported behaviour that we didn't explicitly add...

[notatallshaw]

Maybe I should make a new issue? I'm thinking of rather than having a flag that says "Pip uses System certificate store" it could be a flag that says "Pip uses 'load default certs' for the SSL Context as provided by Python's standard library"

Then users can only expect whatever behavior the standard library provides and not any specific claims about what that it will actually do for them beyond what's documented in the standard library.

Edit: I just made a new issue to match the proposal: #11038

[pfmoore]

I don't think this needs another issue opening. We have this one and #10961. Why do we need a third? What has changed since this was discussed previously?

It's still IMO something that should be handled in our (vendored) http layer, not by pip directly. It seems that the only new point is that someone else is asking for it - maybe it's affecting more people, but those people still seem to be mainly on corporate networks, so maybe the companies can find some funding (either individually or jointly) to get SChannel support added to the Python networking stack? I know that at one point there were people looking at adding SChannel in the Python stdlib (I know almost nothing about this subject, and I didn't pay much attention at the time, so apologies if that mischaracterises what was proposed at the time...) I believe it got abandoned because there wasn't enough manpower to complete the work. That sounds very much like something that would have benefitted from corporate funding, and it would have solved this issue once and for all for every Python http library/application.

I don't see the value in adding a flag that mostly works but sometimes doesn't. Saying it's not supported won't stop people using it, and won't stop them flagging it to us when it doesn't do what they want. And the mere fact that we added the option in the first place is precedent for the idea that if people nag at us enough, we'll cave in. That's not a message I want to give, to be honest.

[notatallshaw]

I don't think this needs another issue opening. We have this one and #10961. Why do we need a third? What has changed since this was discussed previously?

The existing 2 issues are asking pip to explicitly support the system certificate store, the new issue is asking for pip to support the "load_default_certs" from the standard library.

This solves most people's issue without Pip over promising what is being provided.

It's still IMO something that should be handled in our (vendored) http layer, not by pip directly.

Requests has an API to to use the load_default_certs, they have left it up to libraries that want to use it to use that API.

Pip uses many APIs and extensions to requests that require more than a simple flag, this is one of them.

but those people still seem to be mainly on corporate networks, so maybe the companies can find some funding (either individually or jointly) to get SChannel support added to the Python networking stack

This is a completely seperate issue and something I agree with but not what's being asked for here.

I don't see the value in adding a flag that mostly works but sometimes doesn't.

It works a 100% of the time if what it's advertised for is "uses load_default_certs from the Standard library".

It works 99.99%+ of the time if what is advertised is "uses system certificate store", which is better than some other other of Pip's advertised features (such as correctly backtracking 😛 )

[pfmoore]

Thanks, that clarifies a lot. Sorry for not understanding this from the existing context.