gh-107361: strengthen default SSL context flags

[woodruffw]

See #107361: this adds VERIFY_X509_STRICT to make the default SSL context perform stricter (per RFC 5280) validation, as well as VERIFY_X509_PARTIAL_CHAIN to enforce more standards-compliant path-building behavior.

As part of this changeset, I had to tweak make_ssl_certs.py slightly to emit 5280-conforming CA certs. This changeset includes the regenerated certificates after that change.

CC @sethmlarson

• Issue: ssl.create_default_context(): add VERIFY_X509_STRICT and VERIFY_X509_PARTIAL_CHAIN to the default verify_flags #107361

---

📚 Documentation preview 📚: https://cpython-previews--112389.org.readthedocs.build/

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

[pitrou]

Side note: perhaps we mark all PEM files as generated?

[woodruffw]

Side note: perhaps we mark all PEM files as generated?

Done: I've marked certdata/*.{pem,0} as generated (the .0 files should also really be .pem I think -- I can do some cleanup there in a follow-up PR, if there's interest).

[vstinner]

In practice, this should have no effect on >99% of users: most root programs use 5280 as their baseline, and the Web PKI's CABF rules are also built on 5280.

Your PR required to regenerate all PEM test certificates. Does it mean that they belong to the 1% of affected users?

[woodruffw]

Your PR required to regenerate all PEM test certificates. Does it mean that they belong to the 1% of affected users?

I made the numbers up for emphasis, but I'll stand by this having no effect on the vast majority of users: the vast majority use case is the Web PKI, which is significantly stricter than even VERIFY_X509_STRICT. The fact that the PEMs had to be regenerated is more a testament to how lax OpenSSL is by default than any actual common practice 🙂

Or put another way: most users are not manually generating certs with openssl req or, if they are, they're massaging them into RFC 5280 compliance because other ecosystems (like Go, Java, and OSes/major software) are much more strict.

Additional datapoints:

• The MS root program is at least as strict as these changes: https://learn.microsoft.com/en-us/security/trusted-root/program-requirements
• The Java root program specifies RFC 5280, making it at least as strict: https://www.oracle.com/java/technologies/javase/carootcertsprogram.html
• Same with Adobe: https://helpx.adobe.com/acrobat/kb/approved-trust-list2.html

(I picked these semi-arbitrarily from https://pkic.org/ltl/)

[sethmlarson]

I'm ++ on the partial chain changes, my comments below apply to VERIFY_X509_STRICT:

Since OpenSSL will likely be the way folks are generating certificates and like you said, OpenSSL allows you to do things that isn't acceptable for Web PKI, is there any value in providing guidance on how one does generate a certificate with OpenSSL that works with VERIFY_X509_STRICT?

Also since we're recommending folks down a new code path to restore old behavior, should we create a new test which exercises that code path and verifies that "bad" certificates would be accepted?

What about non-Web PKI, does this change have any affect for certificates in that domain? I am less versed for this, so am relying a bit on others' experience.

[pitrou]

Since OpenSSL will likely be the way folks are generating certificates

Is that still the case? Aren't most users relying on something like Let's Encrypt nowadays (at least for Web certificates, but also perhaps e-mail)?

is there any value in providing guidance on how one does generate a certificate with OpenSSL that works with VERIFY_X509_STRICT?

I don't think writing an OpenSSL command line tutorial is really a responsibility for CPython maintainers.

[woodruffw]

Since OpenSSL will likely be the way folks are generating certificates and like you said, OpenSSL allows you to do things that isn't acceptable for Web PKI, is there any value in providing guidance on how one does generate a certificate with OpenSSL that works with VERIFY_X509_STRICT?

To clarify: I think most users won't be using OpenSSL to generate certs -- I think the majority will be getting their certs from a CA service (like Let's Encrypt), with a very small minority doing bespoke things. Putting an OpenSSL example in the docs might cause additional confusion, and IMO won't be too helpful anyways (since the context here is verification, in which case the user might not have any control over the certs they're being presented with).

What about non-Web PKI, does this change have any affect for certificates in that domain? I am less versed for this, so am relying a bit on others' experience

Nope, this should be wholly compatible with those -- "strict" validation for OpenSSL means "approximate RFC 5280," and the Web PKI profiles are a superset of RFC 5280 🙂

In other words: all public web certificates should be compatible with these changes.

[malemburg]

These are the checks that OpenSSL applies when VERIFY_X509_STRICT is set: https://www.openssl.org/docs/man3.0/man1/openssl-verification-options.html#x509_strict (search for -x509_strict in case the link doesn't take you there directly)

These checks all make sense for certs generated with a trusted CA, e.g. Let's Encrypt.

The other common use case is having self-signed certs (e.g. for intranet/VPC devices). Those will already fail to verify with the current standard settings, since self-signed certs are rejected by OpenSSL's verify. You have to use the ssl._create_unverified_context() or a custom one to the same effect for those.

Note: The POP/FTP/IMAP and SMTP modules use the unverified context, since intranet/VPC mail and FTP servers will often not use certs created by a trusted CA for various reasons or they are referenced using IP addresses, which don't work with certs at all.

[woodruffw]

@sethmlarson This test provides a backstop check on VERIFY_X509_STRICT, PTAL!

[woodruffw]

Sorry for the delay here! I've updated this PR to include a testcase for strict verification, including a "backstop" test that ensures that disabling VERIFY_X509_STRICT allows this to verify as it did before the change.

[woodruffw]

I'm a little mystified by this error, which only happens on OpenSSL 1.1.1w in CI:

 0:00:10 load avg: 2.63 [18/43/1] test_ssl failed (1 failure)
test test_ssl failed -- Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2971, in test_verify_strict
    with self.assertRaises(ssl.SSLError):
        s.connect((HOST, server.port))
AssertionError: SSLError not raised

AFAICT, the strict flag exists in that version and has the same semantics as in OpenSSL 3.0 and 3.1, which both pass.

Edit: This is probably why: openssl/openssl@1e41dad (and openssl/openssl#10688)

[woodruffw]

This should be good to go again: I've added a "backstop" test that confirms that VERIFY_X509_STRICT enables and disables as expected.

[sethmlarson]

LGTM, apologies for the delay!

[woodruffw]

LGTM, apologies for the delay!

It's "only" 3 months late on my end, so no hard feelings 😉

[cpython-cla-bot]

All commit authors signed the Contributor License Agreement.

[woodruffw]

Gentle ping on this! I've deconflicted the changelog again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @gpshead for commit cef6950 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[gpshead]

LGTM - I'm just running a round of buildbot tests to look for any surprises before merging.

[sethmlarson]

@woodruffw Looks like this change breaks trustme generated certificates, see python-trio/trustme#642

[woodruffw]

@woodruffw Looks like this change breaks trustme generated certificates, see python-trio/trustme#642

Thanks for sharing! Looking now.

[vstinner]

@woodruffw Looks like this change breaks trustme generated certificates, see python-trio/trustme#642

If an application is affected by these changes, they can "just" disable the two options, no?

Using ctx.verify_flags &= ~(ssl.VERIFY_X509_STRICT | ssl.VERIFY_X509_PARTIAL_CHAIN):

$ ./python
Python 3.13.0a5+ (heads/main:b44898299a2, Mar 28 2024, 09:01:21) [GCC 13.2.1 20240316 (Red Hat 13.2.1-7)] on linux
>>> import ssl

>>> ctx=ssl.create_default_context()
>>> ctx.verify_flags & ssl.VERIFY_X509_STRICT
<VerifyFlags.VERIFY_X509_STRICT: 32>
>>> ctx.verify_flags & ssl.VERIFY_X509_PARTIAL_CHAIN
<VerifyFlags.VERIFY_X509_PARTIAL_CHAIN: 524288>

>>> ctx.verify_flags &= ~(ssl.VERIFY_X509_STRICT | ssl.VERIFY_X509_PARTIAL_CHAIN)
>>> ctx.verify_flags & ssl.VERIFY_X509_STRICT
<VerifyFlags.VERIFY_DEFAULT: 0>
>>> ctx.verify_flags & ssl.VERIFY_X509_PARTIAL_CHAIN
<VerifyFlags.VERIFY_DEFAULT: 0>

[woodruffw]

If an application is affected by these changes, they can "just" disable the two options, no?

Yep, correct. I think the above is also generating certs, so they wanted to get it right more generally, so that each Python downstream consumer doesn't have to also download the strict option.