Wrong X509 chain selected when we have multiple viable options

[ayende]

Description

Let's Encrypt root certificate is expiring soon: DST Root CA X3.

We run into a problem where we load a certificate that is signed by ISRG Root X1, but when loading the PEM (and when calling Export), will be marked as signed by the soon to be expired DST Root CA X3.

The ca-certificates package has already removed the trust in this certificate, and that lead to generation of invalid certificates for the X509Certificate2 API.

Reproducing code

using System;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using System.Text;

namespace ConsoleApp5
{
    class Program
    {
        static void Main(string[] args)
        {
                Console.WriteLine(DateTime.Today);
                
                Console.WriteLine("-------------");
                var pem = File.ReadAllText(@"C:\Users\Oren\source\repos\ConsoleApp5\ConsoleApp5\t.pem");
                var cert = new X509Certificate2(Encoding.UTF8.GetBytes(pem), (string)null, X509KeyStorageFlags.MachineKeySet);
                var chain = new X509Chain();
                var r = chain.Build(cert);
                
                foreach (var item in chain.ChainElements)
                {
                    Console.WriteLine(item.Certificate.Subject);
                }
        }
    }
}

The certificate in question is:

t.pem.txt

Problem description

While the PEM file is properly signed by ISRG Root X1, this will output:

24/09/2021 0:00:00
-------------
CN=*.ravenclustertest2ppekrolwork.development.run
CN=R3, O=Let's Encrypt, C=US
CN=DST Root CA X3, O=Digital Signature Trust Co.

If I change the date to one month in the future, after the DST Root CA X3 is already expired, it will output:

24/10/2021 0:00:00

CN=*.ravenclustertest2ppekrolwork.development.run
CN=R3, O=Let's Encrypt, C=US
CN=ISRG Root X1, O=Internet Security Research Group, C=US

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Let's Encrypt root certificate is expiring soon: DST Root CA X3.

We run into a problem where we load a certificate that is signed by ISRG Root X1, but when loading the PEM (and when calling Export), will be marked as signed by the soon to be expired DST Root CA X3.

The ca-certificates package has already removed the trust in this certificate, and that lead to generation of invalid certificates for the X509Certificate2 API.

Reproducing code

using System;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using System.Text;

namespace ConsoleApp5
{
    class Program
    {
        static void Main(string[] args)
        {
                Console.WriteLine(DateTime.Today);
                
                Console.WriteLine("-------------");
                var pem = File.ReadAllText(@"C:\Users\Oren\source\repos\ConsoleApp5\ConsoleApp5\t.pem");
                var cert = new X509Certificate2(Encoding.UTF8.GetBytes(pem), (string)null, X509KeyStorageFlags.MachineKeySet);
                var chain = new X509Chain();
                var r = chain.Build(cert);
                
                foreach (var item in chain.ChainElements)
                {
                    Console.WriteLine(item.Certificate.Subject);
                }
        }
    }
}

The certificate in question is:

t.pem.txt

Problem description

While the PEM file is properly signed by ISRG Root X1, this will output:

24/09/2021 0:00:00
-------------
CN=*.ravenclustertest2ppekrolwork.development.run
CN=R3, O=Let's Encrypt, C=US
CN=DST Root CA X3, O=Digital Signature Trust Co.

If I change the date to one month in the future, after the DST Root CA X3 is already expired, it will output:

24/10/2021 0:00:00

CN=*.ravenclustertest2ppekrolwork.development.run
CN=R3, O=Let's Encrypt, C=US
CN=ISRG Root X1, O=Internet Security Research Group, C=US

Author:	ayende
Assignees:	-
Labels:	area-System.Security, untriaged
Milestone:	-

[vcsjones]

ca-certificates

Since you mentioned ca-certificates it sounds like this is on Linux.

Which .NET / .NET Core version are you using, and do you know which version of OpenSSL is present on the system exhibiting this behavior?

I ask because this is an upstream issue for OpenSSL 1.0.2 or earlier: https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire

[ayende]

I'll get the exact version of OpenSSL shortly. We are running both .NET 3.1 and .NET 5.0.

Note that this is also happening on Windows, where we are seeing the impact by removing: 48504e974c0dac5b5cd476c8202274b24c8c7172 from the Intermediate Certificate Authorities.

That is the Let's Encrypt R3.

[ayende]

Verified that this happened on OpenSSL 1.1.1 as well

[ayende]

Okay, I think that I understand a little better what is going on now.
The issue is when you have a long running process and a certificate that expires.

Consider the following below, which loads the certificate and holds it in memory.
We build the chain at the current time and in the future (past the certificate expiration).

In both cases, however, we'll get the old certificate: CN=DST Root CA X3, O=Digital Signature Trust Co..

For the first call, this is valid. But for the second, that is no longer the case.

What I believe is happening in our scenario is that we start a Kestrel instance with the certificate at a time when the
CN=DST Root CA X3, O=Digital Signature Trust Co. was valid. It will then send the intermediate for this chain (valid at this point).

However, a ca-certificates update will then update to revoke that certificate early. The server will still serve the same chain (now invalid), because it doesn't update this on the fly.

Does it make sense from your perspective?

For reference, this is what I'm getting, which is wrong.

---
Certificate chain
 0 s:CN = *.test.g1706-t3st.cloudtest.ravendb.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

However, the actual PFX file is setup with the ISRG Root X1. When I updated the time to the future (past the certificate expiration), it still gave the same (bad) chain.

Is there a way to affect what chain will be sent back from the server in this case?

[vcsjones]

I have a few thoughts here...

These issues seem related or this duplicates. There is some additional information in those issues, but on the Windows side, it looks like some things get cached and that results in re-using the expired chain.

• X509Chain.Build() fails to return valid certificate chain on Linux (openssl) if CA store contains expired certificate for CA, even if store contains valid cross-signed certificate for the same CA #43884
• Caching of SCHANNEL_CRED may lead to SslStream using expired certificate chain #43879

Is there a way to affect what chain will be sent back from the server in this case?

@wfurt may know from #37933 and if or how this is plumbed in Kerstrel.

[wfurt]

AFAIK the chain construction is out of our hands on Windows and there is no way how to pick exactly what to send. The best we could do is to make certificates available to CAPI and Schannel.
On Linux and macOS we give it array of certificates to send so we have total control.

[bartonjs]

Since it sounds like this is mainly about chains that were being served out of a TLS server instance, closing as duplicate of #43879.