CentOS 7 - SSL issues, both HttpRequest and SQL Connections

[las3r]

I'm building my asp.net core app (2.1) on windows 10 with VSTS 2017. I publish my app through FTP to a CentOS 7 Linux box (freshly installed with dotnet hosting packages, latest SDKS and SQL server 2017 developer edition).

I have the same issue as described in #26590, but I am also unable to connect my app on linux to SQL Server 2017 that's on the same machine. This turns into:

Jun 24 20:09:37 An exception occurred in the database while iterating the results of a query for context type 'WebApplication2.Data.ApplicationDbContext'.
Jun 24 20:09:37 System.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 31 - Encryption(ssl/tls) handshake failed) ---> Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
Jun 24 20:09:37 at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
Jun 24 20:09:37 at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
Jun 24 20:09:37 at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
Jun 24 20:09:37 at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
Jun 24 20:09:37 at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet1 downloaded, HashSet1 systemTrusted, TimeSpan& remainingDownloadTime)
Jun 24 20:09:37 at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
Jun 24 20:09:37 at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
Jun 24 20:09:37 at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
Jun 24 20:09:37 at System.Net.Security.CertificateValidation.BuildChainAndVerifyProperties(X509Chain chain, X509Certificate2 remoteCertificate, Boolean checkCertName, String hostName)
Jun 24 20:09:37 at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
Jun 24 20:09:37 at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:09:37 at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
Jun 24 20:09:37 at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
Jun 24 20:09:37 at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
Jun 24 20:09:37 at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost)
Jun 24 20:09:37 at System.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)
Jun 24 20:09:37 at System.Data.SqlClient.SNI.SNIProxy.EnableSsl(SNIHandle handle, UInt32 options)
Jun 24 20:09:37 at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling)
Jun 24 20:09:37 at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
Jun 24 20:09:37 at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
Jun 24 20:09:37 at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
Jun 24 20:09:37 at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
Jun 24 20:09:37 at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
Jun 24 20:09:37 at System.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
Jun 24 20:09:37 --- End of stack trace from previous location where exception was thrown ---

As for the issue described by #26590 my stacktrace looks like this:

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
Jun 24 20:22:12 An unhandled exception has occurred while executing the request.
Jun 24 20:22:12 System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
Jun 24 20:22:12 at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
Jun 24 20:22:12 at Internal.Cryptography.Pal.StorePal.LoadMachineStores()
Jun 24 20:22:12 at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
Jun 24 20:22:12 at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
Jun 24 20:22:12 at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.FindCandidates(X509Certificate2 leaf, X509Certificate2Collection extraStore, HashSet1 downloaded, HashSet1 systemTrusted, TimeSpan& remainingDownloadTime)
Jun 24 20:22:12 at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
Jun 24 20:22:12 at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
Jun 24 20:22:12 at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
Jun 24 20:22:12 at System.Net.Security.CertificateValidation.BuildChainAndVerifyProperties(X509Chain chain, X509Certificate2 remoteCertificate, Boolean checkCertName, String hostName)
Jun 24 20:22:12 at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
Jun 24 20:22:12 at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken)
Jun 24 20:22:12 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
Jun 24 20:22:12 --- End of stack trace from previous location where exception was thrown ---

I've reproduced this on 2 different (freshly minted) CentOS 7 boxes with the latest SDKs and hosting runtimes installed. It seems something is wrong with how .NET Core is verifiying the SSL certificates / secure connection. It's sad, because this makes the whole proposition for .net core on linux a bit less feasible.

[joshfree]

@bartonjs @davidsh

[pjanotti]

This one also looks like https://github.com/dotnet/corefx/issues/29942

[pjanotti]

Per stack this is the same as #26294 that was already merged on servicing release, closing this one.