Unable to make HTTPS requests with C#'s built-in class (not HTTPRequest) (fixed in master)

[esbennn]

Godot version: 3.2 stable

OS/device including version: Windows 10

Issue description:
We have a C# library, that includes a Log service, which we use in most our projects (of any kind). This service runs in a thread an POST's a bunch of strings to our server.
Implementing this Log service works just fine on my Linux machine, but when i deploy on a Windows machine (or my coworker runs it in the editor on his Windows machine), the web request fails with
Error: SecureChannelFailure (The authentication or decryption has failed.).
Here's a full stack trace:

at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.WebOperation.Run () [0x0009a] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.HttpWebRequest.GetRequestStream () [0x00016] in <8c9baba7add14c22be562e4c18e5738f>:0
  at Playground.Sandbox.Utility.WebRequest.Post (System.String url, System.Collections.Generic.Dictionary`2[TKey,TValue] values) [0x000b6] in <efaba92892e84c979ad8940e61632f93>:0
  at Playground.Sandbox.Services.LogService`1[T].Upload () [0x001f8] in <efaba92892e84c979ad8940e61632f93>:0 : Error: SecureChannelFailure (The authentication or decryption has failed.)

If i remove the s from the url, and stop our server from redirecting to https, it works, so this definitely has to do with tls.

I have tried overriding the ServerCertificateValidationCallback with

ServicePointManager.ServerCertificateValidationCallback += (p1, p2, p3, p4) => true;

as per this stackoverflow post

I have also tried forcing tls 1.2:

System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;

According to monos documentation, you can import certificates with a tool called mozroots, but it does not seem to be bundled with Godot. Is there a way to use this tool from my code, so I don't have to manually run it for every pc i deploy to?

Steps to reproduce:
Make a https request from C#/mono on a Windows machine

Minimal reproduction project:
New C# project, that creates an HTTP request to a https-url, using either System.Net.WebRequest or System.Net.WebClient.

[nonunknown]

Did you tried to recreate the issue using gdscript, if using gdscript the problem may be with c# lib, not the engine itself!
Error: SecureChannelFailure (The authentication or decryption has failed.)
this problem seems to be security issue related to System.Net.WebRequest or System.Net.WebClient Libs, try tu use Godot's HttpRequest Class instead!

[groud]

Sorry, this place is to report bug about Godot, not really about thirdparty libraries.

Regarding your question about how you could use mozroot inside Godot, you should ask on the other community channels first.

[esbennn]

Sorry, this place is to report bug about Godot, not really about thirdparty libraries.

Regarding your question about how you could use mozroot inside Godot, you should ask on the other community channels first.

Well, the point is that making http requests as one normally would in C# doesn't seem to work with how Mono is implemented - whether this is in a 3rd party library or not

[neikeq]

Yes, this is not a bug in a thitdparty library, but Godot and how it uses mono.

[groud]

Yes, this is not a bug in a thitdparty library, but Godot and how it uses mono.

I mean, if OP does not use a Godot internal class (like HTTPRequest), how can this be a Godot issue ? We don't have the hand on the code made by the user.

[esbennn]

Yes, this is not a bug in a thitdparty library, but Godot and how it uses mono.

I mean, if OP does not use a Godot internal class (like HTTPRequest), how can this be a Godot issue ? We don't have the hand on the code made by the user.

Fair point - i don't get it either. It seems to me that the issue is related to validating ssl/tls, specifically on Windows.

I just tried the endpoint with a Godot HTTPRequest. This works, but our function (which we use cross platform in both mono and .NET) does not.

Here's the request function

public static String Post(string url, Dictionary<string, string> values)
        {
            var request = (HttpWebRequest)System.Net.WebRequest.Create(url);
            var postData = "";
            
            foreach (string key in values.Keys)
            {
                postData += (postData.Length > 0 ? "&" : "") + key + "=" + values[key];
            }

            var data = Encoding.UTF8.GetBytes(postData);

            request.Method = "POST";
            request.ContentType = "application/x-www-form-urlencoded";
            request.ContentLength = data.Length;

            using (var stream = request.GetRequestStream())
            {
                stream.Write(data, 0, data.Length);
            }

            using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            { 
                using (StreamReader responseString = new StreamReader(response.GetResponseStream()))
                {
                    return responseString.ReadToEnd();
                }
            }
        }

It works just fine in a standard dotnet application, but not from withing Godot.

[neikeq]

What happens if you run Godot with the environment variable MONO_TLS_PROVIDER=btls?

[neikeq]

Also, for clarification, on Windows this works on the Godot editor but not exported games?

[esbennn]

What happens if you run Godot with the environment variable MONO_TLS_PROVIDER=btls?

I tried putting Environment.SetEnvironmentVariable("MONO_TLS_PROVIDER", "btls"); in the _EnterTree function of my root node.
That gives this exception:

at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x00207] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.WebOperation.Run () [0x0009a] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <8c9baba7add14c22be562e4c18e5738f>:0
  at System.Net.HttpWebRequest.GetRequestStream () [0x00016] in <8c9baba7add14c22be562e4c18e5738f>:0
  at Playground.Sandbox.Utility.WebRequest.Post (System.String url, System.Collections.Generic.Dictionary`2[TKey,TValue] values) [0x000b6] in <7c62243f7d9f4a399e5f28f251f59a42>:0
  at Playground.Sandbox.Services.LogService`1[T].Upload () [0x001f8] in <7c62243f7d9f4a399e5f28f251f59a42>:0 : Error: ConnectFailure (TLS Support not available.)

[esbennn]

Also, for clarification, on Windows this works on the Godot editor but not exported games?

No, it fails both in the editor and when exported.
Works on Linux though

[esbennn]

Followup:
Making a very simple request like the following works:

WebClient client = new System.Net.WebClient();
string response = client.DownloadString("http://isitfriday.org/");

If the url is changed to https..., it does not work, but throws the Exception in the comment above.

I also tried using the RestSharp library to perform the requests - same story, works with http.
Using https the status code on the response is 0, seemingly indicating that the request does not make it to the server.

[neikeq]

Thanks for the example code. I'll be testing this later this week.

[X52p]

Any news or workarounds for this so far? My best guess is, that it doesn't find the certificates.

By the way, I tried all of these:
https://stackoverflow.com/questions/4926676/mono-https-webrequest-fails-with-the-authentication-or-decryption-has-failed
non worked.

[Calinou]

@X52p You should be able to use Godot's HTTPRequest class from C#. It uses the Mozilla certificate bundle included in the Godot binary by default.

[esbennn]

Any news or workarounds for this so far? My best guess is, that it doesn't find the certificates.

By the way, I tried all of these:
https://stackoverflow.com/questions/4926676/mono-https-webrequest-fails-with-the-authentication-or-decryption-has-failed
non worked.

I haver not been able to find a proper workaround, other than using a non-https endpoint - but i haven't tried it again since 3.2.2 came out.
As @Calinou says, the built-in HTTPRequest does work from C# with https, but dotnet standard approach (useful for threaded background tasks and such) does not

[X52p]

The problem is, that I'm using a library that is doing the requests. So I can't just replace the commands.
(currently using v3.2.2.stable.mono.official)

[esbennn]

I'm using a library that is doing the requests.

Excactly

[X52p]

Did anyone try to use a different build tool? (In Editor Settings -> Mono -> Builds and than something other than VS Build Tools)
I tried but I could not get them to compile (It is not finding the libraries).

Is there any documentation on how to use them? Then I could set them up correctly and give it a try. (I'm new to using Mono and NuGet, sorry if this is a stupid question)

[X52p]

I just compiled Godot 3.2 with Mono myself. Also doing this before building:

cert-sync --user myCert.pem

Were myCert.pem is the certificate of the server I'm trying to connect to.
This seems to fix it. (I got other errors now, but these are probably library related)

[TechnoPorg]

I think this issue is because of how the official Godot binaries are built. In Mono, there's an abstract class called MobileTlsProvider used by the C# built-in web requests. A MobileTlsProvider is obtained via MonoTlsProviderFactory, which provides an implementation of MobileTlsProvider that will work with the system it's running on. It seems to me like for Windows, Mono uses the BTLS system and without it can't use TLS. BTLS is only available in Mono if MONO_FEATURE_BTLS is defined when MonoTlsProviderFactory.cs is compiled.
Example:

#if MONO_FEATURE_BTLS
		[MethodImpl (MethodImplOptions.InternalCall)]
		internal extern static bool IsBtlsSupported ();
#endif

The above function points to a C function that returns true if --enable-btls is passed to autogen.sh.
In my Mono 4.5 installation from the binary on their website, the BTLS code is still present when I look at it with DnSpy. In the Mono 4.5 from the Godot 3.3 official binary, however, the same code didn't get included. C# HTTPS requests work when you build Godot with Mono yourself, I think because it copies the existing Mono installation which for most people comes from the official Mono binaries.

My Godot build

No BTLS in the official Godot binary

[akien-mga]

Here's a build of Godot 3.4 beta 2 with the fixed Mono builds from godotengine/godot-mono-builds#47

https://downloads.tuxfamily.org/godotengine/testing/Godot_v3.4-beta2_mono_with-btls_win64.zip

Could you confirm that it works as expected?

[sitiom]

Here's a build of Godot 3.4 beta 2 with the fixed Mono builds from godotengine/godot-mono-builds#47

https://downloads.tuxfamily.org/godotengine/testing/Godot_v3.4-beta2_mono_with-btls_win64.zip

Could you confirm that it works as expected?

Still getting the error.

[akien-mga]

Thanks, something seems to still be off. Continuing debugging in godotengine/godot-mono-builds#47.

[sitiom]

Note: Fixed in Godot 4 which uses .NET 6 (#64089 (comment))

[akien-mga]

This was closed again by mistake when updating a fork. But since this was fixed in 4.0, and no contributor is working on it for 3.x, I think we can leave it closed.