Failure to start .net-5 function dotnet-isolated: grpc error connection refused

[akatakritos]

Hello, I'm having trouble getting my functions app to boot on a .net functions for linux app service plan.

I have a little HTTP health function and a TimerTrigger. Hitting the /health Function i made with the function key spins for a long time before timing out. Luckily we configured Serilog to write to files during app boot, and this is what I'm seeing in the files when I go visit through Kudu.

[2021-06-24 22:38:59.679 +00:00 Monitor.Functions.Program                        INF] Booting function app
[2021-06-24 22:38:59.832 +00:00 Monitor.Functions.Program                        INF] Loading secrets from keyvault
[2021-06-24 22:39:00.818 +00:00 Monitor.Functions.Program                        FTL] Error starting up function
Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: Connection refused (127.0.0.1:80) SocketException: Connection refused", DebugException="System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:80)
 ---> System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")
   at Grpc.Net.Client.Internal.HttpContentClientStreamWriter`2.WriteAsyncCore[TState](Func`5 writeFunc, TState state)
   at Microsoft.Azure.Functions.Worker.GrpcWorker.SendStartStreamMessageAsync(IClientStreamWriter`1 requestStream) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 83
   at Microsoft.Azure.Functions.Worker.GrpcWorker.StartAsync(CancellationToken token) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 60
   at Microsoft.Azure.Functions.Worker.WorkerHostedService.StartAsync(CancellationToken cancellationToken) in D:\a\1\s\src\DotNetWorker.Core\WorkerHostedService.cs:line 25
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Monitor.Functions.Program.Main() in /home/vsts/work/1/s/src/Monitor.Functions/Program.cs:line 56

Using Serilog to write to files, but heres the start up code:

            try
            {
                logger.Information("Booting function app");

                var host = new HostBuilder()
                    .ConfigureFunctionsWorkerDefaults()
                    .ConfigureAppConfiguration(configuration =>
                    {
                        var settings = configuration.Build();
                        var env = settings["ASPNETCORE_ENVIRONMENT"];

                        configuration.AddJsonFile("local.settings.json", optional: true);
                        configuration.AddJsonFile($"appsettings.{env}.json", optional: true);
                        configuration.AddEnvironmentVariables();

                        var keyVaultUrl = settings["KEYVAULT_URL"];
                        if (!string.IsNullOrWhiteSpace(keyVaultUrl))
                        {
                            logger.Information("Loading secrets from keyvault");

                            configuration.AddAzureKeyVault(new Uri(keyVaultUrl),
                                new DefaultAzureCredential(),
                                new KeyVaultSecretManager());
                        }


                    })
                    .ConfigureServices((context, services) =>
                    {
                        var connectionString = context.Configuration.GetConnectionString("MyContext");

                        services.AddDbContext<MyContext>(options =>
                        {
                            options.UseSqlServer(connectionString);
                        });

                        // some other internal services im not sure are relevant here
                    })
                    .Build();

                host.Run();
            }
            catch (Exception ex)
            {
                logger.Fatal(ex, "Error starting up function");
                throw;
            }

We previously had some issues with a transitive dependency on ASP.NET Core libraries that caused the app to be unable to boot, but I'm pretty sure those are no longer present; besides, that yielded a different error.

What can I be doing to troubleshoot this?

Update

Also getting this sometimes after i commented out my TimerTrigger function

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The request was aborted.
 ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
   at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
   at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
   at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")
   at Grpc.Net.Client.Internal.HttpContentClientStreamWriter`2.WriteAsyncCore[TState](Func`5 writeFunc, TState state)
   at Microsoft.Azure.Functions.Worker.GrpcWorker.SendStartStreamMessageAsync(IClientStreamWriter`1 requestStream) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 83
   at Microsoft.Azure.Functions.Worker.GrpcWorker.StartAsync(CancellationToken token) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 60
   at Microsoft.Azure.Functions.Worker.WorkerHostedService.StartAsync(CancellationToken cancellationToken) in D:\a\1\s\src\DotNetWorker.Core\WorkerHostedService.cs:line 25
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Monitor.Functions.Program.Main() in /Users/matt.burke/projects/opendorse/Opendorse-Monitor/src/Monitor.Functions/Program.cs:line 117

[david-peden-q2]

https://feedback.azure.com/forums/169385-web-apps/suggestions/40585333-grpc-support-in-azure-app-service

[akatakritos]

I'm sorry @david-peden-q2, im not sure thats relevant; i'm not specifically, as far as i know, trying to use GRPC; I thought this was some internal error related to the runtime? Is Keyvault using GPRC internally?

[fabiocav]

@akatakritos that's correct, the error above is about the internal gRPC communication with the workers, so the User Voice item is not related to the issue.

We're investigating a similar issue with Linux and VNet, is this your case by chance? If so, unfortunately, the only current workaround until the issue is resolved is to use VNet with the Windows offerings.

[akatakritos]

Thank you, we do have some linux & vnets going on, i'll reach out to my cloud engineer and get back to you. I know we've had some trouble with appservice plans related to the networks in some environments too, and he has a ticket open about that as well.

[akatakritos]

We're going to try removing the vnets. They were a nice-to-have feature but we'd rather ship :)

[akatakritos]

We removed the vnet from the function service plan and it now runs successfully. Thanks for the hint, never would have guessed at that!

[jincod]

Get the same error. Linux and Docker setup without VNet

[fabiocav]

Get the same error. Linux and Docker setup without VNet

@jincod can you please open a separate issue with the exception and error details of what you're seeing?

@akatakritos thank you for confirming. We're investigating the VNet issue and hope to have that resolved soon. Closing this as resolved.

[akatakritos]

Can you update here when the vnet issue is resolved? We'd be interested in re-enabling and trying again.

[metoule]

@jincod you might have the same issue I faced: #524

[yuriburger]

We are experiencing the same issue (dotnet-isolated + VNet integration breaks stuff), so leaving a comment to hopefully get notified if it gets resolved.