Cannot run ASP.NET 5 default app on Debian 8 Jessie

[Alienwaren]

Hello!

I wanted to play around with new ASP.NET 5 on Linux (Debian Jesse 8). It's an MS Azure VM A2 Basic Tier. (2 cores, 3.5GB RAM)

I've used yo to scaffold an app to play with. It ran without any problems. so I think the app scaffolded succesfully.
I'm using Mono as runtime, version: (Stable 4.2.2.30/996df3c Fri Jan 22 00:02:19 UTC 2016) and Debian is running: Linux netrunnerdevelopment 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux kernel.

So, I ran dnu restore and then compiled the app using dnu build and it compiles without errors:

Microsoft .NET Development Utility Mono-x64-1.0.0-rc1-16231


Building test for DNX,Version=v4.5.1
Using Project dependency test 1.0.0
...
...
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time elapsed 00:00:16.7066640
Total build time elapsed: 00:00:16.7540983
Total projects built: 1

but when I run dnx web the server starts, but the app is stuck at loading:

Hosting environment: Production
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]
      Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]
      Executing action method test.Controllers.HomeController.Index with arguments () - ModelState is Valid'

I can't even kill it with CTRL-C, only using kill -9 <pid>.
Any ideas what to do?

[Tratcher]

@CesarBS Didn't you see a bunch of hangs on mono? Did it look anything like this?

[Alienwaren]

To add, I ran the app on Windows Server 2012 (a local Hyper-V VM, not Azure one), using the same commands, app ran without problems.

[davidfowl]

There were a bunch of mono hangs but nobody ever figured out why they were happening. I'd file a mono issue.

Here's some help on debugging mono hangs:

• http://www.debugthings.com/2015/04/07/debugging-vnext-mono/
• http://www.debugthings.com/2015/04/10/debugging-vnext-mono-pt2/

[davidfowl]

If you give @jldgit beer he may help you debug it 😄

[Alienwaren]

@jldgit Tell me what to buy and where and I will send it right now! :D

[debugthings]

Haha, I'll take a look later tonight and see if I can get it to repro. Did you have code can share?

[Alienwaren]

Here you go (I did not even modified scaffolded application):
Compilation and startup script

#!/bin/bash
dnu restore
dnu build > build.log
dnx web 

Startup.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNet.Builder;
using Microsoft.AspNet.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

namespace test
{
    public class Startup
    {
        public Startup(IHostingEnvironment env)
        {
            // Set up configuration sources.
            var builder = new ConfigurationBuilder()
                .AddJsoxnFile("appsettings.json")
                .AddEnvironmentVariables();
            Configuration = builder.Build();
        }

        public IConfigurationRoot Configuration { get; set; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddMvc();
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();

            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                app.UseExceptionHandler("/Home/Error");
            }

            app.UseIISPlatformHandler();

            app.UseStaticFiles();

            app.UseMvc(routes =>
            {
                routes.MapRoute(
                    name: "default",
                    template: "{controller=Home}/{action=Index}/{id?}");
            });
        }

        // Entry point for the application.
        public static void Main(string[] args) => Microsoft.AspNet.Hosting.WebApplication.Run<Startup>(args);
    }
}

appsettings.json

{
  "Logging": {
    "IncludeScopes": false,
    "LogLevel": {
      "Default": "Verbose",
      "System": "Information",
      "Microsoft": "Information"
    }
  }
}

Should I attach Views and Controllers code?
Also a screenshot to show the hang:
Here it is
Some informations(maybe useful):
Tested on all browsers, on screenshot it is IceWeasel/Mozilla Firefox 43.
Desktop Evirnoment on VNC: MATE
VNC Server: TightVNCServe

[davidfowl]

Best to provide a github repository with the failing app and instructions (specific and down to the mono version)

[Alienwaren]

Alright, I will provide it soon.

///EDIT: This is the repo: https://github.com/Alienwaren/failingmonoaspapp

[debugthings]

I was able to repro with that code example. Using the same environment. Steps to repro

1. Create A2 Debian VM on Azure
2. Install Mono from this guide
3. Follow ASPNET guide here
4. Follow Yeoman guide here
5. Run yo aspnet
6. Select Web Application
7. Run dnu restore
8. Run dnx web &
9. Run curl http://localhost:5000/ &

So far it looks to be a mono bug, would need to try on a few more systems to be sure.

[Alienwaren]

Alright, I will try out other Linux distos. To be clear, the code works on Windows Server 2012/Windows 10 Pro.

[debugthings]

Just some extra context around this so it can be reviewed further. This thread is blocked by a manual reset event that is held by the thread when trying to compile Index.cshtml. Specifically it's blocked when trying to read the file to generate the SHA1 hash for the Razor engine.

When the IAsyncResult is passed to the EndRead () method the internal wait handle is null. Stepping through the code in the debugger shows that the mono_threadpool_ms_end_invoke () native method will attempt to locate the handle and if it's not there (it's not) it will create one and wait for it indefinitely.

Using the scaffolding that includes other objects, like database and identity doesn't exhibit this behavior. The reason being is the AsyncResult shows a state of completed when it enters this code. Depending on the VM, the disks, the OS, etc., etc. this issue could crop up elsewhere.

This is definitely something that needs to be reviewed by Mono as it's probably an edge case that shouldn't happen in a normal environment.

BeginInvoke
https://github.com/mono/mono/blob/mono-4.2.2.30/mono/metadata/threadpool-ms.c#L1274
EndInvoke
https://github.com/mono/mono/blob/mono-4.2.2.30/mono/metadata/threadpool-ms.c#L1314

Managed Stack Trace

"Threadpool worker" tid=0x0x7fffec237700 this=0x0x7ffff7f31f30 thread handle 0x1005c state : waiting on 0x10060 : Event  owns ()
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_delegate_end_invoke (object,intptr) <IL 0x0000e, 0xffffffff>
  at (wrapper delegate-end-invoke) <Module>.end_invoke_int__this___IAsyncResult (System.IAsyncResult) <IL 0x0001a, 0xffffffff>
  at System.IO.FileStream.EndRead (System.IAsyncResult) [0x00063] in /home/debugthings/mono/mono-4.2.2/mcs/class/corlib/System.IO/FileStream.cs:605
  at System.IO.FileStream.Read (byte[],int,int) [0x000b1] in /home/debugthings/mono/mono-4.2.2/mcs/class/corlib/System.IO/FileStream.cs:523
  at System.Security.Cryptography.HashAlgorithm.ComputeHash (System.IO.Stream) [0x0001d] in /home/debugthings/mono/mono-4.2.2/external/referencesource/mscorlib/system/security/cryptography/hashalgorithm.cs:69
  at Microsoft.AspNet.Razor.RazorTemplateEngine.ComputeChecksum (System.IO.Stream) <IL 0x00008, 0x00053>
  at Microsoft.AspNet.Razor.RazorTemplateEngine.GenerateCode (System.IO.Stream,string,string,string) <IL 0x00040, 0x000f3>
  at Microsoft.AspNet.Mvc.Razor.MvcRazorHost.GenerateCode (string,System.IO.Stream) <IL 0x00021, 0x0009b>
  at Microsoft.AspNet.Mvc.Razor.Compilation.RazorCompilationService.GenerateCode (string,System.IO.Stream) <IL 0x00008, 0x00035>
  at Microsoft.AspNet.Mvc.Razor.Compilation.RazorCompilationService.Compile (Microsoft.AspNet.Mvc.Razor.Compilation.RelativeFileInfo) <IL 0x00022, 0x0005e>
  at Microsoft.AspNet.Mvc.Razor.Compilation.CompilerCache.CreateCacheEntry (string,System.Func`2<Microsoft.AspNet.Mvc.Razor.Compilation.RelativeFileInfo, Microsoft.AspNet.Mvc.Razor.Compilation.CompilationResult>) <IL 0x00048, 0x0012d>
  at Microsoft.AspNet.Mvc.Razor.Compilation.CompilerCache.GetOrAdd (string,System.Func`2<Microsoft.AspNet.Mvc.Razor.Compilation.RelativeFileInfo, Microsoft.AspNet.Mvc.Razor.Compilation.CompilationResult>) <IL 0x00047, 0x0009f>
  at Microsoft.AspNet.Mvc.Razor.VirtualPathRazorPageFactory.CreateInstance (string) <IL 0x00032, 0x00095>
  at Microsoft.AspNet.Mvc.Razor.RazorViewEngine.LocatePageFromViewLocations (Microsoft.AspNet.Mvc.ActionContext,string,bool) <IL 0x00177, 0x00753>
  at Microsoft.AspNet.Mvc.Razor.RazorViewEngine.GetRazorPageResult (Microsoft.AspNet.Mvc.ActionContext,string,bool) <IL 0x00051, 0x00163>
  at Microsoft.AspNet.Mvc.Razor.RazorViewEngine.FindView (Microsoft.AspNet.Mvc.ActionContext,string) <IL 0x0002a, 0x00063>
  at Microsoft.AspNet.Mvc.ViewEngines.CompositeViewEngine.FindView (Microsoft.AspNet.Mvc.ActionContext,string,bool) <IL 0x00021, 0x000a8>
  at Microsoft.AspNet.Mvc.ViewEngines.CompositeViewEngine.FindView (Microsoft.AspNet.Mvc.ActionContext,string) <IL 0x00020, 0x00037>
  at Microsoft.AspNet.Mvc.ViewFeatures.ViewResultExecutor.FindView (Microsoft.AspNet.Mvc.ActionContext,Microsoft.AspNet.Mvc.ViewResult) <IL 0x00044, 0x0009e>

Native Stack Trace

(gdb) mono_backtrace 15
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
185     in ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S
#1  0x00000000005fe3ab in _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x1005c, timeout=timeout@entry=0x0, alertable=alertable@entry=1, poll=poll@entry=0) at handles.c:1609

1609                                    res = mono_cond_wait (cond, mutex);

#2  0x00000000005fe42b in _wapi_handle_wait_signal_handle (handle=handle@entry=0x1005c, alertable=alertable@entry=1) at handles.c:1554

1554            return _wapi_handle_timedwait_signal_handle (handle, NULL, alertable, FALSE);

#3  0x000000000061051b in wapi_WaitForSingleObjectEx (handle=handle@entry=0x1005c, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at wait.c:194

194                             waited = _wapi_handle_wait_signal_handle (handle, alertable);

#4  0x0000000000589e47 in mono_threadpool_ms_end_invoke (ares=ares@entry=0x7ffff6c7ffb8, out_args=out_args@entry=0x7fffef998348, exc=exc@entry=0x7fffef998340) at threadpool-ms.c:1349

1349                    WaitForSingleObjectEx (wait_event, INFINITE, TRUE);

#5  0x000000000054aa88 in mono_delegate_end_invoke (delegate=0x7ffff6c7fd30, params=0x7fffef9983e0) at marshal.c:2826

2826                    res = mono_threadpool_ms_end_invoke (ares, &out_args, &exc);

#6 0x4055f118 in  (wrapper managed-to-native) object:__icall_wrapper_mono_delegate_end_invoke (object,intptr) + 0x58 (0x4055f0c0 0x4055f149) [0x9d2200 - Microsoft.Dnx.Host.Mono.dll]

Break Point on EndInvoke

Breakpoint 1, mono_threadpool_ms_end_invoke (ares=ares@entry=0x7ffff6c81b20, out_args=out_args@entry=0x7fffef998348, exc=exc@entry=0x7fffef998340) at threadpool-ms.c:1315
1315    {
(gdb) p (MonoAsyncResult)*0x7ffff6c81b20
$1 = {
  object = {
    vtable = vtable(%s), 
    synchronisation = 0x0
    }, 
    async_state = 0x0, 
    handle = 0x0, 
    async_delegate = 0x7ffff6c81898, 
    data = 0x0, 
    object_data = 0x7ffff6c81ae0,
    sync_completed = 0 '\000', 
    completed = 0 '\000', 
    endinvoke_called = 0 '\000', 
    async_callback = 0x0, 
    execution_context = 0x7ffff6c82f00, 
    original_context = 0x0,
    add_time = 0
    }
  }

[Alienwaren]

Alright, I will post it to Mono repo. Will I need some other informations in order to sumbit the bug to Mono issue tracker?

[davidfowl]

Closing this out.

[jijiechen]

@Alienwaren Did you post it to Mono repo? What's the current status for this topic?
I just ran into same issue on a CentOS 7 server with a 4.2.1 version Mono.

[jijiechen]

@Alienwaren I don't know if you are still interested in this topic, but I finally made a workaround. I upgrade my Mono runtime to 4.2.3 and the error continues.

Configure Mvc to use a wrapped synchronous IFileProvider instance instead of the builtin
async one (which may produce errors)
Please refer to this code:
https://gist.github.com/jijiechen/2b5e88923c156ed17e29cf6900d6cfd5#file-aspnetcore-mono-usesynchronousfileprovider-cs-L33

Thanks to details from @jldgit I understand it may be an issue from Mono. Someone reported one in Xamarin bugzilla, may be related to this one.

[Alienwaren]

Awesome, I will try it out :)