containerd mount is invalid

[adeewu]

the error is

> Executing task: dotnet build /workspace/template-dotnet-core-cli-csharp/dotnetcore.csproj /property:GenerateFullPaths=true /consoleloggerparameters:NoSummary <

Microsoft (R) Build Engine version 16.10.1+2fd48ab73 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
/home/gitpod/dotnet/sdk/5.0.301/Roslyn/Microsoft.CSharp.Core.targets(71,5): error MSB6004: The specified task executable location "/run/containerd/io.containerd.runtime.v2.task/k8s.io/187c32ce2e4a85b2ac876a589b6e58b37c4e3151e6b4a93e96d14977cc9ba2fa/rootfs/home/gitpod/dotnet/dotnet" is invalid. [/workspace/template-dotnet-core-cli-csharp/dotnetcore.csproj]

Terminal will be reused by tasks, press any key to close it.

is suport debug in gitpod?

[ghuntley]

Yes, debugging is supported in Gitpod. See this demonstration app over at https://community.gitpod.io/t/debugging-net-unit-tests/2928/10 from March. It looks like something is not working correctly related to containerd and there has been a regression?

[ghuntley]

I've been able to reproduce this in https://github.com/ghuntley/gitpod-dotnet-csharp-breakpoints.

The specified task executable location "/run/containerd/io.containerd.runtime.v2.task/k8s.io/d2b9c1e9b814de15f8b189a4b78844dcb296adaa3adc1ef14f3d69ee3fe84871/rootfs/home/gitpod/dotnet/dotnet" is invalid.

The dotnet application is installed in /home/gitpod/dotnet/dotnet.

[akosyakov]

Also reported by the community:

• https://community.gitpod.io/t/cannot-run-dotnet-tool-restore/4388
• https://community.gitpod.io/t/frequent-permission-denied-when-using-dotnet-core/1473

[howard-pinsley-lab49]

We're impacted by this bug as well.

[csweichel]

I'm absolutely amazed to see dotnet/csharp digging this path out. It's something that should not necessarily be possible.

Either way, this is not so much a regression in containerd but most likely a bug in dotnet, or our own workspacekit. I have yet to find out where this path is coming from - I'm looking around https://github.com/dotnet/roslyn/blob/1cca63b5d8ea170f8d8e88e1574aa3ebe354c23b/src/Compilers/Core/MSBuildTask/Utilities.cs#L174-L176 right now. Does anyone know this codebase well enough to be able to point me to where those paths come from?

[csweichel]

I suspect it's the Assembly.Location used to determine the location of the Utilities, which in turn is used to try and find the dotnet binary.

I don't expect the code that implements this location bit to be open source, no?

I've seen there's a DOTNET_HOST_PATH env var that could be used to override this errornous detection behaviour.

[csweichel]

/schedule

[csweichel]

/assign

[ghuntley]

I don't expect the code that implements this location bit to be open source, no?

It is indeed open-source under MIT over at https://github.com/dotnet/runtime.

Does anyone know this codebase well enough to be able to point me to where those paths come from?

I've got some contacts, going to ask. Park for 12 hours? Twooted at https://twitter.com/GeoffreyHuntley/status/1423561511285362690

Also going to generate a https://msbuildlog.com/ support bundle and attach shortly.

[ghuntley]

Attached you will find a msbuild binlog:

• debug.binlog.zip

Here's how you configure vscode to generate them @csweichel:

[ghuntley]

From https://live.msbuildlog.com/ and the attached binlog:

[ghuntley]

If you want to repro this then click this link https://gitpod.io/#https://github.com/ghuntley/gitpod-dotnet-csharp-breakpoints

[ghuntley]

Setting DOTNET_HOST_PATH does not work; the value is not considered and the containerd path is used.

gitpod /workspace/gitpod-dotnet-csharp-breakpoints $ export DOTNET_HOST_PATH="/home/gitpod/dotnet/dotnet"
gitpod /workspace/gitpod-dotnet-csharp-breakpoints $ dotnet build
Microsoft (R) Build Engine version 16.9.0+57a23d249 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
/home/gitpod/dotnet/sdk/5.0.205/Roslyn/Microsoft.CSharp.Core.targets(71,5): error MSB6004: The specified task executable location "/run/containerd/io.containerd.runtime.v2.task/k8s.io/07774a84e8446c0d66b09e1941c8cf218b849eebc930da6888a0f814ef482951/rootfs/home/gitpod/dotnet/dotnet" is invalid. [/workspace/gitpod-dotnet-csharp-breakpoints/BreakpointTest.csproj]

Build FAILED.

/home/gitpod/dotnet/sdk/5.0.205/Roslyn/Microsoft.CSharp.Core.targets(71,5): error MSB6004: The specified task executable location "/run/containerd/io.containerd.runtime.v2.task/k8s.io/07774a84e8446c0d66b09e1941c8cf218b849eebc930da6888a0f814ef482951/rootfs/home/gitpod/dotnet/dotnet" is invalid. [/workspace/gitpod-dotnet-csharp-breakpoints/BreakpointTest.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:02.46

v5.0.205 of .NET

gitpod /workspace/gitpod-dotnet-csharp-breakpoints $ dotnet --version
5.0.205

[akoeplinger]

Setting DOTNET_HOST_PATH explicitly doesn't work because it's overridden here: https://github.com/dotnet/sdk/blob/2c011f2aa7a91a386430233d5797452ca0821ed3/src/Cli/Microsoft.DotNet.Cli.Utils/MSBuildForwardingAppWithoutLogging.cs#L44

.. and GetDotnetPath() eventually calls into
https://github.com/dotnet/sdk/blob/a30e465a2e2ea4e2550f319a2dc088daaafe5649/src/Cli/Microsoft.DotNet.Cli.Utils/Muxer.cs#L39

Can you check what Process.GetCurrentProcess().MainModule.FileName prints in a simple console app? If I followed the code correctly that gets read from /proc/<pid>/exe

[csweichel]

I've modified @ghuntley's example like this:

Console.WriteLine("Hello {0}!", System.Diagnostics.Process.GetCurrentProcess().MainModule.FileName);

Because of this very issue I could not build it in the workspace directly.
However, in a Docker container that worked no problem:

docker run --rm -it -v $PWD:/app mcr.microsoft.com/dotnet/sdk:5.0 bash -c "cd /app && dotnet publish -c release --self-contained --runtime linux-x64"`

The result is as expected:

[ghuntley]

strace(80) available at https://gist.github.com/ghuntley/83c981cc9c038a52a447753c5ccb55e2

A larger strace(365) available at https://gist.github.com/ghuntley/2eb8a54146c4882c0ada767d3e1b346c

$ strace -s 80 -f dotnet build > strace.log 2>&1
$ ack containerd strace.log 
read(7, "48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
read(7, "48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2635] read(14, "000 r--p 00000000 00:1b9f 4396653                  /run/containerd/io.containerd"..., 1024) = 1024
[pid  2635] read(14, "n/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef4569"..., 1024) = 1024
[pid  2635] read(14, "6e29a2000 ---p 001e7000 00:1b9f 4396346                  /run/containerd/io.cont"..., 1024) = 1024
[pid  2635] read(14, "f 3616988                  /run/containerd/io.containerd.runtime.v2.task/k8s.io/"..., 1024) = 1024
[pid  2635] read(14, "2cf1000 r--p 001cf000 00:1b9f 3617058                  /run/containerd/io.contai"..., 1024) = 1024
[pid  2635] read(14, "003000 00:1b9f 4396387                  /run/containerd/io.containerd.runtime.v2"..., 1024) = 1024
[pid  2635] read(14, "000 r--p 00000000 00:1b9f 4396653                  /run/containerd/io.containerd"..., 1024) = 1024
[pid  2635] read(14, "n/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef4569"..., 1024) = 1024
[pid  2635] read(14, "6e29a2000 ---p 001e7000 00:1b9f 4396346                  /run/containerd/io.cont"..., 1024) = 1024
[pid  2635] read(14, "f 3616988                  /run/containerd/io.containerd.runtime.v2.task/k8s.io/"..., 1024) = 1024
[pid  2635] read(14, "2cf1000 r--p 001cf000 00:1b9f 3617058                  /run/containerd/io.contai"..., 1024) = 1024
[pid  2635] read(14, "003000 00:1b9f 4396387                  /run/containerd/io.containerd.runtime.v2"..., 1024) = 1024
[pid  2635] read(17, "48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2635] read(17, "48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2635] read(95, "rd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef45692ffef0052b"..., 4096) = 3990
[pid  2635] <... read resumed>"0:1b9f 7754111                  /run/containerd/io.containerd.runtime.v2.task/k8"..., 4096) = 4096
[pid  2635] <... read resumed>"/run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef4"..., 4096) = 4096
[pid  2635] <... read resumed>"184                  /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002"..., 4096) = 4096
[pid  2635] <... read resumed>"/run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef4"..., 4096) = 4096
[pid  2635] read(95, " /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef"..., 4096) = 4029
[pid  2635] <... read resumed>"      /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69"..., 4096) = 4044
[pid  2635] <... read resumed>"123                  /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002"..., 4096) = 4096
[pid  2635] <... read resumed>" /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef"..., 4096) = 4096
[pid  2635] <... read resumed>" 00:1b9f 4396151                  /run/containerd/io.containerd.runtime.v2.task/"..., 4096) = 803
[pid  2654] read(7, "48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2654] <... read resumed>"48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2654] <... read resumed>"000 r--p 00000000 00:1b9f 4396653                  /run/containerd/io.containerd"..., 1024) = 1024
[pid  2654] <... read resumed>"n/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef4569"..., 1024) = 1024
[pid  2654] <... read resumed>"c2def4000 ---p 001e7000 00:1b9f 4396346                  /run/containerd/io.cont"..., 1024) = 1024
[pid  2654] <... read resumed>"f 3616988                  /run/containerd/io.containerd.runtime.v2.task/k8s.io/"..., 1024) = 1024
[pid  2654] <... read resumed>"e243000 r--p 001cf000 00:1b9f 3617058                  /run/containerd/io.contai"..., 1024) = 1024
[pid  2654] <... read resumed>"003000 00:1b9f 4396387                  /run/containerd/io.containerd.runtime.v2"..., 1024) = 1024
[pid  2654] <... read resumed>"000 r--p 00000000 00:1b9f 4396653                  /run/containerd/io.containerd"..., 1024) = 1024
[pid  2654] <... read resumed>"n/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef4569"..., 1024) = 1024
[pid  2654] <... read resumed>"c2def4000 ---p 001e7000 00:1b9f 4396346                  /run/containerd/io.cont"..., 1024) = 1024
[pid  2654] <... read resumed>"f 3616988                  /run/containerd/io.containerd.runtime.v2.task/k8s.io/"..., 1024) = 1024
[pid  2654] read(17, "e243000 r--p 001cf000 00:1b9f 3617058                  /run/containerd/io.contai"..., 1024) = 1024
[pid  2654] <... read resumed>"003000 00:1b9f 4396387                  /run/containerd/io.containerd.runtime.v2"..., 1024) = 1024
[pid  2654] read(18, "48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2654] <... read resumed>"48 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/c489f09f69190bb17"..., 1024) = 1024
[pid  2654] <... read resumed>"run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef45"..., 4096) = 4096
[pid  2654] <... read resumed>"000-7f4bb4e62000 rw-p 00000000 00:1b9f 7754102                  /run/containerd/"..., 4096) = 4096
[pid  2654] <... read resumed>"  /run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774e"..., 4096) = 4096
[pid  2677] stat("/run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef45692ffef0052bcb82b082ebf2ac2d94fe2f2e/rootfs/home/gitpod/dotnet/dotnet",  <unfinished ...>
[pid  2677] lstat("/run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef45692ffef0052bcb82b082ebf2ac2d94fe2f2e/rootfs/home/gitpod/dotnet/dotnet", 0x7f372fffcdd0) = -1 ENOENT (No such file or directory)
/home/gitpod/dotnet/sdk/5.0.205/Roslyn/Microsoft.CSharp.Core.targets(71,5): error MSB6004: The specified task executable location "/run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef45692ffef0052bcb82b082ebf2ac2d94fe2f2e/rootfs/[pid  2680] <... sched_yield resumed>)  = 0
[pid  2677] sched_yield(/home/gitpod/dotnet/sdk/5.0.205/Roslyn/Microsoft.CSharp.Core.targets(71,5): error MSB6004: The specified task executable location "/run/containerd/io.containerd.runtime.v2.task/k8s.io/3d4002b8437b0021f2b69774ef45692ffef0052bcb82b082ebf2ac2d94fe2f2e/rootfs/ <unfinished ...>

[csweichel]

It would seem that mountinfo is incorrect as is /sys. The latter would explain why I was able to run dotnet build in a Docker container in the workspace: the Docker container gets a proper sysfs mount, the workspace ring2 doesn't.

I'll file an issue.

[lawrencetaur]

I'm facing the same issue with dotnet build on #5191 Is there a temporary work around we can work with until this is fixed?

[mrsimonemms]

I'm facing the same issue with dotnet build on #5191 Is there a temporary work around we can work with until this is fixed?

@lawrencetaur You can run it inside a Docker container in the workspace. For example:

docker run -it --rm -w /opt/app -v $PWD:/opt/app mcr.microsoft.com/dotnet/sdk:5.0 dotnet build

[thepherm]

docker run -it --rm -w /opt/app -v $PWD:/opt/app mcr.microsoft.com/dotnet/sdk:5.0 dotnet build

This works thanks!

[bdescalzo]

Is there any news related to the fix of this bug? Thanks!

[csweichel]

The root cause is this issue: #5090

I'll close this one in favour of #5090