Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alpine 3.13 image is crashing on dotnet test #24189

Closed
ChaosEngine opened this issue Feb 18, 2021 · 39 comments
Closed

Alpine 3.13 image is crashing on dotnet test #24189

ChaosEngine opened this issue Feb 18, 2021 · 39 comments

Comments

@ChaosEngine
Copy link

Describe the Bug

Running dotnet test on Alpine 3.13 docker based image crashes test runner.

When running on mcr.microsoft.com/dotnet/sdk:5.0-alpine3.12 it succeeds
When running on mcr.microsoft.com/dotnet/sdk:5.0-alpine it fails

Steps to Reproduce

Dockerized repro based on https://github.com/dotnet/EntityFramework.Docs (https://github.com/dotnet/EntityFramework.Docs/tree/master/samples/core/Miscellaneous/Testing/ItemsWebApi/Tests specifically)

Start docker sdk alpine image
docker run -it --rm mcr.microsoft.com/dotnet/sdk:5.0-alpine

and then. execute this in interactive terminal..

#working project dir
cd /srv/
#I just need proper ll, c'mon
alias ll='ls -lah --color'
#usefull debugging and testing stuff
apk add --update vim strace gdb

#our repo to reproduce
git clone https://github.com/dotnet/EntityFramework.Docs.git

#working dir with tests
cd /srv//EntityFramework.Docs/samples/core/Miscellaneous/Testing/ItemsWebApi/Tests/

#build for restoring packages, simple.
dotnet build
#now tests - this crashes!!!!
dotnet test

#deeper investigation?
#strace dotnet test --logger trx --results-directory _temp -v m --blame-crash
#vim -p _temp/*.trx _temp/*/*.xml

Other Information

The same stuff when run with docker run -it --rm mcr.microsoft.com/dotnet/sdk:5.0-alpine works and just properly fails saying LocalDB is not avaialblre under Linux :-)

Output of docker version

Client:
 Version:           19.03.15
 API version:       1.40
 Go version:        go1.15.7
 Git commit:        420b1d3625
 Built:             Thu Feb  4 20:43:34 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.15
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.15.7
  Git commit:       420b1d3625
  Built:            Thu Feb  4 20:43:19 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.9
  GitCommit:        ea765ab
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683b971d9c3ef73f284f176672c44b448662

Output of docker info

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 40
 Server Version: 19.03.15
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ea765ab
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683b971d9c3ef73f284f176672c44b448662
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-gentoo
 Operating System: Gentoo/Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.29GiB
 Name: chaos2
 ID: OUE5:YKZ7:KCDZ:CTG3:BDT2:KLUH:7BW4:FRYG:47XX:25NZ:F5AL:Y67R
 Docker Root Dir: /mnt/samsung512GB/var/lib/docker
 Debug Mode: false
 Username: chaosengine
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
@mthalman
Copy link
Member

I've confirmed it's a segfault crash. It occurs when calling context.Database.EnsureCreated() when using a file-based connection for SQLite. So it seems to be related to Entity Framework or native implementation of SQLite. I saw that the db file does get created on the file system.

I've created a simple repro project that doesn't involve dotnet test. The crux of the repro is here:

using (var context = new ItemsContext(options))
{
  context.Database.EnsureCreated();
}

The call to EnsureCreated is what causes the crash.

Repro steps

  1. Unzip repro.zip to new folder
  2. docker build -t app .
  3. docker run --rm -it app

I've also collected a crash dump:
core.zip

@mthalman
Copy link
Member

Moving to efcore repo for further investigation.

@mthalman mthalman transferred this issue from dotnet/dotnet-docker Feb 18, 2021
@ajcvickers
Copy link
Member

@ericsink @bricelam Is SQLite expected to work on Alpine Linux?

@ericsink
Copy link

Yes, it is supposed to work on Alpine linux. It seems like it breaks 1-2 times per year, usually due to something in the environment changing, IIRC.

@ajcvickers
Copy link
Member

@ericsink Is there anything we can do help understand the issue on Alpine? I don't think there is anything we can change on our end, unless I'm missing something.

@ericsink
Copy link

ericsink commented Mar 1, 2021

Based on what I saw above, it looks like everything succeeded with Alpine docker image 3.12 and then failed with 3.13. So I had been assuming something changed in between that caused the break.

One possibility I have not investigated yet is that maybe "the thing that changed" would require me to build the alpine sqlite library differently, such as, for example, with a more recent compiler or C lib, etc.

I'll try the repro above and see if it yields any clues.

@ajcvickers
Copy link
Member

Thanks @ericsink

@AndriySvyryd AndriySvyryd added this to the 6.0.0 milestone Mar 1, 2021
@ericsink
Copy link

ericsink commented Mar 1, 2021

I tried the repro.zip above and it didn't fail. On the possibility that I did something wrong, one of my coworkers tried it, and no crash for them either.

Still quite possible we're doing the repro incorrectly somehow.

Can someone point me to a list of things that changed in the docker image between 3.12 and 3.13 ?

@mthalman
Copy link
Member

mthalman commented Mar 1, 2021

I tried the repro.zip above and it didn't fail. On the possibility that I did something wrong, one of my coworkers tried it, and no crash for them either.

Did it successfully output created in the console? If not, then it crashed.

@ericsink
Copy link

ericsink commented Mar 1, 2021

Ah, I see. No, it did not output created.

@ericsink
Copy link

ericsink commented Mar 1, 2021

Not sure, but perhaps something in #14504 would be helpful here.

@ChaosEngine
Copy link
Author

I tried the repro.zip above and it didn't fail. On the possibility that I did something wrong, one of my coworkers tried it, and no crash for them either.

Still quite possible we're doing the repro incorrectly somehow.

Can someone point me to a list of things that changed in the docker image between 3.12 and 3.13 ?

Sure thing, I went this path also, and found this comparison:
https://pkgs.alpinelinux.org/packages?name=sqlite&branch=v3.13
vs
https://pkgs.alpinelinux.org/packages?name=sqlite&branch=v3.12

@ericsink
Copy link

ericsink commented Mar 1, 2021

Oh. I didn't realize the 3.12 and 3.13 referred to versions numbers of Alpine Linux itself.

@ChaosEngine FWIW, the links you gave appear to refer to Alpine's sqlite package, which is probably not involved here, but I cannot say for certain. The most likely case here is that my e_sqlite3 build for linux/musl/alpine is somehow incompatible with 3.13.

@ericsink
Copy link

ericsink commented Mar 1, 2021

More info, in case it helps anyone recognize a problem:

My e_sqlite3 builds for alpine have always been built on a regular ubuntu system using the package musl-dev. The actual command I use to build is musl-gcc.

eric@LAPTOP-6A6HQVJ9:/mnt/c/Users/eric/dev/cb/bld$ musl-gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609

My current version of ubuntu for this build is 16.04 xenial.

My current version of the musl-dev package is 1.1.9-1. apt update does not indicate an available upgrade for this package.

A current question is whether this build configuration results in things that are compatible with alpine 3.12 but not with alpine 3.13.

@mthalman
Copy link
Member

mthalman commented Mar 1, 2021

My current version of the musl-dev package is 1.1.9-1. apt update does not indicate an available upgrade for this package.

A current question is whether this build configuration results in things that are compatible with alpine 3.12 but not with alpine 3.13.

Alpine 3.13 was upgraded to musl 1.2 and also had some changes around time64. I wonder if either of those two things are related. See the Alpine 3.13 release notes.

@ChaosEngine
Copy link
Author

FWIW, my alpine3.13 dotnet-dump core dump analysis gave me those unmanaged suspects (based on repro.zip):

> clrstack
OS Thread Id: 0x374 (0)
        Child SP               IP Call Site
00007FFE38176CF0 000000000000e5c6 [InlinedCallFrame: 00007ffe38176cf0]
00007FFE38176CF0 00007fab608d71e0 [InlinedCallFrame: 00007ffe38176cf0]
00007FFE38176CE0 00007FAB608D71E0 <unknown method>
00007FFE38176D80 00007FAB60BDAE8C /srv/app/bin/Debug/net5.0/SQLitePCLRaw.provider.dynamic_cdecl.dll!Unknown
00007FFE38176DB0 00007FAB60BD95C4 /srv/app/bin/Debug/net5.0/Microsoft.Data.Sqlite.dll!Unknown
00007FFE38176DF0 00007FAB60BD9159 /srv/app/bin/Debug/net5.0/Microsoft.Data.Sqlite.dll!Unknown
00007FFE38176EF0 00007FAB60BD8D71 /srv/app/bin/Debug/net5.0/Microsoft.Data.Sqlite.dll!Unknown
00007FFE38176F10 00007FAB60BD8CFB /srv/app/bin/Debug/net5.0/Microsoft.Data.Sqlite.dll!Unknown
00007FFE38176F70 00007FAB60BD55E5 /srv/app/bin/Debug/net5.0/Microsoft.EntityFrameworkCore.Relational.dll!Unknown
00007FFE381770D0 00007FAB60BD322E /srv/app/bin/Debug/net5.0/Microsoft.EntityFrameworkCore.Sqlite.dll!Unknown
00007FFE38177170 00007FAB608C4848 /srv/app/bin/Debug/net5.0/Microsoft.EntityFrameworkCore.Relational.dll!Unknown
00007FFE381771B0 00007FAB603E052D /srv/app/bin/Debug/net5.0/Microsoft.EntityFrameworkCore.dll!Unknown
00007FFE381771E0 00007FAB5F436877 /srv/app/bin/Debug/net5.0/app.dll!Unknown

@ChaosEngine
Copy link
Author

Also I've observed following when playing with vanilla libsqlite:

/srv/app # apk add sqlite-dev
/srv/app # cd /srv/app
/srv/app # dotnet build
Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /srv/app/app.csproj (in 618 ms).
  app -> /srv/app/bin/Debug/net5.0/app.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

/srv/app # /srv/app/bin/Debug/net5.0/app
Segmentation fault (core dumped)

/srv/app # cp /usr/lib/libsqlite3.so.0.8.6  /srv/app/bin/Debug/net5.0/runtimes/linux-x64/native/libe_sqlite3.so

/srv/app # /srv/app/bin/Debug/net5.0/app
created

Looks like when installed original libsqlite3 (musl compiled and alpine-3.13 build) and overwriting dotnet build linux-x64 native version of sqlite - the problem is gone, otherwise it is very reproducible.

@ericsink
Copy link

ericsink commented Mar 2, 2021

I'm going to try and get setup to build an e_sqlite3 actually under alpine 3.13.

@ericsink
Copy link

OK, I haven't gotten anywhere on this.

I have no experience with alpine linux, and very little experience with docker. I want to learn these things, but having my education in the critical path of this issue may not be a good idea.

I assume that if I just build sqlite on alpine 3.13 the resulting library would work.

Anybody want to give me just enough docker knowledge to run gcc under alpine 3.13 ?

I have wsl 2 installed, which I use regularly, but I don't see a wsl image for alpine 3.13 yet.

Feel free to just tell me to go back and do my homework, but if you want to move this bug along, telling me exactly what commands to type would probably do that. :-)

@mthalman
Copy link
Member

You could try these instructions to upgrade your WSL Alpine installation to 3.13. And here are instructions for for installing gcc on Alpine.

@ChaosEngine
Copy link
Author

Ok, looks like a homework for me :-) Will try to give this a shot during weekend (lockdown on my side so what else).

@ajcvickers
Copy link
Member

Many thanks to everyone helping out on this. Much appreciated. :-)

@ericsink
Copy link

Another thought I had:

Let's assume we find that e_sqlite3 for alpine 3.13 needs to be built on 3.13. I speculate that the resulting build will therefore not work on alpine before 3.13. So are we going to need to maintain two builds?

@ChaosEngine
Copy link
Author

ChaosEngine commented Mar 13, 2021

While doing the compilation I noticed something interesting.
When added runtime suffix for alpine (-r alpine-x64 )the repro.zip case worked:

/app # dotnet build -r alpine-x64
Microsoft (R) Build Engine version 16.9.0+57a23d249 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
  app -> /app/bin/Debug/net5.0/alpine-x64/app.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:04.07
/app # rm -rf bi^C
/app # rm -rf bin/ obj/ Test.db
/app # dotnet build -r alpine-x64
Microsoft (R) Build Engine version 16.9.0+57a23d249 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /app/app.csproj (in 874 ms).
  app -> /app/bin/Debug/net5.0/alpine-x64/app.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:10.50

/app # bin/Debug/net5.0/alpine-x64/app
created

also my original repro from the docs works with test project when run with
dotnet test -v m --runtime alpine-x64

Is it possible that default dotnet runtime on alpine3.13 is not detected properly somehow?
Digging deeper into this....:monocle_face:

@ChaosEngine
Copy link
Author

ChaosEngine commented Mar 13, 2021

It seems to be the same as reported here: dotnet/runtime#28303 (comment)

 $ docker run -it --rm -v $(pwd):/app --workdir /app mcr.microsoft.com/dotnet/sdk:5.0-alpine
/app # dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     alpine
 OS Version:  3.13
 OS Platform: Linux
 RID:         linux-musl-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.201/

RID: linux-musl-x64

$ docker run -it --rm -v $(pwd):/app --workdir /app mcr.microsoft.com/dotnet/sdk:5.0-alpine3.12
/app # dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     alpine
 OS Version:  3.12
 OS Platform: Linux
 RID:         alpine.3.12-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.201/

RID: alpine.3.12-x64
and this could be the imain issue here, app assembly not linking to proper native SQLite runtime???

I think this proves it. I run built app (/app/bin/Debug/net5.0/app) in gdb and force a crash, than showed loaded shared libraries.

/app # gdb /app/bin/Debug/net5.0/app

Type "apropos word" to search for commands related to "word"...
Reading symbols from /app/bin/Debug/net5.0/app...
(No debugging symbols found in /app/bin/Debug/net5.0/app)
(gdb) r
Starting program: /app/bin/Debug/net5.0/app
warning: Error disabling address space randomization: Operation not permitted
[New LWP 192]
[New LWP 193]
[New LWP 194]
[New LWP 195]
[New LWP 196]
[New LWP 197]
[New LWP 198]
[New LWP 199]
[New LWP 200]
[New LWP 201]

Thread 1 "app" received signal SIGSEGV, Segmentation fault.
0x000000000000e5c6 in ?? ()
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007f46227c4070  0x00007f462280b6e3  Yes (*)     /lib/ld-musl-x86_64.so.1
0x00007f46226cc820  0x00007f4622755cf1  Yes (*)     /usr/lib/libstdc++.so.6
0x00007f46225f62f0  0x00007f46226065b1  Yes (*)     /usr/lib/libgcc_s.so.1
0x00007f4622591840  0x00007f46225cc2b7  Yes (*)     /usr/share/dotnet/host/fxr/5.0.4/libhostfxr.so
0x00007f4622545900  0x00007f46225781a1  Yes (*)     /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.4/libhostpolicy.so
0x00007f4621dd1290  0x00007f462228daa4  Yes (*)     /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.4/libcoreclr.so
0x00007f458710aff0  0x00007f45872d3d34  Yes (*)     /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.4/libclrjit.so
0x00007f4586ac09c0  0x00007f4586ac899a  Yes (*)     /usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.4/libSystem.Native.so
0x00007f458696f480  0x00007f4586a17341  Yes (*)     /usr/lib/libicuuc.so.67
                                        Yes (*)     /usr/lib/libicudata.so.67
0x00007f4584c57560  0x00007f4584d63031  Yes (*)     /usr/lib/libicui18n.so.67
0x00007f45842c0d50  0x00007f458437b0f0  Yes (*)     /app/bin/Debug/net5.0/runtimes/linux-x64/native/libe_sqlite3.so
(*): Shared library is missing debugging information.

loading /app/bin/Debug/net5.0/runtimes/linux-x64/native/libe_sqlite3.so

@mthalman
Copy link
Member

Great investigative work, @ChaosEngine. Definitely looks related to the rid.

It's strange because I've confirmed that alpine.3.13 is defined as an available rid:

$ docker run -it --rm mcr.microsoft.com/dotnet/sdk:5.0-alpine
/ # cat ./usr/share/dotnet/sdk/5.0.201/RuntimeIdentifierGraph.json | grep alpine.3.13
    "alpine.3.13": {
    "alpine.3.13-arm": {
        "alpine.3.13",
    "alpine.3.13-arm64": {
        "alpine.3.13",
    "alpine.3.13-x64": {
        "alpine.3.13",

Adding @wfurt who added this rid to see if he has an idea here. @wfurt , take a look at the output shown here: #24189 (comment). Why would dotnet --info print out linux-musl-x64 for the rid on Alpine 3.13 while it prints out alpine.3.12-x64 on Alpine 3.12?

@wfurt
Copy link
Member

wfurt commented Mar 15, 2021

It is missing in Microsoft.NETCore.App/5.0.4/Microsoft.NETCore.App.deps.json Adding it there gives expected RID

/usr/share/dotnet/shared # dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.201
 Commit:    a09bd5c86c

Runtime Environment:
 OS Name:     alpine
 OS Version:  3.13
 OS Platform: Linux
 RID:         alpine.3.13-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.201/

@wfurt
Copy link
Member

wfurt commented Mar 15, 2021

cc: @Anipik @aik-jahoda . The sources looks ok to me.

@LiangZugeng
Copy link

@ajcvickers @wfurt Is there any plan (before 6.0) to add the alpine.3.13 RID into Microsoft.NETCore.App.deps.json? I tested the 5.0.7 SDK docker image and the RID is still incorrect.

/ # dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.301
 Commit:    ef17233f86

Runtime Environment:
 OS Name:     alpine
 OS Version:  3.13
 OS Platform: Linux
 RID:         linux-musl-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.301/

Host (useful for support):
  Version: 5.0.7
  Commit:  556582d964

.NET SDKs installed:
  5.0.301 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Could it be fixed in the next minor version of 5.0.7? So we won't get the "Test host process crashed" error in the container while running unit tests, our code is being upgraded from .NET Core 3.1 to .NET 5.0 and we would like to maintain the OS version unchanged in the docker image. Thanks.

@wfurt
Copy link
Member

wfurt commented Jul 12, 2021

Any idea @ericstj ? It seems like the RID does not propagate consistently.

@ektrah
Copy link
Member

ektrah commented Jul 12, 2021

(See also dotnet/runtime#50739)

@ericstj
Copy link
Member

ericstj commented Jul 12, 2021

@Anipik's change dotnet/runtime@95fe690 did this. My best guess is that the runtime consolidation in 5.0 missed picking up the live built runtime.json and is using some restored copy (or SDK copy). cc @jkoritzinsky @ViktorHofer

@wfurt
Copy link
Member

wfurt commented Jul 12, 2021

Thanks @ericstj. So if we make rid update to add 3.14 it would be missed as well?

@ericstj
Copy link
Member

ericstj commented Jul 12, 2021

I didn't check that, but you could by examining the tar.gz file. 3.x was done in a different way (due to separate repos) and underwent a lot more additions that I expect would have noticed this, so I suspect the bug is not there. Can't know for sure until someone root-causes the deps file generation on 5.0. cc @dagood

@ericstj
Copy link
Member

ericstj commented Jul 13, 2021

I think I misread your statement as .NETCore 3.x, but you meant an addition to of 3.14 to 5.0. There's probably a way to make it work correctly. I would ask @dagood or @jkoritzinsky as they know the most about this part of the shared framework creation process. It looks to me like the 5.0 tools relied on getting the runtime graph from some package that was restored as part of the project: https://github.com/dotnet/arcade/blob/7e1e1ff990487b0c74e0fb17b4169bc5378fe778/src/Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk/src/ProcessSharedFrameworkDeps.cs#L56-L57

@ericstj
Copy link
Member

ericstj commented Jul 13, 2021

Here's another 5.0 RID change: dotnet/runtime#48203 I wonder if that has the same problem?

@dagood
Copy link
Member

dagood commented Jul 13, 2021

It looks like the Microsoft.NETCore.Platforms version was pinned to 5.0.0 by dotnet/runtime#43182 and it hasn't been updated since then to match the live version, so it faithfully continues to use 5.0.0. It seems odd to me that it used 5.0.0 rather than $(ProductVersion), and that it was merged with a * NO MERGE * label active(explained in PR description).

The Microsoft.NETCore.Platforms 5.0.0 package name/version shows up in the deps.json file, so I don't think it'll be hard to spot for specific tar.gzs. And yeah, fedora.35-x64 is also missing in a tar.gz I pulled for 5.0.7.

/cc @dleeapho @NikolaMilosavljevic

@ericstj
Copy link
Member

ericstj commented Jul 13, 2021

Let's move this discussion to dotnet/runtime#50739. Let's make that a servicing issue and address it in the next release.

@ajcvickers ajcvickers removed this from the 6.0.0 milestone Jul 14, 2021
@UsamaKhalil-KSO
Copy link

faced the same issue today.

@ajcvickers ajcvickers reopened this Oct 16, 2022
@ajcvickers ajcvickers closed this as not planned Won't fix, can't repro, duplicate, stale Oct 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests