Using EF & SQLite crashes in alpine asp.net core 2.2 (but not 2.1) runtime docker image with Segmentation Fault (aka silently)

[smuellener]

Segmentation fault when using EF Core and Sqlite in official alpine 2.2 runtime docker image from microsoft. Can be easily reproduced with a standard example project from microsoft.

Steps to reproduce the issue

1. clone repro repo https://github.com/smuellener/SqliteSegmentationFaultAlpineDetCoreApp2_2.git
2. dotnet publish -c Release -v n -o ./obj/Docker/publish SqliteTestApp.csproj
3. docker build -t sqlitetestapp .
4. docker run -it sqlitetestapp /bin/sh
5. dotnet SqliteTestApp.dll

Expected behavior

Application runs without Segmentation Fault crash

Actual behavior

Segmentation Fault crash

Additional information

• A standard example project from microsoft is used for reproduction: https://docs.microsoft.com/en-us/ef/core/get-started/netcore/new-db-sqlite
• Built using Current SDK version (2.2.103)
• Does not Segmentation Fault crash when microsoft/dotnet:2.1-aspnetcore-runtime-alpine is used

Output of docker version

Client: Docker Engine - Community
Version: 18.09.1
API version: 1.39
Go version: go1.10.6
Git commit: 4c52b90
Built: Wed Jan 9 19:34:26 2019
OS/Arch: windows/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.1
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 4c52b90
Built: Wed Jan 9 19:41:49 2019
OS/Arch: linux/amd64
Experimental: false

[bricelam]

Confirmed. Just using Microsoft.Data.Sqlite seems to work fine, but something about EF Core is causing the process to terminate early.

The last things logged (by both EF Core and SQLite) are:

DetectChanges completed for 'BloggingContext'.
BEGIN IMMEDIATE;

[marius1]

I have same/similar issue. Not using EF just Microsoft.Data.Sqlite. I can read from the database but once the application starts writing is crashes.

For now moved away from the alpine image which seams to work fine.

[kristianhald]

We are using Sqlite for our automated tests using Docker with microsoft/dotnet:2.2-sdk-alpine.
The test runner crashes on tests using Sqlite, without providing any information about the error.

Same tests work fine with microsoft/dotnet:2.1-sdk-alpine.

[bricelam]

@ericsink this looks like a problem (or missing dependency) with e_sqlite3 on the microsoft/dotnet:2.2-sdk-alpine docker image. If I switch to SQLitePCLRaw.provider.sqlite3.netstandard11 (and apk add sqlite-dev) it works. Here is a simple repro using only SQLitePCL.raw.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.2</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="SQLitePCLRaw.bundle_green" Version="1.1.11"/>
  </ItemGroup>

</Project>

using System;
using System.Diagnostics;
using static SQLitePCL.raw;

class Program
{
    static void Main()
    {
        SQLitePCL.Batteries_V2.Init();

        Console.WriteLine("Opening database...");

        var rc = sqlite3_open("test.sqlite3", out var db);
        Debug.Assert(rc == SQLITE_OK);

        Console.WriteLine("Creating schema...");

        rc = sqlite3_exec(db, "CREATE TABLE IF NOT EXISTS test(value)");
        Debug.Assert(rc == SQLITE_OK);

        Console.WriteLine("Done");
    }
}

Output:

# dotnet run
Opening database...
Creating schema...
# _

[ericsink]

Quick reply to acknowledge I've seen this:

First I'll take a look at the alpine-specific parts and see if I can find something that looks wrong.

I don't currently have anything setup for testing that configuration, but I can pursue that if I need to.

[ericsink]

So it's broken on 2.2 but not 2.1. Do you know what SQLitePCLRaw versions correspond to those versions?

From your example above, it looks like it's broken in my 1.1.11.

I'm trying to figure out in which version of SQLitePCLRaw the alpine linux build was last working. If ever. But I assume it was at one point, as I recall somebody reporting so.

[bricelam]

It worked on the older 2.1 Docker images (e.g. microsoft/dotnet:2.1-sdk-alpine), but broke in the 2.2 images. They may have removed or updated some dependency.

@divega Do we have a contact for the Docker images that can also take a look?

[kristianhald]

We are using the NuGet package for getting SQLite (can’t remember which and version) and version is kept.
The only thing we did change was changing our Docker image from alpine-2.1-sdk to alpine-2.2-sdk.

[ajcvickers]

@glennc We think this is likely an issue with the 2.2 Alpine docker image. Do you know who would be a good person to contact about this?

[ajcvickers]

@richlander This is the issue I mentioned. It looks like it is a issue with the alpine docker image for 2.2 that could be a regression from 2.1. Can you recommend someone to help investigate?

[richlander]

Do these libraries depend on any native code (besides .NET Core itself)?

/cc @MichaelSimons @janvorli

[ajcvickers]

Yes, they depend on https://www.nuget.org/packages/SQLitePCLRaw.core/ which pulls in the native SQLite library.

[richlander]

If it depends on glibc, then it probably won't work. Alpine uses musl. Why the scenario worked in 2.1 and not 2.2, I don't know. Has anyone run the scenario under a native debugger to at least produce a good error message?

[ericsink]

Those SQLite builds are specifically built for musl.

[janvorli]

Let me try to repro it locally to see where it crashes.

[MichaelSimons]

It looks like the source of the regression is related to the version of Alpine being used. Note that 2.1 .NET image ships on Alpine 3.7 while 2.2 ships on Alpine 3.8. I did the following quick test with the provided repro steps. The seg fault does not occure with alpine:3.7 but does with alpine:3.8

FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj .
RUN dotnet restore

# copy everything else and build app
COPY . .
RUN dotnet publish -c Release -o out


#FROM alpine:3.7
FROM alpine:3.8

RUN apk add --no-cache \
        ca-certificates \
        \
        # .NET Core dependencies
        krb5-libs \
        libgcc \
        libintl \
        libssl1.0 \
        libstdc++ \
        lttng-ust \
        tzdata \
        userspace-rcu \
        zlib

# Configure web servers to bind to port 80 when present
ENV ASPNETCORE_URLS=http://+:80 \
    # Enable detection of running in a container
    DOTNET_RUNNING_IN_CONTAINER=true \
    # Set the invariant mode since icu_libs isn't included (see https://github.com/dotnet/announcements/issues/20)
    DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true

# Install .NET Core
ENV DOTNET_VERSION 2.2.3

RUN wget -O dotnet.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Runtime/$DOTNET_VERSION/dotnet-runtime-$DOTNET_VERSION-linux-musl-x64.tar.gz \
    && dotnet_sha512='b11e8731dd2e6b8738fb3a2762ed90de08df6661a8720ed76ef9429b99d763d0913ee100042a2995d72a13b50394a7e357397cecb23402c3104075efda04f62b' \
    && echo "$dotnet_sha512  dotnet.tar.gz" | sha512sum -c - \
    && mkdir -p /usr/share/dotnet \
    && tar -C /usr/share/dotnet -xzf dotnet.tar.gz \
    && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet \
    && rm dotnet.tar.gz

WORKDIR /app
COPY --from=build /app/out ./
#ENTRYPOINT ["dotnet", "SqliteTestApp.dll"]

[janvorli]

I've tried to run the repro as described at the beginning of this github issue. It has crashed in /app/runtimes/linux-x64/native/libe_sqlite3.so. Looking at the library dependencies, it is obvious that it was built for glibc and not musl:

scanelf -n /app/runtimes/linux-x64/native/libe_sqlite3.so
 TYPE   NEEDED FILE
ET_DYN libc.so.6 /app/runtimes/linux-x64/native/libe_sqlite3.so

[janvorli]

It seems that the issue may be caused by the fact that the alpine.3.8 rid is not present in the shared/Microsoft.NETCore.App/2.2.1/Microsoft.NETCore.App.deps.json in the SDK.

[ericsink]

Oh my. If that lib was built for libc, does that mean I screwed up the musl build? If so, I'm wondering how this ever worked.

Or wait -- looking at your latest comment, is it that the wrong lib is somehow getting included?

[janvorli]

@ericsink - I don't think it is your fault. Modifying the /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.1/Microsoft.NETCore.App.deps.json in the docker image by adding the alpine.3.8 rid fixes the crash. Now the libe_sqlite3.so is correctly loaded from /app/runtimes/alpine-x64/native/libe_sqlite3.so (note that without that change, it was being taken from /app/runtimes/linux-x64/native/libe_sqlite3.so

[MichaelSimons]

Related to https://github.com/dotnet/corefx/issues/34316 which added Alpine 3.8 and 3.9 to the rid graph.

[ericsink]

Ah, this is making more sense now. Thanks.

[ajcvickers]

@MichaelSimons Since that was fixed in 2.2.3, should this issue be fixed by using the latest 2.2 SDK and runtime?

[MichaelSimons]

I was thinking it should but I see the Microsoft.NETCore.App/2.2.3/Microsoft.NETCore.App.deps.json doesn't include alpine 3.8 or 3.9. @joperezr - can you answer why?

[ericstj]

That would be determined by the version of Microsoft.NETCore.Platforms used when building the shared framework in core-setup.

2.1 is using 2.1.3
https://github.com/dotnet/core-setup/blob/9013a2af4bec2618af36aab7edbbdd9a52403597/dependencies.props#L19

2.2 is using 2.2.0
https://github.com/dotnet/core-setup/blob/f95848e5243fafee7e535d6db4f8a7d227346cc3/dependencies.props#L19

If I crack open the 2.2.0 package it doesn't have alpine3.8 or 3.9, it was built prior to those changes. It looks to me like a new version of Microsoft.NETCore.Platforms was never shipped from release/2.2 after it's initial release in November. @wtgodbe do you know if anything is tracking updating the platforms package in release/2.2?

[wtgodbe]

I didn't realize that we'd shipped a version of Platforms out of both 2.1 & 2.2, that seems wrong. I was under the impression we were only shipping it out of 2.1 - we're shipping another new version of it with the next release of 2.1: https://github.com/dotnet/corefx/blob/release/2.1/src/packages.builds#L27-L29. Am I misunderstanding something about Platforms that would give it a different policy than Library packages? If so, then perhaps we would want to ship it out of both branches. Otherwise the intention should be for consumers to use the latest 2.1 version of that package

[ajcvickers]

@wtgodbe @ericstj @danmosemsft @livarcocc Looks like the root cause of this is below EF. Where would be a good place to continue tracking this issue?

[wtgodbe]

The solution would be either to ship the package out of both 2.x branches in CoreFx, or to have Core-Setup 2.2 consume the latest 2.1 version of the package. Therefore we should probably track it in CoreFx, I'll open an issue there & close this one.

[wtgodbe]

https://github.com/dotnet/core-setup/issues/5562

(I can't close issues in this repo, so I'll leave that to someone who has permissions)

[ajcvickers]

Thanks @wtgodbe

[WeihanLi]

Any solution here?
I use asp.net core 2.1 alpine based image, also crashed microsoft/dotnet:2.1-aspnetcore-runtime-alpine
you can get my project here for confirm or test

[ajcvickers]

@WeihanLi This issue has been closed as "external". If you want to continue to follow the progress on or add information to this issue, then please follow the link above to the new location.

If you would like to report a different problem, then please file a new issue and include a small, runnable project/solution or complete code listing that demonstrates the behavior you are seeing.